Skip to content

Commit eec2e5c

Browse files
ZStriker19ZStriker19
authored
Merge branch 'main' into zachg/llmobs_annotate_context_fix
2 parents dbbfd8e + f23eb40 commit eec2e5c

File tree

15 files changed

+998
-32
lines changed

15 files changed

+998
-32
lines changed

‎.claude/settings.local.json‎

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -35,8 +35,10 @@
3535
"mcp__github__get_pull_request_review_comments",
3636
"mcp__github__pull_request_read",
3737
"Skill(run-tests)",
38-
"Skill(lint)"
38+
"Skill(lint)",
39+
"Skill(find-cpython-usage)",
40+
"Skill(compare-cpython-versions)"
3941
],
4042
"deny": []
4143
}
42-
}
44+
}

‎.claude/skills/compare-cpython-versions/SKILL.md‎

Lines changed: 171 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,171 @@
1+
---
2+
name: compare-cpython-versions
3+
description: >
4+
Compare CPython source code between two Python versions to identify changes in
5+
headers and structs. Use this when adding support for a new Python version to
6+
understand what changed between versions.
7+
allowed-tools:
8+
- Bash
9+
- Read
10+
- Grep
11+
- WebSearch
12+
- TodoWrite
13+
---
14+
15+
# Compare CPython Versions Skill
16+
17+
This skill helps compare CPython source code between two Python versions to identify
18+
changes in headers, structs, and APIs that affect our codebase.
19+
20+
## When to Use This Skill
21+
22+
Use this skill when:
23+
- Adding support for a new Python version
24+
- Need to understand what changed between versions
25+
- Investigating compatibility issues
26+
- Have a list of headers/structs from `find-cpython-usage` skill
27+
28+
## Key Principles
29+
30+
1. **Compare systematically** - Focus on headers and structs identified in Step 1
31+
2. **Use multiple methods** - Git diff, manual diff, or AI-assisted comparison
32+
3. **Document changes** - Note all breaking changes and API modifications
33+
4. **Check context** - Understand why changes were made (PEPs, GitHub issues)
34+
35+
## How This Skill Works
36+
37+
### Step 1: Prepare CPython Repository
38+
39+
```bash
40+
# Create ~/dd directory if it doesn't exist
41+
mkdir -p ~/dd
42+
43+
# Clone CPython repository if needed (to ~/dd/cpython)
44+
if [ ! -d ~/dd/cpython ]; then
45+
git clone https://github.com/python/cpython.git ~/dd/cpython
46+
cd ~/dd/cpython
47+
git fetch --tags
48+
else
49+
cd ~/dd/cpython
50+
# Update existing repository
51+
git fetch --tags
52+
git fetch origin
53+
fi
54+
```
55+
56+
### Step 2: Compare Specific Headers
57+
58+
Using the list of headers from `find-cpython-usage`, compare each header between
59+
the old version and new version. Replace `OLD_VERSION` and `NEW_VERSION` with the
60+
actual version tags (e.g., `v3.13.0`, `v3.14.0`):
61+
62+
```bash
63+
# Compare specific headers between versions
64+
git diff OLD_VERSION NEW_VERSION -- Include/internal/pycore_frame.h
65+
git diff OLD_VERSION NEW_VERSION -- Include/frameobject.h
66+
67+
# Compare all internal headers
68+
git diff OLD_VERSION NEW_VERSION -- 'Include/internal/pycore*.h'
69+
70+
# Compare specific struct definitions
71+
git diff OLD_VERSION NEW_VERSION -- Include/internal/pycore_frame.h | grep -A 20 "struct _PyInterpreterFrame"
72+
```
73+
74+
75+
### Step 3: Identify Changes
76+
77+
For each header/struct, look for:
78+
79+
**Struct Changes:**
80+
- Field additions
81+
- Field removals
82+
- Field type changes
83+
- Field reordering
84+
- Struct moves to different headers
85+
86+
**API Changes:**
87+
- Removed functions/structures
88+
- New functions/structures
89+
- Changed function signatures
90+
- Deprecated APIs
91+
92+
**Header Changes:**
93+
- Headers moved to different locations
94+
- Headers split or merged
95+
- New headers introduced
96+
97+
### Step 4: Analyze Impact
98+
99+
For each change identified:
100+
101+
1. **Understand the change:**
102+
- Why was it changed? (Check Python's What's New, PEPs, or GitHub issues)
103+
- Is it a breaking change or backward compatible?
104+
- What's the replacement API?
105+
- **Find the specific commit(s) that introduced the change:**
106+
107+
```bash
108+
# Find commits that modified a specific file between versions
109+
git log OLD_VERSION..NEW_VERSION -- Include/internal/pycore_frame.h
110+
111+
# Find commits that mention a specific struct or function
112+
git log OLD_VERSION..NEW_VERSION --all --grep="_PyInterpreterFrame" -- Include/
113+
114+
# Show the commit that introduced a specific change
115+
git log -p OLD_VERSION..NEW_VERSION -S "struct _PyInterpreterFrame" -- Include/
116+
```
117+
118+
- **Find related GitHub issues:**
119+
- Check commit messages for issue references (e.g., `gh-123923`, `#123923`)
120+
- Search CPython GitHub issues: `https://github.com/python/cpython/issues`
121+
- Look for "What's New" documentation: `https://docs.python.org/3/whatsnew/`
122+
- Check PEPs if the change is part of a larger feature
123+
124+
2. **Assess impact:**
125+
- Which files in our codebase are affected?
126+
- What functionality might break?
127+
- Are there alternative approaches?
128+
129+
3. **Document findings:**
130+
- Create a summary document of key changes
131+
- Note any breaking changes
132+
- List files that need updates
133+
134+
### Step 5: Use AI Tools (Optional)
135+
136+
You can use AI coding assistants to help analyze differences by:
137+
- Providing header file contents from both versions
138+
- Asking about specific struct changes
139+
- Understanding migration paths
140+
141+
## Common Change Patterns
142+
143+
When comparing versions, look for these types of changes (examples):
144+
145+
**Struct Field Changes:**
146+
- Field type changes (e.g., pointer types → tagged pointer types)
147+
- Field renamed
148+
- Field removed and replaced with different mechanism
149+
- Field reordering
150+
151+
**Header Moves:**
152+
- Internal headers moved to new locations
153+
- Structs moved between headers
154+
- Headers split or merged
155+
156+
**API Deprecations:**
157+
- Internal functions removed
158+
- Public API replacements available
159+
- Function signature changes
160+
161+
## Output Format
162+
163+
After running this skill, you should have:
164+
1. A list of all changed headers
165+
2. A list of all changed structs with details
166+
3. Impact assessment for each change
167+
4. Files in our codebase that need updates
168+
169+
## Related
170+
171+
- **find-cpython-usage skill**: Use to identify what to compare

‎.claude/skills/find-cpython-usage/SKILL.md‎

Lines changed: 167 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,167 @@
1+
---
2+
name: find-cpython-usage
3+
description: >
4+
Find all CPython internal headers and structs used in the codebase, particularly
5+
for profiling functionality. Use this when adding support for a new Python version
6+
to identify what CPython internals we depend on.
7+
allowed-tools:
8+
- Bash
9+
- Read
10+
- Grep
11+
- Glob
12+
- TodoWrite
13+
---
14+
15+
# Find CPython Internal Usage Skill
16+
17+
This skill helps identify all CPython internal headers and structures used in the
18+
codebase, which is essential when adding support for new Python versions.
19+
20+
## When to Use This Skill
21+
22+
Use this skill when:
23+
- Adding support for a new Python version
24+
- Investigating CPython API dependencies
25+
- Understanding what internal APIs the profiler uses
26+
- Preparing to compare CPython versions
27+
28+
## Key Principles
29+
30+
1. **Focus on internal headers** - These are most likely to change between versions
31+
2. **Check all native extensions** - CPython internals are used in profiling, AppSec, and internal modules
32+
3. **Look for struct field access** - Direct field access is version-sensitive
33+
4. **Document findings** - Keep track of what you find for comparison
34+
35+
## How This Skill Works
36+
37+
### Step 1: Find CPython Header Includes
38+
39+
Search for CPython header includes across all C/C++/Cython files:
40+
41+
```bash
42+
# Find all CPython internal header includes
43+
grep -r "include.*internal/pycore" ddtrace/ --include="*.c" --include="*.cpp" --include="*.h" --include="*.hpp" --include="*.pyx"
44+
45+
# Find all CPython cpython header includes
46+
grep -r "include.*cpython" ddtrace/ --include="*.c" --include="*.cpp" --include="*.h" --include="*.hpp" --include="*.pyx"
47+
48+
# Find all Python.h includes (indicates CPython API usage)
49+
grep -r "#include.*Python\.h" ddtrace/ --include="*.c" --include="*.cpp" --include="*.h" --include="*.hpp" --include="*.pyx"
50+
51+
# Find frameobject.h includes (common CPython API)
52+
grep -r "#include.*frameobject\.h" ddtrace/ --include="*.c" --include="*.cpp" --include="*.h" --include="*.hpp" --include="*.pyx"
53+
54+
# Find PyO3 FFI usage in Rust (Rust extensions may use CPython internals)
55+
grep -r "pyo3_ffi::\|pyo3::ffi::" src/native/ --include="*.rs" || true
56+
```
57+
58+
**Note:** These patterns search across all native extension files (`.c`, `.cpp`, `.h`, `.hpp`, `.pyx`, `.rs`)
59+
regardless of their location in the codebase. Check `setup.py` to see which extensions are built.
60+
Rust extensions use PyO3 which may access CPython internals through the `pyo3_ffi` module.
61+
62+
### Step 2: Find Struct Field Access
63+
64+
Search for direct struct field access and struct definitions across all native files:
65+
66+
```bash
67+
# Find struct field accesses (arrow operator)
68+
grep -r "->f_\|->[a-z_]*\." ddtrace/ --include="*.c" --include="*.cpp" --include="*.h" --include="*.hpp"
69+
70+
# Find struct definitions
71+
grep -r "struct.*Py" ddtrace/ --include="*.c" --include="*.cpp" --include="*.h" --include="*.hpp"
72+
73+
# Find common CPython struct usage
74+
grep -r "PyFrameObject\|PyThreadState\|_PyInterpreterFrame" ddtrace/ --include="*.c" --include="*.cpp" --include="*.h" --include="*.hpp" --include="*.pyx"
75+
grep -r "PyFrameObject\|PyThreadState\|PyInterpreterState" src/native/ --include="*.rs" || true
76+
77+
# Find PyCodeObject usage
78+
grep -r "PyCodeObject" ddtrace/ --include="*.c" --include="*.cpp" --include="*.h" --include="*.hpp" --include="*.pyx"
79+
grep -r "PyCodeObject" src/native/ --include="*.rs" || true
80+
```
81+
82+
**Look for patterns like:**
83+
- Frame structure access (`PyFrameObject`, `_PyInterpreterFrame`)
84+
- Thread state access (`PyThreadState`)
85+
- Code object access (`PyCodeObject`)
86+
- Generator/coroutine structures
87+
- Asyncio task structures
88+
89+
### Step 3: Identify Common Structures
90+
91+
Common CPython structures we typically access:
92+
93+
**Frame structures:**
94+
- `PyFrameObject` / `struct _frame`
95+
- `_PyInterpreterFrame`
96+
97+
**State structures:**
98+
- `PyThreadState`
99+
- `PyInterpreterState`
100+
- `_PyRuntimeState`
101+
102+
**Code structures:**
103+
- `PyCodeObject`
104+
105+
**Generator structures:**
106+
- `PyGenObject`
107+
- `PyAsyncGenASend`
108+
109+
**Asyncio structures:**
110+
- `FutureObj`
111+
- `TaskObj`
112+
113+
### Step 4: Document Findings
114+
115+
Create a list of:
116+
- All headers that are included
117+
- All structs that are accessed
118+
- All struct fields that are used directly
119+
120+
This will be used in the next step to compare against the new Python version.
121+
122+
## Native Extensions Using CPython APIs
123+
124+
To find all native extensions that may use CPython APIs, check `setup.py`:
125+
126+
```bash
127+
# View all native extensions defined in setup.py
128+
grep -A 5 "Extension\|CMakeExtension\|Cython.Distutils.Extension\|RustExtension" setup.py
129+
```
130+
131+
The `setup.py` file defines all native extensions (C, C++, CMake, Cython, and Rust) that
132+
are built for the project. Not all extensions use CPython internals - focus on those
133+
that access frame objects, thread state, or internal structures when searching for
134+
CPython API usage.
135+
136+
**Note:** Rust extensions use PyO3 bindings which may access CPython internals through
137+
`pyo3_ffi` module. Search Rust source files (`.rs`) for CPython API usage as well.
138+
139+
## Common Headers to Look For
140+
141+
The grep commands above will identify which CPython headers are actually used in the codebase.
142+
Common patterns include:
143+
144+
**Public Headers:**
145+
- Headers matching `*.h` in `Include/` directory (e.g., `frameobject.h`, `unicodeobject.h`)
146+
- Headers in `Include/cpython/` directory (e.g., `cpython/genobject.h`)
147+
148+
**Internal Headers (require `Py_BUILD_CORE`):**
149+
- Headers matching `internal/pycore*.h` pattern
150+
- These are most likely to change between Python versions
151+
152+
Focus on headers that are actually found by the grep commands rather than maintaining
153+
a hardcoded list, as the headers used may change over time.
154+
155+
## Output Format
156+
157+
After running this skill, you should have:
158+
1. A list of all CPython headers included in the codebase
159+
2. A list of all CPython structs accessed
160+
3. A list of struct fields accessed directly
161+
4. Files that use each header/struct
162+
163+
This information can then be used with the `compare-cpython-versions` skill to identify what changed.
164+
165+
## Related
166+
167+
- **compare-cpython-versions skill**: Use findings from this skill to compare versions

‎.github/CODEOWNERS‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -83,7 +83,7 @@ ddtrace/internal/ci_visibility @DataDog/ci-app-libraries
8383
ddtrace/internal/test_visibility @DataDog/ci-app-libraries
8484
ddtrace/internal/codeowners.py @DataDog/apm-core-python @DataDog/ci-app-libraries
8585
ddtrace/internal/coverage @DataDog/apm-core-python @DataDog/ci-app-libraries
86-
tests/internal/test_codeowners.py @datadog/ci-app-libraries
86+
tests/internal/test_codeowners.py @DataDog/ci-app-libraries
8787
tests/ci_visibility @DataDog/ci-app-libraries
8888
tests/coverage @DataDog/apm-core-python @DataDog/ci-app-libraries
8989
tests/tracer/test_ci.py @DataDog/ci-app-libraries

0 commit comments

Comments
 (0)