-
Notifications
You must be signed in to change notification settings - Fork 475
fix(er): don't persist frames while snapshots are queued #15610
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
|
|
|
@P403n1x87 I need some help on an approach here, I tried: Since it seemed safe, but ended up breaking things. Any ideas? |
625bc6f to
0ea47e3
Compare
Bootstrap import analysisComparison of import times between this PR and base. SummaryThe average import time from this PR is: 256 ± 4 ms. The average import time from base is: 261 ± 4 ms. The import time difference between this PR and base is: -5.2 ± 0.2 ms. Import time breakdownThe following import paths have shrunk:
|
Performance SLOsComparing candidate tyler.finethy/DEBUG-4851-2 (e8d6258) with baseline main (38fdcc5) 📈 Performance Regressions (3 suites)📈 iastaspects - 118/118✅ add_aspectTime: ✅ 0.382µs (SLO: <10.000µs 📉 -96.2%) vs baseline: -0.7% Memory: ✅ 38.497MB (SLO: <41.500MB -7.2%) vs baseline: +4.9% ✅ add_inplace_aspectTime: ✅ 0.384µs (SLO: <10.000µs 📉 -96.2%) vs baseline: +0.8% Memory: ✅ 38.699MB (SLO: <41.500MB -6.7%) vs baseline: +4.8% ✅ add_inplace_noaspectTime: ✅ 0.291µs (SLO: <10.000µs 📉 -97.1%) vs baseline: +1.1% Memory: ✅ 38.424MB (SLO: <41.500MB -7.4%) vs baseline: +4.5% ✅ add_noaspectTime: ✅ 0.357µs (SLO: <10.000µs 📉 -96.4%) vs baseline: +1.2% Memory: ✅ 38.617MB (SLO: <41.500MB -6.9%) vs baseline: +5.0% ✅ bytearray_aspectTime: ✅ 1.307µs (SLO: <10.000µs 📉 -86.9%) vs baseline: -0.2% Memory: ✅ 38.675MB (SLO: <41.500MB -6.8%) vs baseline: +5.2% ✅ bytearray_extend_aspectTime: ✅ 1.441µs (SLO: <10.000µs 📉 -85.6%) vs baseline: -3.3% Memory: ✅ 38.460MB (SLO: <41.500MB -7.3%) vs baseline: +4.6% ✅ bytearray_extend_noaspectTime: ✅ 0.614µs (SLO: <10.000µs 📉 -93.9%) vs baseline: -0.6% Memory: ✅ 38.693MB (SLO: <41.500MB -6.8%) vs baseline: +5.5% ✅ bytearray_noaspectTime: ✅ 0.486µs (SLO: <10.000µs 📉 -95.1%) vs baseline: -1.0% Memory: ✅ 38.492MB (SLO: <41.500MB -7.2%) vs baseline: +4.6% ✅ bytes_aspectTime: ✅ 1.270µs (SLO: <10.000µs 📉 -87.3%) vs baseline: ~same Memory: ✅ 38.362MB (SLO: <41.500MB -7.6%) vs baseline: +4.2% ✅ bytes_noaspectTime: ✅ 0.498µs (SLO: <10.000µs 📉 -95.0%) vs baseline: ~same Memory: ✅ 38.539MB (SLO: <41.500MB -7.1%) vs baseline: +4.9% ✅ bytesio_aspectTime: ✅ 1.314µs (SLO: <10.000µs 📉 -86.9%) vs baseline: ~same Memory: ✅ 38.505MB (SLO: <41.500MB -7.2%) vs baseline: +4.3% ✅ bytesio_noaspectTime: ✅ 0.501µs (SLO: <10.000µs 📉 -95.0%) vs baseline: +0.7% Memory: ✅ 38.515MB (SLO: <41.500MB -7.2%) vs baseline: +5.1% ✅ capitalize_aspectTime: ✅ 0.741µs (SLO: <10.000µs 📉 -92.6%) vs baseline: +0.5% Memory: ✅ 38.574MB (SLO: <41.500MB -7.0%) vs baseline: +5.1% ✅ capitalize_noaspectTime: ✅ 0.438µs (SLO: <10.000µs 📉 -95.6%) vs baseline: +0.2% Memory: ✅ 38.471MB (SLO: <41.500MB -7.3%) vs baseline: +4.9% ✅ casefold_aspectTime: ✅ 0.739µs (SLO: <10.000µs 📉 -92.6%) vs baseline: -0.3% Memory: ✅ 38.449MB (SLO: <41.500MB -7.4%) vs baseline: +4.5% ✅ casefold_noaspectTime: ✅ 0.374µs (SLO: <10.000µs 📉 -96.3%) vs baseline: +0.2% Memory: ✅ 38.605MB (SLO: <41.500MB -7.0%) vs baseline: +5.2% ✅ decode_aspectTime: ✅ 0.737µs (SLO: <10.000µs 📉 -92.6%) vs baseline: +1.6% Memory: ✅ 38.750MB (SLO: <41.500MB -6.6%) vs baseline: +4.8% ✅ decode_noaspectTime: ✅ 0.423µs (SLO: <10.000µs 📉 -95.8%) vs baseline: +0.8% Memory: ✅ 38.445MB (SLO: <41.500MB -7.4%) vs baseline: +4.5% ✅ encode_aspectTime: ✅ 0.716µs (SLO: <10.000µs 📉 -92.8%) vs baseline: +0.5% Memory: ✅ 38.563MB (SLO: <41.500MB -7.1%) vs baseline: +5.1% ✅ encode_noaspectTime: ✅ 0.405µs (SLO: <10.000µs 📉 -96.0%) vs baseline: +0.3% Memory: ✅ 38.597MB (SLO: <41.500MB -7.0%) vs baseline: +5.3% ✅ format_aspectTime: ✅ 3.417µs (SLO: <10.000µs 📉 -65.8%) vs baseline: -0.3% Memory: ✅ 38.526MB (SLO: <41.500MB -7.2%) vs baseline: +4.5% ✅ format_map_aspectTime: ✅ 3.657µs (SLO: <10.000µs 📉 -63.4%) vs baseline: +0.6% Memory: ✅ 38.575MB (SLO: <41.500MB -7.0%) vs baseline: +4.9% ✅ format_map_noaspectTime: ✅ 0.830µs (SLO: <10.000µs 📉 -91.7%) vs baseline: +1.0% Memory: ✅ 38.424MB (SLO: <41.500MB -7.4%) vs baseline: +4.8% ✅ format_noaspectTime: ✅ 0.604µs (SLO: <10.000µs 📉 -94.0%) vs baseline: +1.9% Memory: ✅ 38.393MB (SLO: <41.500MB -7.5%) vs baseline: +4.1% ✅ index_aspectTime: ✅ 0.343µs (SLO: <10.000µs 📉 -96.6%) vs baseline: -0.4% Memory: ✅ 38.537MB (SLO: <41.500MB -7.1%) vs baseline: +5.0% ✅ index_noaspectTime: ✅ 0.313µs (SLO: <10.000µs 📉 -96.9%) vs baseline: +0.2% Memory: ✅ 38.520MB (SLO: <41.500MB -7.2%) vs baseline: +4.9% ✅ join_aspectTime: ✅ 1.318µs (SLO: <10.000µs 📉 -86.8%) vs baseline: ~same Memory: ✅ 38.427MB (SLO: <41.500MB -7.4%) vs baseline: +4.4% ✅ join_noaspectTime: ✅ 0.539µs (SLO: <10.000µs 📉 -94.6%) vs baseline: +1.2% Memory: ✅ 38.562MB (SLO: <41.500MB -7.1%) vs baseline: +4.7% ✅ ljust_aspectTime: ✅ 2.630µs (SLO: <20.000µs 📉 -86.9%) vs baseline: +3.0% Memory: ✅ 38.556MB (SLO: <41.500MB -7.1%) vs baseline: +4.8% ✅ ljust_noaspectTime: ✅ 0.411µs (SLO: <10.000µs 📉 -95.9%) vs baseline: +1.3% Memory: ✅ 38.567MB (SLO: <41.500MB -7.1%) vs baseline: +5.0% ✅ lower_aspectTime: ✅ 2.258µs (SLO: <10.000µs 📉 -77.4%) vs baseline: +1.1% Memory: ✅ 38.600MB (SLO: <41.500MB -7.0%) vs baseline: +5.2% ✅ lower_noaspectTime: ✅ 0.373µs (SLO: <10.000µs 📉 -96.3%) vs baseline: -1.0% Memory: ✅ 38.556MB (SLO: <41.500MB -7.1%) vs baseline: +4.7% ✅ lstrip_aspectTime: ✅ 2.203µs (SLO: <20.000µs 📉 -89.0%) vs baseline: +0.6% Memory: ✅ 38.635MB (SLO: <41.500MB -6.9%) vs baseline: +4.9% ✅ lstrip_noaspectTime: ✅ 0.389µs (SLO: <10.000µs 📉 -96.1%) vs baseline: +1.4% Memory: ✅ 38.697MB (SLO: <41.500MB -6.8%) vs baseline: +5.3% ✅ modulo_aspectTime: ✅ 0.974µs (SLO: <10.000µs 📉 -90.3%) vs baseline: +0.1% Memory: ✅ 38.472MB (SLO: <41.500MB -7.3%) vs baseline: +4.8% ✅ modulo_aspect_for_bytearray_bytearrayTime: ✅ 1.484µs (SLO: <10.000µs 📉 -85.2%) vs baseline: ~same Memory: ✅ 38.691MB (SLO: <41.500MB -6.8%) vs baseline: +5.3% ✅ modulo_aspect_for_bytesTime: ✅ 0.951µs (SLO: <10.000µs 📉 -90.5%) vs baseline: -0.3% Memory: ✅ 38.577MB (SLO: <41.500MB -7.0%) vs baseline: +4.3% ✅ modulo_aspect_for_bytes_bytearrayTime: ✅ 1.172µs (SLO: <10.000µs 📉 -88.3%) vs baseline: -0.2% Memory: ✅ 38.828MB (SLO: <41.500MB -6.4%) vs baseline: +5.2% ✅ modulo_noaspectTime: ✅ 0.671µs (SLO: <10.000µs 📉 -93.3%) vs baseline: -0.2% Memory: ✅ 38.509MB (SLO: <41.500MB -7.2%) vs baseline: +4.3% ✅ replace_aspectTime: ✅ 5.018µs (SLO: <10.000µs 📉 -49.8%) vs baseline: +2.2% Memory: ✅ 38.465MB (SLO: <41.500MB -7.3%) vs baseline: +4.9% ✅ replace_noaspectTime: ✅ 0.465µs (SLO: <10.000µs 📉 -95.3%) vs baseline: +0.6% Memory: ✅ 38.581MB (SLO: <41.500MB -7.0%) vs baseline: +4.9% ✅ repr_aspectTime: ✅ 0.952µs (SLO: <10.000µs 📉 -90.5%) vs baseline: ~same Memory: ✅ 38.487MB (SLO: <41.500MB -7.3%) vs baseline: +5.1% ✅ repr_noaspectTime: ✅ 0.458µs (SLO: <10.000µs 📉 -95.4%) vs baseline: +0.8% Memory: ✅ 38.585MB (SLO: <41.500MB -7.0%) vs baseline: +4.6% ✅ rstrip_aspectTime: ✅ 1.878µs (SLO: <20.000µs 📉 -90.6%) vs baseline: +1.4% Memory: ✅ 38.537MB (SLO: <41.500MB -7.1%) vs baseline: +5.0% ✅ rstrip_noaspectTime: ✅ 0.380µs (SLO: <10.000µs 📉 -96.2%) vs baseline: -1.0% Memory: ✅ 38.574MB (SLO: <41.500MB -7.0%) vs baseline: +4.7% ✅ slice_aspectTime: ✅ 0.484µs (SLO: <10.000µs 📉 -95.2%) vs baseline: -0.6% Memory: ✅ 38.536MB (SLO: <41.500MB -7.1%) vs baseline: +5.2% ✅ slice_noaspectTime: ✅ 0.450µs (SLO: <10.000µs 📉 -95.5%) vs baseline: +0.1% Memory: ✅ 38.632MB (SLO: <41.500MB -6.9%) vs baseline: +4.9% ✅ stringio_aspectTime: ✅ 1.701µs (SLO: <10.000µs 📉 -83.0%) vs baseline: +0.6% Memory: ✅ 38.548MB (SLO: <41.500MB -7.1%) vs baseline: +4.9% ✅ stringio_noaspectTime: ✅ 0.919µs (SLO: <10.000µs 📉 -90.8%) vs baseline: +0.7% Memory: ✅ 38.459MB (SLO: <41.500MB -7.3%) vs baseline: +4.6% ✅ strip_aspectTime: ✅ 2.449µs (SLO: <20.000µs 📉 -87.8%) vs baseline: 📈 +12.5% Memory: ✅ 38.580MB (SLO: <41.500MB -7.0%) vs baseline: +4.9% ✅ strip_noaspectTime: ✅ 0.390µs (SLO: <10.000µs 📉 -96.1%) vs baseline: +0.1% Memory: ✅ 38.630MB (SLO: <41.500MB -6.9%) vs baseline: +5.3% ✅ swapcase_aspectTime: ✅ 2.445µs (SLO: <10.000µs 📉 -75.5%) vs baseline: +0.8% Memory: ✅ 38.548MB (SLO: <41.500MB -7.1%) vs baseline: +4.3% ✅ swapcase_noaspectTime: ✅ 0.541µs (SLO: <10.000µs 📉 -94.6%) vs baseline: +0.7% Memory: ✅ 38.571MB (SLO: <41.500MB -7.1%) vs baseline: +4.8% ✅ title_aspectTime: ✅ 2.384µs (SLO: <10.000µs 📉 -76.2%) vs baseline: -0.1% Memory: ✅ 38.462MB (SLO: <41.500MB -7.3%) vs baseline: +4.6% ✅ title_noaspectTime: ✅ 0.502µs (SLO: <10.000µs 📉 -95.0%) vs baseline: -1.0% Memory: ✅ 38.661MB (SLO: <41.500MB -6.8%) vs baseline: +5.4% ✅ translate_aspectTime: ✅ 3.265µs (SLO: <10.000µs 📉 -67.3%) vs baseline: -0.2% Memory: ✅ 38.772MB (SLO: <41.500MB -6.6%) vs baseline: +5.6% ✅ translate_noaspectTime: ✅ 1.049µs (SLO: <10.000µs 📉 -89.5%) vs baseline: +0.1% Memory: ✅ 38.700MB (SLO: <41.500MB -6.7%) vs baseline: +5.2% ✅ upper_aspectTime: ✅ 2.394µs (SLO: <10.000µs 📉 -76.1%) vs baseline: +5.5% Memory: ✅ 38.560MB (SLO: <41.500MB -7.1%) vs baseline: +4.6% ✅ upper_noaspectTime: ✅ 0.376µs (SLO: <10.000µs 📉 -96.2%) vs baseline: +0.6% Memory: ✅ 38.628MB (SLO: <41.500MB -6.9%) vs baseline: +4.9% 📈 iastaspectsospath - 24/24✅ ospathbasename_aspectTime: ✅ 5.156µs (SLO: <10.000µs 📉 -48.4%) vs baseline: 📈 +21.9% Memory: ✅ 38.535MB (SLO: <41.000MB -6.0%) vs baseline: +4.7% ✅ ospathbasename_noaspectTime: ✅ 1.087µs (SLO: <10.000µs 📉 -89.1%) vs baseline: +0.1% Memory: ✅ 38.673MB (SLO: <41.000MB -5.7%) vs baseline: +5.2% ✅ ospathjoin_aspectTime: ✅ 6.131µs (SLO: <10.000µs 📉 -38.7%) vs baseline: +2.2% Memory: ✅ 38.516MB (SLO: <41.000MB -6.1%) vs baseline: +4.5% ✅ ospathjoin_noaspectTime: ✅ 2.296µs (SLO: <10.000µs 📉 -77.0%) vs baseline: -0.4% Memory: ✅ 38.535MB (SLO: <41.000MB -6.0%) vs baseline: +4.5% ✅ ospathnormcase_aspectTime: ✅ 3.473µs (SLO: <10.000µs 📉 -65.3%) vs baseline: +0.1% Memory: ✅ 38.574MB (SLO: <41.000MB -5.9%) vs baseline: +4.8% ✅ ospathnormcase_noaspectTime: ✅ 0.571µs (SLO: <10.000µs 📉 -94.3%) vs baseline: -0.6% Memory: ✅ 38.614MB (SLO: <41.000MB -5.8%) vs baseline: +5.0% ✅ ospathsplit_aspectTime: ✅ 4.855µs (SLO: <10.000µs 📉 -51.4%) vs baseline: -0.2% Memory: ✅ 38.555MB (SLO: <41.000MB -6.0%) vs baseline: +4.7% ✅ ospathsplit_noaspectTime: ✅ 1.603µs (SLO: <10.000µs 📉 -84.0%) vs baseline: +0.6% Memory: ✅ 38.535MB (SLO: <41.000MB -6.0%) vs baseline: +4.7% ✅ ospathsplitdrive_aspectTime: ✅ 3.710µs (SLO: <10.000µs 📉 -62.9%) vs baseline: -0.8% Memory: ✅ 38.614MB (SLO: <41.000MB -5.8%) vs baseline: +4.9% ✅ ospathsplitdrive_noaspectTime: ✅ 0.699µs (SLO: <10.000µs 📉 -93.0%) vs baseline: -1.1% Memory: ✅ 38.574MB (SLO: <41.000MB -5.9%) vs baseline: +4.7% ✅ ospathsplitext_aspectTime: ✅ 4.558µs (SLO: <10.000µs 📉 -54.4%) vs baseline: -1.3% Memory: ✅ 38.535MB (SLO: <41.000MB -6.0%) vs baseline: +4.5% ✅ ospathsplitext_noaspectTime: ✅ 1.382µs (SLO: <10.000µs 📉 -86.2%) vs baseline: -0.6% Memory: ✅ 38.614MB (SLO: <41.000MB -5.8%) vs baseline: +5.0% 📈 telemetryaddmetric - 30/30✅ 1-count-metric-1-timesTime: ✅ 3.359µs (SLO: <20.000µs 📉 -83.2%) vs baseline: 📈 +11.2% Memory: ✅ 34.741MB (SLO: <35.500MB -2.1%) vs baseline: +4.5% ✅ 1-count-metrics-100-timesTime: ✅ 203.693µs (SLO: <220.000µs -7.4%) vs baseline: -1.3% Memory: ✅ 34.800MB (SLO: <35.500MB 🟡 -2.0%) vs baseline: +4.6% ✅ 1-distribution-metric-1-timesTime: ✅ 3.361µs (SLO: <20.000µs 📉 -83.2%) vs baseline: -1.4% Memory: ✅ 34.918MB (SLO: <35.500MB 🟡 -1.6%) vs baseline: +5.2% ✅ 1-distribution-metrics-100-timesTime: ✅ 219.789µs (SLO: <230.000µs -4.4%) vs baseline: ~same Memory: ✅ 34.800MB (SLO: <35.500MB 🟡 -2.0%) vs baseline: +4.5% ✅ 1-gauge-metric-1-timesTime: ✅ 2.198µs (SLO: <20.000µs 📉 -89.0%) vs baseline: -0.2% Memory: ✅ 34.819MB (SLO: <35.500MB 🟡 -1.9%) vs baseline: +4.8% ✅ 1-gauge-metrics-100-timesTime: ✅ 136.823µs (SLO: <150.000µs -8.8%) vs baseline: +0.5% Memory: ✅ 34.780MB (SLO: <35.500MB -2.0%) vs baseline: +4.5% ✅ 1-rate-metric-1-timesTime: ✅ 3.150µs (SLO: <20.000µs 📉 -84.3%) vs baseline: -0.7% Memory: ✅ 34.839MB (SLO: <35.500MB 🟡 -1.9%) vs baseline: +4.9% ✅ 1-rate-metrics-100-timesTime: ✅ 216.291µs (SLO: <250.000µs 📉 -13.5%) vs baseline: -1.1% Memory: ✅ 34.839MB (SLO: <35.500MB 🟡 -1.9%) vs baseline: +4.8% ✅ 100-count-metrics-100-timesTime: ✅ 20.399ms (SLO: <22.000ms -7.3%) vs baseline: -2.2% Memory: ✅ 34.800MB (SLO: <35.500MB 🟡 -2.0%) vs baseline: +4.5% ✅ 100-distribution-metrics-100-timesTime: ✅ 2.297ms (SLO: <2.550ms -9.9%) vs baseline: +0.2% Memory: ✅ 34.859MB (SLO: <35.500MB 🟡 -1.8%) vs baseline: +4.7% ✅ 100-gauge-metrics-100-timesTime: ✅ 1.399ms (SLO: <1.550ms -9.8%) vs baseline: +0.4% Memory: ✅ 34.839MB (SLO: <35.500MB 🟡 -1.9%) vs baseline: +4.9% ✅ 100-rate-metrics-100-timesTime: ✅ 2.232ms (SLO: <2.550ms 📉 -12.5%) vs baseline: -1.6% Memory: ✅ 34.839MB (SLO: <35.500MB 🟡 -1.9%) vs baseline: +4.4% ✅ flush-1-metricTime: ✅ 4.617µs (SLO: <20.000µs 📉 -76.9%) vs baseline: -0.4% Memory: ✅ 34.996MB (SLO: <35.500MB 🟡 -1.4%) vs baseline: +4.9% ✅ flush-100-metricsTime: ✅ 174.325µs (SLO: <250.000µs 📉 -30.3%) vs baseline: +0.3% Memory: ✅ 35.252MB (SLO: <35.500MB 🟡 -0.7%) vs baseline: +5.0% ✅ flush-1000-metricsTime: ✅ 2.174ms (SLO: <2.500ms 📉 -13.0%) vs baseline: +0.3% Memory: ✅ 36.038MB (SLO: <36.500MB 🟡 -1.3%) vs baseline: +5.0% 🟡 Near SLO Breach (15 suites)🟡 coreapiscenario - 10/10 (1 unstable)
|
|
Do we understand what's holding references to those frames? In the ER case, for instance, the frame is held by the snapshot, but as soon as that is serialised we drop it, and with it the reference to the frame object 🤔 |
20fe3fb to
e8d6258
Compare
e8d6258 to
7f58de8
Compare
|
This was an artifact of the test harness queueing the Python objects unlike our real system which enqueues JSON strings |
Description
This test shows that we persist frames in memory until messages are sent to Datadog. We should find a way to dereference these frames so the garbage collector can clean them up as soon as possible, otherwise we risk raising the application high water mark.
Testing
Risks
Additional Notes