feat: Phase 2 Modernization: httpx2, Pydantic v2, Retry Engine, and Security Guards#51
Open
skupriienko-mailgun wants to merge 45 commits into
Open
feat: Phase 2 Modernization: httpx2, Pydantic v2, Retry Engine, and Security Guards#51skupriienko-mailgun wants to merge 45 commits into
skupriienko-mailgun wants to merge 45 commits into
Conversation
The async ecosystem is rapidly evolving. Relying on older Python runtimes or outdated HTTP clients introduces performance bottlenecks and blocks us from adopting modern async features safely. - Bumped 'requires-python' requirement to '>=3.11'. - Added 'httpx2 >=2.7.0' as the new primary async engine, keeping 'httpx >=0.24.0' as a fallback safety net. - Added fastapi' to dev dependencies. This keeps the SDK aligned with modern Python standards, guaranteeing our users benefit from the latest performance optimizations and security patches while preventing dependency hell.
… normalization Developers often unintentionally trigger spam filters with invalid HTML or send duplicate emails during network outages, harming their domain's reputation. Additionally, Internationalized Domain Names (IDN) can cause Unicode routing crashes in HTTP clients. - Introduced 'SpamGuard' (a zero-network static HTML deliverability analyzer). - Introduced 'IdempotencyGuard' to generate deterministic SHA-256 payload fingerprints. - Added 'DeliverabilityError' to gracefully handle SpamGuard rule violations. - Implemented 'normalize_domain' to natively convert IDNs to safe RFC 3490 Punycode, which allows domain names to contain non-Latin characters. By failing fast locally, we actively protect our users' domain reputations and prevent billing spikes from duplicate sends—all without adding external network overhead or API latency.
Hardcoded retry logic directly inside HTTP adapters leads to 'thundering herd' problems when the API is rate-limiting us (429s) or experiencing transient cloud errors (50x). - Extracted retry configurations into a dedicated 'RetryPolicy' class. - Implemented mathematical exponential backoff with Full Jitter to prevent collisions. - Added native support for respecting the 'Retry-After' HTTP header. This aligns the SDK with cloud-native resilience best practices, ensuring we behave gracefully as a client while maximizing delivery success rates under heavy load.
…drails Loading massive attachments (e.g., 20MB PDFs) entirely into memory crashes Serverless functions (OOM/CWE-400). Building complex inline HTML templates is clunky, and remembering to manually add idempotency headers is error-prone. - Implemented generator-based 'ChunkedStreamer' for lazy file loading. - Added 'attach_stream' for large files and updated 'attach_inline' to support explicit Content-IDs. - Injected 'check_deliverability()' into the builder. - Added '.set_idempotency_safe()' and automated 'h:X-Idempotency-Key' injection inside the '.build()' method. This guarantees memory-safe execution in constrained environments and provides a bulletproof, 'pit-of-success' developer experience right out of the box.
…x intercepts Orchestrators like Kubernetes require a lightweight way to check if an API client is healthy. During local testing, our 'dry_run' mode lacked rich payload previews. Furthermore, automatic retries weren't resetting file stream pointers, resulting in empty file uploads on the second attempt. - Added a low-overhead, fail-safe 'ping()' method to both Client and AsyncClient. - Implemented the explicit RetryPolicy loop inside endpoint requests. - Added '_reset_stream_pointers' to guarantee idempotency during retry iterations. - Upgraded the 'dry_run' interceptor to trigger LocalSandbox for rich email previews. These changes drastically improve production reliability (K8s readiness probes) while offering a vastly superior, zero-leak testing environment for local development.
Directly importing underlying dependencies inside fuzzers and type definition files creates tight coupling, making it incredibly painful to migrate HTTP clients or introduce fallback engines later. - Centralized all HTTPX imports across the fuzzing suite and types.py through mailgun._httpx_compat. This pays down technical debt and centralizes dependency management, making the SDK's core far more robust to upstream changes in the Python ecosystem.
- Added README snippets for explicit CID 'attach_inline' attachments and 'client.ping()' readiness probes.
- Expanded 'builder_examples.py' with real-world scenarios: chunked streams ('send_large_report_sync'), spam analysis ('send_marketing_campaign'), and client-side exactly-once delivery ('test_idempotency_guard_in_action').
By providing ready-to-copy, proven patterns, we lower the barrier to entry and empower developers to adopt our safest features immediately.
Introducing an exponential backoff retry policy can drastically and artificially inflate the execution time of our CI/CD test pipelines. Concurrently, new integrations need rigorous regression testing without colliding with each other.
- Added 'bypass_retry_delays' fixture in 'conftest.py' to globally mock 'time.sleep' and 'asyncio.sleep' during tests.
- Added assertions for 'LocalSandbox' dry runs, 'AsyncClient' teardown loops, and updated mock transports.
- Segmented list addresses ('python_sdk_sync@...' vs 'python_sdk_async@...') in integration tests to prevent parallel collisions.
This maintains high developer velocity by keeping the test suite lightning fast, while strictly validating that all new features and guardrails operate flawlessly.
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
Remove the Python 3.10 classifier from pyproject configuration. Eliminate the typing-extensions dependency from all package environments. Update imports across builders, client, security, and types modules to pull Self, TypedDict, and NotRequired directly from the standard typing library. Remove legacy version checks that conditionally loaded types for pre-3.11 Python versions. BREAKING CHANGE: Support for Python 3.10 has been dropped. Consumers must upgrade to Python 3.11 or higher to use this library version. build: bump mypy target python_version to 3.11
Add Pydantic to the optional dependencies list to support strict payload validation. Update the FastAPI optional dependency group to explicitly require Pydantic alongside it. Register Pydantic in the developer environment specification for local testing. Add unit tests for strict payload schema validation via Pydantic extensions.
…ries Introduce TestChunkedStreamer to verify that file attachments are read precisely within memory-bounded chunk limits. Add TestRetryPolicy to validate the mathematical boundaries of the network backoff engine. Ensure exponential growth in the retry policy calculates full jitter accurately without breaching the maximum delay ceiling. Confirm that memory-efficient slots on the retry policy prevent dynamic dictionary allocation. Introduce security guard tests to validate path sanitization and boundary invariants.
Document version 1.9.0 in the changelog, highlighting new features like LocalSandbox, IdempotencyGuard, RetryPolicy, and httpx2 engine compatibility. Add comprehensive examples to the README for Local Sandbox local browser previews. Expand the README to include sections on Advanced Retry Policy, Exactly-Once Delivery, Strict Payload Validation with Pydantic, and Pre-Flight Validation. Clarify the memory-safe benefits of Streaming Pagination in the documentation.
- Mitigate CWE-113 (Header Injection) in builders and pydantic models via strict CRLF checks - Mitigate CWE-400 (Resource Exhaustion) by clamping timeouts to 300s max and enforcing 25MB limits on Pydantic body schemas - Mitigate CWE-294 (Replay Attacks) by adding 15-minute TTL to webhook signature verification - Mitigate CWE-79 (XSS) by injecting strict CSP meta tags in LocalSandbox previews - Mitigate CWE-319 by strictly enforcing TLS 1.2+ for connection pools - Fix false-positive idempotency deduplication by hashing attachment signatures - Update fuzzer dictionary with targeted control character byte sequences
- Fix URL corruption in handlers by removing global '.replace()' calls that mangled enterprise proxies - Fix silent webhook v3-to-v4 upgrade failure by explicitly passing data/filters to the routing handler - Fix async iteration in ChunkedStreamer by offloading file I/O to 'asyncio.to_thread' - Fix missing 'await' on domains.get() inside AsyncClient.ping() - Fix type hints in suppression handlers returning Any instead of str - Enforce string casting for all HTTP headers to prevent serialization crashes - Clamp 'Retry-After' headers against max_delay to prevent infinite sleeping
…efaults - Fix TypeError in RedactingFilter by safely handling standard lists vs NamedTuple unpacking - Prevent LocalSandbox from popping browser tabs in CI/CD environments by default - Abstract MockResponse HTTP error to use framework-agnostic native ApiError - Ensure Config audit_hook initialization state is stable across test reloads - Add async large report streaming example to demonstrate safe memory handling
- Add fuzz_pydantic_models.py to target schema validation and CRLF bounds - Add fuzz_webhooks.py to target temporal offsets and TTL limits - Update fuzz_builders harness to expect CWE-20 and CWE-113 security exceptions - Suppress LocalSandbox browser execution during fuzzing via environment variables - Expand seed_harvester.py with new template, lists, and domain endpoint targets - Add regression tests for Header Injection, Log Overrides, and Proxy URL boundaries
…mailgun/mailgun-python into feat/sdk-modernization-phase-2
Co-authored-by: Copilot Autofix powered by AI <223894421+github-code-quality[bot]@users.noreply.github.com>
skupriienko-mailgun
marked this pull request as ready for review
July 22, 2026 19:45
…ce updates This commit addresses three primary architectural concerns to ensure the SDK is enterprise-ready and fail-safe: 1. Resource Management (CWE-400 & CWE-22): Added a __del__ safety net to AsyncClient preventing silent socket exhaustion when developers bypass context managers. Hardened ChunkedStreamer to satisfy Mypy and Pyright strict typing for path traversal defenses. 2. Deep Redaction (CWE-316): Patched a Late Stringification Bypass in RedactingFilter._deep_redact(). It now recursively sanitizes nested Pydantic models and dictionaries, preventing high-entropy secrets from leaking into APM observability tools (like Datadog) before falling back to stringification. 3. Resilience & Routing: Replaced blind IO tuple indexing in IdempotencyGuard with explicit SHA-256 hashing (first 512 bytes) for raw byte streams, guaranteeing collision-free retries. Structurally hardened dynamic path construction in endpoints.py to safely process iterable kwargs.
…to safely handle teardowns when initialization fails partway; update tests
…on, and integration tests
Co-authored-by: Copilot Autofix powered by AI <223894421+github-code-quality[bot]@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <223894421+github-code-quality[bot]@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Links:
Jira
Actions:
Architecture & Developer Experience (DX):
LocalSandboxEmail Preview capability. Standard routes now natively intercept email payloads whendry_run=Trueand generate local browser previews without executing real network calls.RetryPolicywith stateless exponential backoff and jitter to intelligently handle network partitions and mitigate the "Thundering Herd" effect.httpx2compatibility via themailgun/_httpx_compat.pybridge, ensuring a graceful fallback to legacyhttpxfor older environments.mailgun.extecosystem namespace, featuring strictPydanticv2 payload schemas (e.g.,SendMessageSchema) for runtime validation, alongside FastAPI webhook dependencies and Django backend integrations.ChunkedStreamerto lazily stream large file attachments (up to 25MB) using 512KB partitions, locking memory consumption toSecurity & Guardrails:
SpamGuard, a zero-network static HTML analyzer that fails-fast on deliverability risks (like XSS scripts, missingaltattributes, or excessive payload size) before dispatching to Mailgun.IdempotencyGuardto generate deterministic SHA-256 payload fingerprints, preventing duplicate email charges during network retries.normalize_domainto natively convert Internationalized Domain Names (IDNs) to safe RFC 3490 Punycode, neutralizing Unicode routing crashes.DeliverabilityErrorto gracefully catch and handleSpamGuardrule violations.Bug Fixes:
MockResponse) tomailgun/types.pyand strictly routing them through the compatibility bridge.slotscheckAST superclass errors specifically for_SpamGuardParser(which inherits fromhtml.parser.HTMLParser).Testing, CI/CD & Supply Chain:
commit_checks.yaml) to explicitly validate against Python 3.14.typing-extensionsfrom package environments, replacing them with standard librarytypingmodules (Self,TypedDict,NotRequired).SpamGuard,IdempotencyGuard, andRetryPolicyto maintain 100% branch coverage across the new release features.Verification & Testing:
To verify these changes locally, ensure your environment variables (
APIKEYandDOMAIN, and others) are set, then run the following commands:1. Run the Unit Test Suite (Fast):
Validates the core routing logic, new guardrails (
SpamGuard,IdempotencyGuard),RetryPolicy, and strictPydanticpayload schemas.2. Run the Live Routing Meta-Test (No state mutation):
Proves the SDK correctly constructs URLs for all supported endpoints by hitting live Mailgun servers (expects 200, 400, 401, or 403 responses; tests fail if the Python SDK crashes or generates a 404 bad route).
3. Run the Full Integration Suite (State mutation):
Executes end-to-end flows against your Sandbox domain (creates/deletes real resources).
4. Execute the Interactive Smoke Test:
Runs the executable documentation script demonstrating cross-version routing and payload serialization.
5. Run the Performance & Cold-Boot Benchmarks:
Validates the new O(1) routing dispatch and
__slots__memory optimizations using our unified DX script.