Merge origin/main into eric-wang/kernel-azure-oauth

Resolve conflicts from #921 (JWT private-key M2M on use_kernel=True):
- CHANGELOG: keep main's JWT M2M entry + the U2M/Azure entries; the Azure
  entry supersedes main's stale 'azure-oauth not supported' note.
- auth_bridge.py docstring/comments: adopt main's resolution-order numbering
  (JWT M2M = step 1) while keeping azure-oauth as a U2M type in step 4.
- Extend the JWT-vs-U2M ambiguity guard to cover azure-oauth (main only
  guarded databricks-oauth); azure-oauth is now a U2M type, so
  oauth_jwt_key_file + azure-oauth must be rejected as ambiguous rather than
  falling through to JWT M2M. Parametrized the guard test over both U2M types.
- Final catch-all error message: list JWT M2M alongside the Azure flows.

Author: eric-wang-1990

CHANGELOG.md

@@ -1,6 +1,7 @@
 # Release History
 
 # Unreleased
+- Kernel backend (`use_kernel=True`): OAuth **M2M with a JWT private-key client assertion** (RFC 7523) is now supported. Pass `oauth_client_id` + `oauth_jwt_key_file` + `oauth_jwt_kid` (with optional `oauth_jwt_passphrase` for an encrypted PKCS#8 key, `oauth_jwt_algorithm` defaulting to `RS256`, `oauth_scopes`, and `token_url` for the IdP token endpoint) and the connector routes them to the kernel's `auth_type="oauth-m2m-jwt"`, which signs a short-lived assertion with the private key instead of sending a client secret. The kernel owns the token lifecycle. A private-key file is treated as unambiguous JWT M2M intent and is mutually exclusive with `oauth_client_secret` / `credentials_provider` (both raise `NotSupportedError`). Verified end-to-end against an Azure Databricks workspace with the service principal's public certificate registered on its Entra ID app registration. Requires `databricks-sql-kernel >= 0.2.0` with JWT support.
 - Kernel backend (`use_kernel=True`): OAuth U2M with `auth_type="databricks-oauth"` now forwards the connector's `databricks-sql-python` OAuth-app bundle (`client_id` + `sql offline_access` scopes + redirect port) into the kernel, so a bare U2M connection authenticates as `databricks-sql-python` — parity with the Thrift path — instead of inheriting the kernel's own `databricks-sql-connector` default. A caller-supplied `oauth_client_id` (with its coupled `oauth_redirect_port`) is honored, as is a caller-supplied `oauth_scopes`; absent one, the connector default (`sql offline_access`) is forwarded. Note: the kernel binds a single U2M redirect port, so unlike the Thrift path (which tries the full `8020..8024` range) the kernel path uses only one port and does not fall back to the next port if it is already bound — pass `oauth_redirect_port` (with `oauth_client_id`) to pick a free one on a port collision (PECOBLR-4040)
 - Kernel backend (`use_kernel=True`): **Azure Entra (Azure AD) service-principal M2M is now supported.** `auth_type="azure-sp-m2m"` forwards `azure_client_id` / `azure_client_secret`; the kernel is the Azure-aware auth core — it builds the Entra v2.0 token endpoint and the `{app_id}/.default` scope, and **auto-discovers the tenant** from the workspace's `/aad/auth` redirect when `azure_tenant_id` is omitted (matching Thrift). The `Authorization` bearer is the Databricks-audience data token, which alone authenticates a workspace-member SP. Set `azure_workspace_resource_id` and the kernel also sends the Azure SP management token (`X-Databricks-Azure-SP-Management-Token`) + `X-Databricks-Azure-Workspace-Resource-Id` header (matching the JDBC driver), so a service principal with an Azure RBAC role but no workspace membership can authenticate; omit it and no ARM management-scope token is fetched. Azure AD **U2M** (`auth_type="azure-oauth"`) now routes to the kernel's OAuth U2M flow, identically to `auth_type="databricks-oauth"`: the kernel runs the in-house workspace-federated browser flow, which Azure workspaces support (the workspace federates login to Entra). It forwards the connector's `databricks-sql-python` OAuth app, not the Thrift Azure app (`96eecda7` / port 8030), which is registered for Thrift's direct-Entra flow the kernel does not perform (PECOBLR-4141; PECOBLR-4120)
 

CONNECTION_PARAMETERS.md

@@ -73,6 +73,11 @@ to change without notice.
 | `oauth_client_id` (U2M)                             | `str`                |   ✅   |   ✅   | built-in client id        | Custom U2M client id. Forwarded on both; when absent, each path applies its own built-in default.                                                              |
 | `oauth_redirect_port` (U2M)                         | `int`                |   ✅   |   ✅   | `None`                    | Localhost redirect port for the browser flow. On **both** backends it is only honored when a custom `oauth_client_id` is also supplied — then that single port becomes the redirect URI. With the built-in client id (or when omitted) the connector uses the full registered range 8020–8024 and binds the first free port, so a bare `oauth_redirect_port` has no effect. (Thrift: `auth.py` `oauth_redirect_port_range`; Kernel: same logic, forwarded as `redirect_ports`.) |
 | `oauth_client_secret` (OAuth M2M)                   | `str`                |   ❌   |   ✅   | `None`                    | **Kernel-only in practice.** The Thrift auth path never reads `oauth_client_secret`; use `credentials_provider` or an Azure service principal for M2M on Thrift. |
+| `oauth_jwt_key_file` (OAuth M2M, JWT private key)   | `str`                |   ❌   |   ✅   | `None`                    | **Kernel-only.** Path to the PEM private key for JWT private-key M2M (RFC 7523 client assertion). Supplying it selects the JWT flow: the kernel signs a short-lived assertion with the key instead of sending a client secret. Requires `oauth_client_id` + `oauth_jwt_kid`; mutually exclusive with `oauth_client_secret` / `credentials_provider`. |
+| `oauth_jwt_kid` (OAuth M2M, JWT private key)        | `str`                |   ❌   |   ✅   | `None`                    | **Kernel-only.** Key id written into the JWT header so the IdP can select the registered public key. Required with `oauth_jwt_key_file`. (For Entra ID this is the certificate's `x5t` thumbprint.) |
+| `oauth_jwt_passphrase` (OAuth M2M, JWT private key) | `str`                |   ❌   |   ✅   | `None`                    | **Kernel-only.** Passphrase for an encrypted PKCS#8 private key; omit for an unencrypted key.                                                                    |
+| `oauth_jwt_algorithm` (OAuth M2M, JWT private key)  | `str`                |   ❌   |   ✅   | `RS256`                   | **Kernel-only.** JWT signing algorithm (`RS256`/`384`/`512`, `PS256`/`384`/`512`, `ES256`, `ES384`).                                                            |
+| `token_url` (OAuth M2M)                             | `str`                |   ❌   |   ✅   | `None` ⇒ OIDC discovery   | **Kernel-only.** OAuth IdP token endpoint override. Required for JWT M2M against an external-IdP-backed workspace (e.g. Entra ID for Azure Databricks), since Databricks-native OIDC does not advertise the `private_key_jwt` method. Applies to shared-secret M2M too. |
 | `oauth_scopes`                                      | `List[str]`          |   ❌   |   ✅   | `["sql","offline_access"]`| **Thrift ignores custom scopes** — it always uses the built-in scope set. Only the kernel honors a custom `oauth_scopes`.                                       |
 | `credentials_provider`                              | `CredentialsProvider`|   ✅   |   ❌   | `None`                    | Custom external credentials provider. **Rejected on the kernel path** (`NotSupportedError`) — it is an opaque token source, so the kernel cannot own the token lifecycle; use `oauth_client_id` + `oauth_client_secret` for M2M, or the Thrift backend. |
 | `identity_federation_client_id`                     | `str`                |   ✅   |   ✅   | `None`                    | Workload identity / token-federation client id (kernel support added in #910).                                                                                 |

CONTRIBUTING.md

@@ -153,7 +153,7 @@ reproduce on a SEA or kernel connection, and vice versa:
 | Backend | Select via (connect kwarg / `extra_params`) | Where its tests live |
 | --- | --- | --- |
 | **Thrift** (default) | *(nothing — the default path)* | the general `tests/e2e` suite (the `{}` parametrize case) and mocked `tests/unit` |
-| **SEA** (Statement Execution API) | `use_sea=True` | the general `tests/e2e` suite (the `{"use_sea": True}` parametrize case, e.g. `tests/e2e/test_driver.py`) and mocked `tests/unit` |
+| **SEA** (Statement Execution API) *(deprecated — use Kernel for SEA-native connections)* | `use_sea=True` | the general `tests/e2e` suite (the `{"use_sea": True}` parametrize case, e.g. `tests/e2e/test_driver.py`) and mocked `tests/unit` |
 | **Kernel** (Rust, optional) | `use_kernel=True` | the dedicated `tests/e2e/test_kernel_backend.py` / `test_kernel_tls.py`, plus the offline routing test `tests/unit/test_session.py -m realkernel` |
 
 Notes that matter when running the suite:

KERNEL_REV

@@ -1 +1 @@
-45a0d6ae1de2f203220913ba96c994ebb2d7aae4
+9e3dbf9c40733b176151e001c9a15202030b967a

examples/experimental/sea_connector_test.py

@@ -1,6 +1,11 @@
 """
 Main script to run all SEA connector tests.
 
+DEPRECATED: the pure-Python SEA backend (``use_sea=True``) exercised by
+these examples is incomplete (e.g. no positional ``?`` parameter binding)
+and slated for removal. For a SEA-native connection use ``use_kernel=True``
+instead — install it with ``pip install 'databricks-sql-connector[kernel]'``.
+
 This script runs all the individual test modules and displays
 a summary of test results with visual indicators.
 

osv-scanner.toml

@@ -16,3 +16,26 @@
 # This file starts empty -- populate iteratively as the first scan run
 # surfaces real false positives. Do not pre-populate with speculative
 # suppressions.
+
+[[IgnoredVulns]]
+id = "GHSA-g6cj-pr64-35w5"
+# CVE-2026-69247 / PYSEC-2026-3552 -- a Bleichenbacher-style timing/error
+# oracle in cryptography's PKCS#7 *EnvelopedData* decryption
+# (pkcs7_decrypt_der/pem/smime), exploitable only against an endpoint that
+# auto-decrypts attacker-supplied S/MIME EnvelopedData and responds
+# adaptively. Affects cryptography >=44.0.0,<50.0.0; fixed in 50.0.0.
+#
+# Not reachable here: `cryptography` is a transitive dependency (via
+# PyJWT[crypto] / oauthlib, for OAuth token signing/verification). The
+# connector never decrypts PKCS#7 / S/MIME EnvelopedData -- the vulnerable
+# code path is never invoked. Pre-existing on main (transitive, not
+# introduced by any connector change); a clean bump awaits PyJWT/oauthlib
+# floors that pull cryptography>=50. Re-evaluate when the dependency tree
+# resolves to a fixed cryptography.
+
+[[IgnoredVulns]]
+# Same advisory under its PYSEC alias -- OSV reports both ids for this
+# finding, so suppress both to be safe (see GHSA-g6cj-pr64-35w5 above for
+# the full justification: unreachable PKCS#7/S-MIME decryption oracle in a
+# transitive dependency).
+id = "PYSEC-2026-3552"

src/databricks/sql/backend/kernel/auth_bridge.py

@@ -178,13 +178,18 @@ def kernel_auth_kwargs(
 
     (The ``azure-sp-m2m`` Azure Entra auth type is handled up front, before
     these guards, forwarding to the kernel's Azure SP flow. ``azure-oauth`` is
-    a U2M type and is handled by step 3 alongside ``databricks-oauth``. See the
-    module docstring.)
-    1. **OAuth M2M** — ``oauth_client_id`` + ``oauth_client_secret``
+    a U2M type and is handled by the OAuth U2M step alongside
+    ``databricks-oauth``. See the module docstring.)
+    1. **OAuth M2M (JWT private key)** — ``oauth_jwt_key_file`` present →
+       forward the private-key + ``oauth_client_id`` + ``oauth_jwt_kid``
+       to the kernel's ``oauth-m2m-jwt`` (RFC 7523 client assertion). The
+       kernel signs the assertion and owns the token lifecycle. Checked
+       first because a private-key file is unambiguous JWT M2M intent.
+    2. **OAuth M2M** — ``oauth_client_id`` + ``oauth_client_secret``
        both present → forward raw creds to the kernel's ``oauth-m2m``.
-    2. **PAT** — the built provider is (or wraps) an
+    3. **PAT** — the built provider is (or wraps) an
        ``AccessTokenAuthProvider`` → extract the bearer token.
-    3. **OAuth U2M** — ``auth_type`` is ``databricks-oauth`` or ``azure-oauth``
+    4. **OAuth U2M** — ``auth_type`` is ``databricks-oauth`` or ``azure-oauth``
        → forward the connector's coupled ``databricks-sql-python`` bundle
        (``client_id`` + ``redirect_ports`` list, defaulting scopes to
        ``PYSQL_OAUTH_SCOPES`` when the caller supplies none) to the kernel's
@@ -194,9 +199,9 @@ def kernel_auth_kwargs(
        (PECOBLR-4039/4040). ``azure-oauth`` resolves identically: the kernel's
        workspace-federated flow serves Azure workspaces too (PECOBLR-4120).
        Unlike the Thrift path, a caller-supplied ``oauth_scopes`` is honored here.
-    4. **Custom credentials_provider** → ``NotSupportedError`` (opaque
+    5. **Custom credentials_provider** → ``NotSupportedError`` (opaque
        token source; no raw creds for the kernel to own).
-    5. Anything else → ``NotSupportedError``.
+    6. Anything else → ``NotSupportedError``.
 
     M2M is checked before PAT so that a workload passing both an
     access token *and* M2M creds resolves to the (refreshing) M2M path
@@ -211,7 +216,12 @@ def kernel_auth_kwargs(
     client_secret = opts.get("oauth_client_secret")
     federation_client_id = opts.get("identity_federation_client_id")
     auth_type = opts.get("auth_type")
+    jwt_key_file = opts.get("oauth_jwt_key_file")
     has_m2m = bool(client_id and client_secret)
+    # A private-key file is unambiguous JWT client-assertion M2M intent
+    # (RFC 7523): the kernel signs a short-lived assertion with the key
+    # rather than sending a client secret.
+    has_jwt_m2m = bool(jwt_key_file)
 
     # Azure Entra (Azure AD) auth types route to the kernel's GENERIC OAuth
     # flows with Azure values supplied as overrides — the kernel needs no
@@ -288,22 +298,96 @@ def kernel_auth_kwargs(
             "(machine-to-machine). Drop oauth_client_secret for U2M, or drop "
             "auth_type for M2M."
         )
+    if has_jwt_m2m and client_secret:
+        raise NotSupportedError(
+            "Ambiguous auth on use_kernel=True: both oauth_jwt_key_file "
+            "(JWT private-key M2M) and oauth_client_secret (shared-secret "
+            "M2M) were provided. Pass exactly one — a private key for "
+            "JWT client-assertion M2M, or a client secret for shared-secret M2M."
+        )
+    if has_jwt_m2m and opts.get("credentials_provider") is not None:
+        raise NotSupportedError(
+            "Ambiguous auth on use_kernel=True: both a custom "
+            "credentials_provider and oauth_jwt_key_file were provided. "
+            "Pass exactly one — oauth_client_id + oauth_jwt_key_file for "
+            "kernel-managed JWT private-key M2M, or use the Thrift backend "
+            "(default) for credentials_provider."
+        )
+    if has_jwt_m2m and auth_type in ("databricks-oauth", "azure-oauth"):
+        raise NotSupportedError(
+            f"Ambiguous auth on use_kernel=True: auth_type={auth_type!r} selects "
+            "the U2M browser flow, but oauth_jwt_key_file was also provided "
+            "(JWT private-key M2M). Drop oauth_jwt_key_file for U2M, or drop "
+            "auth_type for JWT M2M."
+        )
 
-    # 1. OAuth M2M — raw client-credentials pair forwarded to the kernel.
-    if has_m2m:
+    # 1. OAuth M2M (JWT private-key client assertion) — the kernel signs a
+    #    short-lived assertion with the private key and runs the
+    #    client-credentials grant. Checked before shared-secret M2M and PAT
+    #    because a private-key file is unambiguous JWT M2M intent. Requires
+    #    oauth_client_id (the service principal / OAuth client) and
+    #    oauth_jwt_kid (the key id the IdP uses to select the registered
+    #    public key). Optional oauth_jwt_passphrase / oauth_jwt_algorithm /
+    #    oauth_scopes / token_url are forwarded when present; the kernel
+    #    fills defaults (RS256 algorithm, all-apis scope, OIDC discovery)
+    #    for any omitted.
+    if has_jwt_m2m:
+        if not client_id:
+            raise ProgrammingError(
+                "use_kernel=True JWT private-key M2M (oauth_jwt_key_file) "
+                "requires oauth_client_id (the service principal / OAuth "
+                "client id used as the assertion issuer and subject)."
+            )
+        jwt_kid = opts.get("oauth_jwt_kid")
+        if not jwt_kid:
+            raise ProgrammingError(
+                "use_kernel=True JWT private-key M2M (oauth_jwt_key_file) "
+                "requires oauth_jwt_kid (the key id written into the JWT "
+                "header so the IdP can select the registered public key)."
+            )
         kwargs: Dict[str, Any] = {
+            "auth_type": "oauth-m2m-jwt",
+            "client_id": client_id,
+            "jwt_key_file": jwt_key_file,
+            "jwt_kid": jwt_kid,
+        }
+        jwt_passphrase = opts.get("oauth_jwt_passphrase")
+        if jwt_passphrase:
+            kwargs["jwt_passphrase"] = jwt_passphrase
+        jwt_algorithm = opts.get("oauth_jwt_algorithm")
+        if jwt_algorithm:
+            kwargs["jwt_algorithm"] = jwt_algorithm
+        token_url = opts.get("token_url")
+        if token_url:
+            kwargs["token_url"] = token_url
+        scopes = _normalize_scopes(opts.get("oauth_scopes"))
+        if scopes is not None:
+            kwargs["oauth_scopes"] = scopes
+        if federation_client_id:
+            kwargs["identity_federation_client_id"] = federation_client_id
+        return kwargs
+
+    # 2. OAuth M2M — raw client-credentials pair forwarded to the kernel.
+    if has_m2m:
+        kwargs = {
             "auth_type": "oauth-m2m",
             "client_id": client_id,
             "client_secret": client_secret,
         }
         scopes = _normalize_scopes(opts.get("oauth_scopes"))
         if scopes is not None:
             kwargs["oauth_scopes"] = scopes
+        # token_url is an auth-method-agnostic token-endpoint override (JDBC's
+        # OAuth2ConnAuthTokenEndpoint applies it to client-secret M2M too), so
+        # forward it here as well as on the JWT path.
+        token_url = opts.get("token_url")
+        if token_url:
+            kwargs["token_url"] = token_url
         if federation_client_id:
             kwargs["identity_federation_client_id"] = federation_client_id
         return kwargs
 
-    # 2. PAT (including TokenFederationProvider-wrapped PAT).
+    # 3. PAT (including TokenFederationProvider-wrapped PAT).
     if _is_pat(auth_provider):
         token = _extract_bearer_token(auth_provider)
         if not token:
@@ -316,7 +400,7 @@ def kernel_auth_kwargs(
             kwargs["identity_federation_client_id"] = federation_client_id
         return kwargs
 
-    # 3. OAuth U2M — browser authorization-code flow; the kernel runs it.
+    # 4. OAuth U2M — browser authorization-code flow; the kernel runs it.
     #    Both databricks-oauth and azure-oauth reach here: on the kernel path
     #    they are the same in-house workspace-federated flow (Azure workspaces
     #    federate the browser login to Entra), so azure-oauth is an alias for
@@ -351,7 +435,7 @@ def kernel_auth_kwargs(
             kwargs["identity_federation_client_id"] = federation_client_id
         return kwargs
 
-    # 4. Custom credentials_provider — the connector's primary M2M path
+    # 5. Custom credentials_provider — the connector's primary M2M path
     #    on Thrift/SEA, but unusable on the kernel: it's an opaque token
     #    source with no extractable client_id/secret, so the kernel
     #    can't own the token lifecycle. Point the caller at the raw
@@ -365,15 +449,16 @@ def kernel_auth_kwargs(
             "credentials_provider."
         )
 
-    # 5. Everything else (including no usable credentials at all —
+    # 6. Everything else (including no usable credentials at all —
     #    ``auth_provider`` is None on the kernel path when no access
     #    token was supplied and no OAuth kwargs resolved above).
     provider_desc = (
         type(auth_provider).__name__ if auth_provider is not None else "no credentials"
     )
     raise NotSupportedError(
         f"use_kernel=True requires PAT (access_token), OAuth M2M "
-        f"(oauth_client_id + oauth_client_secret), OAuth U2M "
+        f"(oauth_client_id + oauth_client_secret), OAuth M2M JWT "
+        f"(oauth_client_id + oauth_jwt_key_file + oauth_jwt_kid), OAuth U2M "
         f"(auth_type='databricks-oauth' or 'azure-oauth'), or Azure SP M2M "
         f"(auth_type='azure-sp-m2m'), but got "
         f"{provider_desc} with auth_type={auth_type!r}. Use the Thrift "