Add MySQL backend#872
Open
phdoerfler wants to merge 9 commits into
Open
Conversation
phdoerfler
marked this pull request as ready for review
July 14, 2026 19:57
Latest tag is v0.29.0; the SQLite backend work also changes the SqlMappingLike dialect API (new abstract members, defaultOffsetForSubquery renamed to normalizeOffsetLimit), so the next release must start a new binary-incompatible minor line for MiMa.
Datetime tests in the MSSQL suite (MovieSuite's `moviesShownBetween` and computed-field cases) fail on a developer machine in a non-UTC time zone, though they pass in CI. The MSSQL test codec encodes an OffsetDateTime argument via a zone-naive `java.sql.Timestamp`, and mssql-jdbc binds it into a DATETIMEOFFSET parameter using the ambient JVM time zone. On a UTC+2 host the filter bound shifts by two hours, admitting an extra row. Postgres is immune (native `Meta[OffsetDateTime]`); Oracle uses the same codec but ojdbc binds it without the shift; CI runs in UTC so it never surfaces there.
A new Doobie-based SQL backend for SQLite, mirroring the doobie-oracle/
doobie-mssql module shape: a single dialect mapping supplying the
SqlMappingLike dialect hooks, plus a test harness wiring up the shared
sql-core suites against a fresh temp-file SQLite database per test
suite (no docker service needed, since SQLite runs in-process).
Two dialect-specific choices, documented inline: LIMIT/OFFSET uses
SQLite's legacy comma form (`LIMIT offset, limit`), since the shared
query builder's fixed offset-then-limit render order is incompatible
with SQLite's OFFSET-must-follow-LIMIT grammar; and LIKE
case-sensitivity relies on a connection-level `case_sensitive_like`
pragma paired with `UPPER()`-normalization for case-insensitive matches.
SQLite has two grammar gaps no pre-existing dialect hook could route
around, addressed in sql-core with existing dialects' behavior
unchanged:
- No correlated FROM-clause subqueries (no LATERAL equivalent).
`SqlSelect.nest()` embeds a parent-constraint equality predicate into
a nested subquery's own WHERE clause - redundant with the enclosing
JOIN ... ON clause, and only useful as an optimization for genuinely
lateral-evaluated subqueries (Postgres/Oracle LATERAL, MSSQL
CROSS/OUTER APPLY). On a dialect with no lateral join at all, that
predicate references a column out of scope and fails at runtime with
"no such column". A derived capability, `supportsLateralJoin` (false
exactly when `mkLateral` has no lateral form to render and answers
`NotLateral`, as SQLite's does), makes `addFilterOrderByOffsetLimit`
skip embedding it.
- No parenthesized UNION branches. `SqlUnion.toFragment` used to wrap
each UNION ALL branch in parentheses unconditionally; SQLite's
compound-select grammar forbids that. Branch rendering is now a
dialect hook, `unionBranchToFragment`, which the existing dialects
implement with the previous parenthesizing behavior and SQLite
implements as the identity - safe for any number of branches since
grackle only ever uses UNION ALL, a purely associative bag union.
`DoobieSqliteMappingLike.encapsulateUnionBranch` is also extended to
wrap branches carrying their own offset/limit (not just order) into
a derived-table subquery, since a branch-level LIMIT is rejected by
SQLite regardless of parenthesization.
Fixing the LATERAL gap exposed a real correctness bug, also fixed
here: `addFilterOrderByOffsetLimit` has two "Case 1" fast paths that
apply LIMIT/OFFSET directly to a query built from pre-existing joins,
trusting `oneToOne && predIsOneToOne` to mean one row per key. That's
only guaranteed when a lateral-evaluated subquery produced those joins
(at most one contribution per outer row); a join chain containing a
nested one-to-many hop (e.g. a windowed/limited grandchild list) can
produce both a real match row and a NULL-extended LEFT JOIN row
sharing the same key, and an ORDER BY/LIMIT on top with no
de-duplication picks whichever the database's tie-breaking favors.
Both fast paths are now gated behind `supportsLateralJoin`, safe to
skip only when the dialect genuinely evaluates subqueries laterally,
or when there's provably no risk (no joins, or no offset/limit at all).
The offsetDateTime test codec normalizes to UTC on decode: SQLite has
no native timestamptz, so it stores and returns whatever literal
offset is written in the seed data, whereas Postgres/Oracle/MSSQL's
drivers hand back a UTC-normalized value regardless of storage;
without normalizing, otherwise-correct results failed pure
string-equality against the shared UTC ("Z")-formatted expected-JSON
fixtures.
Also applies the shared utcTestSettings (pinning the forked test JVM to
UTC) to doobiesqlite for consistency with the other backends, even
though SQLite's own datetime handling is UTC-normalized independent of
host time zone.
munit-cats-effect's 30s default is not enough headroom for the database-backed suites on a constrained machine: sbt runs each backend's tests in its own forked JVM, several of which contend for one docker daemon, and under that load individual tests have been observed to exceed 30s and fail with spurious TimeoutExceptions unrelated to what they test. Introduce a shared SqlDatabaseSuite base trait in sql-core's test scope, extended by DoobieDatabaseSuite and SqlPgDatabaseSuite (a single base rather than per-trait overrides, since DoobiePgDatabaseSuite mixes in both and separate overrides of munitIOTimeout would conflict), raising the timeout to 2 minutes for every backend suite.
defaultOffsetForSubquery supplied dialect-required offset/limit defaults only for selects rendered as subqueries, leaving the true query root unnormalized. That gap was invisible for Postgres and Oracle (identity implementations) and harmless for MSSQL (ORDER BY without OFFSET is legal at the root, unlike inside a derived table), but fatal for SQLite's comma-form LIMIT clause: a flat root query with an offset and no limit - the one shape whose offset survives to the top level unwrapped - rendered a dangling "LIMIT n," and failed at runtime with an incomplete-input error. Rename the hook to normalizeOffsetLimit and apply it once in SqlSelect.toFragment, so every rendered select is normalized wherever it appears; SubqueryRef no longer applies it separately. The only rendering change for existing dialects is that MSSQL root-level ordered queries now carry a redundant-but-legal "OFFSET 0 ROWS". Adds a shared regression test covering the flat root-offset shape, run by all backends.
SQLite sorts NULLs low by default (first in ASC, last in DESC), but orderToFragment carried the Postgres conditional, which only emits an explicit NULLS FIRST/LAST clause on the cases that deviate from a nulls-high default. On SQLite that renders the explicit clause exactly where the engine default already matches and stays silent on the two cases that need correcting, so orderings requesting the non-default placement (ascending nulls-last, descending nulls-first) put NULLs at the wrong end. Emit the clause on the mirror-image cases instead - the same polarity correction the MSSQL dialect makes, expressed with SQLite's native syntax. No existing fixture could observe this: grackle core re-sorts fetched rows in memory (stripCompiled preserves OrderBy), so the SQL-side placement only matters when it decides which rows survive a LIMIT cut, and no shared fixture orders a NULL-containing nullable column. The new NullOrderingSuite pins all four ascending/nullsLast combinations through a limit for exactly that reason.
phdoerfler
force-pushed
the
topic/mysql-backend
branch
from
July 17, 2026 13:50
21eaec0 to
9942895
Compare
Superseded by the shared SqlNullOrderingMapping/SqlNullOrderingSuite in sql-core, landing via typelevel#870. This branch will pick that up once it rebases past typelevel#870's merge.
Superseded by the shared SqlNullOrderingMapping/SqlNullOrderingSuite in sql-core, landing via typelevel#870. This branch will pick that up once it rebases past typelevel#870's merge.
phdoerfler
force-pushed
the
topic/mysql-backend
branch
from
July 17, 2026 20:49
9942895 to
69012fd
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This adds MySQL as a backend, same procedure as the SQLite (#864) and H2 (#866) work. Mostly done by AI, test first, with a review from the LLM itself and me looking over the mapping code. This one depends on #864 (it needs the lateral-join workaround machinery that PR introduces for the shared
SqlMappingLikehooks), so please merge that one first.I decided to use the mysql-connector-j driver rather than the mariadb driver in mysql compatibility mode. It is GPLv2 with the Universal FOSS Exception. It is added as a plain compile dependency (following the
ojdbc8precedent already in the tree for Oracle). The module has zero compile-time references to driver classes, so% Testscope would also work if that's preferred.This targets MySQL 8.0.14+ (first version with native
LATERALderived tables, which is the load-bearing feature this backend leans on). That floor is tested, not just asserted: the suite is green against MySQL 8.0.14, 8.0.46, and 8.4 (current LTS), via aMYSQL_VERSIONoverride on the docker-compose service. Not wired into permanent CI (didn't want to double CI cost/time for a one-off verification, and no other backend in the tree pins more than one version), but reproducible any time withMYSQL_VERSION=8.0.14 sbt doobiemysql/test.Summary
grackle-doobie-mysql: a doobie-based MySQL backend with full shared-test-suite parity against every other SQL backend (pg/oracle/mssql/sqlite/h2) — same suite classes, same fixtures ported over, 249 tests green on both Scala 2.13 and 3.3.8.Dialect notes
NULLS FIRST/LAST; its default is nulls-low (NULLs sort first under ASC, last under DESC). Emulated with aCASE WHEN col IS NULL THEN 1 ELSE 0 ENDsort-key prefix, added only where the requested placement differs from the default. Pinned by a dedicatedNullOrderingSuitethat orders-and-limits so the SQL-level placement is actually observable (grackle re-sorts fetched rows in memory with engine-independent semantics, so a naive whole-response comparison can't catch a dialect ordering bug, only a LIMIT cut can).OFFSETis illegal withoutLIMIT, same as SQLite, so it reuses SQLite's comma-form rendering (LIMIT offset, limit) and the samenormalizeOffsetLimitmechanism. MySQL has noLIMIT -1idiom, so an offset-without-limit query getsInt.MaxValueas a practical "unbounded" sentinel instead.*_cicollations make plainLIKEcase-insensitive already, so it's the case-sensitive branch that needs an explicitCOLLATE(now an overridablebinaryCollationmember, defaultutf8mb4_bin, for mappings over a non-default charset/collation).DISTINCT+ theFirstValueColumnwindow-function strategy already used by the MSSQL backend.CASTonly accepts a small fixed set of target names, not arbitrary column types — mapped explicitly; anything without a safe CAST target (e.g.FLOAT/DOUBLE, sinceCAST(... AS DOUBLE)needs 8.0.17, above this backend's 8.0.14 floor) renders a bareNULLinstead, which is safe since an ascribed NULL is only ever an inference hint.ORDER BY/LIMIT/OFFSETare legal, so no derived-table encapsulation is needed (unlike SQLite, which forbids this entirely).INSERT ... RETURNING, no sequences.AUTO_INCREMENT+ JDBC generated-keys (withUniqueGeneratedKeys), same pattern as the Oracle backend.Test infrastructure
The official
mysqldocker image'sdocker-entrypoint-initdb.dfixture loader negotiates alatin1client handshake by default, which double-encodes the UTF-8 fixture literals (accented city/country names).--skip-character-set-client-handshakewas removed in MySQL 8.3+, and a bind-mountedmy.cnfisn't reliable in every dev environment (some setups force bind-mounted files world-writable, whichmysqlrefuses to read). Worked around with a smallentrypoint.shthat writes a[client] default-character-set=utf8mb4config file fresh inside the container before handing off to the stock entrypoint.Known follow-up (not in this PR)
While implementing this I found MSSQL's
orderToFragmenthas an inverted CASE-key polarity in its DESC branch (a real bug,desc + nullsFirstsorts NULLs last instead of first). The LLM filed that separately as #870 rather than folding an unrelated fix into this PR.