fix(pgsql-source): use CAST(col AS text) instead of col::text for Redshift#1746
Merged
Conversation
…shift (#1549) Redshift does not support the col::text cast operator for all type combinations — in particular 'cannot cast type boolean to character varying' is raised for any boolean column when reading from a Redshift source. The ANSI SQL CAST(expression AS type) syntax is supported by both standard PostgreSQL and Redshift, so replace the PostgreSQL-specific ::text shorthand with it everywhere a source SELECT casts columns to text: v3 src/sources/pgsql/pgsql.lisp SELECT col::text → SELECT CAST(col AS text) v4 clojure/src/pgloader/source/pgsql.clj SELECT "col" → SELECT CAST("col" AS text) (v4 previously relied on JDBC .getObject()+str for implicit conversion; explicit casting is safer and makes Redshift compatibility explicit) v4 clojure/src/pgloader/ddl/citus.clj tbl.col::text → CAST(tbl.col AS text) (Citus backfill SELECT, pgsql-source path)
…rce compatibility The Citus backfill SELECT is generated against the SOURCE database. When the source is Redshift, col::text fails for boolean and other types that Redshift does not support via the :: operator. Using CAST(col AS text) instead is semantically identical on PostgreSQL and Redshift both supports it for all types. Update the test/citus/README.md example to reflect the new SQL.
The test-augment-catalog-pg-source-type test was asserting that the generated Citus backfill SQL contained ::text, which is no longer true after switching to CAST(col AS text) for Redshift compatibility. Update the assertion and description to match the new syntax, and add pgloader.ddl.citus-test to the local Makefile test-unit target so it matches what CI runs.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #1549.
Problem
When using Redshift as the data source for a pgsql-to-pgsql migration, pgloader generates a SELECT that casts every column to text using PostgreSQL's
::shorthand:Redshift raises:
The
::cast operator is a PostgreSQL extension that Redshift only partially supports — boolean→text is not allowed via::in Redshift. The ANSI SQLCAST(expr AS type)syntax works for all types on both engines.Affected code paths
There are four call-sites, all now fixed:
src/sources/pgsql/pgsql.lisp~s::textCAST(~s AS text)src/utils/citus.lisp~a::textCAST(~a AS text)clojure/src/pgloader/source/pgsql.clj"col"CAST("col" AS text)clojure/src/pgloader/ddl/citus.cljtbl.col::textCAST(tbl.col AS text)The Citus backfill SELECT is generated against the source database. In a Redshift→Citus migration (Redshift as source, Citus-distributed PostgreSQL as target), pgloader must JOIN source tables to retrieve the distribution key — that JOIN query runs on Redshift, so it also needs
CASTsyntax. The test/citus/README.md example is updated to reflect the generated SQL.Testing
CAST(col AS text)andcol::textare semantically identical; existing regression tests cover this path.test/citus/documents the generated SQL; the updated README shows the new form.