Skip to content

Commit 275d301

Browse files
author
Wojciech Padło
committed
Snowflake: parse the EXTERNAL TABLE statement family
Snowflake's `CREATE EXTERNAL TABLE` grammar did not parse: the fork's `parse_create_external_table` is the Hive-shaped one and the Snowflake dialect never intercepted `CREATE EXTERNAL TABLE`, so realistic DDL died in the parser. This teaches the Snowflake dialect the whole external-table statement family, while leaving the Hive path (which other dialects — and Snowflake itself, for the `STORED AS … LOCATION '<path>'` form — rely on) untouched via a tail-shape fallback. Added under the Snowflake dialect: * `CREATE [OR REPLACE] EXTERNAL TABLE [IF NOT EXISTS]` with virtual column definitions (`<col> <type> AS <expr>`, parenthesised or bare), `LOCATION`, `FILE_FORMAT` (named and inline), `PATTERN`, `REFRESH_ON_CREATE`, `AUTO_REFRESH`, `PARTITION BY`, `PARTITION_TYPE`, `TABLE_FORMAT`, `AWS_SNS_TOPIC`, `COPY GRANTS`, tags, row-access policy and `COMMENT`. * `DROP EXTERNAL TABLE` and `DESC[RIBE] EXTERNAL TABLE` (new `ObjectType::ExternalTable` / `DescribeObjectType::ExternalTable`). * `ALTER EXTERNAL TABLE … ADD FILES / REMOVE FILES / SET AUTO_REFRESH / ADD PARTITION / DROP PARTITION`, alongside the existing `REFRESH`. New `CreateTable` fields (`pattern`, `refresh_on_create`, `partition_type`, `table_format`, `aws_sns_topic`) and `AlterTableOperation` variants carry the Snowflake-only clauses; `Display` round-trips every member back to the same AST. The `ADD PARTITION` column/value pairs use a dedicated `ExternalTablePartitionColumn` struct so the `visitor` derive is satisfied. Deferred/rejected members (`TABLE_FORMAT = DELTA`, `PARTITION_TYPE = USER_SPECIFIED`, `ADD`/`DROP PARTITION`) parse so they can be rejected downstream rather than aborting a batch in the parser. Also brings the branch to a green CI baseline: fills in several stale full `CreateTable` struct literals in the duckdb/mssql/postgres tests, applies `cargo fmt`, clears `clippy -D warnings`, and fixes two rustdoc errors.
1 parent 94d37ca commit 275d301

15 files changed

Lines changed: 714 additions & 108 deletions

‎examples/cli.rs‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -71,9 +71,9 @@ $ cargo run --example cli - [--dialectname]
7171
.expect("failed to read from stdin");
7272
String::from_utf8(buf).expect("stdin content wasn't valid utf8")
7373
} else {
74-
println!("Parsing from file '{}' using {:?}", &filename, dialect);
74+
println!("Parsing from file '{}' using {:?}", filename, dialect);
7575
fs::read_to_string(&filename)
76-
.unwrap_or_else(|_| panic!("Unable to read the file {}", &filename))
76+
.unwrap_or_else(|_| panic!("Unable to read the file {}", filename))
7777
};
7878
let without_bom = if contents.chars().next().unwrap() as u64 != 0xfeff {
7979
contents.as_str()

‎src/ast/ddl.rs‎

Lines changed: 161 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -121,6 +121,24 @@ impl fmt::Display for ReplicaIdentity {
121121
}
122122
}
123123

124+
/// A single `<column> = '<value>'` pair of a Snowflake external-table
125+
/// `ADD PARTITION` clause.
126+
#[derive(Debug, Clone, PartialEq, PartialOrd, Eq, Ord, Hash)]
127+
#[cfg_attr(feature = "serde", derive(Serialize, Deserialize))]
128+
#[cfg_attr(feature = "visitor", derive(Visit, VisitMut))]
129+
pub struct ExternalTablePartitionColumn {
130+
/// The partition column.
131+
pub column: Ident,
132+
/// The partition value, always a string literal.
133+
pub value: String,
134+
}
135+
136+
impl fmt::Display for ExternalTablePartitionColumn {
137+
fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
138+
write!(f, "{} = '{}'", self.column, self.value)
139+
}
140+
}
141+
124142
/// An `ALTER TABLE` (`Statement::AlterTable`) operation
125143
#[derive(Debug, Clone, PartialEq, PartialOrd, Eq, Ord, Hash)]
126144
#[cfg_attr(feature = "serde", derive(Serialize, Deserialize))]
@@ -498,6 +516,44 @@ pub enum AlterTableOperation {
498516
/// Optional subpath for external table refresh
499517
subpath: Option<String>,
500518
},
519+
/// `ADD FILES ( '<path>' [, ... ] )`
520+
///
521+
/// Snowflake external table: register specific staged files.
522+
/// <https://docs.snowflake.com/en/sql-reference/sql/alter-external-table>
523+
AddFiles {
524+
/// Relative staged file paths to register.
525+
files: Vec<String>,
526+
},
527+
/// `REMOVE FILES ( '<path>' [, ... ] )`
528+
///
529+
/// Snowflake external table: unregister specific staged files.
530+
RemoveFiles {
531+
/// Relative staged file paths to unregister.
532+
files: Vec<String>,
533+
},
534+
/// `SET AUTO_REFRESH = { TRUE | FALSE }`
535+
///
536+
/// Snowflake external table auto-refresh toggle.
537+
SetAutoRefresh {
538+
/// The new auto-refresh value.
539+
value: bool,
540+
},
541+
/// `ADD PARTITION ( <col> = '<value>' [, ... ] ) LOCATION '<path>'`
542+
///
543+
/// Snowflake user-specified partition addition (external table).
544+
AddExternalPartition {
545+
/// Column/value pairs defining the partition.
546+
partitions: Vec<ExternalTablePartitionColumn>,
547+
/// The staged subpath the partition maps to.
548+
location: String,
549+
},
550+
/// `DROP PARTITION LOCATION '<path>'`
551+
///
552+
/// Snowflake user-specified partition removal (external table).
553+
DropExternalPartition {
554+
/// The staged subpath whose partition is dropped.
555+
location: String,
556+
},
501557
/// `SUSPEND`
502558
///
503559
/// Note: this is Snowflake specific for dynamic tables <https://docs.snowflake.com/en/sql-reference/sql/alter-table>
@@ -1068,6 +1124,48 @@ impl fmt::Display for AlterTableOperation {
10681124
}
10691125
Ok(())
10701126
}
1127+
AlterTableOperation::AddFiles { files } => {
1128+
write!(
1129+
f,
1130+
"ADD FILES ({})",
1131+
files
1132+
.iter()
1133+
.map(|file| format!("'{file}'"))
1134+
.collect::<Vec<_>>()
1135+
.join(", ")
1136+
)
1137+
}
1138+
AlterTableOperation::RemoveFiles { files } => {
1139+
write!(
1140+
f,
1141+
"REMOVE FILES ({})",
1142+
files
1143+
.iter()
1144+
.map(|file| format!("'{file}'"))
1145+
.collect::<Vec<_>>()
1146+
.join(", ")
1147+
)
1148+
}
1149+
AlterTableOperation::SetAutoRefresh { value } => {
1150+
write!(
1151+
f,
1152+
"SET AUTO_REFRESH = {}",
1153+
if *value { "TRUE" } else { "FALSE" }
1154+
)
1155+
}
1156+
AlterTableOperation::AddExternalPartition {
1157+
partitions,
1158+
location,
1159+
} => {
1160+
write!(
1161+
f,
1162+
"ADD PARTITION ({}) LOCATION '{location}'",
1163+
display_comma_separated(partitions)
1164+
)
1165+
}
1166+
AlterTableOperation::DropExternalPartition { location } => {
1167+
write!(f, "DROP PARTITION LOCATION '{location}'")
1168+
}
10711169
AlterTableOperation::Suspend => {
10721170
write!(f, "SUSPEND")
10731171
}
@@ -2477,7 +2575,7 @@ pub(crate) fn display_option_spaced<T: fmt::Display>(option: &Option<T>) -> impl
24772575
///
24782576
/// `ENABLE`/`DISABLE`, `VALIDATE`/`NOVALIDATE` and `RELY`/`NORELY` are only
24792577
/// parsed for dialects returning true from
2480-
/// [`Dialect::supports_informational_constraint_properties`].
2578+
/// `Dialect::supports_informational_constraint_properties`.
24812579
#[derive(Debug, Copy, Clone, PartialEq, PartialOrd, Default, Eq, Ord, Hash)]
24822580
#[cfg_attr(feature = "serde", derive(Serialize, Deserialize))]
24832581
#[cfg_attr(feature = "visitor", derive(Visit, VisitMut))]
@@ -3324,7 +3422,7 @@ pub struct CreateTable {
33243422
/// bare keyword such as `DISABLE`); the value is stored verbatim.
33253423
/// <https://docs.snowflake.com/en/sql-reference/sql/create-dynamic-table>
33263424
pub scheduler: Option<String>,
3327-
/// Snowflake "IMMUTABLE WHERE (<predicate>)" clause for dynamic tables.
3425+
/// Snowflake `IMMUTABLE WHERE (<predicate>)` clause for dynamic tables.
33283426
/// Stored as the serialized predicate text so its casing survives.
33293427
/// <https://docs.snowflake.com/en/sql-reference/sql/create-dynamic-table>
33303428
pub immutable_where: Option<String>,
@@ -3349,6 +3447,34 @@ pub struct CreateTable {
33493447
/// Redshift `BACKUP` option: `BACKUP { YES | NO }`
33503448
/// <https://docs.aws.amazon.com/redshift/latest/dg/r_CREATE_TABLE_NEW.html>
33513449
pub backup: Option<bool>,
3450+
/// Snowflake external table `PATTERN = '<regex>'` clause.
3451+
/// <https://docs.snowflake.com/en/sql-reference/sql/create-external-table>
3452+
pub pattern: Option<String>,
3453+
/// Snowflake external table `REFRESH_ON_CREATE = { TRUE | FALSE }` clause.
3454+
pub refresh_on_create: Option<bool>,
3455+
/// Snowflake external table `PARTITION_TYPE = { USER_SPECIFIED | ... }` clause.
3456+
pub partition_type: Option<String>,
3457+
/// Snowflake external table `TABLE_FORMAT = { DELTA | ... }` clause.
3458+
pub table_format: Option<String>,
3459+
/// Snowflake external table `AWS_SNS_TOPIC = '<arn>'` clause.
3460+
pub aws_sns_topic: Option<String>,
3461+
}
3462+
3463+
impl CreateTable {
3464+
/// Whether this is a Snowflake-shaped `CREATE EXTERNAL TABLE` (which renders
3465+
/// `LOCATION=@stage FILE_FORMAT=(...)` etc.) as opposed to the Hive form
3466+
/// (`STORED AS ... LOCATION '...'`). The two grammars share the `external`
3467+
/// flag but never the Snowflake-only clauses.
3468+
fn is_snowflake_external(&self) -> bool {
3469+
self.external
3470+
&& self.file_format.is_none()
3471+
&& (self.stage_file_format.is_some()
3472+
|| self.pattern.is_some()
3473+
|| self.refresh_on_create.is_some()
3474+
|| self.partition_type.is_some()
3475+
|| self.table_format.is_some()
3476+
|| self.aws_sns_topic.is_some())
3477+
}
33523478
}
33533479

33543480
impl fmt::Display for CreateTable {
@@ -3510,14 +3636,41 @@ impl fmt::Display for CreateTable {
35103636
}
35113637
}
35123638
}
3513-
if self.external {
3639+
if self.external && !self.is_snowflake_external() {
35143640
if let Some(file_format) = self.file_format {
35153641
write!(f, " STORED AS {file_format}")?;
35163642
}
35173643
if let Some(location) = &self.location {
35183644
write!(f, " LOCATION '{location}'")?;
35193645
}
35203646
}
3647+
if self.is_snowflake_external() {
3648+
if let Some(location) = &self.location {
3649+
write!(f, " LOCATION={location}")?;
3650+
}
3651+
if let Some(stage_file_format) = &self.stage_file_format {
3652+
write!(f, " FILE_FORMAT=({stage_file_format})")?;
3653+
}
3654+
if let Some(pattern) = &self.pattern {
3655+
write!(f, " PATTERN='{pattern}'")?;
3656+
}
3657+
if let Some(refresh_on_create) = self.refresh_on_create {
3658+
write!(
3659+
f,
3660+
" REFRESH_ON_CREATE={}",
3661+
if refresh_on_create { "TRUE" } else { "FALSE" }
3662+
)?;
3663+
}
3664+
if let Some(partition_type) = &self.partition_type {
3665+
write!(f, " PARTITION_TYPE={partition_type}")?;
3666+
}
3667+
if let Some(table_format) = &self.table_format {
3668+
write!(f, " TABLE_FORMAT={table_format}")?;
3669+
}
3670+
if let Some(aws_sns_topic) = &self.aws_sns_topic {
3671+
write!(f, " AWS_SNS_TOPIC='{aws_sns_topic}'")?;
3672+
}
3673+
}
35213674

35223675
match &self.table_options {
35233676
options @ CreateTableOptions::With(_)
@@ -3601,8 +3754,10 @@ impl fmt::Display for CreateTable {
36013754
)?;
36023755
}
36033756

3604-
if let Some(stage_file_format) = &self.stage_file_format {
3605-
write!(f, " STAGE_FILE_FORMAT=({stage_file_format})")?;
3757+
if !self.is_snowflake_external() {
3758+
if let Some(stage_file_format) = &self.stage_file_format {
3759+
write!(f, " STAGE_FILE_FORMAT=({stage_file_format})")?;
3760+
}
36063761
}
36073762

36083763
if let Some(data_retention_time_in_days) = self.data_retention_time_in_days {
@@ -5034,7 +5189,7 @@ impl fmt::Display for AlterTable {
50345189
if self.only {
50355190
write!(f, "ONLY ")?;
50365191
}
5037-
write!(f, "{} ", &self.name)?;
5192+
write!(f, "{} ", self.name)?;
50385193
if let Some(cluster) = &self.on_cluster {
50395194
write!(f, "ON CLUSTER {cluster} ")?;
50405195
}

‎src/ast/helpers/stmt_create_table.rs‎

Lines changed: 50 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -196,6 +196,16 @@ pub struct CreateTableBuilder {
196196
pub sortkey: Option<Vec<Expr>>,
197197
/// Redshift `BACKUP` option.
198198
pub backup: Option<bool>,
199+
/// Snowflake external table `PATTERN` clause.
200+
pub pattern: Option<String>,
201+
/// Snowflake external table `REFRESH_ON_CREATE` clause.
202+
pub refresh_on_create: Option<bool>,
203+
/// Snowflake external table `PARTITION_TYPE` clause.
204+
pub partition_type: Option<String>,
205+
/// Snowflake external table `TABLE_FORMAT` clause.
206+
pub table_format: Option<String>,
207+
/// Snowflake external table `AWS_SNS_TOPIC` clause.
208+
pub aws_sns_topic: Option<String>,
199209
}
200210

201211
impl CreateTableBuilder {
@@ -267,6 +277,11 @@ impl CreateTableBuilder {
267277
distkey: None,
268278
sortkey: None,
269279
backup: None,
280+
pattern: None,
281+
refresh_on_create: None,
282+
partition_type: None,
283+
table_format: None,
284+
aws_sns_topic: None,
270285
}
271286
}
272287
/// Set `OR REPLACE` for the CREATE TABLE statement.
@@ -595,6 +610,31 @@ impl CreateTableBuilder {
595610
self.backup = backup;
596611
self
597612
}
613+
/// Set the Snowflake external table `PATTERN` clause.
614+
pub fn pattern(mut self, pattern: Option<String>) -> Self {
615+
self.pattern = pattern;
616+
self
617+
}
618+
/// Set the Snowflake external table `REFRESH_ON_CREATE` clause.
619+
pub fn refresh_on_create(mut self, refresh_on_create: Option<bool>) -> Self {
620+
self.refresh_on_create = refresh_on_create;
621+
self
622+
}
623+
/// Set the Snowflake external table `PARTITION_TYPE` clause.
624+
pub fn partition_type(mut self, partition_type: Option<String>) -> Self {
625+
self.partition_type = partition_type;
626+
self
627+
}
628+
/// Set the Snowflake external table `TABLE_FORMAT` clause.
629+
pub fn table_format(mut self, table_format: Option<String>) -> Self {
630+
self.table_format = table_format;
631+
self
632+
}
633+
/// Set the Snowflake external table `AWS_SNS_TOPIC` clause.
634+
pub fn aws_sns_topic(mut self, aws_sns_topic: Option<String>) -> Self {
635+
self.aws_sns_topic = aws_sns_topic;
636+
self
637+
}
598638
/// Consume the builder and produce a `CreateTable`.
599639
pub fn build(self) -> CreateTable {
600640
CreateTable {
@@ -663,6 +703,11 @@ impl CreateTableBuilder {
663703
distkey: self.distkey,
664704
sortkey: self.sortkey,
665705
backup: self.backup,
706+
pattern: self.pattern,
707+
refresh_on_create: self.refresh_on_create,
708+
partition_type: self.partition_type,
709+
table_format: self.table_format,
710+
aws_sns_topic: self.aws_sns_topic,
666711
}
667712
}
668713
}
@@ -750,6 +795,11 @@ impl From<CreateTable> for CreateTableBuilder {
750795
distkey: table.distkey,
751796
sortkey: table.sortkey,
752797
backup: table.backup,
798+
pattern: table.pattern,
799+
refresh_on_create: table.refresh_on_create,
800+
partition_type: table.partition_type,
801+
table_format: table.table_format,
802+
aws_sns_topic: table.aws_sns_topic,
753803
}
754804
}
755805
}

0 commit comments

Comments
 (0)