Skip to content

Handle Azure Date column inferred as DateType by Spark 4#255

Merged
jnioche merged 1 commit into
mainfrom
246
Jul 16, 2026
Merged

Handle Azure Date column inferred as DateType by Spark 4#255
jnioche merged 1 commit into
mainfrom
246

Conversation

@jnioche

@jnioche jnioche commented Jul 16, 2026

Copy link
Copy Markdown
Member

Fixes #246

Spark 4 infers ISO-formatted Date columns in Azure CSV exports as DateType, so the getString() call in the azure FOCUSColumns module threw a ClassCastException. As discussed on the issue, this keeps the richer type rather than casting back to string.

Changes

  • SparkJob: the Azure CSV post-load fix-ups are extracted into a testable normalizeAzureColumns() method, which now also normalises Date to DateType with coalesce(try_to_date(...)) over the known formats (ISO, MM/dd/yyyy, M/d/yyyy). try_to_date rather than a cast because ANSI mode makes a failed cast throw instead of yielding null.
  • AzureColumn: DATE is now declared DateType.
  • RowColumn: new getDate(Row) accessor returning LocalDate; accepts java.sql.Date, LocalDate (Java 8 datetime API) and falls back to parsing strings.
  • azure/FOCUSColumns: reads the date with getDate() and does plusDays(1); the format-guessing nextDay() is gone. Side effect: ChargePeriodStart/End are now always emitted as ISO dates, even for MM/dd/yyyy inputs.
  • pom.xml: surefire gets the standard Spark-on-JDK17+ --add-opens list (keeping the jacoco agent via @{argLine}), needed for the new test.
  • Tests: FOCUSColumnsTest builds rows with real date values plus string-fallback coverage, and a new SparkJobTest reads two small CSVs through actual inferSchema — asserting the inference premise and that enrichment no longer throws — since rows built by hand could not catch this.

Verification

Beyond the unit tests (316 pass), ran SparkJob end-to-end on azure-examples/EA-Cost-Actual.csv both as-is (US dates) and with the Date column converted to ISO (the Spark 4 DateType inference scenario from the issue): both runs complete, Date lands as a proper DATE in the output parquet, charge periods are ISO strings and the emissions columns are populated.

Left out deliberately

Typing ChargePeriodStart/End as dates (raised in the issue discussion): the AWS bridge writes hourly CUR timestamps into the same columns, so that needs its own PR.

Spark 4 infers ISO-formatted Date columns in Azure CSV exports as
DateType, making the getString() call in the azure FOCUSColumns module
throw a ClassCastException.

Keep the richer type instead of casting back to string:
- normalise Date to DateType at load time with try_to_date over the
  known Azure formats (a plain cast would throw under ANSI mode for
  MM/dd/yyyy values)
- declare AzureColumn.DATE as DateType and add RowColumn.getDate(),
  which accepts java.sql.Date, LocalDate and, as a fallback, strings
- drop the format-guessing nextDay() from the module; ChargePeriodStart
  and ChargePeriodEnd are now always emitted as ISO dates
- extract the Azure CSV fix-ups into SparkJob.normalizeAzureColumns()
  and cover it with tests that read real CSVs through inferSchema,
  which rows built by hand could not catch; surefire gets the add-opens
  needed to run an embedded Spark session on JDK 17+
@jnioche jnioche added this to the 1.2 milestone Jul 16, 2026
@jnioche jnioche added bug Something isn't working Azure labels Jul 16, 2026
@jnioche
jnioche merged commit 6cba43a into main Jul 16, 2026
5 checks passed
@jnioche
jnioche deleted the 246 branch July 16, 2026 09:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Azure bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: Azure FinOpsColumns fails when Spark infers Date column as DateType

1 participant