Conversation
Spark 4 infers ISO-formatted Date columns in Azure CSV exports as DateType, making the getString() call in the azure FOCUSColumns module throw a ClassCastException. Keep the richer type instead of casting back to string: - normalise Date to DateType at load time with try_to_date over the known Azure formats (a plain cast would throw under ANSI mode for MM/dd/yyyy values) - declare AzureColumn.DATE as DateType and add RowColumn.getDate(), which accepts java.sql.Date, LocalDate and, as a fallback, strings - drop the format-guessing nextDay() from the module; ChargePeriodStart and ChargePeriodEnd are now always emitted as ISO dates - extract the Azure CSV fix-ups into SparkJob.normalizeAzureColumns() and cover it with tests that read real CSVs through inferSchema, which rows built by hand could not catch; surefire gets the add-opens needed to run an embedded Spark session on JDK 17+
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #246
Spark 4 infers ISO-formatted
Datecolumns in Azure CSV exports asDateType, so thegetString()call in the azureFOCUSColumnsmodule threw aClassCastException. As discussed on the issue, this keeps the richer type rather than casting back to string.Changes
normalizeAzureColumns()method, which now also normalisesDatetoDateTypewithcoalesce(try_to_date(...))over the known formats (ISO,MM/dd/yyyy,M/d/yyyy).try_to_daterather than a cast because ANSI mode makes a failed cast throw instead of yielding null.DATEis now declaredDateType.getDate(Row)accessor returningLocalDate; acceptsjava.sql.Date,LocalDate(Java 8 datetime API) and falls back to parsing strings.getDate()and doesplusDays(1); the format-guessingnextDay()is gone. Side effect:ChargePeriodStart/Endare now always emitted as ISO dates, even forMM/dd/yyyyinputs.--add-openslist (keeping the jacoco agent via@{argLine}), needed for the new test.FOCUSColumnsTestbuilds rows with real date values plus string-fallback coverage, and a newSparkJobTestreads two small CSVs through actualinferSchema— asserting the inference premise and that enrichment no longer throws — since rows built by hand could not catch this.Verification
Beyond the unit tests (316 pass), ran
SparkJobend-to-end onazure-examples/EA-Cost-Actual.csvboth as-is (US dates) and with theDatecolumn converted to ISO (the Spark 4DateTypeinference scenario from the issue): both runs complete,Datelands as a properDATEin the output parquet, charge periods are ISO strings and the emissions columns are populated.Left out deliberately
Typing
ChargePeriodStart/Endas dates (raised in the issue discussion): the AWS bridge writes hourly CUR timestamps into the same columns, so that needs its own PR.