Conversation
report.py and dashboard.py now query only standard FOCUS columns (ServiceName, ChargeCategory, BilledCost, ListCost, SkuMeter, Tags) plus the SPRUCE-generated ones, so the same queries run on the enriched output of every pipeline (AWS CUR, AWS FOCUS, Azure, Azure FOCUS). No provider-specific x_ columns are used. - Instance types are derived from SkuMeter alone (FOCUS 1.2); inputs that do not populate it get an empty section - Tag breakdowns support Tags stored as a MAP (AWS) or a JSON string (Azure) - BILLING_PERIOD is inferred from ChargePeriodStart when the input is not hive-partitioned by billing period - Known columns are aliased to their canonical case when loading, so e.g. a billing_period parquet column works like the BILLING_PERIOD hive partition - Missing columns are injected as typed NULLs so every query runs everywhere - Update reporting README and docs to match
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #239
Makes the reporting scripts (
reporting/report.py,reporting/dashboard.py) provider-neutral by querying the FOCUS (FinOps Open Cost & Usage Specification) columns emitted by all SPRUCE pipelines, instead of raw AWS CUR columns.What changed
Pipeline side
FOCUSColumnsbridge modules map provider-native billing columns to FOCUS names (BilledCost,ServiceName,ChargeCategory,RegionId,SubAccountId,ChargePeriodStart/End,Tags,SkuMeter, ...) for the native AWS and Azure pipelines; FOCUS exports already carry themInstanceTypeExtractionmodules andSpruceColumn.INSTANCE_TYPEwere removed; the instance type is now derived fromSkuMeteron the reporting sideReporting side
x_columnsSkuMeter(FOCUS 1.2, e.g.EUW2-BoxUsage:t3.xlarge); inputs that do not populate it get an empty sectionTagsstored as aMAP(AWS) or a JSON string (Azure)BILLING_PERIODis inferred fromChargePeriodStartwhen the input is not hive-partitioned by billing period (handles ISO timestamps, ISO strings, and US-style Azure dates)Testing
Report and dashboard verified end-to-end against four enriched datasets: AWS CUR (hourly, hive-partitioned), AWS FOCUS (daily, flat), Azure cost details, and Azure FOCUS. Totals match across export formats of the same account; per-service splits legitimately differ where the provider's own
ServiceNametaxonomy differs from meter categories.