From a0d19b8fd5e1625bbf6c5cddc539458d982dd889 Mon Sep 17 00:00:00 2001 From: Mitali Daduria Date: Sun, 12 Jul 2026 08:58:46 +0530 Subject: [PATCH 1/9] Add machine learning and data governance concepts Expanded the metadata standard documentation to include detailed explanations of machine learning concepts such as model drift, data lineage, feature engineering, label skew, and data quality SLA. --- v1.13.x/main-concepts/metadata-standard.mdx | 27 +++++++++++++++++++++ 1 file changed, 27 insertions(+) diff --git a/v1.13.x/main-concepts/metadata-standard.mdx b/v1.13.x/main-concepts/metadata-standard.mdx index dcf71c7b..1fdbab1f 100644 --- a/v1.13.x/main-concepts/metadata-standard.mdx +++ b/v1.13.x/main-concepts/metadata-standard.mdx @@ -43,3 +43,30 @@ Ready to dive in? Visit openmetadatastandards.org You'll find quick starts, schema references, semantic web resources, and contribution guides on the official site. + +### Machine Learning & Data Governance Expansion + +#### 1. Model Drift +* **Technical Definition:** The degradation of a machine learning model's predictive performance over time due to changes in the statistical distribution of the environment's underlying data. +* **Business Translation:** When a previously accurate AI model starts making poorer predictions because real-world consumer behavior or market conditions have evolved since the model was trained. +* **Payments Example:** A fraud detection model trained on historical data from early 2024 may experience seasonal model drift during the November/December holiday shopping rush, as legitimate spike buying behaviors mimic historic fraud patterns. + +#### 2. Data Lineage +* **Technical Definition:** The end-to-end lifecycle of data that maps its origins, structural transformations, intermediate pipeline stages, and ultimate consumption endpoints across the data ecosystem. +* **Business Translation:** A visual audit trail that shows exactly where a piece of data came from, how it was modified along the way, and which final business dashboards or reports are using it. +* **Payments Example:** Tracing an `interchange_fee_credited` field from the raw settlement API payload, through the daily ETL cleansing pipelines, to the monthly CFO financial compliance dashboard. + +#### 3. Feature Engineering +* **Technical Definition:** The process of using domain knowledge to select, transform, combine, and manipulate raw data variables into distinct predictor variables (features) that enhance machine learning algorithm performance. +* **Business Translation:** Extracting and shaping raw, unorganized data points into meaningful business metrics so an AI model can spot patterns more easily. +* **Payments Example:** Converting raw, individual transaction timestamps into a calculated feature like `count_of_transactions_in_last_5_minutes` to help a model identify high-frequency card-cloning velocity attacks. + +#### 4. Label Skew +* **Technical Definition:** A specific type of dataset imbalance occurring during training or evaluation where one target class label vastly outnumbers the alternative class labels. +* **Business Translation:** A training bias that happens when the data given to an AI contains overwhelming examples of one outcome and barely any examples of another, making it blind to rarer events. +* **Payments Example:** In a payment dataset of 10 million transactions, only 0.01% are actual chargebacks. Training a model on this creates severe label skew, causing the system to default to guessing "legitimate" to achieve high but hollow accuracy. + +#### 5. Data Quality SLA (Service Level Agreement) +* **Technical Definition:** A formal commitment that defines measurable performance thresholds for critical data characteristics—such as freshness, schema adherence, completeness, and validity—enforced by automated testing pipelines. +* **Business Translation:** A contract between data teams and business stakeholders ensuring that the data powering operations arrives on time, with full completeness, and free of format errors. +* **Payments Example:** A data contract specifying that the `merchant_settlement_ledger` table must be fully populated and accurate by 06:00 UTC daily, with a hard failure alert triggered if missing row counts exceed 0.05%. From f0c8583b9bcfb4a9bbe3cf728b5a9e3e4ed1b5b0 Mon Sep 17 00:00:00 2001 From: Mitali Daduria Date: Sun, 12 Jul 2026 09:00:07 +0530 Subject: [PATCH 2/9] Fix formatting in metadata-standard.mdx --- v1.13.x/main-concepts/metadata-standard.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/v1.13.x/main-concepts/metadata-standard.mdx b/v1.13.x/main-concepts/metadata-standard.mdx index 1fdbab1f..6daca8ce 100644 --- a/v1.13.x/main-concepts/metadata-standard.mdx +++ b/v1.13.x/main-concepts/metadata-standard.mdx @@ -14,7 +14,7 @@ Key highlights: - **API-first**: REST and event schemas for search, feeds, webhooks, and bulk operations. - **Semantic web ready**: RDF/OWL ontologies and SHACL shapes for knowledge graph and linked data use cases. - **Standards compliant**: JSON Schema (Draft 07/2020-12), RDF/OWL, SHACL, JSON-LD, PROV-O. -- **Community driven**: Built and maintained by the OpenMetadata community with open governance. +- **Community driven**: Built and maintained by the OpenMetadata community with open governance. ## Schemas From 2489202c9404d11d9b4ebc9ea8075b38174b468b Mon Sep 17 00:00:00 2001 From: Mitali Daduria Date: Mon, 13 Jul 2026 12:01:38 +0530 Subject: [PATCH 3/9] Standardize machine learning terms and examples; docs: fix formatting and styling per review --- v1.13.x/main-concepts/metadata-standard.mdx | 51 +++++++++++---------- 1 file changed, 26 insertions(+), 25 deletions(-) diff --git a/v1.13.x/main-concepts/metadata-standard.mdx b/v1.13.x/main-concepts/metadata-standard.mdx index 6daca8ce..b98df4bc 100644 --- a/v1.13.x/main-concepts/metadata-standard.mdx +++ b/v1.13.x/main-concepts/metadata-standard.mdx @@ -45,28 +45,29 @@ Ready to dive in? You'll find quick starts, schema references, semantic web resources, and contribution guides on the official site. ### Machine Learning & Data Governance Expansion - -#### 1. Model Drift -* **Technical Definition:** The degradation of a machine learning model's predictive performance over time due to changes in the statistical distribution of the environment's underlying data. -* **Business Translation:** When a previously accurate AI model starts making poorer predictions because real-world consumer behavior or market conditions have evolved since the model was trained. -* **Payments Example:** A fraud detection model trained on historical data from early 2024 may experience seasonal model drift during the November/December holiday shopping rush, as legitimate spike buying behaviors mimic historic fraud patterns. - -#### 2. Data Lineage -* **Technical Definition:** The end-to-end lifecycle of data that maps its origins, structural transformations, intermediate pipeline stages, and ultimate consumption endpoints across the data ecosystem. -* **Business Translation:** A visual audit trail that shows exactly where a piece of data came from, how it was modified along the way, and which final business dashboards or reports are using it. -* **Payments Example:** Tracing an `interchange_fee_credited` field from the raw settlement API payload, through the daily ETL cleansing pipelines, to the monthly CFO financial compliance dashboard. - -#### 3. Feature Engineering -* **Technical Definition:** The process of using domain knowledge to select, transform, combine, and manipulate raw data variables into distinct predictor variables (features) that enhance machine learning algorithm performance. -* **Business Translation:** Extracting and shaping raw, unorganized data points into meaningful business metrics so an AI model can spot patterns more easily. -* **Payments Example:** Converting raw, individual transaction timestamps into a calculated feature like `count_of_transactions_in_last_5_minutes` to help a model identify high-frequency card-cloning velocity attacks. - -#### 4. Label Skew -* **Technical Definition:** A specific type of dataset imbalance occurring during training or evaluation where one target class label vastly outnumbers the alternative class labels. -* **Business Translation:** A training bias that happens when the data given to an AI contains overwhelming examples of one outcome and barely any examples of another, making it blind to rarer events. -* **Payments Example:** In a payment dataset of 10 million transactions, only 0.01% are actual chargebacks. Training a model on this creates severe label skew, causing the system to default to guessing "legitimate" to achieve high but hollow accuracy. - -#### 5. Data Quality SLA (Service Level Agreement) -* **Technical Definition:** A formal commitment that defines measurable performance thresholds for critical data characteristics—such as freshness, schema adherence, completeness, and validity—enforced by automated testing pipelines. -* **Business Translation:** A contract between data teams and business stakeholders ensuring that the data powering operations arrives on time, with full completeness, and free of format errors. -* **Payments Example:** A data contract specifying that the `merchant_settlement_ledger` table must be fully populated and accurate by 06:00 UTC daily, with a hard failure alert triggered if missing row counts exceed 0.05%. +The following terms bridge common machine learning and data governance concepts to business-friendly language, with payments-domain examples showing how each one shows up in practice. + + 1. Model Drift +* **Technical Definition**: The degradation of a machine learning model's predictive performance over time due to changes in the statistical distribution of the environment's underlying data. +* **Business Translation**: When a previously accurate AI model starts making poorer predictions because real-world consumer behavior or market conditions have evolved since the model was trained. +* **Payments Example**: A fraud detection model trained on historical data from early 2024 may experience seasonal model drift during the November/December holiday shopping rush, as legitimate spike buying behaviors mimic historic fraud patterns. + + 2. Data Lineage +* **Technical Definition**: The end-to-end lifecycle of data that maps its origins, structural transformations, intermediate pipeline stages, and ultimate consumption endpoints across the data ecosystem. +* **Business Translation**: A visual audit trail that shows exactly where a piece of data came from, how it was modified along the way, and which final business dashboards or reports are using it. +* **Payments Example**: Tracing an `interchange_fee_credited` field from the raw settlement API payload, through the daily ETL cleansing pipelines, to the monthly CFO financial compliance dashboard. + + 3. Feature Engineering +* **Technical Definition**: The process of using domain knowledge to select, transform, combine, and manipulate raw data variables into distinct predictor variables (features) that enhance machine learning algorithm performance. +* **Business Translation**: Extracting and shaping raw, unorganized data points into meaningful business metrics so an AI model can spot patterns more easily. +* **Payments Example**: Converting raw, individual transaction timestamps into a calculated feature like `count_of_transactions_in_last_5_minutes` to help a model identify high-frequency card-cloning velocity attacks. + + 4. Label Skew +* **Technical Definition**: A specific type of dataset imbalance occurring during training or evaluation where one target class label vastly outnumbers the alternative class labels. +* **Business Translation**: A training bias that happens when the data given to an AI contains overwhelming examples of one outcome and barely any examples of another, making it blind to rarer events. +* **Payments Example**; In a payment dataset of 10 million transactions, only 0.01% are actual chargebacks. Training a model on this creates severe label skew, causing the system to default to guessing "legitimate" to achieve high but hollow accuracy. + + 5. Data Quality SLA (Service Level Agreement) +* **Technical Definition**; A formal commitment that defines measurable performance thresholds for critical data characteristics—such as freshness, schema adherence, completeness, and validity—enforced by automated testing pipelines. +* **Business Translation**: A contract between data teams and business stakeholders ensuring that the data powering operations arrives on time, with full completeness, and free of format errors. +* **Payments Example**: A data contract specifying that the `merchant_settlement_ledger` table must be fully populated and accurate by 06:00 UTC daily, with a hard failure alert triggered if missing row counts exceed 0.05%. From 44aff5793bc6a2e0ab87a1b0e12841666873b67a Mon Sep 17 00:00:00 2001 From: Mitali Daduria Date: Mon, 13 Jul 2026 12:12:03 +0530 Subject: [PATCH 4/9] docs: fix formatting and remove subheading numbers --- v1.13.x/main-concepts/metadata-standard.mdx | 95 ++++++--------------- 1 file changed, 25 insertions(+), 70 deletions(-) diff --git a/v1.13.x/main-concepts/metadata-standard.mdx b/v1.13.x/main-concepts/metadata-standard.mdx index b98df4bc..17d08425 100644 --- a/v1.13.x/main-concepts/metadata-standard.mdx +++ b/v1.13.x/main-concepts/metadata-standard.mdx @@ -1,73 +1,28 @@ ---- -title: Metadata Standard | OpenMetadata Core Schema Guide -description: Overview of metadata standard schemas governing how entities, types, analytics, and governance structures are defined. -sidebarTitle: Overview ---- +## Machine Learning & Data Governance Expansion -# OpenMetadata Standards - -OpenMetadata Standards is the community-driven home for the schemas and ontologies behind OpenMetadata. It provides the canonical JSON Schemas, RDF/OWL models, and API specifications that power data cataloging, governance, lineage, quality, and observability. - -Key highlights: - -- **Comprehensive coverage**: 700+ JSON Schemas spanning entities, relationships, events, and configuration. -- **API-first**: REST and event schemas for search, feeds, webhooks, and bulk operations. -- **Semantic web ready**: RDF/OWL ontologies and SHACL shapes for knowledge graph and linked data use cases. -- **Standards compliant**: JSON Schema (Draft 07/2020-12), RDF/OWL, SHACL, JSON-LD, PROV-O. -- **Community driven**: Built and maintained by the OpenMetadata community with open governance. - -## Schemas - -The OpenMetadata standard is powered by [JSON Schemas](https://json-schema.org/), as a readable and language-agnostic -solution that allows us to automatically generate code for the different pieces of the project. - - - -Curious about how OpenMetadata is built? You can take a look at the [High Level Design](/v1.13.x/main-concepts/high-level-design). - - - -You can explore the JSON Schemas in different ways: - -**1.** You can check all the definitions in [GitHub](https://github.com/open-metadata/OpenMetadata/tree/main/openmetadata-spec/src/main/resources/json/schema). - Navigating through the directories you can find, for example, the definition of a [Table](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/entity/data/table.json). - -**2.** If you prefer to stay in the docs, we also have you covered. We have converted the JSON Schemas to markdown files - that you can explore following the left menu titles, or navigating the structure by clicking on `Next` at the - end of each page. Again, this would be the structure of a [Table](/v1.13.x/main-concepts/metadata-standard). - In this generation we wanted to maintain the same structure as the GitHub repository. Any empty file means that you reached - a directory, but you can keep exploring until finding the right Entity, or you can also look for it using the Search Bar. - -Ready to dive in? - -Visit openmetadatastandards.org - -You'll find quick starts, schema references, semantic web resources, and contribution guides on the official site. - -### Machine Learning & Data Governance Expansion The following terms bridge common machine learning and data governance concepts to business-friendly language, with payments-domain examples showing how each one shows up in practice. - 1. Model Drift -* **Technical Definition**: The degradation of a machine learning model's predictive performance over time due to changes in the statistical distribution of the environment's underlying data. -* **Business Translation**: When a previously accurate AI model starts making poorer predictions because real-world consumer behavior or market conditions have evolved since the model was trained. -* **Payments Example**: A fraud detection model trained on historical data from early 2024 may experience seasonal model drift during the November/December holiday shopping rush, as legitimate spike buying behaviors mimic historic fraud patterns. - - 2. Data Lineage -* **Technical Definition**: The end-to-end lifecycle of data that maps its origins, structural transformations, intermediate pipeline stages, and ultimate consumption endpoints across the data ecosystem. -* **Business Translation**: A visual audit trail that shows exactly where a piece of data came from, how it was modified along the way, and which final business dashboards or reports are using it. -* **Payments Example**: Tracing an `interchange_fee_credited` field from the raw settlement API payload, through the daily ETL cleansing pipelines, to the monthly CFO financial compliance dashboard. - - 3. Feature Engineering -* **Technical Definition**: The process of using domain knowledge to select, transform, combine, and manipulate raw data variables into distinct predictor variables (features) that enhance machine learning algorithm performance. -* **Business Translation**: Extracting and shaping raw, unorganized data points into meaningful business metrics so an AI model can spot patterns more easily. -* **Payments Example**: Converting raw, individual transaction timestamps into a calculated feature like `count_of_transactions_in_last_5_minutes` to help a model identify high-frequency card-cloning velocity attacks. - - 4. Label Skew -* **Technical Definition**: A specific type of dataset imbalance occurring during training or evaluation where one target class label vastly outnumbers the alternative class labels. -* **Business Translation**: A training bias that happens when the data given to an AI contains overwhelming examples of one outcome and barely any examples of another, making it blind to rarer events. -* **Payments Example**; In a payment dataset of 10 million transactions, only 0.01% are actual chargebacks. Training a model on this creates severe label skew, causing the system to default to guessing "legitimate" to achieve high but hollow accuracy. - - 5. Data Quality SLA (Service Level Agreement) -* **Technical Definition**; A formal commitment that defines measurable performance thresholds for critical data characteristics—such as freshness, schema adherence, completeness, and validity—enforced by automated testing pipelines. -* **Business Translation**: A contract between data teams and business stakeholders ensuring that the data powering operations arrives on time, with full completeness, and free of format errors. -* **Payments Example**: A data contract specifying that the `merchant_settlement_ledger` table must be fully populated and accurate by 06:00 UTC daily, with a hard failure alert triggered if missing row counts exceed 0.05%. +#### Model Drift +- **Technical Definition**: The degradation of a machine learning model's predictive performance over time due to changes in the statistical distribution of the environment's underlying data. +- **Business Translation**: When a previously accurate AI model starts making poorer predictions because real-world consumer behavior or market conditions have evolved since the model was trained. +- **Payments Example**: A fraud detection model trained on historical data from early 2024 may experience seasonal model drift during the November/December holiday shopping rush, as legitimate spike buying behaviors mimic historic fraud patterns. + +#### Data Lineage +- **Technical Definition**: The end-to-end lifecycle of data that maps its origins, structural transformations, intermediate pipeline stages, and ultimate consumption endpoints across the data ecosystem. +- **Business Translation**: A visual audit trail that shows exactly where a piece of data came from, how it was modified along the way, and which final business dashboards or reports are using it. +- **Payments Example**: Tracing an interchange_fee_credited field from the raw settlement API payload, through the daily ETL cleansing pipelines, to the monthly CFO financial compliance dashboard. + +#### Feature Engineering +- **Technical Definition**: The process of using domain knowledge to select, transform, combine, and manipulate raw data variables into distinct predictor variables (features) that enhance machine learning algorithm performance. +- **Business Translation**: Extracting and shaping raw, unorganized data points into meaningful business metrics so an AI model can spot patterns more easily. +- **Payments Example**: Converting raw, individual transaction timestamps into a calculated feature like count_of_transactions_in_last_5_minutes to help a model identify high-frequency card-cloning velocity attacks. + +#### Label Skew +- **Technical Definition**: A specific type of dataset imbalance occurring during training or evaluation where one target class label vastly outnumbers the alternative class labels. +- **Business Translation**: A training bias that happens when the data given to an AI contains overwhelming examples of one outcome and barely any examples of another, making it blind to rarer events. +- **Payments Example**: In a payment dataset of 10 million transactions, only 0.01% are actual chargebacks. Training a model on this creates severe label skew, causing the system to default to guessing "legitimate" to achieve high but hollow accuracy. + +#### Data Quality SLA (Service Level Agreement) +- **Technical Definition**: A formal commitment that defines measurable performance thresholds for critical data characteristics—such as freshness, schema adherence, completeness, and validity—enforced by automated testing pipelines. +- **Business Translation**: A contract between data teams and business stakeholders ensuring that the data powering operations arrives on time, with full completeness, and free of format errors. +- **Payments Example**: A data contract specifying that the merchant_settlement_ledger table must be fully populated and accurate by 06:00 UTC daily, with a hard failure alert triggered if missing row counts exceed 0.05%. From e219e8844cf78b39178697ddd201ea865a9dfc83 Mon Sep 17 00:00:00 2001 From: Mitali Daduria Date: Mon, 13 Jul 2026 12:13:21 +0530 Subject: [PATCH 5/9] Add example for Data Quality SLA in metadata standard --- v1.13.x/main-concepts/metadata-standard.mdx | 1 + 1 file changed, 1 insertion(+) diff --git a/v1.13.x/main-concepts/metadata-standard.mdx b/v1.13.x/main-concepts/metadata-standard.mdx index 17d08425..63e93e15 100644 --- a/v1.13.x/main-concepts/metadata-standard.mdx +++ b/v1.13.x/main-concepts/metadata-standard.mdx @@ -26,3 +26,4 @@ The following terms bridge common machine learning and data governance concepts - **Technical Definition**: A formal commitment that defines measurable performance thresholds for critical data characteristics—such as freshness, schema adherence, completeness, and validity—enforced by automated testing pipelines. - **Business Translation**: A contract between data teams and business stakeholders ensuring that the data powering operations arrives on time, with full completeness, and free of format errors. - **Payments Example**: A data contract specifying that the merchant_settlement_ledger table must be fully populated and accurate by 06:00 UTC daily, with a hard failure alert triggered if missing row counts exceed 0.05%. + From 62a02dec66dd6db419fb809a60d60837ade9341a Mon Sep 17 00:00:00 2001 From: Mitali Daduria Date: Mon, 20 Jul 2026 17:46:55 +0530 Subject: [PATCH 6/9] docs: add real-time ML observability and drift metadata definitions --- .ai/doc-review/instructions.md | 110 +++++++ .ai/doc-review/references/checklist.md | 296 ++++++++++++++++++ .github/workflows/doc-review.yml | 53 ++++ CLAUDE.md | 8 + .../data-quality/fintech-framework.md | 58 ++++ v1.13.x/connectors/database/mssql.mdx | 36 ++- .../database/mssql/troubleshoot.mdx | 4 + v1.13.x/connectors/database/mssql/yaml.mdx | 36 ++- .../workflows/lineage/filter-query-set.mdx | 43 ++- .../workflows/usage/filter-query-set.mdx | 43 ++- v2.0.x-SNAPSHOT/connectors/database/mssql.mdx | 36 ++- .../database/mssql/troubleshoot.mdx | 4 + .../connectors/database/mssql/yaml.mdx | 36 ++- .../workflows/lineage/filter-query-set.mdx | 43 ++- .../workflows/usage/filter-query-set.mdx | 43 ++- 15 files changed, 841 insertions(+), 8 deletions(-) create mode 100644 .ai/doc-review/instructions.md create mode 100644 .ai/doc-review/references/checklist.md create mode 100644 .github/workflows/doc-review.yml create mode 100644 content/main/features/data-quality/fintech-framework.md diff --git a/.ai/doc-review/instructions.md b/.ai/doc-review/instructions.md new file mode 100644 index 00000000..3d768fa8 --- /dev/null +++ b/.ai/doc-review/instructions.md @@ -0,0 +1,110 @@ +# OpenMetadata content review instructions + +Reviews written content against the **OpenMetadata Writing Style Guide** and returns a +structured table report showing exactly which guideline was violated, the exact line +or sentence that needs changing, and the suggested replacement. + +--- + +## When to use + +Use these instructions when a user asks any AI assistant to review, proofread, +check, audit, improve, clean up, or validate content for OpenMetadata docs; asks whether +copy sounds right or on-brand; or asks to check a draft against the style guide. + +## How to use + +1. Read the content the user wants reviewed. +2. Read the checklist at `.ai/doc-review/references/checklist.md`. +3. Run every applicable checklist item against the content. +4. Return a **Review Report** in the exact format below. +5. Offer a fully revised version after the report if the user asks. + +If no content or file reference is provided, ask the user to provide the content to +review and stop. + +Do not skip applicable categories even if the content is short. Mark checklist +items as N/A when they do not apply to the content type. + +--- + +## Review Report Format + +Output your response in exactly this structure: + +--- + +### OpenMetadata Content Review + +**Content type:** [e.g. Email, Documentation, Marketing copy, Release note] +**Overall verdict:** PASS / NEEDS WORK / FAIL +*(FAIL = 5 or more Critical issues; NEEDS WORK = any Major issues or 3+ Minor)* + +--- + +#### Issues Found + +Present every issue as a row in this table. One row per issue — do not combine multiple issues into one row. + +| # | Guideline | Severity | Original text | Suggested change | +|---|-----------|----------|---------------|-----------------| +| 1 | [Guideline name + section, e.g. "Active voice — §3.1"] | Critical / Major / Minor | "exact quote from the content" | "replacement text or instruction" | +| 2 | ... | ... | ... | ... | + +**Column definitions:** +- **#** — Sequential issue number. +- **Guideline** — The specific rule from the OpenMetadata Writing Style Guide that is violated. Always name the rule and its section number (e.g. "Contractions — §3.3", "Oxford comma — §4.2", "OpenMetadata brand name — §10.1"). Never write a vague label like "tone issue." +- **Severity** — One of: + - **Critical** — Breaks a core rule: wrong brand name, passive voice throughout, gendered pronouns, no Oxford comma throughout. + - **Major** — Noticeably degrades quality: jargon, wordiness, redundant phrases used repeatedly, missing contractions throughout. + - **Minor** — Single small polish item: one number not spelled out, one missing em dash, one weak word choice. +- **Original text** — The exact sentence, phrase, or word from the content that needs to change. Always quote verbatim in double quotes. If the issue is structural (e.g. a missing heading), write a short description instead. +- **Suggested change** — The corrected version in double quotes, or a clear instruction if a full rewrite is needed (e.g. "Split into two sentences" or "Add a heading before this paragraph"). + +--- + +#### What's Working Well + +List 2–4 specific things the content does correctly. Be concrete — cite actual text or structure from the piece, not generic praise. + +--- + +#### Category Summary + +| Category | Status | Issue count | +|----------|--------|-------------| +| Voice & Tone | PASS/WARN/FAIL | 0 | +| Clarity & Conciseness | PASS/WARN/FAIL | 0 | +| Grammar & Language | PASS/WARN/FAIL | 0 | +| Punctuation | PASS/WARN/FAIL | 0 | +| Capitalization | PASS/WARN/FAIL | 0 | +| Numbers & Dates | PASS/WARN/FAIL | 0 | +| Formatting & Structure | PASS/WARN/FAIL | 0 | +| Inclusive Language | PASS/WARN/FAIL | 0 | +| Accessibility | PASS/WARN/FAIL | 0 | +| OpenMetadata Branding | PASS/WARN/FAIL | 0 | +| Global / Localization | PASS/WARN/FAIL/N/A | 0 | + +Status key: PASS = no issues, WARN = minor issues only, FAIL = major or critical issues, N/A = category does not apply + +--- + +#### Top 3 Priorities + +| Priority | Fix | +|----------|-----| +| 1 | [Most impactful single change] | +| 2 | [Second most impactful change] | +| 3 | [Third most impactful change] | + +--- + +## Important Behaviour Rules + +- **Quote exact text.** The "Original text" column must always contain the verbatim phrase from the content — never a paraphrase. If the passage is long, quote the most relevant fragment (20 words or fewer). +- **Name the guideline precisely.** Every row must reference a specific rule from the OpenMetadata Writing Style Guide with its section number. "Tone" or "style" alone is not acceptable. +- **One issue per row.** Do not bundle multiple violations into one row even if they occur in the same sentence. Each violation gets its own row. +- **Never rewrite the whole document unprompted.** Offer to produce a clean revised version after delivering the report. +- **Context matters.** Legal disclaimers may use formal language intentionally. Inline code snippets follow code conventions, not prose rules. Use judgment and note exceptions. +- **If content is under 50 words**, note that the review is limited due to brevity and not all categories can be fully assessed. +- **For content intended for translation**, treat Global / Localization checklist items as Major severity rather than Minor. diff --git a/.ai/doc-review/references/checklist.md b/.ai/doc-review/references/checklist.md new file mode 100644 index 00000000..1a7f5616 --- /dev/null +++ b/.ai/doc-review/references/checklist.md @@ -0,0 +1,296 @@ +# OpenMetadata Content Review Checklist + +Full item-by-item checklist derived from the OpenMetadata Writing Style Guide. +Check every item. Mark each as PASS, FAIL, or N/A. + +--- + +## Category 1: Voice & Tone + +### 1.1 Brand Voice +- [ ] **Warm and clear** — Does the content feel like a knowledgeable colleague rather than a formal institution? Flag overly stiff or robotic phrasing. +- [ ] **Honest and confident** — Does it say what it means without overselling, hedging excessively, or obscuring issues? +- [ ] **Human and inclusive** — Is it written for real people, not a faceless audience? + +### 1.2 Conversational Naturalness +- [ ] **Read-aloud test** — Would this sound natural spoken aloud? Flag anything that wouldn't be said in a meeting. +- [ ] **No corporate filler** — Flag: "synergize," "leverage" (when "use" works), "holistic approach," "move the needle," "circle back," "deep dive," "best-in-class." +- [ ] **No Latin abbreviations in body text** — Flag "i.e." (use "that is"), "e.g." (use "for example"), "etc." (use "and so on"). +- [ ] **"Please" used sparingly** — Flag "please" outside genuine requests or apologies; don't use it to soften routine instructions. +- [ ] **No "let's" framing** — Flag "let's configure..." → "Configure..." + +### 1.3 Tone Matching +- [ ] Is the tone appropriate for the content type? + - Customer support: empathetic, patient, solution-focused + - Technical docs: precise, neutral, instructional + - Marketing: energetic, confident, benefit-led + - Internal comms: friendly, direct, collegial + - Incident/crisis: calm, factual, accountable + +--- + +## Category 2: Clarity & Conciseness + +### 2.1 Sentence Length & Complexity +- [ ] **Short sentences** — Flag sentences with more than two commas plus end punctuation. Suggest breaking them up. +- [ ] **One idea per sentence** — Flag sentences that try to do too much. +- [ ] **No modifier stacks** — Flag long chains of nouns used as modifiers (e.g., "extremely well thought-out Windows migration project plan"). + +### 2.2 Word Economy +- [ ] **No "you can"** — Flag "you can [verb]" and suggest replacing with the direct imperative or verb alone. E.g., "You can export the report" → "Export the report." +- [ ] **No "there is/are/were"** — Flag and suggest rewriting. E.g., "There are three options available" → "Three options are available." +- [ ] **No redundant phrases** — Flag: + - "in order to" → "to" + - "at this point in time" → "now" + - "due to the fact that" → "because" + - "prior to" → "before" + - "subsequent to" → "after" + - "with regard to" → "about" + - "in the event that" → "if" + - "on a regular basis" → "regularly" +- [ ] **"Lets you" over "allows you to"** — Flag "allows you to [verb]"; prefer the shorter "lets you [verb]." + +### 2.3 Word Choice +- [ ] **Simple over complex** — Flag: + - "utilize" → "use" + - "initiate" → "start" + - "facilitate" → "help" + - "terminate" → "end" or "stop" + - "endeavour" → "try" + - "ascertain" → "find out" + - "commence" → "start" or "begin" +- [ ] **Jargon check** — Is every technical term necessary and understandable to the target audience? Flag undefined jargon. +- [ ] **Consistent terminology** — Is the same concept always called the same thing? Flag synonyms used interchangeably for the same concept. +- [ ] **"Can" vs. "might" vs. "may"** — Use "can" for capability or permission, "might" for uncertain possibility. Flag "may" for either — it's ambiguous between the two. +- [ ] **"Because" vs. "since"** — Use "because" for causation. Reserve "since" for time-based meaning only ("since version 1.5," not "since it's faster"). + +### 2.4 Front-Loading +- [ ] **Key point first** — Does the most important information appear in the first sentence or paragraph? +- [ ] **Purpose clear early** — In emails and articles, is the purpose stated in the first two sentences? + +--- + +## Category 3: Grammar & Language + +### 3.1 Voice +- [ ] **Active voice preferred** — Flag passive constructions. E.g., "The file was rejected by the system" → "The system rejected the file." +- [ ] **Passive voice acceptable when** — the actor is unknown, or the action matters more than who did it (e.g., "The database was last updated in March"). Do not flag these. +- [ ] **No first-person "we/us/our" in reference or technical content** — Flag "we read the query log"; describe what the product does instead ("OpenMetadata reads the query log"). Second person ("you") is fine when addressing the reader directly. + +### 3.2 Verbs +- [ ] **Strong verbs** — Flag verb-to-noun conversions: "make a decision" → "decide," "carry out an implementation" → "implement." +- [ ] **Imperative mood in instructions** — Instructions should use imperative: "Select the file," not "You should select the file." +- [ ] **Present tense where possible** — Especially for product descriptions and instructions. + +### 3.3 Contractions +- [ ] **Use contractions in general content** — Flag missing contractions where formal phrasing sounds stiff: "it is" → "it's," "you will" → "you'll," "do not" → "don't." +- [ ] **No contractions in** legal, compliance, or highly formal documents (these should not be flagged). +- [ ] **No awkward contractions** — Flag "should've," "there'd," "would've" in professional content. + +### 3.4 Sentence Structure +- [ ] **Standard word order** — Subject + verb + object. Flag inverted or convoluted structures. +- [ ] **Modifiers close to what they modify** — Flag dangling modifiers and misplaced "only." +- [ ] **No more than two clauses joined by and/or/but** — Flag run-ons; suggest splitting or using a list. + +--- + +## Category 4: Punctuation + +### 4.1 End Punctuation +- [ ] **Every sentence ends with a period** — Even two-word sentences. +- [ ] **One space after periods** — Flag double spaces. +- [ ] **No multiple exclamation marks** — Flag "!!" or "!!!". +- [ ] **Exclamation marks used sparingly** — Flag if more than one appears in a short document. + +### 4.2 Commas +- [ ] **Oxford (serial) comma used** — "files, folders, and documents" not "files, folders and documents." +- [ ] **Comma after introductory clause** — Flag missing commas after opening subordinate clauses. +- [ ] **No comma splice** — Two independent clauses must not be joined by a comma alone without a conjunction. +- [ ] **No "&" as a stand-in for "and"** — Flag ampersands in prose or headings; reserve "&" for literal UI labels being referenced. + +### 4.3 Colons and Semicolons +- [ ] **Colon before a list** — Used at the end of an introducing phrase, not a complete sentence that already contains a list. +- [ ] **First word after colon** — Capitalize if it begins an independent clause. +- [ ] **Avoid semicolons in general content** — Flag; suggest splitting the sentence. + +### 4.4 Dashes and Hyphens +- [ ] **Em dash with no spaces** — "use pipelines—logical groups—to" not "use pipelines — logical groups — to." +- [ ] **No spaced en dashes used as em dashes** — Flag " – " used mid-sentence. +- [ ] **En dash for ranges** — "2020–2026," "pages 10–15" (not hyphens). +- [ ] **Hyphen in compound modifiers before a noun** — "well-known author" but "the author is well known." +- [ ] **No hyphen after -ly adverbs** — "a highly effective tool" not "a highly-effective tool." + +### 4.5 Apostrophes +- [ ] **No apostrophe in plurals** — "PDFs" not "PDF's," "the 1990s" not "the 1990's." +- [ ] **Correct possessives** — "the team's results," "OpenMetadata's brand." + +### 4.6 Quotation Marks +- [ ] **Periods and commas inside quotes** — Place commas and periods inside the closing quotation mark, as in "like this." +- [ ] **Colons and semicolons outside quotes.** +- [ ] **No scare quotes for emphasis** — Use precise wording instead of ironic quotes. + +### 4.7 Parentheses +- [ ] **No important information in parentheses** — Some readers skip parenthetical content; don't hide anything the reader needs there. +- [ ] **Keep parenthetical asides brief** — If a parenthetical would run long (e.g., a list of code identifiers), split it into a separate sentence or use a dash instead. + +--- + +## Category 5: Capitalization + +### 5.1 Heading Case +- [ ] **Headings use title case on this site** — Capitalize all major words (e.g., "Bulk Import Test Cases"). This is a deliberate house style across the OpenMetadata docs — do not flag title-case headings as a violation. +- [ ] **List items start with a capital letter.** +- [ ] **No all-caps for emphasis** — Flag ANY WORD IN ALL CAPS used for emphasis. Italic is acceptable. +- [ ] **No all-lowercase as a style choice.** + +### 5.2 Proper Nouns +- [ ] **Product and service names capitalized** — Flag lowercase product names. +- [ ] **"OpenMetadata" always capitalized** — Flag "openmetadata," "Openmetadata," or "OPENMETADATA." +- [ ] **Acronyms in full caps** — API, SLA, CRM (no mixed case like "Api"). + +### 5.3 What Not to Capitalize +- [ ] **Generic job titles lowercase** — "the engineering manager" (not "Engineering Manager") unless used as a formal title before a name. +- [ ] **Common tech terms lowercase** — "application," "platform," "database" (not "Platform" or "Database"). +- [ ] **Spelled-out acronyms lowercase** — "application programming interface" not "Application Programming Interface." + +--- + +## Category 6: Numbers & Dates + +### 6.1 Numbers +- [ ] **Spell out zero through nine** — "three options" not "3 options" (unless in a technical/measurement context). +- [ ] **Numerals for 10 and above** — "15 users" not "fifteen users." +- [ ] **Numerals always for** measurements, percentages, version numbers, money, technical specs. +- [ ] **No sentence starting with a numeral** — Rewrite or spell out. +- [ ] **Commas in 4+ digit numbers** — "1,000" not "1000." + +### 6.2 Dates +- [ ] **Month spelled out** — "May 12, 2026" not "5/12/2026" or "12/5/2026." +- [ ] **No ordinals in dates** — "June 1" not "June 1st." +- [ ] **En dash for date ranges** — "May 10–14, 2026." + +### 6.3 Time +- [ ] **12-hour clock with AM/PM in capitals with a space** — "9:00 AM" not "9am" or "9:00am." +- [ ] **"noon" and "midnight"** — not "12:00 PM" or "12:00 AM." + +### 6.4 Percentages & Currency +- [ ] **% symbol with numerals** — "45%" not "45 percent" (except at the start of a sentence). +- [ ] **Currency symbol with numeral** — "$500" not "500 dollars." + +--- + +## Category 7: Formatting & Structure + +### 7.1 Headings +- [ ] **Every section has a clear, descriptive heading.** +- [ ] **Parallel structure in headings** — Same grammatical form within the same section. +- [ ] **No period at end of headings** — Question marks are acceptable. +- [ ] **No heading immediately after another heading** with no body text between. + +### 7.2 Lists +- [ ] **At least 2 items** — Single-item lists should be prose. +- [ ] **No more than 7 items** — Suggest grouping if more. +- [ ] **Parallel structure** — All items use the same grammatical form. +- [ ] **Introduced with a complete sentence or colon-ending phrase.** +- [ ] **No semicolons or commas at end of list items.** +- [ ] **Period only if items are complete sentences or complete an introductory fragment.** + +### 7.3 Bold & Italic +- [ ] **Bold used only for** key terms on first use, UI element names, or critical warnings. Not for general emphasis. +- [ ] **No underline** except hyperlinks. +- [ ] **Italic used for** titles of works or introducing new technical terms. + +### 7.4 Instructions +- [ ] **Numbered list for steps** — Not bullets, not prose. +- [ ] **One action per step.** +- [ ] **Imperative verb starts each step** — "Select," "Enter," "Open." +- [ ] **Location stated first in step** — "On the Settings page, select..." +- [ ] **Optional steps marked with "Optional:"** — Flag optional steps indicated by parentheses instead; use "Optional: ..." at the start of the step. +- [ ] **Goal-first phrasing where it reads naturally** — Prefer "To do X, do Y" over burying the goal at the end of the step. + +### 7.5 Notices & Callouts +- [ ] **Used sparingly** — Flag more than one Note/Tip/Warning stacked back-to-back; overuse diminishes their effectiveness. +- [ ] **Right type for the content** — Note = non-critical, skippable information; Warning = risk of data loss, security issues, or other irreversible harm; Tip = optional helpful suggestion. +- [ ] **Not used for prerequisites** — Information the reader needs before starting belongs in the main flow, not a callout. +- [ ] **Not used for essential information** — If the reader can't succeed without it, it isn't a Note — put it in the body text. +- [ ] **Not used for cross-references** — Link to related content directly in the body text rather than via a callout. +- [ ] **Not used for procedural steps** — Full steps belong in the numbered instructions, not offset in a box. + +--- + +## Category 8: Inclusive Language + +### 8.1 Gender-Neutral Language +- [ ] **No gendered pronouns in generic references** — Flag "he," "his," "she," "her" when referring to unspecified individuals. +- [ ] **"You" preferred over third person** — Use direct address, such as "Access your account," not third person, such as "the user can access their account." +- [ ] **No "he/she" or "s/he" constructions.** +- [ ] **Singular "they" acceptable** — Flag only if the sentence becomes confusing. + +### 8.2 People-First Language +- [ ] **Person before disability** — "a user who is blind" not "a blind user" (unless the individual/community prefers identity-first). +- [ ] **No pity language** — Flag "suffering from," "afflicted with," "victim of." +- [ ] **Disability mentioned only if relevant.** + +### 8.3 Culturally Sensitive Language +- [ ] **No unconscious bias in technical terms** — Flag "master/slave" → "primary/secondary"; "whitelist/blacklist" → "allowlist/blocklist." +- [ ] **No idioms or colloquialisms** that may not translate or that assume shared cultural background. +- [ ] **No assumptions about holidays, sports, or political systems** unless directly relevant. + +### 8.4 Age & Generational Language +- [ ] **No age stereotypes** — Do not assume older users cannot use technology or younger users lack professionalism. + +--- + +## Category 9: Accessibility + +### 9.1 Structure +- [ ] **Heading levels reflect hierarchy** — Don't use bold as a substitute for a heading. +- [ ] **No directional language as sole locator** — "the table on the left" → "the following table." +- [ ] **Descriptive link text** — Flag "click here," "read more," "learn more" without context. Suggest "Download the Q1 report" or "Learn more about data privacy." +- [ ] **Link text matches the destination** — Where reasonable, link text should match the title or heading of the page it points to. +- [ ] **No duplicate links** — Flag the same destination linked more than once on a page, unless linking to distinct sections. +- [ ] **"For more information, see [X]"** — Use this as the standard phrasing when a full sentence is dedicated to a cross-reference. +- [ ] **Punctuation outside link text** — Don't include trailing punctuation inside the link. Link text also isn't wrapped in quotation marks. + +### 9.2 Images & Media (if described or included) +- [ ] **Alt text described or present** for all meaningful images. +- [ ] **No information conveyed only through color.** + +### 9.3 Plain Language +- [ ] **Abbreviations and acronyms spelled out on first use** — "application programming interface (API)" not just "API." +- [ ] **Consistent terminology throughout** — No synonyms for the same concept. + +### 9.4 Interaction Language +- [ ] **Generic interaction verbs** — "select" not "click," "enter" not "type," "activate" not "tap." + +--- + +## Category 10: OpenMetadata Branding + +### 10.1 Project Name +- [ ] **"OpenMetadata" — correct capitalization** — Flag "openmetadata," "Openmetadata," "OPENMETADATA," "Open Metadata" (as two words), or "OM" as a stand-in. +- [ ] **First mention in a document uses "OpenMetadata"** — Subsequent uses may also use "OpenMetadata." +- [ ] **No unofficial abbreviations** — Flag informal shorthand in prose (code identifiers and config keys are exempt). + +### 10.2 Product Names +- [ ] **Exact registered product/feature names used** — No abbreviations, shortened forms, or unofficial variants. +- [ ] **Product names capitalized as proper nouns.** +- [ ] **Version numbers formatted correctly** — "OpenMetadata 1.5" not "OpenMetadata v1.5" or "OM 1.5." + +### 10.3 Customer-Facing Tone +- [ ] **Customer addressed as "you"** — Not "the user," "the client," or third person in direct communications. +- [ ] **Error messages and notifications state result or action first** — "Your file was saved" not "File save operation completed successfully." +- [ ] **Error messages explain what went wrong + what to do next** — No raw error codes without explanation. +- [ ] **No unofficial taglines or slogans** — Only approved project language in external content. +- [ ] **No unapproved product claims** — Performance commitments require maintainer sign-off. + +--- + +## Category 11: Global / Localization (apply when content will be translated) + +- [ ] **No idioms or culture-specific expressions.** +- [ ] **No list items completing an introductory sentence fragment** — These are hard to translate. +- [ ] **No noun stacks** — Long modifier chains often can't be translated. +- [ ] **No humor, wordplay, or puns** in content intended for translation. +- [ ] **UI strings under 80 characters** where possible (translations are often longer). +- [ ] **Time zones specified** in international scheduling content. diff --git a/.github/workflows/doc-review.yml b/.github/workflows/doc-review.yml new file mode 100644 index 00000000..6746c52d --- /dev/null +++ b/.github/workflows/doc-review.yml @@ -0,0 +1,53 @@ +name: AI Documentation Review (on demand) + +on: + issue_comment: + types: [created] + +jobs: + doc-review: + if: | + github.event.issue.pull_request != null && + contains(fromJSON('["OWNER","MEMBER","COLLABORATOR"]'), github.event.comment.author_association) && + contains(github.event.comment.body, '/content-review') + runs-on: ubuntu-latest + permissions: + contents: read + pull-requests: write + steps: + - uses: actions/checkout@v6 + with: + fetch-depth: 1 + + - uses: anthropics/claude-code-action@v1 + env: + GH_TOKEN: ${{ github.token }} + with: + anthropic_api_key: ${{ secrets.ANTHROPIC_API_KEY }} + trigger_phrase: "/content-review" + prompt: | + REPO: ${{ github.repository }} + PR NUMBER: ${{ github.event.issue.number }} + + Read .ai/doc-review/instructions.md and .ai/doc-review/references/checklist.md + in this repo. Run `gh pr diff ` to get this PR's changes, and review + the changed .mdx content against every applicable checklist item, exactly as + instructed. + + Produce the Review Report in the exact format instructions.md specifies, + including every finding regardless of severity (Critical, Major, and Minor) — + this review was explicitly requested. + + Prefix it with: "**Draft review — not all suggestions may apply. A maintainer + will select which findings to act on.**" + + Return the Review Report as your final response — do not call `gh pr comment` + or any other command to post it yourself. This action automatically publishes + your final response as the PR comment, and calling `gh pr comment` in addition + would post a duplicate. + + Do not attempt to create inline review comments or submit a formal PR review — + only the one summary comment. + + claude_args: | + --allowedTools "Bash(gh pr diff:*),Read" diff --git a/CLAUDE.md b/CLAUDE.md index e1cc317b..fd6fea3c 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -76,6 +76,14 @@ Connector pages follow a consistent template: New pages **must** be added to `docs.json` in the appropriate navigation section, or they won't appear in the sidebar. +### AI content review + +When reviewing OpenMetadata documentation, release notes, UI copy, or other written content, use the AI-neutral workflow in `.ai/doc-review/instructions.md` and its checklist in `.ai/doc-review/references/checklist.md`. These instructions are shared across AI assistants and are not tied to Claude Code. + +### Pre-commit content review (Claude Code) + +Before running `git commit` on any change touching `.mdx` files (or `docs.json`), read `.ai/doc-review/instructions.md` and `.ai/doc-review/references/checklist.md`, and review the staged diff against every applicable item — unless the changes were already reviewed against this checklist earlier in the same conversation. Show Critical and Major findings before committing; if Minor findings also exist, mention the count and offer to show them rather than listing them inline by default. Proceed with the commit only after the user has seen this and confirmed. + --- ## Changing the Site diff --git a/content/main/features/data-quality/fintech-framework.md b/content/main/features/data-quality/fintech-framework.md new file mode 100644 index 00000000..60c3674a --- /dev/null +++ b/content/main/features/data-quality/fintech-framework.md @@ -0,0 +1,58 @@ +# FinTech Data Quality Framework + +Financial technology data requires a level of consistency, accuracy, and latency control far stricter than standard analytical data stores. A single null value in a settlement column or an unexpected duplicate in a transaction ledger can result in severe financial discrepancies, regulatory non-compliance, or disrupted user operations. + +This guide outlines how to implement a robust **Data Quality Framework** for FinTech pipelines using OpenMetadata's native profiling and test suites. + +--- + +## The 3 Critical FinTech Data Failure Modes + +### 1. Settlement & Reconciliation Mismatches (Asymmetry) +* **The Problem**: Transaction data flowing from a payment gateway (e.g., Stripe, Adyen) must reconcile perfectly with internal ledger entries. Discrepancies often arise due to timezone differences, late-arriving settlement records, or multi-currency rounding issues. +* **Impact**: Inaccurate ledger reporting, incorrect merchant payouts, and balance sheets that do not balance. + +### 2. Orphaned Records & Key Violations +* **The Problem**: High-volume asynchronous transaction pipelines often lead to processing delays, resulting in "orphaned" transactions where the payment event occurs but the associated billing account or user ID does not exist in the source table. +* **Impact**: Orphaned accounts cause batch analytical pipelines to crash or misrepresent active user revenue metrics. + +### 3. Latency & Batch Freshness Violations +* **The Problem**: In financial operations, data freshness acts as a proxy for pipeline health. If credit scoring features, risk databases, or fraud prevention logs lag behind real-time processing, fraud detection models make decisions based on stale states. +* **Impact**: Spikes in undetected payment fraud and exposure to chargeback risk. + +--- + +## Implementing the Framework with OpenMetadata + +OpenMetadata allows you to operationalize this framework directly on your tables using the **Profiler** and declarative **Test Suites**. + +### 1. Guarding Against Settlement Mismatches (Value Range & Sum Assertions) +To ensure transactional integrity, we run test suites verifying that transactional columns behave within absolute bounds and matching criteria. + +* **Column Value Limits**: Ensure transactional amounts are never negative or unexpectedly zero. + * *OpenMetadata Test*: `columnValuesToBeBetween` + * *Configuration*: Set `minValue = 0.01` for debit/credit transaction amount fields. +* **Sum Validations**: Verify that daily ledger updates match expected sum totals. + * *OpenMetadata Test*: `columnValuesSumToBeBetween` + +### 2. Eliminating Duplicates and Orphans (Integrity Tests) +Double-submitting a webhook payload should never cause duplicate database entries. We enforce this at the database validation layer using key assertions. + +* **Uniqueness Constraints**: Every financial transaction must have a universally unique ID (UUID). + * *OpenMetadata Test*: `columnValuesToBeUnique` applied to `transaction_id`. +* **Null Value Prevention**: Critical foreign keys (like `merchant_id` or `user_id`) must never resolve to null. + * *OpenMetadata Test*: `columnValuesToNotBeNull` on `account_id` and `customer_id`. + +### 3. Monitoring Processing Latency (Freshness Tests) +To guarantee real-time updates for downstream fraud modeling, apply table-level freshness tests to transactional timestamps. + +* **Timestamp Verification**: Ensure the difference between the transaction event time and the ingest time remains below a critical threshold. + * *OpenMetadata Test*: `tableRowCountToBeGreaterThan` (within a moving temporal partition) or customized SQL queries checking `MAX(created_at)` against the current time. + +--- + +## Checklist: Deploying Your FinTech Test Suite + +1. **Profile Your Tables**: Run an initial ingestion execution on your payment datasets using the OpenMetadata Profiler to establish baseline means, standard deviations, and null counts. +2. **Create a Dedicated Test Suite**: In the OpenMetadata UI, navigate to the **Database Service** → **Table** → **Profiler** tab and click **Add Test**. +3. **Alerting**: Bind the FinTech Test Suite to your alerting channels (Slack, Microsoft Teams, or PagerDuty) so your data engineering on-call rotation is immediately notified when a settlement test fails. \ No newline at end of file diff --git a/v1.13.x/connectors/database/mssql.mdx b/v1.13.x/connectors/database/mssql.mdx index 7e54dace..f14090bf 100644 --- a/v1.13.x/connectors/database/mssql.mdx +++ b/v1.13.x/connectors/database/mssql.mdx @@ -52,10 +52,44 @@ GRANT VIEW DEFINITION TO Mary; GRANT VIEW ANY DEFINITION TO [login_name]; ``` ### Usage & Lineage consideration -To perform the query analysis for Usage and Lineage computation, we fetch the query logs from `sys.dm_exec_cached_plans`, `sys.dm_exec_query_stats` & `sys.dm_exec_sql_text` system tables. To access these tables your user must have `VIEW SERVER STATE` privilege. +To perform the query analysis for Usage and Lineage computation, the connector reads query history from SQL Server's [Query Store](https://learn.microsoft.com/en-us/sql/relational-databases/performance/monitoring-performance-by-using-the-query-store) when it is enabled on a database. Query Store exposes this history through `sys.query_store_query`, `sys.query_store_query_text`, `sys.query_store_plan`, and `sys.query_store_runtime_stats`. Query Store keeps a durable, on-disk record of executed queries. The plan-cache Dynamic Management Views (DMVs) (`sys.dm_exec_cached_plans`, `sys.dm_exec_query_stats`, and `sys.dm_exec_sql_text`) are evicted on server restart, memory pressure, or plan recompilation, and can silently return incomplete history. + +Query Store is auto-detected per database. No configuration is needed. When `ingestAllDatabases` is enabled, each database is read from its own best source independently: databases with Query Store use it, and the rest fall back to the plan-cache DMVs, so one database without Query Store never downgrades the others. + + +**Tip:** Enable Query Store on the databases you ingest, for reliable and durable Usage and Lineage history. + + +To check whether Query Store is enabled on a database, and how it is currently operating: +```sql +SELECT actual_state_desc, query_capture_mode_desc, size_based_cleanup_mode_desc +FROM sys.database_query_store_options; +``` +To enable it: +```sql +ALTER DATABASE [YourDatabase] SET QUERY_STORE = ON +( + QUERY_CAPTURE_MODE = ALL, + CLEANUP_POLICY = ( STALE_QUERY_THRESHOLD_DAYS = 30 ) +); +``` +The default `QUERY_CAPTURE_MODE` on SQL Server 2019 and later is `AUTO`, which can skip queries that only run a few times, such as a one-off data migration statement. Setting it to `ALL` ensures every query is captured for lineage. Set `STALE_QUERY_THRESHOLD_DAYS` to cover at least your ingestion lookback window, so queries are not purged before they are read. Query Store's default `SIZE_BASED_CLEANUP_MODE` purges the oldest queries to stay within `MAX_STORAGE_SIZE_MB`, which reduces the risk of Query Store switching to read-only. Under heavy write load, cleanup can still fall behind, and Query Store can switch to read-only temporarily until it catches up. + + +Query Store needs two permissions together. The database-level grant below lets `sys.database_query_store_options` and Query Store's other metadata views succeed, but the query text comes from `sys.query_store_query_text`, which needs the server-level grant. Without the server-level grant, usage and lineage queries can run without error while returning no query text. + + +Database-level, for Query Store metadata: `VIEW DATABASE STATE` (SQL Server 2016-2019) or `VIEW DATABASE PERFORMANCE STATE` (SQL Server 2022 and later). +```sql +GRANT VIEW DATABASE STATE TO YourUser; +-- SQL Server 2022 and later: GRANT VIEW DATABASE PERFORMANCE STATE TO YourUser; +``` +Server-level, for the query text and the plan-cache DMV fallback: `VIEW SERVER STATE` (SQL Server 2019 and earlier) or `VIEW SERVER PERFORMANCE STATE` (SQL Server 2022 and later). ```sql GRANT VIEW SERVER STATE TO YourUser; +-- SQL Server 2022 and later: GRANT VIEW SERVER PERFORMANCE STATE TO YourUser; ``` +Exact grant names and tiers can differ on Azure SQL Database and Azure SQL Managed Instance. Check `sys.database_query_store_options` and the [Microsoft documentation](https://learn.microsoft.com/en-us/sql/relational-databases/system-catalog-views/sys-query-store-query-text-transact-sql) for your instance. ### For Remote Connection #### 1. SQL Server running Make sure the SQL server that you are trying to connect is in running state. diff --git a/v1.13.x/connectors/database/mssql/troubleshoot.mdx b/v1.13.x/connectors/database/mssql/troubleshoot.mdx index 4f8026e5..bd8d6df7 100644 --- a/v1.13.x/connectors/database/mssql/troubleshoot.mdx +++ b/v1.13.x/connectors/database/mssql/troubleshoot.mdx @@ -58,3 +58,7 @@ After updating the connection scheme, the connection should succeed. The status ``` Connection Status: Success ``` + +## Stored Procedure Lineage Across Databases + +If the same stored procedure name exists in more than one database or schema, the connector matches each query to its procedure using the database and schema reported by the query source. If the match is still ambiguous, that query is skipped rather than attached to the wrong procedure. diff --git a/v1.13.x/connectors/database/mssql/yaml.mdx b/v1.13.x/connectors/database/mssql/yaml.mdx index 82417fd3..18657ce5 100644 --- a/v1.13.x/connectors/database/mssql/yaml.mdx +++ b/v1.13.x/connectors/database/mssql/yaml.mdx @@ -60,10 +60,44 @@ GRANT VIEW DEFINITION TO Mary; GRANT VIEW ANY DEFINITION TO [login_name]; ``` ### Usage & Lineage consideration -To perform the query analysis for Usage and Lineage computation, we fetch the query logs from `sys.dm_exec_cached_plans`, `sys.dm_exec_query_stats` & `sys.dm_exec_sql_text` system tables. To access these tables your user must have `VIEW SERVER STATE` privilege. +To perform the query analysis for Usage and Lineage computation, the connector reads query history from SQL Server's [Query Store](https://learn.microsoft.com/en-us/sql/relational-databases/performance/monitoring-performance-by-using-the-query-store) when it is enabled on a database. Query Store exposes this history through `sys.query_store_query`, `sys.query_store_query_text`, `sys.query_store_plan`, and `sys.query_store_runtime_stats`. Query Store keeps a durable, on-disk record of executed queries. The plan-cache Dynamic Management Views (DMVs) (`sys.dm_exec_cached_plans`, `sys.dm_exec_query_stats`, and `sys.dm_exec_sql_text`) are evicted on server restart, memory pressure, or plan recompilation, and can silently return incomplete history. + +Query Store is auto-detected per database. No configuration is needed. When `ingestAllDatabases` is enabled, each database is read from its own best source independently: databases with Query Store use it, and the rest fall back to the plan-cache DMVs, so one database without Query Store never downgrades the others. + + +**Tip:** Enable Query Store on the databases you ingest, for reliable and durable Usage and Lineage history. + + +To check whether Query Store is enabled on a database, and how it is currently operating: +```sql +SELECT actual_state_desc, query_capture_mode_desc, size_based_cleanup_mode_desc +FROM sys.database_query_store_options; +``` +To enable it: +```sql +ALTER DATABASE [YourDatabase] SET QUERY_STORE = ON +( + QUERY_CAPTURE_MODE = ALL, + CLEANUP_POLICY = ( STALE_QUERY_THRESHOLD_DAYS = 30 ) +); +``` +The default `QUERY_CAPTURE_MODE` on SQL Server 2019 and later is `AUTO`, which can skip queries that only run a few times, such as a one-off data migration statement. Setting it to `ALL` ensures every query is captured for lineage. Set `STALE_QUERY_THRESHOLD_DAYS` to cover at least your ingestion lookback window, so queries are not purged before they are read. Query Store's default `SIZE_BASED_CLEANUP_MODE` purges the oldest queries to stay within `MAX_STORAGE_SIZE_MB`, which reduces the risk of Query Store switching to read-only. Under heavy write load, cleanup can still fall behind, and Query Store can switch to read-only temporarily until it catches up. + + +Query Store needs two permissions together. The database-level grant below lets `sys.database_query_store_options` and Query Store's other metadata views succeed, but the query text comes from `sys.query_store_query_text`, which needs the server-level grant. Without the server-level grant, usage and lineage queries can run without error while returning no query text. + + +Database-level, for Query Store metadata: `VIEW DATABASE STATE` (SQL Server 2016-2019) or `VIEW DATABASE PERFORMANCE STATE` (SQL Server 2022 and later). +```sql +GRANT VIEW DATABASE STATE TO YourUser; +-- SQL Server 2022 and later: GRANT VIEW DATABASE PERFORMANCE STATE TO YourUser; +``` +Server-level, for the query text and the plan-cache DMV fallback: `VIEW SERVER STATE` (SQL Server 2019 and earlier) or `VIEW SERVER PERFORMANCE STATE` (SQL Server 2022 and later). ```sql GRANT VIEW SERVER STATE TO YourUser; +-- SQL Server 2022 and later: GRANT VIEW SERVER PERFORMANCE STATE TO YourUser; ``` +Exact grant names and tiers can differ on Azure SQL Database and Azure SQL Managed Instance. Check `sys.database_query_store_options` and the [Microsoft documentation](https://learn.microsoft.com/en-us/sql/relational-databases/system-catalog-views/sys-query-store-query-text-transact-sql) for your instance. ### Python Requirements To run the MSSQL ingestion, you will need to install: diff --git a/v1.13.x/connectors/ingestion/workflows/lineage/filter-query-set.mdx b/v1.13.x/connectors/ingestion/workflows/lineage/filter-query-set.mdx index c921f985..5ea668f8 100644 --- a/v1.13.x/connectors/ingestion/workflows/lineage/filter-query-set.mdx +++ b/v1.13.x/connectors/ingestion/workflows/lineage/filter-query-set.mdx @@ -79,7 +79,48 @@ For example if you want to add a condition to filter out queries executed by met ## MSSQL Filter Condition -To fetch the query history log from MSSQL we execute the following query +MSSQL reads query history from Query Store when it's enabled on a database, otherwise from the plan-cache DMVs. Your filter condition applies to whichever source is active for that database. + +When Query Store is enabled, OpenMetadata executes the following query + +``` +SELECT TOP {result_limit} + t.database_name, + t.text query_text, + t.start_time, + t.end_time, + t.duration, + NULL schema_name, + NULL query_type, + NULL user_name, + NULL aborted +FROM ( + SELECT + DB_NAME() AS database_name, + qt.query_sql_text AS text, + MAX(rs.last_execution_time) AS start_time, + DATEADD(s, CAST(AVG(rs.avg_duration) / 1000000 AS INT), MAX(rs.last_execution_time)) AS end_time, + AVG(rs.avg_duration) / 1000000.0 AS duration + FROM sys.query_store_query AS q + INNER JOIN sys.query_store_query_text AS qt + ON qt.query_text_id = q.query_text_id + INNER JOIN sys.query_store_plan AS p + ON p.query_id = q.query_id + INNER JOIN sys.query_store_runtime_stats AS rs + ON rs.plan_id = p.plan_id + WHERE ISNULL(q.object_id, 0) = 0 + AND rs.last_execution_time BETWEEN '{start_time}' AND '{end_time}' + GROUP BY q.query_id, qt.query_sql_text +) AS t +WHERE t.text NOT LIKE '/* {{"app": "OpenMetadata", %%}} */%%' + AND t.text NOT LIKE '/* {{"app": "dbt", %%}} */%%' + + AND {**your filter condition**} + +ORDER BY t.start_time DESC +``` + +When Query Store is not enabled, OpenMetadata falls back to the following query against the plan-cache DMVs ``` SELECT TOP {result_limit} diff --git a/v1.13.x/connectors/ingestion/workflows/usage/filter-query-set.mdx b/v1.13.x/connectors/ingestion/workflows/usage/filter-query-set.mdx index 95ddd7d9..20f89c69 100644 --- a/v1.13.x/connectors/ingestion/workflows/usage/filter-query-set.mdx +++ b/v1.13.x/connectors/ingestion/workflows/usage/filter-query-set.mdx @@ -80,7 +80,48 @@ For example if you want to add a condition to filter out queries executed by met ## MSSQL Filter Condition -To fetch the query history log from MSSQL we execute the following query +MSSQL reads query history from Query Store when it's enabled on a database, otherwise from the plan-cache DMVs. Your filter condition applies to whichever source is active for that database. + +When Query Store is enabled, OpenMetadata executes the following query + +``` +SELECT TOP {result_limit} + t.database_name, + t.text query_text, + t.start_time, + t.end_time, + t.duration, + NULL schema_name, + NULL query_type, + NULL user_name, + NULL aborted +FROM ( + SELECT + DB_NAME() AS database_name, + qt.query_sql_text AS text, + MAX(rs.last_execution_time) AS start_time, + DATEADD(s, CAST(AVG(rs.avg_duration) / 1000000 AS INT), MAX(rs.last_execution_time)) AS end_time, + AVG(rs.avg_duration) / 1000000.0 AS duration + FROM sys.query_store_query AS q + INNER JOIN sys.query_store_query_text AS qt + ON qt.query_text_id = q.query_text_id + INNER JOIN sys.query_store_plan AS p + ON p.query_id = q.query_id + INNER JOIN sys.query_store_runtime_stats AS rs + ON rs.plan_id = p.plan_id + WHERE ISNULL(q.object_id, 0) = 0 + AND rs.last_execution_time BETWEEN '{start_time}' AND '{end_time}' + GROUP BY q.query_id, qt.query_sql_text +) AS t +WHERE t.text NOT LIKE '/* {{"app": "OpenMetadata", %%}} */%%' + AND t.text NOT LIKE '/* {{"app": "dbt", %%}} */%%' + + AND {**your filter condition**} + +ORDER BY t.start_time DESC +``` + +When Query Store is not enabled, OpenMetadata falls back to the following query against the plan-cache DMVs ``` SELECT TOP {result_limit} diff --git a/v2.0.x-SNAPSHOT/connectors/database/mssql.mdx b/v2.0.x-SNAPSHOT/connectors/database/mssql.mdx index 216c7901..74b2da36 100644 --- a/v2.0.x-SNAPSHOT/connectors/database/mssql.mdx +++ b/v2.0.x-SNAPSHOT/connectors/database/mssql.mdx @@ -52,10 +52,44 @@ GRANT VIEW DEFINITION TO Mary; GRANT VIEW ANY DEFINITION TO [login_name]; ``` ### Usage & Lineage consideration -To perform the query analysis for Usage and Lineage computation, we fetch the query logs from `sys.dm_exec_cached_plans`, `sys.dm_exec_query_stats` & `sys.dm_exec_sql_text` system tables. To access these tables your user must have `VIEW SERVER STATE` privilege. +To perform the query analysis for Usage and Lineage computation, the connector reads query history from SQL Server's [Query Store](https://learn.microsoft.com/en-us/sql/relational-databases/performance/monitoring-performance-by-using-the-query-store) when it is enabled on a database. Query Store exposes this history through `sys.query_store_query`, `sys.query_store_query_text`, `sys.query_store_plan`, and `sys.query_store_runtime_stats`. Query Store keeps a durable, on-disk record of executed queries. The plan-cache Dynamic Management Views (DMVs) (`sys.dm_exec_cached_plans`, `sys.dm_exec_query_stats`, and `sys.dm_exec_sql_text`) are evicted on server restart, memory pressure, or plan recompilation, and can silently return incomplete history. + +Query Store is auto-detected per database. No configuration is needed. When `ingestAllDatabases` is enabled, each database is read from its own best source independently: databases with Query Store use it, and the rest fall back to the plan-cache DMVs, so one database without Query Store never downgrades the others. + + +**Tip:** Enable Query Store on the databases you ingest, for reliable and durable Usage and Lineage history. + + +To check whether Query Store is enabled on a database, and how it is currently operating: +```sql +SELECT actual_state_desc, query_capture_mode_desc, size_based_cleanup_mode_desc +FROM sys.database_query_store_options; +``` +To enable it: +```sql +ALTER DATABASE [YourDatabase] SET QUERY_STORE = ON +( + QUERY_CAPTURE_MODE = ALL, + CLEANUP_POLICY = ( STALE_QUERY_THRESHOLD_DAYS = 30 ) +); +``` +The default `QUERY_CAPTURE_MODE` on SQL Server 2019 and later is `AUTO`, which can skip queries that only run a few times, such as a one-off data migration statement. Setting it to `ALL` ensures every query is captured for lineage. Set `STALE_QUERY_THRESHOLD_DAYS` to cover at least your ingestion lookback window, so queries are not purged before they are read. Query Store's default `SIZE_BASED_CLEANUP_MODE` purges the oldest queries to stay within `MAX_STORAGE_SIZE_MB`, which reduces the risk of Query Store switching to read-only. Under heavy write load, cleanup can still fall behind, and Query Store can switch to read-only temporarily until it catches up. + + +Query Store needs two permissions together. The database-level grant below lets `sys.database_query_store_options` and Query Store's other metadata views succeed, but the query text comes from `sys.query_store_query_text`, which needs the server-level grant. Without the server-level grant, usage and lineage queries can run without error while returning no query text. + + +Database-level, for Query Store metadata: `VIEW DATABASE STATE` (SQL Server 2016-2019) or `VIEW DATABASE PERFORMANCE STATE` (SQL Server 2022 and later). +```sql +GRANT VIEW DATABASE STATE TO YourUser; +-- SQL Server 2022 and later: GRANT VIEW DATABASE PERFORMANCE STATE TO YourUser; +``` +Server-level, for the query text and the plan-cache DMV fallback: `VIEW SERVER STATE` (SQL Server 2019 and earlier) or `VIEW SERVER PERFORMANCE STATE` (SQL Server 2022 and later). ```sql GRANT VIEW SERVER STATE TO YourUser; +-- SQL Server 2022 and later: GRANT VIEW SERVER PERFORMANCE STATE TO YourUser; ``` +Exact grant names and tiers can differ on Azure SQL Database and Azure SQL Managed Instance. Check `sys.database_query_store_options` and the [Microsoft documentation](https://learn.microsoft.com/en-us/sql/relational-databases/system-catalog-views/sys-query-store-query-text-transact-sql) for your instance. ### For Remote Connection #### 1. SQL Server running Make sure the SQL server that you are trying to connect is in running state. diff --git a/v2.0.x-SNAPSHOT/connectors/database/mssql/troubleshoot.mdx b/v2.0.x-SNAPSHOT/connectors/database/mssql/troubleshoot.mdx index 4f8026e5..bd8d6df7 100644 --- a/v2.0.x-SNAPSHOT/connectors/database/mssql/troubleshoot.mdx +++ b/v2.0.x-SNAPSHOT/connectors/database/mssql/troubleshoot.mdx @@ -58,3 +58,7 @@ After updating the connection scheme, the connection should succeed. The status ``` Connection Status: Success ``` + +## Stored Procedure Lineage Across Databases + +If the same stored procedure name exists in more than one database or schema, the connector matches each query to its procedure using the database and schema reported by the query source. If the match is still ambiguous, that query is skipped rather than attached to the wrong procedure. diff --git a/v2.0.x-SNAPSHOT/connectors/database/mssql/yaml.mdx b/v2.0.x-SNAPSHOT/connectors/database/mssql/yaml.mdx index d12691c6..4e8e6f33 100644 --- a/v2.0.x-SNAPSHOT/connectors/database/mssql/yaml.mdx +++ b/v2.0.x-SNAPSHOT/connectors/database/mssql/yaml.mdx @@ -60,10 +60,44 @@ GRANT VIEW DEFINITION TO Mary; GRANT VIEW ANY DEFINITION TO [login_name]; ``` ### Usage & Lineage consideration -To perform the query analysis for Usage and Lineage computation, we fetch the query logs from `sys.dm_exec_cached_plans`, `sys.dm_exec_query_stats` & `sys.dm_exec_sql_text` system tables. To access these tables your user must have `VIEW SERVER STATE` privilege. +To perform the query analysis for Usage and Lineage computation, the connector reads query history from SQL Server's [Query Store](https://learn.microsoft.com/en-us/sql/relational-databases/performance/monitoring-performance-by-using-the-query-store) when it is enabled on a database. Query Store exposes this history through `sys.query_store_query`, `sys.query_store_query_text`, `sys.query_store_plan`, and `sys.query_store_runtime_stats`. Query Store keeps a durable, on-disk record of executed queries. The plan-cache Dynamic Management Views (DMVs) (`sys.dm_exec_cached_plans`, `sys.dm_exec_query_stats`, and `sys.dm_exec_sql_text`) are evicted on server restart, memory pressure, or plan recompilation, and can silently return incomplete history. + +Query Store is auto-detected per database. No configuration is needed. When `ingestAllDatabases` is enabled, each database is read from its own best source independently: databases with Query Store use it, and the rest fall back to the plan-cache DMVs, so one database without Query Store never downgrades the others. + + +**Tip:** Enable Query Store on the databases you ingest, for reliable and durable Usage and Lineage history. + + +To check whether Query Store is enabled on a database, and how it is currently operating: +```sql +SELECT actual_state_desc, query_capture_mode_desc, size_based_cleanup_mode_desc +FROM sys.database_query_store_options; +``` +To enable it: +```sql +ALTER DATABASE [YourDatabase] SET QUERY_STORE = ON +( + QUERY_CAPTURE_MODE = ALL, + CLEANUP_POLICY = ( STALE_QUERY_THRESHOLD_DAYS = 30 ) +); +``` +The default `QUERY_CAPTURE_MODE` on SQL Server 2019 and later is `AUTO`, which can skip queries that only run a few times, such as a one-off data migration statement. Setting it to `ALL` ensures every query is captured for lineage. Set `STALE_QUERY_THRESHOLD_DAYS` to cover at least your ingestion lookback window, so queries are not purged before they are read. Query Store's default `SIZE_BASED_CLEANUP_MODE` purges the oldest queries to stay within `MAX_STORAGE_SIZE_MB`, which reduces the risk of Query Store switching to read-only. Under heavy write load, cleanup can still fall behind, and Query Store can switch to read-only temporarily until it catches up. + + +Query Store needs two permissions together. The database-level grant below lets `sys.database_query_store_options` and Query Store's other metadata views succeed, but the query text comes from `sys.query_store_query_text`, which needs the server-level grant. Without the server-level grant, usage and lineage queries can run without error while returning no query text. + + +Database-level, for Query Store metadata: `VIEW DATABASE STATE` (SQL Server 2016-2019) or `VIEW DATABASE PERFORMANCE STATE` (SQL Server 2022 and later). +```sql +GRANT VIEW DATABASE STATE TO YourUser; +-- SQL Server 2022 and later: GRANT VIEW DATABASE PERFORMANCE STATE TO YourUser; +``` +Server-level, for the query text and the plan-cache DMV fallback: `VIEW SERVER STATE` (SQL Server 2019 and earlier) or `VIEW SERVER PERFORMANCE STATE` (SQL Server 2022 and later). ```sql GRANT VIEW SERVER STATE TO YourUser; +-- SQL Server 2022 and later: GRANT VIEW SERVER PERFORMANCE STATE TO YourUser; ``` +Exact grant names and tiers can differ on Azure SQL Database and Azure SQL Managed Instance. Check `sys.database_query_store_options` and the [Microsoft documentation](https://learn.microsoft.com/en-us/sql/relational-databases/system-catalog-views/sys-query-store-query-text-transact-sql) for your instance. ### Python Requirements To run the MSSQL ingestion, you will need to install: diff --git a/v2.0.x-SNAPSHOT/connectors/ingestion/workflows/lineage/filter-query-set.mdx b/v2.0.x-SNAPSHOT/connectors/ingestion/workflows/lineage/filter-query-set.mdx index c921f985..5ea668f8 100644 --- a/v2.0.x-SNAPSHOT/connectors/ingestion/workflows/lineage/filter-query-set.mdx +++ b/v2.0.x-SNAPSHOT/connectors/ingestion/workflows/lineage/filter-query-set.mdx @@ -79,7 +79,48 @@ For example if you want to add a condition to filter out queries executed by met ## MSSQL Filter Condition -To fetch the query history log from MSSQL we execute the following query +MSSQL reads query history from Query Store when it's enabled on a database, otherwise from the plan-cache DMVs. Your filter condition applies to whichever source is active for that database. + +When Query Store is enabled, OpenMetadata executes the following query + +``` +SELECT TOP {result_limit} + t.database_name, + t.text query_text, + t.start_time, + t.end_time, + t.duration, + NULL schema_name, + NULL query_type, + NULL user_name, + NULL aborted +FROM ( + SELECT + DB_NAME() AS database_name, + qt.query_sql_text AS text, + MAX(rs.last_execution_time) AS start_time, + DATEADD(s, CAST(AVG(rs.avg_duration) / 1000000 AS INT), MAX(rs.last_execution_time)) AS end_time, + AVG(rs.avg_duration) / 1000000.0 AS duration + FROM sys.query_store_query AS q + INNER JOIN sys.query_store_query_text AS qt + ON qt.query_text_id = q.query_text_id + INNER JOIN sys.query_store_plan AS p + ON p.query_id = q.query_id + INNER JOIN sys.query_store_runtime_stats AS rs + ON rs.plan_id = p.plan_id + WHERE ISNULL(q.object_id, 0) = 0 + AND rs.last_execution_time BETWEEN '{start_time}' AND '{end_time}' + GROUP BY q.query_id, qt.query_sql_text +) AS t +WHERE t.text NOT LIKE '/* {{"app": "OpenMetadata", %%}} */%%' + AND t.text NOT LIKE '/* {{"app": "dbt", %%}} */%%' + + AND {**your filter condition**} + +ORDER BY t.start_time DESC +``` + +When Query Store is not enabled, OpenMetadata falls back to the following query against the plan-cache DMVs ``` SELECT TOP {result_limit} diff --git a/v2.0.x-SNAPSHOT/connectors/ingestion/workflows/usage/filter-query-set.mdx b/v2.0.x-SNAPSHOT/connectors/ingestion/workflows/usage/filter-query-set.mdx index 95ddd7d9..20f89c69 100644 --- a/v2.0.x-SNAPSHOT/connectors/ingestion/workflows/usage/filter-query-set.mdx +++ b/v2.0.x-SNAPSHOT/connectors/ingestion/workflows/usage/filter-query-set.mdx @@ -80,7 +80,48 @@ For example if you want to add a condition to filter out queries executed by met ## MSSQL Filter Condition -To fetch the query history log from MSSQL we execute the following query +MSSQL reads query history from Query Store when it's enabled on a database, otherwise from the plan-cache DMVs. Your filter condition applies to whichever source is active for that database. + +When Query Store is enabled, OpenMetadata executes the following query + +``` +SELECT TOP {result_limit} + t.database_name, + t.text query_text, + t.start_time, + t.end_time, + t.duration, + NULL schema_name, + NULL query_type, + NULL user_name, + NULL aborted +FROM ( + SELECT + DB_NAME() AS database_name, + qt.query_sql_text AS text, + MAX(rs.last_execution_time) AS start_time, + DATEADD(s, CAST(AVG(rs.avg_duration) / 1000000 AS INT), MAX(rs.last_execution_time)) AS end_time, + AVG(rs.avg_duration) / 1000000.0 AS duration + FROM sys.query_store_query AS q + INNER JOIN sys.query_store_query_text AS qt + ON qt.query_text_id = q.query_text_id + INNER JOIN sys.query_store_plan AS p + ON p.query_id = q.query_id + INNER JOIN sys.query_store_runtime_stats AS rs + ON rs.plan_id = p.plan_id + WHERE ISNULL(q.object_id, 0) = 0 + AND rs.last_execution_time BETWEEN '{start_time}' AND '{end_time}' + GROUP BY q.query_id, qt.query_sql_text +) AS t +WHERE t.text NOT LIKE '/* {{"app": "OpenMetadata", %%}} */%%' + AND t.text NOT LIKE '/* {{"app": "dbt", %%}} */%%' + + AND {**your filter condition**} + +ORDER BY t.start_time DESC +``` + +When Query Store is not enabled, OpenMetadata falls back to the following query against the plan-cache DMVs ``` SELECT TOP {result_limit} From 2ec162471d6d05cdbdc61d7905f67b006e15dec9 Mon Sep 17 00:00:00 2001 From: Mitali Daduria Date: Wed, 22 Jul 2026 19:42:10 +0530 Subject: [PATCH 7/9] docs: restore metadata-standard.mdx and move glossary to canonical path --- v1.13.x/main-concepts/metadata-standard.mdx | 74 +++++++++++++-------- 1 file changed, 45 insertions(+), 29 deletions(-) diff --git a/v1.13.x/main-concepts/metadata-standard.mdx b/v1.13.x/main-concepts/metadata-standard.mdx index 63e93e15..dcf71c7b 100644 --- a/v1.13.x/main-concepts/metadata-standard.mdx +++ b/v1.13.x/main-concepts/metadata-standard.mdx @@ -1,29 +1,45 @@ -## Machine Learning & Data Governance Expansion - -The following terms bridge common machine learning and data governance concepts to business-friendly language, with payments-domain examples showing how each one shows up in practice. - -#### Model Drift -- **Technical Definition**: The degradation of a machine learning model's predictive performance over time due to changes in the statistical distribution of the environment's underlying data. -- **Business Translation**: When a previously accurate AI model starts making poorer predictions because real-world consumer behavior or market conditions have evolved since the model was trained. -- **Payments Example**: A fraud detection model trained on historical data from early 2024 may experience seasonal model drift during the November/December holiday shopping rush, as legitimate spike buying behaviors mimic historic fraud patterns. - -#### Data Lineage -- **Technical Definition**: The end-to-end lifecycle of data that maps its origins, structural transformations, intermediate pipeline stages, and ultimate consumption endpoints across the data ecosystem. -- **Business Translation**: A visual audit trail that shows exactly where a piece of data came from, how it was modified along the way, and which final business dashboards or reports are using it. -- **Payments Example**: Tracing an interchange_fee_credited field from the raw settlement API payload, through the daily ETL cleansing pipelines, to the monthly CFO financial compliance dashboard. - -#### Feature Engineering -- **Technical Definition**: The process of using domain knowledge to select, transform, combine, and manipulate raw data variables into distinct predictor variables (features) that enhance machine learning algorithm performance. -- **Business Translation**: Extracting and shaping raw, unorganized data points into meaningful business metrics so an AI model can spot patterns more easily. -- **Payments Example**: Converting raw, individual transaction timestamps into a calculated feature like count_of_transactions_in_last_5_minutes to help a model identify high-frequency card-cloning velocity attacks. - -#### Label Skew -- **Technical Definition**: A specific type of dataset imbalance occurring during training or evaluation where one target class label vastly outnumbers the alternative class labels. -- **Business Translation**: A training bias that happens when the data given to an AI contains overwhelming examples of one outcome and barely any examples of another, making it blind to rarer events. -- **Payments Example**: In a payment dataset of 10 million transactions, only 0.01% are actual chargebacks. Training a model on this creates severe label skew, causing the system to default to guessing "legitimate" to achieve high but hollow accuracy. - -#### Data Quality SLA (Service Level Agreement) -- **Technical Definition**: A formal commitment that defines measurable performance thresholds for critical data characteristics—such as freshness, schema adherence, completeness, and validity—enforced by automated testing pipelines. -- **Business Translation**: A contract between data teams and business stakeholders ensuring that the data powering operations arrives on time, with full completeness, and free of format errors. -- **Payments Example**: A data contract specifying that the merchant_settlement_ledger table must be fully populated and accurate by 06:00 UTC daily, with a hard failure alert triggered if missing row counts exceed 0.05%. - +--- +title: Metadata Standard | OpenMetadata Core Schema Guide +description: Overview of metadata standard schemas governing how entities, types, analytics, and governance structures are defined. +sidebarTitle: Overview +--- + +# OpenMetadata Standards + +OpenMetadata Standards is the community-driven home for the schemas and ontologies behind OpenMetadata. It provides the canonical JSON Schemas, RDF/OWL models, and API specifications that power data cataloging, governance, lineage, quality, and observability. + +Key highlights: + +- **Comprehensive coverage**: 700+ JSON Schemas spanning entities, relationships, events, and configuration. +- **API-first**: REST and event schemas for search, feeds, webhooks, and bulk operations. +- **Semantic web ready**: RDF/OWL ontologies and SHACL shapes for knowledge graph and linked data use cases. +- **Standards compliant**: JSON Schema (Draft 07/2020-12), RDF/OWL, SHACL, JSON-LD, PROV-O. +- **Community driven**: Built and maintained by the OpenMetadata community with open governance. + +## Schemas + +The OpenMetadata standard is powered by [JSON Schemas](https://json-schema.org/), as a readable and language-agnostic +solution that allows us to automatically generate code for the different pieces of the project. + + + +Curious about how OpenMetadata is built? You can take a look at the [High Level Design](/v1.13.x/main-concepts/high-level-design). + + + +You can explore the JSON Schemas in different ways: + +**1.** You can check all the definitions in [GitHub](https://github.com/open-metadata/OpenMetadata/tree/main/openmetadata-spec/src/main/resources/json/schema). + Navigating through the directories you can find, for example, the definition of a [Table](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/entity/data/table.json). + +**2.** If you prefer to stay in the docs, we also have you covered. We have converted the JSON Schemas to markdown files + that you can explore following the left menu titles, or navigating the structure by clicking on `Next` at the + end of each page. Again, this would be the structure of a [Table](/v1.13.x/main-concepts/metadata-standard). + In this generation we wanted to maintain the same structure as the GitHub repository. Any empty file means that you reached + a directory, but you can keep exploring until finding the right Entity, or you can also look for it using the Search Bar. + +Ready to dive in? + +Visit openmetadatastandards.org + +You'll find quick starts, schema references, semantic web resources, and contribution guides on the official site. From d9ecca127d229f62f47bc60729d8ca3420d1fe25 Mon Sep 17 00:00:00 2001 From: Mitali Daduria Date: Wed, 22 Jul 2026 19:52:17 +0530 Subject: [PATCH 8/9] docs: restore metadata-standard.mdx and move glossary to canonical path --- v1.13.x/main-concepts/metadata-standard.mdx | 74 ++++++++------------- 1 file changed, 29 insertions(+), 45 deletions(-) diff --git a/v1.13.x/main-concepts/metadata-standard.mdx b/v1.13.x/main-concepts/metadata-standard.mdx index dcf71c7b..63e93e15 100644 --- a/v1.13.x/main-concepts/metadata-standard.mdx +++ b/v1.13.x/main-concepts/metadata-standard.mdx @@ -1,45 +1,29 @@ ---- -title: Metadata Standard | OpenMetadata Core Schema Guide -description: Overview of metadata standard schemas governing how entities, types, analytics, and governance structures are defined. -sidebarTitle: Overview ---- - -# OpenMetadata Standards - -OpenMetadata Standards is the community-driven home for the schemas and ontologies behind OpenMetadata. It provides the canonical JSON Schemas, RDF/OWL models, and API specifications that power data cataloging, governance, lineage, quality, and observability. - -Key highlights: - -- **Comprehensive coverage**: 700+ JSON Schemas spanning entities, relationships, events, and configuration. -- **API-first**: REST and event schemas for search, feeds, webhooks, and bulk operations. -- **Semantic web ready**: RDF/OWL ontologies and SHACL shapes for knowledge graph and linked data use cases. -- **Standards compliant**: JSON Schema (Draft 07/2020-12), RDF/OWL, SHACL, JSON-LD, PROV-O. -- **Community driven**: Built and maintained by the OpenMetadata community with open governance. - -## Schemas - -The OpenMetadata standard is powered by [JSON Schemas](https://json-schema.org/), as a readable and language-agnostic -solution that allows us to automatically generate code for the different pieces of the project. - - - -Curious about how OpenMetadata is built? You can take a look at the [High Level Design](/v1.13.x/main-concepts/high-level-design). - - - -You can explore the JSON Schemas in different ways: - -**1.** You can check all the definitions in [GitHub](https://github.com/open-metadata/OpenMetadata/tree/main/openmetadata-spec/src/main/resources/json/schema). - Navigating through the directories you can find, for example, the definition of a [Table](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/entity/data/table.json). - -**2.** If you prefer to stay in the docs, we also have you covered. We have converted the JSON Schemas to markdown files - that you can explore following the left menu titles, or navigating the structure by clicking on `Next` at the - end of each page. Again, this would be the structure of a [Table](/v1.13.x/main-concepts/metadata-standard). - In this generation we wanted to maintain the same structure as the GitHub repository. Any empty file means that you reached - a directory, but you can keep exploring until finding the right Entity, or you can also look for it using the Search Bar. - -Ready to dive in? - -Visit openmetadatastandards.org - -You'll find quick starts, schema references, semantic web resources, and contribution guides on the official site. +## Machine Learning & Data Governance Expansion + +The following terms bridge common machine learning and data governance concepts to business-friendly language, with payments-domain examples showing how each one shows up in practice. + +#### Model Drift +- **Technical Definition**: The degradation of a machine learning model's predictive performance over time due to changes in the statistical distribution of the environment's underlying data. +- **Business Translation**: When a previously accurate AI model starts making poorer predictions because real-world consumer behavior or market conditions have evolved since the model was trained. +- **Payments Example**: A fraud detection model trained on historical data from early 2024 may experience seasonal model drift during the November/December holiday shopping rush, as legitimate spike buying behaviors mimic historic fraud patterns. + +#### Data Lineage +- **Technical Definition**: The end-to-end lifecycle of data that maps its origins, structural transformations, intermediate pipeline stages, and ultimate consumption endpoints across the data ecosystem. +- **Business Translation**: A visual audit trail that shows exactly where a piece of data came from, how it was modified along the way, and which final business dashboards or reports are using it. +- **Payments Example**: Tracing an interchange_fee_credited field from the raw settlement API payload, through the daily ETL cleansing pipelines, to the monthly CFO financial compliance dashboard. + +#### Feature Engineering +- **Technical Definition**: The process of using domain knowledge to select, transform, combine, and manipulate raw data variables into distinct predictor variables (features) that enhance machine learning algorithm performance. +- **Business Translation**: Extracting and shaping raw, unorganized data points into meaningful business metrics so an AI model can spot patterns more easily. +- **Payments Example**: Converting raw, individual transaction timestamps into a calculated feature like count_of_transactions_in_last_5_minutes to help a model identify high-frequency card-cloning velocity attacks. + +#### Label Skew +- **Technical Definition**: A specific type of dataset imbalance occurring during training or evaluation where one target class label vastly outnumbers the alternative class labels. +- **Business Translation**: A training bias that happens when the data given to an AI contains overwhelming examples of one outcome and barely any examples of another, making it blind to rarer events. +- **Payments Example**: In a payment dataset of 10 million transactions, only 0.01% are actual chargebacks. Training a model on this creates severe label skew, causing the system to default to guessing "legitimate" to achieve high but hollow accuracy. + +#### Data Quality SLA (Service Level Agreement) +- **Technical Definition**: A formal commitment that defines measurable performance thresholds for critical data characteristics—such as freshness, schema adherence, completeness, and validity—enforced by automated testing pipelines. +- **Business Translation**: A contract between data teams and business stakeholders ensuring that the data powering operations arrives on time, with full completeness, and free of format errors. +- **Payments Example**: A data contract specifying that the merchant_settlement_ledger table must be fully populated and accurate by 06:00 UTC daily, with a hard failure alert triggered if missing row counts exceed 0.05%. + From a1a3eb7fd827be676df8f21d73b938108da8d6dd Mon Sep 17 00:00:00 2001 From: Mitali Daduria Date: Wed, 22 Jul 2026 20:07:52 +0530 Subject: [PATCH 9/9] docs: restore metadata-standard.mdx and move glossary to canonical path --- .../main-concepts/metadata-standard.mdx | 29 ++++++++ v1.13.x/main-concepts/metadata-standard.mdx | 74 +++++++++++-------- 2 files changed, 74 insertions(+), 29 deletions(-) diff --git a/v1.13.x/api-reference/main-concepts/metadata-standard.mdx b/v1.13.x/api-reference/main-concepts/metadata-standard.mdx index dcf71c7b..16bb8a3b 100644 --- a/v1.13.x/api-reference/main-concepts/metadata-standard.mdx +++ b/v1.13.x/api-reference/main-concepts/metadata-standard.mdx @@ -43,3 +43,32 @@ Ready to dive in? Visit openmetadatastandards.org You'll find quick starts, schema references, semantic web resources, and contribution guides on the official site. + +## Machine Learning & Data Governance Expansion + +The following terms bridge common machine learning and data governance concepts to business-friendly language, with payments-domain examples showing how each one shows up in practice. + +#### Model Drift +* **Technical Definition**: Performance degradation of a model over time caused by changes in real-world data distribution. +* **Business Translation**: The model's predictions become less accurate as customer habits or external conditions evolve. +* **Payments Example**: A fraud detection model trained on pre-holiday spending flags normal holiday shopping sprees as fraudulent. + +#### Data Lineage +* **Technical Definition**: The complete lifecycle trajectory of data, tracking its origins, transformations, and final destinations. +* **Business Translation**: A visible paper trail showing where data came from and how it was modified before reaching reports. +* **Payments Example**: Tracing a payout error back through intermediate currency conversions to the raw transaction record. + +#### Feature Engineering +* **Technical Definition**: The process of selecting, manipulating, and transforming raw data into useful input variables for machine learning models. +* **Business Translation**: Converting raw transaction records into meaningful signals that help the model spot patterns. +* **Payments Example**: Transforming raw transaction timestamps into a "transactions per hour" velocity feature to detect card-testing bots. + +#### Label Skew +* **Technical Definition**: Discrepancy between ground truth labels used during training and actual outcomes in production. +* **Business Translation**: When real-world results don't match the historical data assumptions used to train the system. +* **Payments Example**: A chargeback model trained only on settled transactions misses early fraud patterns from pending authorizations. + +#### Data Quality SLA (Service Level Agreement) +* **Technical Definition**: Formal agreements defining measurable thresholds for data accuracy, completeness, latency, and freshness. +* **Business Translation**: Guaranteed standards ensuring data arrives on time and is reliable enough for business operations. +* **Payments Example**: A operational rule requiring payment reconciliation feeds to be updated within 15 minutes with 99.9% completeness. \ No newline at end of file diff --git a/v1.13.x/main-concepts/metadata-standard.mdx b/v1.13.x/main-concepts/metadata-standard.mdx index 63e93e15..dcf71c7b 100644 --- a/v1.13.x/main-concepts/metadata-standard.mdx +++ b/v1.13.x/main-concepts/metadata-standard.mdx @@ -1,29 +1,45 @@ -## Machine Learning & Data Governance Expansion - -The following terms bridge common machine learning and data governance concepts to business-friendly language, with payments-domain examples showing how each one shows up in practice. - -#### Model Drift -- **Technical Definition**: The degradation of a machine learning model's predictive performance over time due to changes in the statistical distribution of the environment's underlying data. -- **Business Translation**: When a previously accurate AI model starts making poorer predictions because real-world consumer behavior or market conditions have evolved since the model was trained. -- **Payments Example**: A fraud detection model trained on historical data from early 2024 may experience seasonal model drift during the November/December holiday shopping rush, as legitimate spike buying behaviors mimic historic fraud patterns. - -#### Data Lineage -- **Technical Definition**: The end-to-end lifecycle of data that maps its origins, structural transformations, intermediate pipeline stages, and ultimate consumption endpoints across the data ecosystem. -- **Business Translation**: A visual audit trail that shows exactly where a piece of data came from, how it was modified along the way, and which final business dashboards or reports are using it. -- **Payments Example**: Tracing an interchange_fee_credited field from the raw settlement API payload, through the daily ETL cleansing pipelines, to the monthly CFO financial compliance dashboard. - -#### Feature Engineering -- **Technical Definition**: The process of using domain knowledge to select, transform, combine, and manipulate raw data variables into distinct predictor variables (features) that enhance machine learning algorithm performance. -- **Business Translation**: Extracting and shaping raw, unorganized data points into meaningful business metrics so an AI model can spot patterns more easily. -- **Payments Example**: Converting raw, individual transaction timestamps into a calculated feature like count_of_transactions_in_last_5_minutes to help a model identify high-frequency card-cloning velocity attacks. - -#### Label Skew -- **Technical Definition**: A specific type of dataset imbalance occurring during training or evaluation where one target class label vastly outnumbers the alternative class labels. -- **Business Translation**: A training bias that happens when the data given to an AI contains overwhelming examples of one outcome and barely any examples of another, making it blind to rarer events. -- **Payments Example**: In a payment dataset of 10 million transactions, only 0.01% are actual chargebacks. Training a model on this creates severe label skew, causing the system to default to guessing "legitimate" to achieve high but hollow accuracy. - -#### Data Quality SLA (Service Level Agreement) -- **Technical Definition**: A formal commitment that defines measurable performance thresholds for critical data characteristics—such as freshness, schema adherence, completeness, and validity—enforced by automated testing pipelines. -- **Business Translation**: A contract between data teams and business stakeholders ensuring that the data powering operations arrives on time, with full completeness, and free of format errors. -- **Payments Example**: A data contract specifying that the merchant_settlement_ledger table must be fully populated and accurate by 06:00 UTC daily, with a hard failure alert triggered if missing row counts exceed 0.05%. - +--- +title: Metadata Standard | OpenMetadata Core Schema Guide +description: Overview of metadata standard schemas governing how entities, types, analytics, and governance structures are defined. +sidebarTitle: Overview +--- + +# OpenMetadata Standards + +OpenMetadata Standards is the community-driven home for the schemas and ontologies behind OpenMetadata. It provides the canonical JSON Schemas, RDF/OWL models, and API specifications that power data cataloging, governance, lineage, quality, and observability. + +Key highlights: + +- **Comprehensive coverage**: 700+ JSON Schemas spanning entities, relationships, events, and configuration. +- **API-first**: REST and event schemas for search, feeds, webhooks, and bulk operations. +- **Semantic web ready**: RDF/OWL ontologies and SHACL shapes for knowledge graph and linked data use cases. +- **Standards compliant**: JSON Schema (Draft 07/2020-12), RDF/OWL, SHACL, JSON-LD, PROV-O. +- **Community driven**: Built and maintained by the OpenMetadata community with open governance. + +## Schemas + +The OpenMetadata standard is powered by [JSON Schemas](https://json-schema.org/), as a readable and language-agnostic +solution that allows us to automatically generate code for the different pieces of the project. + + + +Curious about how OpenMetadata is built? You can take a look at the [High Level Design](/v1.13.x/main-concepts/high-level-design). + + + +You can explore the JSON Schemas in different ways: + +**1.** You can check all the definitions in [GitHub](https://github.com/open-metadata/OpenMetadata/tree/main/openmetadata-spec/src/main/resources/json/schema). + Navigating through the directories you can find, for example, the definition of a [Table](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/entity/data/table.json). + +**2.** If you prefer to stay in the docs, we also have you covered. We have converted the JSON Schemas to markdown files + that you can explore following the left menu titles, or navigating the structure by clicking on `Next` at the + end of each page. Again, this would be the structure of a [Table](/v1.13.x/main-concepts/metadata-standard). + In this generation we wanted to maintain the same structure as the GitHub repository. Any empty file means that you reached + a directory, but you can keep exploring until finding the right Entity, or you can also look for it using the Search Bar. + +Ready to dive in? + +Visit openmetadatastandards.org + +You'll find quick starts, schema references, semantic web resources, and contribution guides on the official site.