Publish standalone Apache Airflow Mypy documentation

.github/workflows/publish-docs-to-s3.yml

@@ -153,7 +153,7 @@ jobs:
         run: |
           # Keep the non-provider list in sync with NON_PROVIDER_TOKENS in
           # dev/registry/derive_wave_providers.py.
-          NON_PROVIDER_TOKENS=("apache-airflow" "apache-airflow-ctl" "task-sdk" "helm-chart" "docker-stack")
+          NON_PROVIDER_TOKENS=("apache-airflow" "apache-airflow-ctl" "apache-airflow-mypy" "task-sdk" "helm-chart" "docker-stack")
           has_provider=false
           for token in $INCLUDE_DOCS; do
             is_non_provider=false

airflow-core/docs/howto/static-type-checking.rst

@@ -20,56 +20,6 @@
 Static Type Checking for Dags
 =============================
 
-Airflow publishes a set of `mypy <https://mypy-lang.org/>`_ plugins as a standalone, independently
-versioned distribution: `apache-airflow-mypy <https://pypi.org/project/apache-airflow-mypy/>`_.
-
-When to use it
---------------
-
-If you run ``mypy`` over your Dags, custom operators, or hooks, install the plugins to get accurate
-results for Airflow-specific patterns that plain ``mypy`` cannot reason about and would otherwise report
-as false positives. The plugins teach ``mypy`` about:
-
-* **Typed decorators** -- decorators that inject keyword arguments at runtime (for example
-  ``GoogleBaseHook.fallback_to_default_project_id``), so ``mypy`` does not flag those arguments as missing.
-* **Operator outputs** -- the ``.output`` attribute of operators and the return value of ``@task``-decorated
-  functions (an ``XComArg``) are resolved to the underlying runtime type. This lets you wire a task's output
-  into a downstream task without spurious type errors:
-
-  .. code-block:: python
-
-     @task
-     def f(a: str) -> int:
-         return len(a)
-
-
-     @task
-     def g(b: int) -> None: ...
-
-
-     g(f("hello"))  # mypy understands the output of f() is an int
-
-The package is entirely optional -- Airflow does not require it at runtime; it only improves the accuracy
-of static type checking for Dag authors.
-
-Installation
-------------
-
-Install it alongside ``mypy``:
-
-.. code-block:: bash
-
-   pip install apache-airflow-mypy
-
-The package follows `SemVer <https://semver.org/>`_ and is released on its own cadence, so you can adopt it
-independently of your Airflow version.
-
-Configuration
--------------
-
-Enable the plugins in your ``mypy`` configuration (``mypy.ini``, ``setup.cfg`` or ``pyproject.toml``):
-
-.. code-block:: ini
-
-   [mypy]
-   plugins = airflow_mypy.plugins.decorators, airflow_mypy.plugins.outputs
+Airflow publishes optional `mypy <https://mypy-lang.org/>`_ plugins as the independently versioned
+`apache-airflow-mypy <https://airflow.apache.org/docs/apache-airflow-mypy/stable/index.html>`_ distribution.
+See the plugin documentation for installation, configuration, supported Airflow patterns, and release notes.

dev/README_RELEASE_MYPY.md

@@ -31,6 +31,7 @@
   - [Add tags in git](#add-tags-in-git)
   - [Commit the source packages to Apache SVN repo](#commit-the-source-packages-to-apache-svn-repo)
   - [Publish the distributions to PyPI (release candidates)](#publish-the-distributions-to-pypi-release-candidates)
+  - [Publish release candidate documentation](#publish-release-candidate-documentation)
   - [Prepare voting email](#prepare-voting-email)
   - [Verify the release candidate by PMC members](#verify-the-release-candidate-by-pmc-members)
   - [Verify the release candidate by Contributors](#verify-the-release-candidate-by-contributors)
@@ -39,6 +40,7 @@
   - [Publish release to SVN](#publish-release-to-svn)
   - [Publish the packages to PyPI](#publish-the-packages-to-pypi)
   - [Add tags in git](#add-tags-in-git-1)
+  - [Publish documentation](#publish-documentation)
   - [Notify developers of release](#notify-developers-of-release)
 
 <!-- END doctoc generated TOC please keep comment here to allow auto update -->
@@ -166,6 +168,18 @@ twine upload dist/apache_airflow_mypy-<VERSION>rc<RC>*
 
 Use a short-lived (throw-away) PyPI API token for the upload and delete it afterwards.
 
+## Publish release candidate documentation
+
+Run the `Publish Docs to S3` workflow from the `apache/airflow` repository with:
+
+- **ref**: `apache-airflow-mypy-<VERSION>rc<RC>`
+- **destination**: `staging`
+- **include-docs**: `apache-airflow-mypy`
+
+After it completes, run the `Build docs` workflow from the `staging` branch of the `apache/airflow-site`
+repository. Verify the release candidate documentation at
+`https://airflow.staged.apache.org/docs/apache-airflow-mypy/stable/index.html` before starting the vote.
+
 ## Prepare voting email
 
 Send a voting email to dev@airflow.apache.org with the following template:
@@ -408,6 +422,20 @@ git tag -s apache-airflow-mypy-<VERSION> -m "Apache Airflow Mypy <VERSION>"
 git push origin apache-airflow-mypy-<VERSION>
 ```
 
+## Publish documentation
+
+Run the `Publish Docs to S3` workflow from the `apache/airflow` repository with:
+
+- **ref**: `apache-airflow-mypy-<VERSION>`
+- **destination**: `live`
+- **include-docs**: `apache-airflow-mypy`
+
+After it completes, run the `Build docs` workflow from the `main` branch of the `apache/airflow-site`
+repository. Verify both the versioned and stable documentation URLs:
+
+- `https://airflow.apache.org/docs/apache-airflow-mypy/<VERSION>/index.html`
+- `https://airflow.apache.org/docs/apache-airflow-mypy/stable/index.html`
+
 ## Notify developers of release
 
 Send an announcement email to dev@airflow.apache.org and announce@apache.org:
@@ -422,8 +450,11 @@ Apache Airflow Mypy provides Mypy plugins for Apache Airflow to enhance type che
 The release is available at:
 https://pypi.org/project/apache-airflow-mypy/<VERSION>/
 
+Documentation:
+https://airflow.apache.org/docs/apache-airflow-mypy/stable/index.html
+
 Release notes:
-https://github.com/apache/airflow/blob/main/dev/mypy/RELEASE_NOTES.rst
+https://airflow.apache.org/docs/apache-airflow-mypy/stable/release_notes.html
 
 Installation:
 pip install apache-airflow-mypy

dev/breeze/doc/ci/04_selective_checks.md

@@ -467,8 +467,8 @@ together using `pytest-xdist` (pytest-xdist distributes the tests among parallel
 * If there are some build dependencies changed (`hatch_build.py` and updated system dependencies in
   the `pyproject.toml` - then `upgrade to newer dependencies` is enabled.
 * If docs are build, the `docs-list-as-string` will determine which docs packages to build. This is based on
-  several criteria: if any of the airflow core, charts, docker-stack, providers files or docs have changed,
-  then corresponding packages are build (including cross-dependent providers). If any of the core files
+  several criteria: if any of the airflow core, charts, docker-stack, Apache Airflow Mypy, providers files or
+  docs have changed, then corresponding packages are built (including cross-dependent providers). If any of the core files
   changed, also providers docs are built because all providers depend on airflow docs. If any of the docs
   build python files changed or when build is "canary" type in main - all docs packages are built.
 

dev/breeze/doc/images/output_build-docs.txt

@@ -1 +1 @@
-bd120b39a206e374a2867ca6020fba81
+6869c62bf6dd00ce2c54810da1e09c95

dev/breeze/doc/images/output_release-management_add-back-references.txt

@@ -1 +1 @@
-541760eea56b065a0bd0573fba31bdb0
+da17572185eeb81b9714aa9514be1970

dev/breeze/doc/images/output_release-management_publish-docs.txt

@@ -1 +1 @@
-314fc7a431405a85bfe701e396ea09fb
+ea0870d76dc087db72a64b430dc321fe

dev/breeze/doc/images/output_workflow-run_publish-docs.txt

@@ -1 +1 @@
-9845cbadb564c5fe56361e299dc1fb66
+f4e0b4ba219bf74c7f8935ef27799253

dev/breeze/src/airflow_breeze/commands/release_management_commands.py

@@ -102,6 +102,7 @@
     MULTI_PLATFORM,
     SCHEMA_DESTINATION_LOCATIONS,
     UV_VERSION,
+    get_airflow_mypy_version,
     get_airflow_version,
     get_airflowctl_version,
     get_task_sdk_version,
@@ -2044,6 +2045,9 @@ def get_package_version_possibly_from_stable_txt(package_name: str) -> str | Non
     if package_name == "apache-airflow-ctl":
         return get_airflowctl_version()
 
+    if package_name == "apache-airflow-mypy":
+        return get_airflow_mypy_version()
+
     if package_name == "task-sdk":
         return get_task_sdk_version()
 

dev/breeze/src/airflow_breeze/global_constants.py

@@ -33,6 +33,7 @@
     AIRFLOW_CTL_SOURCES_PATH,
     AIRFLOW_ROOT_PATH,
     AIRFLOW_TASK_SDK_SOURCES_PATH,
+    MYPY_SOURCES_PATH,
 )
 
 PUBLIC_AMD_RUNNERS = '["ubuntu-22.04"]'
@@ -316,6 +317,7 @@
     "task-sdk",
     "ts-sdk",
     "apache-airflow-ctl",
+    "apache-airflow-mypy",
 ]
 
 
@@ -711,6 +713,19 @@ def get_airflowctl_version():
     return airflowctl_version
 
 
+def get_airflow_mypy_version():
+    mypy_init_py_file = MYPY_SOURCES_PATH / "airflow_mypy" / "__init__.py"
+    mypy_version = "unknown"
+    with open(mypy_init_py_file) as init_file:
+        while line := init_file.readline():
+            if "__version__ = " in line:
+                mypy_version = line.split()[2][1:-1]
+                break
+    if mypy_version == "unknown":
+        raise RuntimeError("Unable to determine Apache Airflow Mypy version")
+    return mypy_version
+
+
 def get_airflow_version():
     airflow_init_py_file = AIRFLOW_CORE_SOURCES_PATH / "airflow" / "__init__.py"
     airflow_version = "unknown"

dev/breeze/src/airflow_breeze/utils/add_back_references.py

@@ -158,6 +158,9 @@ def start_generating_back_references(
     if "apache-airflow-ctl" in short_provider_ids:
         console_print("[info]Skipping airflowctl package. No back-reference needed.")
         short_provider_ids.remove("apache-airflow-ctl")
+    if "apache-airflow-mypy" in short_provider_ids:
+        console_print("[info]Skipping apache-airflow-mypy package. No back-reference needed.")
+        short_provider_ids.remove("apache-airflow-mypy")
 
     if short_provider_ids:
         for p in short_provider_ids:

dev/breeze/src/airflow_breeze/utils/docs_publisher.py

@@ -20,6 +20,7 @@
 import shutil
 
 from airflow_breeze.global_constants import (
+    get_airflow_mypy_version,
     get_airflow_version,
     get_airflowctl_version,
     get_java_sdk_version,
@@ -88,6 +89,8 @@ def _current_version(self):
             return chart_version()
         if self.package_name == "apache-airflow-ctl":
             return get_airflowctl_version()
+        if self.package_name == "apache-airflow-mypy":
+            return get_airflow_mypy_version()
         if self.package_name == "java-sdk":
             return get_java_sdk_version()
         raise SystemExit(f"Unsupported package: {self.package_name}")

dev/breeze/src/airflow_breeze/utils/publish_docs_to_s3.py

@@ -34,6 +34,7 @@
 NON_SHORT_NAME_PACKAGES = [
     "apache-airflow",
     "apache-airflow-ctl",
+    "apache-airflow-mypy",
     "docker-stack",
     "helm-chart",
     "java-sdk",

dev/breeze/src/airflow_breeze/utils/selective_checks.py

@@ -367,6 +367,10 @@ def __hash__(self):
             r"^airflow-ctl/docs",
             r"^airflow-ctl/src/.*\.py$",
             r"^airflow-ctl/tests/.*\.py$",
+            r"^dev/mypy/docs/",
+            r"^dev/mypy/src/.*\.py$",
+            r"^dev/mypy/RELEASE_NOTES\.rst$",
+            r"^dev/mypy/pyproject\.toml$",
             r"^CHANGELOG\.txt",
             r"^airflow-core/src/airflow/config_templates/config\.yml",
             r"^chart/RELEASE_NOTES\.rst",
@@ -1670,6 +1674,8 @@ def docs_list_as_string(self) -> str | None:
             packages.append("task-sdk")
         if any(file.startswith("airflow-ctl/") for file in self._files):
             packages.append("apache-airflow-ctl")
+        if any(file.startswith("dev/mypy/") for file in self._files):
+            packages.append("apache-airflow-mypy")
         if providers_affected:
             suspended = set(get_suspended_provider_ids())
             for provider in providers_affected:

dev/breeze/tests/test_add_back_references.py

@@ -0,0 +1,32 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+from pathlib import Path
+from unittest import mock
+
+from airflow_breeze.utils.add_back_references import start_generating_back_references
+
+
+@mock.patch("airflow_breeze.utils.add_back_references.generate_back_references", autospec=True)
+def test_mypy_docs_do_not_generate_provider_back_references(mock_generate_back_references):
+    package_ids = ["apache-airflow-mypy"]
+
+    start_generating_back_references(Path("airflow-site"), package_ids)
+
+    assert package_ids == []
+    mock_generate_back_references.assert_not_called()

dev/breeze/tests/test_docs_publisher.py

@@ -114,6 +114,13 @@ def test_publish_java_sdk_version_falls_back_to_gradle_properties(
     assert (java_sdk_archive / "stable.txt").read_text() == "9.9.9\n"
 
 
+def test_mypy_version_falls_back_to_package_source(monkeypatch):
+    monkeypatch.setattr(docs_publisher, "get_airflow_mypy_version", lambda: "0.2.0")
+    publisher = DocsPublisher(package_name="apache-airflow-mypy", output=None, verbose=False)
+
+    assert publisher._current_version == "0.2.0"
+
+
 def test_publish_ts_sdk_version_falls_back_to_package_json(generated_path, airflow_site_dir, monkeypatch):
     monkeypatch.setattr(docs_publisher, "get_ts_sdk_version", lambda: "0.1.0-alpha.0")
     _stage_ts_sdk_docs(generated_path, version=None)

dev/breeze/tests/test_publish_docs_to_s3.py

@@ -60,6 +60,7 @@ def test_get_all_eligible_docs(self, mock_listdir):
             "apache-airflow-providers-apache-cassandra",
             "helm-chart",
             "apache-airflow-ctl",
+            "apache-airflow-mypy",
         ]
 
         self.publish_docs_to_s3.exclude_docs = "amazon docker-stack apache.kafka"
@@ -71,6 +72,7 @@ def test_get_all_eligible_docs(self, mock_listdir):
                 "apache-airflow-providers-apache-cassandra",
                 "helm-chart",
                 "apache-airflow-ctl",
+                "apache-airflow-mypy",
             ]
         )
 

dev/breeze/tests/test_release_management_commands.py

@@ -296,6 +296,12 @@ def test_get_package_version_possibly_from_stable_txt_for_java_sdk(
     assert get_package_version_possibly_from_stable_txt("java-sdk") == expected_version
 
 
+def test_get_package_version_possibly_from_stable_txt_for_mypy(monkeypatch):
+    monkeypatch.setattr(release_management_commands, "get_airflow_mypy_version", lambda: "0.2.0")
+
+    assert get_package_version_possibly_from_stable_txt("apache-airflow-mypy") == "0.2.0"
+
+
 @pytest.mark.parametrize(
     ("stable_txt_content", "expected_version"),
     [

dev/breeze/tests/test_selective_checks.py

@@ -3104,6 +3104,13 @@ def test_upgrade_to_newer_dependencies(
             },
             id="Only Airflow docs changed",
         ),
+        pytest.param(
+            ("dev/mypy/docs/index.rst",),
+            {
+                "docs-list-as-string": "apache-airflow-mypy",
+            },
+            id="Only Apache Airflow Mypy docs changed",
+        ),
         pytest.param(
             ("providers/celery/src/airflow/providers/celery/file.py",),
             {"docs-list-as-string": "celery cncf.kubernetes common.compat"},