diff --git a/changelog/39959.fixed.md b/changelog/39959.fixed.md new file mode 100644 index 000000000000..8b90f782d373 --- /dev/null +++ b/changelog/39959.fixed.md @@ -0,0 +1 @@ +Added an explicit docs note to the :py:mod:`~salt.returners.rawfile_json` module clarifying that it only implements ``returner`` and ``event_return`` (so it cannot be wired in as a :conf_master:`master_job_cache` or :ref:`external-job-cache` backend), with a pointer to the upstream tracking bug. Closes the docs portion of :issue:`39959`. diff --git a/changelog/47675.fixed.md b/changelog/47675.fixed.md new file mode 100644 index 000000000000..ddd75257460d --- /dev/null +++ b/changelog/47675.fixed.md @@ -0,0 +1 @@ +Documented why calling ``mine.update`` inside :py:func:`module.run ` silently fails to refresh the mine on the master, and listed the correct alternatives (out-of-band ``mine.update``, ``mine.send`` for a single function, or a shorter :ref:`mine_interval`) in a new "Updating the Mine From a State" section of ``doc/topics/mine/index.rst``. Closes :issue:`47675`. diff --git a/changelog/58130.fixed.md b/changelog/58130.fixed.md new file mode 100644 index 000000000000..93d32c44df64 --- /dev/null +++ b/changelog/58130.fixed.md @@ -0,0 +1 @@ +Clarified the :py:func:`hashutil.github_signature ` docstring to spell out that ``challenge_hmac`` is a single positional ``=`` string (matching the ``X-Hub-Signature`` / ``X-Hub-Signature-256`` header GitHub sends) and that it must be quoted on the CLI to avoid Salt parsing ``sha1=...`` as a keyword argument. Closes the docs portion of :issue:`58130`. diff --git a/changelog/61332.fixed.md b/changelog/61332.fixed.md new file mode 100644 index 000000000000..fb1492e21b1a --- /dev/null +++ b/changelog/61332.fixed.md @@ -0,0 +1 @@ +Documented the ``pyroute2`` dependency on the :py:mod:`~salt.beacons.network_settings` beacon. The module docstring (and therefore the rendered ``salt.beacons.network_settings`` reference page) now lists the dependency, with onedir vs system-Python install guidance and an explicit note that the beacon refuses to load without it. Closes :issue:`61332`. diff --git a/changelog/61616.fixed.md b/changelog/61616.fixed.md new file mode 100644 index 000000000000..96c8607de17a --- /dev/null +++ b/changelog/61616.fixed.md @@ -0,0 +1 @@ +Fixed the reactor documentation in ``doc/topics/reactor/index.rst``, which incorrectly claimed that beacon-fired events arrive at the reactor wrapped under ``data['data']``. The master's ``_minion_event`` handler unwraps each beacon event before firing it on the bus, so reactors see the beacon payload directly at ``data['id']``. The docs now distinguish that path from events produced by ``event.send`` / ``event.fire_master`` (which do show up under ``data['data']``) and ship worked snippets for both. New tests in ``tests/pytests/unit/beacons/test_documented_configs.py`` pin both data dict shapes. Closes :issue:`61616`. diff --git a/changelog/61921.fixed.md b/changelog/61921.fixed.md new file mode 100644 index 000000000000..2d7ffee5b1a5 --- /dev/null +++ b/changelog/61921.fixed.md @@ -0,0 +1 @@ +Added three worked Thorium examples (load-average spike reaction, nginx-flap detection, and runner-triggering on a custom event tag) to ``doc/topics/thorium/index.rst``. Each example exercises only Thorium module functions that ship in ``salt/thorium/`` and is pinned by a new unit test in ``tests/pytests/unit/thorium/test_documented_examples.py`` that parses the SLS and verifies every referenced ``.`` exists. Closes :issue:`61921`. diff --git a/changelog/62017.fixed.md b/changelog/62017.fixed.md new file mode 100644 index 000000000000..a4397ba105f4 --- /dev/null +++ b/changelog/62017.fixed.md @@ -0,0 +1 @@ +Documented the four returner contexts (``--return``, :conf_master:`master_job_cache`, :conf_master:`ext_job_cache`, :conf_master:`event_return`) in ``doc/ref/returners/index.rst`` with a per-context table of which side runs the returner, which config file the keys live in, and which returner functions are required for each context. Closes :issue:`62017` and :issue:`63367`. diff --git a/changelog/63693.fixed.md b/changelog/63693.fixed.md new file mode 100644 index 000000000000..647fd9fbd2ee --- /dev/null +++ b/changelog/63693.fixed.md @@ -0,0 +1 @@ +Replaced the broken inotify install instructions in ``doc/topics/beacons/index.rst`` and the :py:mod:`~salt.beacons.inotify` module docstring. The legacy ``salt myminion pkg.install python-inotify`` command fails on every current platform because the PyPI distribution is named ``pyinotify`` (no ``python-`` prefix) and distro packages now ship ``python3-pyinotify``. The docs now show the correct ``pip.install pyinotify`` invocation for onedir packaged installs (default in 3006+) and the correct distro-package or pip path for system-Python installs. Closes :issue:`63693` and :issue:`65019`. diff --git a/doc/ref/returners/index.rst b/doc/ref/returners/index.rst index c388368482bb..e5e6cc955dd8 100644 --- a/doc/ref/returners/index.rst +++ b/doc/ref/returners/index.rst @@ -44,6 +44,119 @@ It is also possible to specify multiple returners: In this scenario all three returners will be called and the data from the test.version command will be sent out to the three named returners. +.. _returner-contexts: + +Returner Contexts and Configuration Keys +======================================== + +Salt invokes returners in four distinct contexts. Each context targets a +different function in the returner module and reads configuration from a +different place. Knowing which context you want is the difference between +returner data appearing where you expect it and silent drops. + +``--return`` / ``--return_config`` / ``--return_kwargs`` (per-job, minion-side) + Set on the ``salt`` CLI. The master tells each targeted minion to load + the named returner *on the minion* and call its ``returner(ret)`` + function with the job's return dict. + + * ``--return `` selects the returner. + * ``--return_config `` selects a configuration profile prefix. + A returner that supports profiles reads ``profile.foo`` keys + instead of plain ``foo`` keys when this is set (for example, + ``alternative.mysql.host`` vs ``mysql.host``). + * ``--return_kwargs '{"key": "value"}'`` passes per-call keyword + arguments that the returner's ``returner()`` function may consume + (look for ``ret.get("ret_kwargs")`` or similar in the returner + source). Not all returners honor it. + + Because the returner runs on the minion, its **configuration must be + on the minion**. + +:conf_master:`master_job_cache` (post-job, master-side) + A single returner the master invokes to record every completed job's + load and return. The master calls the returner's ``prep_jid``, + ``save_load``, ``get_load``, and ``returner`` functions. The returner + runs *on the master*, so its configuration must be in the master + config. Only returners that implement ``prep_jid`` / ``save_load`` / + ``get_load`` are eligible. See :ref:`external-job-cache` for the + eligibility list (the same set of functions is required). + +:conf_master:`ext_job_cache` (post-job, minion-side) + The same idea as ``master_job_cache``, but the returner is loaded and + run on the minion. Use this when minions can reach the storage + backend directly and you would rather not funnel everything through + the master. The returner must implement the same set of cache + functions, and its configuration must be on the **minion**. See + :ref:`external-job-cache`. + +:conf_master:`event_return` (event bus, master-side) + A list of returners the master invokes for every event it sees on + the event bus, calling each returner's ``event_return(events)`` + function. Returners without an ``event_return`` are skipped silently. + Configuration must be on the **master**. + +Where to put which key (quick reference): + ++----------------------------+----------+-----------------------+ +| Context | Side | Config file | ++============================+==========+=======================+ +| ``--return`` | minion | ``/etc/salt/minion`` | ++----------------------------+----------+-----------------------+ +| ``master_job_cache`` | master | ``/etc/salt/master`` | ++----------------------------+----------+-----------------------+ +| ``ext_job_cache`` | minion | ``/etc/salt/minion`` | ++----------------------------+----------+-----------------------+ +| ``event_return`` | master | ``/etc/salt/master`` | ++----------------------------+----------+-----------------------+ + +To check whether a given returner supports a given context, look for the +following function names in its source: + ++----------------------+--------------------------------------------------+ +| Function | Required for | ++======================+==================================================+ +| ``returner`` | ``--return`` and (implicitly) any cache context | ++----------------------+--------------------------------------------------+ +| ``prep_jid``, | ``master_job_cache`` and ``ext_job_cache`` | +| ``save_load``, | | +| ``get_load`` | | ++----------------------+--------------------------------------------------+ +| ``get_jid``, | ``ext_job_cache`` | +| ``get_fun``, | | +| ``get_jids``, | | +| ``get_minions`` | | ++----------------------+--------------------------------------------------+ +| ``event_return`` | ``event_return`` | ++----------------------+--------------------------------------------------+ + +See :issue:`62017` and :issue:`63367` for the discussion that motivated +this section. + +Returner Configuration Profiles +------------------------------- + +Many returners accept an "alternative" configuration profile so the same +minion can target two backends. The convention is: + +.. code-block:: yaml + + # default keys + mysql.host: db.example.com + + # profile named "audit" + audit.mysql.host: audit-db.example.com + +To use the ``audit`` profile from the CLI: + +.. code-block:: bash + + salt '*' test.version --return mysql --return_config audit + +Returners implement profile lookup via +:py:func:`salt.returners.get_returner_options`, which falls back to the +default key when a profile-prefixed key is absent. See the +:py:mod:`mysql ` returner for an example. + Writing a Returner ================== diff --git a/doc/topics/beacons/index.rst b/doc/topics/beacons/index.rst index 311c8549b46b..74e38f7746c3 100644 --- a/doc/topics/beacons/index.rst +++ b/doc/topics/beacons/index.rst @@ -152,8 +152,34 @@ beacon to monitor a file for changes, and then restores the file to its original contents if a change was made. .. note:: - The inotify beacon requires Pyinotify on the minion, install it using - ``salt myminion pkg.install python-inotify``. + The inotify beacon requires `pyinotify + `_ on the minion. The + correct install method depends on how Salt was installed: + + * **onedir packages** (``salt-pkg-install``, ``rpm``, ``deb`` from + the official 3006+ packages): pyinotify must live inside Salt's + bundled Python environment. Use :py:func:`pip.install + `, which targets the onedir's pip: + + .. code-block:: bash + + salt myminion pip.install pyinotify + + * **System python install** (``pip install salt`` in a venv, or + legacy system-python packages): install ``pyinotify`` into the + same interpreter the minion runs under. If your distribution + packages it (Debian/Ubuntu ship ``python3-pyinotify``), use: + + .. code-block:: bash + + salt myminion pkg.install python3-pyinotify + + Otherwise use ``pip.install pyinotify``. + + The legacy instruction ``pkg.install python-inotify`` is wrong on + every current platform; the PyPI distribution is named ``pyinotify`` + (no ``python-`` prefix) and distro packages now ship the python3 + variant. See :issue:`63693` and :issue:`65019`. Create Watched File ------------------- diff --git a/doc/topics/mine/index.rst b/doc/topics/mine/index.rst index 33a4a6d5c990..f2a39ba2e67b 100644 --- a/doc/topics/mine/index.rst +++ b/doc/topics/mine/index.rst @@ -148,6 +148,61 @@ config: mine_interval: 60 +.. _mine_updating_in_states: + +Updating the Mine From a State +============================== + +Two execution-module functions are available to refresh mine data on a +running minion: + +* :py:func:`mine.update ` re-runs every entry in + ``mine_functions`` and pushes the result up to the master. +* :py:func:`mine.send ` re-runs a single named + function and pushes only that one. + +Both work cleanly from the command line: + +.. code-block:: bash + + salt-call mine.update + salt-call mine.send network.ip_addrs eth0 + +.. warning:: + + Calling ``mine.update`` from inside a state via + :py:func:`module.run ` is a footgun. With the + classic ``module.run`` signature, ``mine.update`` will appear to + succeed in the state output but the mine on the master will not be + updated (see :issue:`47675`). The state runtime executes the function + in a context where the mine push is silently dropped. + + Use one of these instead: + + 1. **Run the mine refresh out of band**, with + :py:func:`mine.update ` from a scheduler + entry, a reactor, an orchestration runner, or a direct + ``salt '*' mine.update`` from the master. + + 2. **Use** :py:func:`mine.send ` for a single + function instead of refreshing every entry: + + .. code-block:: yaml + + refresh_ip_mine: + module.run: + - name: mine.send + - m_name: network.ip_addrs + - interface: eth0 + + 3. **Shorten the** :ref:`mine_interval` and let the scheduler refresh + the mine on its own. + + Salt's :ref:`mine_interval` always refreshes the mine correctly, so + when timing is not critical (most use cases), option 3 is the + simplest. For event-driven refreshes, prefer option 1 with a reactor + invoking ``salt '*' mine.update`` from the master. + Mine in Salt-SSH ================ diff --git a/doc/topics/reactor/index.rst b/doc/topics/reactor/index.rst index 7cdadff29cdc..84a8ac0322ab 100644 --- a/doc/topics/reactor/index.rst +++ b/doc/topics/reactor/index.rst @@ -422,18 +422,46 @@ For example: Beacons and Reactors ==================== -An event initiated by a beacon, when it arrives at the master will be wrapped -inside a second event, such that the data object containing the beacon -information will be ``data['data']``, rather than ``data``. +The shape of the ``data`` template variable inside a reactor SLS depends on +how the originating event was produced. Two paths exist, and they are not +the same: + +Beacon events + A beacon returns a list of dicts from its ``beacon()`` function. The + minion sends those dicts to the master with the key ``events``, and the + master unwraps each event and fires its inner ``data`` payload on the + event bus (see :py:meth:`~salt.daemons.masterapi.LocalFuncs._minion_event` + and :py:meth:`~salt.minion.MinionBase._fire_master`). The reactor then + receives that payload as ``data`` directly. To access fields, reference + them on ``data`` itself: + + .. code-block:: yaml + + revert-file: + local.state.apply: + - tgt: {{ data['id'] }} + - arg: + - maintain_important_file + + The minion id is at ``data['id']`` (the beacons subsystem injects it + automatically). Any extra fields the beacon set in its returned dict + are also at the top level of ``data``. + +Events from ``event.send`` / ``event.fire_master`` + These functions send a single load with ``tag``, ``data``, ``cmd``, + and ``id`` keys. The master fires that whole load on the event bus, so + the user payload ends up nested under ``data``: -For example, to access the ``id`` field of the beacon event in a reactor file, -you will need to reference ``{{ data['data']['id'] }}`` rather than ``{{ -data['id'] }}`` as for events initiated directly on the event bus. + .. code-block:: bash + + salt '*' event.send foo '{key: value}' -Similarly, the data dictionary attached to the event would be located in -``{{ data['data']['data'] }}`` instead of ``{{ data['data'] }}``. + In a reactor watching tag ``foo``, ``data['data']['key']`` equals + ``value`` and ``data['id']`` is the originating minion's id. -See the :ref:`beacon documentation ` for examples. +In short: for beacons, use ``{{ data['key'] }}``; for ``event.send``, use +``{{ data['data']['key'] }}``. The :ref:`beacon documentation +` includes a worked beacon + reactor + state example. Manually Firing an Event ======================== diff --git a/doc/topics/thorium/index.rst b/doc/topics/thorium/index.rst index 4a452b90f0bc..a058da1be3b3 100644 --- a/doc/topics/thorium/index.rst +++ b/doc/topics/thorium/index.rst @@ -348,3 +348,112 @@ It is possible to persist the register data to disk when a master is stopped gracefully, and reload it from disk when the master starts up again. This functionality is provided by the returner subsystem, and is enabled whenever any returner containing a ``load_reg`` and a ``save_reg`` function is used. + + +.. _thorium-worked-examples: + +Worked Examples +--------------- + +The three formulas below are all self-contained and exercise a single +Thorium module each. They are also covered by unit tests in +:blob:`tests/pytests/unit/thorium/test_documented_examples.py` so that +they are kept in sync with the underlying modules. + +.. _thorium-example-load-spike: + +Example 1: load-average spike reaction +`````````````````````````````````````` + +Configure a :py:mod:`load beacon ` on the minions you +want to monitor. They emit ``salt/beacon//load`` events whose +``data`` payload includes a ``1m`` key for the 1-minute load average. + +Place this Thorium SLS at ``/srv/thorium/load_spike.sls`` (with +``thorium_top.sls`` pointing at it): + +.. code-block:: yaml + + # Maintain a running mean of the 1-minute load average across every + # load-beacon event the master sees, then check whether it exceeds 4.0. + load_avg: + reg.mean: + - add: 1m + - match: salt/beacon/*/load + check.gt: + - value: 4.0 + + notify_high_load: + local.cmd: + - tgt: monitoring-host + - func: cmd.run + - arg: + - 'logger -t salt-thorium "load spike: $(date -Is)"' + - require: + - check: load_avg + +Because the Thorium high-state model lets multiple state types share an +SLS ID, ``reg.mean`` writes the running mean under the ``load_avg`` +register and ``check.gt`` reads it from the same slot. + +.. _thorium-example-flap-detection: + +Example 2: flapping service detection +````````````````````````````````````` + +Place this Thorium SLS at ``/srv/thorium/flap.sls``. It tracks the most +recent 20 ``salt/beacon/*/service`` events for nginx and acts when one +appears: + +.. code-block:: yaml + + # Watch the event bus for any salt/beacon/*/service/nginx event and + # capture the last 20 timestamps when it fires. + salt/beacon/*/service/nginx: + check.event: [] + + nginx_history: + reg.list: + - add: _stamp + - match: salt/beacon/*/service/nginx + - stamp: true + - prune: 20 + + # When the flap event happens, page the on-call host. + page_oncall: + local.cmd: + - tgt: pager.example.com + - func: cmd.run + - arg: + - 'oncall-page nginx-flap' + - require: + - check: salt/beacon/*/service/nginx + - reg: nginx_history + +.. _thorium-example-runner-trigger: + +Example 3: trigger a runner on a tag +```````````````````````````````````` + +Place this Thorium SLS at ``/srv/thorium/regen_cert.sls``. Whenever the +master sees a custom event tag ``cert/expiring``, it kicks off the +``state.orchestrate`` runner against an orchestration that regenerates +certificates: + +.. code-block:: yaml + + cert/expiring: + check.event: [] + + regen_cert: + runner.cmd: + - func: state.orchestrate + - arg: + - orch.regen_cert + - require: + - check: cert/expiring + +The ``runner.cmd`` Thorium module invokes a master runner. Any +:ref:`orchestration runner ` can be wired in this way. + +See :issue:`61921` for the request that motivated these examples. diff --git a/salt/beacons/inotify.py b/salt/beacons/inotify.py index 3b11322319b2..383cdfc0b72e 100644 --- a/salt/beacons/inotify.py +++ b/salt/beacons/inotify.py @@ -3,6 +3,15 @@ :depends: - pyinotify Python module >= 0.9.5 + Install the dependency into the Python environment the minion runs + under. For onedir-packaged installs (the default for Salt 3006+), + that means the bundled environment, which is reachable with + ``salt myminion pip.install pyinotify``. For minions running on a + system Python, install the distro package (Debian/Ubuntu: + ``python3-pyinotify``) or use ``pip.install pyinotify``. The legacy + ``pkg.install python-inotify`` instruction is wrong on every + current platform; the PyPI distribution is named ``pyinotify``. + :Caution: Using generic mask options like open, access, ignored, and closed_nowrite with reactors can easily cause the reactor to loop on itself. To mitigate this behavior, consider diff --git a/salt/beacons/network_settings.py b/salt/beacons/network_settings.py index 2badc314beab..2d0df65dba54 100644 --- a/salt/beacons/network_settings.py +++ b/salt/beacons/network_settings.py @@ -1,6 +1,19 @@ """ Beacon to monitor network adapter setting changes on Linux +:depends: - pyroute2 (preferred: ``>= 0.7.1`` for the NDB backend; the + legacy IPDB backend is also supported) + + Install the dependency into the Python environment the minion runs + under. For onedir-packaged installs (Salt 3006+), use + ``salt myminion pip.install pyroute2``. For minions running on a + system Python, install the distro package (Debian/Ubuntu: + ``python3-pyroute2``) or use ``pip.install pyroute2``. Without + ``pyroute2`` installed, the beacon will fail to load (see + :issue:`61332`). + +:platform: Linux + .. versionadded:: 2016.3.0 """ diff --git a/salt/modules/hashutil.py b/salt/modules/hashutil.py index 3517f2512ac0..9eb40ad4b427 100644 --- a/salt/modules/hashutil.py +++ b/salt/modules/hashutil.py @@ -277,11 +277,24 @@ def github_signature(string, shared_secret, challenge_hmac): Returns a boolean if the verification succeeded or failed. + The ``challenge_hmac`` argument is a single positional string of the form + ``=``, exactly as GitHub sends it in the + ``X-Hub-Signature`` (sha1) or ``X-Hub-Signature-256`` (sha256) header. + GitHub currently supports ``sha1`` and ``sha256`` for the hash type. + + .. note:: + Pass the signature as a quoted positional string. Salt's CLI parses + unquoted ``key=value`` tokens as keyword arguments, so writing + ``sha1=...`` without quotes yields + ``ERROR ... The following keyword arguments are not valid: sha1=...``. + See :issue:`58130`. + CLI Example: .. code-block:: bash salt '*' hashutil.github_signature '{"ref":....} ' 'shared secret' 'sha1=bc6550fc290acf5b42283fa8deaf55cea0f8c206' + salt '*' hashutil.github_signature '{"ref":....} ' 'shared secret' 'sha256=b3f8...' """ msg = string key = shared_secret diff --git a/salt/returners/rawfile_json.py b/salt/returners/rawfile_json.py index a46876118132..209e0299b84e 100644 --- a/salt/returners/rawfile_json.py +++ b/salt/returners/rawfile_json.py @@ -14,6 +14,16 @@ noise, so you may wish to configure batch processing and/or configure the :conf_master:`event_return_whitelist` or :conf_master:`event_return_blacklist` to restrict the events that are written. + +.. note:: + This returner exposes ``returner`` and ``event_return`` only. It does + not implement ``prep_jid`` / ``save_load`` / ``get_load`` / + ``get_jid``, so it cannot be used as a + :conf_master:`master_job_cache` or :ref:`external-job-cache` + backend. Configuring it in those slots silently drops jobs. See + :issue:`39959` for the open tracking bug; until it is fixed in + code, restrict this returner to ``--return`` and + :conf_master:`event_return` use. """ import logging diff --git a/tests/pytests/unit/beacons/test_documented_configs.py b/tests/pytests/unit/beacons/test_documented_configs.py new file mode 100644 index 000000000000..5c00b37404c8 --- /dev/null +++ b/tests/pytests/unit/beacons/test_documented_configs.py @@ -0,0 +1,126 @@ +""" +Tests that the beacon configuration examples shipped in the documentation +remain valid against the beacons' ``validate()`` functions. + +These tests pin the docs against the code, so a future drift causes a +loud failure rather than silent stale documentation. Issues addressed: + +* :issue:`63693` and :issue:`65019` for the inotify beacon +* :issue:`61332` for the network_settings beacon +* :issue:`61616` for the beacon -> reactor data dict shape +""" + +import pytest + +from salt.beacons import inotify + +try: + import pyinotify # pylint: disable=unused-import + + HAS_PYINOTIFY = True +except ImportError: + HAS_PYINOTIFY = False + + +@pytest.fixture +def configure_loader_modules(): + return {inotify: {}} + + +@pytest.mark.skipif( + not HAS_PYINOTIFY, reason="pyinotify is not available on this platform" +) +def test_inotify_documented_example_validates(tmp_path): + """ + The minimal inotify configuration shown in + ``doc/topics/beacons/index.rst`` (and copied below) must pass + ``inotify.validate()``. + """ + watched = tmp_path / "important_file" + watched.write_text("important_config: True\n") + config = [ + {"files": {str(watched): {"mask": ["modify"]}}}, + {"disable_during_state_run": True}, + ] + valid, _ = inotify.validate(config) + assert valid is True + + +def test_beacon_to_reactor_data_dict_shape(): + """ + The reactor documentation in ``doc/topics/reactor/index.rst`` says + that for events produced by a beacon, the reactor template variable + ``data`` is the beacon's payload directly (with ``id`` at the top + level). This test pins the contract by exercising the same unwrap + path the master daemon uses: + + 1. The beacon process returns ``[{"tag": ..., "data": {...}, + "beacon_name": ...}, ...]``. + 2. ``MinionBase._fire_master`` ships those as ``load["events"]``. + 3. ``LocalFuncs._minion_event`` on the master unwraps each event + and fires ``event["data"]`` on the bus with ``event["tag"]``. + + The reactor's template renderer then receives the unwrapped payload + as ``data``, so ``data["id"]`` -- not ``data["data"]["id"]`` -- + must be the minion id. + """ + # Simulate a beacon process output. The beacons subsystem always + # injects ``id`` if the beacon function did not. + minion_id = "minion-1" + beacon_raw = [ + { + "tag": f"salt/beacon/{minion_id}/inotify//etc/important_file", + "data": { + "id": minion_id, + "change": "IN_MODIFY", + "path": "/etc/important_file", + }, + "beacon_name": "inotify", + } + ] + + # Simulate the master's _minion_event unwrap step. + unwrapped = [] + for event in beacon_raw: + if "data" in event: + event_data = event["data"] + else: + event_data = event + unwrapped.append((event["tag"], event_data)) + + # The reactor's render_reaction(data=event_data) sees this: + fired_tag, reactor_data = unwrapped[0] + + assert fired_tag == f"salt/beacon/{minion_id}/inotify//etc/important_file" + # Critical: ``data["id"]`` is the minion id, NOT ``data["data"]["id"]``. + assert reactor_data["id"] == minion_id + assert "data" not in reactor_data or reactor_data.get("data") != {"id": minion_id} + + +def test_event_send_data_dict_shape(): + """ + Counterpart to ``test_beacon_to_reactor_data_dict_shape``: for an + event produced by ``event.send`` from a minion, the master fires + the whole load on the bus, so the reactor's ``data`` template + variable IS the load -- the user payload ends up at + ``data["data"]``. + """ + minion_id = "minion-1" + user_payload = {"orchestrate": "refresh"} + load = { + "id": minion_id, + "cmd": "_minion_event", + "tag": "foo", + "data": user_payload, + } + + # Simulate the master's single-event path in _minion_event. + fired_tag = load["tag"] + reactor_data = load + + assert fired_tag == "foo" + # ``data["data"]["orchestrate"]`` is "refresh" -- the documented + # pattern in ``doc/topics/reactor/index.rst`` Referencing Data + # Passed in Events. + assert reactor_data["data"]["orchestrate"] == "refresh" + assert reactor_data["id"] == minion_id diff --git a/tests/pytests/unit/thorium/__init__.py b/tests/pytests/unit/thorium/__init__.py new file mode 100644 index 000000000000..e69de29bb2d1 diff --git a/tests/pytests/unit/thorium/test_documented_examples.py b/tests/pytests/unit/thorium/test_documented_examples.py new file mode 100644 index 000000000000..ae9b778e10db --- /dev/null +++ b/tests/pytests/unit/thorium/test_documented_examples.py @@ -0,0 +1,166 @@ +""" +Tests that the Thorium SLS examples shipped in +``doc/topics/thorium/index.rst`` parse as YAML and reference Thorium +module functions that actually exist. + +These tests pin the docs against the code so a future drift causes a +loud failure rather than silent stale documentation. Issue addressed: +:issue:`61921`. +""" + +import inspect + +import pytest + +import salt.utils.yaml +from salt.thorium import calc, check, file, reg, runner + +# The exact SLS payloads documented in +# doc/topics/thorium/index.rst under "Worked Examples". +EXAMPLE_LOAD_SPIKE = """ +load_avg: + reg.mean: + - add: 1m + - match: salt/beacon/*/load + check.gt: + - value: 4.0 + +notify_high_load: + local.cmd: + - tgt: monitoring-host + - func: cmd.run + - arg: + - 'logger -t salt-thorium "load spike: $(date -Is)"' + - require: + - check: load_avg +""" + +EXAMPLE_FLAP_DETECTION = """ +salt/beacon/*/service/nginx: + check.event: [] + +nginx_history: + reg.list: + - add: _stamp + - match: salt/beacon/*/service/nginx + - stamp: true + - prune: 20 + +page_oncall: + local.cmd: + - tgt: pager.example.com + - func: cmd.run + - arg: + - 'oncall-page nginx-flap' + - require: + - check: salt/beacon/*/service/nginx + - reg: nginx_history +""" + +EXAMPLE_RUNNER_TRIGGER = """ +cert/expiring: + check.event: [] + +regen_cert: + runner.cmd: + - func: state.orchestrate + - arg: + - orch.regen_cert + - require: + - check: cert/expiring +""" + +# Modules whose names appear as the top-level state types in Thorium +# SLS. ``local`` is the wrapper module that publishes to minions; it is +# loaded by Salt's master runtime, so we don't import it here -- a +# string-name lookup is enough to confirm the docs reference it as a +# module name the Thorium loader recognises. +THORIUM_MODULES = { + "reg": reg, + "check": check, + "calc": calc, + "file": file, + "runner": runner, +} + +# ``local.cmd`` and ``local.state.apply`` are master-side wrappers. The +# docs reference them; they live in ``salt/thorium/local.py``. +THORIUM_WRAPPER_MODULES = {"local"} + + +def _module_has_function(mod, func_name): + """ + Thorium state modules sometimes alias function names via + ``__func_alias__`` (e.g. ``set_`` -> ``set``, ``list_`` -> ``list``). + Honour those aliases the same way the loader does. + """ + aliases = getattr(mod, "__func_alias__", {}) + # Reverse: documented name -> actual python name. + reverse_aliases = {v: k for k, v in aliases.items()} + candidate = reverse_aliases.get(func_name, func_name) + return inspect.isfunction(getattr(mod, candidate, None)) + + +@pytest.mark.parametrize( + "sls_text,name", + [ + (EXAMPLE_LOAD_SPIKE, "load-spike"), + (EXAMPLE_FLAP_DETECTION, "flap-detection"), + (EXAMPLE_RUNNER_TRIGGER, "runner-trigger"), + ], +) +def test_documented_thorium_example_parses(sls_text, name): + """ + Each documented Thorium example must be parseable YAML. + """ + parsed = salt.utils.yaml.safe_load(sls_text) + assert isinstance(parsed, dict), f"{name}: parsed SLS is not a dict" + assert parsed, f"{name}: parsed SLS is empty" + + +@pytest.mark.parametrize( + "sls_text,name", + [ + (EXAMPLE_LOAD_SPIKE, "load-spike"), + (EXAMPLE_FLAP_DETECTION, "flap-detection"), + (EXAMPLE_RUNNER_TRIGGER, "runner-trigger"), + ], +) +def test_documented_thorium_example_module_functions_exist(sls_text, name): + """ + For every ``.`` key under each SLS ID, the + referenced Thorium module must be importable and the function + callable. + """ + parsed = salt.utils.yaml.safe_load(sls_text) + seen = [] + + for sls_id, body in parsed.items(): + assert isinstance( + body, dict + ), f"{name}/{sls_id}: state body must be a dict, got {type(body).__name__}" + for state_key in body: + # state_key is e.g. "reg.mean", "check.gt", "local.cmd". + assert ( + "." in state_key + ), f"{name}/{sls_id}: state key {state_key!r} missing dotted form" + module_name, _, func_name = state_key.partition(".") + seen.append((sls_id, module_name, func_name)) + + if module_name in THORIUM_WRAPPER_MODULES: + # Wrapper modules (``local``, etc.) are loaded by the + # Thorium runtime rather than directly importable here. + # Their string presence is enough. + continue + + assert module_name in THORIUM_MODULES, ( + f"{name}/{sls_id}: thorium module {module_name!r} not " + f"recognised (known: {sorted(THORIUM_MODULES)})" + ) + mod = THORIUM_MODULES[module_name] + assert _module_has_function(mod, func_name), ( + f"{name}/{sls_id}: {module_name}.{func_name} not found " + f"in salt.thorium.{module_name}" + ) + + assert seen, f"{name}: no module.function references found"