Rework pillar/renderers/encryption documentation

changelog/56239.fixed.md

@@ -0,0 +1 @@
+Rewrote the pillar / renderers / encryption documentation to address twelve outstanding doc issues: clarified the pillar data structure and merge rules (#56239, #61283), documented that custom modules used in pillar must be synced to the **master** with `salt-run saltutil.sync_all` (#62802), documented the `key:` subkey separator on pillar `include:` directives (#66622), explained how to target with `-I` against a specific pillarenv (#63110), tightened the pillar encryption page with a working shebang-renderer example (#62733), added the required `--homedir` flag to every documented `gpg` encrypt command (#65682), added a consistent GPG-terminology note (#59539), added a `Module.run`+pyobjects `**kwargs` workaround for dotted function names (#61828), expanded the py renderer guide with new examples and the highstate return-data shape (#63698), clarified the meaning of `default=<class 'KeyError'>` in `pillar.get` (#60378), and added a per-pillarenv `root` example to the `git_pillar` walkthrough (#58929). New regression tests verify that every documented example renders or round-trips end-to-end.

doc/topics/pillar/index.rst

@@ -337,6 +337,114 @@ Since both pillar SLS files contained a ``bind`` key which contained a nested
 dictionary, the pillar dictionary's ``bind`` key contains the combined contents
 of both SLS files' ``bind`` keys.
 
+.. _pillar-data-structure:
+
+The Pillar Data Structure
+=========================
+
+Pillar SLS files are parsed by their renderer into Python data types and
+then collapsed into a single, per-minion dictionary.  Knowing the rules
+the compiler uses makes pillar files much easier to read and write.
+
+* A pillar dictionary is **always a dict** at the top level.  Keys in
+  that dict come from every SLS file the minion was assigned in the top
+  file.
+
+* When two SLS files declare the **same top-level key** the values are
+  combined according to the **types** involved:
+
+  * Two ``dict`` values are recursively merged, key-by-key.
+  * Two ``list`` values are merged by appending; whether the merge
+    de-duplicates or appends raw depends on
+    :conf_minion:`pillar_merge_lists` (default ``False``).
+  * Any **scalar collision** (string, int, bool, etc.) uses the
+    *last-one-wins* rule: the value from the file processed last
+    overwrites the earlier one.  Files are processed in the order they
+    appear in the top file.
+
+* The colon-delimited subkey syntax used throughout pillar
+  (:py:func:`pillar.get('foo:bar:baz') <salt.modules.pillar.get>`,
+  :conf_master:`decrypt_pillar`, the include ``key`` option) refers to
+  Python dict nesting -- ``foo:bar:baz`` reads
+  ``pillar['foo']['bar']['baz']``.
+
+Worked example
+--------------
+
+Top file declares both pillars apply to every minion:
+
+``/srv/pillar/top.sls``:
+
+.. code-block:: yaml
+
+    base:
+      '*':
+        - users
+        - sudoers
+
+``/srv/pillar/users.sls``:
+
+.. code-block:: yaml
+
+    users:
+      alice:
+        uid: 1001
+      bob:
+        uid: 1002
+
+``/srv/pillar/sudoers.sls``:
+
+.. code-block:: yaml
+
+    users:
+      alice:
+        sudo: True
+      carol:
+        uid: 1003
+
+After compilation the pillar dictionary contains a single merged
+``users`` key:
+
+.. code-block:: python
+
+    {
+        "users": {
+            "alice": {"uid": 1001, "sudo": True},
+            "bob": {"uid": 1002},
+            "carol": {"uid": 1003},
+        }
+    }
+
+``alice`` is the recursive-merge case (both files contributed dict
+fragments under the same key path).  ``bob`` and ``carol`` are present
+because each file added a unique sub-key.  If, instead, ``sudoers.sls``
+had declared ``users: 'alice'`` (a scalar), it would overwrite the
+``users`` dictionary entirely under the "last one wins" rule.
+
+.. note::
+
+    The same compiled dictionary is exposed three different ways in
+    templates: ``pillar['users']['alice']['uid']`` (raw access),
+    ``salt['pillar.get']('users:alice:uid')`` (safe traversal with a
+    default value) and ``__pillar__['users']['alice']['uid']`` (inside
+    execution modules and the ``py`` renderer).
+
+Custom modules in pillar
+------------------------
+
+Execution modules that are referenced by ``ext_pillar`` or by pillar SLS
+files (via ``salt['custom_mod.foo']``) run on the **master**, not on
+the minion.  Custom modules placed in ``_modules/`` on the master will
+not be available in pillar until they have been synced to the master:
+
+.. code-block:: bash
+
+    salt-run saltutil.sync_all
+
+Running ``salt '*' saltutil.sync_all`` only syncs custom modules to the
+minions.  Pillar compilation will continue to fail to find the custom
+function until the master itself has the synced copy.
+
 .. _pillar-include:
 
 Including Other Pillars
@@ -369,6 +477,22 @@ With this form, the included file (users.sls) will be nested within the 'users'
 key of the compiled pillar. Additionally, the 'sudo' value will be available
 as a template variable to users.sls.
 
+.. note::
+
+    The ``key`` option uses the same colon-delimited subkey syntax as
+    :py:func:`pillar.get <salt.modules.pillar.get>`.  To nest the included
+    pillar deeper than one level, use ``parent:child:leaf``:
+
+    .. code-block:: yaml
+
+        include:
+          - users:
+              key: accounts:internal:users
+
+    With this form, the contents of ``users.sls`` will be made available
+    under ``pillar['accounts']['internal']['users']``.  The colon is the
+    only separator accepted by ``key``.
+
 .. _pillar-in-memory:
 
 In-Memory Pillar Data vs. On-Demand Pillar Data
@@ -586,6 +710,15 @@ Salt's renderer system can be used to decrypt pillar data. This allows for
 pillar items to be stored in an encrypted state, and decrypted during pillar
 compilation.
 
+.. note:: GPG terminology
+
+    Throughout this documentation the term "GPG" is used for all
+    OpenPGP/PGP/GPG-related material and implementations, matching the
+    convention used by the GNU Privacy Guard project itself.  The
+    armored block markers (``-----BEGIN PGP MESSAGE-----`` and
+    ``-----END PGP MESSAGE-----``) come from the underlying OpenPGP
+    standard and appear regardless of which tool produced them.
+
 Encrypted Pillar SLS
 --------------------
 
@@ -693,6 +826,33 @@ So, the first example above could be rewritten as:
     decrypt_pillar:
       - 'secrets:vault'
 
+Per-file Shebang Renderers
+**************************
+
+When :conf_master:`decrypt_pillar` is not configured, individual pillar
+SLS files can still be marked as encrypted by setting the rendering
+pipeline with a shebang on the first line of the file:
+
+.. code-block:: yaml
+
+    #!yaml|gpg
+
+    api_key: |
+      -----BEGIN PGP MESSAGE-----
+      ...
+      -----END PGP MESSAGE-----
+
+The ``#!yaml|gpg`` shebang tells the renderer system to parse the file
+as YAML and then pass every string value through the
+:mod:`gpg <salt.renderers.gpg>` renderer, which decrypts the PGP
+message blocks in place.  The same pattern works for any decryption
+renderer listed in :conf_master:`decrypt_pillar_renderers` (for example
+``#!yaml|nacl``).
+
+This is often simpler than :conf_master:`decrypt_pillar` when only a
+handful of pillar files contain secrets: the encryption is declared in
+the file itself rather than via a separate path-based configuration.
+
 Encrypted Pillar Data on the CLI
 --------------------------------
 
@@ -719,7 +879,7 @@ Triggering decryption of this CLI pillar data can be done in one of two ways:
 
    .. code-block:: bash
 
-       $ echo -n bar | gpg --armor --trust-model always --encrypt -r user@domain.tld | sed ':a;N;$!ba;s/\n/\\n/g'
+       $ echo -n bar | gpg --homedir /etc/salt/gpgkeys --armor --trust-model always --encrypt -r user@domain.tld | sed ':a;N;$!ba;s/\n/\\n/g'
 
    .. note::
        Using ``pillar_enc`` will perform the decryption minion-side, so for

doc/topics/targeting/pillar.rst

@@ -32,3 +32,50 @@ is being traversed. The below example would match minions with a pillar named
 .. code-block:: bash
 
     salt -I 'foo:bar:baz*' test.version
+
+Targeting against a specific pillarenv
+======================================
+
+Pillar match commands on the CLI evaluate against the **in-memory** pillar
+data that lives on each minion -- i.e. the data that was already compiled
+and shipped to the minion the last time it refreshed its pillar.  By
+default that in-memory pillar is the merge of every configured pillar
+environment.
+
+To target on data sourced exclusively from a single
+:conf_minion:`pillarenv`, the targeted minions must first be made to
+compile their in-memory pillar against that environment.  Two patterns
+are supported:
+
+1. Set :conf_minion:`pillarenv` in the minion configuration file (or
+   override it on the master with
+   :conf_master:`pillarenv`/``pillarenv_from_saltenv``).  The minion's
+   in-memory pillar will then only contain keys from that environment
+   and pillar matching with ``-I`` will only see those keys:
+
+   .. code-block:: bash
+
+       salt '*' saltutil.refresh_pillar
+       salt -I 'key_only_in_qa_env:value' test.version
+
+2. Run states with an explicit ``pillarenv`` keyword argument when you
+   need targeting and pillar selection in the same call:
+
+   .. code-block:: bash
+
+       salt -I 'key:value' state.apply mystates pillarenv=qa
+
+   In this form ``-I 'key:value'`` is still evaluated against the
+   minion's *in-memory* pillar; the ``pillarenv=qa`` kwarg only
+   controls the pillar made available during state rendering.  If
+   ``key`` is not present in the default in-memory pillar the minion
+   will not be matched.
+
+.. note::
+
+    Pillar targeting cannot pick which pillarenv to evaluate the match
+    against on the master side.  If a key lives only in the ``qa``
+    pillarenv, minions must already be running with ``pillarenv: qa``
+    (or equivalent) for ``-I 'key:value'`` to find them.  See
+    :ref:`How Pillar Environments Are Handled <pillar-environments>`
+    for the complete set of options.

salt/modules/pillar.py

@@ -65,6 +65,29 @@ def get(
         unless :conf_minion:`pillar_raise_on_missing` is set to ``True``, in
         which case an error will be raised.
 
+        .. note::
+
+            The function signature uses a private ``NOT_SET`` sentinel for
+            ``default`` and the auto-generated documentation may render
+            this as ``default=<class 'KeyError'>``.  Both are internal
+            implementation details and not intended to be passed by the
+            caller.  The supported behavior is:
+
+            * Omit ``default`` (the common case): missing keys return an
+              empty string, or raise :py:exc:`KeyError` when
+              :conf_minion:`pillar_raise_on_missing` is ``True``.
+            * Pass any value for ``default`` -- scalars, dictionaries
+              and lists are all accepted.  When ``merge`` is also
+              ``True`` a dictionary or list default is recursively
+              merged with the retrieved pillar value.
+            * Explicitly passing the :py:class:`KeyError` class is
+              equivalent to enabling
+              :conf_minion:`pillar_raise_on_missing`: a missing key
+              raises ``KeyError`` rather than returning a value.
+
+            See ``salt/modules/pillar.py`` ``get()`` for the canonical
+            behavior.
+
     merge : ``False``
         If ``True``, the retrieved values will be merged into the passed
         default. When the default and the retrieved value are both

salt/pillar/git_pillar.py

@@ -329,6 +329,45 @@
     - Salt versions prior to 2018.3.4 ignore the ``root`` parameter when
       ``mountpoint`` is set.
 
+.. _git-pillar-per-saltenv-root:
+
+Per-saltenv ``root``
+~~~~~~~~~~~~~~~~~~~~
+
+The ``root`` per-remote parameter sets where in the repository the
+pillar SLS files live for a given remote.  When the same remote serves
+more than one pillarenv, ``root`` can be set per pillarenv so that one
+branch can contain pillar data for several environments at different
+subdirectories.
+
+The per-pillarenv ``root`` is **not additive** to a global
+:conf_master:`git_pillar_root`: when set it replaces it completely for
+that pillarenv.
+
+.. code-block:: yaml
+
+    git_pillar_root: pillar
+
+    ext_pillar:
+      - git:
+        - __env__ https://mydomain.tld/secrets.git:
+          - saltenv:
+            - dev:
+              - root: pillar/dev
+            - prod:
+              - root: pillar/prod
+            - test:
+              - root: pillar/test/v2
+
+With the above:
+
+* In the ``dev`` pillarenv, pillar SLS files for this remote are read
+  from ``pillar/dev`` (NOT ``pillar/dev`` *plus* ``pillar``).
+* In the ``prod`` pillarenv they are read from ``pillar/prod``.
+* In the ``test`` pillarenv they are read from ``pillar/test/v2``.
+* In any other pillarenv the global ``git_pillar_root: pillar`` is
+  used.
+
 .. _git-pillar-all_saltenvs:
 
 all_saltenvs

salt/renderers/gpg.py

@@ -173,7 +173,15 @@
 
 .. code-block:: bash
 
-   $ echo -n 'supersecret' | gpg --trust-model always -ear <KEY-ID>
+   $ echo -n 'supersecret' | gpg --homedir /etc/salt/gpgkeys --trust-model always -ear <KEY-ID>
+
+.. note::
+
+    The ``--homedir`` option is required whenever the recipient's public
+    key lives in the salt-master's gpg keyring rather than the calling
+    user's ``~/.gnupg`` directory.  Omitting it produces
+    ``gpg: [stdin]: encryption failed: No public key`` because gpg
+    searches the wrong keyring.
 
 To apply the renderer on a file-by-file basis add the following line to the
 top of any pillar with gpg data in it:
@@ -244,9 +252,9 @@
 .. code-block:: bash
 
     # awk
-    ciphertext=`echo -n "supersecret" | gpg --armor --batch --trust-model always --encrypt -r user@domain.com | awk '{printf "%s\\n",$0} END {print ""}'`
+    ciphertext=`echo -n "supersecret" | gpg --homedir /etc/salt/gpgkeys --armor --batch --trust-model always --encrypt -r user@domain.com | awk '{printf "%s\\n",$0} END {print ""}'`
     # Perl
-    ciphertext=`echo -n "supersecret" | gpg --armor --batch --trust-model always --encrypt -r user@domain.com | perl -pe 's/\n/\\n/g'`
+    ciphertext=`echo -n "supersecret" | gpg --homedir /etc/salt/gpgkeys --armor --batch --trust-model always --encrypt -r user@domain.com | perl -pe 's/\n/\\n/g'`
 
 With Python:
 
@@ -255,7 +263,8 @@
     import subprocess
 
     secret, stderr = subprocess.Popen(
-        ['gpg', '--armor', '--batch', '--trust-model', 'always', '--encrypt',
+        ['gpg', '--homedir', '/etc/salt/gpgkeys',
+         '--armor', '--batch', '--trust-model', 'always', '--encrypt',
          '-r', 'user@domain.com'],
         stdin=subprocess.PIPE,
         stdout=subprocess.PIPE,
@@ -294,9 +303,9 @@
 .. code-block:: bash
 
     # awk
-    ciphertext=`echo -n "{'secret_a': 'CorrectHorseBatteryStaple', 'secret_b': 'GPG is fun!'}" | gpg --armor --batch --trust-model always --encrypt -r user@domain.com | awk '{printf "%s\\n",$0} END {print ""}'`
+    ciphertext=`echo -n "{'secret_a': 'CorrectHorseBatteryStaple', 'secret_b': 'GPG is fun!'}" | gpg --homedir /etc/salt/gpgkeys --armor --batch --trust-model always --encrypt -r user@domain.com | awk '{printf "%s\\n",$0} END {print ""}'`
     # Perl
-    ciphertext=`echo -n "{'secret_a': 'CorrectHorseBatteryStaple', 'secret_b': 'GPG is fun!'}" | gpg --armor --batch --trust-model always --encrypt -r user@domain.com | perl -pe 's/\n/\\n/g'`
+    ciphertext=`echo -n "{'secret_a': 'CorrectHorseBatteryStaple', 'secret_b': 'GPG is fun!'}" | gpg --homedir /etc/salt/gpgkeys --armor --batch --trust-model always --encrypt -r user@domain.com | perl -pe 's/\n/\\n/g'`
 
 With Python:
 
@@ -308,7 +317,8 @@
                    'secret_b': 'GPG is fun!'}
 
     secret, stderr = subprocess.Popen(
-        ['gpg', '--armor', '--batch', '--trust-model', 'always', '--encrypt',
+        ['gpg', '--homedir', '/etc/salt/gpgkeys',
+         '--armor', '--batch', '--trust-model', 'always', '--encrypt',
          '-r', 'user@domain.com'],
         stdin=subprocess.PIPE,
         stdout=subprocess.PIPE,