Caching functionality does not work

[RobinR1]

I have a job running each morning to check/(re-)register all managed hosts into Zabbix.
For this I set variables ansible_user and ansible_httpapi_pass using anatomicjc.passbolt.cached lookups before starting the role on all these hosts:

ansible_httpapi_pass: "{{ lookup('anatomicjc.passbolt.cached', 'anatomicjc.passbolt.passbolt', '<uuid>', per_uuid='true').password }}"

It seems that ansible, for each and every task it starts, tries to resolve these ansible_ variables by querying passbolt, even when they are not used in the task.
I already had my suspicion but this case made it very clear: the passwords are not cached.

Currently this play runs over 1100+ hosts, bombarding passbolt with 2000+ queries for the same user/password for each and every task that needs to be performed in this play.
Passbolt and it's database can hardly follow that much requests, memory usage peaks to 12G and CPU usage of 4 CPU's is 100% during the play, currently with a runtime of about 2 hours. And several queries give a timeout.

End result is that out of the 1100+ hosts about 700 hosts fail due to a passbolt lookup timeout sooner or later in the play.

[Bigouden]

I checked by displaying the content of this variable: https://github.com/passbolt/lab-passbolt-ansible-collection/blob/main/plugins/lookup/cached.py#L49 and by running the playbook in verbose mode.

With each task change, the dictionary seems to be reset.

I asked GPT to generate a similar behavior but with file storage, and the difference is astonishing:

cached.py

"""
Source: https://github.com/GoodRx/ansible-cached-lookup/pull/5

An Ansible lookup plugin that caches the results of any other lookup, most
useful in group/host vars.
By default, Ansible evaluates any lookups in a group/host var whenever the var
is accessed. For example, given a group/host var:
.. code-block:: yaml
    content: "{{ lookup('pipe', 'a-very-slow-command') }}"
any tasks that access ``content`` (e.g. in a template) will re-evaluate
the lookup, which adds up very quickly.
.. seealso:: :attr:`.DOCUMENTATION`, :attr:`.EXAMPLES`, `ansible/ansible#9623
    <https://github.com/ansible/ansible/issues/9623>`_
"""
from __future__ import absolute_import
from __future__ import division
from __future__ import print_function

import os.path
import json

from ansible import constants as C
from ansible.errors import AnsibleError
from ansible.plugins.loader import lookup_loader
from ansible.plugins.lookup import LookupBase

__version__ = "1.0.1"


DOCUMENTATION = """
lookup: cached
short_description: cache the result of a lookup in memory
description:
  - Run a lookup and cache the result for the duration of the play. This is
    most useful for lookups in group/host vars, which are typically
    re-evaluated every time they are used
options:
  _terms:
    description: the lookup and any arguments
    required: True
notes:
  - This code has been updated to do in-memory cache
"""

EXAMPLES = """
group_var1: "{{ lookup('cached', 'pipe', 'a-very-slow-command') }}"
"""

try:
    from __main__ import display
except ImportError:
    from ansible.utils.display import Display

    display = Display()


cache_file_path = '/tmp/passbolt_cache.json'

try:
    if os.path.exists(cache_file_path):
        with open(cache_file_path, 'r') as cache_file:
            passbolt_cache = json.load(cache_file)
    else:
        passbolt_cache = {}
except (IOError, ValueError):
    passbolt_cache = {}

class LookupModule(LookupBase):

    def run(self, terms, variables=None, **kwargs):
        lookup_name, terms = terms[0], terms[1:]
        key = terms[0]

        try:
            result = passbolt_cache[key]
            display.verbose("'cached' lookup cache hit for %r" % (key,))
        except KeyError:
            lookup = lookup_loader.get(
                lookup_name, loader=self._loader, templar=self._templar
            )
            if lookup is None:
                raise AnsibleError("lookup plugin (%s) not found" % lookup_name)

            result = lookup.run(terms, variables=variables, **kwargs)
            passbolt_cache[key] = result

            with open(cache_file_path, 'w') as cache_file:
                json.dump(passbolt_cache, cache_file)

            display.verbose("'cached' lookup cache miss for %r" % (key,))

        return result

Try it !

[Bigouden]

I release a collection who fix cache. You can try it : https://galaxy.ansible.com/ui/repo/published/bigouden/utils/

[AnatomicJC]

Thanks @Bigouden for your update and fix ! If @RobinR1 can check and confirm it is ok, I can merge it in the collection :)

[Bigouden]

The corrected source code is here: https://github.com/Bigouden/utils/blob/main/plugins/lookup/cache.py.

I just renamed "cached" to "cache" and store the cache file in the path associated with "DEFAULT_LOCAL_TMP". (https://docs.ansible.com/ansible/latest/reference_appendices/config.html#default-local-tmp)

This allows you to automatically delete the cache file at the end of each playbook execution.

If you take it, don't forget to rename "cache" to "cached" :)

[RobinR1]

Thanks @Bigouden . I have tested the new code and this seems to work much better in terms of caching. However I encountered a new problem with the caching, now that that functionality actually works:
The caching seems to only take the passbolt search string as key and leaves out the other variables. This works for calls like:

lookup('anatomicjc.passbolt.cached', 'anatomicjc.passbolt.passbolt', '<uuid>', per_uuid='true').password

where a unique string is given (the uuid).
However when called like this:

lookup('anatomicjc.passbolt.cached', 'anatomicjc.passbolt.passbolt', '<password name>', username=<username>).password

<password name> on its own is not unique, only the combination of <password name> and its extra parameters, like in this case username=<username> should be unique.
But the cache plugin seems to only use <password name> as key. In a play on multiple hosts each with an identical <password name> but unique username=<username>, this causes only the first host to get the correct password, but any subsequent hosts get the password of the first host which is of course the wrong password.

[Bigouden]

Ok cool, so it would just be a matter of adding the additional parameters passed to the Passbolt plugin in the dictionary key that stores and retrieves the result from the cache. I'm trying to see when I can change that.

[Bigouden]

Could you try this piece of code.

cached.py

#!/usr/bin/env python3
# coding: utf-8
# pyright: reportMissingImports=false
# pyright: reportArgumentType=false
# pylint: disable=import-error
# pylint: disable=no-member
# pylint: disable=ungrouped-imports
# pylint: disable=too-few-public-methods

"""
Source: https://github.com/GoodRx/ansible-cached-lookup/pull/5

An Ansible lookup plugin that caches the results of any other lookup.

By default, Ansible evaluates any lookups in a group/host var whenever the var
is accessed. For example, given a group/host var:

.. code-block:: yaml

    content: "{{ lookup('pipe', 'command') }}"

Any tasks that access ``content`` (e.g. in a template) will re-evaluate
the lookup, which adds up very quickly.

.. seealso:: :attr:`.DOCUMENTATION`, :attr:`.EXAMPLES`, `ansible/ansible#9623
    <https://github.com/ansible/ansible/issues/9623>`_
"""

from __future__ import absolute_import, division, print_function

import json
import os.path

from ansible import constants as C
from ansible.errors import AnsibleError
from ansible.plugins.loader import lookup_loader
from ansible.plugins.lookup import LookupBase

__version__ = "1.0.0"


DOCUMENTATION = """
lookup: cached
short_description: Cache the result of a lookup
description:
  - Run a lookup and cache the result for the duration of the play. This is
    most useful for lookups, which are typically re-evaluated every time they are used
options:
  Terms:
    description: The lookup and any arguments to perform.
    required: True
notes:
  - Results are cached in C(DEFAULT_LOCAL_TMP) and will be deleted at the end of
    the play.
"""

EXAMPLES = """
var: "{{ lookup('anatomicjc.passbolt.cache', 'pipe', 'command') }}"
"""

try:
    from __main__ import display
except ImportError:
    from ansible.utils.display import Display

    display = Display()

CACHE_FILE_PATH = os.path.join(C.DEFAULT_LOCAL_TMP, "cache_lookup")

try:
    if os.path.exists(CACHE_FILE_PATH):
        with open(CACHE_FILE_PATH, "r", encoding="utf-8") as CACHE_FILE:
            CACHE = json.load(CACHE_FILE)
    else:
        CACHE = {}
except (IOError, ValueError):
    CACHE = {}


class LookupModule(LookupBase):
    """Lookup Module Class"""

    def run(self, terms, variables=None, **kwargs):
        """Run"""

        lookup_name, terms = terms[0], terms[1:]
        key = f"{terms[0]}{kwargs}"

        try:
            result = CACHE[key]
            display.verbose(f"'cache' lookup cache hit for {key}")
        except KeyError as exception:
            lookup = lookup_loader.get(
                lookup_name, loader=self._loader, templar=self._templar
            )
            if lookup is None:
                raise AnsibleError(
                    f"lookup plugin ({lookup_name}) not found"
                ) from exception

            result = lookup.run(terms, variables=variables, **kwargs)
            CACHE[key] = result

            with open(CACHE_FILE_PATH, "w", encoding="utf-8") as cache_file:
                json.dump(CACHE, cache_file)

            display.verbose(f"'cache' lookup cache miss for {key}")

        return result

[RobinR1]

I have not yet been able to test the new code, but as I reviewed it, I understand you are now concatenating all parameters as they are, as key to the cache dict?
I'm not very educated in Python but two possible problems came to mind:

I'm not sure about allowed characters in dict keys in Python, but can this not possibly cause problems with certain characters when those are used in the key ?
In that case, it should be easily solved by base64 encoding that string first and use the base64 string as key? That feels a bit safer to me anyway.

Second possible 'problem' that came to mind, if the lookup plugin is called with multiple arguments like in:

lookup('anatomicjc.passbolt.cached', 'anatomicjc.passbolt.passbolt', 'Passbolt Cloud', username='admin@passbolt.dev', uri='https://cloud.passbolt.com')

it would be cached with these arguments in exactly this order, I assume ? So calling a subsequent call to the same password using

lookup('anatomicjc.passbolt.cached', 'anatomicjc.passbolt.passbolt', 'Passbolt Cloud', uri='https://cloud.passbolt.com', username='admin@passbolt.dev')

would not be found in the cache?
So, if possible, I think this can be avoided by first sorting the parameters alphabetically before concatenating.

[Bigouden]

Not a bad idea to sort the kwargs dictionary and encode the key ! 👍

The kwargs dictionary is now sorted, and a SHA-256 hash of the key is generated.

Can you test the following code ?

cached.py

#!/usr/bin/env python3
# coding: utf-8
# pyright: reportMissingImports=false
# pyright: reportArgumentType=false
# pylint: disable=import-error
# pylint: disable=no-member
# pylint: disable=ungrouped-imports
# pylint: disable=too-few-public-methods

"""
Source: https://github.com/GoodRx/ansible-cached-lookup/pull/5

An Ansible lookup plugin that caches the results of any other lookup.

By default, Ansible evaluates any lookups in a group/host var whenever the var
is accessed. For example, given a group/host var:

.. code-block:: yaml

    content: "{{ lookup('pipe', 'command') }}"

Any tasks that access ``content`` (e.g. in a template) will re-evaluate
the lookup, which adds up very quickly.

.. seealso:: :attr:`.DOCUMENTATION`, :attr:`.EXAMPLES`, `ansible/ansible#9623
    <https://github.com/ansible/ansible/issues/9623>`_
"""

from __future__ import absolute_import, division, print_function

import hashlib
import json
import os.path

from ansible import constants as C
from ansible.errors import AnsibleError
from ansible.plugins.loader import lookup_loader
from ansible.plugins.lookup import LookupBase

__version__ = "1.0.0"


DOCUMENTATION = """
lookup: cached
short_description: Cache the result of a lookup
description:
  - Run a lookup and cache the result for the duration of the play. This is
    most useful for lookups, which are typically re-evaluated every time they are used
options:
  Terms:
    description: The lookup and any arguments to perform.
    required: True
notes:
  - Results are cached in C(DEFAULT_LOCAL_TMP) and will be deleted at the end of
    the play.
"""

EXAMPLES = """
var: "{{ lookup('anatomicjc.passbolt.cached', 'pipe', 'command') }}"
"""

try:
    from __main__ import display
except ImportError:
    from ansible.utils.display import Display

    display = Display()

CACHE_FILE_PATH = os.path.join(C.DEFAULT_LOCAL_TMP, "cache_lookup")

try:
    if os.path.exists(CACHE_FILE_PATH):
        with open(CACHE_FILE_PATH, "r", encoding="utf-8") as CACHE_FILE:
            CACHE = json.load(CACHE_FILE)
    else:
        CACHE = {}
except (IOError, ValueError):
    CACHE = {}

def gen_hash(key):
    """Generate Hash"""
    sha256_hash = hashlib.sha256()
    string_key = str(key).encode("utf-8")
    sha256_hash.update(string_key)
    return sha256_hash.hexdigest()


class LookupModule(LookupBase):
    """Lookup Module Class"""

    def run(self, terms, variables=None, **kwargs):
        """Run"""

        lookup_name, terms = terms[0], terms[1:]
        sorted_kwargs = {key: kwargs[key] for key in sorted(kwargs)}
        key = gen_hash(f"{terms[0]}{sorted_kwargs}")

        try:
            result = CACHE[key]
            display.verbose(f"'cache' lookup cache hit for {key}")
        except KeyError as exception:
            lookup = lookup_loader.get(
                lookup_name, loader=self._loader, templar=self._templar
            )
            if lookup is None:
                raise AnsibleError(
                    f"lookup plugin ({lookup_name}) not found"
                ) from exception

            result = lookup.run(terms, variables=variables, **kwargs)
            CACHE[key] = result

            with open(CACHE_FILE_PATH, "w", encoding="utf-8") as cache_file:
                json.dump(CACHE, cache_file)

            display.verbose(f"'cache' lookup cache miss for {key}")

        return result

[RobinR1]

I tried the new code and it seems to works a lot better now. Many thanks!