python-jose not installed during install

[etieng]

Describe the bug

When installing Kresus for the first time, Woob cannot fetch data due to missing python library python-jose

Context

• Hardware: VPS @ barebone @ home
• YunoHost version: 11.2.10.3
• I have access to my server: Through SSH | through the webadmin
• Are you in a special context or did you perform some particular tweaking on your YunoHost instance?: yes
  • If yes, please explain:
    • ZFS raid for yunohost.app directory
    • Wireguard tunnel for an IP facing the world wide web
• Using, or trying to install package version/branch: stable 0.20.1~ynh3

Steps to reproduce

• Install Kresus from webmin on maindomain.tld/kresus > OK
• setup new bank account on Kresus webpage > cannot fetch data
• logs from Kresus said missing python-jose
• Do : #: sudo -u kresus pip install python-jose
• setup new bank on Kresus on webpage > OK

Expected behavior

python-jose should be installed during install of Kresus_ynh

Logs

[2024-04-25T11:31:34.162] [INFO] HTTP - POST /***sus/api/settings - 200 (17 ms)
[2024-04-25T11:31:34.459] [INFO] accounts-manager - Retrieve all accounts from access creditcooperatif with login **********
[2024-04-25T11:31:34.467] [INFO] providers/woob - Calling woob: command accounts...
[2024-04-25T11:31:40.586] [INFO] providers/woob - exited with code 1.
[2024-04-25T11:31:40.586] [INFO] providers/woob - Command returned an error code.
[2024-04-25T11:31:40.587] [INFO] providers/woob - Checking that woob is installed and can actually be called…
[2024-04-25T11:31:40.587] [INFO] providers/woob - Calling woob: command test...
[2024-04-25T11:31:41.029] [INFO] providers/woob - exited with code 0.
[2024-04-25T11:31:41.030] [INFO] providers/woob - OK: woob exited normally with non-empty JSON content.
[2024-04-25T11:31:41.030] [ERROR] providers/woob - Got error while running command "3": Unable to load module creditcooperatif.
Traceback (most recent call last):
  File "/var/www/***sus/venv/lib/python3.9/site-packages/woob/core/modules.py", line 281, in check_version
    pkg = pkg_resources.get_distribution(name)
  File "/var/www/***sus/venv/lib/python3.9/site-packages/pkg_resources/__init__.py", line 497, in get_distribution
    dist = get_provider(dist)
  File "/var/www/***sus/venv/lib/python3.9/site-packages/pkg_resources/__init__.py", line 384, in get_provider
    return working_set.find(moduleOrReq) or require(str(moduleOrReq))[0]
  File "/var/www/***sus/venv/lib/python3.9/site-packages/pkg_resources/__init__.py", line 937, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/var/www/***sus/venv/lib/python3.9/site-packages/pkg_resources/__init__.py", line 798, in resolve
    dist = self._resolve_dist(
  File "/var/www/***sus/venv/lib/python3.9/site-packages/pkg_resources/__init__.py", line 839, in _resolve_dist
    raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'python-jose' distribution was not found and is required by the application

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/var/www/***sus/build/server/providers/woob/py/main.py", line 924, in main
    woob_connector.create_backend(bank_module, params, session)
  File "/var/www/***sus/build/server/providers/woob/py/main.py", line 525, in create_backend
    self.backend = self.woob.build_backend(
  File "/var/www/***sus/venv/lib/python3.9/site-packages/woob/core/woob.py", line 504, in build_backend
    return super().build_backend(module_name, params, storage, name, nofail, logger)
  File "/var/www/***sus/venv/lib/python3.9/site-packages/woob/core/woob.py", line 141, in build_backend
    module = self.modules_loader.get_or_load_module(module_name)
  File "/var/www/***sus/venv/lib/python3.9/site-packages/woob/core/modules.py", line 197, in get_or_load_module
    self.load_module(module_name)
  File "/var/www/***sus/venv/lib/python3.9/site-packages/woob/core/modules.py", line 239, in load_module
    self.check_version(module_name, module_spec)
  File "/var/www/***sus/venv/lib/python3.9/site-packages/woob/core/modules.py", line 283, in check_version
    raise ModuleLoadError(
woob.exceptions.ModuleLoadError: Module requires python package "python-jose" but not installed.

[nicofrand]

It should be installed already:

kresus_ynh/manifest.toml

Line 62 in 2016f08

packages = "locales, git, python3, python3-dev, python3-pip, python3-selenium, python3-josepy, libffi-dev, libxml2-dev, libxslt-dev, libyaml-dev, libtiff-dev, libjpeg-dev, libopenjp2-7-dev, zlib1g-dev, libfreetype6-dev, libwebp-dev, build-essential, gcc, g++, wget, unzip, mupdf-tools, libnss3-tools, python3-nss, virtualenv, postgresql"

[etieng]

Well, definitely python3-josepy debian package was installed

2024-04-25 11:25:35,586: DEBUG - Selecting previously unselected package python3-josepy.
2024-04-25 11:25:35,593: DEBUG - Preparing to unpack .../64-python3-josepy_1.2.0-2_all.deb ...
2024-04-25 11:25:35,599: DEBUG - Unpacking python3-josepy (1.2.0-2) ...
>><<
2024-04-25 11:25:40,494: DEBUG - Setting up python3-josepy (1.2.0-2) ...

With no errors returned, but at some point python-jose was missing for Woob, as stated in logs of first publication.

How can I investigate further ?

[nicofrand]

I don't really know… I kind of remember of a similar issue in the other way (debian package was working, not the one from pip) so I am not eager to replace one by the other :/

[etieng]

Ok, if I'm the only one concerned...
I posted to have a record somewhere.

[tseignette]

Hi there, I am facing the same issue.

Context

• Hardware: old MacBook Pro with Debian (bare-metal amd64, Linux 6.1.0-34-amd64, Debian 12.10

• YunoHost version: 12.0.16 (stable)

• I have access to my server: Through SSH | through the webadmin

• Are you in a special context or did you perform some particular tweaking on your YunoHost instance?: no

• Using, or trying to install package version/branch: stable 0.22.1~ynh2

Here are the logs when I try to add an account:

[2025-05-26T11:21:45.630] [ERROR] providers/woob - 	(error code: UNKNOWN_WOOB_MODULE)
[2025-05-26T11:21:45.658] [ERROR] controllers/accesses - The access process creation failed, cleaning up...
[2025-05-26T11:21:45.658] [INFO] controllers/accesses - 	deleting access...
[2025-05-26T11:21:45.673] [ERROR] helpers - when creating a bank access: Unable to load module creditcooperatif.
Traceback (most recent call last):
  File "/usr/lib/python3.11/importlib/metadata/__init__.py", line 563, in from_name
    return next(cls.discover(name=name))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
StopIteration

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/var/www/***sus/venv/lib/python3.11/site-packages/woob/core/modules.py", line 286, in check_version
    pkg = metadata.distribution(name)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/importlib/metadata/__init__.py", line 981, in distribution
    return Distribution.from_name(distribution_name)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/importlib/metadata/__init__.py", line 565, in from_name
    raise PackageNotFoundError(name)
importlib.metadata.PackageNotFoundError: No package metadata was found for python-jose

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/var/www/***sus/build/server/providers/woob/py/main.py", line 948, in main
    woob_connector.create_backend(bank_module, params, session)
  File "/var/www/***sus/build/server/providers/woob/py/main.py", line 528, in create_backend
    self.backend = self.woob.build_backend(
                   ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/www/***sus/venv/lib/python3.11/site-packages/woob/core/woob.py", line 504, in build_backend
    return super().build_backend(module_name, params, storage, name, nofail, logger)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/www/***sus/venv/lib/python3.11/site-packages/woob/core/woob.py", line 141, in build_backend
    module = self.modules_loader.get_or_load_module(module_name)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/www/***sus/venv/lib/python3.11/site-packages/woob/core/modules.py", line 202, in get_or_load_module
    self.load_module(module_name)
  File "/var/www/***sus/venv/lib/python3.11/site-packages/woob/core/modules.py", line 244, in load_module
    self.check_version(module_name, module_spec)
  File "/var/www/***sus/venv/lib/python3.11/site-packages/woob/core/modules.py", line 288, in check_version
    raise ModuleLoadError(
woob.exceptions.ModuleLoadError: Module requires python package "python-jose" but not installed.

If I try @etieng suggestion to sudo -u kresus pip install python-jose I get the following error : error: externally-managed-environment so I manually installed python-jose with sudo apt install python3-jose. I then get this error when I re-try to add my account:

[2025-05-26T11:13:09.526] [ERROR] providers/woob - Got error while running command "3": Unknown error: 'NoneType' object has no attribute 'get_snid'.
Traceback (most recent call last):
  File "/var/www/***sus/build/server/providers/woob/py/main.py", line 711, in fetch
    results["values"] = self.get_accounts()
                        ^^^^^^^^^^^^^^^^^^^
  File "/var/www/***sus/build/server/providers/woob/py/main.py", line 553, in get_accounts
    for account in list(self.backend.iter_accounts()):
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/yunohost.app/***sus/woob-data/modules/3.7/woob_modules/caissedepargne/module.py", line 78, in iter_accounts
    return self.browser.iter_accounts()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/www/***sus/venv/lib/python3.11/site-packages/woob/browser/switch.py", line 98, in wrapper
    return val(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^
  File "/var/www/***sus/venv/lib/python3.11/site-packages/woob/browser/browsers.py", line 1154, in inner
    browser.do_login()
  File "/var/www/***sus/venv/lib/python3.11/site-packages/woob/browser/mfa.py", line 201, in do_double_authentication
    self.init_login()
  File "/home/yunohost.app/***sus/woob-data/modules/3.7/woob_modules/caissedepargne/browser.py", line 231, in init_login
    self.do_api_pre_login()
  File "/home/yunohost.app/***sus/woob-data/modules/3.7/woob_modules/caissedepargne/browser.py", line 257, in do_api_pre_login
    self.snid = self.page.get_snid(self.enseigne.upper())
                ^^^^^^^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'get_snid'

@etieng I see that you are trying to add an account from Crédit Coopératif also, it may be related to that?

[tseignette]

It should be installed already:

kresus_ynh/manifest.toml

Line 62 in 2016f08
packages = "locales, git, python3, python3-dev, python3-pip, python3-selenium, python3-josepy, libffi-dev, libxml2-dev, libxslt-dev, libyaml-dev, libtiff-dev, libjpeg-dev, libopenjp2-7-dev, zlib1g-dev, libfreetype6-dev, libwebp-dev, build-essential, gcc, g++, wget, unzip, mupdf-tools, libnss3-tools, python3-nss, virtualenv, postgresql"

@nicofrand The missing package is python-jose, is python3-josepy the same thing?

[nicofrand]

It should be installed already:
kresus_ynh/manifest.toml
Line 62 in 2016f08
packages = "locales, git, python3, python3-dev, python3-pip, python3-selenium, python3-josepy, libffi-dev, libxml2-dev, libxslt-dev, libyaml-dev, libtiff-dev, libjpeg-dev, libopenjp2-7-dev, zlib1g-dev, libfreetype6-dev, libwebp-dev, build-essential, gcc, g++, wget, unzip, mupdf-tools, libnss3-tools, python3-nss, virtualenv, postgresql"

@nicofrand The missing package is python-jose, is python3-josepy the same thing?

I believe so yes https://packages.debian.org/search?keywords=python3-jose https://packages.debian.org/search?keywords=python-jose

[tseignette]

It should be installed already:
kresus_ynh/manifest.toml
Line 62 in 2016f08
packages = "locales, git, python3, python3-dev, python3-pip, python3-selenium, python3-josepy, libffi-dev, libxml2-dev, libxslt-dev, libyaml-dev, libtiff-dev, libjpeg-dev, libopenjp2-7-dev, zlib1g-dev, libfreetype6-dev, libwebp-dev, build-essential, gcc, g++, wget, unzip, mupdf-tools, libnss3-tools, python3-nss, virtualenv, postgresql"

@nicofrand The missing package is python-jose, is python3-josepy the same thing?

I believe so yes https://packages.debian.org/search?keywords=python3-jose https://packages.debian.org/search?keywords=python-jose

Sorry I don't know if I was clear but the code you quoted mentioned the package python3-josepy (with a py suffix). There is also a debian package called python3-jose (without the py suffix), which is the one which seems to be mentioned in the errors we got.
If we look at the debian packages pages:

• python3-josepy version for bookworm: 1.13.0-1
• python3-jose version for bookworm: 3.3.0+dfsg-4

This could be the culprit, right? I can test it out in the next days.

[nicofrand]

This was intended as far as I remember (which is: not much): 9f8f303

[nicofrand]

Hm, I think (I should have left better comments) that at the time the python3-jose package did not exist and only the python3-josepy (relying on a pure python implementation) existed.

Confirmed in #186 (comment).

So, maybe we now switch to python3-jose indeed.

It would be nice to check if python3-josepy still works though.

[tseignette]

Thanks for the investigation.

I ran the installation again and got the same missing jose error. Here is what I found:

• sudo apt install python3-josepy: python3-josepy is already the newest version (1.13.0-1). so the package was installed correctly but does not seem to work
• sudo apt install python3-jose > Add an account:

[2025-05-28T06:15:53.400] [INFO] providers/woob - exited with code 1.
[2025-05-28T06:15:53.400] [WARN] providers/woob - stderr: 2025-05-28 06:15:52,301:DEBUG:woob.core.modules.loader:modules.py:255:load_module Loaded module "creditcooperatif" from /home/yunohost.app/***sus/woob-data/modules/3.7/woob_modules/creditcooperatif
2025-05-28 06:15:52,301:DEBUG:woob.backend:modules.py:163:create_instance Created backend "creditcooperatif" for module "creditcooperatif"
2025-05-28 06:15:52,302:DEBUG:urllib3.util.retry:retry.py:286:from_int Converted retries value: 2 -> Retry(total=2, connect=None, read=None, redirect=None, status=None)
2025-05-28 06:15:52,304:DEBUG:urllib3.util.retry:retry.py:286:from_int Converted retries value: 2 -> Retry(total=2, connect=None, read=None, redirect=None, status=None)
2025-05-28 06:15:52,306:DEBUG:urllib3.util.retry:retry.py:286:from_int Converted retries value: 2 -> Retry(total=2, connect=None, read=None, redirect=None, status=None)
2025-05-28 06:15:52,306:DEBUG:urllib3.util.retry:retry.py:286:from_int Converted retries value: 2 -> Retry(total=2, connect=None, read=None, redirect=None, status=None)
2025-05-28 06:15:52,306:INFO:woob.creditcooperatif.browser:switch.py:88:set_browser using 'main' browser
2025-05-28 06:15:52,307:DEBUG:urllib3.util.retry:retry.py:286:from_int Converted retries value: 2 -> Retry(total=2, connect=None, read=None, redirect=None, status=None)
2025-05-28 06:15:52,307:DEBUG:urllib3.util.retry:retry.py:286:from_int Converted retries value: 2 -> Retry(total=2, connect=None, read=None, redirect=None, status=None)
2025-05-28 06:15:52,309:DEBUG:urllib3.util.retry:retry.py:286:from_int Converted retries value: 2 -> Retry(total=2, connect=None, read=None, redirect=None, status=None)
2025-05-28 06:15:52,309:DEBUG:urllib3.util.retry:retry.py:286:from_int Converted retries value: 2 -> Retry(total=2, connect=None, read=None, redirect=None, status=None)
2025-05-28 06:15:52,310:INFO:woob.creditcooperatif.browser:switch.py:88:set_browser using 'main' browser
2025-05-28 06:15:52,311:DEBUG:urllib3.connectionpool:connectionpool.py:1049:_new_conn Starting new HTTPS connection (1): www.credit-cooperatif.coop:443
2025-05-28 06:15:52,540:DEBUG:urllib3.connectionpool:connectionpool.py:544:_make_request https://www.credit-cooperatif.coop:443 "GET /se-connecter/sso?service=dei HTTP/1.1" 302 255
2025-05-28 06:15:52,541:DEBUG:urllib3.connectionpool:connectionpool.py:1049:_new_conn Starting new HTTPS connection (1): www.icgauth.credit-cooperatif.coop:443
2025-05-28 06:15:53,062:DEBUG:urllib3.connectionpool:connectionpool.py:544:_make_request https://www.icgauth.credit-cooperatif.coop:443 "GET /se-connecter/sso?service=dei HTTP/1.1" 200 1749
2025-05-28 06:15:53,075:DEBUG:woob.creditcooperatif.browser:browsers.py:1040:internal_callback Handle https://www.icgauth.credit-cooperatif.coop/se-connecter/sso?service=dei with LoginPage
2025-05-28 06:15:53,170:DEBUG:urllib3.connectionpool:connectionpool.py:544:_make_request https://www.icgauth.credit-cooperatif.coop:443 "GET /se-connecter/main-GYV2L3NN.js HTTP/1.1" 200 None
2025-05-28 06:15:53,282:DEBUG:woob.creditcooperatif.browser:browsers.py:1054:internal_callback Unable to handle https://www.icgauth.credit-cooperatif.coop/se-connecter/main-GYV2L3NN.js
[2025-05-28T06:15:53.401] [INFO] providers/woob - Command returned an error code.
[2025-05-28T06:15:53.401] [INFO] providers/woob - Checking that woob is installed and can actually be called…
[2025-05-28T06:15:53.401] [INFO] providers/woob - Calling woob: command test...
[2025-05-28T06:15:53.745] [INFO] providers/woob - exited with code 0.
[2025-05-28T06:15:53.745] [INFO] providers/woob - OK: woob exited normally with non-empty JSON content.
[2025-05-28T06:15:53.746] [ERROR] providers/woob - Got error while running command "3": Unknown error: 'NoneType' object has no attribute 'get_snid'.
Traceback (most recent call last):
  File "/var/www/***sus/build/server/providers/woob/py/main.py", line 711, in fetch
    results["values"] = self.get_accounts()
                        ^^^^^^^^^^^^^^^^^^^
  File "/var/www/***sus/build/server/providers/woob/py/main.py", line 553, in get_accounts
    for account in list(self.backend.iter_accounts()):
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/yunohost.app/***sus/woob-data/modules/3.7/woob_modules/caissedepargne/module.py", line 78, in iter_accounts
    return self.browser.iter_accounts()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/www/***sus/venv/lib/python3.11/site-packages/woob/browser/switch.py", line 98, in wrapper
    return val(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^
  File "/var/www/***sus/venv/lib/python3.11/site-packages/woob/browser/browsers.py", line 1154, in inner
    browser.do_login()
  File "/var/www/***sus/venv/lib/python3.11/site-packages/woob/browser/mfa.py", line 201, in do_double_authentication
    self.init_login()
  File "/home/yunohost.app/***sus/woob-data/modules/3.7/woob_modules/caissedepargne/browser.py", line 231, in init_login
    self.do_api_pre_login()
  File "/home/yunohost.app/***sus/woob-data/modules/3.7/woob_modules/caissedepargne/browser.py", line 257, in do_api_pre_login
    self.snid = self.page.get_snid(self.enseigne.upper())
                ^^^^^^^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'get_snid'

[2025-05-28T06:15:53.746] [ERROR] providers/woob - 	(error code: GENERIC_EXCEPTION)
[2025-05-28T06:15:53.761] [ERROR] controllers/accesses - The access process creation failed, cleaning up...
[2025-05-28T06:15:53.761] [INFO] controllers/accesses - 	deleting access...
[2025-05-28T06:15:53.768] [ERROR] helpers - when creating a bank access: Unknown error: 'NoneType' object has no attribute 'get_snid'.
Traceback (most recent call last):
  File "/var/www/***sus/build/server/providers/woob/py/main.py", line 711, in fetch
    results["values"] = self.get_accounts()
                        ^^^^^^^^^^^^^^^^^^^
  File "/var/www/***sus/build/server/providers/woob/py/main.py", line 553, in get_accounts
    for account in list(self.backend.iter_accounts()):
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/yunohost.app/***sus/woob-data/modules/3.7/woob_modules/caissedepargne/module.py", line 78, in iter_accounts
    return self.browser.iter_accounts()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/www/***sus/venv/lib/python3.11/site-packages/woob/browser/switch.py", line 98, in wrapper
    return val(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^
  File "/var/www/***sus/venv/lib/python3.11/site-packages/woob/browser/browsers.py", line 1154, in inner
    browser.do_login()
  File "/var/www/***sus/venv/lib/python3.11/site-packages/woob/browser/mfa.py", line 201, in do_double_authentication
    self.init_login()
  File "/home/yunohost.app/***sus/woob-data/modules/3.7/woob_modules/caissedepargne/browser.py", line 231, in init_login
    self.do_api_pre_login()
  File "/home/yunohost.app/***sus/woob-data/modules/3.7/woob_modules/caissedepargne/browser.py", line 257, in do_api_pre_login
    self.snid = self.page.get_snid(self.enseigne.upper())
                ^^^^^^^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'get_snid'

[2025-05-28T06:15:53.768] [INFO] helpers - Error
    at new KError (/var/www/***sus/build/server/helpers.js:62:22)
    at callWoob (/var/www/***sus/build/server/providers/woob/index.js:234:15)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async _fetchHelper (/var/www/***sus/build/server/providers/woob/index.js:291:16)
    at async Object.fetchAccounts (/var/www/***sus/build/server/providers/woob/index.js:305:12)
    at async /var/www/***sus/build/server/lib/accounts-manager.js:137:20
    at async retryCallProvider (/var/www/***sus/build/server/lib/accounts-manager.js:96:20)
    at async pollAccounts (/var/www/***sus/build/server/lib/accounts-manager.js:136:34)
    at async AccountManager.syncAccounts (/var/www/***sus/build/server/lib/accounts-manager.js:324:24)
    at async createAndRetrieveData (/var/www/***sus/build/server/controllers/accesses.js:120:33)
[2025-05-28T06:15:53.769] [ERROR] HTTP - POST /api/accesses - 500 (1935 ms)

But this is a Woob issue now right? So switching from python3-josepy to python3-jose seems to work.

[nicofrand]

But this is a Woob issue now right?

Yes.

Thanks for testing and investigating! I'll create a MR to change python3-josepy to python3-jose then. Or would you be eager to do it :)?

[tseignette]

Yes.

Thanks for testing and investigating! I'll create a MR to change python3-josepy to python3-jose then. Or would you be eager to do it :)?

Happy to help :) Yes I can do it tomorrow, and I'll open an issue in Woob

[tseignette]

And here is the Woob issue: https://gitlab.com/woob/woob/-/issues/777