Description
Json config settings like [core] dag_bundle_config_list and [secrets] backend_kwargs can become quite large and cumbersome. The linux kernel sets a maximum total size limit of 256MB for environment variables, so adding large json blobs to the environment variables risks running into these limits. To work around this, json files could be loaded into the config using the same technique as secrets by specifying a path to a json file in the config and reading the file at runtime using json.load().
Use case/motivation
Like with config settings tagged as secrets, update the conf.getjson() method to also attempt to load json config settings from a path specified using a suffix, e.g. dag_bundle_config_list_file=/opt/airflow/config/dag_bundles.json would read the setting dag_bundle_config_list from /opt/airflow/config/dag_bundles.json
Related issues
#65449
Are you willing to submit a PR?
Code of Conduct
Description
Json config settings like
[core] dag_bundle_config_listand[secrets] backend_kwargscan become quite large and cumbersome. The linux kernel sets a maximum total size limit of 256MB for environment variables, so adding large json blobs to the environment variables risks running into these limits. To work around this, json files could be loaded into the config using the same technique as secrets by specifying a path to a json file in the config and reading the file at runtime usingjson.load().Use case/motivation
Like with config settings tagged as secrets, update the
conf.getjson()method to also attempt to load json config settings from a path specified using a suffix, e.g.dag_bundle_config_list_file=/opt/airflow/config/dag_bundles.jsonwould read the settingdag_bundle_config_listfrom/opt/airflow/config/dag_bundles.jsonRelated issues
#65449
Are you willing to submit a PR?
Code of Conduct