diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 0423cc0..20292be 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -124,7 +124,7 @@ jobs: working-directory: modflow-devtools/autotest env: REPOS_PATH: ${{ github.workspace }} - MODFLOW_DEVTOOLS_NO_AUTO_SYNC: 1 + MODFLOW_DEVTOOLS_AUTO_SYNC: 0 TEST_DFN_PATH: ${{ github.workspace }}/modflow6/doc/mf6io/mf6ivar/dfn # use --dist loadfile to so tests requiring pytest-virtualenv run on the same worker run: uv run pytest -v -n auto --dist loadfile --durations 0 --ignore test_download.py --ignore test_models.py --ignore test_dfns_registry.py diff --git a/HISTORY.md b/HISTORY.md index 39e75f6..355b7be 100644 --- a/HISTORY.md +++ b/HISTORY.md @@ -1,35 +1,19 @@ -### Version 1.9.0 +### Version 1.9.1 -#### New features +#### Bug fixes -* [feat(misc)](https://github.com/MODFLOW-ORG/modflow-devtools/commit/910a1f1d201c262e218da27865b7e9487271e4d2): Add verbose option to set_dir (#258). Committed by wpbonelli on 2025-11-06. -* [feat(models)](https://github.com/MODFLOW-ORG/modflow-devtools/commit/c8a136872198e1d9f36be74e958d003cdc75f6eb): Improve models API (#268). Committed by wpbonelli on 2026-01-15. -* [feat(programs)](https://github.com/MODFLOW-ORG/modflow-devtools/commit/5e85417270cea1b4e5081d531c8c98b072acc371): Add programs API (#270). Committed by wpbonelli on 2026-01-16. -* [feat(version)](https://github.com/MODFLOW-ORG/modflow-devtools/commit/f9e26de1efdda95d509c0cc0cb1cf886a3f6c278): Add project version management tool (#284). Committed by wpbonelli on 2026-02-13. -* [feat(dfns)](https://github.com/MODFLOW-ORG/modflow-devtools/commit/aa99c56f5bc5fc2e846262d5bb5c6d80dc9d69b9): Improve DFNs API (#271). Committed by wpbonelli on 2026-02-23. -* [feat(models)](https://github.com/MODFLOW-ORG/modflow-devtools/commit/c430e8b26157e954a5543344b083125dc5b2ed18): Add copy command (#295). Committed by wpbonelli on 2026-02-25. +* [fix(dfn)](https://github.com/MODFLOW-ORG/modflow-devtools/commit/2dd65bf57307aa42a6043aca3e6da8f8253dc4f9): Restore old dfn2toml logic (#299). Committed by wpbonelli on 2026-03-04. +* [fix](https://github.com/MODFLOW-ORG/modflow-devtools/commit/1e94d1bae8d33acf27becce3560bec092922dc35): Fix version comparison, improve autosync behavior (#300). Committed by wpbonelli on 2026-03-04. -#### Bug fixes +### Version 1.9.0 + +#### New features -* [fix(models)](https://github.com/MODFLOW-ORG/modflow-devtools/commit/d736dfd881cbc39d2fe82180d98d1d2a0130f146): Backwards-compatibility adjustments (#272). Committed by wpbonelli on 2026-01-20. -* [fix(models)](https://github.com/MODFLOW-ORG/modflow-devtools/commit/a1497797fc9b1a7cf4ed7cd72e43f0b290fd49d4): Miscellaneous fixes (#273). Committed by wpbonelli on 2026-01-21. -* [fix(models)](https://github.com/MODFLOW-ORG/modflow-devtools/commit/5dce2af85369063d037c58a3bfc12a2e32c74770): Check env var before auto-sync, fix tests (#275). Committed by wpbonelli on 2026-01-23. -* [fix(models)](https://github.com/MODFLOW-ORG/modflow-devtools/commit/f6cc9bbbcc8577a4384e18159597c128181e3ecf): Exclude output files from registries (#274). Committed by wpbonelli on 2026-01-23. -* [fix(models)](https://github.com/MODFLOW-ORG/modflow-devtools/commit/5aa924836fe700bb7ccf4c56faeea52b72368bb5): Auto-sync only on api commands (#277). Committed by wpbonelli on 2026-02-10. -* [fix(programs)](https://github.com/MODFLOW-ORG/modflow-devtools/commit/2674176f6151814fa158269478015b1b175ba884): Exe path at program or dist level, add tests, update docs (#281). Committed by wpbonelli on 2026-02-11. -* [fix(programs)](https://github.com/MODFLOW-ORG/modflow-devtools/commit/a9584012258e5ca52be86a7d631da2f0bf10b75c): Multiple fixes/improvements (#282). Committed by wpbonelli on 2026-02-12. -* [fix](https://github.com/MODFLOW-ORG/modflow-devtools/commit/18af51ae53bf1c8fb877620b0fa26e7be28ea210): More fixes/improvements (#283). Committed by wpbonelli on 2026-02-12. -* [fix](https://github.com/MODFLOW-ORG/modflow-devtools/commit/448993522772773ea650206e94185ad8ec72bb83): Multiple fixes (#285). Committed by wpbonelli on 2026-02-20. -* [fix(models)](https://github.com/MODFLOW-ORG/modflow-devtools/commit/02ab50dac6540191914506329dcc9ef52820f7ff): Fix model registry generation from release assets (#290). Committed by wpbonelli on 2026-02-23. -* [fix(dfn)](https://github.com/MODFLOW-ORG/modflow-devtools/commit/ebaffb9e2d364210789cd003a5654f4d6741436d): Restore dfn module and tests (#291). Committed by wpbonelli on 2026-02-23. -* [fix(tests)](https://github.com/MODFLOW-ORG/modflow-devtools/commit/d83dfa9470c619693c36da2aacd526916141aec1): Fix test branch (#292). Committed by wpbonelli on 2026-02-24. -* [fix(models)](https://github.com/MODFLOW-ORG/modflow-devtools/commit/9dfa972efffe310f5de79fa31cdc0235716edc2c): Don't exclude .obs files when copying input files (#293). Committed by wpbonelli on 2026-02-25. +This release improves the architecture of the models API, decoupling model repositories from devtools and adding a CLI. The improvements should all be backwards-compatible. -#### Refactoring +This release also contains an experimental programs API that can be used to inspect available versions of MODFLOW 6 and related programs, and help with installs, as well as an experimental redesign of the MF6 DFN specification file utilities. -* [refactor(programs)](https://github.com/MODFLOW-ORG/modflow-devtools/commit/bb76dc1c36f235b3e8bd5b10231661081dda7fe9): Multiple programs api improvements (#276). Committed by wpbonelli on 2026-02-09. -* [refactor(registries)](https://github.com/MODFLOW-ORG/modflow-devtools/commit/75c6909a6202da02b30a1b7b5743e75498aa26a7): Leaner model/program registry file contents (#279). Committed by wpbonelli on 2026-02-10. -* [refactor(dependencies)](https://github.com/MODFLOW-ORG/modflow-devtools/commit/fa8c8d9753da38a9a400c0f7d736944d08a3a059): Introduce ecosystem group (#289). Committed by wpbonelli on 2026-02-22. +The aim here is to unify this project's ecosystem management utilities with consistent design and UX once we reach version 2. No breaking changes (and new stuff will stay experimental) for the remainder of 1.x. ### Version 1.8.0 diff --git a/autotest/test_dfns.py b/autotest/test_dfns.py index 52b0217..ab5c281 100644 --- a/autotest/test_dfns.py +++ b/autotest/test_dfns.py @@ -4,15 +4,15 @@ import pytest from packaging.version import Version -from modflow_devtools.dfn2toml import convert, is_valid from modflow_devtools.dfns import Dfn, _load_common, load, load_flat +from modflow_devtools.dfns.dfn2toml import convert, is_valid from modflow_devtools.dfns.fetch import fetch_dfns from modflow_devtools.dfns.schema.v1 import FieldV1 from modflow_devtools.dfns.schema.v2 import FieldV2 from modflow_devtools.markers import requires_pkg PROJ_ROOT = Path(__file__).parents[1] -DFN_DIR = PROJ_ROOT / "autotest" / "temp" / "dfn" +DFN_DIR = PROJ_ROOT / "autotest" / "temp" / "dfns" TOML_DIR = DFN_DIR / "toml" SPEC_DIRS = {1: DFN_DIR, 2: TOML_DIR} MF6_OWNER = "MODFLOW-ORG" diff --git a/docs/conf.py b/docs/conf.py index 09bf7a0..ed1dcb5 100644 --- a/docs/conf.py +++ b/docs/conf.py @@ -8,7 +8,7 @@ project = "modflow-devtools" author = "MODFLOW Team" -release = "1.9.0" +release = "1.9.1" # -- General configuration --------------------------------------------------- # https://www.sphinx-doc.org/en/master/usage/configuration.html#general-configuration diff --git a/docs/md/dev/dfns.md b/docs/md/dev/dfns.md index 3fd39b2..0ca75dd 100644 --- a/docs/md/dev/dfns.md +++ b/docs/md/dev/dfns.md @@ -476,7 +476,7 @@ status = get_sync_status() - **At install time**: Best-effort sync to default refs during package installation (fail silently on network errors) - **On first use**: If registry cache is empty for requested ref, attempt to sync before raising errors - **Lazy loading**: Don't sync until DFN access is actually requested -- **Configurable**: Users can disable auto-sync via environment variable: `MODFLOW_DEVTOOLS_NO_AUTO_SYNC=1` +- **Configurable (Experimental)**: Auto-sync is opt-in via environment variable: `MODFLOW_DEVTOOLS_AUTO_SYNC=1` (set to "1", "true", or "yes") ### Source repository integration @@ -1295,7 +1295,7 @@ dfn.name # attribute access 5. Implement `sync_dfns()` function 6. Add registry metadata caching with hash verification 7. Implement version-controlled registry discovery -8. Add auto-sync on first use (with opt-out via `MODFLOW_DEVTOOLS_NO_AUTO_SYNC`) +8. Add auto-sync on first use (opt-in via `MODFLOW_DEVTOOLS_AUTO_SYNC` while experimental) 9. **Implement `DfnSpec` dataclass** with `Mapping` protocol for single canonical hierarchical representation with flat dict access **CLI and module API** (depends on Registry infrastructure): diff --git a/docs/md/dev/programs.md b/docs/md/dev/programs.md index 1c5d7ba..0426f5e 100644 --- a/docs/md/dev/programs.md +++ b/docs/md/dev/programs.md @@ -509,7 +509,7 @@ status = get_sync_status() - **At install time**: Best-effort sync during package installation (fail silently on network errors) - **On first use**: If registry cache is empty, attempt to sync before raising errors -- **Configurable**: Users can disable auto-sync via environment variable: `MODFLOW_DEVTOOLS_NO_AUTO_SYNC=1` +- **Configurable (Experimental)**: Auto-sync is opt-in via environment variable: `MODFLOW_DEVTOOLS_AUTO_SYNC=1` (set to "1", "true", or "yes") #### Force semantics diff --git a/docs/md/models.md b/docs/md/models.md index 9261abc..40eb37d 100644 --- a/docs/md/models.md +++ b/docs/md/models.md @@ -156,7 +156,7 @@ python -m modflow_devtools.models cp mf6/example/ex-gwf-twri01 /path/to/workspac ``` The copy command: -- Automatically attempts to sync registries before copying (unless `MODFLOW_DEVTOOLS_NO_AUTO_SYNC=1`) +- Automatically attempts to sync registries before copying (if `MODFLOW_DEVTOOLS_AUTO_SYNC=1`) - Creates the workspace directory if it doesn't exist - Copies all input files for the specified model - Preserves subdirectory structure within the workspace @@ -316,16 +316,16 @@ mf models clear --force ## Automatic Synchronization -By default, `modflow-devtools` attempts to sync registries: -- On first import (best-effort, fails silently on network errors) -- When accessing models (unless `MODFLOW_DEVTOOLS_NO_AUTO_SYNC=1`) - -To disable auto-sync: +Auto-sync is **opt-in** (experimental). To enable: ```bash -export MODFLOW_DEVTOOLS_NO_AUTO_SYNC=1 +export MODFLOW_DEVTOOLS_AUTO_SYNC=1 # or "true" or "yes" ``` +When enabled, `modflow-devtools` attempts to sync registries: +- On first access (best-effort, fails silently on network errors) +- When accessing models via the API or CLI + Then manually sync when needed: ```bash diff --git a/docs/md/programs.md b/docs/md/programs.md index 24a98bb..8da048f 100644 --- a/docs/md/programs.md +++ b/docs/md/programs.md @@ -347,17 +347,17 @@ mf programs install mf6 --force ## Automatic Synchronization -By default, `modflow-devtools` attempts to sync registries: -- On first import (best-effort, fails silently on network errors) -- Before installation (unless `MODFLOW_DEVTOOLS_NO_AUTO_SYNC=1`) -- Before listing available programs - -To disable auto-sync: +Auto-sync is **opt-in** (experimental). To enable: ```bash -export MODFLOW_DEVTOOLS_NO_AUTO_SYNC=1 +export MODFLOW_DEVTOOLS_AUTO_SYNC=1 # or "true" or "yes" ``` +When enabled, `modflow-devtools` attempts to sync registries: +- On first access (best-effort, fails silently on network errors) +- Before installation +- Before listing available programs + Then manually sync when needed: ```bash diff --git a/modflow_devtools/__init__.py b/modflow_devtools/__init__.py index b75c884..5e65b73 100644 --- a/modflow_devtools/__init__.py +++ b/modflow_devtools/__init__.py @@ -1,6 +1,6 @@ __author__ = "Joseph D. Hughes" -__date__ = "Feb 25, 2026" -__version__ = "1.9.0" +__date__ = "Mar 05, 2026" +__version__ = "1.9.1" __maintainer__ = "Joseph D. Hughes" __email__ = "jdhughes@usgs.gov" __status__ = "Production" diff --git a/modflow_devtools/cli.py b/modflow_devtools/cli.py index 563cf61..22978bd 100644 --- a/modflow_devtools/cli.py +++ b/modflow_devtools/cli.py @@ -2,6 +2,11 @@ Root CLI for modflow-devtools. Usage: + mf sync + mf dfns sync + mf dfns info + mf dfns list + mf dfns clean mf models sync mf models info mf models list @@ -17,6 +22,56 @@ import argparse import sys +import warnings + + +def _sync_all(): + """Sync all registries (dfns, models, programs).""" + print("Syncing all registries...") + print() + + # Sync DFNs + print("=== DFNs ===") + try: + from modflow_devtools.dfns.registry import sync_dfns + + registries = sync_dfns() + for registry in registries: + meta = registry.registry_meta + print(f" {registry.ref}: {len(meta.files)} files") + print(f"Synced {len(registries)} DFN registry(ies)") + except Exception as e: + print(f"Error syncing DFNs: {e}") + print() + + # Sync Models + print("=== Models ===") + try: + from modflow_devtools.models import ModelSourceConfig + + config = ModelSourceConfig.load() + config.sync() + print("Models synced successfully") + except Exception as e: + print(f"Error syncing models: {e}") + print() + + # Sync Programs + print("=== Programs ===") + try: + # Suppress experimental warning + with warnings.catch_warnings(): + warnings.filterwarnings("ignore", message=".*modflow_devtools.programs.*experimental.*") + from modflow_devtools.programs import ProgramSourceConfig + + config = ProgramSourceConfig.load() + config.sync() + print("Programs synced successfully") + except Exception as e: + print(f"Error syncing programs: {e}") + print() + + print("All registries synced!") def main(): @@ -27,6 +82,12 @@ def main(): ) subparsers = parser.add_subparsers(dest="subcommand", help="Available commands") + # Sync subcommand (syncs all APIs) + subparsers.add_parser("sync", help="Sync all registries (dfns, models, programs)") + + # DFNs subcommand + subparsers.add_parser("dfns", help="Manage MODFLOW 6 definition files") + # Models subcommand subparsers.add_parser("models", help="Manage MODFLOW model registries") @@ -41,7 +102,14 @@ def main(): sys.exit(1) # Dispatch to the appropriate module CLI with remaining args - if args.subcommand == "models": + if args.subcommand == "sync": + _sync_all() + elif args.subcommand == "dfns": + from modflow_devtools.dfns.__main__ import main as dfns_main + + sys.argv = ["mf dfns", *remaining] + sys.exit(dfns_main()) + elif args.subcommand == "models": from modflow_devtools.models.__main__ import main as models_main # Replace sys.argv to make it look like we called the submodule directly diff --git a/modflow_devtools/dfn2toml.py b/modflow_devtools/dfn2toml.py index 7ec012d..96a6866 100644 --- a/modflow_devtools/dfn2toml.py +++ b/modflow_devtools/dfn2toml.py @@ -1,126 +1,46 @@ """Convert DFNs to TOML.""" import argparse -import sys -import textwrap -from dataclasses import asdict from os import PathLike from pathlib import Path import tomli_w as tomli from boltons.iterutils import remap -from modflow_devtools.dfns import Dfn, is_valid, load, load_flat, map, to_flat, to_tree -from modflow_devtools.dfns.schema.block import block_sort_key -from modflow_devtools.misc import drop_none_or_empty +from modflow_devtools.dfn import Dfn # mypy: ignore-errors -def convert(inpath: PathLike, outdir: PathLike, schema_version: str = "2") -> None: - """ - Convert DFN files in `inpath` to TOML files in `outdir`. - By default, convert the definitions to schema version 2. - """ - inpath = Path(inpath).expanduser().absolute() +def convert(indir: PathLike, outdir: PathLike): + indir = Path(indir).expanduser().absolute() outdir = Path(outdir).expanduser().absolute() outdir.mkdir(exist_ok=True, parents=True) + for dfn in Dfn.load_all(indir).values(): + with Path.open(outdir / f"{dfn['name']}.toml", "wb") as f: - if inpath.is_file(): - if inpath.name == "common.dfn": - raise ValueError("Cannot convert common.dfn as a standalone file") + def drop_none_or_empty(path, key, value): + if value is None or value == "" or value == [] or value == {}: + return False + return True - common_path = inpath.parent / "common.dfn" - if common_path.exists(): - with common_path.open() as f: - from modflow_devtools.dfn import parse_dfn - - common, _ = parse_dfn(f) - else: - common = {} - - with inpath.open() as f: - dfn = load(f, name=inpath.stem, common=common, format="dfn") - - dfn = map(dfn, schema_version=schema_version) - _convert(dfn, outdir / f"{inpath.stem}.toml") - else: - dfns = { - name: map(dfn, schema_version=schema_version) for name, dfn in load_flat(inpath).items() - } - tree = to_tree(dfns) - flat = to_flat(tree) - for dfn_name, dfn in flat.items(): - _convert(dfn, outdir / f"{dfn_name}.toml") - - -def _convert(dfn: Dfn, outpath: Path) -> None: - with Path.open(outpath, "wb") as f: - # TODO if we start using c/attrs, swap out - # all this for a custom unstructuring hook - dfn_dict = asdict(dfn) - dfn_dict["schema_version"] = str(dfn_dict["schema_version"]) - if blocks := dfn_dict.pop("blocks", None): - for block_name, block_fields in blocks.items(): - if block_name not in dfn_dict: - dfn_dict[block_name] = {} - for field_name, field_data in block_fields.items(): - dfn_dict[block_name][field_name] = field_data - - tomli.dump( - dict( - sorted( - remap(dfn_dict, visit=drop_none_or_empty).items(), - key=block_sort_key, - ) - ), - f, - ) + tomli.dump(remap(dfn, visit=drop_none_or_empty), f) if __name__ == "__main__": - """ - Convert DFN files in the original format and schema version 1 - to TOML files, by default also converting to schema version 2. - """ + """Convert DFN files to TOML.""" - parser = argparse.ArgumentParser( - description="Convert DFN files to TOML.", - epilog=textwrap.dedent( - """\ -Convert DFN files in the original format and schema version 1 -to TOML files, by default also converting to schema version 2. -""" - ), - ) + parser = argparse.ArgumentParser(description="Convert DFN files to TOML.") parser.add_argument( "--indir", "-i", type=str, - help="Directory containing DFN files, or a single DFN file.", + help="Directory containing DFN files.", ) parser.add_argument( "--outdir", "-o", help="Output directory.", ) - parser.add_argument( - "--schema-version", - "-s", - type=str, - default="2", - help="Schema version to convert to.", - ) - parser.add_argument( - "--validate", - "-v", - action="store_true", - help="Validate DFN files without converting them.", - ) args = parser.parse_args() - - if args.validate: - if not is_valid(args.indir): - sys.exit(1) - else: - convert(args.indir, args.outdir, args.schema_version) + convert(args.indir, args.outdir) diff --git a/modflow_devtools/dfns/__init__.py b/modflow_devtools/dfns/__init__.py index d141424..eedf9b6 100644 --- a/modflow_devtools/dfns/__init__.py +++ b/modflow_devtools/dfns/__init__.py @@ -575,11 +575,12 @@ def map( schema_version: str | Version = "2", ) -> Dfn: """Map a MODFLOW 6 specification to another schema version.""" - if dfn.schema_version == schema_version: + version = Version(str(schema_version)) + if version == dfn.schema_version: return dfn - elif Version(str(schema_version)) == Version("1"): + elif version == Version("1"): raise NotImplementedError("Mapping to schema version 1 is not implemented yet.") - elif Version(str(schema_version)) == Version("2"): + elif version == Version("2"): return MapV1To2().map(dfn) raise ValueError(f"Unsupported schema version: {schema_version}. Expected 1 or 2.") diff --git a/modflow_devtools/dfns/__main__.py b/modflow_devtools/dfns/__main__.py index 7a39ad4..bdfe4d7 100644 --- a/modflow_devtools/dfns/__main__.py +++ b/modflow_devtools/dfns/__main__.py @@ -2,10 +2,10 @@ Command-line interface for the DFNs API. Usage: - python -m modflow_devtools.dfns sync [--ref REF] [--force] - python -m modflow_devtools.dfns info - python -m modflow_devtools.dfns list [--ref REF] - python -m modflow_devtools.dfns clean [--all] + mf dfns sync [--ref REF] [--force] + mf dfns info + mf dfns list [--ref REF] + mf dfns clean [--all] """ from __future__ import annotations @@ -138,7 +138,7 @@ def cmd_list(args: argparse.Namespace) -> int: except DfnRegistryNotFoundError as e: print(f"Error: {e}", file=sys.stderr) - print("Try running 'python -m modflow_devtools.dfn sync' first.", file=sys.stderr) + print("Try running 'mf dfns sync' first.", file=sys.stderr) return 1 except Exception as e: print(f"Error: {e}", file=sys.stderr) @@ -198,7 +198,7 @@ def _format_size(size_bytes: int) -> str: def main(argv: list[str] | None = None) -> int: """Main entry point for the CLI.""" parser = argparse.ArgumentParser( - prog="python -m modflow_devtools.dfn", + prog="mf dfns", description="MODFLOW 6 definition file tools", ) parser.add_argument( diff --git a/modflow_devtools/dfns/dfn2toml.py b/modflow_devtools/dfns/dfn2toml.py new file mode 100644 index 0000000..3376028 --- /dev/null +++ b/modflow_devtools/dfns/dfn2toml.py @@ -0,0 +1,125 @@ +"""Convert DFNs to TOML.""" + +import argparse +import sys +import textwrap +from dataclasses import asdict +from os import PathLike +from pathlib import Path + +import tomli_w as tomli +from boltons.iterutils import remap + +from modflow_devtools.dfns import Dfn, is_valid, load, load_flat, map, to_flat, to_tree +from modflow_devtools.dfns.parse import parse_dfn +from modflow_devtools.dfns.schema.block import block_sort_key +from modflow_devtools.misc import drop_none_or_empty + +# mypy: ignore-errors + + +def convert(inpath: PathLike, outdir: PathLike, schema_version: str = "2") -> None: + """ + Convert DFN files in `inpath` to TOML files in `outdir`. + By default, convert the definitions to schema version 2. + """ + inpath = Path(inpath).expanduser().absolute() + outdir = Path(outdir).expanduser().absolute() + outdir.mkdir(exist_ok=True, parents=True) + + if inpath.is_file(): + if inpath.name == "common.dfn": + raise ValueError("Cannot convert common.dfn as a standalone file") + + common_path = inpath.parent / "common.dfn" + if common_path.exists(): + with common_path.open() as f: + common, _ = parse_dfn(f) + else: + common = {} + + with inpath.open() as f: + dfn = load(f, name=inpath.stem, common=common, format="dfn") + + dfn = map(dfn, schema_version=schema_version) + _convert(dfn, outdir / f"{inpath.stem}.toml") + else: + dfns = { + name: map(dfn, schema_version=schema_version) for name, dfn in load_flat(inpath).items() + } + tree = to_tree(dfns) + flat = to_flat(tree) + for dfn_name, dfn in flat.items(): + _convert(dfn, outdir / f"{dfn_name}.toml") + + +def _convert(dfn: Dfn, outpath: Path) -> None: + with Path.open(outpath, "wb") as f: + # TODO if we start using c/attrs, swap out + # all this for a custom unstructuring hook + dfn_dict = asdict(dfn) + dfn_dict["schema_version"] = str(dfn_dict["schema_version"]) + if blocks := dfn_dict.pop("blocks", None): + for block_name, block_fields in blocks.items(): + if block_name not in dfn_dict: + dfn_dict[block_name] = {} + for field_name, field_data in block_fields.items(): + dfn_dict[block_name][field_name] = field_data + + tomli.dump( + dict( + sorted( + remap(dfn_dict, visit=drop_none_or_empty).items(), + key=block_sort_key, + ) + ), + f, + ) + + +if __name__ == "__main__": + """ + Convert DFN files in the original format and schema version 1 + to TOML files, by default also converting to schema version 2. + """ + + parser = argparse.ArgumentParser( + description="Convert DFN files to TOML.", + epilog=textwrap.dedent( + """\ +Convert DFN files in the original format and schema version 1 +to TOML files, by default also converting to schema version 2. +""" + ), + ) + parser.add_argument( + "--indir", + "-i", + type=str, + help="Directory containing DFN files, or a single DFN file.", + ) + parser.add_argument( + "--outdir", + "-o", + help="Output directory.", + ) + parser.add_argument( + "--schema-version", + "-s", + type=str, + default="2", + help="Schema version to convert to.", + ) + parser.add_argument( + "--validate", + "-v", + action="store_true", + help="Validate DFN files without converting them.", + ) + args = parser.parse_args() + + if args.validate: + if not is_valid(args.indir): + sys.exit(1) + else: + convert(args.indir, args.outdir, args.schema_version) diff --git a/modflow_devtools/dfns/registry.py b/modflow_devtools/dfns/registry.py index 462d655..51a6dfc 100644 --- a/modflow_devtools/dfns/registry.py +++ b/modflow_devtools/dfns/registry.py @@ -734,7 +734,7 @@ def get_sync_status(source: str = "modflow6") -> dict[str, bool]: def get_registry( source: str = "modflow6", ref: str = "develop", - auto_sync: bool = True, + auto_sync: bool = False, path: str | PathLike | None = None, ) -> DfnRegistry: """ @@ -747,8 +747,9 @@ def get_registry( ref : str, optional Git ref (branch, tag, or commit hash). Default is "develop". auto_sync : bool, optional - If True and registry is not cached, automatically sync. Default is True. - Can be disabled via MODFLOW_DEVTOOLS_NO_AUTO_SYNC environment variable. + If True and registry is not cached, automatically sync. Default is False + (opt-in while experimental). Can be enabled via MODFLOW_DEVTOOLS_AUTO_SYNC + environment variable (set to "1", "true", or "yes"). Ignored when path is provided. path : str or PathLike, optional Path to a local directory containing DFN files. If provided, returns @@ -775,9 +776,9 @@ def get_registry( if path is not None: return LocalDfnRegistry(path=Path(path), source=source, ref=ref) - # Check for auto-sync opt-out - if os.environ.get("MODFLOW_DEVTOOLS_NO_AUTO_SYNC", "").lower() in ("1", "true", "yes"): - auto_sync = False + # Check for auto-sync opt-in (experimental - off by default) + if os.environ.get("MODFLOW_DEVTOOLS_AUTO_SYNC", "").lower() in ("1", "true", "yes"): + auto_sync = True registry = RemoteDfnRegistry(source=source, ref=ref) diff --git a/modflow_devtools/models/__init__.py b/modflow_devtools/models/__init__.py index ca96eaa..00902c7 100644 --- a/modflow_devtools/models/__init__.py +++ b/modflow_devtools/models/__init__.py @@ -1062,7 +1062,7 @@ def _load(self): Load registry data from cache. Raises an error if no cached registries are found. - Run 'python -m modflow_devtools.models sync' to populate the cache. + Run 'mf models sync' to populate the cache. """ # Try to load from cache loaded_from_cache = self._try_load_from_cache() @@ -1070,7 +1070,7 @@ def _load(self): if not loaded_from_cache: raise RuntimeError( "No model registries found in cache. " - "Run 'python -m modflow_devtools.models sync' to download registries, " + "Run 'mf models sync' to download registries, " "or use ModelSourceConfig.load().sync() programmatically." ) @@ -1322,6 +1322,9 @@ def get_default_registry(): This allows the module to import successfully even if the cache is empty, with a clear error message on first use. + Auto-sync can be enabled via MODFLOW_DEVTOOLS_AUTO_SYNC environment variable + (currently opt-in while experimental). Set to "1", "true", or "yes" to enable. + Returns ------- PoochRegistry @@ -1329,6 +1332,9 @@ def get_default_registry(): """ global _default_registry_cache if _default_registry_cache is None: + # Opt-in auto-sync (experimental - off by default) + if os.environ.get("MODFLOW_DEVTOOLS_AUTO_SYNC", "").lower() in ("1", "true", "yes"): + _try_best_effort_sync() _default_registry_cache = PoochRegistry(base_url=_DEFAULT_BASE_URL, env=_DEFAULT_ENV) return _default_registry_cache diff --git a/modflow_devtools/models/__main__.py b/modflow_devtools/models/__main__.py index 49752fa..f578631 100644 --- a/modflow_devtools/models/__main__.py +++ b/modflow_devtools/models/__main__.py @@ -108,7 +108,7 @@ def cmd_sync(args): def cmd_info(args): """Info command handler.""" # Attempt auto-sync before showing info (unless disabled) - if not os.environ.get("MODFLOW_DEVTOOLS_NO_AUTO_SYNC"): + if os.environ.get("MODFLOW_DEVTOOLS_AUTO_SYNC", "").lower() in ("1", "true", "yes"): _try_best_effort_sync() config = ModelSourceConfig.load() @@ -176,7 +176,7 @@ def cmd_info(args): def cmd_list(args): """List command handler.""" # Attempt auto-sync before listing (unless disabled) - if not os.environ.get("MODFLOW_DEVTOOLS_NO_AUTO_SYNC"): + if os.environ.get("MODFLOW_DEVTOOLS_AUTO_SYNC", "").lower() in ("1", "true", "yes"): _try_best_effort_sync() cached = _DEFAULT_CACHE.list() @@ -285,7 +285,7 @@ def cmd_clear(args): def cmd_copy(args): """Copy command handler.""" # Attempt auto-sync before copying (unless disabled) - if not os.environ.get("MODFLOW_DEVTOOLS_NO_AUTO_SYNC"): + if os.environ.get("MODFLOW_DEVTOOLS_AUTO_SYNC", "").lower() in ("1", "true", "yes"): _try_best_effort_sync() from . import copy_to diff --git a/modflow_devtools/programs/__main__.py b/modflow_devtools/programs/__main__.py index df793ad..43bf679 100644 --- a/modflow_devtools/programs/__main__.py +++ b/modflow_devtools/programs/__main__.py @@ -139,13 +139,15 @@ def cmd_info(args): def cmd_list(args): """List command handler.""" # Attempt auto-sync before listing (unless disabled) - if not os.environ.get("MODFLOW_DEVTOOLS_NO_AUTO_SYNC"): + if os.environ.get("MODFLOW_DEVTOOLS_AUTO_SYNC", "").lower() in ("1", "true", "yes"): _try_best_effort_sync() cached = _DEFAULT_CACHE.list() if not cached: - print("No cached program registries. Run 'sync' first.") + print( + "No program registries found in cache. Run 'mf programs sync' to download registries." + ) return # Apply filters @@ -193,7 +195,7 @@ def cmd_list(args): def cmd_install(args): """Install command handler.""" # Attempt auto-sync before installation (unless disabled) - if not os.environ.get("MODFLOW_DEVTOOLS_NO_AUTO_SYNC"): + if os.environ.get("MODFLOW_DEVTOOLS_AUTO_SYNC", "").lower() in ("1", "true", "yes"): _try_best_effort_sync() # Parse program@version syntax if provided diff --git a/version.txt b/version.txt index abb1658..ee672d8 100644 --- a/version.txt +++ b/version.txt @@ -1 +1 @@ -1.9.0 \ No newline at end of file +1.9.1 \ No newline at end of file