Migrate to the PowerIO parser backend for broad non-MATPOWER format support#50
Migrate to the PowerIO parser backend for broad non-MATPOWER format support#50samtalki wants to merge 6 commits into
Conversation
PowerIO is the default parser, with parser = :exapowerio as the MATPOWER and PGLib fallback. The vendored GO Challenge 3 parser is replaced by PowerIO.parse_goc3_json: bus ids come from its bus_id_by_uid table instead of regex uid parsing, and UC startup and shutdown flags come from PowerIO.goc3_add_status_flags!. Parser parity and format routing are covered by test/powerio_parser_tests.jl.
|
Added GO Challenge 3 parsing in the last few commits. In commit ce6eef9 I tried out removing the ExaPowerIO fallback because it made things a lot simpler and made the PR nearly -100 LOC net. Please feel free to cherry pick and edit as desired. |
|
Thank you for the contribution; running CI now |
|
@sshin23 thank you for your contributions with ExaModels! I will resolve these CI errors today and will ping you. |
|
@samtalki, I'm curious if there are benchmark results available |
|
@sshin23 yes, some benchmarks for the powerio Rust API are here: https://github.com/eigenergy/powerio/blob/main/benchmarks/RESULTS.md I recently found a serialization bug that is hurting the Julia side, but this will be fixed in the next patch: https://github.com/eigenergy/PowerIO.jl/tree/agent/matrix-axis-maps-jl/benchmark On a related note, we will hopefully soon have support for the Arrow C Data Interface in Julia: apache/arrow-julia#603 which one of the maintainers seems supportive of: apache/arrow-julia#594 Once that is upstreamed, PowerIO.jl will behave identically to the Rust API. please do not hesitate to let me know if you have any other thoughts |
|
Awesome! Congratulations on the very impressive results |
* Ingest mpopf loads through PowerIO.LoadSeries Replace the readdlm + update_load_data path with PowerIO.LoadSeries. The network is parsed once via parse_mp_network and feeds both the powerdata NamedTuple and the load series, so bus order aligns without positional guessing. apply_load_series! writes per-period pd/qd onto each (bus, period) entry; the matrix path is per-unit identical to before. The curve path now scales loads only, leaving fixed bus shunts gs/bs at their base value. * Consume PowerIO GOC3 static index-set builders in sc_parser Replace the moved pure-topology construction (parse_sc_data_static, the p_sdpc / energy-window / price-block / contingency-survivor blocks, and the index helpers) with calls into PowerIO's new goc3_* functions. The UC-coupled and model-shaped code stays: T_supc/T_sdpc accumulations, u_on attachment and period expansion over the AC contingency survivors, the sc_time_data assembly, and save_go3_solution. sc_parser.jl shrinks by ~625 lines; parse_sc_data output is byte-identical, so scopf.jl needs no changes. * Thread GOC3 stacked variable indices locally in sc_parser PowerIO no longer emits the model's stacked j/j_pr/j_cs/j_prcs/j_sh indices. Define the is_pr/get_j_* helpers and goc3_shutdown_power_cap here, and add a pass that threads those indices back onto PowerIO's general uid-keyed rows, reproducing the exact numbering scopf.jl consumes. * Build the multiperiod busarray immutably from the load series parse_mp_power_data now takes the LoadSeries and bakes each period's pd/qd onto the bus entries as the busarray is constructed, replacing the separate apply_load_series! mutation pass (removed). The result is identical per (bus, period). Drop the hand-written empty_stor fallback and its stray assignment: PowerIO now returns concrete storage rows, so the storarray comprehension is type-stable even when a case has no storage. * Own combined GOC3 indices in the SCOPF decorator goc3_static_data returns per-class indices only now, so thread the combined j_ac (AC lines and transformers) and n (reserve zones) onto the rows here, mirroring how the DC-line j is already stacked client-side. Drop the goc3_static_data producers_first argument and the per-parse @info that materialized the full sc_data type string. Precompute the cumulative interval times and per-device values once in goc3_shutdown_power_cap. GOC3 model data is unchanged. * Compute SCOPF transformer indices in parser * Compute reserve membership indices in parser * Tighten PowerIO parser boundary * Fix SCOPF sentinel types, load-series N guard, drop dead dep Type the reactive-capability empty sentinels' sum2_T_supc/sum2_T_sdpc fields as Float64 to match the populated branches and their zeros(...) sources, removing a branch-dependent eltype flip on capability-free cases. Drop the stray empty_data = ... assignment from the fpd/fwr/vpd/vwr array ternaries (the binding was never read). Assert N == size(data.busarray, 2) in both build_mpopf methods so a load-series period-count mismatch fails with a clear message instead of a downstream BoundsError. Remove the unused DelimitedFiles dependency. Tests: assert the multiperiod busarray element type is concrete/isbits on CPU; pin the positional lengths-tuple index contract in the GOC3 boundary test. * Pass through PowerIO lengths NamedTuple; dedup uid parsing Replace the positional 18-tuple of static index-set sizes with PowerIO's lengths NamedTuple passed through and extended via merge with the five model-owned window counts and K. scopf.jl, sc_parser.jl, and the boundary test now read lengths by name instead of positional destructure/indexing. Parsed scenario_303 output is identical (same 18 length fields, same sc_time_data). Route the twelve inline parse(Int, match(r"\d+", uid)) sites through the existing _uidnum helper, and drop the unused goc3_bus_id alias. Assert element-type concreteness across the SCOPF static arrays so a PowerIO regression that widened a row eltype surfaces on CPU without a GPU runner. * Consume PowerIO.goc3_scopf_data in one call Replace the six qualified calls into PowerIO GOC3 internals (goc3_static_data, goc3_energy_windows, goc3_price_blocks, goc3_ac_contingency_survivors, goc3_dc_contingency_flows) with a single PowerIO.goc3_scopf_data(data) and read its fields. The stacked-index threading and UC attachment are unchanged, so the parsed sc_time_data is identical (verified field-for-field on scenario_303). ExaModelsPower now depends only on exported PowerIO names. * Error clearly when only one of pd/qd is given to mpopf_model The matrix mpopf_model methods treated the pd/qd override as all-or-nothing: passing only `pd` fell through to PowerIO.LoadSeries(net, pd, nothing), which has no Nothing method and threw a MethodError. Require both matrices or neither (read both from file), with a clear ArgumentError otherwise.
|
@sshin23 thanks for your patience. We are ready for another CI run, PowerIO.jl v0.6.4 just merged into the Julia general registry: JuliaRegistries/General#160390. v0.6.4 implements type-stable summaryJust squash merged this PR in my fork into this PR's branch, which improves the integration of PowerIO.jl and ExaModelsPower.jl. I believe this improves the UX and DX of ExaModelsPower.jl and also opens the pathway to more capabilities in the future without breaking changes. architectural changes
I have set up issues to track this so you can understand what is coming:
I set this up so that there will hopefully be no breaking changes for ExaModelsPower, the new features will simply appear and you can choose to use them if you wish. extra bug fixes for ExaModelsPower.jlfixes without behavioral changes
one fix with a behavioral change #52See #52. On I think holding the shunts at their base values, as you do with the
I could be totally wrong, but I think the only alternative explanations would be:
I made both input paths hold |
2 participants
This PR routes network parsing through PowerIO while keeping ExaPowerIO available for PGLib artifact lookup.
Changes
PowerIO.parse_ac_power_datafromparse_ac_power_dataExaPowerIO.get_path(:pglib)so existing PGLib case names still workfromthrough for ambiguous formats such as PowerModels JSONValidation
PowerIO parser backend | 2107 passedFull
Pkg.test("ExaModelsPower")was not rerun in this refresh.Closes #52