Skip to content

Process Network

This module provides functions for processing power networks and scenarios.

init_julia

Initialize Julia interface with PowerModels.jl.

Sets up Julia environment and defines AC OPF/PF/DCPF entrypoints.

Solver output is governed by a single :class:SolverOutputConfig. Pass output directly, or rely on the legacy solver_log_dir / print_level arguments, which are folded into an equivalent config.

Parameters:

Name Type Description Default
max_iter int

Maximum number of iterations for AC OPF solver.

required
solver_log_dir str

Legacy. Enables solver logging to files under this directory when output is None.

None
dc_max_iter Optional[int]

Maximum number of iterations for DC OPF solver (default 1000).

None
print_level Optional[int]

Legacy. Explicit Ipopt print level override.

None
output Optional[SolverOutputConfig]

Solver-output config; takes precedence over the legacy args.

None
opf_formulation str

AC OPF coordinate formulation. "polar" preserves the historical formulation; "rectangular" uses ACR coordinates with starts from the current network state and is often faster.

'polar'
coinhsl Optional[Dict[str, Any]]

Optional mapping with enabled, linear_solver, and hsllib. Applies to all Ipopt solves; disabled by default.

None

Returns:

Type Description
Any

Julia interface object for running power flow calculations.

Raises:

Type Description
ValueError

If the OPF formulation or CoinHSL settings are invalid.

RuntimeError

If Julia initialization fails.

Source code in gridfm_datakit/process/process_network.py
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
def init_julia(
    max_iter: int,
    solver_log_dir: str = None,
    dc_max_iter: Optional[int] = None,
    print_level: Optional[int] = None,
    output: Optional[SolverOutputConfig] = None,
    opf_formulation: str = "polar",
    coinhsl: Optional[Dict[str, Any]] = None,
) -> Any:
    """Initialize Julia interface with PowerModels.jl.

    Sets up Julia environment and defines AC OPF/PF/DCPF entrypoints.

    Solver output is governed by a single :class:`SolverOutputConfig`. Pass
    ``output`` directly, or rely on the legacy ``solver_log_dir`` /
    ``print_level`` arguments, which are folded into an equivalent config.

    Args:
        max_iter: Maximum number of iterations for AC OPF solver.
        solver_log_dir: Legacy. Enables solver logging to files under this
            directory when ``output`` is None.
        dc_max_iter: Maximum number of iterations for DC OPF solver (default 1000).
        print_level: Legacy. Explicit Ipopt print level override.
        output: Solver-output config; takes precedence over the legacy args.
        opf_formulation: AC OPF coordinate formulation. ``"polar"`` preserves
            the historical formulation; ``"rectangular"`` uses ACR coordinates
            with starts from the current network state and is often faster.
        coinhsl: Optional mapping with ``enabled``, ``linear_solver``, and
            ``hsllib``. Applies to all Ipopt solves; disabled by default.

    Returns:
        Julia interface object for running power flow calculations.

    Raises:
        ValueError: If the OPF formulation or CoinHSL settings are invalid.
        RuntimeError: If Julia initialization fails.
    """
    if opf_formulation not in ("polar", "rectangular"):
        raise ValueError(
            "opf_formulation must be 'polar' or 'rectangular', "
            f"got {opf_formulation!r}",
        )

    hsl = CoinHSLConfig.model_validate({} if coinhsl is None else coinhsl)
    coinhsl_options = ""
    if hsl.enabled:
        # JSON quoting escapes paths; additionally escape Julia interpolation.
        library_literal = json.dumps(hsl.hsllib, ensure_ascii=False).replace("$", "\\$")
        coinhsl_options = (
            f'"linear_solver" => "{hsl.linear_solver}",\n"hsllib" => {library_literal},'
        )

    if output is None:
        output = SolverOutputConfig.from_settings(
            log_dir=solver_log_dir,
            enable_solver_logs=solver_log_dir is not None,
        )
    # An explicit legacy print_level still wins over the verbosity-derived one.
    ipopt_print_level = output.ipopt_print_level if print_level is None else print_level
    # Ensure spawned workers use the same Julia project so package resolution is consistent.
    julia_project = None
    try:
        from juliapkg.state import STATE

        julia_project = STATE.get("project")
        if julia_project:
            os.environ.setdefault("JULIA_PROJECT", julia_project)
    except Exception:
        julia_project = None

    from juliacall import Main as jl

    # Importing juliacall resolves every packaged juliapkg.json against
    # JULIA_PROJECT before exposing Main. Running Pkg.instantiate again here is
    # redundant and serializes concurrent workers on the project lock.

    # Route every sub-system's output through one process-wide policy. The
    # solver entrypoints below are plain aliases to their cores; capture happens
    # in Python around each call (see process.solvers), not baked into Julia.
    router = build_router(output)
    set_active_router(router)

    print_level = ipopt_print_level

    try:
        # If dc_max_iter not provided, use 1000
        dc_iter = 1000 if dc_max_iter is None else dc_max_iter
        # Memento.config! sets the root logger; PowerModels.silence() is what
        # actually quiets PowerModels' own Info/Warn (see SolverOutputConfig).
        memento_level = output.memento_level
        silence_pm = "PowerModels.silence()" if output.silence_powermodels else ""
        # Base imports and logging config in Julia
        try:
            jl.seval(f"""
            using PowerModels
            using Ipopt
            using Memento
            Memento.config!("{memento_level}")
            {silence_pm}
            """)
        except Exception as e:
            msg = str(e)
            missing_pm = "Package PowerModels not found" in msg
            missing_ipopt = "Package Ipopt not found" in msg
            missing_memento = "Package Memento not found" in msg
            if missing_pm or missing_ipopt or missing_memento:
                jl.seval(f"""
                using Pkg
                Pkg.add("Ipopt")
                Pkg.add("PowerModels")
                Pkg.add("Memento")
                using PowerModels
                using Ipopt
                using Memento
                Memento.config!("{memento_level}")
                {silence_pm}
                """)
            else:
                raise

        # ----- AC-OPF cores -----
        # ACR is opt-in: it is generally faster on the benchmarked large cases,
        # but AC OPF is nonconvex and changing coordinates can change which local
        # optimum Ipopt reaches.  Keep the historical ACP path as the default.
        jl.seval(f'global _GFM_OPF_FORMULATION = "{opf_formulation}"')
        jl.seval(
            f"""
        function _gfm_set_acr_starts!(data)
            for (_, bus) in data["bus"]
                vm = bus["vm"]
                va = bus["va"]
                bus["vr_start"] = vm * cos(va)
                bus["vi_start"] = vm * sin(va)
            end
            for (_, gen) in data["gen"]
                gen["pg_start"] = gen["pg"]
                gen["qg_start"] = gen["qg"]
            end
            return data
        end

        function _gfm_set_acp_starts!(data)
            for (_, bus) in data["bus"]
                bus["vm_start"] = bus["vm"]
                bus["va_start"] = bus["va"]
            end
            for (_, gen) in data["gen"]
                gen["pg_start"] = gen["pg"]
                gen["qg_start"] = gen["qg"]
            end
            return data
        end

        function _gfm_clear_acr_starts!(data)
            for (_, bus) in data["bus"]
                delete!(bus, "vr_start")
                delete!(bus, "vi_start")
            end
            for (_, gen) in data["gen"]
                delete!(gen, "pg_start")
                delete!(gen, "qg_start")
            end
            return data
        end

        function _gfm_clear_acp_starts!(data)
            for (_, bus) in data["bus"]
                delete!(bus, "vm_start")
                delete!(bus, "va_start")
            end
            for (_, gen) in data["gen"]
                delete!(gen, "pg_start")
                delete!(gen, "qg_start")
            end
            return data
        end

        function _run_opf_polar_core(case_or_data)
            start_time = time()
            result = solve_ac_opf(
                case_or_data,
                optimizer_with_attributes(
                    Ipopt.Optimizer,
                    "tol" => 1e-6,
                    "print_level" => {print_level},
                    "max_iter" => {max_iter},
                    {coinhsl_options}
                ),
            )
            result["runtime"] = time() - start_time
            result["solution"]["pf"] = false
            return result
        end

        function _run_opf_rectangular_core(case_or_data)
            start_time = time()
            data = case_or_data isa AbstractString ?
                PowerModels.parse_file(case_or_data) : case_or_data
            _gfm_set_acr_starts!(data)
            result = nothing
            try
                result = solve_opf(
                    data,
                    ACRPowerModel,
                    optimizer_with_attributes(
                        Ipopt.Optimizer,
                        "tol" => 1e-6,
                        "print_level" => {print_level},
                        "max_iter" => {max_iter},
                        {coinhsl_options}
                    );
                    solution_processors=[sol_data_model!],
                )
            finally
                _gfm_clear_acr_starts!(data)
            end
            result["runtime"] = time() - start_time
            result["solution"]["pf"] = false
            return result
        end

        function _run_opf_core(case_or_data)
            if _GFM_OPF_FORMULATION == "rectangular"
                return _run_opf_rectangular_core(case_or_data)
            end
            return _run_opf_polar_core(case_or_data)
        end
        """,
        )

        # Output routing is handled in Python (fd-level capture around the call),
        # so the entrypoint is a plain alias to its core.
        jl.seval("const run_opf = _run_opf_core")

        # ----- DC-OPF core -----
        jl.seval(
            """
        function _run_dcopf_core(case_file)
            start_time = time()  # record start (seconds since epoch)
            result = solve_dc_opf(
                case_file,
                optimizer_with_attributes(
                    Ipopt.Optimizer,
                    "tol" => 1e-6,
                    "print_level" => {},
                    "max_iter" => {},
                    {coinhsl_options}
                ),
            )
            end_time = time()  # record end time
            result["runtime"] = end_time - start_time  # elapsed seconds
            result["solution"]["pf"] = false
            return result
        end
        """.format(print_level, dc_iter, coinhsl_options=coinhsl_options),
        )

        jl.seval("const run_dcopf = _run_dcopf_core")

        # ----- Fast PF (direct computation) -----
        # *_data variants take a PowerModels data dict (built by _gfm_state);
        # the file entrypoints wrap them for warmups and direct file solves.
        # The _data variants mutate their argument (update_data!). The next
        # _gfm_state call fully resets the worker-local buffer before reuse.
        jl.seval("""
        function run_pf_fast_data(network)
            _gfm_set_acp_starts!(network)
            result = nothing
            try
                result = compute_ac_pf(network)
            finally
                _gfm_clear_acp_starts!(network)
            end

            if result["termination_status"] == false
                return result
            end

            update_data!(network, result["solution"])
            flows = calc_branch_flow_ac(network)

            result["solution"]["branch"] = flows["branch"]
            result["solution"]["pf"] = true
            return result
        end
        run_pf_fast(case_file) = run_pf_fast_data(PowerModels.parse_file(case_file))
        """)

        # ----- Fast DC-PF (direct computation) -----
        jl.seval("""
        function run_dcpf_fast_data(network)
            result = compute_dc_pf(network)

            if result["termination_status"] == false
                return result
            end

            update_data!(network, result["solution"])
            flows = calc_branch_flow_dc(network)

            result["solution"]["branch"] = flows["branch"]
            result["solution"]["pf"] = true
            return result
        end
        run_dcpf_fast(case_file) = run_dcpf_fast_data(PowerModels.parse_file(case_file))
        """)

        # ----- AC-PF core -----
        jl.seval(
            """
        function run_pf_data(network)
            _gfm_set_acp_starts!(network)
            result = nothing
            try
                result = solve_ac_pf(
                    network,
                    optimizer_with_attributes(
                        Ipopt.Optimizer,
                        "tol" => 1e-6,
                        "print_level" => {},
                        "max_iter" => {},
                        {coinhsl_options}
                    ),
                )
            finally
                _gfm_clear_acp_starts!(network)
            end

            if string(result["termination_status"]) != "LOCALLY_SOLVED"
                return result
            end

            update_data!(network, result["solution"])
            flows = calc_branch_flow_ac(network)

            result["solution"]["branch"] = flows["branch"]
            result["solution"]["pf"] = true
            return result
        end
        _run_pf_core(case_file) = run_pf_data(PowerModels.parse_file(case_file))
        """.format(print_level, max_iter, coinhsl_options=coinhsl_options),
        )

        jl.seval("const run_pf = _run_pf_core")

        # ----- DC-PF core -----
        jl.seval(
            """
        function run_dcpf_data(network)
            result = solve_dc_pf(
                network,
                optimizer_with_attributes(
                    Ipopt.Optimizer,
                    "tol" => 1e-6,
                    "print_level" => {},
                    "max_iter" => {},
                    {coinhsl_options}
                ),
            )

            if string(result["termination_status"]) != "LOCALLY_SOLVED"
                return result
            end

            update_data!(network, result["solution"])
            flows = calc_branch_flow_dc(network)

            result["solution"]["branch"] = flows["branch"]
            result["solution"]["pf"] = true
            return result
        end
        _run_dcpf_core(case_file) = run_dcpf_data(PowerModels.parse_file(case_file))
        """.format(print_level, dc_iter, coinhsl_options=coinhsl_options),
        )

        jl.seval("const run_dcpf = _run_dcpf_core")

        # ----- In-memory data path -----
        # Parse the MATPOWER case once per process (_gfm_init_base), then per
        # solve push only the fields the pipeline mutates (_gfm_state). The
        # transforms mirror PowerModels.parse_file bit-for-bit: make_per_unit!
        # (÷baseMVA, deg2rad on va), _rescale_cost_model! (cost[k]·mva^(n-k))
        # followed by _simplify_cost_terms! (leading zero coefficients
        # trimmed). Every other step of correct_network_data! only reads or
        # mutates fields the pipeline never changes between solves, so the
        # base parse covers them (verified in tests/test_pm_data_path.py).
        # A zero-pd load component is materialized for every bus so any load
        # scenario can be applied without re-parsing.
        jl.seval("""
        function _gfm_init_base(case_file)
            data = PowerModels.parse_file(case_file)
            bus_load = Dict{Int,String}()
            for (k, l) in data["load"]
                bus_load[l["load_bus"]] = k
            end
            next = isempty(data["load"]) ? 1 : maximum(parse(Int, k) for k in keys(data["load"])) + 1
            for (_, b) in data["bus"]
                bi = b["index"]
                if !haskey(bus_load, bi)
                    data["load"][string(next)] = Dict{String,Any}(
                        "source_id" => Any["bus", bi], "load_bus" => bi,
                        "status" => 1, "pd" => 0.0, "qd" => 0.0, "index" => next)
                    bus_load[bi] = string(next)
                    next += 1
                end
            end
            global _GFM_BASE = data
            global _GFM_WORK = deepcopy(data)
            global _GFM_BUS_LOAD = bus_load
            return nothing
        end

        function _gfm_state(bus_ids, bus_type, pd, qd, vm, va_deg,
                            pg, qg, vg, gen_status, cost,
                            br_status, br_r, br_x, br_b)
            # One worker executes solves serially, so reset and reuse a single
            # work dictionary instead of deep-copying the full parsed network
            # for every PF/OPF call.
            d = _GFM_WORK
            mva = d["baseMVA"]
            for r in eachindex(bus_ids)
                bi = Int(bus_ids[r])
                b = d["bus"][string(bi)]
                b["bus_type"] = Int(bus_type[r])
                b["vm"] = Float64(vm[r])
                b["va"] = deg2rad(Float64(va_deg[r]))
                l = d["load"][_GFM_BUS_LOAD[bi]]
                l["pd"] = Float64(pd[r]) / mva
                l["qd"] = Float64(qd[r]) / mva
            end
            ncost = size(cost, 2)
            for i in axes(cost, 1)
                g = d["gen"][string(i)]
                g["pg"] = Float64(pg[i]) / mva
                g["qg"] = Float64(qg[i]) / mva
                g["vg"] = Float64(vg[i])
                g["gen_status"] = Int(gen_status[i])
                c = [Float64(cost[i, k]) * Float64(mva)^(ncost - k) for k in 1:ncost]
                while !isempty(c) && c[1] == 0.0
                    popfirst!(c)
                end
                g["cost"] = c
                g["ncost"] = length(c)
            end
            for i in eachindex(br_status)
                br = d["branch"][string(i)]
                br["br_status"] = Int(br_status[i])
                br["br_r"] = Float64(br_r[i])
                br["br_x"] = Float64(br_x[i])
                br["b_fr"] = Float64(br_b[i]) / 2
                br["b_to"] = Float64(br_b[i]) / 2
            end
            return d
        end

        function _gfm_pack(result, branch_ids, gen_ids, bus_ids)
            sol = result["solution"]
            B = fill(NaN, length(branch_ids), 4)
            if haskey(sol, "branch")
                sb = sol["branch"]
                for (r, i) in enumerate(branch_ids)
                    br = sb[string(Int(i))]
                    B[r, 1] = br["pf"]
                    B[r, 2] = get(br, "qf", NaN)
                    B[r, 3] = br["pt"]
                    B[r, 4] = get(br, "qt", NaN)
                end
            end
            G = fill(NaN, length(gen_ids), 2)
            if haskey(sol, "gen")
                sg = sol["gen"]
                for (r, i) in enumerate(gen_ids)
                    g = sg[string(Int(i))]
                    G[r, 1] = g["pg"]
                    G[r, 2] = get(g, "qg", NaN)
                end
            end
            V = fill(NaN, length(bus_ids), 2)
            if haskey(sol, "bus")
                sbus = sol["bus"]
                for (r, i) in enumerate(bus_ids)
                    b = sbus[string(Int(i))]
                    V[r, 1] = get(b, "vm", NaN)
                    V[r, 2] = b["va"]
                end
            end
            return B, G, V
        end
        """)

        # Warm start all functions on a dummy case. Ipopt prints its one-time
        # license banner here at the C level; each warm-up runs inside its
        # channel's capture, so the banner lands wherever that channel points
        # (a file, or /dev/null when silent) instead of leaking to the console.
        dummy_case_file = str(
            resources.files("gridfm_datakit.process").joinpath("dummy.m"),
        )
        if output.to_console:
            print("\n ======= warm starting Julia interface =======\n", flush=True)

        # (channel name, Julia entrypoint). Fast paths share their channel with
        # the optimizer-based one; they warm up under the same capture.
        warmups = [
            ("opf", jl.run_opf),
            ("dcopf", jl.run_dcopf),
            ("pf", jl.run_pf_fast),
            ("pf", jl.run_pf),
            ("dcpf", jl.run_dcpf),
            ("dcpf", jl.run_dcpf_fast),
        ]
        for name, fn in warmups:
            channel = router.channel(name)
            channel.write_header(f" ======= warm starting {name} function =======")
            with channel.capture():
                fn(dummy_case_file)

        if output.to_console:
            print("\n ======= warm starting completed =======\n", flush=True)

    except Exception as e:
        raise RuntimeError("Error initializing Julia: {}".format(e))

    return jl

pf_preprocessing

Set variables to the results of OPF.

Copies the complete OPF operating point needed to initialize PF:

  • gen.pg / gen.qg: active and reactive generator dispatch
  • bus.vm / bus.va: bus voltage magnitude and angle
  • gen.vg: generator voltage setpoints, synced to the OPF bus voltage so gen.vg == bus.vm in the written case and PowerModels does not emit voltage-setpoint mismatch warnings.

Parameters:

Name Type Description Default
net Network

The power network to preprocess.

required
res Dict[str, Any]

OPF result dictionary containing solution data.

required

Returns:

Type Description
Network

Updated network with OPF results applied.

Source code in gridfm_datakit/process/process_network.py
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
def pf_preprocessing(net: Network, res: Dict[str, Any]) -> Network:
    """Set variables to the results of OPF.

    Copies the complete OPF operating point needed to initialize PF:

    - gen.pg / gen.qg: active and reactive generator dispatch
    - bus.vm / bus.va: bus voltage magnitude and angle
    - gen.vg: generator voltage setpoints, synced to the OPF bus voltage so
      gen.vg == bus.vm in the written case and PowerModels does not emit
      voltage-setpoint mismatch warnings.

    Args:
        net: The power network to preprocess.
        res: OPF result dictionary containing solution data.

    Returns:
        Updated network with OPF results applied.
    """
    _, gen_pq, bus_vmva = _solution_arrays(res, net)

    net.Pg_gen = gen_pq[:, 0] * net.baseMVA
    net.Qg_gen = gen_pq[:, 1] * net.baseMVA
    net.Vm = bus_vmva[:, 0]
    # PowerModels reports radians; MATPOWER stores degrees.
    net.Va = np.rad2deg(bus_vmva[:, 1])

    # Sync each in-service generator's voltage setpoint (VG) to the OPF bus
    # voltage at its terminal bus, so gen.vg == bus.vm in the written .m file.
    for gen_idx in net.idx_gens_in_service:
        bus_idx = int(net.gens[gen_idx, GEN_BUS])
        net.gens[gen_idx, VG] = bus_vmva[bus_idx, 0]

    return net

pf_post_processing

Post-process solved network results into numpy arrays for CSV export.

This function extracts power flow results and builds four arrays matching the column schemas defined in gridfm_datakit.utils.column_names:

  • Bus data with BUS_COLUMNS (+ DC_BUS_COLUMNS if include_dc_res=True)
  • Generator data with GEN_COLUMNS
  • Branch data with BRANCH_COLUMNS
  • Y-bus nonzero entries with [index1, index2, G, B]

Parameters:

Name Type Description Default
net Network

The power network to process (must have solved power flow results).

required
res Dict[str, Any]

Power flow result dictionary containing solution data.

required
include_dc_res bool

If True, include DC power flow voltage magnitude/angle (Vm_dc, Va_dc).

required

Returns:

Type Description
Dict[str, ndarray]

Dictionary containing:

Dict[str, ndarray]
  • "bus": np.ndarray with bus-level features
Dict[str, ndarray]
  • "gen": np.ndarray with generator features
Dict[str, ndarray]
  • "branch": np.ndarray with branch features and admittances
Dict[str, ndarray]
  • "Y_bus": np.ndarray with nonzero Y-bus entries
Source code in gridfm_datakit/process/process_network.py
 777
 778
 779
 780
 781
 782
 783
 784
 785
 786
 787
 788
 789
 790
 791
 792
 793
 794
 795
 796
 797
 798
 799
 800
 801
 802
 803
 804
 805
 806
 807
 808
 809
 810
 811
 812
 813
 814
 815
 816
 817
 818
 819
 820
 821
 822
 823
 824
 825
 826
 827
 828
 829
 830
 831
 832
 833
 834
 835
 836
 837
 838
 839
 840
 841
 842
 843
 844
 845
 846
 847
 848
 849
 850
 851
 852
 853
 854
 855
 856
 857
 858
 859
 860
 861
 862
 863
 864
 865
 866
 867
 868
 869
 870
 871
 872
 873
 874
 875
 876
 877
 878
 879
 880
 881
 882
 883
 884
 885
 886
 887
 888
 889
 890
 891
 892
 893
 894
 895
 896
 897
 898
 899
 900
 901
 902
 903
 904
 905
 906
 907
 908
 909
 910
 911
 912
 913
 914
 915
 916
 917
 918
 919
 920
 921
 922
 923
 924
 925
 926
 927
 928
 929
 930
 931
 932
 933
 934
 935
 936
 937
 938
 939
 940
 941
 942
 943
 944
 945
 946
 947
 948
 949
 950
 951
 952
 953
 954
 955
 956
 957
 958
 959
 960
 961
 962
 963
 964
 965
 966
 967
 968
 969
 970
 971
 972
 973
 974
 975
 976
 977
 978
 979
 980
 981
 982
 983
 984
 985
 986
 987
 988
 989
 990
 991
 992
 993
 994
 995
 996
 997
 998
 999
1000
1001
1002
1003
1004
1005
1006
1007
1008
1009
1010
1011
1012
1013
1014
1015
1016
1017
1018
1019
1020
1021
1022
1023
1024
1025
1026
1027
1028
1029
1030
1031
1032
1033
1034
1035
1036
1037
1038
1039
1040
1041
1042
1043
1044
1045
1046
1047
1048
1049
1050
1051
1052
1053
1054
def pf_post_processing(
    scenario_index: int,
    net: Network,
    res: Dict[str, Any],
    res_dc: Dict[str, Any],
    include_dc_res: bool,
) -> Dict[str, np.ndarray]:
    """Post-process solved network results into numpy arrays for CSV export.

    This function extracts power flow results and builds four arrays matching
    the column schemas defined in `gridfm_datakit.utils.column_names`:

    - Bus data with BUS_COLUMNS (+ DC_BUS_COLUMNS if include_dc_res=True)
    - Generator data with GEN_COLUMNS
    - Branch data with BRANCH_COLUMNS
    - Y-bus nonzero entries with [index1, index2, G, B]

    Args:
        net: The power network to process (must have solved power flow results).
        res: Power flow result dictionary containing solution data.
        include_dc_res: If True, include DC power flow voltage magnitude/angle (Vm_dc, Va_dc).

    Returns:
        Dictionary containing:
        - "bus": np.ndarray with bus-level features
        - "gen": np.ndarray with generator features
        - "branch": np.ndarray with branch features and admittances
        - "Y_bus": np.ndarray with nonzero Y-bus entries
    """

    # --- Edge (branch) info ---
    n_branches = net.branches.shape[0]
    n_cols = (
        len(BRANCH_COLUMNS) + len(DC_BRANCH_COLUMNS)
        if include_dc_res
        else len(BRANCH_COLUMNS)
    )
    X_branch = np.zeros((n_branches, n_cols))
    X_branch[:, 0] = scenario_index
    X_branch[:, 1] = np.arange(n_branches)
    X_branch[:, 2] = np.real(net.branches[:, F_BUS])
    X_branch[:, 3] = np.real(net.branches[:, T_BUS])

    # pf, qf, pt, qt
    # (solutions that already hold dense arrays have no per-element dicts to count:
    # _solution_arrays checks their shapes against the in-service elements instead)
    if "arrays" in res["solution"]:
        pass
    elif res["solution"]["pf"]:
        # when solving pf, the flow of all branches is computed, so the number of branches in solution should match the number of branches in network
        assert len(res["solution"]["branch"]) == n_branches, (
            "Number of branches in solution should match number of branches in network"
        )
    else:
        # when solving opf, the flow of only the in-service branches is computed, so the number of branches in solution should match the number of in-service branches in network
        assert len(res["solution"]["branch"]) == len(net.idx_branches_in_service), (
            "Number of branches in solution should match number of branches in network"
        )

    branch_flows, gen_pq, bus_vmva = _solution_arrays(res, net)
    if len(net.idx_branches_in_service) > 0:
        X_branch[net.idx_branches_in_service, 4:8] = branch_flows * net.baseMVA

    X_branch[:, 8] = net.branches[:, BR_R]
    X_branch[:, 9] = net.branches[:, BR_X]
    X_branch[:, 10] = net.branches[:, BR_B]

    # admittances
    branch_vecs = branch_vectors(net.branches, net.branches.shape[0])
    Ytt, Yff, Yft, Ytf = branch_vecs
    X_branch[:, 11] = np.real(Yff)
    X_branch[:, 12] = np.imag(Yff)
    X_branch[:, 13] = np.real(Yft)
    X_branch[:, 14] = np.imag(Yft)
    X_branch[:, 15] = np.real(Ytf)
    X_branch[:, 16] = np.imag(Ytf)
    X_branch[:, 17] = np.real(Ytt)
    X_branch[:, 18] = np.imag(Ytt)

    X_branch[:, 19] = net.branches[:, TAP]
    # assign 1 to tap = 0
    X_branch[net.branches[:, TAP] == 0, 19] = 1

    X_branch[:, 20] = net.branches[:, SHIFT]
    X_branch[:, 21] = net.branches[:, ANGMIN]
    X_branch[:, 22] = net.branches[:, ANGMAX]
    X_branch[:, 23] = net.branches[:, RATE_A]
    X_branch[:, 24] = net.branches[:, BR_STATUS]

    if include_dc_res:
        if res_dc is not None:
            dc_flows, gen_pq_dc, bus_vmva_dc = _solution_arrays(res_dc, net)
            dc_flows = dc_flows * net.baseMVA
            pf_dc = dc_flows[:, 0]
            pt_dc = dc_flows[:, 2]
            X_branch[net.idx_branches_in_service, 25] = pf_dc
            X_branch[net.idx_branches_in_service, 26] = pt_dc
        else:
            X_branch[net.idx_branches_in_service, 25] = np.nan
            X_branch[net.idx_branches_in_service, 26] = np.nan

    # --- Bus data ---
    n_buses = net.buses.shape[0]
    n_cols = (
        len(BUS_COLUMNS) + len(DC_BUS_COLUMNS) if include_dc_res else len(BUS_COLUMNS)
    )
    X_bus = np.zeros((n_buses, n_cols))

    # --- Loads ---
    X_bus[:, 0] = scenario_index
    X_bus[:, 1] = net.buses[:, BUS_I]  # bus
    X_bus[:, 2] = net.buses[:, PD]
    X_bus[:, 3] = net.buses[:, QD]

    # --- Generator injections
    assert "arrays" in res["solution"] or len(res["solution"]["gen"]) == len(
        net.idx_gens_in_service,
    ), "Number of generators in solution should match number of generators in network"
    gen_pq = gen_pq * net.baseMVA
    pg_gen = gen_pq[:, 0]
    qg_gen = gen_pq[:, 1]
    gen_bus = net.gens[net.idx_gens_in_service, GEN_BUS].astype(int)
    Pg_bus = np.bincount(gen_bus, weights=pg_gen, minlength=n_buses)
    Qg_bus = np.bincount(gen_bus, weights=qg_gen, minlength=n_buses)

    # Bus ids follow table order, so Pg_bus[i] is the generation at row i.
    bus_type_by_idx = np.zeros(n_buses)
    bus_type_by_idx[net.buses[:, BUS_I].astype(int)] = net.buses[:, BUS_TYPE]

    assert np.all(Pg_bus[bus_type_by_idx == PQ] == 0)
    assert np.all(Qg_bus[bus_type_by_idx == PQ] == 0)

    if include_dc_res:
        if res_dc is not None:
            # check if "gen" key is in res_dc["solution"]
            if "gen" in res_dc["solution"] or "arrays" in res_dc["solution"]:
                pg_gen_dc = gen_pq_dc[:, 0] * net.baseMVA
            else:
                pg_gen_dc = apply_slack_single_gen(net, pg_gen, Pg_bus, pf_dc, pt_dc)
            Pg_bus_dc = np.bincount(gen_bus, weights=pg_gen_dc, minlength=n_buses)
            assert np.all(Pg_bus_dc[bus_type_by_idx == PQ] == 0)

    # BUS_I equals the row, so this copies Pg/Qg onto the matching bus row.
    bus_row_idx = net.buses[:, BUS_I].astype(int)
    X_bus[:, 4] = Pg_bus[bus_row_idx]
    X_bus[:, 5] = Qg_bus[bus_row_idx]

    # Voltage. Extraction (_solution_arrays) raises on any missing expected
    # bus key, so together with this length check the key sets must match.
    assert "arrays" in res["solution"] or len(res["solution"]["bus"]) == n_buses, (
        "Buses in solution should match buses in network"
    )

    X_bus[:, 6] = bus_vmva[:, 0]
    va = np.rad2deg(bus_vmva[:, 1])

    # convert to range [-180, 180]
    va = (va + 180) % 360 - 180
    X_bus[:, 7] = va

    # one-hot encoding of bus type
    assert np.all(np.isin(net.buses[:, BUS_TYPE], [PQ, PV, REF])), (
        "Bus type should be PQ, PV, or REF, no disconnected buses (4)"
    )

    X_bus[np.arange(n_buses), 8 + net.buses[:, BUS_TYPE].astype(int) - 1] = (
        1  # because type is 1, 2, 3, not 0, 1, 2
    )

    # base_kv, min_vm_pu, max_vm_pu
    X_bus[:, 11] = net.buses[:, BASE_KV]
    X_bus[:, 12] = net.buses[:, VMIN]
    X_bus[:, 13] = net.buses[:, VMAX]

    X_bus[:, 14] = net.buses[:, GS] / net.baseMVA
    X_bus[:, 15] = net.buses[:, BS] / net.baseMVA

    if include_dc_res:
        if res_dc is not None:
            va = np.rad2deg(bus_vmva_dc[:, 1])
            # convert to range [-180, 180]
            va = (va + 180) % 360 - 180
            X_bus[:, 16] = va
            X_bus[:, 17] = Pg_bus_dc[bus_row_idx]
        else:
            X_bus[:, 16] = np.nan
            X_bus[:, 17] = np.nan

    # --- Generator data ---

    n_cost = net.gencosts[0, NCOST]
    assert np.all(net.gencosts[:, NCOST] == n_cost), (
        "NCOST should be the same for all generators"
    )
    n_gens = net.gens.shape[0]
    n_cols = (
        len(GEN_COLUMNS) + len(DC_GEN_COLUMNS) if include_dc_res else len(GEN_COLUMNS)
    )

    X_gen = np.zeros((n_gens, n_cols))
    X_gen[:, 0] = scenario_index
    X_gen[:, 1] = np.arange(n_gens)
    X_gen[:, 2] = net.gens[:, GEN_BUS]
    X_gen[net.idx_gens_in_service, 3] = pg_gen  # 0 if not in service
    X_gen[net.idx_gens_in_service, 4] = qg_gen  # 0 if not in service
    X_gen[:, 5] = net.gens[:, PMIN]
    X_gen[:, 6] = net.gens[:, PMAX]
    X_gen[:, 7] = net.gens[:, QMIN]
    X_gen[:, 8] = net.gens[:, QMAX]

    if n_cost == 3:  # order in .m file is c2, c1, c0
        X_gen[:, 9] = net.gencosts[:, COST + 2]
        X_gen[:, 10] = net.gencosts[:, COST + 1]
        X_gen[:, 11] = net.gencosts[:, COST]

    if n_cost == 2:  # order in .m file is c1, c0, and there is no cp2 cost
        X_gen[:, 9] = net.gencosts[:, COST + 1]
        X_gen[:, 10] = net.gencosts[:, COST]
        X_gen[:, 11] = 0  # no cp2 cost for linear cost function

    if n_cost == 1:  # order in .m file is c0, and there is no cp1 or cp2 cost
        X_gen[:, 9] = net.gencosts[:, COST]
        X_gen[:, 10] = 0  # no cp1 cost for constant cost function
        X_gen[:, 11] = 0  # no cp2 cost for constant cost function

    X_gen[net.idx_gens_in_service, 12] = 1

    # slack gen (can be any generator connected to the ref node)
    slack_gen_idx = np.where(net.gens[:, GEN_BUS] == net.ref_bus_idx)[0]
    X_gen[slack_gen_idx, 13] = 1

    if include_dc_res:
        if res_dc is not None:
            X_gen[net.idx_gens_in_service, 14] = pg_gen_dc
        else:
            X_gen[net.idx_gens_in_service, 14] = np.nan

    # --- Y-bus ---
    # lightsim2grid already built it for its power flow
    Y_bus = res["solution"].get("Ybus")
    if Y_bus is None:
        Y_bus, _, _ = makeYbus(net.baseMVA, net.buses, net.branches, branch_vecs)

    # Non-zero entries in row-major order, read straight from the CSR storage
    # (makeYbus already eliminated the explicit zeros).
    # note that Y_bus[i,j] can be != 0 even if a branch from i to j is not in service because there might be other branches connected to the same buses
    Y_bus = Y_bus.tocsr()
    Y_bus.sort_indices()
    Y_coo = Y_bus.tocoo()

    Y_bus = np.zeros((Y_coo.nnz, 5))
    Y_bus[:, 0] = scenario_index
    Y_bus[:, 1] = Y_coo.row
    Y_bus[:, 2] = Y_coo.col
    Y_bus[:, 3] = Y_coo.data.real
    Y_bus[:, 4] = Y_coo.data.imag

    # ---- runtime data ----
    n_cols = (
        len(RUNTIME_COLUMNS) + len(DC_RUNTIME_COLUMNS)
        if include_dc_res
        else len(RUNTIME_COLUMNS)
    )
    X_runtime = np.zeros((1, n_cols))
    X_runtime[0, 0] = scenario_index
    X_runtime[0, 1] = res["solve_time"]
    if include_dc_res:
        if res_dc is not None:
            X_runtime[0, 2] = res_dc["solve_time"]
        else:
            X_runtime[0, 2] = np.nan
    return {
        "bus": X_bus,
        "gen": X_gen,
        "branch": X_branch,
        "Y_bus": Y_bus,
        "runtime": X_runtime,
    }

process_scenario_opf_mode

Processes a load scenario in OPF mode

In OPF mode, perturbations are applied first, then OPF is run to get generator setpoints that account for the perturbed topology. This ensures all constraints are satisfied in the final operating point.

Parameters:

Name Type Description Default
net Network

The power network.

required
scenarios ndarray

Array of load scenarios with shape (n_loads, n_scenarios, 2).

required
scenario_index int

Index of the current scenario to process.

required
topology_generator TopologyGenerator

Generator for topology perturbations (line/transformer outages).

required
generation_generator GenerationGenerator

Generator for generation cost perturbations.

required
admittance_generator AdmittanceGenerator

Generator for line admittance perturbations.

required
local_processed_data List[ndarray]

List to accumulate processed data tuples.

required
error_log_file str

Path to error log file for recording failures.

required
include_dc_res bool

Whether to include DC power flow results in output.

required
jl Any

Julia interface object for running power flow calculations.

required
scenario_data_index Optional[int]

Optional local index into a sliced scenarios tensor. Defaults to scenario_index.

None

Returns:

Type Description
List[ndarray]

Updated list of processed data (bus, gen, branch, Y_bus arrays)

Note

Random seed is controlled by the calling context (process_scenario_chunk).

Source code in gridfm_datakit/process/process_network.py
1576
1577
1578
1579
1580
1581
1582
1583
1584
1585
1586
1587
1588
1589
1590
1591
1592
1593
1594
1595
1596
1597
1598
1599
1600
1601
1602
1603
1604
1605
1606
1607
1608
1609
1610
1611
1612
1613
1614
1615
1616
1617
1618
1619
1620
1621
1622
1623
1624
1625
1626
1627
1628
1629
1630
1631
1632
1633
1634
1635
1636
1637
1638
1639
1640
1641
1642
1643
1644
1645
1646
1647
1648
1649
1650
1651
1652
1653
1654
1655
1656
1657
1658
1659
1660
1661
1662
1663
1664
1665
1666
1667
1668
1669
1670
def process_scenario_opf_mode(
    net: Network,
    scenarios: np.ndarray,
    scenario_index: int,
    topology_generator: TopologyGenerator,
    generation_generator: GenerationGenerator,
    admittance_generator: AdmittanceGenerator,
    local_processed_data: List[np.ndarray],
    error_log_file: str,
    include_dc_res: bool,
    jl: Any,
    *,
    scenario_data_index: Optional[int] = None,
) -> List[np.ndarray]:
    """Processes a load scenario in OPF mode

    In OPF mode, perturbations are applied first, then OPF is run to get
    generator setpoints that account for the perturbed topology. This ensures
    all constraints are satisfied in the final operating point.

    Args:
        net: The power network.
        scenarios: Array of load scenarios with shape (n_loads, n_scenarios, 2).
        scenario_index: Index of the current scenario to process.
        topology_generator: Generator for topology perturbations (line/transformer outages).
        generation_generator: Generator for generation cost perturbations.
        admittance_generator: Generator for line admittance perturbations.
        local_processed_data: List to accumulate processed data tuples.
        error_log_file: Path to error log file for recording failures.
        include_dc_res: Whether to include DC power flow results in output.
        jl: Julia interface object for running power flow calculations.
        scenario_data_index: Optional local index into a sliced ``scenarios``
            tensor. Defaults to ``scenario_index``.

    Returns:
        Updated list of processed data (bus, gen, branch, Y_bus arrays)

    Note:
        Random seed is controlled by the calling context (process_scenario_chunk).
    """

    # apply the load scenario to the network
    data_index = scenario_index if scenario_data_index is None else scenario_data_index
    net.Pd = scenarios[:, data_index, 0]
    net.Qd = scenarios[:, data_index, 1]

    # Generate perturbed topologies
    perturbations = topology_generator.generate(net)

    # Apply generation perturbations
    perturbations = generation_generator.generate(perturbations)

    # Apply admittance perturbations
    perturbations = admittance_generator.generate(perturbations)

    for perturbation in (
        perturbations
    ):  # (that returns copies of the network with the topology perturbation applied)
        res_dcopf = None
        if include_dc_res:
            try:
                res_dcopf = run_dcopf(perturbation, jl)
            except Exception as e:
                with open(error_log_file, "a", encoding="utf-8") as f:
                    f.write(
                        f"Caught an exception at scenario {scenario_index} in run_dcopf function: {e}\n",
                    )
        try:
            # run OPF to get the gen set points. Here the set points account for the topology perturbation.
            res = run_opf(perturbation, jl)
        except Exception as e:
            with open(error_log_file, "a", encoding="utf-8") as f:
                f.write(
                    f"Caught an exception at scenario {scenario_index} in run_opf function: {e}\n",
                )
            continue

        # Append processed power flow data
        pf_data = pf_post_processing(
            scenario_index,
            perturbation,
            res,
            res_dcopf,
            include_dc_res,
        )
        local_processed_data.append(
            (
                pf_data["bus"],
                pf_data["gen"],
                pf_data["branch"],
                pf_data["Y_bus"],
                pf_data["runtime"],
            ),
        )
    return local_processed_data

process_scenario_pf_mode

Processes a load scenario in PF mode.

In PF mode, OPF is run first to get generator setpoints, then topology perturbations are applied. This can lead to constraint violations (overloads, voltage violations) since the setpoints are not re-optimized for the new topology.

Parameters

net: The base power network (copied internally before mutation). scenarios: Array of load scenarios with shape (n_loads, n_scenarios, 2). scenario_index: Global index of the current scenario to process and write. topology_generator: Generator for topology perturbations (line/transformer outages). generation_generator: Generator for generation cost perturbations. admittance_generator: Generator for line admittance perturbations. local_processed_data: List to accumulate processed data tuples. error_log_file: Path to error log file for recording failures. include_dc_res: Whether to include DC power flow results in output. pf_fast: Whether to use the fast AC PF solver (compute_ac_pf from PowerModels.jl). Only consulted when pf_solver='powermodel'. dcpf_fast: Whether to use the fast DC PF solver (compute_dc_pf from PowerModels.jl). Only consulted when pf_solver='powermodel'. jl: Julia interface object. Always required — even when pf_solver='powsybl' Julia is used for the OPF step that produces the generator set-points before topology perturbation. pf_solver: Which engine to use for the power flow solve after topology perturbation. Must be 'powermodel' (default), 'powsybl' or 'lightsim2grid'. OPF is always solved by PowerModels regardless of this value.

Keyword-only arguments (only required when pf_solver='powsybl')

meta: Optional dictionary containing metadata for PowSyBl processing, with keys: - pp_net: the PowSyBl network. - mapping_p2g: dictionary mapping from PowSyBl to GFM. scenario_data_index: Optional local index into a sliced scenarios tensor. Defaults to scenario_index for callers that pass the full tensor.

Returns

List[np.ndarray] Updated local_processed_data list with one tuple (bus, gen, branch, Y_bus, runtime) appended per successfully solved perturbation.

Note

Random seed is controlled by the calling context (process_scenario_chunk or generate_power_flow_data).

Source code in gridfm_datakit/process/process_network.py
1057
1058
1059
1060
1061
1062
1063
1064
1065
1066
1067
1068
1069
1070
1071
1072
1073
1074
1075
1076
1077
1078
1079
1080
1081
1082
1083
1084
1085
1086
1087
1088
1089
1090
1091
1092
1093
1094
1095
1096
1097
1098
1099
1100
1101
1102
1103
1104
1105
1106
1107
1108
1109
1110
1111
1112
1113
1114
1115
1116
1117
1118
1119
1120
1121
1122
1123
1124
1125
1126
1127
1128
1129
1130
1131
1132
1133
1134
1135
1136
1137
1138
1139
1140
1141
1142
1143
1144
1145
1146
1147
1148
1149
1150
1151
1152
1153
1154
1155
1156
1157
1158
1159
1160
1161
1162
1163
1164
1165
1166
1167
1168
1169
1170
1171
1172
1173
1174
1175
1176
1177
1178
1179
1180
1181
1182
1183
1184
1185
1186
1187
1188
1189
1190
1191
1192
1193
1194
1195
1196
1197
1198
1199
1200
1201
1202
1203
1204
1205
1206
1207
1208
1209
1210
1211
1212
1213
1214
1215
1216
1217
1218
1219
1220
1221
1222
1223
1224
1225
1226
1227
1228
1229
1230
1231
1232
1233
1234
1235
1236
1237
1238
1239
1240
1241
1242
1243
1244
1245
1246
1247
1248
1249
1250
1251
1252
1253
1254
1255
1256
1257
1258
1259
1260
1261
1262
1263
1264
1265
1266
1267
1268
1269
1270
1271
1272
1273
1274
1275
1276
1277
1278
1279
1280
1281
1282
1283
1284
1285
1286
1287
1288
1289
1290
1291
1292
1293
1294
1295
1296
1297
1298
1299
1300
1301
1302
1303
1304
1305
1306
1307
1308
1309
1310
1311
1312
1313
1314
1315
1316
1317
def process_scenario_pf_mode(
    net: Network,
    scenarios: np.ndarray,
    scenario_index: int,
    topology_generator: TopologyGenerator,
    generation_generator: GenerationGenerator,
    admittance_generator: AdmittanceGenerator,
    local_processed_data: List[np.ndarray],
    error_log_file: str,
    include_dc_res: bool,
    pf_fast: bool,
    dcpf_fast: bool,
    jl: Any,
    pf_solver: str = "powermodel",
    *,
    meta: Optional[Dict] = None,
    scenario_data_index: Optional[int] = None,
) -> List[np.ndarray]:
    """Processes a load scenario in PF mode.

    In PF mode, OPF is run first to get generator setpoints, then topology
    perturbations are applied. This can lead to constraint violations (overloads,
    voltage violations) since the setpoints are not re-optimized for the new topology.

    Parameters
    ----------
    net:
        The base power network (copied internally before mutation).
    scenarios:
        Array of load scenarios with shape ``(n_loads, n_scenarios, 2)``.
    scenario_index:
        Global index of the current scenario to process and write.
    topology_generator:
        Generator for topology perturbations (line/transformer outages).
    generation_generator:
        Generator for generation cost perturbations.
    admittance_generator:
        Generator for line admittance perturbations.
    local_processed_data:
        List to accumulate processed data tuples.
    error_log_file:
        Path to error log file for recording failures.
    include_dc_res:
        Whether to include DC power flow results in output.
    pf_fast:
        Whether to use the fast AC PF solver (``compute_ac_pf`` from
        PowerModels.jl).  Only consulted when ``pf_solver='powermodel'``.
    dcpf_fast:
        Whether to use the fast DC PF solver (``compute_dc_pf`` from
        PowerModels.jl).  Only consulted when ``pf_solver='powermodel'``.
    jl:
        Julia interface object.  Always required — even when
        ``pf_solver='powsybl'`` Julia is used for the OPF step that
        produces the generator set-points before topology perturbation.
    pf_solver:
        Which engine to use for the power flow solve after topology
        perturbation.  Must be ``'powermodel'`` (default),
        ``'powsybl'`` or ``'lightsim2grid'``.  OPF is always solved by
        PowerModels regardless of this value.

    Keyword-only arguments (only required when ``pf_solver='powsybl'``)
    -------------------------------------------------------------------
    meta:
    Optional dictionary containing metadata for PowSyBl processing, with keys:
        - pp_net: the PowSyBl network.
        - mapping_p2g: dictionary mapping from PowSyBl to GFM.
    scenario_data_index:
        Optional local index into a sliced ``scenarios`` tensor. Defaults to
        ``scenario_index`` for callers that pass the full tensor.

    Returns
    -------
    List[np.ndarray]
        Updated ``local_processed_data`` list with one tuple
        ``(bus, gen, branch, Y_bus, runtime)`` appended per successfully
        solved perturbation.

    Note
    ----
    Random seed is controlled by the calling context
    (``process_scenario_chunk`` or ``generate_power_flow_data``).
    """
    net = net.copy_for_perturbation()

    # apply the load scenario to the network
    data_index = scenario_index if scenario_data_index is None else scenario_data_index
    net.Pd = scenarios[:, data_index, 0]
    net.Qd = scenarios[:, data_index, 1]

    # Apply generation perturbations before OPF.
    perturbations = generation_generator.generate((x for x in [net]))

    # Apply admittance perturbations
    perturbations = admittance_generator.generate(perturbations)

    net = next(perturbations)

    # first run OPF to get the gen set points
    try:
        res = run_opf(net, jl)
    except Exception as e:
        with open(error_log_file, "a", encoding="utf-8") as f:
            f.write(
                f"Caught an exception at scenario {scenario_index} in run_opf function: {e}\n",
            )
        return local_processed_data

    net_pf = pf_preprocessing(net, res)

    # Generate perturbed topologies
    perturbations = topology_generator.generate(net_pf)

    if pf_solver == "powsybl":
        powsybl.check_powsybl_available()
        if meta is None or "pp_net" not in meta or "mapping_p2g" not in meta:
            raise ValueError("Network seems to not be initialized for PowSyBl solver")
        pp_net = meta["pp_net"]
        mapping_p2g = meta["mapping_p2g"]
        base_variant_id = pp_net.get_working_variant_id()
        lf_params = powsybl.get_default_lf_params()

    if pf_solver == "lightsim2grid":
        lightsim2grid.check_lightsim2grid_available()

    # to get PF points that can violate some OPF inequality constraints (to train PF solvers that can handle points outside of normal operating limits), we apply the topology perturbation after OPF.
    # The setpoints are then no longer adapted to the new topology, and might lead to e.g. abranch overload or a voltage magnitude violation once we drop an element.
    for pert_index, perturbation in enumerate(perturbations):
        if pf_solver == "powermodel":
            res_dcpf = None
            if include_dc_res:
                try:
                    res_dcpf = run_dcpf(perturbation, jl, fast=dcpf_fast)

                except Exception as e:
                    with open(error_log_file, "a", encoding="utf-8") as f:
                        f.write(
                            f"Caught an exception at scenario {scenario_index} when solving dcpf function: {e}\n",
                        )
            try:
                res = run_pf(perturbation, jl, fast=pf_fast)
            except Exception as e:
                with open(error_log_file, "a", encoding="utf-8") as f:
                    f.write(
                        f"Caught an exception at scenario {scenario_index} when solving in run_pf function: {e}\n",
                    )
                continue

        if pf_solver == "lightsim2grid":
            try:
                # kept in meta so the same LSGrid is updated in place across the
                # perturbations (and scenarios) handled by this worker
                converted = lightsim2grid.update_lightsim2grid(
                    perturbation,
                    None if meta is None else meta.get("ls_converted"),
                )
                if meta is not None:
                    meta["ls_converted"] = converted
            except Exception as e:
                with open(error_log_file, "a", encoding="utf-8") as f:
                    f.write(
                        f"Caught an exception at scenario {scenario_index} when building the lightsim2grid model: {e}\n",
                    )
                continue

            res_dcpf = None
            if include_dc_res:
                try:
                    res_dcpf = lightsim2grid.run_ls_pf(
                        converted.ls_net,
                        perturbation,
                        converted.mapping_l2g,
                        dc=True,
                        as_arrays=True,
                    )
                except Exception as e:
                    with open(error_log_file, "a", encoding="utf-8") as f:
                        f.write(
                            f"Caught an exception at scenario {scenario_index} when solving dcpf function with lightsim2grid solver: {e}\n",
                        )
            try:
                res = lightsim2grid.run_ls_pf(
                    converted.ls_net,
                    perturbation,
                    converted.mapping_l2g,
                    as_arrays=True,
                )
            except Exception as e:
                with open(error_log_file, "a", encoding="utf-8") as f:
                    f.write(
                        f"Caught an exception at scenario {scenario_index} when solving in run_pf function with lightsim2grid solver: {e}\n",
                    )
                continue

        if pf_solver == "powsybl":
            variant_id = f"scenario_{scenario_index}_perturbation_{pert_index}"
            pp_net.clone_variant(base_variant_id, variant_id)
            pp_net.set_working_variant(variant_id)
            try:
                powsybl.update_powsybl(pp_net, perturbation, mapping_p2g)

                res_dcpf = None
                if include_dc_res:
                    try:
                        start_time = time.perf_counter()
                        dcpf_metadata = powsybl.pypowsybl.loadflow.run_dc(
                            pp_net,
                            lf_params,
                        )
                        end_time = time.perf_counter()
                        solve_time = end_time - start_time
                        res_dcpf = powsybl.get_pf_res(
                            pp_net,
                            solve_time,
                            dcpf_metadata,
                            mapping_p2g,
                        )

                    except Exception as e:
                        with open(error_log_file, "a", encoding="utf-8") as f:
                            f.write(
                                f"Caught an exception at scenario {scenario_index} when solving dcpf function with PowSyBl solver: {e}\n",
                            )
                try:
                    start_time = time.perf_counter()
                    pf_metadata = powsybl.pypowsybl.loadflow.run_ac(pp_net, lf_params)
                    end_time = time.perf_counter()
                    solve_time = end_time - start_time
                    res = powsybl.get_pf_res(
                        pp_net,
                        solve_time,
                        pf_metadata,
                        mapping_p2g,
                    )
                except Exception as e:
                    with open(error_log_file, "a", encoding="utf-8") as f:
                        f.write(
                            f"Caught an exception at scenario {scenario_index} when solving in run_pf function with PowSyBl solver: {e}\n",
                        )
                    continue
            finally:
                pp_net.set_working_variant(base_variant_id)
                pp_net.remove_variant(variant_id)

        # Append processed power flow data
        pf_data = pf_post_processing(
            scenario_index,
            perturbation,
            res,
            res_dcpf,
            include_dc_res,
        )
        local_processed_data.append(
            (
                pf_data["bus"],
                pf_data["gen"],
                pf_data["branch"],
                pf_data["Y_bus"],
                pf_data["runtime"],
            ),
        )
    return local_processed_data

process_scenario_chunk

Process a chunk of scenarios for distributed processing.

This function processes multiple scenarios in a single worker process, accumulating results before returning them to the main process.

Parameters:

Name Type Description Default
mode str

Processing mode ("opf" or "pf").

required
start_idx int

Starting scenario index (inclusive).

required
end_idx int

Ending scenario index (exclusive).

required
scenarios ndarray

Array of load scenarios with shape (n_loads, n_scenarios, 2).

required
net Network

The power network.

required
progress_queue Optional[Any]

Queue for reporting progress to main process.

required
topology_generator TopologyGenerator

Generator for topology perturbations.

required
generation_generator GenerationGenerator

Generator for generation cost perturbations.

required
admittance_generator AdmittanceGenerator

Generator for line admittance perturbations.

required
error_log_path str

Path to error log file for recording failures.

required
include_dc_res bool

Whether to include DC power flow results in output.

required
pf_fast bool

Whether to use fast AC PF solver.

required
dcpf_fast bool

Whether to use fast DC PF solver.

required
solver_log_dir str

Directory for solver logs.

required
max_iter int

Maximum iterations for the solver.

required
seed int

Global random seed for reproducibility.

required
pf_solver str

PF solver to use in pf mode; one of 'powermodel', 'powsybl' or 'lightsim2grid'. OPF is always solved by PowerModels regardless of this value.

'powermodel'
meta Optional[Dict]

metadata dict; when pf_solver='powsybl', must contain 'network_path' and 'mapping_p2g'. 'pp_net' is loaded fresh per worker from 'network_path'.

None
scenario_index_offset int

Global index added to local indices when scenarios contains only the current chunk.

0
opf_formulation str

AC OPF coordinate formulation.

'polar'
coinhsl Optional[Dict[str, Any]]

Optional CoinHSL settings forwarded to Ipopt initialization.

None

Returns:

Type Description
Tuple[Union[None, Exception], Union[None, str], Optional[List[ndarray]]]

Tuple containing: - Exception object (None if successful) - Traceback string (None if successful) - List of processed data tuples (bus, gen, branch, Y_bus arrays)

Source code in gridfm_datakit/process/process_network.py
1420
1421
1422
1423
1424
1425
1426
1427
1428
1429
1430
1431
1432
1433
1434
1435
1436
1437
1438
1439
1440
1441
1442
1443
1444
1445
1446
1447
1448
1449
1450
1451
1452
1453
1454
1455
1456
1457
1458
1459
1460
1461
1462
1463
1464
1465
1466
1467
1468
1469
1470
1471
1472
1473
1474
1475
1476
1477
1478
1479
1480
1481
1482
1483
1484
1485
1486
1487
1488
1489
1490
1491
1492
1493
1494
1495
1496
1497
1498
1499
1500
1501
1502
1503
1504
1505
1506
1507
1508
1509
1510
1511
1512
1513
1514
1515
1516
1517
1518
1519
1520
1521
1522
1523
1524
1525
1526
1527
1528
1529
1530
1531
1532
1533
1534
1535
1536
1537
1538
1539
1540
1541
1542
1543
1544
1545
1546
1547
1548
1549
1550
1551
1552
1553
1554
1555
1556
1557
1558
1559
1560
1561
1562
1563
1564
1565
1566
1567
1568
1569
1570
1571
1572
1573
def process_scenario_chunk(
    mode: str,
    start_idx: int,
    end_idx: int,
    scenarios: np.ndarray,
    net: Network,
    progress_queue: Optional[Any],
    topology_generator: TopologyGenerator,
    generation_generator: GenerationGenerator,
    admittance_generator: AdmittanceGenerator,
    error_log_path: str,
    include_dc_res: bool,
    pf_fast: bool,
    dcpf_fast: bool,
    solver_log_dir: str,
    max_iter: int,
    seed: int,
    pf_solver: str = "powermodel",
    meta: Optional[Dict] = None,
    *,
    scenario_index_offset: int = 0,
    opf_formulation: str = "polar",
    coinhsl: Optional[Dict[str, Any]] = None,
) -> Tuple[
    Union[None, Exception],
    Union[None, str],
    Optional[List[np.ndarray]],
]:
    """Process a chunk of scenarios for distributed processing.

    This function processes multiple scenarios in a single worker process,
    accumulating results before returning them to the main process.

    Args:
        mode: Processing mode ("opf" or "pf").
        start_idx: Starting scenario index (inclusive).
        end_idx: Ending scenario index (exclusive).
        scenarios: Array of load scenarios with shape (n_loads, n_scenarios, 2).
        net: The power network.
        progress_queue: Queue for reporting progress to main process.
        topology_generator: Generator for topology perturbations.
        generation_generator: Generator for generation cost perturbations.
        admittance_generator: Generator for line admittance perturbations.
        error_log_path: Path to error log file for recording failures.
        include_dc_res: Whether to include DC power flow results in output.
        pf_fast: Whether to use fast AC PF solver.
        dcpf_fast: Whether to use fast DC PF solver.
        solver_log_dir: Directory for solver logs.
        max_iter: Maximum iterations for the solver.
        seed: Global random seed for reproducibility.
        pf_solver: PF solver to use in pf mode; one of 'powermodel', 'powsybl' or 'lightsim2grid'.
            OPF is always solved by PowerModels regardless of this value.
        meta: metadata dict; when pf_solver='powsybl', must contain 'network_path'
            and 'mapping_p2g'. 'pp_net' is loaded fresh per worker from 'network_path'.
        scenario_index_offset: Global index added to local indices when ``scenarios``
            contains only the current chunk.
        opf_formulation: AC OPF coordinate formulation.
        coinhsl: Optional CoinHSL settings forwarded to Ipopt initialization.

    Returns:
        Tuple containing:
            - Exception object (None if successful)
            - Traceback string (None if successful)
            - List of processed data tuples (bus, gen, branch, Y_bus arrays)
    """

    global _worker_jl
    completed_scenarios = 0
    try:
        if _worker_jl is None:
            _worker_jl = init_julia(
                max_iter,
                solver_log_dir,
                opf_formulation=opf_formulation,
                coinhsl=coinhsl,
            )
        jl = _worker_jl

        # In distributed (spawn) workers pp_net is not passed; reload it here.
        if (
            pf_solver == "powsybl"
            and meta
            and "network_path" in meta
            and "pp_net" not in meta
        ):
            import gridfm_datakit.powsybl as _powsybl

            loaded_net = _powsybl.load_net(meta["network_path"])
            meta["pp_net"] = loaded_net.pp_net

        local_processed_data = []

        global_start_idx = start_idx + scenario_index_offset

        # Derive a deterministic seed from the chunk's global offset. The shared
        # policy also keeps it distinct from the base seed used for load scenarios.
        with custom_seed(
            _DEFAULT_SEED_POLICY.distributed_seed(seed, global_start_idx),
        ):
            for scenario_data_index in range(start_idx, end_idx):
                scenario_index = scenario_data_index + scenario_index_offset
                if mode == "opf":
                    local_processed_data = process_scenario_opf_mode(
                        net,
                        scenarios,
                        scenario_index,
                        topology_generator,
                        generation_generator,
                        admittance_generator,
                        local_processed_data,
                        error_log_path,
                        include_dc_res,
                        jl,
                        scenario_data_index=scenario_data_index,
                    )
                elif mode == "pf":
                    local_processed_data = process_scenario_pf_mode(
                        net,
                        scenarios,
                        scenario_index,
                        topology_generator,
                        generation_generator,
                        admittance_generator,
                        local_processed_data,
                        error_log_path,
                        include_dc_res,
                        pf_fast,
                        dcpf_fast,
                        jl,
                        pf_solver,
                        meta=meta,
                        scenario_data_index=scenario_data_index,
                    )
                else:
                    raise ValueError(f"Invalid mode: {mode!r}")

                completed_scenarios += 1
                if progress_queue is not None:
                    progress_queue.put(1)

        return (
            None,
            None,
            local_processed_data,
        )
    except Exception as e:
        with open(error_log_path, "a", encoding="utf-8") as f:
            f.write(f"Caught an exception in process_scenario_chunk function: {e}\n")
            f.write(traceback.format_exc())
            f.write("\n")
        if progress_queue is not None:
            for _ in range(end_idx - start_idx - completed_scenarios):
                progress_queue.put(1)
        return e, traceback.format_exc(), None