Skip to content

Option 1: Run data gen using interactive interface

To use the interactive interface, either open scripts/interactive_interface.ipynb or copy the following into a Jupyter notebook and follow the instructions:

from gridfm_datakit.interactive import interactive_interface
interactive_interface()

Option 2: Using the command line interface

Generate Data

Run the data generation routine from the command line:

gridfm-datakit generate path/to/config.yaml

Configuration Overview

Refer to the sections Network, Load Scenarios, and Topology perturbations for a description of the configuration parameters.

To generate time-domain trajectories instead of power flow snapshots, add a dynamic: block to the config; the same generate command then runs the dynamic pipeline. See Dynamic Simulation.

To read the grid or solve the power flow with PowSyBl, see PowSyBl.

Sample configuration files are provided in scripts/config, e.g. default.yaml:

network:
  name: "case24_ieee_rts" # Name of the power grid network (without extension)
  source: "pglib" # Data source for the grid; options: pglib, file
  reader: "native" # Network reader; options: native, powsybl
  # WARNING: the following parameter is only used if source is "file"
  network_dir: "scripts/grids" # if using source "file", this is the directory containing the network file (relative to the project root)

load:
  generator: "agg_load_profile" # Name of the load generator; options: agg_load_profile, powergraph
  agg_profile: "default" # Name of the aggregated load profile
  scenarios: 10000 # Number of different load scenarios to generate
  # WARNING: the following parameters are only used if generator is "agg_load_profile"
  # if using generator "powergraph", these parameters are ignored
  sigma: 0.2 # max local noise
  change_reactive_power: true # If true, changes reactive power of loads. If False, keeps the ones from the case file
  global_range: 0.4 # Range of the global scaling factor. used to set the lower bound of the scaling factor
  max_scaling_factor: 4.0 # Max upper bound of the global scaling factor
  step_size: 0.05 # Step size when finding the upper bound of the global scaling factor
  start_scaling_factor: 0.8 # Initial value of the global scaling factor

topology_perturbation:
  type: "random" # Type of topology generator; options: n_minus_k, random, none
  # WARNING: the following parameters are only used if type is not "none"
  k: 2 # Maximum number of components to drop in each perturbation
  n_topology_variants: 10 # Number of unique perturbed topologies per scenario
  elements: [branch, gen] # elements to perturb. options: branch, gen
  outage_count_probabilities: [0.1, 0.8, 0.1] # Optional: P(N-0), P(N-1), P(N-2)

generation_perturbation:
  type: "cost_permutation" # Type of generation perturbation; options: cost_permutation, cost_perturbation, none
  # WARNING: sigma is required only if type is "cost_perturbation"
  # sigma: 1.0 # Size of range used for sampling scaling factor

admittance_perturbation:
  type: "random_perturbation" # Type of admittance perturbation; options: random_perturbation, none
  # WARNING: the following parameter is only used if type is "random_perturbation"
  sigma: 0.2 # Size of range used for sampling scaling factor

settings:
  num_processes: 10 # Number of parallel processes to use
  data_dir: "./data_out" # Directory to save generated data relative to the project root
  large_chunk_size: 1000 # Number of load scenarios processed before saving
  overwrite: true # If true, overwrites existing files, if false, appends to files
  mode: "pf" # Mode of the script; options: pf, opf. pf: power flow data where one or more operating limits – the inequality constraints defined in OPF, e.g., voltage magnitude or branch limits – may be violated. opf: generates datapoints for training OPF solvers, with cost-optimal dispatches that satisfy all operating limits (OPF-feasible)
  include_dc_res: true # If true, also stores the results of dc power flow and dc optimal power flow
  pf_solver: "powermodel" # Engine solving the power flow in pf mode; options: powermodel, powsybl, lightsim2grid (the OPF is always solved by PowerModels). powsybl requires network.reader: powsybl
  pf_fast: true # Whether to use fast PF solver by default (compute_ac_pf from powermodels.jl); if false, uses Ipopt-based PF. Some networks e.g. case10000_goc do not work with pf_fast: true
  dcpf_fast: true # Whether to use fast DC PF solver (compute_dc_pf from powermodels.jl); if false, uses optimizer-based DC PF
  enable_solver_logs: false # If true, write OPF/PF solver logs to {data_dir}/solver_log; PF fast and DCPF fast ignore logging
  max_iter: 200 # Maximum iterations for Ipopt-based solvers

Parallel generation

The CLI and generate_power_flow_data_distributed support two process backends:

settings:
  num_processes: 4
  process_backend: "multiprocessing_pool"

With process_backend: multiprocessing_pool, static generation uses a spawned multiprocessing.Pool and a Manager queue to display progress after each load scenario. The pool and its initialized solver instances are reused across large chunks. Results are saved in scenario order, regardless of worker completion order. Worker exceptions stop generation and clean up the pool and Manager.

The default, process_pool_executor, uses the existing ProcessPoolExecutor backend; with one worker it processes scenarios in the current process. The Pool backend spawns a worker even when num_processes: 1. Both backends preserve chunk boundaries, seed derivation, and solver options, including CoinHSL and opf_formulation. The setting does not change the sequential generate_power_flow_data API or the dynamic simulation backend.

Scenario timeouts are not part of this backend selection. overwrite: false retains its existing append behavior when checkpointing is disabled.

Checkpoint and resume

Enable checkpoints for distributed static generation (either process backend):

settings:
  enable_checkpointing: true
  overwrite: true
  seed: null

The run writes checkpoint.json before generating scenarios. It saves the actual random seed even when seed is null, then advances the checkpoint only after all five output datasets for a large chunk are saved. To resume an interrupted run, use the same configuration and output directory with overwrite: false. The saved seed is reused; completed large chunks are skipped. An already completed run returns its artifact paths without regenerating scenarios or starting workers.

The checkpoint records the committed parquet files and output sample count. On resume, parquet files written after the last committed checkpoint are removed and that count is restored, allowing an interrupted chunk save to be retried without duplicate samples. Missing or truncated committed files cause an error.

Keep the same scenario count, worker count, chunk size, network, perturbations, and solver settings. An explicit seed must match the saved seed; seed: null uses the saved one. You may change solver logging, switch process backends, or move the complete output directory. Keep input-file contents and the software environment unchanged for reproducible results, and use one generation process per output directory.

Existing data without a checkpoint cannot be adopted into a resumable run. Disabling checkpointing on existing checkpointed data is rejected to prevent an ordinary append from invalidating its state. Use overwrite: true for a fresh run. Checkpointing is disabled by default and applies to the CLI's static pipeline and generate_power_flow_data_distributed, not the sequential or dynamic APIs.

Choosing the right mode

The mode parameter controls how the power flow scenarios are generated and validated:

Optimal Power Flow Mode (mode: "opf")

  • Generation: Solves Optimal Power Flow (OPF) for each perturbed topology
  • Constraints: Enforces all operational constraints (voltage limits, branch loading, generator limits)
  • Example Use Case: Generating data to train a neural optimal power flow solver, or a power flow solver for cases within operating limits.
  • Performance: Slower due to OPF solving for each scenario

Power Flow Mode (mode: "pf")

  • Generation: Solves OPF for base topology, then applies topology perturbations and solves Power Flow (without changing the generator setpoints)
  • Constraints: Since the topology perturbations are performed after solving OPF, the inequality constraints of OPF (e.g. branch loading, voltage magnitude at PQ buses, generator bounds on reactive power, etc) might be violated.
  • Use Case: Training data for power flow, contingency analysis, etc
  • Performance: Faster as it avoids re-solving OPF for each perturbed scenario
  • PF Solver Choice: settings.pf_solver selects the engine: powermodel (default), powsybl, or lightsim2grid. The OPF is always solved by PowerModels. The powsybl backend requires network.reader: "powsybl". See Power flow solver.
  • PF Fast Path: With powermodel, settings.pf_fast selects the fast compute_ac_pf path when true and the Ipopt-based AC PF when false.

Data Validation

The generated data can be validated using the CLI validation command:

# Validate with default sampling (100 partitions of 200 scenarios)
gridfm-datakit validate ./data_out/case24_ieee_rts/raw

# Validate with custom number of partitions
gridfm-datakit validate ./data_out/case24_ieee_rts/raw --n-partitions 50

# Validate all partitions (slower but complete)
gridfm-datakit validate ./data_out/case24_ieee_rts/raw --n-partitions 0

This automatically detects the generation mode and runs appropriate validation checks:

  • Physical Consistency: Power balance equations, Y-bus matrix consistency
  • Data Integrity: File completeness, scenario indexing, bus indexing
  • Constraint Validation: Generator limits, branch loading (OPF mode only)
  • Power Flow Accuracy: Computed vs stored power flows

Statistics

After generating data, you can compute and visualize statistics:

# Generate statistics plots
gridfm-datakit stats ./data_out/case24_ieee_rts/raw

This generates stats_plot.png showing distributions of: - Network topology metrics (generator/branch counts) - Branch loading and overload statistics - Power balance errors

The plots help assess dataset quality and identify scenarios with constraint violations or balance errors.

To visualize individual feature distributions across buses, run:

gridfm-datakit plots ./data_out/case24_ieee_rts/raw --sn-mva 100

This command saves violin plots for each feature to feature_plots/ (or a custom directory specified with --output-dir).


Output Files

The data generation process writes the following artifacts under: {settings.data_dir}/{network.name}/raw

  • tqdm.log: Progress bar log.
  • error.log: Error messages captured during generation.
  • args.log: YAML dump of the configuration used for this run.
  • scenarios_{generator}.parquet: Load scenarios (per-element time series) produced by the selected load generator.
  • scenarios_{generator}.html: Plot of the generated load scenarios.
  • scenarios_{generator}.log: Generator-specific notes (e.g., bounds for the global scaling factor when using agg_load_profile).
  • bus_data.parquet: Bus-level features for each processed scenario (columns BUS_COLUMNS and, if settings.include_dc_res=True, also DC_BUS_COLUMNS).
  • gen_data.parquet: Generator features per scenario (columns GEN_COLUMNS).
  • branch_data.parquet: Branch features per scenario (columns BRANCH_COLUMNS).
  • y_bus_data.parquet: Nonzero Y-bus entries per scenario with columns [scenario, index1, index2, G, B].

Running gridfm-datakit stats additionally writes stats.parquet and stats_plot.png to the data directory.