SCADA Data

Simulation results are stored in ScadaData instances.

The topology of the simulated water distribution network can be accessed by the property network_topo() of a given ScadaData instance. In addition to that, the function topo_adj_matrix() returns the the adjacency matrix, and the function get_topo_edge_indices() returns the topology as edge indices.

Sensor Placements

A sensor placement is necessary for getting actual sensor readings from a ScadaData instance. Such a sensor placement can be set before the simulation is run by calling set_sensors() of a ScenarioSimulator instance, or after when post-processing the results a ScadaData instances – this becomes handy in cases where multiple sensor configurations have to be evaluated without having to re-run the simulation every time.

Note

The default, when loading an .inp file and specifying anything else, is an empty sensor config – i.e. no sensors anywhere. However, note that some networks and scenarios included in EPyT-Flow do come with a default sensor placement – please refer to the documentation for the specific network/scenario.

EPyT-Flow supports different types of sensors:

Identifier	Description
SENSOR_TYPE_NODE_PRESSURE	Pressure at a node.
SENSOR_TYPE_NODE_QUALITY	Water quality (e.g. chemical concentration, water age, etc.) at a node.
SENSOR_TYPE_NODE_DEMAND	Demand (i.e. water consumption) at a node.
SENSOR_TYPE_LINK_FLOW	Flow rate at a link/pipe.
SENSOR_TYPE_LINK_QUALITY	Water quality (e.g. chemical concentration, water age, etc.) at a link/pipe.
SENSOR_TYPE_VALVE_STATE	State of a valve.
SENSOR_TYPE_PUMP_STATE	State of a pump.
SENSOR_TYPE_PUMP_EFFICIENCY	Efficiency of a pump.
SENSOR_TYPE_PUMP_ENERGYCONSUMPTION	Energy consumption of a pump.
SENSOR_TYPE_TANK_VOLUME	Water volume in a tank.
SENSOR_TYPE_NODE_BULK_SPECIES	Bulk species concentrations at a node.
SENSOR_TYPE_LINK_BULK_SPECIES	Bulk species concentrations at a link/pipe.
SENSOR_TYPE_SURFACE_SPECIES	Surface species concentrations at a link/pipe.

Before the simulation run

Example for specifying a sensor placement BEFORE the simulation is run:

# Open/Create a new scenario based on the Hanoi network
network_config = load_hanoi()
with ScenarioSimulator(scenario_config=network_config) as sim:
    # Place pressure sensors at nodes "13", "16", "22", and "30"
    sim.set_sensors(SENSOR_TYPE_NODE_PRESSURE, sensor_locations=["13", "16", "22", "30"])

    # Place a flow sensor at link/pipe "1"
    sim.set_sensors(SENSOR_TYPE_LINK_FLOW, sensor_locations=["1"])

    # Run simulation
    # ....

Alternatively, one can use sensor type-specific functions to specify a sensor placement BEFORE the simulation is run:

Sensor type	Function for specifying sensors
Pressure	`set_pressure_sensors()`
Flow	`set_flow_sensors()`
Demand	`set_demand_sensors()`
Link quality	`set_link_quality_sensors()`
Node quality	`set_node_quality_sensors()`
Valve state	`set_valve_sensors()`
Pump state	`set_pump_state_sensors()`
Pump efficiency	`set_pump_efficiency_sensors()`
Pump energy consumption	`set_pump_energyconsumption_sensors()`
Tank water volume	`set_tank_sensors()`
Bulk species node concentrations	`set_bulk_species_node_sensors()`
Bulk species link concentrations	`set_bulk_species_link_sensors()`
Surface species concentrations	`set_surface_species_sensors()`

# Open/Create a new scenario based on the Hanoi network
network_config = load_hanoi()
with ScenarioSimulator(scenario_config=network_config) as sim:
    # Place pressure sensors at nodes "13", "16", "22", and "30"
    sim.set_pressure_sensors(sensor_locations=["13", "16", "22", "30"])

    # Place a flow sensor at link/pipe "1"
    sim.set_flow_sensors(sensor_locations=["1"])

    # Run simulation
    # ....

Besides specifying sensors manually, it is also possible to easily place sensors everywhere – e.g. placing a pressure sensors at all nodes in the network. This can be done by calling the following functions before BEFORE the simulation is run:

Sensor type	Function for specifying sensors
Pressure	`place_pressure_sensors_everywhere()`
Flow	`place_flow_sensors_everywhere()`
Demand	`place_demand_sensors_everywhere()`
Link quality	`place_link_quality_sensors_everywhere()`
Node quality	`place_node_quality_sensors_everywhere()`
Valve state	`place_valve_sensors_everywhere()`
Pump state	`place_pump_state_sensors_everywhere()`
Pump efficiency	`place_pump_efficiency_sensors_everywhere()`
Pump energy consumption	`place_pump_energyconsumption_sensors_everywhere()`
All pump quantities	`place_pump_sensors_everywhere()`
Tank water volume	`place_tank_sensors_everywhere()`
Bulk species node concentrations	`place_bulk_species_node_sensors_everywhere()`
Bulk species link concentrations	`place_bulk_species_link_sensors_everywhere()`
Surface species concentrations	`place_surface_species_sensors_everywhere()`
All quantities	`place_sensors_everywhere()`

After the simulation run

Besides specifying a sensor placement before the simulation is run, it is also possible to change the sensor configuration of a ScadaData instances if the simulation was run with frozen_sensor_config=False (default).

Example of specifying a sensor placement AFTER the simulation is run by calling change_sensor_config() of a ScadaData instance:

# Load scenario
# ...

# Run simulation
scada_data = sim.run_simulation()

# Set new sensor configuration
sensor_config = scada_data.sensor_config    # Copy current sensor configuration

sensor_config.pressure_sensors = ["13", "16", "22", "30"]   # Change/Set pressure sensors
sensor_config.flow_sensors = ["1"]     # Change/Set flow sensors

scada_data.change_sensor_config(cur_sensor_config)  # Set new sensor configuration

Accessing Sensor Readings

If a sensor placement has been specified, the final sensor readings of all sensors (as a numpy.array) can be obtained by calling get_data() of a given ScadaData instance:

# Load scenario
# ...

# Run simulation
scada_data = sim.run_simulation()

# Compute final sensor readings that are observed
observed_sensor_readings = scada_data.get_data()

Note

The function get_index_of_reading() of the sensor configuration can be used to get the index of a particular sensor in the final sensor reading numpy array.

Example for getting the pressure readings at node “5”:

# Load and run scenario simulation ...

# Compute final sensor readings that are observed
observed_sensor_readings = scada_data.get_data()

# Access pressure readings at node "5"
pressure_sensor_5_idx = scada_data.sensor_config.get_index_of_reading(
    pressure_sensor="5")
pressures_at_node_5 = observed_sensor_readings[:, pressure_sensor_5_idx]

Alternatively, one can use sensor type-specific function for retrieving the readings of all or some sensors of that type - note that the ordering of the columns (i.e. sensors) in the returned array depends on the ordering of the specified sensors:

Sensor type	Function for getting sensor readings
Pressure	`get_data_pressures()`
Flow	`get_data_flows()`
Demand	`get_data_demands()`
Node quality	`get_data_nodes_quality()`
Link quality	`get_data_links_quality()`
Valve state	`get_data_valves_state()`
Pump state	`get_data_pumps_state()`
Pump efficiency	`get_data_pumps_efficiency()`
Pump energy consumption	`get_data_pumps_energyconsumption()`
Tank water volume	`get_data_tanks_water_volume()`
Bulk species node concentration	`get_data_bulk_species_node_concentration()`
Bulk species link concentration	`get_data_bulk_species_link_concentration()`
Surface species concentration	`get_data_surface_species_concentration()`

Example for getting the pressure readings at node “5”:

# Load scenario
# ...

# Run simulation
scada_data = sim.run_simulation()

# Access pressure readings at node "5"
pressure_at_node_5 = scada_data.get_data_pressures(sensor_locations=["5"])

Connecting sensor readings to the topology of the network

Sensor readings can also be directly connected to the topology of the network, which for instance is useful when working with Graph Neural Networks (GNNs) – also refer to get_topo_edge_indices() for getting the topology of the network as edge indices (compatible with PyTorch Geometric).

For this purpose, ScadaData instances have dedicated functions for returning the sensor readings in topology consistent feature matrices and masks indicating the presence of a sensor:

Sensor type	Function for getting a topology consistent feature matrix
Pressure	`get_data_pressures_as_node_features()`
Flow	`get_data_flows_as_edge_features()`
Node quality	`get_data_nodes_quality_as_node_features()`
Link quality	`get_data_links_quality_as_edge_features()`
Surface species concentration	`get_data_surface_species_concentrations_as_edge_features()`
Bulk species node concentration	`get_data_bulk_species_concentrations_as_node_features()`
Bulk species link concentration	`get_data_bulk_species_concentrations_as_edge_features()`

For convience, ScadaData instances also have functions for retrieving all node features get_data_node_features(), and all edges features get_data_edge_features().

Plotting of sensor readings

Similar to the functions for retrieving the final sensor reading, there also exist dedicated functions for plotting the final sensor readings:

Sensor type	Plot function
Pressure	`plot_pressures()`
Flow	`plot_flows()`
Demand	`plot_demands()`
Node quality	`plot_nodes_quality()`
Link quality	`plot_links_quality()`
Valve state	`plot_valves_state()`
Pump state	`plot_pumps_state()`
Pump efficiency	`plot_pumps_efficiency()`
Pump energy consumption	`plot_pumps_energyconsumption()`
Tank water volume	`plot_tanks_water_volume()`
Bulk species node concentration	`plot_bulk_species_node_concentration()`
Bulk species link concentration	`plot_bulk_species_link_concentration()`
Surface species concentration	`plot_surface_species_concentration()`

For more advanced plotting, the function plot_timeseries_data() might be used.

Units of Measurement

The units of measurements are stored in the sensor configuration:

Units of Measurements	Attribute in the sensor configuration
Flow units	`flow_unit()`
Pressure units	`pressure_unit()`
Water quality unit	`quality_unit()`
Bulk species mass unit	`bulk_species_mass_unit()`
Surface species mass unit	`surface_species_mass_unit()`
Surface species area unit	`surface_species_area_unit()`

For a full list of supported measurement units and how they releate to each other can be found in the EPANET documentation.

The units can be changed (i.e., measurements are converted) by calling the function convert_units() of a ScadaData instances.

Example of getting and changing the measurement units:

# Running a simulation of loading a ScadaData instance
# ...

# Show current flow and pressure unit in a human-readable format
print(flowunit_to_str(scada_data.sensor_config.flow_unit))
print(pressureunit_to_str(scada_data.sensor_config.pressure_unit))

# Change flow units to gal/min and pressure units to pounds per square inch (psi) --
# note that this changes the hydraulic units to US CUSTOM
scada_data_new = scada_data.convert_units(flow_unit=EpanetConstants.EN_GPM,
                                          pressure_unit=EpanetConstants.EN_PSI)
print(flowunit_to_str(scada_data_new.sensor_config.flow_unit))
print(pressureunit_to_str(scada_data_new.sensor_config.pressure_unit))

Importing and Exporting

SCADA data can be exported and also imported if stored in a custom binary file – see Serialization for details.

Example for exporting and important ScadaData instances:

# Load Hanoi network with a default sensor configuration
network_config = load_hanoi(include_default_sensor_placement=True)
with ScenarioSimulator(scenario_config=network_config) as sim:
    # Run simulation
    scada_data = sim.run_simulation()

    # Store simulation results in a file
    scada_data.save_to_file("myHanoiResuls.epytflow_scada_data")

# ...

# Load SCADA results from file
scada_data = ScadaData.load_from_file("myHanoiResuls.epytflow_scada_data")

Note

Note that the use of the “.epytflow_scada_data” file extension is mandatory and will be appended automatically if not already present.

Export to other file formats

EPyT-Flow also supports the export of SCADA data to Numpy, .xlsx, MatLab files – see here.

Note

In these cases, the exported SCADA data CANNOT be imported again!

Example for exporting a ScadaData instance to numpy:

# Load Hanoi network with a default sensor configuration
network_config = load_hanoi(include_default_sensor_placement=True)
with ScenarioSimulator(scenario_config=network_config) as sim:
    # Run simulation
    scada_data = sim.run_simulation()

    # Export results (i.e. SCADA for the current sensor configuration) to numpy
    ScadaDataNumpyExport(f_out="myHanoiResults.npz").export(scada_data)

Importing external data

Some use cases might require loading external (real-world) SCADA data into EPyT-Flow for further analysis/processing such as calibration and state estimation tasks where the user wants to use information from both the hydraulic simulation results and sparse SCADA data to correct parameters (like pipe roughnesses) or estimate real-time system-wide pressures and flows.

External SCADA data can be loaded into EPyT-Flow by manually creating a ScadaData instance.

A hypothetical example of how to simulate a given .inp file and loading external (real-world) sensor readings into EPyT-Flow:

# Load C-Town network
with ScenarioSimulator(scenario_config=load_ctown()) as sim:
    # Place a pressure sensor at the tank "T1"
    sim.set_pressure_sensors(sensor_locations=["T1"])
    my_sensor_config = sim.sensor_config

    # Run simulation
    scada_data = sim.run_simulation()

    # Import external sensor measurements for the pressure at "T1" into a ScadaData instance
    my_measurement_time_points = np.arange(0, 3600*24, 3600)
    real_world_pressure_data = np.array([3, 2.82, 2.7, 2.62, 2.7, 2.89, 3.14, 3.26,
                                         3.4, 3.66, 3.73, 3.66, 3.73, 3.88, 4.07,
                                         4.23, 4.41, 4.44, 4.03, 4.03, 4.03, 4.03,
                                         4.03, 4.03])

    # We only have pressure data at the tank --
    # everything else is set to zero and will be ignored by ScadaData
    pressure_measurements = np.zeros((len(my_measurement_time_points),
                                      len(my_sensor_config.nodes)))
    tank_data_idx = my_sensor_config.map_node_id_to_idx("T1")
    pressure_measurements[:, tank_data_idx] = real_world_pressure_data

    # IMPORTANT: frozen_sensor_config=True because we only provide data for some specific sensors!
    my_scada_data = ScadaData(sensor_config=my_sensor_config,
                              frozen_sensor_config=True,
                              sensor_readings_time=my_measurement_time_points,
                              pressure_data_raw=pressure_measurements)

    # Show/Analyze external sensor data in EPyT-Flow
    print(my_scada_data.get_data_pressures())

    # ....