Stale, missing and invalid telemetry

A dashboard that shows every reading as an ordinary number reports a failed meter as a steady load. A reading can be wrong in several ways, and each needs different handling: it can be stale, missing, invalid or estimated.

This guide is part of the data quality and delivery series.

Five states

Some protocols carry these states with each value. Map them in the collector, and keep the state with the value all the way to the final record.

OPC UA defines these codes in Part 8, tables 61 and 62. Uncertain_LastUsableValue means that whatever was updating the value has stopped. Uncertain_SubstituteValue is a value that was manually overwritten. The IEC 60870 quality guide explains the five IEC flags.

BACnet Status_Flags has four bits: IN_ALARM, FAULT, OVERRIDDEN and OUT_OF_SERVICE. FAULT is set when the Reliability property holds anything other than NO_FAULT_DETECTED. Read Reliability as well, because it gives the cause: NO_SENSOR, OVER_RANGE, UNDER_RANGE, OPEN_LOOP, SHORTED_LOOP or COMMUNICATION_FAILURE, among others. When OUT_OF_SERVICE is set, Present_Value no longer follows the physical input. Exclude it from totals.

Modbus has no per-value quality. The collector derives the state from three signals: a timeout, an exception response, and the device's own not-available codes. Exception 04 (server device failure) means the device failed while it processed the request. Exception 0B (gateway target device failed to respond) means the serial device behind a gateway did not answer. The Modbus troubleshooting guide lists the other exception codes.

Stale

A stale value is hard to detect, because it looks like a good reading. It comes from a gateway cache, a register that holds its last value, or a dashboard that keeps the last value it received. A Modbus gateway can go on answering with the last value it read after the meter behind it stops replying.

Mark a point stale when no reading has arrived within a timeout. Set the timeout at two or three reporting intervals. On a radio network, one lost message is normal; two or three in a row is a fault. With a three-interval timeout, a point that reports every 5 minutes is stale after 15 minutes. The timeout must also be longer than the collector's complete retry sequence, or a slow device goes stale on each retry.

Judge freshness by the time of the last reading received from the source, not by the last time the value changed. A steady temperature does not change, and it is still fresh.

A source that reports by exception has no fixed interval, and a quiet source looks the same as a dead one. Base the timeout on a guaranteed heartbeat instead. A Zigbee device with attribute reporting configured sends a report at least once per maximum reporting interval, even when the value has not changed, so set the timeout to two or three of those. A BACnet COV subscription expires unless the client renews it, and a lapsed subscription sends nothing. If a source has no heartbeat, also poll it at a fixed interval. The BACnet COV guide explains the checks.

Stale is a state of the live value. History has no stale state: an interval that closes with no reading in it is missing.

Missing

Show a gap as a gap:

• a chart breaks the line or shades the period;
• a table shows no value, not 0;
• a daily or monthly total states its coverage, for example 44 of 48 half-hours.

For a cumulative energy counter, a communications gap loses no energy. The next reading after the gap includes all the energy used during the gap. Only the split across the missing intervals is unknown. A gap in instantaneous power is different: nothing can recover it.

Whether a figure can be calculated from incomplete data depends on the figure. A daily kWh total from counter readings at each end of the day is exact at any coverage. A maximum demand from incomplete intervals is only a lower bound, because the peak can be in a missing interval; report it as "at least". An average or a baseload is biased when the gaps cluster. An overnight outage removes exactly the intervals that a baseload figure needs. Set a minimum coverage for each figure in the specification, and flag or withhold the figure below it. The kW vs kWh guide explains counters and intervals.

Invalid

A value is invalid when the source flags a fault, or when it is outside the range that the quantity can physically take. Examples are a relative humidity above 100 % and a power factor above 1.

Many Modbus devices return a fixed code for a value they cannot supply. The SunSpec information model uses 0x8000 for int16, 0xFFFF for uint16 and 0xFFFFFFFF for uint32 as "not implemented", and 0 for an acc32 accumulator as "not accumulated". Decoded without a check, a uint16 voltage register scaled at 0.1 V that returns 0xFFFF (65,535) reads as 6,553.5 V. Test the raw register for these codes before scaling. On a SunSpec device, an energy accumulator that reads exactly 0 is a code, not a reset. The Modbus register map guide lists common not-available codes.

Energy counters produce the largest invalid values, from three causes.

Rollover: a counter wraps to zero at its maximum. A 16-bit register that counts whole kWh wraps at 65,535 kWh, which a steady 50 kW load reaches in about 55 days. Correct a wrap by adding the register range to the difference, but only when the result is plausible for the elapsed time.

Reset or replacement: a new meter starts near zero. The difference is a large negative number. A rollover correction applied here creates a false consumption as large as the register range. Record a meter change as a closing reading on the old meter and an opening reading on the new one.

Split reads: a 32-bit counter read with two separate requests can combine words from two moments, and the total jumps by 65,536 counts and back. Read both registers in one request.

Reject any interval delta that is negative, or larger than the circuit can deliver. A 100 A three-phase circuit at 400 V delivers at most about 69 kVA, so it cannot use more than about 35 kWh in 30 minutes.

Drop invalid values from totals and calculations, show a flag, and keep the raw value in the record for diagnosis.

Estimated

A bill or a regulatory report may need a complete series. When you fill a gap:

• Use a documented method. For a counter, spread the known total across the missing intervals, evenly or in proportion to a similar day's profile, so the intervals still add up to the counter difference.
• Do not fill energy from interpolated power. An interpolated kW line has no total to preserve, and the kWh integrated from it will not match the counter.
• Mark every estimated value as estimated, in storage and on screen.
• Never overwrite a measured value with an estimate. Keep estimates separate, so they can be replaced.

Real readings can arrive after a gap was filled: a store-and-forward backlog after an outage, or a meter's own load-profile log read later. Replace the estimates with the measurements. Recalculate every total that used them, and record that the report was revised. The timestamps guide explains how to place late readings by source time.

What each state does downstream

Decide the behaviour for each consumer in advance, and write it into the specification. These defaults suit most energy monitoring projects:

A rule that turns off a chiller when the chilled water temperature is low must not act on a stale low reading. It holds the chiller off while the building warms. The safe output here is to release the chiller to its own controller. The interlock failure guide sets out how to choose safe outputs.

Test each state

At commissioning, create each state on purpose and check what every consumer shows:

1. Disconnect a device's communications link. Check that its points go stale within the timeout, that the communications alarm raises, and that history shows the lost intervals as missing, not zero.
2. Disconnect a sensor input, such as a temperature probe. Where the device can detect this, check that the value is flagged and dropped from totals.
3. Restore both. Check that the stale flag and the alarm clear, and that the first counter reading after recovery closes the gap without a demand spike in one interval.
4. Where a Modbus gateway sits between the collector and a meter, disconnect the meter from the gateway. Check whether the gateway returns exception 0B, times out, or keeps returning the last value.

The commissioning checklist includes these tests.

Data states in Edge

Edge records a timestamp with every reading and a health state for each Modbus and OPC UA device. A Modbus device goes offline after two expected polls are missed, with a five-minute minimum. The fastest enabled sensor on the device sets the expected cadence, so an hourly meter does not go offline after five minutes. An OPC UA device goes offline after five minutes without communication.

For each Modbus and OPC UA sensor, Edge also calculates the percentage of expected readings received over a rolling 15-minute window. For an interval longer than 15 minutes, it reports a count of readings instead. This figure measures delivery against the configured interval. It is not the OPC UA StatusCode, and it does not say whether a value is physically valid.

Gap fill for Modbus and OPC UA devices is off by default. When it is switched on, Edge fills gaps up to a configured maximum, 15 seconds by default, with zeros, the last value or interpolated values, and marks each generated point as synthetic. A downstream database may not keep that marker, so keep gap fill off for data used in bills, alarms or control. A CSV export from the Data page leaves missing values blank, not zero. At each platform that receives data from Edge, run the tests above to check that it applies the same rules.