Sensor data is not just a number beside a timestamp. A trustworthy observation needs enough context to answer four questions:
- What physical property was observed, and where?
- When did that condition apply?
- Is the value valid and fresh enough for this use?
- What measurement and processing chain produced it?
If any answer is missing, a dashboard may still draw a smooth line while the underlying evidence is ambiguous. That ambiguity matters when data is used to allocate energy, diagnose equipment, trigger an alarm, calculate performance or request a control action.
This guide describes a practical data contract and commissioning method. It does not make every point suitable for billing, protection, safety functions or regulatory reporting. Those uses require the applicable equipment, approval, uncertainty, time and audit requirements.
What counts as sensor data?
The W3C Semantic Sensor Network ontology models an observation as an execution that estimates a property of a subject or feature of interest. It distinguishes the property, result, system, procedure, phenomenon time and result time rather than reducing an observation to a bare value.
The OGC SensorThings API uses a comparable model for interoperable sensor services. An Observation belongs to a Datastream and can carry a result, phenomenon time, result time, result quality, valid time and parameters describing conditions around the measurement.
You do not have to implement either information model to apply the lesson: meaning, time, quality and provenance are separate fields.
The minimum useful observation record
The exact schema depends on the system, but the following questions should have explicit answers.
| Field or relationship | Question it answers | Failure if omitted |
|---|---|---|
| Site, asset and point identity | Which physical thing and channel produced this? | Values from different sources can be merged or compared incorrectly |
| Observed property | Is this active power, cumulative energy, temperature, status or something else? | A number has no stable engineering meaning |
| Value and unit | What magnitude and unit were reported? | W, kW and MW, or °C and °F, can be silently confused |
| Phenomenon or source time | When did the measured condition apply? | Buffered old data can look current |
| Result or ingestion time | When was the result produced or received here? | Transport delay and replay cannot be diagnosed |
| Quality or state | Is the value good, uncertain, bad, stale, substituted or unavailable under the agreed model? | Invalid data looks authoritative |
| Procedure and configuration identity | Which range, scaling, firmware, calibration and calculation applied? | A later change cannot be separated from a physical change |
| Sequence or stable event identity | Has this observation already been processed? | Retries can become duplicate records or double-counted events |
Not every source can populate every field. A device without a reliable clock may need the Gateway to timestamp receipt, but that is a different statement from knowing when the physical observation occurred. Record the limitation instead of inventing precision.
Six quality boundaries in the data pipeline
1. Define the measurement before choosing the sensor
Start with the measurand: the quantity intended to be measured. “Temperature” is incomplete if the decision depends on air temperature at a named location, pipe surface temperature, immersion temperature or a calculated average across rooms.
Then specify:
- the expected and permitted range;
- required uncertainty or accuracy evidence over that range;
- response time and relevant frequency content;
- environmental and installation conditions;
- the decision, alarm or calculation the result supports; and
- the consequence of a wrong, missing or delayed value.
An accuracy label on one component does not establish the performance of the installed chain. NIST's metrological-traceability policy treats traceability as a property of a measurement result supported by a documented chain of calibrations, each contributing to measurement uncertainty. Merely having a calibrated instrument does not make every later result traceable or fit for purpose.
For electricity measurement, the high-accuracy metering guide explains why a meter class, complete-chain uncertainty and legal approval are different claims.
2. Preserve scaling, units and transformations
A raw register or analogue input may need signed decoding, word ordering, ratio application, linear scaling or a non-linear transfer function before it becomes an engineering value. Store the parameters or configuration revision that made the conversion.
For a calculated point, preserve:
- every input point identity;
- the formula and formula revision;
- units before and after conversion;
- the time-alignment and aggregation rule;
- invalid, missing and stale-input behaviour; and
- any substitution, clamping or manual override.
Do not silently replace a bad value with zero. Zero may be a valid physical result and may trigger a different operational conclusion from “unknown”.
3. Keep observation time separate from transport time
Three clocks commonly appear in one journey:
- source or phenomenon time: when the measured condition applied;
- result time: when the sensing or calculation procedure produced its result; and
- ingestion time: when a gateway, broker, historian or application received it.
Those times may match during normal operation and diverge during buffering or replay. W3C SOSA explicitly states that phenomenon time is not necessarily the same as result time. OPC UA's DataValue similarly separates source and server timestamps and couples the value with a StatusCode.
Use UTC or another unambiguous time representation, record the time zone where local civil intervals matter, and monitor clock state. A timestamp with millisecond digits is not proof of millisecond accuracy.
4. Treat transport guarantees as one layer
MQTT 5.0 defines three quality-of-service levels between an MQTT sender and receiver: at most once, at least once and exactly once. MQTT 5 also provides message-expiry and session controls.
Use the MQTT over TLS commissioning guide to keep transport protection, broker identity, client authentication, topic access and receiving-application evidence separate.
Those features do not establish end-to-end data quality. They do not prove that:
- the physical observation was correct;
- every source sample became an application message;
- the payload contains the right point, unit or timestamp;
- a second integration hop retained the same guarantee;
- the consumer stored or acted on the message once; or
- a retained value is fresh enough for the present decision.
Define stable identities, idempotent writes, retry limits, queue capacity and what the operator sees when delivery is delayed. Test disconnect and replay behaviour rather than inferring it from a protocol label.
5. Make quality and freshness visible
Quality is contextual. OPC UA uses Good, Uncertain and Bad StatusCode severities with more specific reasons. Another system may use different states or numeric quality fields. The data contract should map those meanings deliberately and retain the source state where possible.
At minimum, consumers need to distinguish:
- a valid current value;
- a valid older observation delivered late;
- a value outside its declared range;
- a source or calculation reporting bad or uncertain state;
- a substituted or manually overridden value;
- no observation for an expected interval; and
- a previously valid value that is now stale.
Freshness is an application rule, not a universal number. A monthly energy register and a fast equipment interlock have radically different limits. Display the observation time and stale state together; changing the chart's timestamp to “now” is not recovery.
6. Preserve raw evidence through aggregation
Aggregation can change meaning. A mean power over an interval, maximum demand, last-known state, count of alarms and change in a cumulative energy register need different functions.
Record:
- interval boundaries and time zone;
- whether the interval is leading or trailing;
- which observations were expected and received;
- the minimum completeness rule;
- how late values revise a closed interval;
- counter rollover and reset handling; and
- whether the result is measured, aggregated, estimated or substituted.
Never average cumulative energy registers. Use a difference between appropriate readings, with explicit reset and rollover handling. Do not sum instantaneous power values without integrating them over time.
Sampling, reporting and dashboard refresh are different
“One-minute data” can describe several different things:
- a sensor physically sampled once per minute;
- a meter that sampled rapidly but calculated a one-minute statistic;
- a device that published its latest value once per minute;
- a platform that stored or displayed a one-minute roll-up; or
- a dashboard that refreshed every minute while the underlying point changed at another rate.
Specify each rate that matters. Choose the reporting interval from the use case, not a universal rule.
| Use | Interval decision | Evidence to retain |
|---|---|---|
| Energy allocation or reporting | Match the contractual or reporting interval and preserve the cumulative register where available | Boundary, completeness, register resets and any estimation |
| Operational trend | Resolve the shortest meaningful change the operator must see without hiding it in an average | Source rate, aggregation and display rate |
| Alarm | Meet the permitted detection delay and prove what happens when values stop | Threshold, persistence, deadband, stale limit and notification delay |
| Demand management | Match the demand calculation and leave enough time for the permitted response | Window definition, update rate, forecast or threshold logic and command evidence |
| Condition monitoring | Capture the feature that indicates degradation; a simple trend may not represent waveform or spectral information | Sensor bandwidth, sampling method, feature calculation and baseline |
| Power quality, protection or safety | Use equipment and methods qualified for that exact function | Applicable standard, instrument configuration, event timing and acceptance evidence |
A faster interval increases message rate, storage, radio airtime and processing. A slower interval can hide a short event or delay an alarm. Where the source has a cumulative register, retain it alongside interval values because it can help reconcile energy after a telemetry gap; it cannot recreate every missing power sample or event.
Commissioning tests that expose bad data
A dashboard showing plausible values is not an acceptance test. Commission the full path with known identities and recorded expected outcomes.
- Identity: trace each displayed point back to the physical device, channel and asset label.
- Units and scaling: apply at least two known values where practical, including a non-zero point, and verify every conversion.
- Direction and phase: confirm polarity, import/export convention and phase association for electrical quantities.
- Time: compare source and receiving clocks, then introduce a known delay and confirm both times remain distinguishable.
- Bad state: disconnect or invalidate the source and verify the consuming application shows bad or unavailable, not zero.
- Stale state: stop updates without clearing the last value and verify it becomes stale at the agreed limit.
- Gap and recovery: interrupt the onward connection for a bounded period, observe queue or buffer state, reconnect and verify replay, ordering and completeness.
- Duplicate handling: replay a record or retry a delivery and confirm idempotent consumers do not double count it.
- Restart: restart each relevant layer in turn and confirm identity, configuration, clock state and delivery recover as specified.
- Aggregation: compare a closed interval with source observations, including a boundary, missing value and counter reset.
- Configuration change: alter an approved range or formula revision and verify the change is visible in lineage and does not rewrite prior meaning silently.
- Independent reference: compare important measurements with a suitable reference method under representative operating conditions.
The acceptance record should name the equipment, configuration, reference, test time, expected result, observed result and person or system that approved it.
Monitoring data quality in operation
Commissioning proves a known state once. Operations must detect when that state changes.
Useful service-level indicators include:
- expected observations versus observations received;
- age of the newest valid source observation;
- late, out-of-order and duplicate counts;
- bad, uncertain, substituted and out-of-range counts;
- clock offset or synchronisation state;
- queue depth and oldest queued observation;
- rejected schema, unit or identity changes; and
- reconciliation difference against a cumulative register or independent source.
Define the denominator. “99.9% data availability” is meaningless unless the expected point set, reporting schedule, exclusions, quality rule and treatment of late data are fixed.
ISO/IEC 25012, confirmed current in 2025, provides a general model for defining and evaluating the quality of structured data. A project still has to translate relevant characteristics into measurable requirements for its own context.
Designing the EpiSensor data boundary
EpiSensor devices provide field measurements, while the Gateway and EpiSensor Edge can bring physical and calculated points into local history, dashboards, rules and onward integrations. Edge can retain queued readings and forward them with their original timestamps when connectivity returns.
That architecture supports resilience, but it does not decide the data contract for the project. Agree:
- the authoritative identity, unit and timestamp for every point;
- which layer applies scaling and calculations;
- the quality and stale-state mapping;
- local-history, replay-queue and long-term retention responsibilities;
- the onward payload and delivery acknowledgement;
- alarm and control behaviour during missing or stale input; and
- who owns commissioning evidence and later configuration changes.
Continue to IoT data-storage architecture once the record and quality contract are defined. For an RTD example, use the Pt100 and Pt1000 calculator to inspect the standards curve and tolerance-class limit, then add the transmitter, wiring, input and installation effects required by the actual measurement chain.
The goal is not perfect-looking data. It is data whose identity, time, state and lineage remain honest when equipment, networks and software do what real systems eventually do: drift, restart, disconnect, retry and change.