Networks of sensors and data validation

Strategies for the validation and reconstruction of data

The validation of data consists on capturing the data with a regular temporary frequency (for example hourly) from the operational data base of the system of telecontrol, which allow to know periodically (once a day) the state of the network, or through express request of the operator of the system of telecontrol, to analyze with different levels of test all the data of the interval concerned and, from the result obtained,  to label the data as valid data, specifying  the tests passed  or non-passed, specifying which are the non-passed tests and keeping them  in the same operating data base of the system of telecontrol.

The collection of tests that has to pass each one of the data with a regulalr frequency (every 10 minutes, hourly or daily) of the state of the network, is inspired in the levels of data quality of the defined automatic meteorological stations, according to the regulation UNE 500540.

• Test 1. Errors of communication: it is a matter of knowing if the datum has arrived or not  in the expected moment (every 10 minutes, hourly or daily).
• Test 2. It is analyzed if a datum exceeds the maximum or minimum physical values, impossible to be exceeded for a coherent datum.
• Test 3. The temporary variation of the datum is analyzed, since the difference between two consecutive values should not exceed certain values of reference.
• Test 4. It is analyzed if the sensors of a same station are coherent among them. For example, if the position of a valve indicates that it is closed, water can not circulate and the sensor of flow has to indicate zero or a value very near to zero.
• Test 5. It is analyzed the space coherence of the data of the network, from a model of relation among measures; and
• Test 6. The temporary coherence of the data is analyzed and the estimation of the datum is compared from all its history to the datum to be validated.