1 Abstract¶

In this technote, we discuss the problem of representing missing values in the EFD.

In particular, we demonstrate that we can use the symbol NaN to represent missing values for floating-point data types and that it works across all systems involved from a CSC to the EFD client.

The symbol NaN, however, does not solve the problem entirely, it works for floating-point data types only, and there is no equivalent symbol for other data types that work as well as NaN across all systems. Currently, we don’t have a solution for representing missing values for other data types, and we discourage using, for example, a valid number like -99 as a sentinel value for integers.

That said, using NaN for floating-point types solves the problem for about 80% of the fields in Telemetry topics, and we recommend using NaN in the CSCs explicitly for missing values and to fill sparse arrays with NaN when appropriate.

We also found that the DDS has system default values for each IDL type. The default value used by DDS for floating-point types is 0. We recommend overwriting the default value to NaN in the IDL schema definition generated by the SAL, in which case DDS will automatically fill in with NaN a field that exists in the topic schema but is omitted by the CSC.

2 Introduction¶

Telescope & Site implements CSCs for about 60 different telescope subsystems. The CSCs publish data to DDS, and this data is recorded in the EFD.

Missing values can occur in different ways:

• A field is not implement. A situation that is often encountered during integration and testing is a DDS topic with a field that exists in the topic schema, but is not implemented in the CSC yet. That happened when testing the Main Telescope Camera Cable Wrap subsystem recently.

• A sensor produces no values. Another situation, which can happen at anytime, is when a sensor goes offline.

• A sensor produces invalid values. For example, the Environment subsystem has a sensor that measures snow depth. The sensor may produce invalid values if there is no snow. In this case the CSC might represent those values as missing.

In these situations, a value is designated as “missing” in the EFD, and high-level context information (e.g. from the Event logs) is usually required to know why the value is missing.

In practice, we need a representation for missing values when one or more field values in a topic are missing. In this case, there is a timestamp, and there may be other fields with valid values to record in the EFD.

In section 3   Representing missing values we discuss alternatives for representing missing values. In section 4   The data flow we review the data flow and how the different systems involved handle missing values. Finally, in section 6   Conclusion we conclude this investigation with a solution that works for most of the cases, and ensures that we can handle missing values in a convenient way when using the EFD client, the supported method for accessing EFD data.

3 Representing missing values¶

Generally, there are two approaches for representing missing values, one uses a mask to indicate the null status of a value, and the other uses a sentinel value.

In the sentinel value approach, a sentinel value indicates the missing value. It can use some data-specific convention such as -99 or use a more global convention, such as NaN (Not a Number), a symbol that is part of the IEEE 754 floating-point specification. The sentinel value may or may not be stored. See, for example, the sparse data structures in Pandas.

The two approaches have trade-offs. In particular, not all systems support masking. While a sentinel value like -99 works for all numeric types, it may cause problems because it is a valid number. The problem with NaN is that it is available for floating-point types only.

Most systems support derived data types, where it is possible to mark a member of the derived data type as optional and/or specify a default value for it. In this case, if the missing value is omitted, the system will automatically fill in a default value on behalf of the application. However, the question of which symbol to use as the default value for every primitive data type still remains.

4 The data flow¶

Different systems use different conventions for representing missing values. We have to ensure the representation we use works throughout the complete data flow:

CSCs -> DDS -> SAL Kafka -> Kafka -> InfluxDB Sink -> InfluxDB -> EFD Client

CSCs are implemented in multiple languages (C++, LabVIEW, Java, and Python) and publish records to DDS topics.

SAL Kafka and InfluxDB Sink are of particular importance because they are responsible for converting the records between the data formats used by the different systems. SAL Kafka converts the DDS records into Avro records and forwards them to Kafka. SAL Kafka is also responsible for converting the IDL schema to Avro schema and registering them with the Kafka schema registry. The InfluxDB Sink connector uses the Avro schemas to deserialize the Avro records and convert them into the InfluxDB line protocol format before writing to InfluxDB.

Finally, the EFD client reads from InfluxDB and returns a Pandas dataframe.

In summary, we need to understand how each system handles missing values and how data conversion between them is done.

struct MyTopic {
  long id; # default value is 0
  float x;  # default value is 0
  @default(NaN) float y; # default value is NaN
};

enum Color {
  GREEN,
  @default_literal RED,
  BLUE
};

{
  "type": "record",
  "name": "foo",
  "fields" : [
    {"name": "bar", "type": "float"},
    {"name": "baz", "type": "float", "default": NaN} // field baz default to NaN
  ]
}

<measurement>[,<tag_key>=<tag_value>[,<tag_key>=<tag_value>]] <field_key>=<field_value> [,<field_key>=<field_value>] [<timestamp>]

<measurement> <field_key>=<field_value>[,<field_key>=<field_value>]

%%bash
docker-compose up -d influxdb

import requests
requests.post(url="http://localhost:8086/query", params={'q':'CREATE DATABASE "mydb"'})

<Response [200]>

p = "foo bar=NaN"
r = requests.post(url="http://localhost:8086/write?db=mydb", data=p)
r.text

'{"error":"unable to parse \'foo bar=NaN\': invalid number"}\n'

p1 = "foo bar=1.0,baz=1.0"
p2 = "foo baz=2.0"
p3 = "foo bar=3.0,baz=3.0"
requests.post(url="http://localhost:8086/write?db=mydb", data=p1)
requests.post(url="http://localhost:8086/write?db=mydb", data=p2)
requests.post(url="http://localhost:8086/write?db=mydb", data=p3)

<Response [204]>

r = requests.get(url="http://localhost:8086/query", params={'q': 'SELECT * FROM "mydb"."autogen"."foo"'})
r.json()['results'][0]['series']

[{'name': 'foo',
  'columns': ['time', 'bar', 'baz'],
  'values': [['2021-03-15T17:17:41.087503933Z', 1, 1],
   ['2021-03-15T17:17:41.107504568Z', None, 2],
   ['2021-03-15T17:17:41.111964009Z', 3, 3]]}]

r = requests.get(url="http://localhost:8086/query", params={'q': 'SELECT mean(baz), mean(baz) FROM "mydb"."autogen"."foo" GROUP BY time(10ms) fill(linear)'})
r.json()['results'][0]['series']

[{'name': 'foo',
  'columns': ['time', 'mean', 'mean_1'],
  'values': [['2021-03-15T17:17:41.08Z', 1, 1],
   ['2021-03-15T17:17:41.09Z', 1.5, 1.5],
   ['2021-03-15T17:17:41.1Z', 2, 2],
   ['2021-03-15T17:17:41.11Z', 3, 3],
   ['2021-03-15T17:17:41.12Z', None, None],
   ['2021-03-15T17:17:41.13Z', None, None],
   ['2021-03-15T17:17:41.14Z', None, None]]}]

from aioinflux import InfluxDBClient
client = InfluxDBClient(db="mydb", output="dataframe")
await client.query('SELECT * FROM foo')

5 Recommendations¶

• We recommend adopting the symbol NaN to represent missing values for floating-point data types in the CSCs.

• For DDS topics with sparse arrays of floating-point data types, we also recommend filling in the missing values with NaN values.

• We should consider using the @default annotation in the IDL specification to set the default values for float, double and long double fields to NaN instead of 0. That would prevent the InfluxDB Sink connector from recording a 0 to InfluxDB when it should instead skip the NaN value.

• There is no obvious symbol to represent missing values for other data types that works across all systems in our data flow. We recommend revisiting this issue only if it becomes a problem in the future and follow as much as possible what Pandas does.

6 Conclusion¶

In this investigation we demonstrate that the symbol NaN can be used to represent missing values for floating-point data types across all systems in our data flow. While this is a partial solution to the problem, we showed that it covers about 80% of the fields in Telemetry topics.

With this solution in place, a field with the value NaN in a DDS record is passed along to an Avro record in Kafka. The InfluxDB Sink connector skip NaN values before writing to InfluxDB so that they are correctly represented as “missing” in InfluxDB. In particular, we showed that when querying InfluxDB in Python, it fills the missing values with None.

Finally, in the EFD client, missing values in InfluxDB are returned back as NaN which is a convenient way of representing missing values in Pandas.