Searching...

Matching results

    MQTT compressed format for time series

    Note: This is an obsolete legacy page, superseded by content at Using MQTT with AirVantage

    Initializing Table Of Contents...

    Introduction

    This page explains the AirVantage Time Series format for MQTT. This is a compressed format dedicated to sending time-series such as large sensor collections or GPS traces.

    This format is based on CBOR, an IETF standard RFC 7049 for Concise Binary Object Representation.

    You can optionally add an additional layer of zlib compression.

    To receive task messages from AirVantage (read, write, commands) you need to use the JSON topic: Using MQTT with AirVantage

    Representation

    This format will let you send a list of data for different timestamps. For example we want to send a car accelerometer X,Y,Z values together with the GPS coordinates (latitude and longitude) and the car speed.

    Example:

    Timestamp (ms) X Y Z Latitude Longitude Speed
    1412320402000 0 2 0 49.455177 0.537743 100
    1412320402100 0 3 0 49.455177 0.537743 98.4

    Encoding

    For basic compression we can use delta encoding , in order to send a smaller number. In place of sending the actual values for each new line, we will just send the delta compared to the previous line. If we look at our previous example, for the second timestamp, rather than sending 1412320402100, we will send the delta: 100.

    To reduce the size of floating numbers with a fixed precision (like GPS coordinates), we can apply a factor to the samples. In place of encoding the actual floating value we will encode the integer delta.

    To reduce the size of the timestamp, we will remove the last two zeros because we know we are always sampling every 100ms.

    CBOR is based on the successful JSON data model: numbers, strings, arrays and maps.

    We can represent CBOR object using JSON, but at the end it will be binary encoded on the wire.

    The payload for the message is a CBOR map with three entries:

    • “h” the list of data path, in our case [“X”,“Y”,“Z”,“Latitude”,“Longitude”,“Speed”] the first column is always timestamps so we don’t need to precise it in this list
    • “f” the list of factor to apply, in our case [0.01,1,1,1,1000000,1000000,1] the first value is the factor to apply to timestamps
    • “s” the list of samples, we will place here the delta encoded values, in a big list

    As a JSON representation:

    {
    "h" : [             "x", "y", "z", "lat",    "long",   , "speed" ],

    "f" : [ 0.01,        1,   1 ,  1,   1000000,  1000000,   1       ],

    "s" : [ 14123204020, 0,   2,   0,   49455177, 537743,    100,
            1,           0,   1,   0,   0,        0,         -1.6    ]
}

    As a binary CBOR encoded buffer:

    a361688661786179617a636c6174646c6f6e67657370656564616687fb3f847ae147ae147b0101011a000f42401a000f42400161738e1b0000000349cefdb40002001a02f2a0491a0008348f1864010001000000fbbff999999999999a

    You can easily verify your CBOR-encoded message using cbor.me

    Extra compression

    If your embedded target permits it, you can add another optional layer of compression using zlib according to the RFC1950 standard. You can use the popular zlib compression library.

    MQTT Topics

    The CBOR-encoded messages will be published on the following MQTT topic: {IMEI or SN}/messages/ts (for Time Series).

    For CBOR+zlib messages: {IMEI or SN}/messages/tsz

    Helpful links

    • cbor.io : for an introduction to CBOR and also a list of implementation in various languages (C, Java, Python, Go, ..)
    • cn-cbor A constrained node implementation of CBOR in C
    • zlib.net : the homepage of one of the most popular zlib implementation in C
    • miniz a one C file implementations of zlib
    • MQTT Errors topic
    TOP