🚀 Optimizing Meta Data Retrieval with ThingsDB 1.7.6

Avatar photoByAdil

We at InfraSonar query large volumes of meta data, resulting in many properties that are often empty lists, false booleans, or empty strings.
This redundant data increases payload size and network traffic unnecessarily.

ThingsDB version 1.7.6 introduces new wrap-prefix flags that solve this by allowing us to exclude properties from the wrapped output if they evaluate to a “false” state.

The Problem: Bloated JSON Payloads

Consider our Metric type definition (partial, in reality it looks slightly different): 

set_type('Metric', {
    id: '#',                // the metric Id as "id"
    key: 'str<1:>',
    reference: '&Ref?',     // nil for most
    description: 'str',     // optional description
    dataType: 'DataType',
    isRequired: 'bool',
    isFileId: 'bool',       // false for most
    displayFunction: 'int', // mapping to a display function
    derived: '&[Derived]',  // list of derived metrics, empty for most
});

A standard Metric(123456).wrap() call often returns a large object dominated by default or empty values:

(JSON) 

{
    "id": 123456,
    "key": "name",
    "reference": null,
    "description": "",
    "dataType": "STRING",
    "isRequired": true,
    "isFileId": false,
    "displayFunction": 0,
    "derived": []
}

The Solution: The ! and ? Wrap-Prefix Flags

The wrap-prefix flags enable a concise way to control data serialization based on truthiness:

  • ! – Excludes the property if its value evaluates to false (e.g., false, 0, "", []).
  • ? – Excludes the property only if its value is explicitly nil.

Applying the Flags

We update the Metric definition by prefixing properties we want to exclude when empty or false: 

set_type('Metric', {
    id: '#',
    key: 'str<1:>',
    reference: '?&Ref?',     // Exclude when explicitly `nil`.
    description: '!str',     // Exclude when false (empty string).
    dataType: 'DataType',
    isRequired: '!bool',     // Exclude when false.
    isFileId: '!bool',       // Exclude when false.
    displayFunction: '!int', // Exclude when false (value is 0).
    derived: '!&[Derived]',  // Exclude when false (empty list).
});

The Compact Result

Running the same query, Metric(123456).wrap(), now yields a significantly more compact result:

(JSON) 

{
    "id": 123456,
    "key": "name",
    "dataType": "STRING",
    "isRequired": true
}

By leveraging the ! and ? flags, we eliminate unnecessary data transfer, resulting in significantly reduced network traffic and faster query response times when dealing with meta data objects.

Similar Posts