Message enrichment for Azure IoT Edge modules

The Azure IoT Hub offers ingestion of D2C and D2C messages for thousands or more devices.

Incoming messages are routed to several endpoints (eg. Event Hub or Blob Storage) using internal IoT Hub routes:

Note: even messages sent to the Event Grid are also enriched.

The messages produced by devices look like these (image taken from the console output of the Microsoft temperature simulation):

Here, messages from the Microsoft temperature simulation module are sent to the cloud.

When these messages arrive in the IoT Hub, the fulll message format looks like this:

{
  "body": {
    "machine": {
      "temperature": 73.05171716227275,
      "pressure": 6.929942461524743
    },
    "ambient": {
      "temperature": 20.795734739068774,
      "humidity": 26
    },
    "timeCreated": "2021-05-10T11:51:42.1835163Z"
  },
  "enqueuedTime": "Mon May 10 2021 13:51:42 GMT+0200 (Central European Summer Time)",
  "properties": {
    "sequenceNumber": "101",
    "batchId": "7f3f847f-c20c-49aa-af17-a22db31f244f"
  },
  "systemProperties": {
    "iothub-connection-device-id": "simulation",
    "iothub-connection-module-id": "sim",
    "iothub-connection-auth-method": "{\"scope\":\"module\",\"type\":\"sas\",\"issuer\":\"iothub\",\"acceptingIpFilterRule\":null}",
    "iothub-connection-auth-generation-id": "637540164221770949",
    "iothub-enqueuedtime": 1620647502173,
    "iothub-message-source": "Telemetry",
    "x-opt-sequence-number": 13975,
    "x-opt-offset": "12885042704",
    "x-opt-enqueued-time": 1620647502314
  }
}

As you can see, the messages are build up in three parts:

1. The actual message body, generated by (your) logic in the device. This part could be encoded so only the receiver knows how to handle this. Normally this is a JSON message format, though.
2. (Application) properties, message enrichment on the device to provide context about the messages. This acts as a property bag, also added by the developer. The values are directly accessible for routing purposes.
3. System properties, added by the IoT Hub. This gives technical context about the incoming message from the perspective of the IoT Hub

So users can choose to add values in the body or to add values in the application properties.

Still, every byte put in the message sent by the device results in a longer transmit duration and extra costs (number of bytes communicated, potentially more message chunks handled by the IoT Hub).

So, values already found in the system properties (eg. the device name or module id) can be left out in the body. This results in a smaller message.

Microsoft also provides message enrichment on the IoT Hub. This can take away some of the pain too.

It also makes it possible to route messages further down the road in other Azure resources which receive these enriched messages.

Let’s check this out.

Message enrichment on the IoT Hub makes it possible to enrich incoming messages with some extra values, provided by either the IoT Hub, device twin, or custom values:

At this time, only $iothubname, $twin.tags, $twin.properties.desired, and $twin.properties.reported are supported variables for message enrichment.

So, let’s add some tags and desired properties to both an Azure IoT Edge device twin and the deployed Microsoft Temperature Simulation module twin deployed on that device:

Here are the Device Twin tags and desired properties:

Here are the Module Twin tags and desired properties:

Check: As you can see, the same tags and desired properties are added at both locations.

Let’s add some message route enrichments:

Here, I add:

• $twin.tags.version
• $twin.tags.customer
• $twin.tags.usage
• $twin.properties.desired.myProperty (reported properties work the same)
• $iothubname
• Custom value meaningOfLife = 42
• $twin.tags.deviceOnly (not added on module)
• $twin.tags.moduleOnly (not added on device)

Once applied, Let’s check incoming messages in the IoT Explorer:

Note: It can take some time until the mapping actually is enabled. These values have to be deployed on the backend of the IoT Hub where incoming messages are partitioned. Until then, the special enrichment values for eg. tags are just shown as text values…

We have several interesting observations:

1. The enrichments are made available an (application) properties. The body is not extended!
2. Tags and properties from the Module Twin are represented over the device properties.
3. Device twin properties are ignored!
4. The integer tag value “Version” is written as a string, so is the extra “meaningOfLife”
5. the IoT Hub name is the short hostname

This should give you quite a few possibilities to filter or aggregate messages later on in other Azure resources receiving these messages.

Device properties are not mapped

The most important take-away is that device properties are not mapped.

This means you have to take extra care of module twins!

If you want to enrich messages from multiple modules of the same device, you have to add these tags to each module! There is no fall-back!

Just to be sure, I tested this on the desired properties too:

There, the mapping is not picked up too. So, test the mapping twice for each module on edge devices!

Accessing the properties in an Azure Function

I created an Azure Function to check out the messages put on the default built-in endpoint.

The default EventHub triggered function template uses a string as a representation of the message (body). The properties are ignored…

This is not optimal.

Then, I noticed the solution I used in a previous blog is deprecated (using the “Microsoft.ServiceBus” libraries).

So, a new and more elegant solution makes use of the EventHubs library. Azure Function has support for this library by default using the “#r” notation:

#r "Newtonsoft.Json"
#r "Microsoft.Azure.EventHubs"

using System;  
using Newtonsoft.Json;
using Microsoft.Azure.EventHubs;

We are interested in both the body, the application properties, and the system properties.

Thus, the full code block of the function looks like this:

#r "Newtonsoft.Json"
#r "Microsoft.Azure.EventHubs"

using System;  
using System.Text;
using Newtonsoft.Json;
using Microsoft.Azure.EventHubs;

public static void Run(EventData myIoTHubMessage, ILogger log)
{
    string messageBody = Encoding.UTF8.GetString(myIoTHubMessage.Body.Array, myIoTHubMessage.Body.Offset, myIoTHubMessage.Body.Count);

    log.LogInformation($"C# IoT Hub trigger function processed a message: {messageBody}");

    var systemProperties = string.Join(" | ", myIoTHubMessage.SystemProperties.Select(x => $"{x.Key} = {x.Value}"));
    log.LogInformation($"System properties: {systemProperties}");

    var properties = string.Join(" || ", myIoTHubMessage.Properties.Select(x => $"{x.Key} = {x.Value}"));
    log.LogInformation($"Properties: {properties}");
}

Note: the system properties are separated by one ‘|’ pipe character. The application properties are separated by two pipe characters.

Once deployed, this results in:

2021-05-10T15:01:31.729 [Information] Executing 'Functions.IoTHub_EventHub1' (Reason='(null)', Id=b6a7b20f-f2a0-42e5-ac26-3f15c738d7e3)
2021-05-10T15:01:31.729 [Information] C# IoT Hub trigger function processed a message: {"machine":{"temperature":101.65851864074759,"pressure":10.188945161604154},"ambient":{"temperature":20.89180557587734,"humidity":26},"timeCreated":"2021-05-10T15:01:31.6466362Z"}
2021-05-10T15:01:31.729 [Information] System properties: iothub-connection-device-id = simulation | iothub-connection-module-id = sim | iothub-connection-auth-method = {"scope":"module","type":"sas","issuer":"iothub","acceptingIpFilterRule":null} | iothub-connection-auth-generation-id = 637540164221770949 | iothub-enqueuedtime = 5/10/2021 3:01:31 PM | iothub-message-source = Telemetry | x-opt-sequence-number = 17984 | x-opt-offset = 12887598784 | x-opt-enqueued-time = 5/10/2021 3:01:31 PM
2021-05-10T15:01:31.729 [Information] Properties: sequenceNumber = 2367 || batchId = 7f3f847f-c20c-49aa-af17-a22db31f244f || version = 8055 || deviceOnly = $twin.tags.deviceOnly || moduleOnly = moduleOnly || customer = mtAcme || myProperty = mtValue || iotHubName = training-demo-weu-ih || meaningOfLife = 42 || usage = mtSimulation
2021-05-10T15:01:31.729 [Information] Executed 'Functions.IoTHub_EventHub1' (Succeeded, Id=b6a7b20f-f2a0-42e5-ac26-3f15c738d7e3, Duration=1ms)

See, we have access to both the body and the properties.

Again, the ‘device only’ values are not mapped. The rest of the enriched values can be read using code.

Conclusion

Message enrichment on the IoT Hub is a nice way to:

1. keep the size of messages low because common data can be added to messages in the IoT Hub, not on the device
2. The enriched values are configurable on the cloud, not static (as in hard-coded on the device)
3. These enriched values can be used to filter or route messages down the stream (eg. in a Stream Analytics job or Azure Function)

For messages coming from Azure IoT Edge modules, the enrichment is based on the Module Twin values, Device Twin values are ignored.

Because routing of messages is based on Device Twin values, the tags, and properties of Azure IoT Edge devices must be propagated to the module twins for modules sending messages to the cloud.

Also check out my post on coping with IoT Hub routing message enrichment limitations.

Advertentie