Message enrichment for Azure IoT Edge modules

The Azure IoT Hub offers ingestion of D2C and D2C messages for thousands or more devices.

Incoming messages are routed to several endpoints (eg. Event Hub or Blob Storage) using internal IoT Hub routes:

Note: Messages sent to the Event Grid are also enriched.

The messages produced by devices look like this:

Here messages from the Microsoft temperature simulation modules are sent to the cloud.

When these messages arrive in the IoT Hub, the format looks like this:

{
  "body": {
    "machine": {
      "temperature": 73.05171716227275,
      "pressure": 6.929942461524743
    },
    "ambient": {
      "temperature": 20.795734739068774,
      "humidity": 26
    },
    "timeCreated": "2021-05-10T11:51:42.1835163Z"
  },
  "enqueuedTime": "Mon May 10 2021 13:51:42 GMT+0200 (Central European Summer Time)",
  "properties": {
    "sequenceNumber": "101",
    "batchId": "7f3f847f-c20c-49aa-af17-a22db31f244f"
  },
  "systemProperties": {
    "iothub-connection-device-id": "simulation",
    "iothub-connection-module-id": "sim",
    "iothub-connection-auth-method": "{\"scope\":\"module\",\"type\":\"sas\",\"issuer\":\"iothub\",\"acceptingIpFilterRule\":null}",
    "iothub-connection-auth-generation-id": "637540164221770949",
    "iothub-enqueuedtime": 1620647502173,
    "iothub-message-source": "Telemetry",
    "x-opt-sequence-number": 13975,
    "x-opt-offset": "12885042704",
    "x-opt-enqueued-time": 1620647502314
  }
}

As you can see, the messages are build up in three parts:

1. The actual message body, generated by the device. This part could be encoded so only the receiver knows how to handle this. Normally this is a JSON message, though.
2. (Application) properties, message enrichment on the device to provide context about the messages. This acts as a property bag, these values are directly accessible for routing purposes.
3. System properties, added by the IoT Hub. This gives technical context about the incoming message from the perspective of the IoT Hub

So users can choose to add values in the body or to add values in the application properties. Values already found in the system properties can be left out.

Still, every byte put in the message sent by the device results in a longer transmit duration and extra costs (number of bytes communicated, potentially more message chunks handled by the IoT Hub).

Microsoft also provides message enrichment on the IoT Hub. This can take away some of the pain.

It also makes it possible to route messages.

Let’s check this out.

How to cope in code with the quota limitation of an IoT Hub

In a previous blog, I showed how to make use of the Azure IoT Device SDK to send data to an Azure IoT Hub.

It is very simple to start with azure IoT making use of those SDKs using either C#, Embedded C, C, Java, NodeJS, Python, or ioS.

The only thing you have to take into account is the IoT Hub message quotas.

An S1 version is capable of receiving 400.000 D2C messages a day of size 4KB, the free tier F1 supports 8000 messages of size 0.5 KB.

Sending C2D messages and Device provisioning is also part of this amount.

Let’s take a further look at this limitation.

Azure Stream Analytics anomaly detection on the edge

Back in 2018, the Azure Stream Analytics team announced Anomaly detection for Azure Stream Analytics. And, it was also supported on the Azure IoT Edge. In November 2018, I was allowed to demonstrate this at the SPS IPC Drives in Nuremberg:

Since then, anomaly detection has become a first citizen of Azure Stream Analytics.

Azure Stream Analytics Anomaly Detection is able to ‘automatically’ detect spikes, dips, and trends in a stream of values. This is based on math and all you need to do is to specify how many values you expect and how sensitive the detection must be. It is in fact Machine Learning, in the end. So it is a prediction with a certain certainty…

And it’s just part of the Stream Analytics query language. So on the edge, we can deploy a Stream Analytics module (the engine of stream analytics). This is a fixed module from Microsoft.

All you have to do is feeding it a query, inputs, outputs and, user-defined functions if available:

We can define all this in the cloud, inside a stream analytics job and we can even test it. Those parts are then packed as a blob and put in blob storage. The Stream Analytics job can then download it and run it.

The inputs and outputs of the Stream Analytics job can be attached to the normal Azure IoT Edge routing mechanism.

Let’s see how this is set up:

IoT on Microsoft Ignite, March 2021

Although 2021 seems to be a copy of 2021, the Microsoft Ignite has brought us a lot of possitive new ragarting new Microsoft technology we can all wait for. Or even better, it’s available right now.

The biggest Azure IoT on the current Ignite is Azure Percept:

 a comprehensive, easy-to-use platform with added security for creating responsible edge AI solutions. With Azure Percept, we have simplified AI on the edge from silicon to service

Documentation can be found here. The dedicated hardeware (Azure Percept DK is designed as a reference implementation) can be found here. (Datasheet DK | Datasheet Vision) (Availability around the world)

But there were also smaller gems. On of them is “30 days to learn it“. You can complete the IoT Developer challenge and get 50 percent off the cost of a Microsoft Certification:

Implement and maintain the cloud and edge components of an IoT solution. Learn how to connect devices, move workloads to the edge, and build scalable, enterprise-grade, secured IoT solutions on Azure to increase performance, reduce costs, and optimize operations. It’ll only take about 15 hours to discover how to securely connect IoT devices to the cloud, build the intelligent edge, and develop solutions with Azure IoT Central.

There are more certificates to earn. See the site for all eight challenges!

All sessions are available now in the session catalog for you to watch. Below, I summerized all IoT related sessions.

Using Influx database and Grafana on the Azure IoT Edge

Over the last couple of years, in this blog, multiple databases are demonstrated which can be deployed on the Azure IoT Edge:

• SQL Server
• MySql
• CrateDB
• SQLite

Persisting incoming telemetry in a local database is useful for multiple purposes. One of them is creating a custom dashboard eg. written in Blazor.

In this blog, we explore how to deploy the popular InfluxDB. This open-source time-series database is specialized in Internet of Things usage. And on top of that, let’s explore how Grafana can be deployed on the edge too. Grafana is an open source analytics and interactive visualization web application.

We see how it can be connected to the Influx database:

As you see here, we need a custom Azure IoT Edge module (called ‘writer’) that is capable of writing incoming telemetry to the Influx database. There, the telemetry is picked up and displayed by Grafana.

Let’s see how this works.

IoT Plug and Play, modeling IoT Central devices

If you have built an IoT Device yourself and are finally able to send telemetry to the cloud, you should be familiar with the scenario where you have to repeat the hard work of describing the messages all again on ingestion.

IoT Devices expose D2C telemetry and it can also support C2D communication. This interface is most of the time unique for that device. To be able to get insights from a device you have to be able to react to its interface.

Wouldn’t it be nice if a device was able to provide metadata about its interface once it connects to the cloud? This way, the incoming D2C telemetry could automatically result in e.g. a full user interface. And all C2D output could be represented by pre-configured input controls.

With IoT Plug and Play this is all possible:

IoT Plug and Play enables solution builders to integrate smart devices with their solutions without any manual configuration. At the core of IoT Plug and Play, is a device model that a device uses to advertise its capabilities to an IoT Plug and Play-enabled application

https://docs.microsoft.com/en-us/azure/iot-pnp/overview-iot-plug-and-play

Microsoft provides a way to describe the interface of a device and annotate every feature.

To experience how this works, we will look at the best example: Azure IoT Central.

This SaaS IoT Dashboard makes perfect use of the ability how devices can expose their meta-model.

Let’s see how.

All Around Azure IoT. Join this free conference at January 19, 2021

TLDR: Missed the event? Here is the recording of my session:

At Janaury 19, 2021, Microsoft organizes a world-wide event about Azure IoT.

Add local storage to Azure IoT Edge modules using Docker Bind

Azure IoT Edge makes use of the Moby container runtime so IoT Edge modules (being Docker containers) can work together and offer logic on the edge.

Docker containers are ‘sandboxed’. This means that the logic within the containers has limited access to the environment they ‘live’ in.

By default, containers have no SUDO rights, no access to the host filesystem, and just limited network capabilities.

Though, containers can be granted elevated rights. One of these is the right to access the filesystem.

In this blog, we will see how to configure a container with access to the filesystem. To demonstrate this, a custom IoT Edge module is introduced, an IoT Edge filewatcher for CSV files:

Handling Advantech Wise 710 OPC-UA telemetry using OPCPublisher

Microsoft has an extensive IoT platform based on Azure.

It provides so many features, this can be overwhelming for customers. Therefore, Microsoft provides Azure IoT solution accelerators, based on the Azure IoT reference architecture.

Some of the original accelerators (it started with Azure IoT suites) like Remote Monitoring are now outdated or even archived. These are replaced by excellent Azure IoT Central apps which demonstrate the capabilities of the IoT platform for numerous markets and verticals:

There is still one original accelerator alive-and-kicking: the Connected Factory. This one demonstrates the use of OPC-UA protocols on the edge and in the cloud.

More than two years ago, I already wrote about this accelerator and the OPC publisher module, the backbone of this accelerator. Since then, a lot has changed. Some functionality is (temporarily) deprecated so I got a lot of questions based on the old blogs.

So it’s time to update it a little and see how the OPC Publisher is doing these days.

The OPC-UA solution is open source and covers a lot:

I limit this blog to the scope of my very first blog, extracting OPC-UA messages and send them to the cloud using a ‘published nodes’ file.

We just need an OPC-UA server to get some sample data from. For this, I used an Advantech Wise 710 as an industrial protocol gateway.

Azure IoT Edge module metrics in action

We are familiar with the Azure IoT Hub metrics which are offered. The Azure cloud tells us eg. how many messages are received or the number of devices that are connected.

If we look at Azure IoT Edge, you had to collect your own made metrics in the past.

Because IoT Edge modules are Docker containers and therefore sandboxed, you had to rely on the (third-party) logic to capture Host metrics. Regarding metrics about the edge agent and hub, these were not available.

Until now.

With the most recent IoT Edge runtimes, agent, and hub, we have access to Edge metrics.

Both the Agent and Hub module expose the metrics over HTTP endpoints:

Within the Moby runtime, port 9600 is exposed on both individual modules. Outside the runtime, we have to assign individual host ports to prevent using the same host port.

Let’s see how this looks like and how we can harvest metrics in a custom container.