How to cope with IoT Hub enrichment restrictions

As seen in my previous post, The IoT Hub routing feature supports message enrichment, both for IoT devices and IoT edge modules.

Using the routing message enrichments, each incoming message gets extra user properties based on either static values, device twin tags, or device twin desired properties.

Unfortunately, only ten enrichments can be added:

If you want to pass more values, this will not work for you.

It would be great if nested JSON properties would count as one.

Again, unfortunately, only simple types (string, decimal, boolean, date/time, etc.) are supported so this excludes nested JSON (complex types).

Below, a viable solution to overcome both restrictions, using Azure Stream Analytics, is presented.

Let’s see how this works out.

Dynamic routing of IoT Hub telemetry to Azure Data Explorer

Azure Data Explorer (ADX) is a great data exploration tool for IoT developers building a full IoT solution. This could be a perfect target for the cold path.

As seen in my previous blog post, ADX even offers a native connector for the IoT Hub. This is based on the ‘default EventHub compatible endpoint’ offered by this cloud IoT gateway (optionally the built-in Events endpoint in the routing section of the IoT Hub or using the fallback mechanism).

Most of the documentation regarding this ADX connector is following this ‘happy flow’ where one connector stores incoming IoT telemetry in one ADX table using static routing.

This is a serious limitation where most IoT Hubs ingest multiple types of messages. These will not fit into that single table.

Luckily, the connector also offers the possibility to allow routing to other databases:

Here, we will check out this dynamic routing option and see how this provides much more flexibility.

IoT sessions during Build 2022

Yes, the Microsft Build 2022 is coming. It will be held on May 24-26. It’s online. It’s free.

So register now and start building your schedule:

There are 500+ sessions of different kinds: keynotes, on-demand sessions, round tables, and connection zone.

Let’s check out the IoT-related sessions.

Free Microsoft AZ-220 Azure IoT Developer Specialty renewal

If you are working with Azure IoT on a daily bases, it’s recommended to get certified for it.

Microsoft offers this AZ-220 Azure IoT Developer Specialty exam and certification program just for that.

If you are new to this AZ-220 exam, please check out this blog post first with information about how to get started with the program.

This exam is valid for three years once you passed it.

I took it already a few years ago. Recently, I got this email saying my exam is expiring so I am granted an extra year once I pass a free exam on MS Learn:

Let’s check out how this works.

Azure Data Explorer connector for IoT Hub with real-time ingest

Azure Data Explorer (ADX) is a fully managed, high-performance, big data analytics platform that makes it easy to analyze high volumes of data in near real-time. The Azure Data Explorer toolbox gives you an end-to-end solution for data ingestion, query, visualization (including third-party dashboards), and management.

ADX can ingest data both from (persisted) storage and data provided as a stream:

Ingested data is managed and optimized by the underlying cluster so it can be queried and made available in (third-party) platforms as a view (like Power BI, Grafana, etc.) based on a powerful query language.

Data can also be ingested directly from Azure IoT Hub, as a stream for real-time analysis.

Let’s check out how this works.

Azure IoT Device lifecycle events Part 2: modules

In my previous blog post, I showed how to use the Azure IoT device lifecycle events.

These events are emitted by the Azure IoT Hub as routing events next to the regular incoming device-to-cloud messages. Routing these device lifecycle events makes it possible for both persisting and reacting at the behavior of devices (creation, deletion, connected, disconnected) and registration changes (device identity twin changes).

This was first demonstrated using a regular direct internet-connected device.

But what about Azure IoT modules?

Event Grid custom Event Source and Handler

Azure Event Grid is one of (many) event mechanisms available in the Azure cloud.

Azure Event Grid works with these moving parts:

• Event sources. Some logic sending messages
• Event handlers. The logic consuming incoming messages
• Event Grid topics. the mechanism receiving incoming messages and sending them to subscribed event handlers
• Event Grid subscriptions. The connection between topic and handlers. Offers filtering and additional features

Note: in this post, we focus on Azure Event Grid. Check out this MS Learn module for comparison with Azure Event Hub and Azure Message Bus.

Azure Event Grid support is offered for a growing number of Azure resources as an extra event mechanism, out-of-the-box. The resource could either be a source or handler or both.

For your convenience, we can also create custom events to custom topic and consume them:

It must be clear that Event Grid uses the fan-in-fan-out pattern, also seen in Event Hubs.

Multiple Event sources can send messages on the same topic. From there, multiple Event handlers can pick their own copy of incoming messages using its subscription. Filtering can be applied if needed.

We ignore the predefined event sources and focus on a custom one to explain the flow.

Adding context using EventData properties in Azure Stream Analytics

The Azure IoT Hub is the preferred Azure Cloud gateway for IoT Devices.

The format of messages sent by devices to the IoTHub is described in the EventData class.

There, a regular message is split into three data parts:

1. The actual message body, the data inside
2. System properties, context added by the system (like IoT Hub context)
3. Application (aka User) properties, context added by the user

These properties are key value pairs. these are not encoded opposite to the message body which has to be decoded before it is accessible.

So, we see that the original messages can get context along the way.

As a developer, using Azure IoT, adding application/user context can be done at two levels:

We can add application properties to the device message. This is normally done on the device when the message is constructed.

The Azure IoT Hub also supports message enrichment. These enrichments are also added as application/user properties, as seen in my previous blog post.

There, I showed how to read the properties using an Azure Function.

Consuming message context is a little bit of a challenge as seen in that post.

Here, we dive into this using Azure Stream Analytics.

Enforce tag usage on Azure resources using Tag policies

How do you find out for which project or environment a certain Azure resource was created?

How do you find out which team is responsible for the costs related to a resource or resource group?

These are just some of the questions that can be answered easily using the support of tags on Azure resources.

Azure supports adding tags to resources. These can be added when the resource is created. These can also be added or updated at a later moment:

Tags are just key-value pairs giving more context to your resources. You can describe eg. the project it is used for, the creation date, the owner, the customer, the environment.

You can just add any key-value pair you have in mind.

Once added, the tags are shown in the portal:

Tags can then be selected while filtering Azure costs, as seen in the portal:

Now, you have a better understanding of the context of this resource group.

Unfortunately, adding tags is optional by default. And in the portal, these must be added by hand at first. Therefore, these are not always added by users or entered wrong just because the purpose is not yet clear to them.

Luckily, Azure offers policies to enforce adding tags.

This way, it’s not optional anymore. And, tags can be inherited too.

Let’s see how this is done.

Azure IoT Hub device query language

The Azure IoT Hub can register thousands of devices. To manage them at scale, several kinds of tooling is made available.

First, using the Device Twin of each device, devices can store extra context (type, brand, vendor, version, location, features, etc.) using the Tags section.

In the Tags section of the device twin, you are free to add a number of (sub) nodes to this JSON document section:

The tags will never be readable by the device itself, these tags are used to query all devices in your IoT Hub and make subsets.

You, both as Azure portal user or as a programmer, can query all devices for specific features.

In the Azure portal, this ability to query is most visible when you open the list of (edge) devices:

This shows a new section where we can enter a SQL-ish query starting with “SELECT FROM devices WHERE”:

Let’s dive further into this query language.