Azure Data Explorer connector for IoT Hub with real-time ingest

Azure Data Explorer (ADX) is a fully managed, high-performance, big data analytics platform that makes it easy to analyze high volumes of data in near real-time. The Azure Data Explorer toolbox gives you an end-to-end solution for data ingestion, query, visualization (including third-party dashboards), and management.

ADX can ingest data both from (persisted) storage and data provided as a stream:

Ingested data is managed and optimized by the underlying cluster so it can be queried and made available in (third-party) platforms as a view (like Power BI, Grafana, etc.) based on a powerful query language.

Data can also be ingested directly from Azure IoT Hub, as a stream for real-time analysis.

Let’s check out how this works.

Azure IoT Device lifecycle events Part 2: modules

In my previous blog post, I showed how to use the Azure IoT device lifecycle events.

These events are emitted by the Azure IoT Hub as routing events next to the regular incoming device-to-cloud messages. Routing these device lifecycle events makes it possible for both persisting and reacting at the behavior of devices (creation, deletion, connected, disconnected) and registration changes (device identity twin changes).

This was first demonstrated using a regular direct internet-connected device.

But what about Azure IoT modules?

Event Grid custom Event Source and Handler

Azure Event Grid is one of (many) event mechanisms available in the Azure cloud.

Azure Event Grid works with these moving parts:

• Event sources. Some logic sending messages
• Event handlers. The logic consuming incoming messages
• Event Grid topics. the mechanism receiving incoming messages and sending them to subscribed event handlers
• Event Grid subscriptions. The connection between topic and handlers. Offers filtering and additional features

Note: in this post, we focus on Azure Event Grid. Check out this MS Learn module for comparison with Azure Event Hub and Azure Message Bus.

Azure Event Grid support is offered for a growing number of Azure resources as an extra event mechanism, out-of-the-box. The resource could either be a source or handler or both.

For your convenience, we can also create custom events to custom topic and consume them:

It must be clear that Event Grid uses the fan-in-fan-out pattern, also seen in Event Hubs.

Multiple Event sources can send messages on the same topic. From there, multiple Event handlers can pick their own copy of incoming messages using its subscription. Filtering can be applied if needed.

We ignore the predefined event sources and focus on a custom one to explain the flow.

Adding context using EventData properties in Azure Stream Analytics

The Azure IoT Hub is the preferred Azure Cloud gateway for IoT Devices.

The format of messages sent by devices to the IoTHub is described in the EventData class.

There, a regular message is split into three data parts:

1. The actual message body, the data inside
2. System properties, context added by the system (like IoT Hub context)
3. Application (aka User) properties, context added by the user

These properties are key value pairs. these are not encoded opposite to the message body which has to be decoded before it is accessible.

So, we see that the original messages can get context along the way.

As a developer, using Azure IoT, adding application/user context can be done at two levels:

We can add application properties to the device message. This is normally done on the device when the message is constructed.

The Azure IoT Hub also supports message enrichment. These enrichments are also added as application/user properties, as seen in my previous blog post.

There, I showed how to read the properties using an Azure Function.

Consuming message context is a little bit of a challenge as seen in that post.

Here, we dive into this using Azure Stream Analytics.