Microsoft Fabric real-time analytics exploration: EventGrid MQTT Broker integration

Microsoft Fabric is an all-in-one analytics solution for enterprises that covers everything from data movement to data science, Real-Time Analytics, and business intelligence.

For IoT Developers, this is a great addition to our Azure IoT resources toolkit because ingesting streams of IoT data is supported too.

In this series of blog posts, we have already learned about integrating with the Azure IoT Hub.

For my recent presentation at the Data Toboggan Winter edition, I wanted to integrate via the MQTT protocol using the new Azure EventGrid MQTT support:

In this blog post, we learn how to integrate using a little extra help offered by an Azure Function.

Streaming data analysis with Free Azure Data Explorer

If you are new to Azure Data Explorer, a simple and cheap way to start is the Free Azure Data Explorer cluster.

There are a number of limitations compared to the paid version of Azure Data Explorer. But hey, you get a Time Series database cluster with data ingestion, KQL querying, and dashboarding for free!

Personally, I moved over to the paid dev/test version because I work a lot with streaming Azure IoT telemetry ingest and this kind of data ingestion was not offered in the free version.

Until last week!

Although a direct data connection to the Azure IoT Hub or ingestion via Event Grid is not offered, we can now ingest streaming data via Event Hub.

This brings us to the following architecture:

This post will explore how to ingest streaming data using the new Event Hub support.

Exploring Azure Digital Twins Graph history

Update: the historical data export functionality now also exports twin lifecycle information. This means a part of the custom logic seen below is now part of the default export.

As seen in previous posts, Azure Digital Twins shows the current state of any twin in the Azure Digital Twins graph.

This current state is a combination of multiple items:

• twin name and model
• twin properties updates based on telemetry and/or business rules related to that telemetry
• twin properties updates based on predictions towards the future
• twin properties updates based on historical data

Azure Digital Twins also offers to store historical Twin property data. That historical data is made available using Azure Data Explorer, part of the data history connection, as explained in my previous post.

There, twin data can also be compared with twin graph knowledge using the ADX Kusto plugin for querying the ADT graph.

Still, that accompanying plugin is missing something. We have no historical knowledge of the graph itself including models and how these changed over time!

If we want this information, we need to be creative.

Can we add the missing pieces?

Let’s check out how we can explore the ADT graph history…

Azure Data Explorer connector for Blob storage (IoT Hub) files

Over the last couple of months, I have written several blog posts regarding Azure Data explorer ingestion connectors.

I checked out the IoT Hub connector, dynamic mapping, and table update policies.

Until now, this is all based on both the IoT Hub connector and the Event Hub connector:

Next to those two, Azure Data Explorer also supports ingesting complete files using the third connector: Blob Storage ingestion in combination with EventGrid events:

Let’s check out what this connector offers and how to set it up.

As we will find out, it even has a surprise for those of you ingesting IoT Hub messages.

Extending the AZ-220 Digital Twins hands-on lab with 3D visualization

Azure Digital Twins is advertised as a “platform as a service (PaaS) offering that enables the creation of twin graphs based on digital models of entire environments, which could be buildings, factories, farms, energy networks, railways, stadiums, and more—even entire cities”.

This sounds promising but it does not really ring a bell, does it?

Fortunately, besides the excellent documentation, Microsoft provides a great learning path in MS Learn as part of the AZ-220 Azure IoT developer exam preparations.

There, you will learn how Azure Digital Twins offers new opportunities for representing an Internet of Things solution via twin models, twin relations, and a runtime environment.

You finish the learning path with a hands-on lab where you build a model around a cheese factory and ingest sensor telemetry:

In the demo, the telemetry flows through the runtime and ends up in Time Series Insights.

Yes, the learning path is a good start and will prepare you for the exam or the assessment (you need to pass this assessment for a free one-year certification renewal).

On the other hand, many extra features could be added to turn this good start into a great start!

Think about propagating Azure Digital Twins events and twin property changes through the graph and visualizing live updates of twins in a 3D model, complete with alerts.

Let’s check out some of these additional features and see what you need to do to extend the ADT example.

Adding context using EventData properties in Azure Stream Analytics

The Azure IoT Hub is the preferred Azure Cloud gateway for IoT Devices.

The format of messages sent by devices to the IoTHub is described in the EventData class.

There, a regular message is split into three data parts:

1. The actual message body, the data inside
2. System properties, context added by the system (like IoT Hub context)
3. Application (aka User) properties, context added by the user

These properties are key value pairs. these are not encoded opposite to the message body which has to be decoded before it is accessible.

So, we see that the original messages can get context along the way.

As a developer, using Azure IoT, adding application/user context can be done at two levels:

We can add application properties to the device message. This is normally done on the device when the message is constructed.

The Azure IoT Hub also supports message enrichment. These enrichments are also added as application/user properties, as seen in my previous blog post.

There, I showed how to read the properties using an Azure Function.

Consuming message context is a little bit of a challenge as seen in that post.

Here, we dive into this using Azure Stream Analytics.

Azure IoT Device lifecycle events (part 1)

The Azure IoT Hub is the main gateway for Azure IoT-related device connectivity.

It features several useful features to make your IoT developer life easy.

It offers a registry for devices so each device has its own credentials. Each device is capable to ingest device-to-cloud telemetry data into Azure and the IoT Hub also offers several ways for cloud-to-device communication (e.g. desired properties and direct methods).

The IoT Hub is also capable to report any device change event during the lifecycle of that device:

It can generate messages when a device is created or deleted. It also generated messages when the device twin is changed. It even generates a message when the ‘digital twin’ is updated.

Note: The digital twin update is related to Azure IoT Plug and Play. This is out of scope for this post. Though, an example of the digital twin change event is seen here.

In this post, we will look at the format of these messages.

First look: The Things Network new Azure IoT Hub integration

Right from the start back in 2015, The Things Network supported connecting their LoraWan backend platform to other (cloud) platforms.

Connecting to the Azure cloud was no exception to this.

There was no default integration but the support for the MQTT protocol opened this door. Back then, I already blogged about using MQTT to connect. Based on that logic, bridges appeared for both uplink and downlink messages.

Later on, Microsoft also introduced the secure and flexible IoT Central bridge to connect to The Things Network, based on uplink support for webhooks.

Still, even with the new V3 portal of The Things Network, this integration was still based on either the Webhook integration or the original V3 MQTT interface.

All these techniques require an extra layer of logic between the two clouds.

But…

Finally, a native Azure IoT Hub integration has been made available:

The main features of the Azure IoT Hub Integration are:

• Devices only need to be registered in the Azure IoT Hub. The related TTN application device registrations are kept in sync by TTN. Just use IoT Hub Device Twin tags to control your TTN device registration
• Uplink messages appear both as telemetry and as Device Twin reported property updates
• Downlink messages are sent using the Device Twin desired properties

The current documentation is pretty straight forward (although the DPS service logo is shown, it’s not part of the standard solution).

Let’s check out how this integration works and let’s dive deeper into this solution architecture.

Event Hub alerts, great for detecting drops in connectivity

If we look at the Azure IoT Reference Architecture we see how streaming data is the heart of the IoT platform:

Data arrives at the IoT Hub and can be routed to any Azure resource using eg. the IoT Hub routing, Stream Analytics jobs and Event Hubs.

This gives us the maximum flexibility to divide the data into three major data streams or storage:

1. Hot path – Event, Alert, Conditions
2. Warm path – aggregated data, data for reporting
3. Cold path – the raw data, mainly untouched and available in large quantities; great for data scientists

But there is an often unseen, even ignored forth stream. And that is the stream of data for monitoring.

In my earlier blogs, I wrote several times hoe heartbeats en watchdogs can tell the story about the quality of the data. And I have shown how eg. Azure Functions and notification services can improve the insights about the quality of the communication.

Today I want to add a little gem to that list.

Let’s check out Monitoring Alerts in Event Hubs. It’s part of the overall Monitoring service in Azure.

Introduction to the IoT Edge SDK V1, part 4: IoTHub routing

Update: Recently, Microsoft introduced the new V2 version of their IoT Edge. This blog is referencing the former but still supported V1.

We have already made great progress understanding and using the Azure Gateway SDK.

What do we have right now? We can send telemetry data from multiple ‘offline’ devices and accept commands from the IoT Hub.

The data we send is well-formatted JSON so we are good to go.

But I am a bit worried. While reading all documentation regarding the transformation from Azure Gateway SDK towards the IoT Edge SDK, it is clear that multiple types of messages are sent to the IoT Hub. For example, I can imagine that a Stream Analytics module generates other data.

And let’s look at a more ‘close to earth’ example. The gateway itself is a potential device too! But I do not want to mix data coming from the gateway and from sensor devices.

Of course, we recently got the ability to route messages using the message sent. But what about using the properties? This keeps the message content clean.

Will this be working?