The Things Network MQTT integration in Azure

Over the years, I have written multiple articles about working with The Things Network community LoraWan platform.

It all started with this workshop back in 2016 that I built together with my friend Jan Willem Groenenberg where we connected the TTN backend with the Azure Cloud. Over the years, we organized many events based on the workshop.

We needed a ‘bridge’ to bring two worlds together: The Things Network backend applications and the Azure cloud.

I created this TTN azure bridge based on the MQTT protocol supporting a stateful exchange of D2C (uplink) messages from Lora devices to an Azure IoT Hub and supporting C2D (downlink) messages back to the devices.

Since then, the TTN backend migrated twice and now we have this new Version 3 backend with lots of goodies!

I got already some questions about the original bridge and I was informed it is not sufficient anymore so I took some time to revisit the MQTT uplink and downlink support in TTN applications:

We will why this is still a solid solution but we will also look at a possible alternative.

Getting started with Azure IoT Plug and Play

Microsoft is the founder of this concept called Azure IoT Plug and Play:

IoT Plug and Play enables solution builders to integrate smart devices with their solutions without any manual configuration.

The idea is that the device describes itself using some identification key. This key, the Model ID (or DTMI, Device Twin Model Identification), is bound to a complete model written in DTDL (Digital Twins Definition Language).

Using this model, the interface (or capabilities) of this device can be read:

• Properties (Azure IoT Device Twin desired properties and reported properties)
• Telemetry (the D2C messages)
• Command methods (based on Azure IoT Direct methods)

Once a device starts communicating using this deterministic interface, a User Interface can be provided dynamically.

This is the same principle of Plug and Play devices like a mouse or a webcam. If you plug it in, the device is identified as a mouse or webcam and the correct device driver is downloaded from the internet and installed. In the time before Plug&Play, each device came with a CD or floppy disk containing the driver. This was always a hassle, Plug&Play has taken away that pain.

The actual model is stored in a global Device Models Repository. You can create your own repository too.

Look at this Azure IoT Plug&Play architecture/flow:

Here are see the different steps needed to build a solution based on Azure IoT Plug&Play:

1. Devices exposing their Model ID
2. IoT Hub storing the Model ID as a reference in the IoT Hub registry
3. Consuming this Model ID by other Azure resources
4. Looking up the actual Model in a Device Models Repository
5. Building up a tree of device capabilities based on the device model

Once the device capabilities are known, an actual UI can be generated for this device so users can interact with it without any extra effort.

Let’s check out how this works.

OPC-UA node browsing using Traeger SDK

OPC-UA is a modern protocol to unlock M2M communcation.

The OPC Unified Architecture (UA) is a platform independent service-oriented architecture that integrates all the functionality of the individual OPC Classic specifications into one extensible framework. Building on the success of OPC Classic, OPC UA was designed to enhance and surpass the capabilities of the OPC Classic specifications.

Its popularity is still growing in many markets and for multiple reasons!

Most importantly, from an IoT Developer view, the protocol supports devices to offer a secure communication layers and the exposed tags can be made human readable.

For example, this OPC-UA clients looks at the exposed tags of an OPC-UA server running on an Advantech Wise 710:

In this Prosys OPC-UA client, the tag values are shown:

The nodes are actually mapped Modbus values read from Wise 4012E.

As you can see, it exposes these six values. The potentio meter values shown is made available as a double.

I was asked to provide documentation like this, listing all exposed nodes together with the DataType.

Is it possible to browse nodes?

Getting weather information and predictions from Azure Maps API

If you think about an Azure service named Azure Maps, what is the main purpose that comes into your mind?

That should be map tiles, of course. And that is ok. Azure maps has many features for representing spatial data in all kinds of interactive maps.

Still, Azure Maps has much more to offer. There are also multiple APIs you can use to make calls against like Mobility or Weather.

The information used to create weather maps like these are also available for you using REST API calls:

Let’s check out how we can consume this interface.

Decompressing Azure IoT messages using Azure Stream Analytics

Azure IoT Edge messages are bound to a maximum size limitation of 256KB. Each message sent is divided into chunks of 4KB. The metering of an IoT Hub is based on these chunks.

So, if a message size of 10KB, it is counted as three separate messages.

Note: the chunk size of the IoT Hub free tier is smaller, just 0.5KB.

In a recent project, we were sending messages of more than 70KB. That means almost twenty chunks or even more in additional cases.

This is technically just fine but we were not feeling comfortable about this:

• This means going rapidly through the IoTHub quotum
• Messages were sent over a metered network

So this could become quite some investment in traffic and chunks.

I checked out if this is possible to limit this message size, there are several solutions.

But first, let’s see how to start compressing messages.

Supporting Azure Defender for IoT agent on IoT Edge device

More than two years ago, I wrote a blog post about introducing Azure cloud security on the Edge.

This, now labeled ‘classic’ version of Azure IoT Security, was based on AuditD and filled an important need: getting insight into the security of IoT devices.

Though, as with many things, the world moves on.

Microsoft reconsidered the solution and decided to spice it up a little bit.

They now offer a new edition supporting both an agent-based and an agent-less solution.

If you are interested (and you are when you own large networks with many devices!) in the agent-less solution, please check out this great demonstration on the Internet of Things show.

This agent-less solution is especially powerful in large solutions with many devices on the network:

You just install this Azure Defender for IoT ‘sensor’ device within the network and it starts inspecting that network for possible threats based on deep-packet inspection and updated threat-analysis logic coming from Microsoft.

In contrast, Here is shown how the current agent-based solution is rolled out:

As seen in the picture an agent, running as a daemon process on your (Linux) host (Azure RTOS is supported too), checks for possible vulnerabilities and passes it on to the cloud, to an IoT Hub.

There, the situation (and possible threats) is visualized in the Azure Defender for IoT portal panes.

When I checked out the documentation, I also expected support for Azure IoT Edge… but it seems there is no Azure IoT Edge Docker module support.

It only describes running that solution based on a daemon. This daemon is just a process that is secured using Azure IoT Device Module Identity credentials.

The identity is a separate security key apart from the Azure IoT device runtime security keys.

So, do we need to register a second Azure IoT Device, deployed side-by-side to an Azure IoT Device just to have the host secured?

Or, how can we integrate this daemon in a device running Azure IoT Edge?

Cloud IoT dashboards using Grafana with Azure IoT

Azure IoT offers a great solution for connecting IoT devices to the cloud and communicating with them in a secure way and in a two-way fashion: D2C and C2D.

Once you start ingesting telemetry you probably, at some point, want to represent the data in some kind of dashboarding.

This can either be a custom dashboard that gives you the most flexible way to represent the data. I have shown how to do this with Blazor. Or, you could choose PowerBI which is a well-known and productive tool used by many Data Scientists already.

Recently, our team invested some time in building dashboards using Grafana.

With Grafana you can create, explore and share all of your data through beautiful, flexible dashboards.

Azure supports Grafana in various ways in the Azure Marketplace.

For this blog post, I selected the official Grafana template which is hosted in a single VM:

The telemetry is ingested by an IoT Hub and send to a SQL Azure database using Azure Stream Analytics.

As you will see, this is quite an elaborate solution due to all the Azure resources being used.

Still, the solution is quite straightforward and certainly interesting if you are already familiar with Grafana.

Reference NuGet packages on GitHub Package Registry in Azure IoT Edge module

NuGet packages are a beautiful addition to your .Net development environment so you can add third-party libraries to your projects with ease.

For .NET (including .NET Core), the Microsoft-supported mechanism for sharing code is NuGet, which defines how packages for .NET are created, hosted, and consumed, and provides the tools for each of those roles.

Making use of a (privately hosted) NuGet feed is also the perfect way to distribute and version your own libraries within your company projects. Then, these libraries are only accessible by you and your friends.

Due to the fact, Azure IoT Edge modules are just docker containers, you can not simply reference other projects in the same solution. This is because of the separate docker container build and push mechanism outside the scope of the projects/solution.

Recently, I did some projects using (private) GitHub repositories to both generate, host, and consume NuGet packages.

Let’s check out how this works.

Can an Azure IoT Edge container be deployed in the cloud?

In the last couple of years, I wrote several blog posts about deploying standard, generic (third-party) Docker container or custom docker containers on the Azure IoT Edge.

Why? Because we can!

Azure IoT Edge is a great container delivery solution for docker containers and can be used in non-IoT-related solutions too.

I agree this is a little bit awkward. Azure IoT Edge has this relationship with IoT. Can we ignore the Internet of Things part?

Yes, we are able to control containers remotely. And as a bonus, IoT related telemetry can be gathered and IoT related logic can be excuted.

So, more generally, we see most of the times this deployment of a mix of both IoT-related modules and generic, but still IoT-related (third-party), modules.

Still, the magic of deploying whatever you want is a big surprise.

The same goes for the opposite idea. Can we roll out IoT Edge modules in the cloud, without the confinements of Azure IoT Edge?

This is an interesting question.

Deploy Prometheus container with Azure IoT Edge

Once you have set up your Azure IoT edge device and runtime, you want to make sure it keeps running as expected. So you need to start measuring how your device, your modules, and runtime are doing.

Microsoft provides built-in metrics for Azure IoT using endpoints on the EdgeHub and EdgeAgent exposing messages in the Prometheus format.

In a previous blog post, I showed how to access these endpoints and not to send messages containing this information to the cloud using a custom Azure IoT Edge metrics module.

Azure IoT Edge can deploy whatever Docker container you have. So, we can also use the original Prometheus service as a Docker container:

This way you can build a local dashboard using well-known tooling on the edge.

Let’s check out how this works.