Reference NuGet packages on GitHub Package Registry in Azure IoT Edge module

NuGet packages are a beautiful addition to your .Net development environment so you can add third-party libraries to your projects with ease.

For .NET (including .NET Core), the Microsoft-supported mechanism for sharing code is NuGet, which defines how packages for .NET are created, hosted, and consumed, and provides the tools for each of those roles.

Making use of a (privately hosted) NuGet feed is also the perfect way to distribute and version your own libraries within your company projects. Then, these libraries are only accessible by you and your friends.

Due to the fact, Azure IoT Edge modules are just docker containers, you can not simply reference other projects in the same solution. This is because of the separate docker container build and push mechanism outside the scope of the projects/solution.

Recently, I did some projects using (private) GitHub repositories to both generate, host, and consume NuGet packages.

Let’s check out how this works.

Can an Azure IoT Edge container be deployed in the cloud?

In the last couple of years, I wrote several blog posts about deploying standard, generic (third-party) Docker container or custom docker containers on the Azure IoT Edge.

Why? Because we can!

Azure IoT Edge is a great container delivery solution for docker containers and can be used in non-IoT-related solutions too.

I agree this is a little bit awkward. Azure IoT Edge has this relationship with IoT. Can we ignore the Internet of Things part?

Yes, we are able to control containers remotely. And as a bonus, IoT related telemetry can be gathered and IoT related logic can be excuted.

So, more generally, we see most of the times this deployment of a mix of both IoT-related modules and generic, but still IoT-related (third-party), modules.

Still, the magic of deploying whatever you want is a big surprise.

The same goes for the opposite idea. Can we roll out IoT Edge modules in the cloud, without the confinements of Azure IoT Edge?

This is an interesting question.

Deploy Prometheus container with Azure IoT Edge

Once you have set up your Azure IoT edge device and runtime, you want to make sure it keeps running as expected. So you need to start measuring how your device, your modules, and runtime are doing.

Microsoft provides built-in metrics for Azure IoT using endpoints on the EdgeHub and EdgeAgent exposing messages in the Prometheus format.

In a previous blog post, I showed how to access these endpoints and not to send messages containing this information to the cloud using a custom Azure IoT Edge metrics module.

Azure IoT Edge can deploy whatever Docker container you have. So, we can also use the original Prometheus service as a Docker container:

This way you can build a local dashboard using well-known tooling on the edge.

Let’s check out how this works.

May 25-27: Build 2021 IoT related sessions

Microsoft Build 2021 is a virtual technology conference scheduled for May 25-27. Microsoft Build is designed mainly for developers who write and support applications for Microsoft Azure, Windows, SQL Server and other applications/software platforms from the company.

Because it’s an digital event (just like last year) registration is free.

Of course, there are also Internet of Things (IoT) related sessions.

Normally, most of these sessions (Except the RSVP ones where typically involve customer interaction) are recorded and available online for registered users. So, that is the second reason to register 🙂

Here is a list of what is scheduled right now regarding IoT:

Azure IoT Edge UDP Client module

Over the past couple of years, I’ve written several blog posts about supporting protocols on Azure IoT Edge.

We have seen examples like Serial communication, Modbus TCP and RTU, OPC-UA, and WebSockets.

This time, a new protocol is added to this impressive list: User Datagram Protocol (UDP).

Communication using UDP is fast and supports broadcasting to several IP addresses at once.

There are some disadvantages (it has no handshaking dialogues, and thus exposes the user’s program to any unreliability of the underlying network; there is no guarantee of delivery, ordering, or duplicate protection) but if you can cope with that at a higher level, UDP is a handy tool in your protocol toolbox.

In this blog post we will see how UDP messages can be received on the Edge with this demonstration IoT Edge module:

Because each UDP message is different (here we send text messages but this could also be binary values, depending on your challenges), this module is just a reference which explains how it works.

Message enrichment for Azure IoT Edge modules

The Azure IoT Hub offers ingestion of D2C and D2C messages for thousands or more devices.

Incoming messages are routed to several endpoints (eg. Event Hub or Blob Storage) using internal IoT Hub routes:

Note: even messages sent to the Event Grid are also enriched.

The messages produced by devices look like these (image taken from the console output of the Microsoft temperature simulation):

Here, messages from the Microsoft temperature simulation module are sent to the cloud.

When these messages arrive in the IoT Hub, the fulll message format looks like this:

{
  "body": {
    "machine": {
      "temperature": 73.05171716227275,
      "pressure": 6.929942461524743
    },
    "ambient": {
      "temperature": 20.795734739068774,
      "humidity": 26
    },
    "timeCreated": "2021-05-10T11:51:42.1835163Z"
  },
  "enqueuedTime": "Mon May 10 2021 13:51:42 GMT+0200 (Central European Summer Time)",
  "properties": {
    "sequenceNumber": "101",
    "batchId": "7f3f847f-c20c-49aa-af17-a22db31f244f"
  },
  "systemProperties": {
    "iothub-connection-device-id": "simulation",
    "iothub-connection-module-id": "sim",
    "iothub-connection-auth-method": "{\"scope\":\"module\",\"type\":\"sas\",\"issuer\":\"iothub\",\"acceptingIpFilterRule\":null}",
    "iothub-connection-auth-generation-id": "637540164221770949",
    "iothub-enqueuedtime": 1620647502173,
    "iothub-message-source": "Telemetry",
    "x-opt-sequence-number": 13975,
    "x-opt-offset": "12885042704",
    "x-opt-enqueued-time": 1620647502314
  }
}

As you can see, the messages are build up in three parts:

1. The actual message body, generated by (your) logic in the device. This part could be encoded so only the receiver knows how to handle this. Normally this is a JSON message format, though.
2. (Application) properties, message enrichment on the device to provide context about the messages. This acts as a property bag, also added by the developer. The values are directly accessible for routing purposes.
3. System properties, added by the IoT Hub. This gives technical context about the incoming message from the perspective of the IoT Hub

So users can choose to add values in the body or to add values in the application properties.

Still, every byte put in the message sent by the device results in a longer transmit duration and extra costs (number of bytes communicated, potentially more message chunks handled by the IoT Hub).

So, values already found in the system properties (eg. the device name or module id) can be left out in the body. This results in a smaller message.

Microsoft also provides message enrichment on the IoT Hub. This can take away some of the pain too.

It also makes it possible to route messages further down the road in other Azure resources which receive these enriched messages.

Let’s check this out.

How to cope in code with the quota limitation of an IoT Hub

In a previous blog, I showed how to make use of the Azure IoT Device SDK to send data to an Azure IoT Hub.

It is very simple to start with azure IoT making use of those SDKs using either C#, Embedded C, C, Java, NodeJS, Python, or ioS.

The only thing you have to take into account is the IoT Hub message quotas.

An S1 version is capable of receiving 400.000 D2C messages a day of size 4KB, the free tier F1 supports 8000 messages of size 0.5 KB.

Sending C2D messages and Device provisioning is also part of this amount.

Let’s take a further look at this limitation.

Azure Stream Analytics anomaly detection on the edge

Back in 2018, the Azure Stream Analytics team announced Anomaly detection for Azure Stream Analytics. And, it was also supported on the Azure IoT Edge. In November 2018, I was allowed to demonstrate this at the SPS IPC Drives in Nuremberg:

Since then, anomaly detection has become a first citizen of Azure Stream Analytics.

Azure Stream Analytics Anomaly Detection is able to ‘automatically’ detect spikes, dips, and trends in a stream of values. This is based on math and all you need to do is to specify how many values you expect and how sensitive the detection must be. It is in fact Machine Learning, in the end. So it is a prediction with a certain certainty…

And it’s just part of the Stream Analytics query language. So on the edge, we can deploy a Stream Analytics module (the engine of stream analytics). This is a fixed module from Microsoft.

All you have to do is feeding it a query, inputs, outputs and, user-defined functions if available:

We can define all this in the cloud, inside a stream analytics job and we can even test it. Those parts are then packed as a blob and put in blob storage. The Stream Analytics job can then download it and run it.

The inputs and outputs of the Stream Analytics job can be attached to the normal Azure IoT Edge routing mechanism.

Let’s see how this is set up:

IoT on Microsoft Ignite, March 2021

Although 2021 seems to be a copy of 2021, the Microsoft Ignite has brought us a lot of possitive new ragarting new Microsoft technology we can all wait for. Or even better, it’s available right now.

The biggest Azure IoT on the current Ignite is Azure Percept:

 a comprehensive, easy-to-use platform with added security for creating responsible edge AI solutions. With Azure Percept, we have simplified AI on the edge from silicon to service

Documentation can be found here. The dedicated hardeware (Azure Percept DK is designed as a reference implementation) can be found here. (Datasheet DK | Datasheet Vision) (Availability around the world)

But there were also smaller gems. On of them is “30 days to learn it“. You can complete the IoT Developer challenge and get 50 percent off the cost of a Microsoft Certification:

Implement and maintain the cloud and edge components of an IoT solution. Learn how to connect devices, move workloads to the edge, and build scalable, enterprise-grade, secured IoT solutions on Azure to increase performance, reduce costs, and optimize operations. It’ll only take about 15 hours to discover how to securely connect IoT devices to the cloud, build the intelligent edge, and develop solutions with Azure IoT Central.

There are more certificates to earn. See the site for all eight challenges!

All sessions are available now in the session catalog for you to watch. Below, I summerized all IoT related sessions.

Using Influx database and Grafana on the Azure IoT Edge

Over the last couple of years, in this blog, multiple databases are demonstrated which can be deployed on the Azure IoT Edge:

• SQL Server
• MySql
• CrateDB
• SQLite

Persisting incoming telemetry in a local database is useful for multiple purposes. One of them is creating a custom dashboard eg. written in Blazor.

In this blog, we explore how to deploy the popular InfluxDB. This open-source time-series database is specialized in Internet of Things usage. And on top of that, let’s explore how Grafana can be deployed on the edge too. Grafana is an open source analytics and interactive visualization web application.

We see how it can be connected to the Influx database:

As you see here, we need a custom Azure IoT Edge module (called ‘writer’) that is capable of writing incoming telemetry to the Influx database. There, the telemetry is picked up and displayed by Grafana.

Let’s see how this works.