Getting started with Azure Arc-enabled servers

Azure Arc, have you tried it out already?

The advertisement of Azure Arc is great:

Azure Arc enables you to extend Azure management capabilities to hybrid environments, including on-premises datacenters and third-party cloud providers. You can use Azure Arc to manage and configure your Windows and Linux server machines and Kubernetes clusters that are hosted outside of Azure. You can also use Azure Arc to introduce Azure data services to hybrid environments.

So, we can manage server machines, Kubernetes, and data services, all over the world as if these live in Azure:

Remote server management, this is a very interesting feature from an IoT point of view because an IoT Developer wants to have control over all edge devices without having to manage them only by visiting them.

Zero-touch maintenance, could that be done using Azure Arc?

Today, we will look at a small but very important part of Azure Arc, Azure Arc-enabled servers, so we can answer that question.

Let’s see how easy it is.

Connect ESP32 to EventGrid over MQTT with Nanoframework

Although I mainly work with Azure IoT Edge solutions, once in a while I program for constrained devices.

These devices are not extending the cloud but devices like the ESP32, are quite powerful in their own way and support wireless communication too.

This makes them perfect for IoT client solutions.

From all client programming platforms, I love to program using the Nanoframework platform the most.

It makes it easy to write C# code for embedded systems so even with limited embedded devices knowledge, developers are successful in building embedded systems.

Last year, I wrote an article about my Azure cloud-connected coffeemaker (a tribute). It was completely written in C#, complete with Azure IoT Hub integration, GPIO, and sound generated by an MP3 player.

For this blog post, I checked out how the Nanoframework can connect to the new MQTT support for the Azure EventGrid namespace:

Let’s see how we securely connect an ESP32 and have it communicate over plain MQTT.

User properties in Azure Stream Analytics, the smart way

The Azure Stream Analytics job is a great solution for routing, aggregating, enriching, and joining streams of Real-Time (IoT) Data.

It supports incoming and outgoing messages using the EventData format.

This means a message typically contains:

• Message body (typically a JSON message)
• System properties, a property bag (typically IoT Hub or Event Hub-related properties)
• User (aka Application) properties, a property bag (optionally)

Where system properties are only touched by the underlying Azure resources, application properties can be added by everyone, even the user.

This means both the (user) application used for transmitting the initial messages and any Azure service capable of transmitting these messages, like the IoT Hub with the message enrichment logic.

User properties are extremely helpful and one of the basic tools for any IoT developer:

• User properties give context to the message in the message body
• Using cloud-side enrichment, the data connection costs can be lowered.
• Routing is offered without touching (deserialization) the message body so both speed and ‘separation of concerns’ are offered (consistency)

Unfortunately, the Azure portal interface of the Azure Stream Analytics job is not the right tool for writing a job where these user properties are queried.

Let’s check out the better solution…

Azure Monitoring alerts based on Azure Data Explorer queries

As seen in many previous blog posts, Azure Data Explorer is my ‘Swiss army knife’ for IoT solutions.

It serves both querying ingested (IoT) time series data for warm path dashboarding, providing long-term raw data storage for the cold path, and supporting exploratory querying the long-term data for data science.

On the other hand, one of the main features of an IoT solution is coping with data in motion. Telemetry from devices streams through the solution and ends somewhere in storage.

The challenge is to keep the flow going, based on queries searching for eg. tipping point changes, anomalies, or complex data joining.

Unfortunately, Azure Data Explorer does not support DB triggers like CosmosDB or SQL Server.

While the streaming data is persisted in the Azure Data Explorer Time-Series database, this is still not the end of the ‘streaming’ story.

Of course, we could create an Azure Function that queries the database on a regular time interval.

Recently, a new solution has been offered based on Azure Monitoring:

You can correlate data in Azure Data Explorer and Azure Resource Graph with data in your Log Analytics workspace and Application Insights resources to enhance your analysis in Azure Monitor Logs.

Technically, you could even join data living in an Azure Log Analytics Workspace with data living in Azure Data Explorer (cross-service queries) or Azure Resource Graph:

Let’s check out how this feature can be used.

Microsoft Fabric real-time analytics exploration: EventGrid MQTT Broker integration

Microsoft Fabric is an all-in-one analytics solution for enterprises that covers everything from data movement to data science, Real-Time Analytics, and business intelligence.

For IoT Developers, this is a great addition to our Azure IoT resources toolkit because ingesting streams of IoT data is supported too.

In this series of blog posts, we have already learned about integrating with the Azure IoT Hub.

For my recent presentation at the Data Toboggan Winter edition, I wanted to integrate via the MQTT protocol using the new Azure EventGrid MQTT support:

In this blog post, we learn how to integrate using a little extra help offered by an Azure Function.

Microsoft Fabric Real-Time Analytics presentation at Data Toboggan 2024 Winter edition

Update: The event was a big success. I want to thank everyone for the kind words regarding my presentation. The recording is available below.

Join me at the free Data Toboggan Winter Edition event on Saturday, February 3, 2024, and learn about Microsoft Fabric Real-Time Analytics:

Data Toboggan is an Azure Synapse Analytics-focused series of events. The events and user group pages contain a list of our events.

This is the Winter edition, offering a full day of free presentations.

A free online presentation about the Azure EventGrid MQTT support on January 18th (recorded)

Update: the presentation has finished. You can watch the recording of the NottsIoT event below.

Join me on January 18 2024 at 19:00 GMT, for my online Notts IoT presentation “Fancy device toppings on a vanilla MQTT broker” where I demonstrate the new Azure EventGrid MQTT broker features:

The presentation is packed with demonstrations which is always fun and challenging.

Join our Free in-person Azure IoT and Azure Data Explorer workshop at BitBash.nl

Together with my colleague Wilko van de Velde, we are hosting the in-person “Experience the basics of Azure IoT, Dashboarding … and beyond” workshop at the next BitBash event on January 26, Veenendaal, The Netherlands.

Check out the details on how to get a free ticket here.

Microsoft Fabric real-time analytics exploration: Home Assistant integration (Christmas edition)

Christmas time seems to be the most wonderful time of the year for IoT Developers.

The combination of both having some spare time and getting geeky gifts (or buying them for yourself) makes it the perfect moment to do some home automation.

Home automation means connecting everything in your home, your garden, etc. Why? Because we can!

So, let’s combine your home automation with Microsoft Fabric!

Microsoft Fabric is an all-in-one analytics solution for enterprises that covers everything from data movement to data science, Real-Time Analytics, and business intelligence.

For IoT Developers, this is a great addition to our Azure IoT resources toolkit because ingesting streams of IoT data is supported too.

What if we could record and query home automation events:

Home Assistant is one of the most popular Open source home automation platforms to automate your own crib.

Let’s see how we can enrich this platform using the Azure Event Hub support.

Can we connect to Microsoft Fabric?

Microsoft Fabric real-time analytics exploration: Managed Grafana integration

Microsoft Fabric is an all-in-one analytics solution for enterprises that covers everything from data movement to data science, Real-Time Analytics, and business intelligence.

For IoT Developers, this is a great addition to our Azure IoT resources toolkit because ingesting streams of IoT data is supported too.

Until now, we have only seen some basic dashboarding using eg. Power BI or rendering in KQL databases. Because building PowerBI dashboards (eg. with KPI or OEE reports) is normally out-of-scope for IoT Developers, this is OK for now.

On the other hand, IoT Developers are always interested in more real-time visualization, and many of you IoT Developers are using Managed Grafana as a dashboard solution:

Although it is not very well documented, it is possible to connect Azure Managed Grafana to Microsoft Fabric via the KQL Database in a kind of experimental way.

Let’s see how this is done.