How to cope with IoT Hub enrichment restrictions

As seen in my previous post, The IoT Hub routing feature supports message enrichment, both for IoT devices and IoT edge modules.

Using the routing message enrichments, each incoming message gets extra user properties based on either static values, device twin tags, or device twin desired properties.

Unfortunately, only ten enrichments can be added:

If you want to pass more values, this will not work for you.

It would be great if nested JSON properties would count as one.

Again, unfortunately, only simple types (string, decimal, boolean, date/time, etc.) are supported so this excludes nested JSON (complex types).

Below, a viable solution to overcome both restrictions, using Azure Stream Analytics, is presented.

Let’s see how this works out.

Advertentie

Creating an Azure Stream Analytics job using VS Code

Azure Stream Analytics is often the centerpiece of our IoT solutions.

It acts like a rule engine where data streams from multiple sources can be combined together, even enriched with static reference data.

Azure Stream Analytics does not come cheap if you only want to do some simple aggregations. For that, probably Azure Functions can help you out.

But, when it comes to more elaborate rules using multiple inputs, multiple outputs, time windowing, custom functions, and Machine learning integration, Azure Stream Analytics (and many more capabilities) should be your first choice:

Normally, I demonstrate Azure Stream Analytics using the Azure Portal.

There, it offers me a convenient browser experience where I can show how inputs, outputs, and user-defined functions are created. I can also copy/paste a (basic) query and demonstrate how it can be tested and run in a simple manner:

For people new to Azure Stream Analytics, this is a perfect starting point.

Still, this is for demonstration purposes only!

The Azure portal lacks (professional) abilities like source control/versioning, superior user-defined functions, and diagnostics.

If you plan to use Azure Stream Analytics in your projects, please consider starting using the VS Code project template.

In this blog, we will see how to start with the Visual Studio Code project for Stream Analytics.

User Defined Function in Stream Analytics

Azure Stream Analytics provides a great solution for temporal queries of streams of data. The query language is pretty simple, especially if you have a background in SQL queries.

The list of built-in functions is a long list, ranging from aggregation, analytics, geospatial, records, scalars to the recently introduced anomaly detection.

But what if you want to write your own functions?

Stream Analytics supports three types of custom functions:

1. user-defined functions (UDF) written in Javascript
2. user-defined aggregates (UDA) written in Javascript
3. Machine learning endpoint disguised as functions

In this blog, I will show how easy it is to write and use your own custom logic in a Stream Analytics job. We will look at the user-defined functions.