Streaming data analysis with Free Azure Data Explorer

If you are new to Azure Data Explorer, a simple and cheap way to start is the Free Azure Data Explorer cluster.

There are a number of limitations compared to the paid version of Azure Data Explorer. But hey, you get a Time Series database cluster with data ingestion, KQL querying, and dashboarding for free!

Personally, I moved over to the paid dev/test version because I work a lot with streaming Azure IoT telemetry ingest and this kind of data ingestion was not offered in the free version.

Until last week!

Although a direct data connection to the Azure IoT Hub or ingestion via Event Grid is not offered, we can now ingest streaming data via Event Hub.

This brings us to the following architecture:

This post will explore how to ingest streaming data using the new Event Hub support.

Advertentie

Programmatically ingest data into Azure Data Explorer

What is the value of a database if adding data is hard?

Over the last year, I played around with ingesting data into Azure Data Explorer in multiple ways.

Most of the time, I used the managed pipelines, aka ADX Database data connections.

But every time, I find new ways to ingest data into database tables:

This time, we look at ingesting data into Azure Data Explorer using the SDKs.

Using Azure Logic apps as business rules for Azure Digital Twins

Although this feature just went into Public Preview, the new Azure Digital Twins connector for Microsoft Power Platform is very promising:

Azure Digital Twins (ADT) connector for Microsoft Power Platform enables you to incorporate Azure Digital Twins into Microsoft Power Automate flows, Power Apps applications, or Azure Logic Apps flows. 

With this connector, you can combine Azure Digital Twins with 700+ other Power Platform connectors to build flows or apps by ingesting data, coming from other systems, into the Azure Digital Twins environment.

You can do graph traversal:

Or, you can send data to Azure Digital Twins (twin update) as seen here:

It’s even possible to develop a flow that creates a digital twin, deriving from a model when an external system emits an event.

Let’s check out what this Azure Logic Apps connector offers for Azure Digital Twins.