Flex your Big Data detective skills with the Kusto detective agency challenge

Flex your Big Data detective skills with multiple cases of the Kusto detective agency challenge.

Solve puzzles using the Kusto Query language and earn awards.

Are you interested?

Doorgaan met het lezen van “Flex your Big Data detective skills with the Kusto detective agency challenge”

Azure Data Explorer connector for Blob storage (IoT Hub) files

Over the last couple of months, I have written several blog posts regarding Azure Data explorer ingestion connectors.

I checked out the IoT Hub connector, dynamic mapping, and table update policies.

Until now, this is all based on both the IoT Hub connector and the Event Hub connector:

Next to those two, Azure Data Explorer also supports ingesting complete files using the third connector: Blob Storage ingestion in combination with EventGrid events:

Let’s check out what this connector offers and how to set it up.

As we will find out, it even has a surprise for those of you ingesting IoT Hub messages.

Doorgaan met het lezen van “Azure Data Explorer connector for Blob storage (IoT Hub) files”

Using ADX table update policies projecting raw messages to target tables

I have blogged about Azure Data Explorer (ADX) integration already a couple of times.

The new ADX database data connection for IoT Hub is a nice addition but appears to be quite static because you can only fill in one (single) table mapping.

ADX also supports dynamic routing (using that same data connection) where the message body is accompanied by application/user properties which tell ADX the table and mapping to use during ingest.

The ultimate goal is to have a solution where one stream of data, with one or more types of messages, is split into multiple tables:

The combination of IoT Hub routing and multiple endpoints should help with that but due to message enrichment limitations, we need to find another way.

A viable solution is the ADX table update policy.

Let’s check it out.

Doorgaan met het lezen van “Using ADX table update policies projecting raw messages to target tables”

Dynamic routing of IoT Hub telemetry to Azure Data Explorer

Azure Data Explorer (ADX) is a great data exploration tool for IoT developers building a full IoT solution. This could be a perfect target for the cold path.

As seen in my previous blog post, ADX even offers a native connector for the IoT Hub. This is based on the ‘default EventHub compatible endpoint’ offered by this cloud IoT gateway (optionally the built-in Events endpoint in the routing section of the IoT Hub or using the fallback mechanism).

Most of the documentation regarding this ADX connector is following this ‘happy flow’ where one connector stores incoming IoT telemetry in one ADX table using static routing.

This is a serious limitation where most IoT Hubs ingest multiple types of messages. These will not fit into that single table.

Luckily, the connector also offers the possibility to allow routing to other databases:

Here, we will check out this dynamic routing option and see how this provides much more flexibility.

Doorgaan met het lezen van “Dynamic routing of IoT Hub telemetry to Azure Data Explorer”

Azure Data Explorer connector for IoT Hub with real-time ingest

Azure Data Explorer (ADX) is a fully managed, high-performance, big data analytics platform that makes it easy to analyze high volumes of data in near real-time. The Azure Data Explorer toolbox gives you an end-to-end solution for data ingestion, query, visualization (including third-party dashboards), and management.

ADX can ingest data both from (persisted) storage and data provided as a stream:

Ingested data is managed and optimized by the underlying cluster so it can be queried and made available in (third-party) platforms as a view (like Power BI, Grafana, etc.) based on a powerful query language.

Data can also be ingested directly from Azure IoT Hub, as a stream for real-time analysis.

Let’s check out how this works.

Doorgaan met het lezen van “Azure Data Explorer connector for IoT Hub with real-time ingest”