Azure Data Explorer connector for Blob storage (IoT Hub) files

Over the last couple of months, I have written several blog posts regarding Azure Data explorer ingestion connectors.

I checked out the IoT Hub connector, dynamic mapping, and table update policies.

Until now, this is all based on both the IoT Hub connector and the Event Hub connector:

Next to those two, Azure Data Explorer also supports ingesting complete files using the third connector: Blob Storage ingestion in combination with EventGrid events:

Let’s check out what this connector offers and how to set it up.

As we will find out, it even has a surprise for those of you ingesting IoT Hub messages.

Doorgaan met het lezen van “Azure Data Explorer connector for Blob storage (IoT Hub) files”

Event Grid custom Event Source and Handler

Azure Event Grid is one of (many) event mechanisms available in the Azure cloud.

Azure Event Grid works with these moving parts:

  • Event sources. Some logic sending messages
  • Event handlers. The logic consuming incoming messages
  • Event Grid topics. the mechanism receiving incoming messages and sending them to subscribed event handlers
  • Event Grid subscriptions. The connection between topic and handlers. Offers filtering and additional features

Note: in this post, we focus on Azure Event Grid. Check out this MS Learn module for comparison with Azure Event Hub and Azure Message Bus.

Azure Event Grid support is offered for a growing number of Azure resources as an extra event mechanism, out-of-the-box. The resource could either be a source or handler or both.

For your convenience, we can also create custom events to custom topic and consume them:

It must be clear that Event Grid uses the fan-in-fan-out pattern, also seen in Event Hubs.

Multiple Event sources can send messages on the same topic. From there, multiple Event handlers can pick their own copy of incoming messages using its subscription. Filtering can be applied if needed.

We ignore the predefined event sources and focus on a custom one to explain the flow.

Doorgaan met het lezen van “Event Grid custom Event Source and Handler”

Subscribe your IoTHub to EventGrid as event source

Azure has all sorts of Publisher/Subscriber resources available to allow sending and receiving messages (events) within Azure. Think of the Service Bus, EventHub, Storage Queue, Stream Analytics, Azure Functions, etc.

At some point, Microsoft realized that a more formal way of the eventing abilities of these resources was needed.

This let to the introduction of the EventGrid, one overall solution to send and receive messages. Azure Event Grid relatively new.

The Azure Event Grid allows you to easily build applications with event-based architectures

The Azure resources are divided into two groups:

  • Event sources. Resources which produce events (or pass-through events)
  • Event handlers. Resources which are able to pick up events and do something with them

These resources all come together in the Event Grid.

Event Grid model of sources and handlers

The Azure IoT Hub can act like an Event Source too. The events exposed are:

Let’s see how we can put this to use.

Doorgaan met het lezen van “Subscribe your IoTHub to EventGrid as event source”