Microsoft Fabric Real-Time Analytics presentation at Data Toboggan 2024 Winter edition

Update: The event was a big success. I want to thank everyone for the kind words regarding my presentation. The recording is available below.

Join me at the free Data Toboggan Winter Edition event on Saturday, February 3, 2024, and learn about Microsoft Fabric Real-Time Analytics:

Data Toboggan is an Azure Synapse Analytics-focused series of events. The events and user group pages contain a list of our events.

This is the Winter edition, offering a full day of free presentations.

Microsoft Fabric real-time analytics exploration: Home Assistant integration (Christmas edition)

Christmas time seems to be the most wonderful time of the year for IoT Developers.

The combination of both having some spare time and getting geeky gifts (or buying them for yourself) makes it the perfect moment to do some home automation.

Home automation means connecting everything in your home, your garden, etc. Why? Because we can!

So, let’s combine your home automation with Microsoft Fabric!

Microsoft Fabric is an all-in-one analytics solution for enterprises that covers everything from data movement to data science, Real-Time Analytics, and business intelligence.

For IoT Developers, this is a great addition to our Azure IoT resources toolkit because ingesting streams of IoT data is supported too.

What if we could record and query home automation events:

Home Assistant is one of the most popular Open source home automation platforms to automate your own crib.

Let’s see how we can enrich this platform using the Azure Event Hub support.

Can we connect to Microsoft Fabric?

Microsoft Fabric real-time analytics exploration: Managed Grafana integration

Microsoft Fabric is an all-in-one analytics solution for enterprises that covers everything from data movement to data science, Real-Time Analytics, and business intelligence.

For IoT Developers, this is a great addition to our Azure IoT resources toolkit because ingesting streams of IoT data is supported too.

Until now, we have only seen some basic dashboarding using eg. Power BI or rendering in KQL databases. Because building PowerBI dashboards (eg. with KPI or OEE reports) is normally out-of-scope for IoT Developers, this is OK for now.

On the other hand, IoT Developers are always interested in more real-time visualization, and many of you IoT Developers are using Managed Grafana as a dashboard solution:

Although it is not very well documented, it is possible to connect Azure Managed Grafana to Microsoft Fabric via the KQL Database in a kind of experimental way.

Let’s see how this is done.

Microsoft Fabric real-time analytics exploration: Eventstream Destination for Custom App egress

Microsoft Fabric is an all-in-one analytics solution for enterprises that covers everything from data movement to data science, Real-Time Analytics, and business intelligence.

For IoT Developers, this is a great addition to our Azure IoT resources toolkit because ingesting streams of IoT data is supported too.

In the previous post, we saw how real-time data ingestion from a Custom App via Eventstreams is supported.

Microsoft Fabric also supports emitting real-time events to non-fabric services via a Custom App:

This way, ingested event stream data can be used by other Azure services living outside Microsoft Fabric.

Let’s check this out.

Microsoft Fabric real-time analytics exploration: Eventstream Source for Custom App ingress

Microsoft Fabric is an all-in-one analytics solution for enterprises that covers everything from data movement to data science, Real-Time Analytics, and business intelligence.

For IoT Developers, this is a great addition to our Azure IoT resources toolkit because ingesting streams of IoT data is supported too.

In the last couple of posts, we focussed on ingesting telemetry ingested from an IoT Hub.

Microsoft Fabric also supports multiple ways of ingesting telemetry coming from a custom app:

Let’s check out what a custom app is for Microsoft Fabric, how ingestion works, and compare it with another way of ingestion.

Microsoft Fabric real-time analytics exploration: KQL Database mirroring

Microsoft Fabric is an all-in-one analytics solution for enterprises that covers everything from data movement to data science, Real-Time Analytics, and business intelligence.

For IoT Developers, this is a great addition to our Azure IoT resources toolkit because ingesting streams of IoT data is supported too.

In previous blog posts, we have seen how data can be streamed to eg. KQL databases or Lakehouses.

Today, we look at providing access to KQL database tables by mirroring the data via OneLake:

This way, already ingested data in a KQL Database is made available as one logical copy in OneLake.

Let’s see how this is done.

Microsoft Fabric real-time analytics exploration: Streaming to a Lakehouse

Microsoft Fabric is an all-in-one analytics solution for enterprises that covers everything from data movement to data science, Real-Time Analytics, and business intelligence.

For IoT Developers, this is a great addition to our Azure IoT resource toolkit because streaming IoT data via IoT Hub integration is supported too.

Next to warm (and potentially cold) path storage, Microsoft Fabric offers a solution to react to certain incoming data, the hot path.

In this post, we try to learn how data engineers could process and analyze our IoT data stream:

Today, we dive into the world of Microsoft Fabrick Lakehouses.

Microsoft Fabric real-time analytics exploration: Activating IoT device reflex data

Microsoft Fabric is an all-in-one analytics solution for enterprises that covers everything from data movement to data science, Real-Time Analytics, and business intelligence.

For IoT Developers, this is a great addition to our Azure IoT resource toolkit because streaming IoT data via IoT Hub integration is supported too.

Next to warm (and potentially cold) path storage, Microsoft Fabric offers a solution to react to certain incoming data too:

Let’s check out this new Microsoft Fabric Data Activator.

Microsoft Fabric real-time analytics exploration: Ingesting IoT device events

Microsoft Fabric is an all-in-one analytics solution for enterprises that covers everything from data movement to data science, Real-Time Analytics, and business intelligence. 

For IoT Developers, this is a great addition to our Azure IoT resource toolkit because streaming IoT data via IoT Hub integration is supported too:

Let’s check out how an IoT Hub device can be integrated with the event streaming capabilities of Microsoft Fabric while transforming ingested messages to our needs.

Microsoft Fabric real-time analytics exploration: Ingesting streams of sample events

Microsoft Fabric is an all-in-one analytics solution for enterprises that covers everything from data movement to data science, Real-Time Analytics, and business intelligence. 

For IoT Developers this is a great addition to our Azure IoT resource toolkit because streaming data is supported:

Let’s check out the event streaming capabilities of Microsoft Fabric.