Test KQL table mappings inline

Azure Data Explorer makes it possible to ingest data from external sources in many ways.

This can be done using eg. the database data connections, using programming code via the SDKs, or using the Azure Stream Analytics output for Azure Data Explorer.

Once a table is created and the incoming file/message format (CSV, JSON, etc) and data structure in the file or message stream are identified, a recurring theme is designing the table mapping for that format:

Here, a typical mapping for a JSON message is shown.

Testing a mapping can be cumbersome if you rely on that default ingest method involving those resources outside Azure Data Explorer.

Can this be done in a more easy way?

Advertentie

Azure IoT PnP from a device client perspective

In the past, this blog has produced a number of posts on the Azure IoT Plug-and-Play (IPNP) topic.

The idea with IPNP is that incoming messages, coming from devices, provide context regarding the format of that message.

If messages follow a certain template, an interface or contract is offered. This makes it easier to exchange messages in the longer term.

For an overview of IoT Plug-and-Play check out this previous post.

Today, I want to look at IPNP from a device client perspective.

All IoT sessions at the Build 2023 event

This May 23–25, the latest Build event is held in Seattle.

While Build presents an opportunity for developers, students, engineers, engineers, and technology professionals to learn more about Microsoft products, it’s also where Microsoft shares major updates for Windows and its suite of productivity apps.

Yes, this is a hybrid event so you can join in-person and on-line.

For those joining in-person, there is a pre-build day on the 22nd with extra workshops.

Please register here to be part of this event.

For developers using Microsoft technology, it’s a great opportunity to see which direction Microsoft is going and to decide in which direction your career is going.

I arranged a long list of Azure IoT-related sessions.

There are many new topics so if you are Azure IoT-minded, this is the Build event for you!

Streaming data analysis with Free Azure Data Explorer

If you are new to Azure Data Explorer, a simple and cheap way to start is the Free Azure Data Explorer cluster.

There are a number of limitations compared to the paid version of Azure Data Explorer. But hey, you get a Time Series database cluster with data ingestion, KQL querying, and dashboarding for free!

Personally, I moved over to the paid dev/test version because I work a lot with streaming Azure IoT telemetry ingest and this kind of data ingestion was not offered in the free version.

Until last week!

Although a direct data connection to the Azure IoT Hub or ingestion via Event Grid is not offered, we can now ingest streaming data via Event Hub.

This brings us to the following architecture:

This post will explore how to ingest streaming data using the new Event Hub support.

Programmatically ingest data into Azure Data Explorer

What is the value of a database if adding data is hard?

Over the last year, I played around with ingesting data into Azure Data Explorer in multiple ways.

Most of the time, I used the managed pipelines, aka ADX Database data connections.

But every time, I find new ways to ingest data into database tables:

This time, we look at ingesting data into Azure Data Explorer using the SDKs.

Using Azure Logic apps as business rules for Azure Digital Twins

Although this feature just went into Public Preview, the new Azure Digital Twins connector for Microsoft Power Platform is very promising:

Azure Digital Twins (ADT) connector for Microsoft Power Platform enables you to incorporate Azure Digital Twins into Microsoft Power Automate flows, Power Apps applications, or Azure Logic Apps flows. 

With this connector, you can combine Azure Digital Twins with 700+ other Power Platform connectors to build flows or apps by ingesting data, coming from other systems, into the Azure Digital Twins environment.

You can do graph traversal:

Or, you can send data to Azure Digital Twins (twin update) as seen here:

It’s even possible to develop a flow that creates a digital twin, deriving from a model when an external system emits an event.

Let’s check out what this Azure Logic Apps connector offers for Azure Digital Twins.

Another year, another AZ-220 Azure IoT Developer Specialty certification renewal

Did you know that Microsoft role-based and specialty certifications expire every year unless they’re renewed?

I got the question to renew my certification at no cost without having to schedule or pass a full exam and keep my skills current and my certification active for another year. (For details, read Renew your Microsoft Certification).

This is not the first time I get this email. Last year, I got the same question and had to take the exam regarding Azure Digital Twins.

Here is some background information about this year’s renewal (without any spoilers).

Azure Logic App actions for Azure Data Explorer

Azure Logic Apps are a convenient solution to add workflow logic, decision-making logic, and rules to your Azure solution.

There are hundreds of connectors with connections to all kinds of both Microsoft and third-party resources so you can literally connect the whole world and move data and insights from one place to another.

Azure Logic Apps also supports Azure Data Explorer, the queryable time-series data solution:

As you can see, you can execute both commands (for altering resources inside Azure Data Explorer) and table queries.

It even supports promises support for rendering charts!

In this post, we check out how to make use of these actions and charts so we can monitor and even control our Azure Data Explorer.

Does your Azure IoT (Edge Ubuntu) device survive the Baltimore certificate migration?

Microsoft is still using the Baltimore certificate for its IoT services. This certificate is used for TLS communication with the IoT Hub and other IoT-related Azure services.

This is done by choice, to give users more time to migrate their devices if they are not (yet) able to support the new DigiCert Global Root G2 certificate automatically.

This year, starting next month, Microsoft will start the migration.

Check out this timeline:

This has been postponed a number of times already. It’s now time to act!

If both your Azure edge devices and Azure IoT edge devices are running Ubuntu, here are some pointers to test your devices.

Azure IoT Community, unified edge event with Microsoft Nederland

Finally, after more than two years of absence due to the global pandemic, the Dutch Azure IoT Community is offering another event related to Azure IoT Topics.

This time, our location sponsor is Microsoft and the event is hosted at their HQ in The Netherlands, near Schiphol.

The Microsoft Technology Center is hosting a Hybrid Computing Event with the Azure IoT Community focusing on Edge and IoT.

We have 25 physical spots available and of course, people are welcome online after registration.