The only thing standing between IoT Hub and Functions are Consumers groups

Traditionally, when telemetry arrived at an Azure IoT Hub, it was a rule of thumb to pass it to a Stream Analytics job. But when routing was introduced in the IoT Hub, it became possible to push the telemetry directly into an EventHub or a Service Bus too.

I still favor the many possibilities of the Stream Analytics job, working with temporal and geospatial data eg. is so much easier that way.

But I can configure a lot of other, new Azure resources after the Event Hub or the Service Bus.

For example, IoT Hub routing to an Event Hub makes it possible to pass telemetry to Azure Functions (using the EventHubTrigger). This opens a whole new world of Azure opportunities using only an EventHub in this case! But it can be done even simpler…

Azure Functions

Azure Functions are truly one of the best additions to the Azure Cloud in the recent past. Although this Azure resource is still growing in functionality and usability, it had already proven to be the ‘glue’ the cloud needs, by executing “server-less” code.

And it’s refreshing to see that these functions can be written in the Azure portal, all you need is a browser and a subscription.

But as pointed out already in a previous blog, the Azure Portal integration is slowly progressing. In that blog, you had to copy/paste the Shared Access Policy key, when you wanted to refer to an EventHub input source connection. This was very uncomfortable.

But there is a new Azure Portal dialog in town!

New Event Hub connection dialog

Azure Functions are actually pieces of code, triggered by some external event. Here, the function code is triggered by an EventHub event (a message received by the EventHub):

To get this triggered, we need some knowledge about some EventHub in your subscription. So you need to pass an EventHub connection:

And as you can see, this dialog knows which EventHubs are available within your subscription. This is so much more convenient compared to the previous solution!

Accessing the IoT Hub Telemetry

Let’s check this out in detail. In this example, I have created both an IoT Hub and a Functions App.

And as you can see, I can select my IoT Hub built-in Events endpoint, the place which exposes all telemetry coming in:

Note: So I do not need to write custom routing to an extra EventHub (within a specific EventHub namespace) anymore!

Select this connection and you are done, ready to receive telemetry.

So save the complete trigger:

Note: The name of the field ‘Event Hub Name’ is ignored in this solution (it is not used in this context).

And check the arrival of messages:

It works. So with just two Azure resources, we can access telemetry and start writing custom code for this telemetry within the Azure Function.

Note: The telemetry available in Stream Analytics is much richer. E.g. ingestion time the name of the device is available, outside of the original message. The time is easy to fix but a message can not be related back to the original device if the name is not available in that original message.

Event Hub-compatible endpoints

Why can we access the telemetry using the EventHub connection? Because Microsoft provides this IoTHub connection as an EventHub compatible endpoint:

This option and the new dialog are both truly a great addition.

Device Explorer shows up as a show stopper?

So far, so good… But there is a little edge on this way of working. and that will explain the title of this blog too!

While writing this blog, just like always, I kept the Device Explorer running along with the test device. I do this just to check if data is arriving at the IoT Hub.

This time, just after I started the Azure Function, I got this exception in the Device Explorer:

Why was my Device Explorer crashing?

Consumer Groups

The answer is simple: The Device Explorer and the Azure Function are both using the same Consumer group.

Consumer groups are a setting similar to other messaging systems that can be used to pull data in specific ways to connect other applications or services to IoT Hub. Every IoT hub is created with a default consumer group. However, you can add or delete consumer groups to your IoT hubs using this setting.

(taken from Azure documentation)

So two applications are ‘fighting’ for the same data. Luckily, we can add a second Consumer group. Then both applications get their own copy of the same telemetry.

Note: There are two Consumer groups now; the $Default group is always existing.

Now go to the Integrate tab page of the Azure Function and see we can actually configure the Consumer group. Fill in the second Consumer group:

And now we have both apps showing the same telemetry:

So it’s a good practice to create a Consumer group for each resource consuming the telemetry (this means Stream Analytics jobs too, btw.).

That’s why Consumer groups are the only thing standing between IoT Hub and Functions!

And try not to depend on the $Default Consumer group. Perhaps there will be another resource trying to read from that ‘always available’ consumer group. This will cause an interruption either way. Remember, there is no way to select the Consumer group in the Device Explorer or during the creation of the Trigger connection.

Operations Monitoring

During the creation of the previous Trigger connection, there was a second type of Endpoint available within the IoTHub. This type is called: Operation monitoring:

Operation monitoring is also a very useful part of the IoTHub. Users can be notified of events like connecting and disconnecting devices, file upload, message routing etc. And it can point out errors in your IoTHub usage.

Just activate the types of events you are interested in. Here I activate the events for connection and disconnection devices. And again, do not forget to give your Operations Monitoring settings a second Consumer group too:

Now create a second Azure function which uses this connection. Ignore the name of the EventHub (you do not need it) and fill in the Consumer group:

To test this Operations Monitoring connection, I disconnected and connected an ESP (running this sketch) multiple times:

2017-06-21T21:45:13.120 Function started (Id=f4dcae0b-e6e7-4f8f-bb2f-6061fb8479d6)
2017-06-21T21:45:13.120 C# Event Hub trigger function processed a message: {"protocol":"Mqtt","authType":"
{ \"scope\": \"device\", \"type\": \"sas\", \"issuer\": \"iothub\" }","time":"2017-06-21T21:45:07.5876233Z",
"operationName":"<strong>deviceDisconnect</strong>","category":"Connections","level":"<strong>Error</strong>","statusCode":404,"statusType":404104,
"statusDescription":"DeviceConnectionClosedRemotely","deviceId":"SomeDevice","ipAddress":"xx.yy.zz.XXX"}
2017-06-21T21:45:13.120 Function completed (Success, Id=f4dcae0b-e6e7-4f8f-bb2f-6061fb8479d6, Duration=0ms)
2017-06-21T21:45:25.252 Function started (Id=b150eeea-128c-49ab-b382-04ae02e27a2f)
2017-06-21T21:45:25.252 C# Event Hub trigger function processed a message: {"protocol":"Mqtt","authType":"
{ \"scope\": \"device\", \"type\": \"sas\", \"issuer\": \"iothub\" }","time":"2017-06-21T21:45:20.5986743Z",
"operationName":"<strong>deviceConnect</strong>","category":"Connections","level":"<strong>Information</strong>","deviceId":"SomeDevice","ipAddress":"xx.yy.zz.XXX"}
2017-06-21T21:45:25.252 Function completed (Success, Id=b150eeea-128c-49ab-b382-04ae02e27a2f, Duration=0ms)

Note: the disconnection is handled as an error which is very true, I just shut down the ESP by disconnecting the USB cable!

Again, with just a few clicks, you have configured the logic to start logging events showing up in the IoTHub.

Conclusion

I do not expect that this is the final version of this Azure Function connection dialog. But this is already a big time saver and less error prone compared to the previous solution where we had to copy Shared Access Policies by hand. And I am pleasantly surprised that this feature does perform so well.

But keep in mind that Consumer groups play an important part in keeping a steady flow of telemetry coming in without any interruption.