Exploring full Azure IoT Hub device support using MQTT(s) only

Most of you Azure IoT developers are connecting devices to the Azure cloud using the Azure IoT Device SDKs.

Using these SDKs, you can connect a device to the cloud in an easy and secure way with your favorite programming language like C#, C, Java, Python, or Node.js.

This is the recommended way because it offers a convenient reand optimized way to support all Azure IoT Hub features like Device Twin, Direct methods, and Cloud messages. It takes away a lot of the code wiring and you can focus on functionality.

Still, in a few instances, like working with very constrained devices, there could be a need for bare MQTT support:

MQTT is the de-facto standard for stateful communication in the IoT World (btw. Bare AMQP is offered too).

Let’s see how the Azure IoT Hub supports bare MQTT.

Credits

The starting point for this blog post was this great article from my friend Pete Gallagher explaining how to connect a Raspberry Pi Pico W to Microsoft Azure IoT Hub using MicroPython and MQTT.

He showed both sending device-to-cloud messages and receiving cloud-to-device messages over MQTT.

It proves the IoT Hub can act like an MQTT broker but only within the constraints of some predefined topics and message formats.

Yes, you can communicate to the IoT Hub using MQTT but do not expect the hub to behave like your regular MQTT broker.

For example, devices are seen as ‘identities’ which connect to the IoT Hub only. Devices cannot register to each other’s topics. If you want to send data from one device to another, you need to add logic (eg. an Azure Function) behind the IoT Hub (an alternative is this parent-child communication using Azure IoT Edge). As a bonus, this is a better architectural choice too due to the separation of concern.

Challenges towards full IoT Hub support

That article from Pete explained very well how to work with the MQTT security and MQTT topics.

It’s a good starting point.

It is missing a good number of abilities regarding full Azure IoT Hub support, though:

• Connecting the device to the MQTT broker as part of the IoT Hub
• Sending device messages
• Sending user properties alongside device messages
• Receiving cloud messages as an event
• Getting cloud messages on startup, those sent to the IoT Hub while the device was offline
• Receiving user properties alongside cloud messages
• Receiving Direct methods as an event, and responding to it
• Receiving desired properties as an event
• Reading desired properties (and reported properties) at the start of the application
• Sending reported properties

Yes, there is a lot more going on regarding device-to-cloud communication and back from an IoT Hub perspective.

So, I bumped my head a couple of times and worked my way through the documentation and some example code I found.

Finally, I present to you this GitHub repo showing all the missing pieces for full IoT Hub support.

Disclaimers

There are a number of disclaimers.

First, when you are able to work with the Device SDKs, please do so. As seen in my sample application, the IoT Hub interaction is quite complicated due to the extensive feature set (like a disconnect, reconnect, and throttling support) and I only implemented the happy flow.

This application shows you how to communicate over MQTT in C#. This will be a great starting point for other languages because we stay close to bare MQTT implementation. Thus, it is not production ready. For example, it is not handling any errors or unexpected data exchange.

Also, it expects the SAS token to be generated outside the application so that one will expire eventually…

Last but not least, it pins the TLS certificate for a secure transport connection to the IoT Hub. Please check that blog to read why this is not a wise thing to do.

This blog post is based on the MQTT version 3.1.1 implementation. Version 5 of MQTT support is in preview (not in scope for this article).

Full Azure IoT Hub support

Now, it’s time to dive into the steps to implement full IoT Hub support for devices only capable to implement bare MQTT. Code snippets are used to explain the logic.

Please check the full example and try to follow along.

Connecting the device to the MQTT broker as part of the IoT Hub

Let’s start with connecting an MQTT client to the IoT Hub.

For this, a .Net Core console app is constructed, referencing the MQTTnet library.

A client is constructed:

// connect
var mqttFactory = new MqttFactory();
mqttClient = mqttFactory.CreateMqttClient();
var mqttClientOptions = new MqttClientOptionsBuilder()
    .WithTcpServer(hostName, 8883)
    .WithCredentials(user_name, passw)
    .WithClientId(clientId)
    .WithCleanSession(cleanSession)
    .WithTls(new MqttClientOptionsBuilderTlsParameters()
    {
        AllowUntrustedCertificates = true,
        Certificates = new List<X509Certificate>
        {
            new X509Certificate2("baltimore.cer")
        },
        UseTls = true,
    })
    .Build();
...
mqttClient.ApplicationMessageReceivedAsync += MqttClient_ApplicationMessageReceivedAsync;
await mqttClient.ConnectAsync(mqttClientOptions, CancellationToken.None);

Update: this example still shows the use of the now deprecated Baltimore TLS certificate. Please check this post regarding the TLS certificate replacement.

The hostname, username, and password have to be provided. See Pete’s blog regarding collecting these secrets.

We need to communicate over port 8883 and TLS has to be enabled.

Also, the public TLS certificate is supplied. I could have left it out because my (windows) machine can provide it.

Note: This ‘Baltimore’ certificate will expire in 2023.

We also register for incoming events via this callback method named ‘MqttClient_ApplicationMessageReceivedAsync’.

This single method will handle all response messages from the cloud. So, we have to check the intent of each incoming message, this is done via the topic of the incoming messages.

We can now set up the connection.

Sending device messages with user properties alongside

Once connected, sending a message is not that hard.

You need to construct a topic like this one, including the IoT Hub DeviceId:

public string send_message_topic => $"devices/{_clientId}/messages/events/";

Note: this topic can be optionally extended, as seen below.

Construct a message and send it:

var payloadJson = new JObject();
payloadJson.Add("temp", DateTime.UtcNow.Millisecond % 20);
payloadJson.Add("hum", DateTime.UtcNow.Millisecond / 10);
string payloadString = JsonConvert.SerializeObject(payloadJson);
var message = new MqttApplicationMessageBuilder()
    .WithTopic(topics.send_message_topic + "$.ct=application%2Fjson&$.ce=utf-8" + "&a=a1&b=bb2")
    .WithPayload(payloadString)
    .WithQualityOfServiceLevel(MQTTnet.Protocol.MqttQualityOfServiceLevel.AtLeastOnce)
.Build();

await mqttClient.PublishAsync(message);

Notice how extra properties are added by extending the topic string:

• User properties like ‘a’ and ‘b’ are added
• Context type and Context encoding are added to support IoT Hub routing based on body properties (this can be omitted if you do not use this specific IoT Hub feature)

The device can send these messages:

In the IoT Hub, we see the messages arrive, complete with the user properties:

Notice that the IoT Explorer tool/IoT Hub omitted the two $.ct and $.ce properties.

Receiving cloud messages as events, complete with user properties alongside

Receiving cloud messages is conform to what we see in Pete’s blog post.

We need to subscribe to this topic:

public string subscribe_topic_filter_cloudmessage => $"devices/{_clientId}/messages/devicebound/#";
...
var mqttSubscribeOptionsCloudMessage = mqttFactory.CreateSubscribeOptionsBuilder()
    .WithTopicFilter(f => { f.WithTopic(topics.subscribe_topic_filter_cloudmessage); })
    .Build();
await mqttClient.SubscribeAsync(mqttSubscribeOptionsCloudMessage, CancellationToken.None);

Once cloud messages arrive over the generic event callback method, we filter them and process them:

if (args.ApplicationMessage.Topic.Contains("devicebound"))
{
    Console.WriteLine("Cloud message event received:");
    Console.WriteLine($"Received application message: {args.ApplicationMessage.ConvertPayloadToString()}");
    //// Notice the topic contains both 'devicebound' and 'deviceBound'
    if (args.ApplicationMessage.Topic.ToLower().Split("devicebound").Length > 2)
    {
        var properties = args.ApplicationMessage.Topic.ToLower().Split("devicebound")[2];
        var propertyList = properties.Split('&', StringSplitOptions.RemoveEmptyEntries);
        foreach (var item in propertyList)
        {
            var p = item.Split('=');
            Console.WriteLine($"\tProperty: {p[0]}={p[1]}");
        }
    }
}

To test sending Cloud messages from the IoT Hub to the device, we use both the Azure Portal and the IoT Explorer tool.

We send a ‘devicebound’ event (notice the custom properties) that is received based on that specific topic:

The payload is simple to extract.

If we inspect the topic string received, it is extended with the custom properties. Here are some examples:

// Received from IoT Explorer tool
"devices/nonsdkdevice/messages/devicebound/%24.mid=f618dedc-210e-4c68-956d-160577b2ca01&%24.to=%2Fdevices%2Fnonsdkdevice%2Fmessages%2Fdevicebound&key1=value1&key2=value2"

// Received from the Azure portal
"devices/nonsdkdevice/messages/devicebound/%24.to=%2Fdevices%2Fnonsdkdevice%2Fmessages%2FdeviceBound&%24.ct=text%2Fplain%3B%20charset%3DUTF-8&%24.ce=utf-8&bbb=bbbb&aaa=aaaa"

Note: for some reason, the second call (made from the Azure Portal) has a ‘deviceBound’ with capital ‘B’…

That is why I added that lousy “.toLower()” to the properties check.

We can separate the properties from the message topic (this one is sent from the IoT Explorer):

Interestingly, the Azure portal also sends extra context type and context encoding information:

Getting cloud messages on startup, those sent to the IoT Hub while the device was offline

Although we can receive cloud messages, this complete ‘cloud messages’ use case is not completed yet!

Cloud messages can have been sent to the IoT Hub while the device was offline!

Then, the IoT Hub supports buffering cloud messages in a queue until the device connects.

The original sample code did not take this into account.

Luckily, this is already covered in our sample, check the connection setup again!

Especially, check the ‘WithCleanSessions’:

var mqttClientOptions = new MqttClientOptionsBuilder()
    ...
    .WithCleanSession(cleanSession) // if false, receive those cloud messages queued while the device was offline 
    ...
    .Build();

If this is set to ‘false’ our client app will receive all queued cloud messages at once.

Note: We do not have to send anything to any topic, just start ‘without clean session’.

Here, I queued three messages. It seems these even arrive in the correct order:

By supporting both kinds of situations(while the device was offline and while the device is online). we do not miss any cloud message anymore.

Receiving Direct methods as events, and responding to it

Sending Direct methods to a device requires the device to have a live connection at that moment of transmission. The cloud call will actually block and wait for a response. The method call can even timeout.

Direct methods have both a name and a payload. A response has to be provided. That response contains at least an HTTP status-like number and optionally it contains a payload too.

To accommodate this, we need a separate topic to listen to, regarding Direct methods requests:

public string subscribe_topic_filter_directmethod => "$iothub/methods/POST/#";
...
var mqttSubscribeOptionsdirectmethod = mqttFactory.CreateSubscribeOptionsBuilder()
    .WithTopicFilter(f => { f.WithTopic(topics.subscribe_topic_filter_directmethod); })
    .Build();
await mqttClient.SubscribeAsync(mqttSubscribeOptionsdirectmethod, CancellationToken.None);

Once a Direct method is received, we need to send a response message over another topic:

internal string send_direct_method_response_topic => "$iothub/methods/res/200/?$rid=";

Notice two things:

1. This topic needs to be extended by this ‘$rid’. This ‘$rid’ stands for ‘RequestId’ and is part of the original Direct method request topic
2. The magic number 200 is sent as a status code. This is actually a confirmation the message is well received. Other status codes are explained here.

We respond to the incoming Direct method request message:

if (args.ApplicationMessage.Topic.Contains("$iothub/methods/POST/"))
{
    var methodName = args.ApplicationMessage.Topic.Split('/')[3];
    var requestid = Convert.ToInt32(args.ApplicationMessage.Topic.Split('=')[1]);
    Console.WriteLine($"Direct method {methodName} event received with id {requestid}:");
    Console.WriteLine($"Received application message: {args.ApplicationMessage.ConvertPayloadToString()}");
    //// Return the Direct Method response.
    var payloadJsonDirectMethod = new JObject();
    payloadJsonDirectMethod.Add("ans", 42);
    string payloadStringDirectMethod = JsonConvert.SerializeObject(payloadJsonDirectMethod);
    var messageDirectMethod = new MqttApplicationMessageBuilder()
    .WithTopic($"{topics.send_direct_method_response_topic}{requestid}") // 200 = success
    .WithPayload(payloadStringDirectMethod)
    .WithQualityOfServiceLevel(MQTTnet.Protocol.MqttQualityOfServiceLevel.AtLeastOnce)
    .Build();
    mqttClient.PublishAsync(messageDirectMethod).Wait();
    Console.WriteLine("Direct method response sent");
    return Task.CompletedTask;
}

Again, getting the body is not so hard.

We check the Request topic for two values:

1. The name of the Direct method. A device can respond to multiple, separate Direct methods so you need to distinguish them from each other
2. We need that request ID, so we can construct the right response topic

Finally, the response payload message (if needed) is sent along the Direct method response (with the right request ID).

So, If we send the following request in the portal:

It will be picked up by the device:

And a response answer is received back in the portal, well within the expiration timeframe:

This covers the happy flow of the Direct methods.

Receiving desired properties as events

Next, we check out another important IoT Hub feature, the support for the Device Twin. This involves both receiving desired properties and writing reported properties.

Receiving desired properties as an event means we have to subscribe to a certain topic:

public string subscribe_topic_filter_desiredprop => "$iothub/twin/PATCH/properties/desired/#";
...
var mqttSubscribeOptionsdesiredprop = mqttFactory.CreateSubscribeOptionsBuilder()
    .WithTopicFilter(f => { f.WithTopic(topics.subscribe_topic_filter_desiredprop); })
    .Build();
await mqttClient.SubscribeAsync(mqttSubscribeOptionsdesiredprop, CancellationToken.None);

Once an IoT Hub change is made to the Device Twin of the device, an event message is received:

if (args.ApplicationMessage.Topic.Contains("$iothub/twin/PATCH/properties/desired/"))
{
    var requestid = Convert.ToInt32(args.ApplicationMessage.Topic.Split('=')[1]);
    Console.WriteLine($"Desired properties event received with id {requestid} :");
    Console.WriteLine($"Received application message: {args.ApplicationMessage.ConvertPayloadToString()}");

    return Task.CompletedTask;
}

The message payload is easy to extract.

The ‘requestId’ can be ignored.

So, if this updated Device twin with a ‘value’ and a ‘text’ is sent:

It is picked up by the application:

Notice, on changing/saving the desired properties in the IoT Hub, the version increased by one to 30 (was 29).

With this being said, the same issue as with the Cloud messages can occur.

How can we react to Device Twin changes which occurred while the device was offline?

Reading desired properties (and reported properties) at the start of the application

Desired and reported properties give us the possibility to save the device state in the cloud.

This is because the device can read both desired and reported properties at any moment it would like to.

Most often this is read once, just after connecting to the cloud (on startup?). After that, you can start reacting to desired properties as seen in the previous section.

To read the Device Twin, we need to send a request message over a topic:

public string request_latest_device_twin_topic => "$iothub/twin/GET/?$rid=42";
...
var messageDeviceTwin = new MqttApplicationMessageBuilder()
    .WithTopic(topics.request_latest_device_twin_topic)
    .WithQualityOfServiceLevel(MQTTnet.Protocol.MqttQualityOfServiceLevel.AtLeastOnce)
.Build();
await mqttClient.PublishAsync(messageDeviceTwin);
Console.WriteLine("Requested latest DeviceTwin with request id 42");

Note: This ‘requestId’ is just a value you can choose yourself. I just went for 42.

Once this request for the Device Twin is made, a response event message is received by the device:

if (args.ApplicationMessage.Topic.Contains("$iothub/twin/res/200"))
{
    var requestid = Convert.ToInt32(args.ApplicationMessage.Topic.Split('=')[1]);
    Console.WriteLine($"Device Twin response on request received with id {requestid}:");
    Console.WriteLine($"Received message: {args.ApplicationMessage.ConvertPayloadToString()}");
    return Task.CompletedTask;
}

Again the ‘requestId’ and message body are easy to extract.

Notice this (full device twin) response is not using the same topic as the desired properties. It’s another topic and we have to register for this too:

public string subscribe_topic_filter_operationresponse => "$iothub/twin/res/#";
...
var mqttSubscribeOptionsreportedpropresponse = mqttFactory.CreateSubscribeOptionsBuilder()
    .WithTopicFilter(f => { f.WithTopic(topics.subscribe_topic_filter_operationresponse); })
    .Build();
await mqttClient.SubscribeAsync(mqttSubscribeOptionsreportedpropresponse, CancellationToken.None);

So, when we ask for this Device Twin (possibly changed while the device was offline):

We get this response:

This is a great starting point for an application to see what its Device Twin looks like.

Still, it is useful to have some kind of local storage so you compare the latest Device Twin with your locally stored Device Twin (version numbers).

Only then you do know for sure it has or has not been changed while the application was offline. Then, you know if you have to respond with any reported properties to the cloud.

Sending reported properties

Last but not least, you want to be able to send reported properties based on logic executed on the device.

Typically, devices echo the desired properties back into the reported properties. This way, from a management standpoint, it is clear if a device has picked up the desired properties already by simply comparing it with the reported properties.

To send a reported properties update, we need to send a message on this topic:

public string send_reported_properties_topic => "$iothub/twin/PATCH/properties/reported/?$rid=45";
...
var payloadStringReportedProp = "{ \"telemetrySendFrequency\": \"" + DateTime.Now.Second.ToString() + "m\"}";
var messageReportedProp = new MqttApplicationMessageBuilder()
    .WithTopic(topics.send_reported_properties_topic)
    .WithPayload(payloadStringReportedProp)
    .WithQualityOfServiceLevel(MQTTnet.Protocol.MqttQualityOfServiceLevel.AtLeastOnce)
    .Build();
mqttClient.PublishAsync(messageReportedProp).Wait();
Console.WriteLine($"Reported properties update sent: '{payloadStringReportedProp}'");

Again, the ‘requestId’ is just some arbitrary number. I went for 45.

Here, we send this arbitrary reported properties update:

We also get a response from the IoT Hub (so we know the version change):

We registered for that same topic related to the DeviceTwin JSON. Only, the reported properties response events are represented by 204 in the topic name instead of 200:

public string subscribe_topic_filter_operationresponse => "$iothub/twin/res/#";
...
if (args.ApplicationMessage.Topic.Contains("$iothub/twin/res/204"))
{
    var parameters = args.ApplicationMessage.Topic.Split('=')[1];
    var requestid = Convert.ToInt32(parameters.Split('&')[0]); 
    var version = Convert.ToInt32(args.ApplicationMessage.Topic.Split('=')[2]);
    Console.WriteLine($"Reported Twin response on request received with id {requestid} and version {version}");
    Console.WriteLine($"Received message: '{args.ApplicationMessage.ConvertPayloadToString()??"Empty payload"}' as response");
    return Task.CompletedTask;
}

Notice we actually receive an empty payload. On the other hand, we received this ‘requestId’ 45 and ‘version’ 66.

You can ignore the ‘requestId’ but the ‘version’ is on par with the Device Twin in the IoT Hub:

You can also remove Reported properties this way.

Each individual (high-level) reported property must be manipulated if you want to remove all Reported properties.

just send a null value for each individual reported property, part of the complete message.

Conclusion

We have demonstrated that using bare MQTT topics and logic, we can securely connect to the IoT Hub over MQTT, and all Azure IoT Hub Device Twin features are matched.

In the end, it can feel very intimidating due to the many Azure IoT Hub features offered. I hope you now understand why the SDKs are the best choice in most cases.

The full C# example application is open-sourced here.

Again, this is a sample application to explain how the IoT Hub supports devices that are not capable of including the regular Azure IoT Device SDKs. It still will need a lot of effort to bring this sample to any production-ready quality level.

Let me know in the comments how this works out for you.

I want to thank Pete for pointing me in the right direction 🙂