First look: The Things Network new Azure IoT Hub integration

Right from the start back in 2015, The Things Network supported connecting their LoraWan backend platform to other (cloud) platforms.

Connecting to the Azure cloud was no exception to this.

There was no default integration but the support for the MQTT protocol opened this door. Back then, I already blogged about using MQTT to connect. Based on that logic, bridges appeared for both uplink and downlink messages.

Later on, Microsoft also introduced the secure and flexible IoT Central bridge to connect to The Things Network, based on uplink support for webhooks.

Still, even with the new V3 portal of The Things Network, this integration was still based on either the Webhook integration or the original V3 MQTT interface.

All these techniques require an extra layer of logic between the two clouds.

But…

Finally, a native Azure IoT Hub integration has been made available:

The main features of the Azure IoT Hub Integration are:

  • Devices only need to be registered in the Azure IoT Hub. The related TTN application device registrations are kept in sync by TTN. Just use IoT Hub Device Twin tags to control your TTN device registration
  • Uplink messages appear both as telemetry and as Device Twin reported property updates
  • Downlink messages are sent using the Device Twin desired properties

The current documentation is pretty straight forward (although the DPS service logo is shown, it’s not part of the standard solution).

Let’s check out how this integration works and let’s dive deeper into this solution architecture.

The TTN console supports multiple kinds of integrations (storage, AWS, etc.) by default, including that generic Webhook integration.

The configuration page of the new Azure IoT Hub integration does not seem that impressive at first because there are only two secrets (at the top) and two settings to fill in:

You only need to provide some details about an IoT Hub (hostname and key) you will be using.

Still, there is done a lot of work under the covers! We will try to find out how things work both on the Azure side and TTN side.

Due to certain design choices made by Microsoft regarding the IoT Hub resource implementation and notifications, that IoT Hub is not the only Azure resource you need for this full, two-way integration. Next to the IoT Hub, an Azure Function and an Event Hub must be installed within your Azure subscription:

Although this seems like some overhead, this is in fact a very smart usage of the message routing capabilities any IoT Hub exposes. Think of messages being exposed which tell when devices are created or deleted or if the Device Twin is updated.

Using this function, the TTN can handle the lifetime of device in the TTN backend for us. Thus, we have a single source of truth.

Note: I was able to connect the IoT Hub integration directly to an (existing) IoT Hub. You lose the capability device registration synchronization and downlink messaging.

ARM template

Luckily, we do not have to deploy and configure all these Azure resources ourselves!

The TTN integration does supply an ARM template together with a nice deployment wizard to guide us:

As you can see, most of the parameters do not need to be changed.

The resource group and related region are free to choose, just like in any Azure resource creation.

The more important parameters are the stack cluster address, stack application ID and stack API key. These can be found the TTN console page where you start the IoT integration.

The first two values are already provided.

Then, just select the IoT Hub integration page and hit that ‘Generate API key’ button:

After that, the three values are all available:

Warning: this page works in conjunction with the ARM template. Do not close this page yet, wait until the ARM template deployment is executed. You then get some additional settings!

Just wait for the ARM template deployment to complete (your deployment is ready):

Then, check out the Outputs tab. There, you see the required two values, hostname and key:

Fill these two values into the integration page and finally save the integration. Only now, the full two-way integration is active.

Warning: The IoT Hub shared access policy key used here is the ‘iothubowner’ key. This key has tremendous power so please be careful not to expose it. The TTN Console does not hide the key on the page. So be careful with exposing that page and/or the key. The product owner from the TTN has promised to look into this.

Finally, all Azure resources are created within your subscription:

Notice that the default ARM template settings let you generate random name parts to keep the resource names unique. You could experiment with the settings to have the names generated the way you want.

These Azure resources are not all free resources. At least, the IoT Hub is a S1 by default. Again, you could experiment with the settings to see if you can deploy it using the F1 (free) tier IoT Hub. The Application Insights service is optional regarding the Azure Function App. If you are confident, you will not miss the logging abilities of Application Insights, you could remove it from the resource group.

It’s time to manage a device.

Device management

The latest version of The Things Network portal has some astonishing features.

One of them is the LORAWAN device repository.

Here, new devices to register can be simply selected from a list. For example, the SeeedStudio Sensecap sensors are supported and selectable:

Therefore, I demonstrate the normal way of how a device could be added in the TTN Console first (this will be ignored later on after we look into the IoT Hub device twins support). This is just an introduction so you see what mechanism is used by the integration…

As an example, I add the Seeedstudio Sensecap Wireless CO2 Sensor. I just select it from the list and a picture of the device pops up:

Regarding the three secrets that identify a individual device, Seeedstudio provides an API where you fill in the EUI and KEY taken from the sticker on the sensor.

Then, this API returns the secrets you need:

As you can see in the TTN console, the brand and model can both be picked from drop-down lists.

If we take this a step further and register the specific device, many other cumbersome settings (like the payload formatters) are automatically filled in into the settings.

This is all thanks to device repository in the TNN V3 backend.

Still, even this easy registration step in the TTN console can be skipped if we go for the Azure integration!

The same registration in the TTN console is automatically performed just using Device management in the Azure IoT Hub integration.

How?

You only need to add an IoT Hub device registration with certain tags in the Device twin and the device is then automatically added to list of end devices in the TTN application.

This is because of the Azure function connected to the integrated IoT Hub routing towards an EventHub:

There are two routes automatically created during the ARM template deployment:

  1. The first route reacts on Devicetwin changes. The route checks if there are either desired properties or tags “IS_OBJECT($body.properties.desired) OR IS_OBJECT($body.tags)”
  2. The second route reacts on device lifecycle events (creation or deletion of devices)

Both routes send updates events to this Event Hub endpoint:

The Azure function, next to the IoT Hub and Event Hub, picks up these events from the Event Hub and processes them:

This function logic just forwards the messages to an The Things Network Endpoint transformed into the right format, using the secrets provided during the ARM template deployment:

If you are interested in the actual code being executed, please check the ARM template JSON:

I checked out the source code and all it does is converting any generated IoT Hub message (device lifetime, device twin changes) into a TTN compatible format and sends that converted message with StackEvents to the TTN endpoint:

var stackEvents = new StackEvent[events.Length];
for (var i = 0; i < events.Length; i++)
{
    var ev = events[i];
    var props = ev.Properties;
    var deviceID = (string)props["deviceId"];

    stackEvents[i] = new StackEvent(
        new EndDeviceIdentifiers(applicationIdentifiers, deviceID),
        JsonDocument.Parse(ev.Body).RootElement,
        props
    );
}

var response = await client.SendAsync(new HttpRequestMessage
{
    RequestUri = uriBuilder.Uri,
    Method = HttpMethod.Post,
    Headers = {
                {"Authorization", $"Bearer {apiKey}"}
            },
    Content = new StringContent(
        JsonSerializer.Serialize(new StackEvents(stackEvents)),
        Encoding.UTF8,
        "application/json"
    ),
}, token);

By adding an Azure Function into your subscription, The Things Network can keep their infrastructure simple. They just need to listen for incoming messages from all possible IoT Hubs (and potentially other sources too) with just one endpoint instead of setting up (stateful) logic for every individual TTN application IoT Hub integration.

Adding a device registration

So, devices must only be added in the IoT Hub.

The first step is to create a regular device registration. Come up with a unique name and accept the default security.

Here, I use the Azure portal because you can see what happens:

As you can see, I added a total of four different Seeedstudio SenseCAP sensors.

Each device has its own Device Twin, a JSON document registration like this, in the IoT Hub ‘device registry’:

Note: see how the desired properties and reported properties are already part of this device twin (but still empty).

Secondly, we need to add ‘Tags’ that describe the registration of the Lora device in the TTN backend:

The AppKEy, DevEui, and JoinEui describe the identification of the device (Remember, we goth them from the Seeedstudio API).

We also have to fill in the device version identification (is this eg. a Sensecap Co2 sensor or a Sensecap barometric sensor). I got these values after some fiddling in the TTN portal where I found the correct brand and model settings using a manual registration:

The region and frequency related information was just taken from the documentation example:

Those sample settings seem to work for this Co2 sensor too. You can find them back in the TTN console here (kind of):

So, after adding these tags and having some patience (this step can take more than a couple of seconds if the Azure function needs to awaken first), the IoT Hub device registration is repeated in the TTN console:

Interestingly, if we check the Device Twin tags afterwards, the device Appkey is suddenly removed from the tags:

I had a conversation about this behavior with the product owner within TTN. As I expected, this is done based on a security related decision. And that’s OK.

Deleting a device

I can be short about device deletion in the IoT Hub: If you delete a device from the IoT Hub registry, the device is also removed from the TTN backend.

It’s just gone.

Incoming uplink telemetry

Now the devices are setup, we would like to see some incoming telemetry.

We do not have to add an extra uplink payload formatter. A formatter from the repository is already active:

The TTN backend is taking the one found in TTN device repository. Nice…

Actually, incoming TTN device telemetry is then forwarded in Azure two ways:

  1. Reported properties in the Device Twin
  2. Device telemetry messages

Let’s exam this.

Reported properties

If we check the Reported properties in the Device Twin, we see that there are suddenly values available (compare this with the live message stream in the TTN application).

Check out the twin of the IoT Hub Co2 device registration:

Here, the measurementID 4100 tells us this is a Co2 measurement. The value 609 is the Co2 level in PPM units. As seen here, I now know my office is identified as a well-ventilated room.

There are two things to notice:

  1. The date and time of the measurement is seen in the related $metadata.$lastUpdated field.
  2. If a device sends multiple, different messages, only the latest message is exposed.

The second point is a bit disappointing.

Why?

Check out this this ‘extended status message’ coming from the Temperature and Humidity sensor in the TTN console.

It is only sent once in a while (eg. when the device starts):

As you can see, it exposes the battery level and message interval (five minutes) too. This is interesting information but not part of the regular messages…

Receiving this elaborate messsage will result in a Reported property update. Still, the other more common messages quickly overwrite that elaborate one:

Thus, the two other values are ‘lost’.

Note: compare the length of the payload. This one is shorter.

I would like to see a ‘merge’ where messages with new or altered properties are added or updated next to the already received properties. This way, over time, we do not wipe interesting Reported properties.

You always can program this behavior yourself with some extra logic (eg. an extra Azure Function or Stream Analytics job) if you add a separate route with reported properties after a Device Twin update (or an extra consumer group on the existing Event Hub).

Telemetry messages

The TTN backend also sends the messages to the IoT Hub as telemetry messages.

Incoming messages can be seen in the IoT Explorer:

As you can see, the whole ‘raw’ TTN message is sent. This includes the payload, gateway information, network information, etc. You can even see the latitude/longitude of the related gateways so you could try to do some kind of triangulation.

This could become a big message but I expect it will be counted as one message by the S1 tier IoT Hub works because it counts with 4Kb chunks (with a daily quota of 400.000 chunks). The Free tier works with only 0.5Kb chunks (quota set to 8000 chunks) and this message seems bigger 😦

Cost Impact on IoT Hub

Both ways of exposing message information are a great addition to the Azure integration and have their own use cases.

Regarding costs, keep in mind you are charged for both ways of messages:

  1. Device-to-cloud messages. Successfully sent messages are charged in 4-KB chunks on ingress into IoT Hub. For example, a 6-KB message is charged two messages.
  2. Device twin updates (tags and properties). Twin updates from the device or module and from the solution backend are charged as messages in 4-KB chunks. For example, reading a 12-KB twin is charged as three messages.

It would be nice to be able to choose which ways of communication you want to have activated. This would be a nice optimization feature.

On the other hand, the number of messages sent per day is not that high.

Still, you need to multiply this by the number of active devices!

In the end, these two ways of communicating at least doubles the consumption on the daily quota.

Sending downlink messages

Sending commands towards the TTN backend is also supported by the IoT Hub integration.

You need two things:

  1. Update the desired properties of a device (in a specific JSON format)
  2. Optional: Implement a downlink payload formatter (JSON to byte array)

Note: The second step is marked as optional. I’m not sure if there is a working payload formatter already active from the repository (I could not find any documentation). Still, you can overrule the default payload formatter and add your own if you want to be sure.

I can demonstrate this downlink support partly by updating desired properties of a device.

For this Light intensity sensor, I added a dummy desired property (again, I have no clue what the repository downlink payload formatter expects):

Notice the properties must be surrounded with this ‘decodedPayload’ element.

I then save the Device Twin changes.

First, we see the Azure Function picking up the routed message (I repeated this step three more times with other values):

Then, in the TTN Console, we see the arrival of this downlink message:

I leave it to the readers imagination regarding the possible further processing of this message towards devices.

Conclusion

We have seen how the Azure IoT Hub integration is a first-class member of The Things Network backend.

This integration is just working as expected. In a simple and robust way, we can support both uplink and downlink LORA messages with only one device repository to worry about (The IoT Hub).

There are some minor points of attention (eg. the exposed shared access policy, the possible Application Insights overhead, the possible overwritten reported properties, paying twice for the same incoming message) but the solution is very powerful and productive.

This solution is more heavy-weight than eg. webhook integration towards an Azure function. There are probably also Azure resources costs involved in the long run. Still, this integration will certainly be less expensive than running a stateful MQTT web job.

If you are planning to connect LoraWan devices to the Azure portal, even without downlink needs, this is probably the most productive way to do this.

If you have already used some kind of integration between TTN and Azure (perhaps the webhook or a custom MQTT connection) consider to upgrade your solution with this new kind of integration.