Introducing The Things Network version 3 stack and portal

Since 2016, I have been involved in the world of LoraWan.

The combination of low powered devices together with long-range communication makes this protocol ideal for sending short messages from remote locations. It even supports two-way communication.

One of the most famous players in this knowledge area is The Things Network. They provide a set of open tools and a global, open network to build your next IoT application at low cost, featuring maximum security and ready to scale with LoraWan.

Its community is thriving on both enthusiastic makers, starters, and companies which are building their IoT solution on that backend.

The team behind The Things Network platform, The Things Industries, are now ramping up the third version of the backend stack.

This is not just an update. This is a completely new stack, built from the ground up and the team invests into a clean, portable, open-sourced backend. This new stack is standards-compliant by default and it will support the Lora 1.1 specification too. The V3 backend is designed for scale, for ‘N’ as they say (N customers, N regions, N devices, N versions):

We see the devices and gateways on the left, the V3 stack in the middle, and the third-party cloud integrations (eg. AWS, Azure) on the right.

In this blog, we look at registering a gateway and a device in the new TTN V3 Stack portal. And we integrate cloud connectivity.

Warning: During this excercise, a preview version of the V3 portal is presented so screens and flows are still subject to change.

We will use both a LoRA gateway, a LoRa device, and a third-party cloud integration to show how things are connected and messages flow through the system. Here we connect the MikroTik wAP LoRa8 gateway, the Bosch parking lot sensor, and the Azure cloud.

Introduction to the new portal

After logging in into the new portal, things still look very familiar. We see both applications and gateways:

You can either start with registering a gateway or checking out your already registered gateways or you can go to your current applications or create one.

Note: The current (V2) stack is not mixed with the V3 stack but please be careful when migrating devices. If you migrate LoRa devices from the current portal to the new V3 stack you can consider deleting the registration of the devices in the old portal. In that case, delete LoRa devices first within an application before deleting the application itself. It is not guaranteed that all devices behind an application are automatically deleted on the deletion of their application.

Choose ‘Go to gateways’. Here, the list of all your registered gateways will be shown (the list should be empty for now). Let’s add a gateway by selecting ‘Add gateway’:

Registering a gateway in the portal

Registering a gateway device in the portal is not that hard.

First, fill in a unique Gateway ID (it should be in lowercase, with numbers and dashes if needed). The value in this field can be of your own choice.

Note: A Gateway ID is unique and this ID cannot be reused for future gateways once you decide to delete your gateway later on.

Next, fill in the GatewayEUI which is dictated by your gateway. Usually, you should find it on the back of your gateway (on the back of my gateway, I have to use the ‘Gw ID’):

Note: the Gateway Server Address is already made available and cannot be changed. Remember this value (eg. ‘microsoft.eu1.cloud.thethings.industries’).

After that, the right frequency plan for your gateway has to be selected:

I live in Europe so I choose the TTN Europe region.

Save the settings.

We will revisit this gateway registration after setting up the actual gateway.

The MikroTik wAP LoRa8

The MikroTik wAP LoRa8 has many features and is very versatile. It can be used in both outdoor and indoor situations using the internal antenna (as seen below, when opened, the green part) or an external antenna using the gold-colored SMA connector:

Image result for microtik wap lora8

In this step, we just connect it to the V3 portal, using the gateway registration we just created.

Note: If you have not done this already, check the Gateway ID on the back of the MikroTik. This makes our gateway unique for the TTN backend, so fill this into the Gateway EUI field in the TTN portal when adding or updating a gateway.

Attach all cables and power the gateway and login into the gateway local dashboard (check the manual for the login credentials and setting the IP address if needed):

I have a wired connection to my local network so I do not have to care about Wifi settings, etc.

What I do care about, is the Gateway Server address of the TTN stack. We fill to in that ‘microsoft.eu1.cloud.thethings.industries’ (copied during the gateway registration in the V3 portal) somewhere in the gateway.

This is because the gateway needs to connect to the V3 stack first before messages from and to LoRa devices can be dropped off or picked up.

Conveniently, in the MikroTik dashboard, the LoRA tab is available on the menu. From there, add a server:

Just fill in the name and the address and save:

This will result in a new network:

Verify the registration of the gateway

Once the network is set, go back to the TTN portal. Open the gateway registration as seen in the previous step.

Within minutes, you will see the first messages arrive in the gateway while the ‘last seen’ counter will be reset, occasionally. Here we see the last communication was less than half a minute ago:

Our gateway is now trusted and ready for end device uplink and downlink messages.

Let’s add a LoRa device.

The Bosch parking lot sensor

The parking lot sensor from Bosch is capable of sensing parked objects (eg. cars) above the sensor. It’s self-calibrating, it is ruggedized and it can operate for several years.

Note: Due to the used technology and being built for a long battery life, do not use this as a ‘vehicle counter’. Expect some delay between occupancy and reporting (~30 seconds).

Each parking lot sensor comes with device credentials in the form of these three values:

  1. The devEUI
  2. The appEUI
  3. The appKey

In this example, we will use these values to register and identity our parking lot sensor.

Note: The TTN stack V3 also supports the concept of join servers which will dramatically simplify the registration of LoRa devices. Bosch is one of the first companies supporting this (as demonstrated during the last The Things Conference). In this blog, we still use the well known way of registering a device.

Adding an application to the portal

First, we will add a new application. An application is a logical container for Lora device registrations. These devices typically share the same message structure which can than be decoded using a so-called payload formatter. They also share the same integrations with third-party clouds (eg. AWS, Azure, etc.).

Login into the portal. In the home screen, go to applications:

Here we see a list all applications created under your account. Please add a new application:

Adding an application is simple:

You have to fill in an Application ID and an application name for your next application.

Note: Application IDs are unique and can only be acquired once. If you delete the application (and its device registrations) somewhere in the future, the Application ID is not available for reuse anymore.

Once the application is created, you see a screen like this:

That’s all you need to do when adding an application. Notice the buttons for importing end devices and adding a single end device.

Adding the uplink payload formatter

Our gateway is now connected to the V3 stack, an application is created and we are almost ready to connect end devices.

The Bosch parking lot sensors (as every LoRa sensor) output binary arrays of data by default. We will have to add some logic to our application to handle the incoming (uplink) byte arrays sent by sensors.

So, we have to add an uplink payload formatter:

Note: The downlink payload formatter is out of scope.

Select Javascript as payload formatter. We see an incoming ‘bytes’ byte array and a JSON structure is returned:

We have to extend this function.

Luckily, Bosch already provides an uplink formatter but that formatter is designed for the V2 stack. In the V3 portal, the port is renamed from ‘port’ to ‘f_port’.

So I took that formatter from Bosch and renamed all ports. This results in this V3 uplink formatter for the Bosch parking lot sensor:

/* 
 * Decoder function for The Things Network to unpack the payload of Bosch Parking Lot Sensors - V3 Stack (preview)
 *
 * More info on the sensors/buy online:
 * https://connectedthings.store/lorawan-sensors/bosch-parking-lot-sensors.html
 *
 * This function was created by Al Bennett at Sensational Systems - al@sensational.systems
 * Altered for application on the TTN V3 (preview) stack
 */

function Decoder(bytes, f_port) {

  var decoded = {
    "bytes": bytes,
    "port": f_port,
    "packet_type": "Unknown"
  };

  // Startup packets are on port 3
  if(3 === f_port) {
    
    decoded.packet_type = "Startup";
    
    // Bytes 0 - 11 are debug info
    decoded.debug = bytes.slice(0, 12);
    
    // Bytes 12 - 14 are firmware version, convert to string
    decoded.firmware_version = bytes[12] + '.' + bytes[13] + '.' + bytes[14];
    
    // Byte 15 is the reset reason, convert to string
    decoded.reset_reason = 0;
    
    switch (bytes[15]) {
      case 1: decoded.reset_reason = "Watchdog";
              break;
      case 2: decoded.reset_reason = "Power On";
              break;
      case 3: decoded.reset_reason = "System Request";
              break;
      case 4: decoded.reset_reason = "Other";
              break;
      default: decoded.reset_reason = "Unknown";
    } 
    
    // Final byte, 16, is seven reserved bits with the LSB being the current occupancy state
    decoded.occupied = false;
    if(1 === (bytes[16] & 1)) {
      decoded.occupied = true;
    }
  }

  // Parking status packets are on port 1
  // Heartbeat packets are on port 2, but the same as status packets, so handled the same
  if(1 === f_port || 2 === f_port) {
    
    if(1 === f_port) {
      decoded.packet_type = "Status";
    } else {
      decoded.packet_type = "Heartbeat";
    }

    decoded.occupied = (1 === bytes[0]) ? true : false;
    
  }

  return decoded;
}

Save the changes:

Note: In the current preview portal, no test tooling is available for this function (as seen in the V2 portal). You have to test this with live messages.

Now we are ready to receive messages from Bosch Parking Lot sensors.

Adding a parking lot sensor

We will make use of OTAA deployment. Again, check if you have the right credentials for OTAA deployment of the Bosch sensor first:

  • devEUI
  • appEUI
  • appKey

Within the application we just created, select the ‘+ Add end device’ button:

The screen you get for adding a device is quite extensive, lots of sections. Yes, there are a lot of ‘moving parts’ in the V3 portal which you are able to manipulate.

Don’t worry, we don’t need much.

Add end device – General settings and the LoraWan options

We only have to fill in a few values, so just follow along. We first look at General settings and LoraWAN options:

In General settings, give the device a unique name like ‘bpls01’ (all lower case; digits, and dashes are supported). The name and description are up to you.

Note: The Device ID is unique and cannot be reused for a future device if you decide to delete this device registration.

Next are the LoRaWAN options. Fill in the LoraWan version, Regional Parameters version, and Frequency plan. The settings I use here are applicable to most recent devices out in the field.

Note: When in doubt, contact your device manufacturer. Or schedule some extra time in your agenda and try to mix and match the dropdown list values 🙂

Add end device – Activation settings

Now it’s time to fill in the three keys of your device. Look at the Activation settings section:

Keep the activation mode to “over the air activation (OTAA)”.

Fill in the AppEUI and fill in the DevEUI. The lenght of the field should match exactly.

The only key left now is the AppKey.

For that, uncheck the External Join Server so we get access to the AppKey field. Fill in your AppKey once that field is accessible.

Note: Normally, this is handled by the join server. The external join server makes it possible to have a new device ‘pre-configured’ by the manufacturer and that is a great way to simplify the registration. For now, this feature is out of scope.

This is everything we have to fill in. Just save the device settings.

The end device is now added:

At this point, we have both a gateway and a parking lot sensor connected to the V3 stack of The Things Network.

Incoming messages

If you have assembled your Bosch parking lot sensor, it is about to send data on startup to the TTN backend. And it will send changes in occupation.

Note: the sensor is automatically switched on using the magnet in the light grey base of the sensor. It triggers a magnetic on/off switch in the sensor itself. If you take out the rubber O ring seal (without damaging it), it is easy to attach and detach the two parts for test purposes.

In the portal, open either the end device page or the application page in the console and look at the Data section. There you can see incoming messages for one device or all devices registered in the application.

On startup, the Bosch parking lot sensor now starts communicating with the V3 stack. We see the first incoming messages, including a forwarded uplink data message on port 3 (third row from above):

The uplink data message is already decoded (using the payload formatter function) in values like firmware version, reset reason, and the current occupancy. Flip open that row:

As you can see, the spot is not occupied (occupied: false).

I experimented with a metal wheelbarrow with some extra metal stuff in it to simulate a car:

This gives me correct readings when the spot is occupied or not:

Note: Apart from vehicles, the sensor seems to be capable to detect other metal objects too. So it can be used in locations other than parking lots, apparently. I encurrage you to experiment a lot, though 🙂

As you can see, we get a simple bit (1) on port 1 which shows the status of the spot:

So we have successfully connected the Bosch Parking Lot sensor and we retreive messages.

If we do not relay these messages to a third party (cloud solution), the messages are dropped eventually when the Time To Live is reached (TTL).

The application support multiple kinds of integrations:

We use a webhook integration.

Azure Function as integration webhook

A webhook integration is nothing more than calling a secured REST endpoint. It’s a simple solution if you only want to offload messages to another platform.

For this occasion, I create this webhook in the Azure Cloud. The azure resource I use is an Azure Function.

First, I have to create an Azure Function App which holds one or more functions:

I use C# as the programming language so I choose .Net Core as runtime stack.

Once the Function app is created, I create a new function, triggered by an HTTP call:

The code behind the function is very simple. I just decode the message coming in to JSON so I can show it in the log:

The Azure function, triggered by an HTTP Post call, is now created. The only thing I still need is the actual URL of this function. So, just select ‘Get function URL’:

With this URL copied, go back to the TTN portal and add a webhook integration to our application:

In the new integration, just fill in the name of your choice, the webhook format being ‘JSON’, and the Azure Function URL:

Note: the API key for security is part of the URL.

The last thing to do is you to check the ‘Uplink message’ checkbox. As you can see, more message types are available but for now, we are only interested in the uplink messages.

Save the webhook registration.

Once a message arrives at the TTN V3 stack, it is relayed to the webhook. Here it arrives at the Azure function:

A more readable version of the message is:

{
    "end_device_ids": {
        "device_id": "blps01",
        "application_ids": {
            "application_id": "bosch-parking-lot"
        },
        "dev_eui": "",
        "join_eui": "",
        "dev_addr": "2700003D"
    },
    "correlation_ids": [
        "as:up:",
        "gs:conn:",
        "gs:uplink:",
        "ns:uplink:",
        "rpc:/ttn.lorawan.v3.GsNs/HandleUplink:"
    ],
    "received_at": "2020-06-09T11:28:08.570295323Z",
    "uplink_message": {
        "session_key_id": "",
        "f_port": 1,
        "f_cnt": 3,
        "frm_payload": "AA==",
        "decoded_payload": {
            "bytes": [
                0
            ],
            "occupied": false,
            "packet_type": "Status",
            "port": 1
        },
        "rx_metadata": [
            {
                "gateway_ids": {
                    "gateway_id": "",
                    "eui": ""
                },
                "time": "2020-06-09T11:27:54.288752Z",
                "timestamp": 3386619906,
                "rssi": -76,
                "channel_rssi": -76,
                "snr": 10.5,
                "uplink_token": "",
                "channel_index": 3
            }
        ],
        "settings": {
            "data_rate": {
                "lora": {
                    "bandwidth": 125000,
                    "spreading_factor": 10
                }
            },
            "data_rate_index": 2,
            "coding_rate": "4/5",
            "frequency": "867100000",
            "timestamp": 3386619906,
            "time": "2020-06-09T11:27:54.288752Z"
        },
        "received_at": "2020-06-09T11:28:08.351414526Z",
        "confirmed": true
    }
}

Note: I removed some keys.

Conclusion

The team of The Things Network has created a brand new portal on top of their V3 stack. There are tons of new features in there, still it feels quite familiar when adding applications, end-devices, and gateways

In this blog, we used the ‘ classic’ credentials to register our device. Although not presented, the support for join servers is very promissing. This will take away most of the hassle regarding the technical settings.

As we have seen, the Bosch parking lot sensor is simple to set up. The provided uplink payload formatter needs some attention. Luckily, you only have to rename the port parameter to get things running.

The Azure Function endpoint makes it possible to connect two worlds by offloading uplink messages. The same technique is used in the Azure IoT Central bridge which makes it possible to connect to the Azure IoT Central Saas solution for IoT.

This blog is just a first impression. For more indept information about the stack and portal, please visit the official The Things Network documentation.

I’d like to acknowledge the assistance of Laurens Slats from The Things Network @thethingsntwrk @LoRaWANAcademy instructor and organizer of The Things Conference for the access to the preview portal. Also, I’d like to acknowledge the support of Sven Nitsche from Bosch Connected Devices and Solutions @Bosch_BCDS for getting the parking lot sensor connected.