Get a bundle of support files from your Azure IoT Edge for remote diagnostics

Azure IoT Edge makes it possible to run extensive logic within your factory, building, vehicle, etc. while it’s connected to the cloud.

This way, we can monitor the underlying sensors and protocols and measure what’s happening on the Edge. We are even capable of making predictions, both on the Edge and in the Cloud, of what is going to happen based on the current measurements and the data received over the last second, minute, hour, etc.

On a meta-level, Azure IoT Edge also comes with features for monitoring the edge device itself. Think about monitoring metrics and logging.

These features are mostly centralized around the edge modules and runtime. Due to the edge logic being sand-boxed, this is fine.

Still, we want to be able to go beyond the logic we deployed.

It would be nice if we would be able to break out of that sandbox and get some information about the Docker/Moby environment, IoT Edge runtime daemon, network, etc.:

This is actually offered!

Azure IoT Edge offers a so-called ‘support bundle’.

It is just a bundle of files with eg. logs, taken from various sources on the edge device and it is made available so you can support your edge device.

It contains:

• Module logs
• IoT Edge security manager logs
• Container engine logs
• ‘iotedge check‘ JSON output
• Other useful debug information

It’s even possible to retrieve these files ‘over-the-air’. This makes remote diagnostics possible for all your Azure IoT Edge devices!

Let’s take a closer look at this.

Note: This blog post can be considered as part two of Towards zero-touch IoT Edge with edgeAgent direct methods.

Let’s start with generating this support bundle right of the device.

Just run this command on the device with Azure IoT Edge deployed:

sudo iotedge support-bundle

This results in this output:

An extensive list of log files being collected is shown. It results in a zip file named ‘support_bundle.zip’:

We can unzip this file locally:

sudo mkdir sb
sudo unzip support_bundle.zip -d ~/sb

Note: perhaps you need to download this unzip app:

sudo apt install unzip

This results in these files:

Notice it even comes with the output of ‘sudo iotedge check’:

The rest of the files are made available over subdirectories:

Note: As you can see, I only installed the simulated temperature sensor 🙂

You can play around with some settings to get a certain timeframe:

E.g. you could limit the output to only the last six hours:

sudo iotedge support-bundle --since 6h

Still, these files are just taken from the same device as you are logged in to. So it only saves you some time to collect the files but you still have to log in by hand to this device.

How convenient should it be if collecting the support bundle could be automated? To the cloud?

EdgeAgent UploadSupportBundle direct method

A short introduction

The EdgeAgent module supports multiple helpful direct methods I blogged about already.

As seen in my previous blog post, the ‘UploadModuleLogs’ is a direct method that can be executed asynchronously. This means you start the method and directly get a response back that only shows the upload has started (and a unique correlation id).

Using the returned correlation id you can make subsequent calls using the ‘GetTaskStatus’ direct method to see the status of the current upload.

Where are the log files made available, in the end?

The output of ‘UploadModuleLogs’ is persisted in an Azure Storage account blob container.

You only have to provide a security token, a SAS key, to the direct method call so the edge device has some limited secure access to that container.

This same flow is also used for the support bundle.

Let’s check this out.

Calling UploadSupportBundle direct method

In this example, I call the edge agent module direct method using the Azure portal:

Note: all direct method calls to the edge agent are made case-insensitive.

As you can see, we need to provide this SAS URL.

How do we get it?

Just add a new (private) container to an Azure Blob storage and go to the specific Settings pane.

This is a security token that gives access to a specific container in a blob storage account:

Notice this token has only write permissions and an expiration date in the (far) future.

When we hit the button, the key is generated:

We now have a key to grand the edge device temporary access to the container.

Once the direct method call is made, we directly get this answer:

It just shows the request is running. A specific correlation id for this single request is made available.

This is done because generating and uploading the files can take some time (eg. due to available compute power and network connection speed). Direct Methods are blocking calls by default. So, this limits the risk of a timeout.

Calling GetTaskStatus direct method

We can now check the request lifetime over time using the GetTaskStatus direct method:

This results in several possible status updates, NotStarted, Running, Completed, Failed, ‘Cancelled’, or Unknown.

Here, the request is already completed:

So, the support bundle should be uploaded already.

We can see it being persisted in the blob storage container when we go back to the container and refresh the content:

A new zip file is shown. See that a ‘folder’ is created for the device we accessed.

If we download the file, we see the same set of files:

Azure Service ticket

If you supply a service ticket to the Azure support, you can attach the support bundle to give the support desk more information.

Do not share the support bundle on a public platform. There is sensitive information like logfiles and module configuration settings in the bundle.

Conclusion

We now have a convenient way to see what’s happening on an Azure IoT Edge device without having to access the device locally.

All parts of this flow can be automated and picked up by cloud logic for further processing.

Advertentie