Bulk import of IoTHub devices

This blog post is for the hardcore IoTHub users. It’s even a bit boring, at first.

The Azure IoTHub does not accept anonymous telemetry. Telemetry has to be presented by devices which are enabled. So you need to have a list of all your devices. You have to manage it.

In this post, we start diving into registering a single device and we will end updating multiple devices in bulk.

What we need is just a single Console app. We will use the NuGet package for Microsoft.Azure.Devices. This package does not accept a UWP app. And it does not support .Net 4.5 or lower. So make sure you select .Net Target Framework 4.6.1 eg.

Now add the following lines of code:

private static RegistryManager registryManager;

private static string connectionString = "[IoTHub connectionstring]";
...

registryManager = RegistryManager.CreateFromConnectionString(connectionString);
...

private async static Task AddDeviceAsync()
{
    string deviceId = "DeviceThree";
    Device device;
    try
    {
        Console.WriteLine("New device:");
 
        var d = new Device(deviceId);
 
        device = await registryManager.AddDeviceAsync(d);
    }
    catch (DeviceAlreadyExistsException)
    {
        Console.WriteLine("Already existing device:");
        device = await registryManager.
        GetDeviceAsync(deviceId);
    }
 
    Console.WriteLine("Generated device key: {0}",
    device.Authentication.SymmetricKey.PrimaryKey);
}

If you fill in the connection string of an IoTHub and you create a RegistryManager, you can create a new device.

To update a device already created, use these lines of code:

private async static Task UpdateDeviceAsync()
{
    try
    {
        Console.WriteLine("Update device");
 
        var d = await registryManager.GetDeviceAsync("DeviceThree");
 
        if (d != null)
        {
            d.Status = DeviceStatus.Disabled;
            d.StatusReason = "Disabled for test";
 
            var dd = await registryManager.UpdateDeviceAsync(d);
        }
        else
        {
            Console.WriteLine("Device not found");
        }
    }
    catch (Exception ex)
    {
        Console.WriteLine($"Exception {ex.Message}");
    }
}

I disable a certain device. This is almost all you can do with a device 🙂

Just for fun, here is the source code for deletion. You can do that too:

private async static Task RemoveDeviceAsync()
{
    try
    {
        Console.WriteLine("Remove device");
 
        await registryManager.RemoveDeviceAsync("DeviceThree");
    }
    catch (Exception ex)
    {
        Console.WriteLine($"Exception {ex.Message}");
    }
}

After creating, updating and removing devices, let’s get a list of registered devices:

private async static Task GetMaxOneThousandDevicesAsync()
{
    try
    {
        Console.WriteLine("Get devices");
 
        var devices = await registryManager.GetDevicesAsync(Int32.MaxValue);
 
        foreach (var device in devices)
        {
            Console.WriteLine($"Id {device.Id} - Status {device.Status} - Reason {device.StatusReason}");
        }
    }
    catch (Exception ex)
    {
        Console.WriteLine($"Exception {ex.Message}");
    }
}

Tadaah, here are the devices. But this list is limited to a thousand devices. And that’s by design. So how do we get all devices? Well, this can be done in a job… It’s just another way to read them asynchronously.

Export devices in batch

What we need is just some blob storage for exporting the full list of devices to. These will we exported to a blob in a blob storage. And this is also a great way to backup your devices including all keys etc. So make sure your blob storage container is secure (not public), you do not want to expose this information.

So register a blob storage account in your Azure portal, add the Azure Storage NuGet package to the code and run the code below:

private async static Task ExportDevicesAsync()
{
    try
    {
        //// Get SAS token of (new) container
 
        string storageAccessKey = "DefaultEndpointsProtocol=https;AccountName=[];AccountKey=[]";
 
        var account = CloudStorageAccount.Parse(storageAccessKey);
 
        var client = account.CreateCloudBlobClient();
 
        var container = client.GetContainerReference("iot");
 
        await container.CreateIfNotExistsAsync();
 
        var permissions = new BlobContainerPermissions
        {
            PublicAccess = BlobContainerPublicAccessType.Off
        };
 
        permissions.SharedAccessPolicies.Add(
            "saspolicy",
            new SharedAccessBlobPolicy()
            {
                SharedAccessExpiryTime = DateTime.UtcNow.AddHours(1),
                Permissions = SharedAccessBlobPermissions.Write
                                    | SharedAccessBlobPermissions.Read
                                    | SharedAccessBlobPermissions.Delete
            });
 
        container.SetPermissions(permissions);
 
        var sasToken = container.GetSharedAccessSignature(new SharedAccessBlobPolicy(), "saspolicy");
 
        //// export devices
 
        Console.WriteLine("Export devices");
 
        var containerSasUri = container.Uri + sasToken;
 
        //// Call an export job on the IoT Hub to retrieve all devices
        var job = await registryManager.ExportDevicesAsync(containerSasUri, false);
 
        while (true)
        {
            job = await registryManager.GetJobAsync(job.JobId);
 
            if (job.Status == JobStatus.Completed
                    || job.Status == JobStatus.Failed
                    || job.Status == JobStatus.Cancelled)
            {
                // Job has finished executing
 
                break;
            }
 
            await Task.Delay(TimeSpan.FromSeconds(5));
        }
 
        var blob = container.GetBlobReference("devices.txt");
 
        Console.WriteLine("Devices Exported");
 
        //// read exported blob
 
        // read blob with devices.txt into list of device objects
 
        var exportedDevices = new List<ExportImportDevice>();
 
        using (var streamReader = new StreamReader(await blob.OpenReadAsync(), Encoding.UTF8))
        {
            while (streamReader.Peek() != -1)
            {
                string line = await streamReader.ReadLineAsync();
                var device = JsonConvert.DeserializeObject<ExportImportDevice>(line);
                exportedDevices.Add(device);
 
                Console.WriteLine($"Device {device.Id} Status {device.Status}");
            }
        }
    }
    catch (Exception ex)
    {
        Console.WriteLine($"Exception {ex.Message}");
    }
}

This code will first construct a container in your blob storage account. For that container, an SAS token is constructed. Now you can create a file named devices.txt in the storage. This file will be filled by code with device information. This takes some time so we loop for a while to check if the job is ready.

When ready, the following file is created:

{ "id":"DeviceTwo","eTag":"MQ==","status":"enabled","authentication":{ "symmetricKey":{ "primaryKey":"[key]","secondaryKey":"[key]"},"x509Thumbprint":{ "primaryThumbprint":null,"secondaryThumbprint":null} } }
{ "id":"DeviceOne","eTag":"MA==","status":"enabled","authentication":{ "symmetricKey":{ "primaryKey":"[key]","secondaryKey":"[key]"},"x509Thumbprint":{ "primaryThumbprint":null,"secondaryThumbprint":null} } }
{ "id":"DeviceThree","eTag":"MQ==","status":"disabled","statusReason":"Disabled for test","authentication":{ "symmetricKey":{ "primaryKey":"[key]","secondaryKey":"[key]"},"x509Thumbprint":{ "primaryThumbprint":null,"secondaryThumbprint":null} } }

And in the end, we iterate through the file to construct the devices  in memory.

Import devices in batch

The file just created can be used to import, add, delete or update devices in batch. To do that, you first have to alter the blob. You can do that using some blob editor (like this one).

Here I have disabled all devices:

{ "id":"DeviceTwo","eTag":"MQ==","status":"disabled","statusReason":"Disable for test 2","authentication":{ "symmetricKey":{ "primaryKey":"[key]","secondaryKey":"[key]"},"x509Thumbprint":{ "primaryThumbprint":null,"secondaryThumbprint":null} } }
{ "id":"DeviceOne","eTag":"MA==","status":"disabled","statusReason":"Disable for test 1","authentication":{ "symmetricKey":{ "primaryKey":"[key]","secondaryKey":"[key]"},"x509Thumbprint":{ "primaryThumbprint":null,"secondaryThumbprint":null} } }
{ "id":"DeviceThree","eTag":"MQ==","status":"disabled","statusReason":"Disabled for test","authentication":{ "symmetricKey":{ "primaryKey":"[key]","secondaryKey":"[key]"},"x509Thumbprint":{ "primaryThumbprint":null,"secondaryThumbprint":null} } }

After changing the file, you will have to consume the blob, telling the logic what has to be done. Do you want to insert? Do you want to update? Or do you want to Delete? In this example, we combine insert and update: createOrUpdate.

This is the default behavior. How? Each device has a property ImportMode. For example, you can change it into update but this will result in an error if a certain device does not exist. For now, known devices will be updated, unknown devices will be added.:

private async static Task UpdateDevicesAsync()
{
    try
    {
        //// Get SAS token of (new) container
 
        string storageAccessKey = "DefaultEndpointsProtocol=https;AccountName=lorastorage;AccountKey=[key]";
 
        var account = CloudStorageAccount.Parse(storageAccessKey);
 
        var client = account.CreateCloudBlobClient();
 
        var container = client.GetContainerReference("iot");
 
        await container.CreateIfNotExistsAsync();
 
        var permissions = new BlobContainerPermissions
        {
            PublicAccess = BlobContainerPublicAccessType.Off
        };
 
        permissions.SharedAccessPolicies.Add(
            "saspolicy",
            new SharedAccessBlobPolicy()
            {
                SharedAccessExpiryTime = DateTime.UtcNow.AddHours(1),
                Permissions = SharedAccessBlobPermissions.Write
                                    | SharedAccessBlobPermissions.Read
                                    | SharedAccessBlobPermissions.Delete
            });
 
        container.SetPermissions(permissions);
 
        var sasToken = container.GetSharedAccessSignature(new SharedAccessBlobPolicy(), "saspolicy");
 
        //// import devices
 
        Console.WriteLine("Import devices");
 
        var containerSasUri = container.Uri + sasToken;
 
        // Call import using the same storage blob to add new devices!
        // This normally takes 1 minute per 100 devices the normal way
        JobProperties importJob = await registryManager.ImportDevicesAsync(containerSasUri, containerSasUri);
 
        // Wait until job is finished
        while (true)
        {
            importJob = await registryManager.GetJobAsync(importJob.JobId);
            if (importJob.Status == JobStatus.Completed ||
                importJob.Status == JobStatus.Failed ||
                importJob.Status == JobStatus.Cancelled)
            {
                // Job has finished executing
                break;
            }
 
            await Task.Delay(TimeSpan.FromSeconds(5));
        }
 
        Console.WriteLine("Devices imported");
    }
    catch (Exception ex)
    {
        Console.WriteLine($"Exception {ex.Message}");
    }
}

Again we access the same container with an SAS token. This time, we tell the logic to import the devices from the blob and we wait until this is ready.

Note: Importing devices is slow!

Do you get this error?

After importing, my code ended with an error:  “ErrorCode:DMImportDevicesNotSupported;ImportDevices not supported on hubs with device management enabled”. No updates were executed.

I had to think hard about this error but then I remembered that checkbox during construction of the IoTHub:

enabledevicemanagement01

That checkbox should give access to the list of devices within the Azure portal. So I constructed a second IoTHub without that checkbox…

And with that hub, the import worked like a charm:

enabledevicemanagement02

Note: I compared both hubs and both have the same features; they both show the same blade, showing the devices in the hub. (update: this is by design during the preview. After the preview this restriction is gone.)

This gives use more power to create a multitude of devices in bulk. When there are multiple programs changing the same devices, it’s good to take the etag property of devices into account too. The devices will only be updated I the etag of devices is still valid.

Advertenties