Introducing Blob storage in Azure IoT on the Edge

Microsoft has introduced the possibility to store data at the edge with Azure Blob Storage on IoT Edge. It is currently in preview. At this moment, that latest version is mcr.microsoft.com/azure-blob-storage:1.0.1-linux-amd64.

This local blob storage is another way to persist data locally on the Edge.

Let’s see how we can use this in our projects.

Azure Blob Storage is similar to SQL Server on the Edge. You deploy a container which stores the data in a local volume, a shared folder on the host system.

The Blob Storage container does not listen to the IoT Edge routing. I hope this will be supported in the near future. No, you will have to call the module using custom source code with a reference to the WindowsAzure.Storage NuGet package.

Deploying the module

Here I deploy to an Edge device running Windows 10 using Linux containers in Docker:

Name: blob
Image URI: mcr.microsoft.com/azure-blob-storage:latest

Container Create options:
{
  "Env": [
    "LOCAL_STORAGE_ACCOUNT_NAME=blobaccount",
    "LOCAL_STORAGE_ACCOUNT_KEY=iU6uTvlF1ysppmft+NO5lAD0E3hwrAORr5Rb5xcBWUgEz/OicrSkFxwZYMNK5XL29/wXZKGOoOVSW040nAOfPg=="
  ],
  "HostConfig": {
    "Binds": [
      "/srv/containerdata:/blobroot"
    ],
    "PortBindings": {
      "11002/tcp": [
        {
          "HostPort": "11002"
        }
      ]
    }
  }
}

The storage account name and key are made up by me, I can reuse the same combination on multiple devices. But I need these credentials later on.

Note: the accountname only supports lowercase characters, maximum length is 24.

The Host Config Bind “blobroot” must not be changed. The name is expected to be there.

The port binding 11002 can be changed if you have multiple Blob Storage containers running.

Deploy the module. You do not need extra routing for this module.

At this moment, I have some issues deploying this module on Linux. But it works like a charm on Windows 10 with Linux containers.

Calling the module

I tested the blob storage with a C# .Net Core application. Yes, it’s possible to access the blob storage outside Azure IoT Edge:

using Microsoft.WindowsAzure.Storage;
using Microsoft.WindowsAzure.Storage.Blob;
using System;
using System.IO;

namespace BlobContainerConsoleApp
{
  class Program
  {
    static void Main(string[] args)
    {
      ReadWriteBlockBlob();

      Console.WriteLine("Press a key to exit");
      Console.ReadKey();
    }

    private static void ReadWriteBlockBlob()
    {
      // Blob container construction
      var account = CloudStorageAccount.Parse("DefaultEndpointsProtocol=https;BlobEndpoint=http://192.168.1.71:11002/blobaccount;AccountName=blobaccount;AccountKey=iU6uTvlF1ysppmft+NO5lAD0E3hwrAORr5Rb5xcBWUgEz/OicrSkFxwZYMNK5XL29/wXZKGOoOVSW040nAOfPg==");

      var client = account.CreateCloudBlobClient();

      var container = client.GetContainerReference("blockblobcontainer");

      container.CreateIfNotExistsAsync().Wait();

      var now = DateTime.Now;

      var filename = "File" + now.ToString("yyyyMMddHHmm");

      var blob = container.GetBlockBlobReference(filename);

      // Do we already have a blob?
      var exists = blob.ExistsAsync().Result;

      if (!exists)
      {
        blob.UploadTextAsync($"Written text at {DateTime.Now.ToLocalTime()}");
        Console.WriteLine("File saved");
      }
      else
      {
        using (Stream stream = blob.OpenReadAsync().Result)
        {
          using (StreamReader reader = new StreamReader(stream))
          {
            while (!reader.EndOfStream)
            {
              Console.WriteLine($"File read: {reader.ReadLine()}");
            }
          }
        }
      }
    }
  }
}

Check out the connection string used:

DefaultEndpointsProtocol=https;BlobEndpoint=http://192.168.1.71:11002/blobaccount;AccountName=blobaccount;AccountKey=iU6uTvlF1ysppmft+NO5lAD0E3hwrAORr5Rb5xcBWUgEz/OicrSkFxwZYMNK5XL29/wXZKGOoOVSW040nAOfPg==

The blob endpoint is a combination of the local IP address and the port assigned to the container. And I need the right credentials.

Note: this is the ‘external’ address of the Blob storage, outside Docker. Inside another container, replace the IP address with the name of the Blob Storage Container.

At this moment, we only use the ‘BlobEndpoint’ locally, we call the blob storage outside the IoT Edge, hence the IP address.

If I run this code the first time, a container is constructed (if not available yet) and a file is created:

And if I run the same code within a minute, I read the same file and show the text inside the blob file:

So here is proof we actually call the same blob.

Check out the storage

Are you interested in the actual storage?

You can peek inside the running module using Powershell:

docker exec -it [blob storagecontainer name] bash
eg. docker exec -it blob bash

This opens a ‘bash’ shell inside the container.

Navigate to the blob storage folder:

Inside this folder, you can navigate to the container used and check out the file:

As you can see, the same text is shown. We can check out the actual blobs created.

Conclusion

This is a very early preview of the local blob storage support of Azure blob storage on the IoT Edge. It’s still limited in usage, we can work with Block Blobs, but we can make good use of it already when working with files.

I was hoping to make use of the CloudAppendBlob. This would make it easy to combine multiple (routed) messages into one blob. But this format is not supported (yet). Check out the documentation to see the supported storage operations.

 

 

 

Advertenties