Microsoft has introduced the possibility to store data at the edge with Azure Blob Storage on IoT Edge. It is currently in preview. At this moment, that latest version is mcr.microsoft.com/azure-blob-storage:1.0.1-linux-amd64.
Update 2019-08-21: This IoT Edge module is now made generally available.
This local blob storage is another way to persist data locally on the Edge.
Let’s see how we can use this in our projects.
Azure Blob Storage is similar to SQL Server on the Edge. You deploy a container which stores the data in a local volume, a shared folder on the host system.
The Blob Storage container does not listen to the IoT Edge routing. I hope this will be supported in the near future. No, you will have to call the module using custom source code with a reference to the WindowsAzure.Storage NuGet package.
Deploying the module
Here I deploy to an Edge device running Linux using Linux containers in Docker:
Name: blob
Image URI: mcr.microsoft.com/azure-blob-storage:latest
Container Create options:
{
"Env": [
"LOCAL_STORAGE_ACCOUNT_NAME=blobaccount",
"LOCAL_STORAGE_ACCOUNT_KEY=iU6uTvlF1ysppmft+NO5lAD0E3hwrAORr5Rb5xcBWUgEz/OicrSkFxwZYMNK5XL29/wXZKGOoOVSW040nAOfPg=="
],
"HostConfig": {
"Binds": [
"/srv/containerdata:/blobroot"
],
"PortBindings": {
"11002/tcp": [
{
"HostPort": "11002"
}
]
}
}
}
The storage account name and key are made up by me, I can reuse the same combination on multiple devices. But I need these credentials later on.
Note: the accountname only supports lowercase characters, maximum length is 24. The key is just some random base64 string of 64 characters in total. It is not related to any real-world azure resource!
I use this ‘key generator’:
using System;
using System.Text;
class Program
{
static void Main(string[] args)
{
var key = GenerateBase64KeyWithLength64();
Console.WriteLine($"Length {key.Length} - {key}");
Console.WriteLine("Press a key to exit...");
Console.ReadKey();
}
public static string GenerateBase64KeyWithLength64()
{
var key = string.Empty;
for (int i = 0; i < 46; i++)
{
key += (char)new Random().Next(32, 127);
}
var plainTextBytes = Encoding.UTF8.GetBytes(key);
return Convert.ToBase64String(plainTextBytes);
}
}
The Host Config Bind “blobroot” must not be changed. The name is expected to be there.
The ‘/srv/containerdata’ is a local folder. This folder must exist already! I gave it elevated write rights.
sudo chmod 666 /srv/containerdata
Note: If you deploy multiple blob modules, try to avoid sharing the same local folder. This can be fixed by using a path like ‘/srv/[module name]/containerdata’.
The port binding 11002 can be changed if you have multiple Blob Storage containers running.
Deploy the module. You do not need extra routing for this module.
At this moment, I have some issues deploying this module on Linux. But it works like a charm on Windows 10 with Linux containers.
Calling the module
I tested the blob storage with a C# .Net Core application. Yes, it’s possible to access the blob storage outside Azure IoT Edge:
using Microsoft.WindowsAzure.Storage;
using Microsoft.WindowsAzure.Storage.Blob;
using System;
using System.IO;
namespace BlobContainerConsoleApp
{
class Program
{
static void Main(string[] args)
{
ReadWriteBlockBlob();
Console.WriteLine("Press a key to exit");
Console.ReadKey();
}
private static void ReadWriteBlockBlob()
{
// Blob container construction
var account = CloudStorageAccount.Parse(
"DefaultEndpointsProtocol=https;BlobEndpoint=http://[ip address]:11002/blobaccount;AccountName=blobaccount;AccountKey=[key]");
var client = account.CreateCloudBlobClient();
var container = client.GetContainerReference("blockblobcontainer");
container.CreateIfNotExistsAsync().Wait();
var now = DateTime.Now;
var filename = "File" + now.ToString("yyyyMMddHHmm");
var blob = container.GetBlockBlobReference(filename);
// Do we already have a blob?
var exists = blob.ExistsAsync().Result;
if (!exists)
{
blob.UploadTextAsync($"Written text at {DateTime.Now.ToLocalTime()}");
Console.WriteLine("File saved");
}
else
{
using (Stream stream = blob.OpenReadAsync().Result)
{
using (StreamReader reader = new StreamReader(stream))
{
while (!reader.EndOfStream)
{
Console.WriteLine($"File read: {reader.ReadLine()}");
}
}
}
}
}
}
}
Check out the connection string used:
DefaultEndpointsProtocol=https;BlobEndpoint=http://[ip address]:11002/blobaccount;AccountName=blobaccount;AccountKey=[key]
The blob endpoint is a combination of the local IP address and the port assigned to the container. And I need the right credentials.
Note: this is the ‘external’ address of the Blob storage, outside Docker. Inside another container, replace the IP address with the name of the Blob Storage Container.
At this moment, we only use the ‘BlobEndpoint’ locally, we call the blob storage outside the IoT Edge, hence the IP address.
If I run this code the first time, a container is constructed (if not available yet) and a file is created:
And if I run the same code within a minute, I read the same file and show the text inside the blob file:
So here is proof we actually call the same blob.
Check out the storage
Are you interested in the actual storage?
You can peek inside the running module using Powershell:
docker exec -it [blob storagecontainer name] bash eg. docker exec -it blob bash
This opens a ‘bash’ shell inside the container.
Navigate to the blob storage folder:
Inside this folder, you can navigate to the container used and check out the file:
As you can see, the same text is shown. We can check out the actual blobs created.
Conclusion
This is a very early preview of the local blob storage support of Azure blob storage on the IoT Edge. It’s still limited in usage, we can work with Block Blobs, but we can make good use of it already when working with files.
I was hoping to make use of the CloudAppendBlob. This would make it easy to combine multiple (routed) messages into one blob. But this format is not supported (yet). Check out the documentation to see the supported storage operations.
Update: this seems possible with the new blockblob support where we could add blocks to the file.
Sander van de Velde 