Running ML.Net models inside an Azure IoT Edge module

Getting started with machine learning is not easy. This is the domain of the Data Scientist and to understand the different models leads you into trying to understand the mathematical part of it.

Still, if you see a machine learning model as a black box, things start to get a little bit easier.

One of the solutions Microsoft offers to developers for getting familiar with machine learning, training models, and deploying them with code, is ML.Net.

Or as Microsoft says:

With ML.NET, you can create custom ML models using C# or F# without having to leave the .NET ecosystem.

In fact, it runs on .Net Core so technically, this should run on multiple operating systems, including Linux; on Intel and Arm processors…

Let’s see how to start with ML.Net and how to integrate it with Azure IoT Edge

For this blog, I make use of Visual Studio. A CLI version of ML.Net is offered too.

Building your first model

So first, open your Visual Studio.

We are looking for the Model builder. This will offer us a wizard for building some models in just a few steps:

  • Read the input data from a text file for model training
  • Build/Setup your data processing and training pipeline
  • Train your model
  • Make predictions using your trained model

Installing the model builder

Currently, the model builder is offered as an extension:

Install it and if needed, update it too.

After being installed, the model builder must be enabled because it’s still in preview. Just type in ‘model builder’ in the Visual Studio search bar:

This brings us to the options dialog:

Just enable the ML.Net Model Builder option.

The last step is to start an empty .Net Core console app. This is a bit of an unusual step but later on, we will see what to do with it.

So, once you have the default console app generated, add ‘Machine learning’ (right-clicking the project):

This will result in a new pane:

This ML.Net Model Builder (1) lets you select the specific scenario for your model (2). For each step, we have to follow a wizard which roughly follows the steps as described before. Any console output during these steps will be shown below (4).

As you can see, some models need Azure ML (2b) to be trained and generated. Others can be trained and generated on the local device (2a).

Note: At this moment, a hand full of ML.Net wizards are offered. There are also a few models in preview (scroll down to the bottom of the pane). Microsoft provides already an extensive list of prebuild models on Github. ML.Net also supports ‘non-ml.net’ models (eg. TenserFlow, ONNX). In this demonstration, we just scratch the surface using a simple model you probably understand.

Creating a Text Classification model

Here, we use the text classification example:

The purpose is of our model will be to make a prediction about the sentiment (positive or negative) of a text.

This model can be generated on my local device with the computing power of it:

Next, we have to feed the model with training material:

I already created a text file with two columns: the ‘comment’ with customer comments and the ‘sentiment’ I have given it for each row. So we offer this to ML.Net to train with (you see the examples on screen).

Note: You are free to add more lines. The more examples you add, the better the model will get.

Then, we ask ML.NET to try to predict the same values as seen in the Sentiment column, based on the other column…

Next, it’s time to train. I have to give up the duration of the training. I just put in 30 seconds because I did not want to wait any longer.

Note: this is running on my local machine so it looks like training comes for free. But it needs memory, CPU/GPU power, electricity, hardware, etc. The same goes for a model trained in the cloud.

During the training, the output becomes available in the output window:

So with just a few comment training lines in the text file, I get a model which is 60-75 percent accurate. This is a bit on the low side (85 and higher is recommended in most cases) but it is ok for now.

I have no expert knowledge about the several types of models which we tested during training. Though, it seems we have a clear winner.

Some machine learling model code seems to be generated but it’s not available yet. We first have to evaluate the model:

I entered a line ‘I do not recommend this restaurant’ which indeed has a negative sentiment. The model is sure at 98%.

Just to compare, I also evaluated a positive line:

99 percent positive, that seems ok to me 🙂

The last step is to add the code to the Visual Studio solution:

This results in two extra projects:

Exploring the code generated

The first project we check out is ‘BlogMLnetAppML.Model’ which acts as a library.

It contains:

  • A zip file which contains the machine learning model
  • A ‘ModelInput’ class and a ‘ModelOutput’ class for interaction with the model
  • A ‘ConsumeModel’ class with the public method to check the model (the class lazy loads the model and model engine)

This is pretty straight forward.

The other project ‘BlogMLnetAppML.ConsoleApp’ is a .Net core console app.

It contains a program.cs which executes:

var predictionResult = ConsumeModel.Predict(sampleData);

So, define the new console app as being the new startup app and run it with the same examples (put the sample comment at line 15):

We get the same values (2% and 98%) for the same negative comment!

And the beauty is, this is available for both C# and F#.

The project also comes with an extra ‘ModelBuilder.cs’.

This class contains code for the same simple wizard we have used in Visual Studio:

...
private static string TRAIN_DATA_FILEPATH = @"[path]training.txt";
...
public static void CreateModel()
{
    // Load Data
    IDataView trainingDataView = mlContext.Data.LoadFromTextFile<ModelInput>(
                                    path: TRAIN_DATA_FILEPATH,
                                    hasHeader: true,
                                    separatorChar: ';',
                                    allowQuoting: true,
                                    allowSparse: false);

    // Build training pipeline
    IEstimator<ITransformer> trainingPipeline = BuildTrainingPipeline(mlContext);

    // Train Model
    ITransformer mlModel = TrainModel(mlContext, trainingDataView, trainingPipeline);

    // Evaluate quality of Model
    Evaluate(mlContext, trainingDataView, trainingPipeline);

    // Save model
    SaveModel(mlContext, mlModel, MODEL_FILE, trainingDataView.Schema);
}
...

Note: Although the training part is not actually demonstrated with sample code, training and model generation should be possible by code too. This means you are able to retrain your model over and over again.

There is one more thing to do… What about that initial console app?

I noticed the initial training file is not part of the generated code…

So, I simply renamed the initial project, deleted that program.cs, and added the training.txt so it’s under source control now:

Migrating to Azure IoT Edge

Does it stop with this?

No 🙂

This blog is all about Azure IoT so let’s see how we can integrate the code into an Azure IoT Edge module.

TLDR: Check out the working IoT edge modules and the final source code examples on GitHub.

Building the module with the model logic

Now it’s time to switch over to Visual Studio Code and start a new Azure IoT Edge solution together with a new C# custom module:

and:

Then, copy the four files of the ‘BlogMLnetAppML.Model’ project to your model:

The .zip file must be part of the deliverables of the project, so we have to make this clear in the .csproj file (1):

We also need to refer to the ML.Net NuGet package (2).

The original code of the generated IoT Edge module is triggered on messages coming from ‘input1’ and sends them to ‘output1’.

For our ML.Net module, we do the same but here we analyze the incoming message comment for the sentiment and send the outcome to the output:

static async Task<MessageResponse> PipeMessage(Message message, object userContext)
{
    int counterValue = Interlocked.Increment(ref counter);

    var moduleClient = userContext as ModuleClient;
    if (moduleClient == null)
    {
        throw new InvalidOperationException("UserContext doesn't contain " + "expected values");
    }

    byte[] messageBytes = message.GetBytes();
    string messageString = Encoding.UTF8.GetString(messageBytes);
    Console.WriteLine($"Received message: {counterValue}, Body: [{messageString}]");

    if (string.IsNullOrEmpty(messageString))
    {
        System.Console.WriteLine($"Request '{messageString}' cannot be converted into the right request. Ignored.");

        return MessageResponse.Completed;
    }

    var request = JsonConvert.DeserializeObject<Request>(messageString);

    ModelInput modelInput = new ModelInput()
    {
        Comment = request.comment,
    };

    try
    {
        // Make a single prediction on the sample data and print results
        var predictionResult = ConsumeModel.Predict(modelInput);

        var scores = predictionResult.Score.Select(x => new Score{entry = x}).ToArray();

        var response = new Response()
        {
            comment = modelInput.Comment,
            prediction = predictionResult.Prediction,
            scores = scores,
        };

        var jsonString = JsonConvert.SerializeObject(response);

        if (!string.IsNullOrEmpty(jsonString))
        {
            using (var pipeMessage = new Message(UTF8Encoding.UTF8.GetBytes(jsonString)))
            {
                await moduleClient.SendEventAsync("output1", pipeMessage);
            
                Console.WriteLine($"Scored '{response.comment}' message ({response.prediction}) sent.");

                System.Console.Write("scores: ");

                foreach(var s in response.scores)
                {
                    System.Console.Write($"{s.entry}; ");
                }

                System.Console.WriteLine();
            }
        }
        else
        {
            System.Console.WriteLine($"Response '{jsonString}' cannot be converted into the right reponse. Ignored.");
        }
    }
    catch(Exception ex)
    {
        System.Console.WriteLine($"Exception: {ex.Message}");
    }

    return MessageResponse.Completed;
}
}

public class Request
{
public string comment {get; set;}
}

public class Response
{
public string comment {get; set;}
public string prediction {get; set;}

public Score[] scores {get; set;}
}

public class Score
{
public float entry {get; set;}
}

Therefore, the code is quite simple. It just reads the comment from the incoming message, it executes ‘var predictionResult = ConsumeModel.Predict(modelInput);’ and then it both sends the response to the module output and shows the response in the console.

Most of the work is put in encoding/decoding messages and JSON strings.

This code of the module can now be build and pushed, except for one extra step.

Docker Build must be made aware of the zip file which comes with the executable code. The model has to be copied into the docker image too:

At line four of the Docker build script, the zip file containing the model is copied into the image.

Pro tip: This resulting module is ready for use already on Docker Hub.

Setting up a test for the text classification module

The text classification module is now connected to the Azure IoT Edge routing. So we need a mechanism to put new comments on the route to be evaluated.

For this, a second module is introduced. So, In the project a second project is available which reacts on calls for the direct method named ‘meassureSentiment’:

    ...
    await ioTHubModuleClient.SetMethodHandlerAsync(
            "meassureSentiment",
            MeassureSentimentCallBack,
            ioTHubModuleClient);

    System.Console.WriteLine("Direct method 'meassureSentiment' is now attached.");
}

private static async Task<MethodResponse> MeassureSentimentCallBack(MethodRequest methodRequest, object userContext)
{
    Console.WriteLine($"MeassureSentiment started at {DateTime.UtcNow}");

    var moduleClient = userContext as ModuleClient;
    
    if (moduleClient == null)
    {
        throw new InvalidOperationException("UserContext doesn't contain " + "expected values");
    }

    try
    {
        var request = JsonConvert.DeserializeObject<Request>(methodRequest.DataAsJson);

        var jsonString = JsonConvert.SerializeObject(request);

        using (var pipeMessage = new Message(UTF8Encoding.UTF8.GetBytes(jsonString)))
        {
            Console.WriteLine($"Message '{request.comment}' sent");

            await moduleClient.SendEventAsync("output1", pipeMessage);
        }

        Console.WriteLine($"MeassureSentiment ready at {DateTime.UtcNow}.");
    }
    catch (Exception ex)
    {
        System.Console.WriteLine($"Exception {ex.Message}");
    }
    
    var response = new MethodResponse(200);

    return response; 
}

}

public class Request
{
public string comment {get; set;}
}

Once the method is called, the message is sent to the output name ‘output1’.

For this module, an image is already available in Docker Hub also.

The only thing left now is a route:

FROM /messages/modules/test/outputs/output1 INTO BrokeredEndpoint("/modules/text/inputs/input1")

This looks like this:

Everything is ready and connected. We are now ready to start testing the model.

Testing ML.Net in Azure IoT Edge

We start by sending a negative comment as a Direct Method to the test module. We just use the same comment as seen before:

From the test module, the message is sent to the text classification module where we can see the response of the text module in the console:

The same goes for the positive message:

This proves we are able to execute the module on Linux (running on Intel architecture on this edge device).

Conclusion

ML.Net has much more to offer than only this single example. See the tutorial above as a starting point. We are now capable to support ML on the edge, yet again in another way.

With ML.Net, creating, training, and deploying machine learning models comes in reach for developers, including IoT developers.

All code is open source on GitHub.

Modules are prebuilt available on Docker Hub: Test module and Text classification.

Geef een reactie

Vul je gegevens in of klik op een icoon om in te loggen.

WordPress.com logo

Je reageert onder je WordPress.com account. Log uit /  Bijwerken )

Google photo

Je reageert onder je Google account. Log uit /  Bijwerken )

Twitter-afbeelding

Je reageert onder je Twitter account. Log uit /  Bijwerken )

Facebook foto

Je reageert onder je Facebook account. Log uit /  Bijwerken )

Verbinden met %s