Deploy Prometheus container with Azure IoT Edge

Once you have set up your Azure IoT edge device and runtime, you want to make sure it keeps running as expected. So you need to start measuring how your device, your modules, and runtime are doing.

Microsoft provides built-in metrics for Azure IoT using endpoints on the EdgeHub and EdgeAgent exposing messages in the Prometheus format.

In a previous blog post, I showed how to access these endpoints and not to send messages containing this information to the cloud using a custom Azure IoT Edge metrics module.

Azure IoT Edge can deploy whatever Docker container you have. So, we can also use the original Prometheus service as a Docker container:

This way you can build a local dashboard using well-known tooling on the edge.

Let’s check out how this works.

For this blog post, we concentrate on the EdgeAgent module.

As seen in the previous blog post, the metrics port must be exposed outside docker.

We add (or merge) the container create options of the EdgeAgent in the deployment manifest:

Here, the internal port 9600 is exposed on the host as 9601. Use another port on the host if this one is already taken.

Note: You need to know the local IP address on which the endpoint is exposed.

Once deployed, you can try to access it in the browser (on the local network) already. You will see the Prometheus format in action.

Prometheus

We will use the Prometheus logic in a docker container. This makes it possible to connect to Prometheus endpoints as seen on the EdgeAgent module and it provides basic filters and charts. On top of this, you can attach Grafana to Prometheus too so this is a very flexible solution.

Prometheus is available as a container for many (Linux based) platforms:

First, you have to create a local folder so Prometheus can store some information there. Plus, we will configure Prometheus using a configuration file at that location:

sudo mkdir /var/prometheus
sudo chmod 777 /var/prometheus

Note: this exposes the /var/prometheus folder on the Host OS to everyone.

Now, add a prometheus.yml file so we configure Prometheus to attach to the EdgeAgent output:

Here, I configure the Prometheus endpoint IP address and port of the edgeAgent:

global:
  scrape_interval: 15s

scrape_configs:
- job_name: edgeagent
  scrape_interval: 5s
  static_configs:
    - targets:
      - 10.3.0.4:9601

I use the local IP address of the machine. The /metrics name in the path is added by default.

Note: If you use an Azure VM, use the internal IP address, not the public IP address.

All we have to do now is rolling out the Prometheus container and binding it to the local /var/prometheus folder.

We extend the deployment manifest again (this is how it’s done in the Azure portal):

Just copy and paste:

Name: prometheus
URI: prom/prometheus:latest
Container Create options:
{
  "ExposedPorts": {
    "9090/tcp": {}
  },
  "HostConfig": {
    "Binds": [
      "/var/prometheus:/etc/prometheus"
    ],
    "PortBindings": {
      "9090/tcp": [
        {
          "HostPort": "9090"
        }
      ]
    }
  }
}

Here, I expose the Prometheus website on port 9090 at the host. If this port is already taken, please provide another host port. Do not change the ‘internal’ port.

The host folder /var/prometheus is mapped to the internal /etc/prometheus folder. Do not change the second part!

Once the module is rolled out, it keeps on running:

Note: because I use an Azure VM to test this, I had to expose the Prometheus port on the Network Security Group, bound to the public IP address.

Warning: limit the access to this port (the same goes for the SSH port) because external access is open for everybody if not specified otherwise.

The Prometheus portal

Once Prometheus is running fine, check out the portal (browse to port 9090 on your edge device).

An opening screen is shown:

Go to menu Status | Targets to see the access to the EdgeAgent metrics endpoint (as we configured in the configuration file):

Make sure, the State is ‘up’. If not, probably the local IP address or port number is wrong.

Go back to the Graphs page and type/select an expression like ‘edgeAgent_used_cpu_sum’:

Prometheus knows which values are available in the incoming interface so just start typing and a list of predefined values is made available:

Note: You can also push the ‘globe’ button to the right.

Once a value is selected, Execute it. Here. a table of CPU values is shown:

Next, select Graph instead of Table:

The current values are now shown as part of a historical chart.

You can filter and highlight specific values too. You can also add multiple panels.

Note: The edgeAgent metrics update frequency is five minutes.

EdgeHub

Once we have the access to the edgeHub in place, we can add the EdgeHub too:

More metrics are exposed once more Azure IoT Edge modules are added.

Conclusion

You have seen how easy it is to deploy common Docker containers on Azure IoT Edge devices.

Here, we deployed the Prometheus container and accessed the EdgeAgent (and EdgeHub) metrics.

If you are interested in a more elaborate of the host os, check out the Prometheus Node explorer for ‘hardware and OS metrics exposed by *NIX kernels’.

You can also put Grafana on top of Prometheus if you want to create stunning dashboards.

Geef een reactie

Vul je gegevens in of klik op een icoon om in te loggen.

WordPress.com logo

Je reageert onder je WordPress.com account. Log uit /  Bijwerken )

Google photo

Je reageert onder je Google account. Log uit /  Bijwerken )

Twitter-afbeelding

Je reageert onder je Twitter account. Log uit /  Bijwerken )

Facebook foto

Je reageert onder je Facebook account. Log uit /  Bijwerken )

Verbinden met %s