Add rolling logging to your Moby containers

Recently we were testing Azure IoT Edge on VMs which were limited in size. All went well, everything worked as expected until the whole solution stopped working.

After some research, we found out the VM disk was full of data.
This came to us as a surprise. What was eating up the hard disk?

After some more research, we checked out the Azure IoT Edge modules logging location: /var/lib/docker/containers/

Here you find folders, one for each container on your system. These are named after a unique GUID which is related to a specific container.

Note: you might need elevated rights to actually access these folders.

You can compare the folder GUID with the container ID in:

docker ps

If you open one of these folders you will find the logging of the related container.

We can, of course, remove the old logging by removing the files and everything seems to be working again. But that only delays the inevitable.

Let’s see if there is a more permanent solution.

After digging even deeper, we ended up using this little Docker gem: “How to set up log rotation post installation”.

Adding a file

The solution is quite simple, add a file named daemon.json in /etc/docker/.

The file can have several options but the most important ones are:

  1. max-size; The maximum size of the log before it is rolled. A positive integer plus a modifier representing the unit of measure (k, m, or g). Defaults to -1 (unlimited).
  2. max-file; The maximum number of log files that can be present. If rolling the logs creates excess files, the oldest file is removed. Only effective when max-size is also set. A positive integer. Defaults to 1.

So the file looks like:

{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "5m",    
    "max-file": "2"    
  }
} 

Note: It is advised to add this file to you IoT Edge after installing the IoT Edge runtime but before actually starting the IoT Edge. Docker says this about the setting: “This process affects only the new containers NOT the old ones. ” A complete restart of your Moby/Docker service could force it to start using these new settings.

Once the modules start to run and the logging is created, you will see the log files being created. In this case, two files with a maximum size of 5Mb will appear.

There are more options which can be set in this file. The ‘labels’, ‘env’ and
‘env-regex’ options are used to Customize log driver output. So you can format the logging lines with a date time eg.

Testing your changes

As said, it is advised to add this file to you IoT Edge after installing the IoT Edge runtime but before actually starting the IoT Edge.

Changing the rolling log settings does not apply to already deployed modules. Only when a module is created (that is: a container folder in /var/lib/docker/containers is created), the rolling log settings are put in place for this module.

Changing the daemon.json settings afterward does not affect this module anymore.

You can check the current settings for an existing module by checking this file:

/var/lib/docker/containers/[container GUID]/hostconfig.json

It will show you the settings active for this module.

Note: changing the settings in this file to update the settings for this module does not work. You have to stop the IoT Edge runtime, remove the image from ‘docker images’ using ‘docker rmi [image ID]’ and start the IoT Edge runtime again.

Conclusion

This simple feature saves us from issues which are hard to fix in production. It is a little step in the long journey in ruggedizing your production environments.

 

Advertenties