How to write "nested" graphs for Munin

Terms Used

dashboard

The default view when opening the Munin web-interface. It shows daily and weekly graphs for each plugin.

year-view

The view when clicking on a traditional graph. It shows daily, weekly, monthly and yearly graphs for the selected plugin.

breakdown

The view when clicking on a nested graph. It shows daily and weekly graphs for each subgraph in the selected plugin.

Essentials

Let's start with a simple, basic plugin. At the very least you need to handle two cases:

1. The plugin is called without arguments, and
2. The plugin is called with the argument config.

Munin runs the plugin with the config argument to figure out what values to expect and how to graph them (lines, areas, stacks, colours, labels, …).

The absolute minimum looks like this:

$ /path/to/plugin config
graph_title Really simple, static graph
graph_vlabel Static value
myvar.label Static value

This will create a graph named Really simple, static graph in the category other (the default). It will expect values for a variable named myvar. You can of course name it however you like. For more configuration options, look up the official Munin documentation.

Next, now that Munin knows what variables to expect, it will get them by calling the plugin without any arguments. It expects a line for each variable with its value. In our case:

$ /path/to/plugin
myvar.value 10

This is all you need for a basic plugin. There is much more to know about plugins. This is out of the scope of this tutorial. See the official documentation for that.

Using the same code for multiple graphs (essentials #2)

Sometimes you can run the same core code for multiple graphs while only changing one variable. A good example is the graph for network interface statistics. Traditionally, this is achieved by creating one core plugin, then symlinking that with other names. This lets you use the basename of the executable to determine the variable. Example:

if_

This plugin on its own does nothing.

if_eth0

Symlink to if_. Accessing the basename and cutting off if_ lets you determine that the stats are supposed to run for eth0.

Now you can run the same code-base for multiple graphs.

Multigraphs

The above method to run multiple graphs has a few downsides:

• The whole plugin has to be executed for each variable. Even if it could technically share some data between executions. This can be wasteful if it's an expensive process.
• For each new created link, the munin-node needs to be restarted. It cannot be dynamically add/remove graphs.
• It potentially create a lot of graphs on the "dashboard". A good culprit for this are the PostgreSQL plugins. This easily creates more than 5 graphs per database, and can be painful to scroll through on a system with many DBs.

Multigraphs tackle these problems one way or another. The latter one is solved by using "nested" graphs which we'll get to shortly.

The Munin documentation claims that multigraphs tend to be complex. While that may be true for some cases, I personally find that they are still very easy to write for basic needs.

To write a multigraph, the only thing you need, is preface each graph with the multigraph statement. Here's an example defining two graphs in one config command:

$ /path/to/plugin config
multigraph myfirstgraph
graph_title Really simple, static graph
graph_vlabel Static value
myvar.label Static value

multigraph mysecondgraph
graph_title Another , static graph
graph_vlabel Static value
myvar.label Static value

This will generate two graphs named myfirstgraph and mysecondgraph. They are both completely independent! The big difference to classical plugins is that they are generated by the same process. And can thus share expensive resources.

Additionally, if the process generates more multigraph sections on subsequent runs, new graphs will dynamically appear. Conversely, if existing multigraph section disappears, the corresponding graph(s) will disappear as well!

This is extremely useful for situations where graphs can appear/disappear during the life-time of the monitored node. A good example is the aforementioned PostgreSQL situation. With the classic plugins, you need to create a new symlink and reload the node each time a new DB is created.

But how about the values? They are generated in the same way as the config:

$ /path/to/plugin
multigraph myfirstgraph
myvar.value 10

multigraph mysecondgraph
myvar.value 20

Note that the variable names and values are local to the corresponding multigraph section of the config! In this example both myvar variables are completely independent!

What are "nested" graphs?

The default Munin view (the "dashboard") shows you the day and week view of each graph.

When clicking on one of those graphs, you will see an overview of the selected plugin including the month and year. The "year-view".

A "nested" graph adds one additional "breakdown" step between the default and "year-view" to this.

So instead of going:

dashboard → year-view

You will go:

dashboard → breakdown → year view

Writing Nested Graphs

Writing nested graphs is extremely easy, and build on Multigraphs and simply uses names separated by dots. For example, if you have one "dashboard" graph named summaryscore and two "breakdown" charts called diskscore and cpuscore, the names shown in the multigraph config are:

• summaryscore
• summaryscore.diskscore
• summaryscore.cpuscore

The config would look like this:

$ /path/to/plugin config
multigraph summaryscore
graph_title Summery Health Score
graph_vlabel Score (higher = better)
health_summary.label Health Summary

multigraph summaryscore.diskscore
graph_title Disk score
graph_vlabel Disk score (higher = better)
temperature_score.label Temperature
iowait_score.label IO Wait
mtbf_score.label MTBF
mtbf_score.info Days until MTBF

multigraph summaryscore.cpuscore
graph_title CPU score
graph_vlabel CPU score (higher = better)
temperature_score.label Temperature
age_score.label Age
age.info Age of the CPU

The corresponding values would look like this:

$ /path/to/plugin
multigraph summaryscore
health_summary.value 98

multigraph summaryscore.diskscore
temperature_score.label 99
iowait_score.label 99
mtbf_score.label 65

multigraph summaryscore.cpuscore
temperature_score.label 87
age_score.label 99

This would create only one graph ("Summary Health Score") on the dashboard. And once you would click on it you would see the week-view of the two sub-graphs "Disk Score" and "CPU Score".

How the values are determined is entirely up to you.

Now, the above example is extremely contrived, and I have no real-world example. But it should get the point across.

Hope this helped and made writing Munin plugins easier.

Happy coding!