Michel Albert

Tidying Up My Python Libraries

2026-10-29T18:00:00+01:00

Last week I spent some time on my older Python libraries. t61codec, x690, gouge and config_resolver all got a new release, and their docs have moved to GitHub Pages.

t61codec 3.0.0

t61codec is a Python codec for T.61 strings. You still run into those in ASN.1 data, where they are called TeletexString. Version 3.0.0 follows the 1988 version of the ITU-T spec to the letter. Accented letters decode to precomposed characters, invalid input raises an error instead of being guessed, and the incremental decoder copes with sequences that are split across chunks.

The expected test data was written from the spec, not from the code. I also compared the output with glibc's iconv over 1696 inputs. All 168 differences are intended and documented.

This is the last feature release. From now on it only gets fixes.

x690 2.0.0

x690 is the BER encoder and decoder behind puresnmp. 2.0.0 is a breaking release:

It uses t61codec 3, so T.61 decoding is strict now.
Context tags are registered per type instead of globally.
More types decode to proper Python values (str, datetime, float).
A decoded value can let go of the input it came from.

puresnmp and NXP's spsdk both pin x690 <2, so they keep working as before until they move to it.

It also got a lot faster. Reading items of a decoded sequence by index was quadratic: 200 items took 40,200 header reads. Now it's at most 600. Decoding a file of 80 MB (on a Ryzen 5 5600X):

Workload	Before	After
Full document	14.4 s / 232 MB	10.0 s / 8 MB
Last record only	2.2 s / 233 MB	0.71 s / 8 MB
10,000 records by index	167 s	10.5 s

For whole documents, x690 is now the fastest of the five libraries I benchmarked. pyasn1 needs 146 s and 9.5 GB for the same file. For streaming, asn1crypto is still faster.

config_resolver 5.2.0

5.2.0 is the last release of config_resolver. With the alternatives around today, I don't see much value in keeping it.

It now emits a deprecation warning, and the docs have a migration page with tested replacements: pydantic-settings for services, ConfigParser for existing INI files, and platformdirs with tomllib for CLI tools. The repository is archived.

gouge 2.3.0

gouge fixes coloured tracebacks (some viewers lost the colour after the first line) and adds JSON output and an option to show extra record attributes, like a trace ID. The default output is byte-identical to 2.2.5, so existing setups don't change.

Docs and Python Support

The docs are now on GitHub Pages:

All four are tested against every supported CPython version, and releases go out through PyPI's Trusted Publishing.

Note

puresnmp still pins x690 <2, so it doesn't use the new x690 yet.

Nine Years of Quiet Building

2026-03-09T12:00:00+01:00

The blog went quiet in 2017. The keyboard didn't.

Here's what happened in the intervening nine years — a quick tour through the projects that got built, shipped, and occasionally finished.

The SNMP Ecosystem

The most sustained body of work, and the one I'm most proud of.

puresnmp is a pure Python SNMPv2 library — no C extensions, no compiled dependencies, just Python. The goal was simple: make SNMP accessible without fighting a build system first. It quietly grew from a weekend experiment into something with actual users, reaching v2.0.1 in July 2024.

Along the way it spawned two companion libraries:

x690 — X.690 / BER encoding, extracted into its own package (v1.0.0 in 2022) so other tools could use it independently.
puresnmp-crypto — DES and AES encryption support as an optional add-on, keeping the core dependency-free.

The biggest design lesson: carving out a focused, composable library from a monolith is worth the effort. x690 being separate means it can evolve independently. More things should work like this.

Infrastructure & Ops Tooling

A few small tools that scratched real itches.

pgflux ships PostgreSQL monitoring metrics to InfluxDB. It's one of those tools that does one thing and stays out of the way — v1.0.0 shipped in 2021 and it hasn't needed much attention since.

weathermonitor captures sensor data from Zigbee/Phoscon devices and stores it for later. Dockerised in 2023, which made it actually usable on a home server without ceremony.

Three smaller utilities that get regular use:

gouge — a small collection of Python logging configurations, because the stdlib defaults are ugly.
clproc — a changelog processor.
strec — a stream coloriser for stdout/stderr output.

None of these are glamorous. They're the kind of thing you build because the alternative is doing it by hand every time.

Web & Frontend Work

schmackhaft is a self-hosted bookmark manager inspired by the old del.icio.us. Six releases from v0.5 to v1.2.3 shipped across 2022. The short answer to "why build your own bookmark manager in 2022" is: the hosted options are either dead, privacy-hostile, or both.

On the component side:

editable-text — a web component that makes any text element inline-editable.
copy-able — a web component for one-click clipboard copy on any element with a .value.
vue-components — reusable VueJS components, started in 2025.

Small, focused, no framework dependencies. The kind of thing that should exist as a package but rarely does.

Hardware & IoT

In 2025 the work moved off the screen entirely.

noise-goblin is a physical button — ESP32-based, WiFi-connected — that fetches a random sound from an API and plays it. Originally on a basic ESP32-DevC, it grew an OLED memory display and eventually moved to the XIAO ESP32S3 module. The commit history from September 2025 tells the story: memory footprint battles, event loop trade-offs, buffer tuning. Hardware is humbling in ways software rarely is.

spriglet is a more ambitious project: a collaborative tabletop device ecosystem for games and activities that need shared counting or tracking. Still early — the design document landed in September 2025 — but it's the most interesting hardware direction to date.

The honest reflection on moving into hardware: everything you know about debugging becomes unreliable. You can't print() your way out of a stack overflow when there's no stack trace. It's frustrating and completely absorbing.

Developer Workflow

Two recent additions to the toolbox:

taskfiles — a collection of reusable Taskfile tasks for common project operations. Started in January 2026 as a way to stop copying the same boilerplate across repos.

timerz — a chess clock implementation, first commit February 2026. Simple scope, well-contained problem, useful result.

What's Next

A few things on the horizon:

JS modernisation — closure-easing, galeshapley-js, and others were written for Google Closure Library circa 2012. Time to bring them to ES6 modules and npm. Issue filed.
npm publishing — several of the web components are useful enough to publish properly.
Profile cleanup — archiving old forks and playgrounds that have been cluttering the profile.

Nine years is a long time to go without a post. Hopefully the next one won't take as long.

Colorize Flask Logging

2017-05-29T10:49:02+02:00

Here's a simple drop-in colorizer for Werkzeug (and thus also Flask) logs. It won't make much sense in production, but makes development a bit more cheerful.

Adding gouge.colourcli to the mix, makes it even more colourful:

To use it, simply put it in a utility module somewhere, import it and call colorize_wekzeug.

You will need blessings. If that's not available, the function will not do any colorizing.

The code is hosted as a gist on GitHub:

import logging import re P_REQUEST_LOG = re.compile(r'^(.*?) - - \[(.*?)\] "(.*?)" (\d+) (\d+|-)$') LOG = logging.getLogger('werkzeug') def colorize_werkzeug(): """ Fetches the werkzeug logger and adds a color filter. This is based on "blessings". If it's not available, this is a no-op! """ try: from blessings import Terminal except ImportError: return term = Terminal() method_colors = { 'GET': term.bold_green, 'POST': term.bold_yellow, 'PUT': term.bold_blue, 'DELETE': term.bold_red, } class WerkzeugColorFilter: def filter(self, record): match = P_REQUEST_LOG.match(record.msg) if match: try: ip, date, request_line, status_code, size = match.groups() method = request_line.split(' ')[0] # key 0 always exists fmt = method_colors.get(method.upper(), term.white) request_line = fmt(request_line) ip = term.blue(ip) date = term.yellow(date) try: status_code_value = int(status_code) if status_code_value >= 500: status_code = term.yellow_on_red(status_code) elif status_code_value >= 400: status_code = term.red(status_code) elif status_code_value >= 300: status_code = term.black_on_yellow(status_code) elif status_code_value >= 200: status_code = term.green(status_code) else: status_code = term.bold_white(status_code) except ValueError: pass record.msg = '%s - - [%s] "%s" %s %s' % ( ip, date, request_line, status_code, size ) except ValueError: pass return record logging.getLogger('werkzeug').addFilter(WerkzeugColorFilter())

Testing the Untestable

2017-04-13T07:52:20+02:00

Abstract

Sometimes you come across a code-base which has no unit-tests. This makes contributing to the project risky. If an application has not been written with unit-tests in mind from the beginning, it is often very difficult to add them later on. This article takes one such project and uses the fantastic unittest.mock Python module to add tests without modifying the code.

Preface

From personal experience with old code-base maintainance, projects become hard to unit-test for four main reasons:

Violations of Demeter's Law.
Fuzzy borders between the main application code and external I/O (like stdout, databases, network services, ...).
Hidden global state.
Function-call side-effects / non-pure functions.

There are other reasons, but these are by far the ones I've ran across the most.

Note

The following article mentions the open-source project munin-influxdb. It is only taken as illustrative example and I do not intend to paint this in a bad light.

As a starting point, we'll look at commit 8b6b4efca of munin-influxdb.

THe project confronts us with several challenges:

Challenge 1 - No Clear Entry-Point

The project contains a shell-script and separate modules in the bin subfolder. Additionally, the setup.py module does not define any entry_points

Issues

As nothing is packaged probably it becomes non-trivial to install. Neither in an isolated virtual environment, nor on the system.
Testing the main entry-point becomes more difficult as it is a bash script.

Solution

Move the scripts from the bin folder into the project itself, and remove the muninflux shell script. Use the argparse module to provide a well-known "UI" to the end user. This has been achieved between commit 8b6b4efca and acea4515f.

Challenge 2 - Global State

In order to add unit-tests it is important that one can look at a piece (a "unit") of code and can say what it's doing without looking through other parts of the code. The project makes this difficult in it's current state. It makes use of a Settings class containing key/value pairs which are populated at run-time by various methods, and are not known in advance. If there's a function taking a Settings instance as argument it's not trivial to "guess" the structure of that instance to be able to run the function.

Getting Flask-Social working

2016-03-04T21:17:42+01:00

A couple of weeks ago I decided to add social login buttons to a small web-page. I am usually wary of "frameworks" and "plugins", and usually only use them sparingly. I do use Flask for web-development though.

For a change I decided to give flask-social a chance. Turns out that one is based on flask-security so I had to ditch flask-login and that's where the fun started.

Flask Security

First, flask-security enforces that user and group DB table have an id column. Historically, the PK in my DB is the email address though. Even "simulating" an id column using a property did not work. Took me a while to figure out the problem as all errors are silent.

Eventually gave up and added an auto-incrementing id for both tables anyway. That made it work. So yay...

My (current) solution

Note

This document should be read as a companion to the current Flask-Social docs in case you want to get it to work on Python 3. The official docs are here:

http://pythonhosted.org/Flask-Social/#getting-started

After much fiddling around, I came up with the following solution. It still requires a lot of cleanup, but at least, this is a working solution.

Assuming you have flask-security properly configured, these are the steps to follow:

For Python 3, install flask-social from my github branch (You may want to check first if the pypi version has since been updated):
```
pip install https://github.com/exhuma/flask-social/tarball/python3-support
```
Again for Facebook login with Python 3, install the Facebook API from github (and check pypi first of course):
```
pip install http://github.com/pythonforfacebook/facebook-sdk/tarball/master
```

... the same goes for python-twitter:

pip install git+https://github.com/bear/python-twitter

The google provider seems to be fine.

This should get you set up for Python 3. On to the next step...

Application Setup

Configuration

Something not well documented:

In case you need to pass on different OAuth scopes (f.ex.: getting e-mail addresses from Google accounts), you need to set the 'request_token_params key for your social provider:

app.config['SOCIAL_GOOGLE'] = {
    'consumer_key': '...',
    'consumer_secret': '...',
    'request_token_params': {
        'scope': ('https://www.googleapis.com/auth/userinfo.profile '
                  'https://www.googleapis.com/auth/plus.me '
                  'https://www.googleapis.com/auth/userinfo.email')

    }
}

Data Model

As mentioned earlier, flask-social uses **kwargs in places. This causes problems with the rank argument. To fix this without modifying the flask-social code, you can just override the __init__ method of your Connection class to accept **kwargs as well. You then of course need to set the values as needed. You also need to add a relationship to your User class:

class Connection(DB.Model):

    ...

    user = relationship('User')

    def __init__(self, *args, **kwargs):
        self.user_id = kwargs['user_id']
        self.provider_id = kwargs['provider_id']
        self.provider_user_id = kwargs['provider_user_id']
        self.access_token = kwargs['access_token']
        self.secret = kwargs['secret']
        self.display_name = kwargs['display_name']
        self.profile_url = kwargs['profile_url']
        self.image_url = kwargs['image_url']
        self.rank = kwargs.get('rank')

Passing connections to your template

This is tivial. Some connections may be None. As such, you need to check those before calling get_connection on them. Additionally, you need to allow the DELETE method so you can remove connections. This seems to be caused by a redirect which is used internally in flask-social. This redirect happens after the connection has been deleted. So you will land on this endpoint, still using the DELETE method. I simply accept it here, and don't treat it in a special manner. This is not fully HTTP compliant, but at least it works:

@app.route('/profile', methods=['GET', 'DELETE'])
@login_required
def profile():
    return render_template(
        'profile.html',
        content='Profile Page',
        twitter_conn=social.twitter.get_connection() if social.twitter else None,
        facebook_conn=social.facebook.get_connection() if social.facebook else None,
        google_conn=social.google.get_connection() if social.google else None)

Talking about disconnecting. The flask-social docs use a FORM element with action="DELETE". This did not work for me and had to replace it with an XhrIO call (even though this uses closure, the gist should be understandable):

var disconnectButtons = goog.dom.getElementsByClass('social-disconnect-button');
goog.array.forEach(disconnectButtons, function(element) {
  goog.events.listen(element, goog.events.EventType.CLICK, function(evt) {
    var span = goog.dom.getAncestorByTagNameAndClass(evt.target, 'span');
    var providerId = span.getAttribute('data-provider-id');
    var userId = span.getAttribute('data-user-id');
    var connectionURL = '/connect/' + providerId + '/' + userId;
    goog.net.XhrIo.send(connectionURL, function(evt) {
      var xhr = evt.target;
      if (xhr.isSuccess()) {
        window.location.reload(true);
      } else {
        var alertDialog = new goog.ui.Dialog();
        alertDialog.setContent('Verbindungstrennung fehlgeschlagen.');
        alertDialog.setTitle('Fehler');
        alertDialog.setButtonSet(goog.ui.Dialog.ButtonSet.OK);
        alertDialog.setVisible(true);
      }
    }, 'DELETE');
  });
});

In order to get a consistend look & feel, I used the same way for the login button and to create connections. This may seem hacky, but it's consisten in both source-code and UI. I had to use form submits in this case to get around the same-origin policy:

var loginButtons = goog.dom.getElementsByClass('login-button');
goog.array.forEach(loginButtons, function(element) {
  goog.events.listen(element, goog.events.EventType.CLICK, function(evt) {
    var form = goog.dom.getAncestorByTagNameAndClass(evt.target, 'form');
    form.submit();
  });
});

var connectButtons = goog.dom.getElementsByClass('social-connect-button');
goog.array.forEach(connectButtons, function(element) {
  goog.events.listen(element, goog.events.EventType.CLICK, function(evt) {
    var form = goog.dom.getAncestorByTagNameAndClass(evt.target, 'form');
    form.submit();
  });
});

Remapping Keys in Windows

2016-01-19T09:10:43+01:00

After having been using Linux for over 20 years now, one of the few things I really miss in Windows, is the "Compose Key". It does exist, but until today, I was never happy with the mapping. For ages I have this mapped on the Caps Lock key. But under Windows this sometimes leads to a "stuck" caps lock. I've tried different Compose Key applications on Windows, and they all had the same problem.

I was looking for something like .Xmodmap on Windows, and today I found it. You can rebind scan codes using the Windows Registry.

An example entry might be the following:

Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Keyboard Layout]
"Scancode Map"=hex:00,00,00,00,00,00,00,00,02,00,00,00,5c,e0,3a,00,00,00,00,00

This maps scan code 3a 00 (usually sent by the caps-lock key) to 5c e0 (usually sent by the right Windows button). So now pressing caps-lock is interpreted as the right windows key by the OS. This one can be safely bound as compose key as it's not a "sticky" key like caps-lock.

Refernces:

Alternative PostgreSQL plugin for munin — Part 4

2015-11-27T13:41:03+01:00

No longer "alpha"

After running the graphs for some time, I am happy with the result, and they have replaced most of the old Postgres graphs at work.

Changes since v1.0.0a2

Added "transaction age" to query ages.

This value was simply missing.

Disabled value scaling

This affects how values are displayed and causes less fractional values.

Added POD-Style documentation

Munin graphs require POD documentation. It is readable with munin-doc.

Graph Labels/Info

Improved the labels and info fields of graphs.

autoconf

The graph is not auto-detectable with autoconf.

Moved to postgresql category

The former category was used for testing to keep the graphs better apart. Now that they are production-ready, this is no longer needed.

Transaction Stats

Added graph of committed/rolled-back transactions.

Fixed counters for query-ages

The value sometimes came out as NaN. This is now properly reported as 0.

Added a breakdown of DB sizes

Opening the DB-Sizes graphs will now show a more detailed view, containing separate sizes for "main data", "indexes", "free-space-map", "visibility map" and "TOAST tables".

The values here usually sum up to less than the raw DB-size value. The DB-size value counts the on-disk size of the database folder. This may contain other/temporary files. In extreme cases (catastrophic server failure), the temporary files may not get cleaned up. This should become visible in this graph.

Row Access is now stacked

This should make the graph more readable.

Added default warning/crtitical values for QueryAges

The default warning is 1 hour, the critical limit is set to 12 hours.

Removed IDLE connections from query ages

This blew up the graph in case an application kept an open connection without active work.

Alternative PostgreSQL plugin for munin — Part 3

2015-11-11T08:08:21+01:00

Comparison with the Official Plugins

The most obvious difference you see at first is that it's only one single file instead of multiple. At this time I'm still pondering to break it apart, but I have a strong tendency to leave it as it is.

If you want to only enable some of the graphs, this can be done using the PG_MULTIGRAPHS environment variable. See Part 2 for details on this.

The main aim of the work was to aggregate those plugins which generated too many graphs. While they show aggregated values on the default munin overview, these new graphs will show a breakdown page once clicking on them.

Quick overview:

Official Plugin Name for new plugin

postgres_bgwriter Not ported (good enough).

postgres_checkpoints Not ported (good enough).

postgres_xlog Not ported (good enough).

postgres_transactions_ Not ported (no use-case).

postgres_cache_
Replaced with:

indexio

sequenceio

tableio

postgres_connections_ connections

postgres_connections_db Merged with connections.

postgres_users Merged with connections

postgres_locks_ locks

postgres_querylength_ query_ages

postgres_scans_ scan_types

postgres_size_ sizes

postgres_tuples_ row_access

Official Plugin	Name for new plugin
`postgres_bgwriter`	Not ported (good enough).
`postgres_checkpoints`	Not ported (good enough).
`postgres_xlog`	Not ported (good enough).
`postgres_transactions_`	Not ported (no use-case).
`postgres_cache_`	Replaced with: `indexio` `sequenceio` `tableio`
`postgres_connections_`	`connections`
`postgres_connections_db`	Merged with `connections`.
`postgres_users`	Merged with `connections`
`postgres_locks_`	`locks`
`postgres_querylength_`	`query_ages`
`postgres_scans_`	`scan_types`
`postgres_size_`	`sizes`
`postgres_tuples_`	`row_access`

Design Decisions

General

Most graph aggregations use stacked areas to display values. The idea behind this is that you are usually interested in the total value. Using stacks gives you a much better view of this. Overlapping lines are more useful to see correlations between values.

Most aggregations show the sum of all the agregates. In the case of the query ages, it shows the maximum of all the aggregates.

Replacing `postgres_cache_` with 3 new I/O Plugins

While investigating, I discovered the following stats tables:

They give a more fine-grained view on buffer I/O than pg_stat_database and decided to split that one up.

Connections to the DB

This currently shows connections by user. It does not show connections per DB (yet?). I currently have no pressing plans to include connections per DB. So I will leave it as-is for the time being.

Locks

The official plugin combines both locks which are granted with locks which aren't (somewhere someone's waiting for the release). Personally, I find it more important to watch for waiting locks than for granted locks. For this reason, this plugin now splits both lock types up, showing granted locks on one side of the X-axis, and waiting locks on the other.

This still keeps all the information from the old graphs, but adds an additional element to it.

Query Ages

I decided to use "age" instead of "length" as, in the context of a graph, this makes more sense. Maybe I was a bit too nitpicky here.

The official graph shows both query and transaction start. The transaction start is currently not yet implemented in the new graph, but I will do so once this blog-post is done.

Scan Types

Nothing of importance to say here.

Sizes

Nothing of importance to say here.

Row Access

Named Tuple Access in the official plugins. I renamed it as the statistics table in PG also talks about "row access". So do the docs.

Alternative PostgreSQL plugin for munin — Part 2

2015-11-10T08:00:49+01:00

First Alpha Releases

I have finished porting all "virtual" graphs from the official plugins. The source of this release can be found on github.

This is currently a very rough cut release. Keep in mind:

There is barely any documentation
The plugin does not support autoconfig
It is likely that some major refactoring will go on once the docs are written.
Maybe things will change. Maybe not.

Trying it out

Installation

Installing the plugin is fairly easy:

Clone the github repository somewhere and switch to the alpha tag:

git clone https://github.com/exhuma/munin-plugins.git /opt/exhuma/munin-plugins
cd /opt/exhuma/munin-plugins
git checkout postgres-multigraph/v1.0.0a2

Install psycopg2:
```
apt-get install python-psycopg2
```

Symlink the plugin into your munin-node:

ln -s /opt/exhuma/munin-plugins/pg_multigraph /etc/munin/plugins/pg_multigraph

Configuration

The plugin reads the following environment variables:

PG_DBNAME (default='template1')

The database name to connect to.

PG_USER (default='postgres')

The username used to connect.

PG_PASSWORD (default='')

The password used to connect. If empty, perform a passwordless login.

PG_HOST (default='')

The DB host. If empty, connect using the unix domain socket.

PG_PORT (default=0)

The port which is used to connect. If 0, connect using the unix domain socket.

PG_MULTIGRAPHS (default='__all__')

A comma-separated list of which graphs to create. At the time of this writing the following names are accepted:

__all__
connections
locks
query_ages
row_access
scantypes
sizes
indexio
sequenceio
tableio

The special name __all__ causes all graphs to be generated.

What's different?

Compared to the plugins bundled with munin, there are a couple of differences in the generated graphs:

Locks show both "granted" and "waiting" locks. The official plugin did not differentiate between both.
postgres_transactions_ has not been ported. I have not use-case for that one.
index scan and sequential scan is removed from tuple access. It is now also related to row access. Both scan fetches are more related to indices and pollute that graph.
Buffer Cache has been removed and replaced by sepearate I/O plugins for Tables, Sequences and Indices.

Screenshots?

I have currently no screenshots at hand which are really representative. All I have is some "work in progress" stuff from my laptop. They don't represent the graphs very well.

Alternative PostgreSQL plugin for munin

2015-11-08T10:22:49+01:00

Abstract

I've been using the PostgreSQL graphs which come bundled with munin for quite some time. But recently I've come annoyed with the amount of graphs those plugins generate. I've decided to fix this in the existing plugins, but ran into a design dilemma. It prompted me to re-write the plugins. During this I discovered a couple of interesting facts about both PostgreSQL stats, and the existing graphs. This blog-post (which is the first in a series) will give some context to the ongoing development and put my thoughs "on paper".

Stipulations

First, the new plugins should work like the snmp__if_multi plugin. Meaning, the main dashboard should show an aggregated graph. Upon clicking on it it should show a more readable breakdown before going to the yearly overview. Second, the graphs should avoid an excessive number of overlapping values on one graph, utilising negative axis where possible.

Why modify the existing graphs?

The current plugins have a couple of "pain-points":

New symlinks have to be created each time a database is created/deleted. If you forget it, the DB will not be graphed.
There are (currently) 17 official postgres plugins. Of these 17 plugins, 11 are "virtual" plugins which have to be symlinked for each database. This means that a system multiple databases will generate 6 + 11 * <num databases> graphs. That is a lot of graphs.
Some graphs (f. ex. "Connections by User") are a bit hard to read as they can generate a lot of overlapping lines.

Why a re-write? (The "design dilemma")

My first reflex was to grab the sources of the existing plugins. I ran into one major issue:

The plugins rely on a perl module bundled with munin. This module is non-trivial, forces the plugin-author to write SQL queries which return a schema as required by the plugin. This makes some queries more complex than necessary. It also makes contributions more difficult as the module tightly couples the plugin with munin itself.

While the library generalises many of the aspects of plugin authoring, it overgeneralises (in my own , personal opinion) which renders the plugins a lot harder to read and modify. Some graphs may warrant top be displayed differently than others. By using the same library for all, this customisation gets lost. And supporting the customisation, is counter-productive to the generalisation.

A re-write allows me to decouple the plugins from the library and make them standalone executables. This makes them easier to debug and more approachable by potential contributors.

Design Decisions

Code Structure

I like munin plugins to be nicely self-contained executables. However, in this case, the plugins have some common properties. In its minimal form, those are:

The DB connection parameters
The graph category

Especially considering the connection parameters, one could argue that this part of the code should be shared amongst the plugins. But I don't want to extract a library. There's no good place to put it if the plugins should remain self-contained.

There are two solutions:

One huge script generating the graphs using the multigraph statement.
Don't share the code.

In a first iteration, I have opted to writing a multigraph plugin. Currently, the amount of shared code is fairly small. I am considering ripping them apart and not share any code.

I did however not yet fully walk through the code and examine shareable parts. I will investigate this further in the future.

Graph Visualisation

Most graphs can be designed such that they have an intermediary "breakdown" graph. A good example is the databse size. It can be written such that it shows all sizes on one stacked graph on the munin overwiew. Then, when clicking on it, you will be sent to a page listing daily/weekly graphs for each database. Clicking on those will send you to the dayly/weekly/monthly/yearly graph of that DB.

Using that method will keep the main munin overview much cleaner than the original graphs while not losing any information.

Another way to improve visualisation is using the negative axis. A good example here are locks and disk/buffer I/O. For locks, granted locks can be drawn on the positive axis, while the waiting locks of the same type can be shown on the negative axis. Fo I/O graphs, buffer hits can be shown as positive values, and disk reads of the same time can be drawn as negative values.

Using both positive and negative values cuts down the number of colors used on a graph by a factor of 2. And thus make it easier to read again.

Current State

I am currently writing a minimal viable product to replace the existing plugins. As of today, I have working graphs for:

Connections (breakdown by user)
Sizes
Locks
QueryAges
TableDiskIO (former buffer-cache)
IndexDiskIO (former buffer-cache)
SequencesDiskIO (former buffer-cache)

The current source can be found on github

How to write "nested" graphs for Munin

2015-10-11T10:58:21+02:00

Preface

To make this tutorial useful to an as wide audience as possible, I will begin to explain the basics of Munin plugins first. If you are only interested in nested graphs, feel free to directly jump to either the Multigraphs or Nested Graphs section.

Terms Used

dashboard: The default view when opening the Munin web-interface. It shows daily and weekly graphs for each plugin.
year-view: The view when clicking on a traditional graph. It shows daily, weekly, monthly and yearly graphs for the selected plugin.
breakdown: The view when clicking on a nested graph. It shows daily and weekly graphs for each subgraph in the selected plugin.

Basics

I will try to show a comprehensive minimal example starting from a simple plugin, but if you have not yet read the tutorial on how to write plugins [1], you should.

I will not show any plugin source-code. But rather focus on the output that Munin needs to see on stdout!

[1]	At the time of this writing, the Munin documentation is undergoing a transition, but the plugin authoring document is still at it's old location!

Walkthrough

Essentials

Note

You can skip this if you already know how to write simple plugins.

Let's start with a simple, basic plugin. At the very least you need to handle two cases:

The plugin is called without arguments, and
The plugin is called with the argument config.

Munin runs the plugin with the config argument to figure out what values to expect and how to graph them (lines, areas, stacks, colours, labels, …).

The absolute minimum looks like this:

$ /path/to/plugin config
graph_title Really simple, static graph
graph_vlabel Static value
myvar.label Static value

This will create a graph named Really simple, static graph in the category other (the default). It will expect values for a variable named myvar. You can of course name it however you like. For more configuration options, look up the official Munin documentation.

Next, now that Munin knows what variables to expect, it will get them by calling the plugin without any arguments. It expects a line for each variable with its value. In our case:

$ /path/to/plugin
myvar.value 10

This is all you need for a basic plugin. There is much more to know about plugins. This is out of the scope of this tutorial. See the official documentation for that.

Using the same code for multiple graphs (essentials #2)

Sometimes you can run the same core code for multiple graphs while only changing one variable. A good example is the graph for network interface statistics. Traditionally, this is achieved by creating one core plugin, then symlinking that with other names. This lets you use the basename of the executable to determine the variable. Example:

if_: This plugin on its own does nothing.
if_eth0: Symlink to if_. Accessing the basename and cutting off if_ lets you determine that the stats are supposed to run for eth0.

Now you can run the same code-base for multiple graphs.

Multigraphs

The above method to run multiple graphs has a few downsides:

The whole plugin has to be executed for each variable. Even if it could technically share some data between executions. This can be wasteful if it's an expensive process.
For each new created link, the munin-node needs to be restarted. It cannot be dynamically add/remove graphs.
It potentially create a lot of graphs on the "dashboard". A good culprit for this are the PostgreSQL plugins. This easily creates more than 5 graphs per database, and can be painful to scroll through on a system with many DBs.

Multigraphs tackle these problems one way or another. The latter one is solved by using "nested" graphs which we'll get to shortly.

The Munin documentation claims that multigraphs tend to be complex. While that may be true for some cases, I personally find that they are still very easy to write for basic needs.

To write a multigraph, the only thing you need, is preface each graph with the multigraph statement. Here's an example defining two graphs in one config command:

$ /path/to/plugin config
multigraph myfirstgraph
graph_title Really simple, static graph
graph_vlabel Static value
myvar.label Static value

multigraph mysecondgraph
graph_title Another , static graph
graph_vlabel Static value
myvar.label Static value

This will generate two graphs named myfirstgraph and mysecondgraph. They are both completely independent! The big difference to classical plugins is that they are generated by the same process. And can thus share expensive resources.

Additionally, if the process generates more multigraph sections on subsequent runs, new graphs will dynamically appear. Conversely, if existing multigraph section disappears, the corresponding graph(s) will disappear as well!

Warning

At the time of this writing, RRD files of graphs are not removed when a multigraph section disappears! So be careful not to fill up the disk with "temporary" graphs!

This is extremely useful for situations where graphs can appear/disappear during the life-time of the monitored node. A good example is the aforementioned PostgreSQL situation. With the classic plugins, you need to create a new symlink and reload the node each time a new DB is created.

But how about the values? They are generated in the same way as the config:

$ /path/to/plugin
multigraph myfirstgraph
myvar.value 10

multigraph mysecondgraph
myvar.value 20

Note that the variable names and values are local to the corresponding multigraph section of the config! In this example both myvar variables are completely independent!

Nested Graphs

What are "nested" graphs?

The default Munin view (the "dashboard") shows you the day and week view of each graph.

When clicking on one of those graphs, you will see an overview of the selected plugin including the month and year. The "year-view".

A "nested" graph adds one additional "breakdown" step between the default and "year-view" to this.

So instead of going:

dashboard → year-view

You will go:

dashboard → breakdown → year view

Note

The values you report for the dashboard do not necessarily have to be related to the breakdown! While this might not seem to make sense at first sight, there may be some intersting applications for this. Just keep in mind that you can report completely different values with a different config on the "dashboard" as on the "breakdown"! An interesting exmple might be a dashboard graph reporting an arbitrry "I/O health" score based on inode usage, disk IOs and IOstat. Then on the breakdown, report each individual graph.

Writing Nested Graphs

Writing nested graphs is extremely easy, and build on Multigraphs and simply uses names separated by dots. For example, if you have one "dashboard" graph named summaryscore and two "breakdown" charts called diskscore and cpuscore, the names shown in the multigraph config are:

summaryscore
summaryscore.diskscore
summaryscore.cpuscore

The config would look like this:

$ /path/to/plugin config
multigraph summaryscore
graph_title Summery Health Score
graph_vlabel Score (higher = better)
health_summary.label Health Summary

multigraph summaryscore.diskscore
graph_title Disk score
graph_vlabel Disk score (higher = better)
temperature_score.label Temperature
iowait_score.label IO Wait
mtbf_score.label MTBF
mtbf_score.info Days until MTBF

multigraph summaryscore.cpuscore
graph_title CPU score
graph_vlabel CPU score (higher = better)
temperature_score.label Temperature
age_score.label Age
age.info Age of the CPU

The corresponding values would look like this:

$ /path/to/plugin
multigraph summaryscore
health_summary.value 98

multigraph summaryscore.diskscore
temperature_score.label 99
iowait_score.label 99
mtbf_score.label 65

multigraph summaryscore.cpuscore
temperature_score.label 87
age_score.label 99

This would create only one graph ("Summary Health Score") on the dashboard. And once you would click on it you would see the week-view of the two sub-graphs "Disk Score" and "CPU Score".

How the values are determined is entirely up to you.

Now, the above example is extremely contrived, and I have no real-world example. But it should get the point across.

Hope this helped and made writing Munin plugins easier.

Happy coding!

Controlling Raspberry-Pi LEDs the Hard Way

2015-07-16T20:39:07+02:00

Yesterday I finally got my hands on a multimeter and could read out resistor values. As a colourblind person, I was unaware that this would bite me.

I decided to go ahead and play around with a simple schematic. The goal: Make the LED light up as long as the button is pressed.

I decided to do all with a multiprocessed Python application. You know... for funsies. But also because I am more of a software than a hardware guy.

To access the GPIOs, I'm using the excellent pigpio library.

The Challenge

Write a Python application which uses multiple processes to react to external stimuli.

Starting Point

Button connected to GPIO 7 (CE1 on the Ext. Board).
LED connected to GPIO 8 (CE0 on the Ext. Board).

Plan of Action

Write three sub-processes (in addition to the main process):
- One process, the "Button Monitor", which reacts to button presses by converting them into "Button State Change Events".
- One process, the "LED Controller", which reacts to "LED State Change" events by converting them to GPIO signals.
- One "Event Dispatch Process". This one combines all message queues and knows which events should be sent to which process.

Material Used

SunFounder Raspberry-Pi-GPIO Extension Board V2.2
1 kΏ Resistor
1 LED
1 Button
3 Jumper Wires
1 Bread Board

Wiring

I don't have anything to draw schematics yet... so a crappy potato photo will have to do:

Code

Now this is the amusing part. And too big to post here. The resulting code can be found in my GitHub repository.

Conclusion

It works. And the code of the individual processes is fairly simple.

But...

I have two processes wach having a connection to the pigpiod daemon. It irks me. I'd prefer having only one connection. But it's good enough for now.
It's not possible to attach event-handlers. The handling is completely hard-coded in the dispatcher process. Again, good enough for a quick-and-dirty script.

Executing snmpwalk on multiple devices simultaneously

2015-02-24T16:22:45+01:00

I tend to run the same SNMP probes on multiple devices quite often. The devices are always the same, the OIDs change. To simplify this I created a small Python script running the commands in a tmux session.

However, there seems to be a small bug/annoyance in tmux which makes it nearly impossible to run a command with special characters in it's parameters directly via tmux. Instead, "injecting" the keystrokes into the running session works fine.

I started off writing this in bash, but due to the issue mentioned above, I ended up writing it in Pythong, making the code a lot cleaner. Albeit unnecessary:

#!/usr/bin/python3
'''
Run "snmpwalk" on multiple devices simultaneously in separate tmux tabs.
'''

from collections import namedtuple
from os import environ
from subprocess import check_call as run
import sys

environ['TERM'] = 'xterm-256color'  # Hint for tmux to enable 256 colours.

Connection = namedtuple('Connection', 'ip, community, name')


TMUX_NAME = 'snmpt-probe'
TARGET_DEVICES = [
    Connection('1.2.3.4', 'helloworld', 'Device 1'),
    Connection('1.2.3.5', 'helloworld', 'Device 2'),
    Connection('1.2.3.6', 'helloworld', 'Device 3'),
    Connection('1.2.3.7', 'helloworld', 'Device 4'),
]


def exec_(ip, oid, community):
    '''
    Run a command using ``tmux send-keys``.

    This is a workaround. Executing the command directly does not work if a
    parameter contains special characters.
    '''
    run(['tmux', 'send', '-t', TMUX_NAME, ' snmpwalk -v2c -c "'])
    run(['tmux', 'send', '-t', TMUX_NAME, community])
    run(['tmux', 'send', '-t', TMUX_NAME, '" %s %s | less' % (ip, oid)])
    run(['tmux', 'send', '-t', TMUX_NAME, 'ENTER'])


def main():

    # Ask the user to enter the OID
    oid = input('OID: ').strip()
    if not oid:
        print('OID cannot be empty!', file=sys.stderr)
        return 1


    # Create a new, empty tmux session
    run(['tmux', 'new', '-s', TMUX_NAME, '-d'])

    # give the first window it's proper name
    run(['tmux', 'rename-window', '-t', '0', TARGET_DEVICES[0].name])

    # create the remaining tabs with appropriate names
    for _, _, name in TARGET_DEVICES[1:]:
        run(['tmux', 'new-window', '-n', name])

    # run the snmpwalk commands
    for ip, community, name in TARGET_DEVICES:
        run(['tmux', 'select-window', '-t', name])
        exec_(ip, oid, community)

    # Attach the tmux session
    run(['tmux', 'attach', '-t', TMUX_NAME])
    return 0


if __name__ == '__main__':
    sys.exit(main())

Maximising xterm and gvim in KDE

2015-02-23T10:52:15+01:00

The Problem

For ages, I was annoyed that neither gvim nor xterm maximised properly in some desktop environment (KDE being one of them). They both have a small "gap" around the outermost screen edges. It is not only visually annoying, but I am also used to "throw" the mouse to the top corner to grab the title-bar. But because this small gap exists, I always end up clicking anything which lies behind that window.

Note the visual gap between the application and the window on it's left. gvim (or xterm) is maximised, and the other window is sitting on a second monitor. The gap is clearly visible.

The reason this happens, is because the applications give the window manager resizing hints. And because both applications are resized based on characters, and not based on pixels, the WM tries it's best to fill as much screen as possible. But only changing the size by the increments given by the application. This depends on font and font-size. It is unlikely that the width and height of the characters can perfectly fill the screen.

Solution

KDE offers you a way to ignore those hints.

The context menu for special application settings.

When the application is open, right-click on the title bar, select "More Actions", and "Special Application Settings". This will open up a dialog for this application".

The "Special Application Settings" dialog.

The "Size & Position" tab contains the option we are looking for: "Obey geometry restrictions".

Override the WM default by ticking the checkbox and selecting "Force" in the drop-down menu and setting the value to "No".

Close-up on the value that you need to set.

End-Result

In this screenshot, you can see that the gap between both windows has disappeared.

After changing these values you can try to maximise the window (or restore, then maximise again). It should now fill the screen up to the last pixel, closuing up the gap.

This fix is applicable to all applications which force size increments on your window. For me, this was both gvim and xterm, but it will surely also fix issues with emacs.

This fix is also not exclusive to KDE. For example "awesome" allows you to do the same via config files, and I am sure other WMs let you override this as well.

Linux LVM and RAID recovery

2014-12-17T15:16:47+01:00

Contents

Creating the Software RAID Array
Creating Logical Volumes
Cleanup
Recovery
- RAID
- LVM
Restarting the array with all devices

This little tutorial will first create a RAID array, and one LVM volume using dummy files. Next it will simulate the case where you would like to recover from just one device, and finally reassemble the complete array again.

Creating the Software RAID Array

Required packages

sudo aptitude install \
    lvm2 \
    mdadm

Creating files for testing

dd if=/dev/zero of=disk1 bs=1M count=30
dd if=/dev/zero of=disk2 bs=1M count=30

Create devices for these files

mdadm only accepts valid block devices for raid members. We use the loopback device for this:

sudo losetup /dev/loop0 disk1
sudo losetup /dev/loop1 disk2

Create the RAID array

sudo mdadm --create /dev/md0 --level=1 --raid-devices=2 /dev/loop0 /dev/loop1

Verify integrity

Running cat /proc/mdstat should show you the following now:

Personalities : [raid1]
md0 : active raid1 loop1[1] loop0[0]
      30656 blocks super 1.2 [2/2] [UU]

unused devices: <none>

The [UU] shows us that both devices are up.

Creating Logical Volumes

As the devices have never been used in LVM, we need to create everything first:

sudo pvcreate /dev/md0

This flags the devices as member in a volume group. Now we can craete a volume group with this raid array as member:

sudo vgcreate foobar /dev/md0

To see that it worked, run the following:

sudo vgs

You should see the following:

VG     #PV #LV #SN Attr   VSize  VFree
foobar   1   0   0 wz--n- 28.00m 28.00m

This shows us that we have one volume group (you can imagine this as a virtual hard-disk) with a bit less than 30MB of free space (we created 30M files in the very first step).

Finally, we can create our logical volumes (virtual partitions). Let's create two 10M partitions:

sudo lvcreate -L10M -n part1 foobar
sudo lvcreate -L10M -n part2 foobar

This will create two volums with the names part1 and part2. They should now be available in /dev/foobar or /dev/mapper and can be used like regular disks. To see a quick status, you can run the commands pvs, vgs and lvs for quick reports about physical volumes, volume groups and logical volumes.

Cleanup

Note

This is only necessary if you followed the above steps to simulate real devices using regular files. We need to free up all references before we can remove the loopback devices.

To simulate the physical connection, you can run the following commands to clean up everything from the previous step:

sudo vgchange -a n foobar  # This deactivates the volume group from LVM
sudo mdadm --stop /dev/md0  # This removes the /dev/md0 device and frees up the loopback devices /dev/loop[01]
sudo losetup -d /dev/loop0  # Finally, this removes the loopback devices
sudo losetup -d /dev/loop1

Recovery

Let's assume that you have only one disk, and you need to recover data from this disk. This section shows how to reassemble everything.

As both the software raid and LVM store metadata in the devices (superblocks), the reassembly is much easier than creation.

Note

If you are using the dummy files as created above, we will set only one of the two files as block device. This is equivalent to physically connecting only one device:

sudo losetup /dev/loop0 disk1

RAID

First we scan for RAID devices:

sudo mdadm --assemble --scan

You should see the following:

mdadm: /dev/md/0 has been started with 1 drive (out of 2).

And a quick look at /proc/mdstat shows the following:

Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md0 : active raid1 loop0[0]
      30656 blocks super 1.2 [2/1] [U_]

unused devices: <none>

Seeing [U_] shows us that one device is up, and the other one missing.

LVM

Now that the raid device /dev/md0 is up and running, we can scan for LVM metadata:

sudo vgscan

This should show up with the following message:

Reading all physical volumes.  This may take a while...
Found volume group "foobar" using metadata type lvm2

Now, running the command pvs, vgs and/or lvs should show the previously created devices.

Restarting the array with all devices

After recovery, you can reassemble the array with all devices in the exact same way as in the previous "recovery" step. But in this case, you should see both devices as "up" in /proc/mdstat.

Automatically Sync Files on WIFI Connect

2014-10-06T19:56:49+02:00

I recently discovered wicd (thanks to Gefoo). It is a really nifty network manager. And it supports running scripts on each connection stage for a network. This immediately gave me an idea:

I like taking photos, and sometimes I unload them from my camera to my laptop for immediate housekeeping. I keep them on an in-house NAS for safe-keeping. Being able to run scripts once the laptop connects, lets me sync the files automagically to my NAS. This is awesome, and this article summarises how I've implemented it.

It is actually quite straight-forward. The wicd documentation mentions that you should modify one the following settings (depending on what you need):

afterscript
postdisconnectscript
beforescript
predisconnectscript

However, this did not work for me. After editing the file, the values reverted to their original value (None). I assume that you need to stop the wicd service before editing it. I did not do that, and I assume that the file is written when wicd ends. This would explain this issue. But there is also another way, which I followed fo the rest of this.

As an alternative, you can just drop executable scripts into one of the /etc/wicd/scripts subfolders. For my task, I chose postconnect. I use this script as an "orchestration" script which launches other scripts as needed. For now it only contains one script, and looks like this:

#!/bin/bash
conntype="$1"
essid="$2"
bssid="$3"

if [ "$conntype" == "wireless" ]; then
    if [ "$essid" == "MichTik" ]; then
        /opt/wicd/scripts/filedrop
    fi
fi

This says, that as soon as a wireless connection is established on the SSID "MichTik", the script /opt/wicd/scripts/filedrop should be executed.

That script in turn looks like this:

 #!/bin/bash
 LOGFILE=/var/log/filedropper.log

 # SSH Private Key for fully unattended operation. This key must be
 # passwordless, so the containing directory should have the proper access
 # rights.
 IDENTITY=/opt/wicd/scripts/sshkeys/filedrop

 # Any file created by this script (the logfile) should be secured.
 umask 0077

 echo "$(date +"%F %T") dropping new files..." >> $LOGFILE

 ##
 # drop(src, dest)
 #
 # Moved files from "src" to "dest".
 #
 # WARNING
 # This uses rsync with --remove-source-files, so you should make sure that
 # files are not modified *while* this is running! Read the according
 # section of the rsync man-page for details!
 ##
 function drop() {
     src=$1
     dst=$2
     rsync -a \
         --partial \
         --remove-source-files \
         -e "ssh -l exhuma -i ${IDENTITY}" \
         ${src} \
         ${dst} >> ${LOGFILE}
 }

 drop /home/exhuma/photos \
      192.168.1.1:/mnt/media/incoming/filedrop/

 echo "$(date +"%F %T") DONE." >> $LOGFILE

And that's it!

Note

Security of this could be improved. For example, I am using my own user on the filedrop target. Which means that if my laptop is compromised, and someone would get their hands on this passwordless key, that person could wreak all kinds of havoc. After I finish this article, I will set up a dedicated user on the NAS for this purpose. And you should do so too :P

Better Testdata for HClustering

2014-09-12T22:00:00+02:00

While I first intended to create the matrices manually, it turned out to take too much time if I wanted to reproduce the exact same error as happened in the unit-tests.

So in order to create the matrices used in the earlier blog-posts, I used the script at the end of this article. I took care to double-check the results however. There are times, where there's a choice of more than one cluster to pick. And as the order is not important, this choice becomes dependent on the concrete implementation, and thus non-deterministic in the unit-tests. This caused a discrepancy in the matrices generated by the script below, and the current implementation of the algorithm in library. If that was the case, I validated the results manually, and they checked out.

Unfortunately though, the data in the existing unit-test do not provoke a difference between UCLUS and average linkage (there are no real outliers). Which is unfortunate. I will (for now) not create a new data-set to verify the difference between UCLUS and average. I trust the current implementation enough to say that it will do the right thing...

Finally, here's the script used to create the test-data:

from __future__ import print_function
from difflib import SequenceMatcher


def mean(numbers):
    """
    Returns the arithmetic mean of a numeric list.
    see: http://mail.python.org/pipermail/python-list/2004-December/294990.html
    """
    return float(sum(numbers)) / float(len(numbers))


def median(numbers):
    """
    Return the median of the list of numbers.
    see: http://mail.python.org/pipermail/python-list/2004-December/294990.html
    """

    # Sort the list and take the middle element.
    n = len(numbers)
    copy = sorted(numbers)
    if n & 1:  # There is an odd number of elements
        return copy[n // 2]
    else:
        return (copy[n // 2 - 1] + copy[n // 2]) / 2.0


def textsim(x, y):
    sm = SequenceMatcher(lambda x: x in ". -", x, y)
    return 1 - sm.ratio()


def simplesim(x, y):
    return abs(x-y)

# sim = textsim
sim = simplesim


def single(a, b):
    return min([sim(ex, ey) for ex in a for ey in b])


def complete(a, b):
    return max([sim(ex, ey) for ex in a for ey in b])


def uclus(a, b):
    return median([sim(ex, ey) for ex in a for ey in b])


def average(a, b):
    return mean([sim(ex, ey) for ex in a for ey in b])


linkage = average


data = [['Lorem'],
        ['ipsum'],
        ['dolor'],
        ['sit'],
        ['amet'],
        ['consectetuer'],
        ['adipiscing'],
        ['elit'],
        ['Ut'],
        ['elit'],
        ['Phasellus'],
        ['consequat'],
        ['ultricies'],
        ['mi'],
        ['Sed'],
        ['congue'],
        ['leo'],
        ['at'],
        ['neque'],
        ['Nullam']]


data = [[_] for _ in [791, 956, 676, 124, 564, 84, 24, 365, 594, 940, 398, 971,
                      131, 365, 542, 336, 518, 835, 134, 391]]


def matrix(data):
    for i, row in enumerate(data):
        for j, cell in enumerate(data[:i]):
            minsim = linkage(row, cell)
            yield (minsim, (i, j))


def step(data):
    while len(data) > 1:
        sorted_matrix = sorted(matrix(data), key=lambda x: x[0])
        closest_items = sorted_matrix[0]
        a, b = closest_items[1]
        new_element = data.pop(max(a, b)) + data.pop(min(a, b))
        data.append(new_element)
        return data


def print_matrix2(data, iteration):
    values = []
    for i, row in enumerate(data):
        simdata = [linkage(row, x) for x in data[:i]]
        values.extend(simdata)
    if values:
        smallest = min(values)

    print('.. csv-table:: Matrix #{}'.format(iteration))
    print('    :header-rows: 1')
    print('    :stub-columns: 1')
    print('    :delim: :\n')
    print('    :' + ': '.join([', '.join([str(x) for x in _]) for _ in data]))
    for i, row in enumerate(data):
        simdata = [linkage(row, x) for x in data[:i]]
        strsimdata = ['**{:.3f}**'.format(_)
                      if _ == smallest else '{:.3f}'.format(_)
                      for _ in simdata]
        foo = ','.join([str(_) for _ in row])
        bar = ': '.join(strsimdata)
        print('    ' + foo, bar, sep=': ')


i = 1
while len(data) > 1:
    print_matrix2(data, i)
    print('\n')
    step(data)
    i += 1
print('\n')
print_matrix2(data, i)
print('\n')

Validating the test-case for clustering string values

2014-09-10T12:43:40+02:00

Here we go again...

After further analysis (the trial & error kind), I found out that the error of modified values only occurs in unsorted data. Which makes perfect sense as Python's list.remove picks the first matching item. So the ordering of data plays a role.

As a quick fix, sorted data is now enforced when using the hierarchical clustering algorithm. Now, the only remaining test failure is the one comparing string values. And this time, I'll prepare a real test-case!

Which means manually calculating the matrix and clusters. So here we go!

The input data (lorem-ipsum split on white-space):

['Lorem', 'ipsum', 'dolor', 'sit', 'amet,', 'consectetuer', 'adipiscing',
 'elit.', 'Ut', 'elit.', 'Phasellus', 'consequat', 'ultricies', 'mi.',
 'Sed', 'congue', 'leo', 'at', 'neque.', 'Nullam.']

As distance function, we'll use Python's SequenceMatcher.

2h30m later

Funnily enough, the matrices are now generated using another quick-and-dirty script. With this input data, doing it manually was a bit too much after all. However, this time, the resulting values look actually correct!

This new algorithm is completely written from scratch, and could possibly replace the existing algorithm in the library. But I have other ideas for that one, so I'll leave it for now...

And indeed, fixing the above mentioned "sorting" issue fixes the problem. Running the tests with the new test-data, against the old code checks out.

There was one discrepancy though: On the final iteration of the test (going from matrix #10 to matrix #11) there are several candidates with a distance of 0.600. And the code which generated the test matrices below picked (leo, dolor), Lorem where the original algorithm picked adipiscing, ipsum. Both values are correct, so the result of the test is indeed valid!

Matrices

Matrix #1
	Lorem	ipsum	dolor	sit	amet	consectetuer	adipiscing	elit	Ut	elit	Phasellus	consequat	ultricies	mi	Sed	congue	leo	at	neque
Lorem
ipsum	0.800
dolor	0.600	1.000
sit	1.000	0.750	1.000
amet	0.778	0.778	1.000	0.714
consectetuer	0.765	0.765	0.765	0.733	0.750
adipiscing	1.000	0.600	0.867	0.846	0.857	0.818
elit	0.778	0.778	0.778	0.429	0.500	0.750	0.857
Ut	1.000	1.000	1.000	0.600	0.667	0.857	1.000	0.667
elit	0.778	0.778	0.778	0.429	0.500	0.750	0.857	0.000	0.667
Phasellus	0.857	0.714	0.857	0.833	0.692	0.714	0.789	0.692	1.000	0.692
consequat	0.714	0.714	0.857	0.667	0.692	0.429	0.789	0.692	0.818	0.692	0.667
ultricies	0.714	0.857	0.714	0.833	0.846	0.810	0.684	0.692	0.818	0.692	0.778	0.778
mi	0.714	0.714	1.000	0.600	0.667	1.000	0.833	0.667	1.000	0.667	1.000	1.000	0.818
Sed	0.750	1.000	0.750	1.000	0.714	0.867	0.846	0.714	1.000	0.714	0.833	0.833	0.833	1.000
congue	0.636	0.818	0.818	1.000	0.800	0.444	0.625	0.800	1.000	0.800	0.867	0.467	0.733	1.000	0.778
leo	0.750	1.000	0.500	1.000	0.714	0.867	1.000	0.714	1.000	0.714	0.833	0.833	0.667	1.000	0.667	0.778
at	1.000	1.000	1.000	0.600	0.333	0.857	0.833	0.667	0.500	0.667	0.818	0.636	0.818	1.000	1.000	1.000	1.000
neque	0.800	0.800	1.000	1.000	0.778	0.529	0.867	0.778	1.000	0.778	0.714	0.429	0.857	1.000	0.750	0.455	0.750	1.000
Nullam	0.818	0.636	0.818	1.000	0.600	0.889	0.875	0.800	1.000	0.800	0.733	0.733	0.733	0.750	1.000	0.833	0.778	0.750	0.818

Matrix #2
	Lorem	ipsum	dolor	sit	amet	consectetuer	adipiscing	Ut	Phasellus	consequat	ultricies	mi	Sed	congue	leo	at	neque	Nullam
Lorem
ipsum	0.800
dolor	0.600	1.000
sit	1.000	0.750	1.000
amet	0.778	0.778	1.000	0.714
consectetuer	0.765	0.765	0.765	0.733	0.750
adipiscing	1.000	0.600	0.867	0.846	0.857	0.818
Ut	1.000	1.000	1.000	0.600	0.667	0.857	1.000
Phasellus	0.857	0.714	0.857	0.833	0.692	0.714	0.789	1.000
consequat	0.714	0.714	0.857	0.667	0.692	0.429	0.789	0.818	0.667
ultricies	0.714	0.857	0.714	0.833	0.846	0.810	0.684	0.818	0.778	0.778
mi	0.714	0.714	1.000	0.600	0.667	1.000	0.833	1.000	1.000	1.000	0.818
Sed	0.750	1.000	0.750	1.000	0.714	0.867	0.846	1.000	0.833	0.833	0.833	1.000
congue	0.636	0.818	0.818	1.000	0.800	0.444	0.625	1.000	0.867	0.467	0.733	1.000	0.778
leo	0.750	1.000	0.500	1.000	0.714	0.867	1.000	1.000	0.833	0.833	0.667	1.000	0.667	0.778
at	1.000	1.000	1.000	0.600	0.333	0.857	0.833	0.500	0.818	0.636	0.818	1.000	1.000	1.000	1.000
neque	0.800	0.800	1.000	1.000	0.778	0.529	0.867	1.000	0.714	0.429	0.857	1.000	0.750	0.455	0.750	1.000
Nullam	0.818	0.636	0.818	1.000	0.600	0.889	0.875	1.000	0.733	0.733	0.733	0.750	1.000	0.833	0.778	0.750	0.818
elit,elit	0.778	0.778	0.778	0.429	0.500	0.750	0.857	0.667	0.692	0.692	0.846	0.667	0.714	0.800	0.714	0.667	0.778	0.800

Matrix #3
	Lorem	ipsum	dolor	sit	consectetuer	adipiscing	Ut	Phasellus	consequat	ultricies	mi	Sed	congue	leo	neque	Nullam	elit, elit
Lorem
ipsum	0.800
dolor	0.600	1.000
sit	1.000	0.750	1.000
consectetuer	0.765	0.765	0.765	0.733
adipiscing	1.000	0.600	0.867	0.846	0.818
Ut	1.000	1.000	1.000	0.600	0.857	1.000
Phasellus	0.857	0.714	0.857	0.833	0.714	0.789	1.000
consequat	0.714	0.714	0.857	0.667	0.429	0.789	0.818	0.667
ultricies	0.714	0.857	0.714	0.833	0.810	0.684	0.818	0.778	0.778
mi	0.714	0.714	1.000	0.600	1.000	0.833	1.000	1.000	1.000	0.818
Sed	0.750	1.000	0.750	1.000	0.867	0.846	1.000	0.833	0.833	0.833	1.000
congue	0.636	0.818	0.818	1.000	0.444	0.625	1.000	0.867	0.467	0.733	1.000	0.778
leo	0.750	1.000	0.500	1.000	0.867	1.000	1.000	0.833	0.833	0.667	1.000	0.667	0.778
neque	0.800	0.800	1.000	1.000	0.529	0.867	1.000	0.714	0.429	0.857	1.000	0.750	0.455	0.750
Nullam	0.818	0.636	0.818	1.000	0.889	0.875	1.000	0.733	0.733	0.733	0.750	1.000	0.833	0.778	0.818
elit,elit	0.778	0.778	0.778	0.429	0.750	0.857	0.667	0.692	0.692	0.846	0.667	0.714	0.800	0.714	0.778	0.800
at,amet	0.778	0.778	1.000	0.600	0.750	0.833	0.500	0.692	0.636	0.818	0.667	0.714	0.800	0.714	0.778	0.600	0.500

Matrix #4
	Lorem	ipsum	dolor	sit	adipiscing	Ut	Phasellus	ultricies	mi	Sed	congue	leo	neque	Nullam	elit, elit	at, amet
Lorem
ipsum	0.800
dolor	0.600	1.000
sit	1.000	0.750	1.000
adipiscing	1.000	0.600	0.867	0.846
Ut	1.000	1.000	1.000	0.600	1.000
Phasellus	0.857	0.714	0.857	0.833	0.789	1.000
ultricies	0.714	0.857	0.714	0.833	0.684	0.818	0.778
mi	0.714	0.714	1.000	0.600	0.833	1.000	1.000	0.818
Sed	0.750	1.000	0.750	1.000	0.846	1.000	0.833	0.833	1.000
congue	0.636	0.818	0.818	1.000	0.625	1.000	0.867	0.733	1.000	0.778
leo	0.750	1.000	0.500	1.000	1.000	1.000	0.833	0.667	1.000	0.667	0.778
neque	0.800	0.800	1.000	1.000	0.867	1.000	0.714	0.857	1.000	0.750	0.455	0.750
Nullam	0.818	0.636	0.818	1.000	0.875	1.000	0.733	0.733	0.750	1.000	0.833	0.778	0.818
elit,elit	0.778	0.778	0.778	0.429	0.857	0.667	0.692	0.846	0.667	0.714	0.800	0.714	0.778	0.800
at,amet	0.778	0.778	1.000	0.600	0.833	0.500	0.692	0.818	0.667	0.714	0.800	0.714	0.778	0.600	0.500
consequat,consectetuer	0.714	0.714	0.765	0.667	0.789	0.818	0.667	0.778	1.000	0.833	0.444	0.833	0.429	0.733	0.692	0.636

Matrix #5
	Lorem	ipsum	dolor	adipiscing	Ut	Phasellus	ultricies	mi	Sed	congue	leo	neque	Nullam	at, amet	consequat, consectetuer
Lorem
ipsum	0.800
dolor	0.600	1.000
adipiscing	1.000	0.600	0.867
Ut	1.000	1.000	1.000	1.000
Phasellus	0.857	0.714	0.857	0.789	1.000
ultricies	0.714	0.857	0.714	0.684	0.818	0.778
mi	0.714	0.714	1.000	0.833	1.000	1.000	0.818
Sed	0.750	1.000	0.750	0.846	1.000	0.833	0.833	1.000
congue	0.636	0.818	0.818	0.625	1.000	0.867	0.733	1.000	0.778
leo	0.750	1.000	0.500	1.000	1.000	0.833	0.667	1.000	0.667	0.778
neque	0.800	0.800	1.000	0.867	1.000	0.714	0.857	1.000	0.750	0.455	0.750
Nullam	0.818	0.636	0.818	0.875	1.000	0.733	0.733	0.750	1.000	0.833	0.778	0.818
at,amet	0.778	0.778	1.000	0.833	0.500	0.692	0.818	0.667	0.714	0.800	0.714	0.778	0.600
consequat,consectetuer	0.714	0.714	0.765	0.789	0.818	0.667	0.778	1.000	0.833	0.444	0.833	0.429	0.733	0.636
elit,elit,sit	0.778	0.750	0.778	0.692	0.600	0.692	0.833	0.600	0.714	0.800	0.714	0.778	0.800	0.500	0.667

Matrix #6
	Lorem	ipsum	dolor	adipiscing	Ut	Phasellus	ultricies	mi	Sed	congue	leo	Nullam	at, amet	elit, elit, sit
Lorem
ipsum	0.800
dolor	0.600	1.000
adipiscing	1.000	0.600	0.867
Ut	1.000	1.000	1.000	1.000
Phasellus	0.857	0.714	0.857	0.789	1.000
ultricies	0.714	0.857	0.714	0.684	0.818	0.778
mi	0.714	0.714	1.000	0.833	1.000	1.000	0.818
Sed	0.750	1.000	0.750	0.846	1.000	0.833	0.833	1.000
congue	0.636	0.818	0.818	0.625	1.000	0.867	0.733	1.000	0.778
leo	0.750	1.000	0.500	1.000	1.000	0.833	0.667	1.000	0.667	0.778
Nullam	0.818	0.636	0.818	0.875	1.000	0.733	0.733	0.750	1.000	0.833	0.778
at,amet	0.778	0.778	1.000	0.833	0.500	0.692	0.818	0.667	0.714	0.800	0.714	0.600
elit,elit,sit	0.778	0.750	0.778	0.692	0.600	0.692	0.833	0.600	0.714	0.800	0.714	0.800	0.500
consequat,consectetuer,neque	0.714	0.714	0.765	0.789	0.818	0.667	0.778	1.000	0.750	0.444	0.750	0.733	0.636	0.667

Matrix #7
	Lorem	ipsum	dolor	adipiscing	Ut	Phasellus	ultricies	mi	Sed	leo	Nullam	at, amet	elit, elit, sit
Lorem
ipsum	0.800
dolor	0.600	1.000
adipiscing	1.000	0.600	0.867
Ut	1.000	1.000	1.000	1.000
Phasellus	0.857	0.714	0.857	0.789	1.000
ultricies	0.714	0.857	0.714	0.684	0.818	0.778
mi	0.714	0.714	1.000	0.833	1.000	1.000	0.818
Sed	0.750	1.000	0.750	0.846	1.000	0.833	0.833	1.000
leo	0.750	1.000	0.500	1.000	1.000	0.833	0.667	1.000	0.667
Nullam	0.818	0.636	0.818	0.875	1.000	0.733	0.733	0.750	1.000	0.778
at,amet	0.778	0.778	1.000	0.833	0.500	0.692	0.818	0.667	0.714	0.714	0.600
elit,elit,sit	0.778	0.750	0.778	0.692	0.600	0.692	0.833	0.600	0.714	0.714	0.800	0.500
consequat,consectetuer,neque,congue	0.636	0.714	0.765	0.625	0.818	0.667	0.733	1.000	0.750	0.750	0.733	0.636	0.667

Matrix #8
	Lorem	ipsum	adipiscing	Ut	Phasellus	ultricies	mi	Sed	Nullam	at, amet	elit, elit, sit	consequat, consectetuer, neque, congue
Lorem
ipsum	0.800
adipiscing	1.000	0.600
Ut	1.000	1.000	1.000
Phasellus	0.857	0.714	0.789	1.000
ultricies	0.714	0.857	0.684	0.818	0.778
mi	0.714	0.714	0.833	1.000	1.000	0.818
Sed	0.750	1.000	0.846	1.000	0.833	0.833	1.000
Nullam	0.818	0.636	0.875	1.000	0.733	0.733	0.750	1.000
at,amet	0.778	0.778	0.833	0.500	0.692	0.818	0.667	0.714	0.600
elit,elit,sit	0.778	0.750	0.692	0.600	0.692	0.833	0.600	0.714	0.800	0.500
consequat,consectetuer,neque,congue	0.636	0.714	0.625	0.818	0.667	0.733	1.000	0.750	0.733	0.636	0.667
leo,dolor	0.600	1.000	0.867	1.000	0.833	0.667	1.000	0.667	0.778	0.714	0.714	0.750

Matrix #9
	Lorem	ipsum	adipiscing	Phasellus	ultricies	mi	Sed	Nullam	elit, elit, sit	consequat, consectetuer, neque, congue	leo, dolor
Lorem
ipsum	0.800
adipiscing	1.000	0.600
Phasellus	0.857	0.714	0.789
ultricies	0.714	0.857	0.684	0.778
mi	0.714	0.714	0.833	1.000	0.818
Sed	0.750	1.000	0.846	0.833	0.833	1.000
Nullam	0.818	0.636	0.875	0.733	0.733	0.750	1.000
elit,elit,sit	0.778	0.750	0.692	0.692	0.833	0.600	0.714	0.800
consequat,consectetuer,neque,congue	0.636	0.714	0.625	0.667	0.733	1.000	0.750	0.733	0.667
leo,dolor	0.600	1.000	0.867	0.833	0.667	1.000	0.667	0.778	0.714	0.750
at,amet,Ut	0.778	0.778	0.833	0.692	0.818	0.667	0.714	0.600	0.500	0.636	0.714

Matrix #10
	Lorem	ipsum	adipiscing	Phasellus	ultricies	mi	Sed	Nullam	consequat, consectetuer, neque, congue	leo, dolor
Lorem
ipsum	0.800
adipiscing	1.000	0.600
Phasellus	0.857	0.714	0.789
ultricies	0.714	0.857	0.684	0.778
mi	0.714	0.714	0.833	1.000	0.818
Sed	0.750	1.000	0.846	0.833	0.833	1.000
Nullam	0.818	0.636	0.875	0.733	0.733	0.750	1.000
consequat,consectetuer,neque,congue	0.636	0.714	0.625	0.667	0.733	1.000	0.750	0.733
leo,dolor	0.600	1.000	0.867	0.833	0.667	1.000	0.667	0.778	0.750
at,amet,Ut,elit,elit,sit	0.778	0.750	0.692	0.692	0.818	0.600	0.714	0.600	0.636	0.714

Matrix #11
	Lorem	Phasellus	ultricies	mi	Sed	Nullam	consequat, consectetuer, neque, congue	leo, dolor	at, amet, Ut, elit, elit, sit
Lorem
Phasellus	0.857
ultricies	0.714	0.778
mi	0.714	1.000	0.818
Sed	0.750	0.833	0.833	1.000
Nullam	0.818	0.733	0.733	0.750	1.000
consequat,consectetuer,neque,congue	0.636	0.667	0.733	1.000	0.750	0.733
leo,dolor	0.600	0.833	0.667	1.000	0.667	0.778	0.750
at,amet,Ut,elit,elit,sit	0.778	0.692	0.818	0.600	0.714	0.600	0.636	0.714
adipiscing,ipsum	0.800	0.714	0.684	0.714	0.846	0.636	0.625	0.867	0.750

The remaining matrices have been left out, as this last one is the one we want for our unit-test!

Stepping through the failing data

2014-09-09T14:04:53+02:00

This is a thumbnail of a visualisation of the separate clustering steps. Open original SVG file for better readability.

The linked image shows the individual steps through the existing clustering algorithm. Each "horizontal line" (the Y-Axis) represents one step of the algorithm. The distance between each "line" represents the level of the created cluster. The first line represents the input data (the black dots).

The X-Axis represents the input values (which ranged roughly between 0 and 1000).

On each iteration, the last generated cluster is marked as a red circle, where previous datums are shown as gray circles. As mentioned, the black dots on the first line are the input values.

Stepping through the code like this has revealed, that the error is not where I think it is. The values here are all straight out of debugging data from a test-run. In this execution, all values were present. Nothing went missing. I will have to continue my search...

So the Test isn't Failing

2014-09-08T18:33:11+02:00

It turns out, the test I prepared earlier passes. So the reported issue needs a bit more work. I wanted to avoid the data-set which reproduces the failure because I'm lazy.

I should have known better.

So I will now do a similar manual step through the data-set as I did earlier... This time, as the data is one-dimensional, I will need to find anotehr way to visualise it. And I also think I will do that by manually writing an SVG file. Clicking all that together with a mouse would be painful. Let's see where this gets me...

Blogging in reStructured Text

2014-09-04T18:22:57+02:00

I have always been quite a big fan of reStructured Text Text as markup format. It's easy to write and supports a lot of different text roles. It's not a lot more difficult than Markdown but can do a lot more. I always thought it was a great blogging "platform". So I went searching, and found Pelican as static site generator. I have been playing around with it now for two days and am impressed so far!

To me the two biggest highlights are:

Publishing from the shell and
Tracking articles via git

As git is a very viable option, publishing via hooks becomes a very nead possibility as well. For now, I'm handling it with fabric, and I don't see the immediate need to publish via hooks.

A Lesson in Unit Testing

2014-09-04T09:00:00+02:00

Contents

Hierarchical Clustering

Quite some time back, I wrote a Python library for simple clustering tasks. Back in the day, I only needed K-Means clustering. But I decided I would add hierarchical clustering as well as it could come in handy later on.

Turns out I messed up the unit tests, and the hierarchical clustering algorithm. But the error went unnoticed as the unit-tests were not testing one obvious fact:

Are all the elements from the input present in the output? And only those from the input?

The main cause of this error is because the way I obtained the expected values was complete heresy! I ran the test run on random values, and simply copy/pasted the results into the expected values. Of course now the test passed, with the important fact that the tests were testing against the wrong values!

In order to fix address this, I first need some proper test values. So now I do what I should have done in the first place: Manually step through the algorithm. This article contains my scribbles which I will use as a "mental strut".

Hierarchical Clustering

We will start of with a sample of non primitive data types. In this case points on a two-dimensional plane. As distance function we will use the euclidian distance. And, for the first step through, we will use single linkage between clusters. The same scenario will be played through with complete- and uclus-linkage later on. As overall method, we will use a bottom-up approach.

The three aproaches will result in the following dendograms:

Initial Step

So we have the following points:

a (1, 1)
b (9, 1)
c (2, 2)
d (3, 2)
e (9, 2)
f (3, 4)

Note

To keep the matrix table more readable, individual points are written in lower-case, while clusters are written in upper-case!

This results in the following initial distance matrix. We will mark the diagonal with an x as we're not interested in comparing a point with itself. Equally, we are only interested in the absolute distance. So we only need the values for pairs on ons side of the diagonal.

Distance Matrix
	a	b	c	d	e	f
a	x	8.00	1.41	2.24	8.06	3.61
b		x	7.07	6.08	1.00	6.71
c			x	1.00	7.00	2.24
d				x	6.00	2.00
e					x	6.32
f						x

Single Linkage

First Step

With this matrix, we see that the first candidates are [b, e] and [c, d]. We'll pick [b, e] as firt cluster (A):

a (1, 1)
A (9, 1), (9, 2) level=1.00
c (2, 2)
d (3, 2)
f (3, 4)

Distance Matrix
	a	A	c	d	f
a	x	8.00	1.41	2.24	3.61
A		x	7.00	6.00	6.32
c			x	1.00	2.24
d				x	2.00
f					x

Second Step

The next candidate is [c, d] as B:

a (1, 1)
A (9, 1), (9, 2) level=1.00
B (2, 2), (3, 2) level=1.00
f (3, 4)

Giving us:

Distance Matrix
	a	A	B	f
a	x	8.00	1.41	3.61
A		x	6.00	6.32
B			x	2.00
f				x

Third Step

Then [f, B] as C:

a (1, 1)
A (9, 1), (9, 2) level=1.00
C (3, 4), ((2, 2), (3, 2) level=1.00) level=2.00

Resulting in:

Distance Matrix
	a	A	C
a	x	8.00	1.41
A		x	6.00
C			x

Fourth Step

Then [a, C] as D:

A (9, 1), (9, 2) level=1.00
D (1, 1), ((3, 4), ((2, 2), (3, 2) level=1.00) level=2.00) level=1.41

Resulting in:

Distance Matrix
	A	D
A	x	6.00
D		x

Which gives us the final cluster E with a level of 6.00.

The end-result is the following dendogram:

               E
               |
   +-----------+-----------+
   |                       |
   |                       D
   |                       |
   |                 +-----+-----+
   |                 |           |
   |                 C           |
   |                 |           |
   |            +----+----+      |
   |            |         |      |
   A            B         |      |
   |            |         |      |
+--+--+      +--+--+      |      |
|     |      |     |      |      |
e     b      c     d      f      a

Complete Linkage

Initial distance matrix for reference:

Distance Matrix
	a	b	c	d	e	f
a	x	8.00	1.41	2.24	8.06	3.61
b		x	7.07	6.08	1.00	6.71
c			x	1.00	7.00	2.24
d				x	6.00	2.00
e					x	6.32
f						x

First Step

First iteration is identical, but distance matrix has different values. The subsequent steps will be displayed without aditional explanation, the idea is the same as above, simply using a different linkage function.

Distance Matrix
	a	A	c	d	f
a	x	8.06	1.41	2.24	3.61
A		x	7.07	6.08	6.71
c			x	1.00	2.24
d				x	2.00
f					x

Second Step

Distance Matrix
	a	A	B	f
a	x	8.06	2.24	3.61
A		x	7.07	6.71
B			x	2.24
f				x

Third Step

Note

We now have to make a choice. I have not yet decided on how to handle this situation to have a detereministic behaviour. My current train of thought is using python sets as data-structure which is unordered. So the algorithm could return either one here.

For a demonstration, we'll pick [Ba] as to have a different result from sinle linkage...

This will give us:

Distance Matrix
	C	A	f
C	x	8.06	3.61
A		x	6.71
f			x

Fourth Step

And finally

Distance Matrix
	D	A
D	x	8.06
A		x

The end-result is the following dendogram:

               E
               |
   +-----------+-----------+
   |                       |
   |                       D
   |                       |
   |                 +-----+-----+
   |                 |           |
   |                 C           |
   |                 |           |
   |            +----+----+      |
   |            |         |      |
   A            B         |      |
   |            |         |      |
+--+--+      +--+--+      |      |
|     |      |     |      |      |
e     b      c     d      a      f

Uclus

Initial distance matrix for reference:

Distance Matrix
	a	b	c	d	e	f
a	x	8.00	1.41	2.24	8.06	3.61
b		x	7.07	6.08	1.00	6.71
c			x	1.00	7.00	2.24
d				x	6.00	2.00
e					x	6.32
f						x

As in complete linkage, the first iteration is identical, but distance matrix has different values. So we will not go into too much detail and will skip the visualisations.

Distance Matrix
	a	A	c	d	f
a	x	8.03	1.41	2.24	3.61
A		x	7.04	6.04	6.52
c			x	1.00	2.24
d				x	2.00
f					x

Distance Matrix
	a	A	B	f
a	x	8.03	1.83	3.61
A		x	6.54	6.52
B			x	2.12
f				x

Distance Matrix
	C	A	f
C	x	7.04	2.24
A		x	6.52
f			x

And finally

Distance Matrix
	D	A
D	x	6.86
A		x

The end-result is the following dendogram:

               E
               |
   +-----------+-----------+
   |                       |
   |                       D
   |                       |
   |                 +-----+-----+
   |                 |           |
   |                 C           |
   |                 |           |
   |            +----+----+      |
   |            |         |      |
   A            B         |      |
   |            |         |      |
+--+--+      +--+--+      |      |
|     |      |     |      |      |
e     b      c     d      a      f

Add creation and modification timestamps to an Excel worksheet

2002-02-19T09:00:00+01:00

Please, for the love of $deity do not hit me….. This is going to be a post about excel!

Excel is a horrid solution for data entry, and even worse for data archival. And yet, it’s one of the most commonly used solutions. One of the most useful information in any given data-set is the information about when the information was created and when it was last modified. This is something that any decent developer in charge of a data collection (let’s just call it that for now) will add to each data record.

Alas, a lot of non-IT people manage and store their data in excel worksheets. And that is OK with me as long as they pay attention to data archival. In it’s most simple form, data archival can be achieved by storing the data as a CSV file and including the following metadata:

Which column represents which value (the name of the variable)
The data type (f.ex.: number, text, date, …) of each column
If a column is “coded”, please also include the meaning of each code. For example a “Yes”, “No”, “Maybe” column might be stored as “1”, “2” and “3”. Which means in it’s most basic nature it’s a numeric variable, but the different values have a meaning attached to them. So: Add this list in your metadata description.
If any computations or checks are performed on the values, please add them to the metadata document as well!

Even if the timestamp values might seem superflous at first, it will be of great help to anyone tracing errors in the data. Imagine that you would at some point need to fix some values that were entered/modified during a specific time period for whatever reason. Without this most basic bit of information you will be up for a treat. However, if it’s been rigurously implemented since the beginning, you’ll have the problem solved in no time.

Now, each halfway serious database system will offer you this kind of functionality out-of-the-box. But Excel is no database system (I intentionally left out the word “management” as this issue is a bit more general!). So it does not offer you a straight-forward way to solve this. But even if it’s not straight-forward, it’s simple enough for about anyone using Excel do add this bit of information.

Assuming that you use the first two columns (numbered 1 and 2 in excel) of your worksheet to add creation- and modification timestamps simply open up the Visual Basic editor (found in Tools->Macro or somesuch), next, in your project tree (in the top left of the screen) select your workbook (the .xls file), and in it’s sub-tree double-click the Worksheet that should have the timestamps set automatically.

Then copy/paste the following text into the just opened code editor and you’re done. I hope the comments will give some insight as to what happens. Note that in this case I will ignore the first row of the sheet, and obviously, the first two columns. If that does not suit your needs, feel free to change this script to your liking.

'
' Callback which is called when a cell in a workbook changes
' @param Target: The cell that changed it's value
'
Private Sub Worksheet_Change(ByVal Target As Range)
  ' We will ignore any changes in the first row, as it contains header
  ' labels
  If Target.Row = 1 Then Exit Sub
  ''

  ' As we set the values of column 1 and 2 we won't need to capture
  ' changes in these either
  If Target.Column = 1 Or Target.Column = 2 Then Exit Sub
  ''

  ' We will update the timestamp in column 2 *always* (last changed time)
  Cells(Target.Row, 2) = Now

  ' We will update the timestamp in column 1 only if it is empty (creation
  ' time)
  If IsEmpty(Cells(Target.Row, 1)) Then
     Cells(Target.Row, 1) = Now
   End If
End Sub

Michel Albert

Tidying Up My Python Libraries

t61codec 3.0.0

x690 2.0.0

config_resolver 5.2.0

gouge 2.3.0

Docs and Python Support

Nine Years of Quiet Building

The SNMP Ecosystem

Infrastructure & Ops Tooling

Web & Frontend Work

Hardware & IoT

Developer Workflow

What's Next

Colorize Flask Logging

Testing the Untestable

Preface

Challenge 1 - No Clear Entry-Point

Issues

Solution

Challenge 2 - Global State

Getting Flask-Social working

Flask Security

Flask Social

My (current) solution

Application Setup

Configuration

Data Model

Passing connections to your template

Remapping Keys in Windows

Alternative PostgreSQL plugin for munin — Part 4

Related Posts

No longer "alpha"

Changes since v1.0.0a2

Alternative PostgreSQL plugin for munin — Part 3

Related Posts

Comparison with the Official Plugins

Design Decisions

General

Replacing postgres_cache_ with 3 new I/O Plugins

Connections to the DB

Locks

Query Ages

Scan Types

Sizes

Row Access

Alternative PostgreSQL plugin for munin — Part 2

Related Posts

First Alpha Releases

Trying it out

Installation

Configuration

What's different?

Screenshots?

Alternative PostgreSQL plugin for munin

Abstract

Related Posts

Stipulations

Why modify the existing graphs?

Why a re-write? (The "design dilemma")

Design Decisions

Code Structure

Graph Visualisation

Current State

How to write "nested" graphs for Munin

Preface

Terms Used

Basics

Walkthrough

Essentials

Using the same code for multiple graphs (essentials #2)

Multigraphs

Nested Graphs

What are "nested" graphs?

Writing Nested Graphs

Controlling Raspberry-Pi LEDs the Hard Way

The Challenge

Starting Point

Plan of Action

Material Used

Replacing `postgres_cache_` with 3 new I/O Plugins