The Problem With Configurations

The Problem With Configurations

On April 3rd and 4th 2015, second DevopsDays conference will be held in Ljubljana, Slovenia. Avishai comes to DevopsDays Ljubljana for the 2nd time to speak; and this time around about what "work" is. We've asked him if he could write a bit about something that he cares about, and from the many topics he could cover, he picked configuration. If you're into systems or app development, you probably know how hard it is to get configuration right. Head on over to read Avishai's write-up on configuration and see what he thinks about how it is and could be done.


Configuration is one of those things we usually take for granted. Yet, as more and more configurations are added the use of configurations becomes hard and complicated. With enough configuration parameters, keeping track of what the current configuration “is” can be quite problematic. Often, people misconfigure systems due to various reasons such as poor documentation or unexpected interactions between different parameters. Sometimes, the configuration itself can be complex and requires a language to express it, raising the bar of required knowledge for defining the configuration.

Despite the “boring” nature of configurations they can and do obliterate entire companies in matter of hours.

Yet another config format

If you take a quick look in your /etc/ directory you will find a pile of configuration files using a large variety of format. From key/value shell include files through ini, toml, json, yaml, xml – and those are the relatively standard ones; If you care to look at sudoers, apt.conf, apache or postfix for example you will find more weird looking formats.

Unfortunately we humans need to remember the syntax for each format, not to mention other pecularities of each program configurations. For example, there is no standard for config file validation and some programs don’t have validation at all!

The vast variety of config formats makes system administration harder. It is hard to keep track and maintain a decent skill; some formats are very hard to generate from configuration management tools and scripts so we must resort to using error prone text templates.

Configuration drop bomb

The conf.d pattern has emerged many years ago as a way to allow different modules and packages to inject configuration snippets to other programs. In the absent of CM tools, it served humans to split large configurations to make management easier.

This pattern is generally considered a good practice yet it causes surprising amount of difficulty in maintenance and automation of systems. Since configurations are merged, values can be overridden from other files and are sometimes merged in unexpected ways. You change a configuration parameter, yet the program behaves as if you changed nothing; you grep through the directory and find this value has been set in a different file, one which you do not control. You must reason about the merge order and move your file to a higher priority, possibly overriding other parameters… and now you need to debug again.

When using CM tools this pattern is particularly annoying – if your CM tool manage files in the conf.d directory and you later remove these file resources the configuration files are not removed, they are simply no longer managed. Moreover, this allows overriding the automatic configurations by dropping files into conf.d manually, circumventing our effort to standardise configurations in servers.

Dude, did you restart just now?

Traditionally, config reload is done by sending a POSIX signal to the process, usually USR1 or HUP. The problem is that signal have no output and there is no obvious feedback telling us whether our reload succeeded. Perhaps the config file is malformed or we have set some parameter to an invalid value; Perhaps the signal was blocked or the program couldn’t handle it. We simply don’t know until we start digging through log files and even then we can’t be sure since the absent of “reload” message in the log file doesn’t actually mean the reload didn’t happen. On top of that it’s extremely difficult to automate such checks, so in most cases we simply give up and assume the configuration made it to our program or take the brutal approach and restart our process needlessly.

When in doubt, nuke from orbit

Some parameters that require a restart to change and some require a reload. CM tools have no way to identify what parameters changed in your configuration file and whether a reload is sufficient to activate the changes. As a result, we are forced to always use the nuclear option – restart. Unfortunately sometimes even the nukes are not enough, such as when changing MySQL innodb ibdata file size – which requires a stop – maintenance – start cycle; This forces us to resort to “nuke from orbit” methods of tearing down compute instances to support a configuration change.

I have no idea what’s going on

Just because the config file contains value X doesn’t actually mean that’s what loaded in the process. Perhaps the file was changed without reloading the configuration or the configuration path is wrong – this is actually very common. So how do you know what configuration is loaded in your server? how do you validate that all servers are configured properly? Most programs do not provide a good mechanism for this.

A better way

Like most operational aspects of programs, configuration issues can and should be resolved by grassroots engineering work rather then after-the-fact makeshift solutions. A good example of an attempt to tackle this at the core is Netflix Archaius project and many others have followed suite.

There are several simple design principles that can help make the configuration of your program much easier to work with. To some degree, you can even apply these principle to 3rd party programs using CM tools:

  1. Separate configuration to 2-3 files based on the impact of configuration parameter change: 1st file contains parameters that require a restart to change, 2nd contains parameters that require a reload and so on.
  2. Avoid using the `conf.d` pattern. Instead, have your CM tools merge values and create a small number of config files – making debugging and validation easier
  3. Create a configuration API. If using REST, GET method should return a complete dump of configuration parameters with e-tag header, and HEAD should return the e-tag header without a body. The e-tag header should be a checksum of the configuration in canonical order allowing for easy comparison between in-ram configuration of all nodes and reference version in CM
  4. If possible, use the REST configuration API to reload configurations using a POST or PUT requests. This allows your CM tools to easily validate that the configuration was successfully loaded and whether any values were updated (200/201/202 response). If not possible, write a small reload wrapper that verifies configuration was reloaded using whatever feedback the program provides.
  5. When writing new programs, choose a serialiazed format for configuration like json, yaml, edn, etc. Although they are not particularly comfortable for users to work with directly, remember that using CM tools and simple utilities people can work with whatever format they feel comformtable with as a long as a conversion utility exists.
  6. Some programs require advanced configuration employing logic (e.g. logstash) which doesn’t easily map to serialized formats; For these, treat the configuration as a plugin and extract variables to an external configuration file.


  1. Jamiie


    18. 9. 2020. in 4:11 pm Reply

    Hi Avishai, this was a great read and I love how you say “survivor of many prod skirmishes”, it made me laugh haha!

    Take care

Leave a Reply

Your email address will not be published.



John Biggs at TechCrunch Balkans in Sofia: There is a Way Out of This Media Black Hole!

The first TechCrunch Balkans meetup in Sofia begun with a loud whistle from Dilyan Dimitrov, founder of the Bulgarian accelerator Eleven, the host of this event. It was startup time! That is right, TechCrunch Balkans, organized by Netocratic, is being held throughout this week – we will have lectures, panels and the most important thing – pitches in front of John Biggs!


Combine Technical And Marketing Knowledge Within Your Startup Team

You've probably heard about Bellabeat, FlyKly, or Layer? Well, three years ago they didn't exist! Now they are at the top of their industries, but how did they do it? Was it enough only to have the right idea or did they also need to have knowledge to conduct business in today’s competitive environment? Specifically, what kind of knowledge do startups need to succeed and not repeat the many mistakes of the 90% of startups that fail?


Slovenian ‘Booze’ Startup Flaviar Raises 400.000 $ from Speedinvest!

Slovene-British startup Flaviar announced today that it raised a seed round of 400k $ from American fund Speedinvest. This one year old company ships monthly tasting packs of 5 sample spirits (whisky, rum, gin, vodka, etc.) to their members, each month with a different theme.

What you missed


Infobip’s Developer Relations Team Wins Two “DevRel Oscars” in London

Development Relations team of global cloud communications platform Infobip won two prestigious awards at the DevRelCon London 2023.


Through a series of developer meetups, Infobip came to the front of technological transformation in Africa

The first Croatian unicorn organized a series of successful developer meetups, which confirmed its leading position in this fast-developing continent.


What are Croatian designers of creative business spaces doing in the Emirates?

Croatian consulting company DC&T, specializing in creative, efficient, and humane office space design, is setting out to conquer the world markets. First step: Dubai.


One the best European developer conferences is coming to Miami

Shift in Croatia attracted thousands of developers and thus made its way to the very top of developer conferences in Europe. Now they are ready to cross the borders of Croatia and come to the USA. Applications for Shift Miami are starting soon!


Infobip (W)intership: Not only for software developers, and not only during summer!

More than 115 mentors will work with 130 interns at 7 Infobip's regional offices in 2023 edition of their internship programme. Last year 87% of interns got - job offers.


How does IT Consulting create experts who know how to integrate IT into all business spheres?

Keeping in mind how much the IT industry influences all other industries but also everyday life, it's surprising how little is said about IT consulting, which is practically the bridge that connects the two. That's why we sat down with a junior and two seniors on different career paths - to explore this profession with those who practice it, love it, and live it.