Avishai Ish-Shalom
13. 3. 2015.
On April 3rd and 4th 2015, second DevopsDays conference will be held in Ljubljana, Slovenia. Avishai comes to DevopsDays Ljubljana for the 2nd time to speak; and this time around about what "work" is. We've asked him if he could write a bit about something that he cares about, and from the many topics he could cover, he picked configuration. If you're into systems or app development, you probably know how hard it is to get configuration right. Head on over to read Avishai's write-up on configuration and see what he thinks about how it is and could be done.
Configuration is one of those things we usually take for granted. Yet, as more and more configurations are added the use of configurations becomes hard and complicated. With enough configuration parameters, keeping track of what the current configuration “is” can be quite problematic. Often, people misconfigure systems due to various reasons such as poor documentation or unexpected interactions between different parameters. Sometimes, the configuration itself can be complex and requires a language to express it, raising the bar of required knowledge for defining the configuration.
Yet another config format
If you take a quick look in your /etc/ directory you will find a pile of configuration files using a large variety of format. From key/value shell include files through ini, toml, json, yaml, xml – and those are the relatively standard ones; If you care to look at sudoers, apt.conf, apache or postfix for example you will find more weird looking formats.
Unfortunately we humans need to remember the syntax for each format, not to mention other pecularities of each program configurations. For example, there is no standard for config file validation and some programs don’t have validation at all!
Configuration drop bomb
The conf.d pattern has emerged many years ago as a way to allow different modules and packages to inject configuration snippets to other programs. In the absent of CM tools, it served humans to split large configurations to make management easier.
This pattern is generally considered a good practice yet it causes surprising amount of difficulty in maintenance and automation of systems. Since configurations are merged, values can be overridden from other files and are sometimes merged in unexpected ways. You change a configuration parameter, yet the program behaves as if you changed nothing; you grep through the directory and find this value has been set in a different file, one which you do not control. You must reason about the merge order and move your file to a higher priority, possibly overriding other parameters… and now you need to debug again.
When using CM tools this pattern is particularly annoying – if your CM tool manage files in the conf.d directory and you later remove these file resources the configuration files are not removed, they are simply no longer managed. Moreover, this allows overriding the automatic configurations by dropping files into conf.d manually, circumventing our effort to standardise configurations in servers.
Dude, did you restart just now?
Traditionally, config reload is done by sending a POSIX signal to the process, usually USR1 or HUP. The problem is that signal have no output and there is no obvious feedback telling us whether our reload succeeded. Perhaps the config file is malformed or we have set some parameter to an invalid value; Perhaps the signal was blocked or the program couldn’t handle it. We simply don’t know until we start digging through log files and even then we can’t be sure since the absent of “reload” message in the log file doesn’t actually mean the reload didn’t happen. On top of that it’s extremely difficult to automate such checks, so in most cases we simply give up and assume the configuration made it to our program or take the brutal approach and restart our process needlessly.
When in doubt, nuke from orbit
I have no idea what’s going on
A better way
Like most operational aspects of programs, configuration issues can and should be resolved by grassroots engineering work rather then after-the-fact makeshift solutions. A good example of an attempt to tackle this at the core is Netflix Archaius project and many others have followed suite.
There are several simple design principles that can help make the configuration of your program much easier to work with. To some degree, you can even apply these principle to 3rd party programs using CM tools:
- Separate configuration to 2-3 files based on the impact of configuration parameter change: 1st file contains parameters that require a restart to change, 2nd contains parameters that require a reload and so on.
- Avoid using the `conf.d` pattern. Instead, have your CM tools merge values and create a small number of config files – making debugging and validation easier
- Create a configuration API. If using REST, GET method should return a complete dump of configuration parameters with e-tag header, and HEAD should return the e-tag header without a body. The e-tag header should be a checksum of the configuration in canonical order allowing for easy comparison between in-ram configuration of all nodes and reference version in CM
- If possible, use the REST configuration API to reload configurations using a POST or PUT requests. This allows your CM tools to easily validate that the configuration was successfully loaded and whether any values were updated (200/201/202 response). If not possible, write a small reload wrapper that verifies configuration was reloaded using whatever feedback the program provides.
- When writing new programs, choose a serialiazed format for configuration like json, yaml, edn, etc. Although they are not particularly comfortable for users to work with directly, remember that using CM tools and simple utilities people can work with whatever format they feel comformtable with as a long as a conversion utility exists.
- Some programs require advanced configuration employing logic (e.g. logstash) which doesn’t easily map to serialized formats; For these, treat the configuration as a plugin and extract variables to an external configuration file.
Comments
Jamiie
18. 9. 2020. in 4:11 pm
Hi Avishai, this was a great read and I love how you say “survivor of many prod skirmishes”, it made me laugh haha!
Take care
Jamie