The Direct Attribute Configuration Pattern

Most programs, especially libraries and frameworks, need "configuration". But exactly how to implement that is a murky subject, mostly because the boundary between "configuration" and "code" is itself ill-defined.

Context

So let's try to define it. The first thing you might notice is that the dictionary definition of "configuration", an arrangement of parts, is quite different from what you typically find in a modern "configuration file". For example, take a typical Apache httpd.conf file. It does contain several directives which identify components: LoadModule, for example. But far outweighing these are directives which set attributes, usually on an object or on the system as a whole. Directives like "Listen 80", "ThreadsPerChild 250", and "LogLevel debug", even though they could be implemented via arrangements of pieces, probably aren't. Instead, the values are most likely implemented as permanent cell variables which never appear, or move, or disappear, but instead only change in value. Even the LoadModule directive doesn't really arrange any pieces within a space; it merely identifies and includes them in an abstract set of "loaded modules". One might argue that the Location context directive deals with arrangements of URL's, but those aren't really arranged; they simply exist. You can't rearrange /path/to/resource to be above /path. No, the dictionary definition of "configuration" as "arrangement" is a holdover from our mostly-hardware past, where even the most dynamic "configuration system" still required moving cards and jumpers around in physical space.

There are some notable exceptions, of course, but the vast majority of software on the market today that is "configurable" consists of a fairly static set of objects, plus a formalized means of tweaking a subset of attributes of those objects. The most common exception to this, the "plugin", is also rarely arranged with respect to other plugins or components; instead, it is merely "turned on" or included. I believe this tendency is due to a natural human limitation: we just don't reason about graphs and networks very well yet, at least not nearly as well as we reason about vectors (of instructions) and sets. We feel good when working on serial problems, and bad when working on parallel ones. As Chris Alexander said:

There is little purpose, then, in saying: It would be better if this force did not exist. For if it does exist anyway, designs based on such wishful thinking will fail.

Conventional approaches and their problems

So then, let's discuss ways to implement this kind of "configuration". Again, let's look at Apache's httpd.conf: here we find almost a DSL, in that http_config.h defines functions to tokenize and parse a config file into another representation, a config vector. Then that intermediate structure is transformed into the actual used values like, say, request_rec->server->keep_alive_timeout.

Or take a typical postgresql.conf file. The entries therein are translated (via the ConfigureNamesBool array) to their internal variable names, and set globally. For example, check_function_bodies is implemented as an extern in guc.h. When a block of code needs to switch on the value of check_function_bodies, it #includes that header and reads the global value directly.

These designs carry with them several problems:

1. The set of configurable attributes is fixed when the program is compiled.
2. The set of configurable attributes must essentially be declared twice; once with an internal name, and again with an external name.
3. Often, these names are different (quite often only by CamelCase for one and names_with_underscores for the other!), increasing the cognitive load for anyone dealing in both.
4. Just as often, the types of the internal and external representations are different. The config file, for example, may allow specifying an attribute as "On" or "Off", but these are translated to the internal values 1 and 0. This mapping also increases cognitive load (I would argue by more than double).
5. The namespace of attributes is flat, often only a single level. This can make searching for the correct directive name more difficult than a hierarchical namespace. The latter also promotes browsing of related attributes.
6. If an intermediate structure is used as a scaffold, then "configuration variables" are declared together in one place, but read independently throughout the code base. In order to know what parts of a code block are configurable, the developer must search through the "config module" scaffolding, which is isolated from the code in question, and match up external names to internal effects.
7. In some implementations, the intermediate structure is not a scaffold, but is the final repository of "config values"; the values are never copied onto their "real" referents. This makes it easier to know which parts of a code block are configurable--just grep for calls to config.get! But often, the config.get call is much more expensive than reading local copies; when that happens, performance can drop sharply with lots of config reads (often multiple reads of the same value).
8. Memory usage is at least double for each configurable value since each one has an intermediate representation whether overridden or not. Quite often, the intermediate structures are retained long after they could have been freed.
9. Conventional config file parsers often are slower and less strict than the parsers for the general-purpose languages they hide. Config reads are slow and errors are delayed.
10. Config layers can be a lot of code. In Apache's server package, for example, config.c has more lines of code than any other C module except core.c.

A solution

There is a way to implement "configuration" as we have defined it above (setting values on named attributes) which avoids the above problems. Rather than defining a layer where external names, types, and values get translated to internal names, types, and values in an ad-hoc mapping, we can define a better translation step by obeying 3 very simple constraints:

1. External names are exactly the same as internal names,
2. External types are exactly the same as internal types, and
3. External values are exactly the same as internal values.

For example, if you have an internal "database" object with a "default_encoding" string attribute, the conventional approach might yield a config file entry like:

DatabaseDefaultEncoding: utf8

But if we follow the above constraints, we instead see config entries like this:

database.default_encoding = 'utf8'

We can generalize that to:

(path.to.object).key = value

...and in fact, we can write a simple parser which performs just that mapping. In the simplest implementation, only the set of objects is defined, and the set of keys is open-ended (that is, any attribute of the given object(s) is overridable):

for key, value in config.pairs():
    objname, attrname = key.rsplit(".", 1)
    obj = configurables[objname]
    setattr(obj, attrname, value)

In contrast to the conventional approach, in the "Direct Attribute Configuration" pattern:

1. The set of configurable attributes automatically changes as the program changes. Code which reads attributes may evolve independently of the config which writes them.
2. The set of configurable attributes must only be declared once. If objects are declared instead of individual attributes, that shrinks to "less than once".
3. The internal names are exactly the same as the external names; no translation required.
4. The internal types are exactly the same as the external types; less guessing which of (On, 1, -1, True, true, t, y, Yes, etc, etc) is allowed.
5. The namespace of attributes is hierarchical, promoting understanding via conceptual chunking.
6. In order to know what parts of a code block are configurable, the developer only needs to remember which few objects are configurable—all the attributes follow.
7. With no (or little) intermediate config structures, reads are fast.
8. With no (or little) intermediate config structures, memory consumption is reduced.
9. Re-using the grammar and parsers of the host language can reduce parsing time and raise syntax errors earlier and more easily.
10. There's less code.

Other considerations

• In an implementation of the Direct Attribute Configuration pattern, there is no longer any slippage between internal and external names. Sometimes that translation layer is useful to ease implementation improvements and deprecations. On the other hand, modern dynamic languages make that sort of renaming/refactoring less painful within the code itself: attributes may become properties, functions may morph into callables, and "get attribute" hooks can rewrite most any get/set as needed.
• Some will also say, "if config is now the same as code, why bother separating them?" The answer lies in another constraint we typically find in configuration files: key-value pairs. All declarations must follow this syntax, wherein a name is mapped to (the result of) an expression. Imperative statements are not allowed; if some are needed, the developer must wrap them in a function and expose it to config. This is the real semantic boundary between config and code.
• Some will further say, "operators aren't programmers", that the vagaries of syntax for various types (number, string, date, list, etc) is too much for busy admins and users. I would counter that by showing any existing config implementation. They all already have various syntax for various types, but unique to the whim of the config language designer; one uses commas to separate list items, another uses spaces; one uses ISO dates only, another allows 23 different date formats.
• If names, types and values are the same for config as for code, then the same authoring tools can often be used for them both. The same tooltips you love when writing code can pop up when writing config.
• It would be nice to extend the dotted-name format to command-line options as well as config file entries; however, some common option parsers (and even some shells) don't allow dots in option names.
• Since the set of configurable attributes is open-ended, it's harder to write a "Configuration entries for program X" document for a DAC implementation than a conventional one.
• The DAC pattern still doesn't address real arrangement-oriented configuration, especially acyclic graph construction.

That's enough for now; feel free to expand in the comments.