Welcome to configman’s documentation!

Contents:

Introduction (start here)

configman is a package that paves over the differences between various configuration methods to achieve a smooth road of cooperation between them.

We use it here at Mozilla to tie together all the different scripts and programs in Socorro.

The modules typically used for configuration in Python applications have inconsistent APIs. You cannot simply swap getopt for argparse and neither of them will do anything at all with configuration files like ini or json. And if applications do work with some configuration file of choice it usually doesn’t support rich types such as classes, functions and Python types that aren’t built in.

For example, it is possible with configman to define configuration in json and then automatically have ini file and command line support. Further, configman enables configuration values to be dynamically loaded Python objects, functions, classes or modules. These dynamically loaded values can, in turn, pull in more configuration definitions and more dynamic loading. This enables configman to offer configurable plugins for nearly any aspect of a Python application.

Getting started

Once you’ve understood what configman is, all you need to do is to install it:

$ pip install configman

The code is available on github: https://github.com/mozilla/configman To clone it all you need to do is:

$ git clone git://github.com/mozilla/configman.git

Once you have it installed, you usually start by importing configman in your scripts and programs, define the options in Python and then start exploring how you can use config files and more advanced type conversions.

Tutorial

We’re going to go from a simple application without configman, to a simple application with configman.

Basics

Suppose we write an app similar to the echo command line program in Unix and Linux. Our app, though reverses the lettering of each word:

from __future__ import absolute_import, division, print_function
def backwards(x):
    return x[::-1]

if __name__ == '__main__':
    import sys
    output_string = ' '.join(sys.argv[1:])
    print(backwards(output_string))

When run it could look something like this:

$ ./tutorial01.py Peter was here
ereh saw reteP

Now, suppose you want to add some more options that can be selected at run time. For example, it could remove all vowels from reversed string. So you can do this:

$ ./tutorial02.py --devowel Lars was here
rh sw srL

First, let’s improve our business logic to add this new feature:

def backwards(x):
    return x[::-1]

import re
vowels_regex = re.compile('[AEIOUY]', re.IGNORECASE)

def devowel(x):
    return vowels_regex.sub('', x)

Now we need configman to enable the run time option for removing the vowels from the input string. It will take the form of a command line switch. If, at run time, the switch is present, we’ll use the devowel function. If the switch is not present, we won’t use the function.

To add this option we create an option container. These containers are called namespaces and have a method that allows us to define our command line options.

Here’s a function that sets up a namespace with a single option:

from configman import Namespace, ConfigurationManager
...
def define_config():
    definition = Namespace()
    definition.add_option(
      name='devowel',
      default=False
    )

Before we can use this option definition, we need to wrap it up in a ConfigurationManager instance that is able to cook it up for us correctly:

...
config_manager = ConfigurationManager(definition)
config = config_manager.get_config()

That’s all! That last line returned an instance of what we call a DotDict. It is essentially a standard Python dict that’s had its __getattr__ cross wired with its __getitem__ method. This means that we can access the values in the dict as if they were attributes. Watch how we access the value of devowel in the full example below:

from __future__ import absolute_import, division, print_function
from configman import Namespace, ConfigurationManager

def backwards(x, capitalize=False):
    return x[::-1]

import re
vowels_regex = re.compile('[AEIOUY]', re.IGNORECASE)

def devowel(x):
    return vowels_regex.sub('', x)

def define_config():
    definition = Namespace()
    definition.add_option(
      name='devowel',
      default=False
    )
    return definition

if __name__ == '__main__':
    definition = define_config()
    config_manager = ConfigurationManager(definition)
    config = config_manager.get_config()
    output_string = ' '.join(config_manager.args)
    if config.devowel:
        output_string = devowel(output_string)
    print(backwards(output_string))

When run, you get what you expect:

$ ./tutorial02.py Peter was here
ereh saw reteP
$ ./tutorial02.py --devowel Peter was here
rh sw rtP

In the tutorial01.py example, we fetched the command line arguments using the reference to argv from the sys module. We couldn’t do that in the second tutorial because sys.argv included the command line switch --devowel. We don’t want that as part of the output. configman offers a version of the command line arguments with the switches removed. That’s the config_manager.args reference inside the join.

Intermediate

Now let’s expand some of the more powerful features of configman to see what it can help us with. Let’s start with the help. You invoke the help simply by running it like this:

$ ./tutorial02.py --help

That’s set up automatically for you. As you can see, it mentions, amongst other things, our --devowel option there. Let’s change the definition of the option slightly to be more helpful:

def define_config():
    definition = Namespace()
    definition.add_option(
      name='devowel',
      default=False,
      doc='Removes all vowels (including Y)',
      short_form='d'
    )

Now, when running --help it will explain our option like this:

-d, --devowel    Removes all vowels (including Y)

Let’s add another option so that we can get our text from a file instead of the command line. The objective is to get a file name from a --file or -f switch. We’ll set the default to be the empty string. If the user doesn’t use the switch, the value for this will be the empty string:

...
definition.add_option(
  name='file',
  default='',
  doc='Filename that contains our text',
  short_form='f'
)

Excellent! The whole thing together looks like this now:

from __future__ import absolute_import, division, print_function
from configman import Namespace, ConfigurationManager

def backwards(x, capitalize=False):
    return x[::-1]

import re
vowels_regex = re.compile('[AEIOUY]', re.IGNORECASE)

def devowel(x):
    return vowels_regex.sub('', x)

def define_config():
    definition = Namespace()
    definition.add_option(
      name='devowel',
      default=False,
      doc='Removes all vowels (including Y)',
      short_form='d'
    )
    definition.add_option(
      name='file',
      default='',
      doc='file name for the input text',
      short_form='f'
    )
    return definition

if __name__ == '__main__':
    definition = define_config()
    config_manager = ConfigurationManager(definition)
    config = config_manager.get_config()
    if config.file:
        with open(config.file) as f:
            output_string = f.read().strip()
    else:
        output_string = ' '.join(config_manager.args)
    if config.devowel:
        output_string = devowel(output_string)
    print(backwards(output_string))

And it’s executed like this:

$ cat > foo.txt
Peter works for Mozilla.^d
$ ./tutorial03.py  --file foo.txt
.allizoM rof skrow reteP
$ ./tutorial03.py --file foo.txt -d
.llzM rf skrw rtP

Persistent config files

Our examples so far have been very much about the command line. The whole point of using configman is so you can use various config file formats to provide configuration information to your programs. The real power of configman isn’t to wrap executable command line scripts but its ability to work ecumenically with config files.

To get started, let’s have our program itself write a configuration file for us. The easiest way is to use the --admin.dump_conf option that is automatically available. It offers different ways to output.

  • ini
  • conf
  • json
  • xml (future enhancement, if requested)

Let’s, for the sake of this tutorial, decide to use .ini files:

$ ./tutorial03.py --admin.dump_conf=./backwards.ini

This will write out a default configation file in ini format. configman figured that out by the file extension that you specified. If you had used ‘json’ instead, configman would have written out a json file:

$ ./tutorial03.py --admin.dump_conf=./backwards.ini
$ cat backwards.ini
[top_level]
# name: devowel
# doc: Removes all vowels (including Y)
devowel=False

# name: file
# doc: Filename that contains our text
file=

Any of the command line switches that you specify along with the --dump_conf switch will appear as the new defaults in the config file that is written:

$ python backwards.py --admin.dump_conf=./backwards.ini --file=/tmp/foo.txt
$ cat backwards.ini
[top_level]
# name: devowel
# doc: Removes all vowels (including Y)
devowel=False

# name: file
# doc: Filename that contains our text
file=/tmp/foo.txt

Next, let’s make our app always read from this file to get its defaults. To do that, we’re going to modify what is known as the hierarchy of value sources. configman, when determining what values to give to your option definitions, uses a list of sources. By default, it first checks the operating system environment. If the names of your options match anything from the environment, configman will pull those values in, overriding any defaults that you specified. Next it looks to the command line. Any values that it fetches will override the defaults as well as the environment variables.

If this default hierarchy of value sources doesn’t suit you, you may specify your own hierarchy. In our example, we’re going to want our configuration file to be the base value source. Then we want the environment variables and finally the command line. We can specify it like this:

value_sources = ('./backwards.ini', os.environ, getopt)

configman will walk this list, applying the values that it finds in turn. First it will read your ini file (you may want to use an absolute path to specify your ini file name). Second, we pass in a dict that represents the operating system environment. Interestingly, you can use any dict-like object that you want as a source. Third, we’re telling configman to use the getopt module to read the command line. In the future, we’ll have the argparse module available here.

To use this value source, we must specify it in the constructor:

config_manager = ConfigurationManager(definition,
                                      values_source_list=value_sources)

Now, the program will read from the ./backwards.ini config file whenever the application is run.

Suppose we change the last line of the file backwards.ini to instead say:

file=/tmp/bar.txt

And then create that file like this:

$ echo "Socorro" > /tmp/bar.txt

Now, our little program is completely self-sufficient:

$ ./tutorial04.py
orrocoS

Even though we’re using a config file, that doesn’t mean that we’ve eliminated the use of the command line. You can override any configuration parameter from command line:

$ ./tutorial04.py --devowel
rrcS
$ ./tutorial04.py We both work at Mozilla --file=
allizoM ta krow htob eW

More advanced options

We just covered how to turn a simple application to one where the configuration is done entirely by a ini file. Note: we could have chosen json or conf instead of ini and the program would be completely unchanged. Only your taste of config file format changed.

Type conversion

configman comes with an advanced set of type conversion utilities. This is necessary since config files don’t allow rich python types to be expressed. The way this is done is by turning things into strings and turning strings into rich python objects by labelling what type conversion script to use.

A basic example is that of booleans as seen in the Tutorial when it dumps the boolean devowel option as into an ini file. It looks like this:

[top_level]
# name: devowel
# doc: Removes all vowels (including Y)
devowel=False

As you can see it automatically figured out that the convertor should be configman.converters.boolean_converter. As you can imagine; under the hood configman does something like this:

# pseudo code
converter = __import__('configman.converters.boolean_converter')
actual_value = converter('False')

So, how did it know you wanted a boolean converter? It picked this up from the definition’s default value’s type itself. Reminder; from the Tutorial:

definition = Namespace()
definition.add_option(
  'devowel',
  default=False
)

Built-ins

The list of configman built-in converters will get you very far for basic python types. The complete list is this:

  • int
  • float
  • str
  • unicode
  • bool (empty string is False, non-empty string is True–use boolean_converter for better boolean conversion)
  • datetime.datetime (%Y-%m-%dT%H:%M:%S or %Y-%m-%dT%H:%M:%S.%f)
  • datetime.date (%Y-%m-%d)
  • datetime.timedelta (for example, 1:2:0:3 becomes days=1, hours=2, minutes=0, seconds=3)
  • type (see below)
  • types.FunctionType (see below)
  • compiled_regexp_type

The type and types.FunctionType built-ins are simpler than they might seem. It’s basically the same example pseudo code above. This example should demostrate how it might work:

import morse
namespace.add_option(
  'morsecode',
  '',
  'Turns morse code into real letters',
  from_string_converter=morse.morse_load
)

What this will do is it will import the python module morse and expect to find a function in there called morse_load. Suppose we have one that looks like this:

# This is morse/__init__.py
dictionary = {
  '.-.': 'p',
  '.': 'e',
  '-': 't',
  '.--.': 'r',
}


def morse_load(s):
    o = []
    for e in s.split(','):
        o.append(dictionary.get(e.lower(), '?'))
    return ''.join(o)

Another more advanced example is to load a class rather than a simple value. To do this you’ll need to use one of the pre-defined configman converters as the from_string_converter value. To our example above we’re going to add a configurable class:

from __future__ import absolute_import, division, print_function
from configman.converters import class_converter
namespace.add_option(
  'dialect',
  'morse.ScottishDialect',
  'A Scottish dialect class for the morse code converter',
  from_string_converter=class_converter
)

That needs to exist as an importable class. So we add it:

# This is morse/__init__.py
class ScottishDialect(object):
    def __init__(self, text):
        self.text = text

    def render(self):
        return self.text.replace('e', 'i').replace('E','I')

Now, this means that the class is configurable and you can refer to a specific class simply by name and it becomes available in your program. For example, in this trivial example we can use it like this:

if __name__ == '__main__':
    config = create_config()
    dialect = config.dialect(config.morsecode)
    print(dialect.render())

If you run this like this:

$ python morse-communicator.py --morsecode=.,-,.--.,-,.
itrti

This is just an example to whet your appetite but a more realistic example is that you might have a configurable class for sending emails. In production you might have it wired to be to something like this:

namespace.add_option(
  'email_send_class',
  'backends.SMTP',
  'Which backend should send the emails',
  from_string_converter=class_converter
)
namespace.add_option(
  'smtp_hostname',
  default='smtp.mozilla.org',
)
namespace.add_option(
  'smtp_username',
  doc='username for using the SMTP server'
)
namespace.add_option(
  'smtp_password',
  doc='password for using the SMTP server'
)

Then, suppose you have different backends for sending SMTP available you might want to run it like this when doing local development:

# name: email_send_class
# doc: Which backend should send the emails
dialect=backends.StdoutLogDumper

So that instead of sending over the network (which was default) it uses another class which knows to just print the emails being sent on the stdout or some log file or something.

Not built-ins

Suppose none of the built-ins in configman is what you want. There’s nothing stopping you from just writing down your own. Consider this tip calculator for example:

from __future__ import absolute_import, division, print_function
import getopt
from configman import Namespace, ConfigurationManager


def create_config():
    namespace = Namespace()
    namespace.add_option(
      'tip',
      default=20
    )
    import decimal
    namespace.add_option(
      'amount',
      from_string_converter=decimal.Decimal
    )
    value_sources = ('tipcalc.ini', getopt, )
    config_manager = ConfigurationManager([namespace], value_sources)
    return config_manager.get_config()


if __name__ == '__main__':
    config = create_config()
    tip_amount = config.amount * config.tip / 100
    print("(exact amount: %r)" % tip_amount)
    print('$%.2f' % tip_amount)

When run it will automatically convert whatever number you give it to a python Decimal type. Note how in the example it prints the repr of the calculated value:

$ python tipcalc.py --amount 100.59 --tip=25
(exact amount: Decimal('25.1475'))
$25.15

Indices and tables