Stuffer - simplified, container-friendly provisioning

Stuffer is a provisioning tool designed to be simple, and to be used in simple scenarios, primarily for provisioning container images.

Documentation is hosted on http://stuffer.readthedocs.io. The source code lives at https://bitbucket.org/mapflat/stuffer.

Project status

Alpha. Raw but usable.

While stuffer is in alpha stage, i.e. version 0.0.x, the interface may change with any release.

Use cases

Stuffer is primarily intended to be used for provisioing container images, Docker in particular. As a secondary use case, it can be used to provision non-production machines, e.g. developer machines.

More complex provisioning tools, such as Puppet, Chef, and Ansible, are intended for bringing a machine in an arbitrary state to a desired state. This turns out not to be possible in practice, and production machines manages with such tools tend to suffer from deviations from intended state, e.g. outdated transitive dependency packages. At the other end of the complexity scale, good old bash has little support for reuse and for encoding decided best practices. Stuffer provides a middle road, aiming to keep the simplicity of shell commands, augmented with support for sharing constructs between projects.

Stuffer is primarily intended for building a machine from scratch to the desired state. Since the initial state is known, much of the complexity of existing provisioning tools is unnecessary. For example, during an image build, running services need not be considered nor restarted.

Overview

Stuffer is installed through pip3. It is based on Python 3, so plain pip will not work unless Python 3 is default on your platform.

sudo apt-get update && apt-get install -y python3-pip
sudo pip3 install stuffer

Stuffer uses a Python embedded DSL for specifying provisioning directives. It is typically invoked with one or more command arguments on the command line, e.g.:

stuffer 'apt.Install("mercurial")'

Multiple arguments are concatenated into a multiple line Python recipe:

stuffer \
  'for pkg in "mercurial", "gradle", "python-nose":' \
  '  print("Installing", pkg)' \
  '  apt.Install(pkg)'

When provisioning Docker containers, stuffer should typically be invoked multiple times, since it allows Docker to use the local cache efficiently:

RUN stuffer 'apt.Install("mercurial")'
RUN stuffer 'apt.Install("gradle")'
RUN stuffer 'apt.Install("python-nose")'

Reused recipes can be factored out into Python modules for easier reuse:

stuffer 'development.Tools()'

In development.py:

from stuffer.core import Group

class Tools(Group):
  def children(self):
    return [apt.Install(p) for p in "mercurial", "gradle", "python-nose"]

It is also possible to create composite actions by explicitly executing other actions. Example from stuffer.contrib.docker:

class Prologue(Action):
    """Check that the Docker base image is sound and prepare the image."""

    def run(self):
        # Check that the image is ok
        ...
        # Always needed, or apt install complains.
        apt.Install(['apt-utils']).execute()

Stuffer comes with builtin knowledge of Docker practices, and helps you steer away from common mistakes, and towards best practices. There is not consensus on a single set of Docker best practices, but stuffer provides a means to express your organisation’s decided practices in code, instead of educating all developers on the right incantations and rituals.

Dockerfile:
  FROM phusion/baseimage:0.9.18

  RUN stuffer 'docker.Prologue()'  # Verifies e.g. that base image is sound.

  RUN stuffer 'apt.Install("some-package")'

  RUN stuffer 'apt.SourceList("deb http://some.external.repository.com stable non-free")'
  RUN stuffer 'apt.Install("other-package")'  # Automatically runs 'apt-get update' before 'apt-get install'

  RUN stuffer 'docker.Epilogue()'  # Cleans temporary files.

Design goals

Stuffer design gives priority to:

  • Simplicity of use. No knowledge about the tool should be required in order to use it for simple scenarios by copying examples. Some simplicity in the implementation is sacrificed in order to make the usage interface simple. Actions are named similarly to the corresponding shell commands.
  • Transparency. Whenever reasonable, actions are translated to shell commands. All actions are logged.
  • Ease of reuse. It should be simple to extract commands from snippets and convert them to reusable modules without a rewrite. Therefore, both the DSL and modules are written in Python.
  • Docker cache friendliness. Images built with similar commands should be able to share a prefix of commands in order to benefit from Docker image caching.
  • No dislike factors. Provisioning tools tend to be loved and/or hated by users, for various reasons. There might be no reason to be passionately enamoured with stuffer, but there should be no reason to have a strong dislike for it, given that you approve of Python and Docker.
  • Ease of debugging. Debugging stuffer recipes should be as easy as debugging standard Python programs.
  • Avoid reinventing wheels. Use existing Python modules or external tools for tasks that have already been solved. Give priority to reusing existing code over minimising dependencies. In particular, use Python 3 and click to save boilerplate.

Moreover, the project model is design to facilitate sharing and reuse of code between users, see below.

DSL

The DSL is designed to be comprehensible by readers that are not familiar with stuffer. For example, the command apt.Install("mypack") runs apt-get install mypack. There is a balance between convenience and comprehensibility. Stuffer in most cases shuns magic that would create convenience in preference for more understandable code.

The DSL is also designed to make it easy to do things that are correct and work well with containers, and difficult to do things that do not harmonise with containers.

The DSL is designed to be tool friendly (with IntelliJ/PyCharm and pylint in particular), both for writing stuff files and for working on stuffer itself. For example, all imports are explicitly declared in order to make package structure comprehensible for tools.

Python conventions are used for naming, i.e. CamelCase classes and snake_case functions.

Actions

Each desired mutation of a container is represented by an Action. There are Actions for installing packages, changing file contents, setting configuration variables, etc. The different types of actions are represented by different subclasses of Action. Implementations of Action should be idempotent; stuffer will not perform any checks whether the Action is redundant, and each Action specified will be run. Many system administration commands are naturally idempotent, e.g. apt-get install. For Actions that are not, the Action implementation needs to include appropriate checks.

Implementations of Action specify what commands to execute by overrinding either Action.command or Action.run.

Prerequisites

Actions may specify that another Action needs to have been executed before Action.run by overriding Action.prerequisites. For example, pip.Install specifies that the pip command must be installed before using it to install other packages. Although the same effect can be achieved by explicitly running the required preparatory steps inside Action.run, it is more natural to separate the prerequisites from the command specified by the user. It also allows a potential future version of stuffer to keep track of executed prerequisites and avoid redundant executions.

Passing state

A container provisioning recipe typically consists of multiple stuffer invocations. The invocations do not share state, except for the container file system. Hence, if you need to pass state between invocations, you will need to save state in the file system.

Stuffer provides a simple key/value store mechanism to pass state between invocations via files in the container file system. Use store.set_value to store values, and store.get to retrieve them. The naming convention for keys is lower snake case, separated by dots for hierarchical organisation, e.g. my_corp.databases.mysql.preferred_driver. The prefix stuffer.` is reserved for stuffer components, which should use key names corresponding to the stuffer package name, e.g. stuffer.apt.update_needed.

The values in the store are plain strings.

Developing stuffer

Collaboration model

Users are allowed to put recipes under sites/ for others to get inspired. This model may not scale, but as long as the number of users is small, there is value in sharing and showing each other code snippets, in order to extract pieces of common value.

Snippets worth reuse can be put under stuffer/contrib/. Files under stuffer/contrib are expected to be maintained by the contributor.

Routines for installing third-party software should also go under stuffer/contrib.

Contributing

New code should be covered with integration tests. Avoid unit tests - since the purpose of stuffer is integration, there is little value testing scenarios that are not authentic. Strive to figure out a way to test functionality with Docker containers.

In order to run the test suite, run tox in the project root directory. The continuous integration build also bulds the documentation and performs a distribution build. See shippable.yml for the exact commands.

When tests pass, fork https://bitbucket.org/mapflat/stuffer, push your code to the fork, and create a pull request.

Build and release

Continuous integration builds are run with Shippable. Shippable builds a release package for every merge or push to master branch. If the version number is higher than the current version on https://pypi.python.org, the CI build uploads a new release. Hence, in order to make a new release, update the version number in main.py and setup.py before merging to master.

Deployment

Install the latest version with pip3 install stuffer, depending on the default python version in your environment.

In order to create an installable distribution package from the source directory, run ./setup.py sdist from the project root directory. Install with pip3 install dist/stuffer-*.tar.gz.

Q & A

Q: Stuffer sounds similar to Packer. What is the relation?

A: Packer is a tool for creating a container, given that you provide stuff to put in the container. Stuffer is a way to express what stuff to put in a container, given that you provide a way to pack the container. They can be used together, if desired. Packer is made by Hashicorp, who have no relation to Stuffer.

Q: I think that Docker containers should be built according to the following principle: <your preference here>. Why doesn’t stuffer do that?

A: There is no single best way to build Docker images. There are tradeoffs involved. Stuffer gives you a way to express your preferences, and package it as code, reusable by your colleagues. Feel free to submit a PR that implements your preferences as an optional strategy.

Q: Does it scale to more complex scenarios? Can I see some examples?

A: You can find some non-trivial examples at https://bitbucket.org/mapflat/stuffer/src/master/sites/mapflat/.

Known issues

There is a name clash between the click command line parser library and a Ubuntu python package for handling the click packaging format. Hence, you might run into trouble if you have the former installed on your machine, or in the Docker images that you wish to build. At this point, you can either solve it by removing the conflicting package, or by installing stuffer in a virtual environment (virtualenv).

API Reference

stuffer Stuffer - simple Docker-friendly provisioning.
stuffer.contrib

Indices and tables