Writing Buildout recipes

There are two kinds of buildout recipes: install and uninstall. Install recipes are by far the most common. Uninstall recipes are very rarely needed because most install recipes add files and directories that can be removed by Buildout.

Install recipes

Install recipes are typically implemented with classes and have 3 important parts:

  • A constructor (typically, __init__) initializes a recipe object.

    The constructor plays a very important role, because it may update the configuration data it’s passed, making information available to other parts and controlling whether a part will need to be re-installed.

    The constructor performs the first of two phases of recipe work, the second phase being the responsibility of either the install or update methods.

  • The install method installs new parts.

  • The update method updates previously installed parts. It’s often an empty method or an alias for install.

Buildout phases

When buildout is run using the default install command, parts are installed in several phases:

  1. Parts are initialized by calling their recipe constructors. This may cause part configuration options to be updated, as described below.

  2. Part options are compared to part options from previous runs [1].

    • Parts from previous runs that are no longer part of the buildout are uninstalled.
    • Parts from previous runs whose options have changed are also uninstalled.
  3. Parts are either installed or updated.

    install() is called on new parts or old parts that were uninstalled.

    update() is called on old parts whose configuration hasn’t changed.

Initialization phase: the constructor

The constructor is passed 3 arguments:

buildout

The buildout configuration

The buildout configuration is a mapping from section names to sections. Sections are mappings from option names to values. The buildout configuration allows the recipe to access configuration data in much the same way as configuration files use value substitutions.

name
The name of the section the recipe was used for
options

The part options

This is a mapping object and may be written to to save derived configuration, to provide information for use by other part recipes, or for value substitutions.

Nothing should be installed in this phase.

If the part being installed isn’t new, options after calling the constructor are compared to the options from the previous Buildout run. If they are different, then the part will be uninstalled and then re-installed by calling the install method, otherwise, the update method will be called.

Install or update phase

In this phase, install() or update() is called, depending on whether the part is new or has new configuration.

This is the phase in which the part does its work. In addition to affecting changes, these methods have some responsibilities that can be a little delicate:

  • If an error is raised, it is the responsibility of the recipe to undo any partial changes.
  • If the recipe created any files or directories, the recipe should return their paths. Doing so allows Buildout to take care of removing them if the part is uninstalled, making a separate uninstall recipe unnecessary.

To make these responsibilities easier to cope with, the option object passed to the constructor has a helper function, created. It should be passed one or more paths just before they are created and returns a list of all of the paths passed as well as any earlier paths created. If an exception is raised, any files or directories created will be removed automatically. When the recipe returns, it can just return the result of calling created() with no arguments.

Example: configuration from template recipe

In this example, we’ll show a recipe that creates a configuration file based on a configuration string computed using value substitutions [2]. A sample usage:

[buildout]
develop = src
parts = server

[config]
recipe = democonfigrecipe
port = 8080
contents =
   <zeo>
     address ${:port}
   </zeo>
   <mappingstorage>
   </mappingstorage>

[server]
recipe = zc.zdaemonrecipe
program = runzeo -C ${config:path}

Some things to note about this example:

  • The config part uses the recipe whose source code we’ll show below. It has a port option, which it uses in its contents option. It could as easily have used options from other sections.
  • The server part uses ${config:path} to get the path to the configuration file generated by the config part. The path option value will be computed by the recipe for use in other parts, as we’ve seen here.
  • We didn’t have to list the config part in the buildout parts option. It’s added automatically by virtue of its use in the server part.
  • We used the develop option to specify a src directory containing our recipe. This allows us to use the recipe locally without having to build a distribution file.

If we were to run this buildout, a parts/config file would be generated:

<zeo>
  address 8080
</zeo>
<mappingstorage>
</mappingstorage>

as would a zdaemon configuration file, parts/server/zdaemon.conf, like:

<runner>
  daemon on
  directory /sample/parts/server
  program runzeo -C /sample/parts/config
  socket-name /sample/parts/server/zdaemon.sock
  transcript /sample/parts/server/transcript.log
</runner>

<eventlog>
  <logfile>
    path /sample/parts/server/transcript.log
  </logfile>
</eventlog>

Here’s the recipe source, src/democonfigrecipe.py:

import os

class Recipe:

    def __init__(self, buildout, name, options):
        options['path'] = os.path.join(
           buildout['buildout']['parts-directory'],
           name,
           )
        self.options = options

    def install(self):
        self.options.created(self.options['path'])
        with open(self.options['path'], 'w') as f:
            f.write(self.options['contents'])
        return self.options.created()

    update = install

The constructor computes the path option. This is then available for use by the server part above. It’s also used later in the install method. We use buildout['buildout']['parts-directory'] to get the buildout parts directory. This is equivalent to using ${buildout:parts-directory} in the configuration. The parts directory is the standard place for recipes to create files or directories. If a recipe uses the parts directory, it should create only one file or directory whose name is the part name, which is passed in as the name argument to the constructor.

The constructor saves the options so that the data and created method are available in install.

The install method calls the option object’s created method before creating a file. The order is important, because if the file-creation fails partially, the file will be removed automatically. The recipe itself doesn’t need an exception handler. The configuration file is then written out. Finally, the created method is called again [3] to return the list of created files (one, in this case).

The update method is just an alias for the install method. We could have used an empty method, however running install again makes sure the file contents are as expected, overwriting manual changes, if any.

Like the install method, the update method returns any paths it created. These are merged with values returned by the install or update in previous runs.

For this recipe to be usable, we need to make it available as a distribution [4], so we need to create a setup script, src/setup.py:

from setuptools import setup

setup(
    name='democonfigrecipe',
    version='0.1.0',
    py_modules = ['democonfigrecipe'],
    entry_points = {"zc.buildout": ["default=democonfigrecipe:Recipe"]},
    )

The setup script specifies a name and version and lists the module to be included.

The setup script also uses an entry_points option. Entry points provide a miniature component systems for setuptools. A project can supply named components of given types. In the example above, the type of the component is "zc.buildout", which is the type used for Buildout recipes. A single components named default is provided. The component is named as the Recipe attribute of the democonfigrecipe module. When you specify a recipe in the recipe option, you name a recipe requirement, which names a project, and optionally provide a recipe name. The default name is default. Most recipe projects provide a single recipe component named default.

If we removed the server part from the configuration, the two configuration files would be removed, because Buildout recorded their paths and would remove them automatically.

Uninstall recipes

Uninstall recipes are very rarely needed, because most recipes just install files and Buildout can handle those automatically.

An uninstall recipe is just a function that takes a name and an options mapping. One of the few packages with an uninstall recipe is zc.recipe.rhrc. The uninstall function there provides the uninstall recipe. Here’s a highly simplified version:

def uninstall(name, options):
   os.system('/sbin/chkconfig --del ' + name)

This was used with a recipe that installed services on older Red Hat Linux servers. When the part was uninstalled, it needed to run /sbin/chkconfig to disable the service. Uninstall recipes don’t need to return anything.

Like install recipes, uninstall recipes need to be registered using entry points, using the type zc.buildout.uninstall as can be seen in the zc.recipe.rhrc setup script.

User interaction: logging and UserError

Recipes communicate to users through logging and errors. Recipes can log information using the Python logging library and messages will be displayed according to buildout’s verbosity setting.

Errors that a user can potentially correct should be reported by raising zc.buildout.UserError exceptions with error messages as arguments.

Buildout will display these as user errors, rather than printing a trace back.

Testing recipes

The recipe API is fairly simple and standard unit-testing approaches can be used. We’ll use a helper class, zc.buildout.testing.Buildout [5] to provide a minimal buildout environment.

Let’s write a test for our configuration recipe. We need to verify that:

  • The recipe generates a path option.
  • The recipe generates a file in the correct place.
  • The recipe returns the path it created from install.

We create a testdemoconfigrecipe.py file containing our tests:

import os
import shutil
import tempfile
import unittest
import zc.buildout.testing

class RecipeTests(unittest.TestCase):

    def setUp(self):
        self.here = os.getcwd()
        self.tmp = tempfile.mkdtemp(prefix='testdemoconfigrecipe-')
        os.chdir(self.tmp)
        self.buildout = buildout = zc.buildout.testing.Buildout()
        self.config = 'some config text\n'
        buildout['config'] = dict(contents=self.config)
        import democonfigrecipe
        self.recipe = democonfigrecipe.Recipe(
            buildout, 'config', buildout['config'])

    def tearDown(self):
        os.chdir(self.here)
        shutil.rmtree(self.tmp)

    def test_path_option(self):
        buildout = self.buildout
        self.assertEqual(os.path.join(buildout['buildout']['parts-directory'],
                                      'config'),
                         buildout['config']['path'])

    def test_install(self):
        buildout = self.buildout
        self.assertEqual(self.recipe.install(), [buildout['config']['path']])
        with open(buildout['config']['path']) as f:
            self.assertEqual(self.config, f.read())

if __name__ == '__main__':
    unittest.main()

In the setUp method, we created a temporary directory and changed to it. This is useful to make sure we have a clean working directory. We clean it up in the tearDown method.

Our test uses zc.buildout so that we can use the zc.buildout.testing.Buildout helper class. We did this so we’d have a more realistic environment, but of course, we could have stubbed this out ourselves. Because we’re using zc.buildout in our test, we’ll add it as a test dependency in our setup script:

from setuptools import setup

setup(
    name='democonfigrecipe',
    version='0.1.0',
    py_modules = ['democonfigrecipe', 'testdemoconfigrecipe'],
    entry_points = {"zc.buildout": ["default=democonfigrecipe:Recipe"]},
    extras_require = dict(test=['zc.buildout >=2.9']),
    )

Here, we defined an “extra” requirement. These are additional dependencies needed to support optional features. In this case, we’re providing an optional test feature. (We specified that we want at least version 2.9, because we’re depending on some testing-support refinements that were added in zc.buildout 2.9.0.)

We’ll write a development buildout to run our tests with:

[buildout]
develop = src
parts = py

[py]
recipe = zc.recipe.egg
eggs = democonfigrecipe [test]
interpreter = py

Running Buildout with this gives is an interpreter script that we can run our tests with. The script will make sure that zc.buildout and our recipe can be imported.

To run our tests:

bin/py src/testdemoconfigrecipe.py

In this example, we’ve tried to keep things simple and as free from external requirements as possible.

More realistically:

  • You’d probably arrange your recipe in a Python package rather than as a top-level module and a top-level testing module.
  • You might use a test runner like nose or pytest. There are recipes that can help set this up. We just used the test runner built into unittest.

zc.buildout.testing reference

The zc.buildout.testing module provides an API that can be used when writing recipe tests. This API is documented below.

Many of the functions documented below take a path argument as multiple arguments. These are joined using os.path.join. This is more convenient than having to call os.path.join before calling the functions.

Buildout()

A class you can use to create buildout and sections objects in your tests

This is a subclass of the main object used to run buildout. Its constructor takes no arguments. You can add data to it by setting section names to dictionaries:

buildout['config'] = dict(contents=self.config)

To get an options object to pass to your recipe, just ask for it back:

buildout['config']

See the recipe example above.

cat(*path)

Display the contents of a file. The file path is provided as one or more strings, to be joined with os.path.join.

On Windows, if the file doesn’t exist, the function will try adding a ‘-script.py’ suffix. This helps to work around a difference in script generation on windows.

clear_here()

Remove all files and directories in the current working directory.

New in buildout 2.9

eqs(got, *expected)

Compare a collection with a collection given as multiple arguments.

Both collections are converted to and compared as sets. If the sets are the same, then no output is returned, otherwise a tuple of extras is returned, so, for example:

>>> eqs([1, 2, 3], 3, 1, 2)
>>> eqs([1, 2, 3], 1, 2, 4) == ({3}, {4})
True

New in buildout 2.9

ls(*path)
List the contents of a directory. The directory path is provided as one or more strings, to be joined with os.path.join.
mkdir(*path)
Create a directory. The directory path is provided as one or more strings, to be joined with os.path.join.
system(command, input='')
Execute a system command with the given input passed to the command’s standard input. The output (error and regular output combined into a single string) from the command is returned.
read(*path)

Read text from a file at the given path. The file path is provided as one or more strings, to be joined with os.path.join.

If no path is given, the 'out' is used.

New in buildout 2.9

remove(*path)
Remove a directory or file. The path is provided as one or more strings, to be joined with os.path.join.
rmdir(*path)
Remove a directory. The directory path is provided as one or more strings, to be joined with os.path.join.
run_buildout_in_process(command='buildout')

Run Buildout in a multiprocessing.Process. The command is must be a buildout command string, starting with ‘buildout’. You can provide additional arguments, as in 'buildout -v'.

Some extra options are added to the command to prevent network access when running the command. Any distribution the buildout needs must already be available for import. So, for example, if you want to use some recipe, include it in your rest dependencies.

All output from the buildout run is captured in the file named out.

This is useful for integration tests or tests of recipes that interact intimately with buildout or other recipes.

New in buildout 2.9

write(*path_and_contents)
Create a file. The file path is provided as one or more strings, to be joined with os.path.join. The last argument is the file contents.

Documenting your recipe

Please, don’t use your doctests to document your recipe. (We did that a lot and it didn’t turn out well.) Just write straightforward documentation that explains to users how to use your recipe.

If you have examples, however, considering testing them using manuel. You can see examples of how to do that by looking at the source of this topic. Otherwise, it’s very easy to end up with mistakes in your examples.

[1]Configuration data from previous runs are saved in a buildout’s installed database, typically saved in a generated .installed.cfg file.
[2]There are a variety of template recipes that provide different features, like using template files and supporting various template engines. Don’t re-use the example here.
[3]Unfortunately, returning the result of calling created() is boilerplate. Future versions of buildout won’t require this return.
[4]Even though we aren’t distributing the recipe in this example, we still need to create a develop distribution so that Buildout can find the recipe and its meta data.
[5]We’re relying on some refinements made to the helper class in zc.buildout 2.9.