Getting started with Buildout¶
Note
In the Buildout documentation, we’ll use the word buildout to refer to:
The Buildout software
We’ll capitalize the word when we do this.
A particular use of Buildout, a directory having a Buildout configuration file.
We’ll use lower case to refer to these.
A
buildout
section in a Buildout configuration (in a particular buildout).We’ll use a lowercase fixed-width font for these.
First steps¶
The recommended way to install Buildout is to use pip within a virtual environment:
virtualenv mybuildout
cd mybuildout
bin/pip install zc.buildout
To use Buildout, you need to provide a Buildout configuration. Here is a minimal configuration:
[buildout]
parts =
By default, Buildout looks for a file named buildout.cfg
to find its configuration.
The configuration hereabove is thus stored in buildout.cfg
.
A minimal (and useless) Buildout configuration has a buildout
section
with a parts
option. If we run Buildout:
buildout
Four directories are created:
- bin
- A directory to hold executables.
- develop-eggs
- A directory to hold develop egg links. More about these later.
- eggs
- A directory that hold installed packages in egg [1] format.
- parts
- A directory that provides a default location for installed parts.
Buildout configuration files use an INI syntax [2]. Configuration is arranged in sections, beginning with section names in square brackets. Section options are names, followed by equal signs, followed by values. Values may be continued over multiple lines as long as the continuation lines start with whitespace.
Buildout is all about building things and the things to be built are
specified using parts. The parts to be built are listed in the
parts
option. For each part, there must be a section with the same
name that specifies the software to build the part and provides
parameters to control how the part is built.
Installing software¶
In this tutorial, we’re going to install a simple web server. The details of the server aren’t important. It just provides a useful example that illustrates a number of ways that Buildout can make things easier.
We’ll start by adding a part to install the server software. We’ll
update our Buildout configuration to add a bobo
part:
[buildout]
parts = bobo
[bobo]
recipe = zc.recipe.egg
eggs = bobo
We added the part name, bobo
to the parts
option in the
buildout
section. We also added a bobo
section with two
options:
- recipe
- The standard
recipe
option names the software component that will implement the part. The value is a Python distribution requirement, as would be used withpip
. In this case, we’ve specified zc.recipe.egg which is the name of a Python project that provides a number of recipe implementations. - eggs
- A list of distribution requirements, one per
line. [3] (The name of this option is
unfortunate, because the values are requirements, not egg names.)
Listed requirements are installed, along with their dependencies. In
addition, any scripts provided by the listed requirements (but not
their dependencies) are installed in the
bin
directory.
If we run this:
buildout
Then a number of things will happen:
zc.recipe.egg
will be downloaded and installed in youreggs
directory.bobo
and its dependencies will be downloaded and installed. (bobo is a small Python database server.)After this, the eggs directory will look something like:
$ ls -l eggs total 0 drwxr-xr-x 4 jim staff 136 Feb 23 09:01 WebOb-1.7.1-py2.7.egg drwxr-xr-x 9 jim staff 306 Feb 23 09:10 bobo-2.3.0-py2.7.egg
A
bobo
script will be installed in thebin
directory:$ ls -l bin total 8 -rwxr-xr-x 1 jim staff 391 Feb 23 09:10 bobo
This script is used to run a bobo server.
Generating configuration and custom scripts¶
The bobo
program doesn’t daemonize itself. Rather, it’s meant to
be used with a dedicated daemonizer like zdaemon or supervisord. We’ll use a recipe to set up zdaemon. Our Buildout
configuration becomes:
[buildout]
parts = bobo server
[bobo]
recipe = zc.recipe.egg
eggs = bobo
[server]
recipe = zc.zdaemonrecipe
program =
${buildout:bin-directory}/bobo
--static /=${buildout:directory}
--port 8200
Here we’ve added a new server
part that uses zc.zdaemonrecipe
.
We used a program
option to define what program should be run.
There are a couple of interesting things to note about this option:
We used variable substitutions:
${buildout:directory}
Expands to the full path of the buildout directory.
${buildout:bin-directory}
Expands to the full path of the buildout’s
bin
directory.
Variable substitution provides a way to access Buildout settings and share information between parts and avoid repetition.
See the reference to see what buildout settings are available.
We spread the program over multiple lines. A configuration value can be spread over multiple lines as long as the continuation lines begin with whitespace.
The interpretation of a value is up to the recipe that uses it. The
zc.zdaemonrecipe
recipe combines the program value into a single line.
If we run Buildout:
buildout
The
zc.zdaemonrecipe
recipe will be downloaded and installed in theeggs
directory.A
server
script is added to thebin
directory. This script is generated by the recipe. It can be run like:bin/server start
to start a server and:
bin/server stop
to stop it. The script references a zdaemon configuration file generated by the recipe in
parts/server/zdaemon.conf
.A zdaemon configuration script is generated in
parts/server/zdaemon.conf
that looks something like:<runner> daemon on directory /Users/jim/t/0214/parts/server program /Users/jim/t/0214/bin/bobo --static /=/Users/jim/t/0214 --port 8200 socket-name /Users/jim/t/0214/parts/server/zdaemon.sock transcript /Users/jim/t/0214/parts/server/transcript.log </runner> <eventlog> <logfile> path /Users/jim/t/0214/parts/server/transcript.log </logfile> </eventlog>
The details aren’t important, other than the fact that the configuration file reflects part options and the actual buildout location.
Version control¶
In this example, the only file that needs to be checked into version
control is the configuration file, buildout.cfg
. Everything else
is generated. Someone else could check out the project, and get the
same result [4].
More than just a package installer¶
The example shown above illustrates how Buildout is more than just a
package installer such as pip
. Using Buildout recipes, we can
install custom scripts and configuration files, and much more. For
example, we could use configure and make to install non-Python
software from source, we could run JavaScript builders, or do anything
else that can be automated with Python.
Buildout is a simple automation framework. There are hundreds of recipes to choose from [5] and writing new recipes is easy.
Repeatability¶
A major goal of Buildout is to provide repeatability. But what does this mean exactly?
If two buildouts with the same configuration are built in the same environments at the same time, they should produce the same result, regardless of their build history.
That definition is rather dense. Let’s look at the pieces:
Buildout environment¶
A Buildout environment includes the operating system and the Python installation Buildout is run with. The more a buildout depends on its environment, the more variation is likely between builds.
If a Python installation is shared, packages installed by one application affect other applications, including buildouts. This can lead to unexpected errors. This is why it’s recommended to use a virtual environment or a “clean python” built from source with no third-party packages installed [6].
To limit dependence on the operating system, people sometimes install libraries or even database servers as Buildout parts.
Modern Linux container technology (e.g. Docker) makes it a lot easier to control the environment. If you develop entirely with respect to a particular container image, you can have repeatability with respect to that image, which is usually good enough because the environment, defined by the image, is itself repeatable and unshared with other applications.
Python requirement versions¶
Another potential source of variation is the versions of Python dependencies used.
Newest versions¶
If you don’t specify versions, Buildout will always try to get the most recent version of everything it installs. This is a major reason that Buildout can be slow. It checks for new versions every time it runs. It does this to satisfy the repeatability requirement above. If it didn’t do this, then an older buildout would likely have different versions of Python packages than newer buildouts.
To speed things up, you can use the -N
Buildout option to tell
Buildout to not check for newer versions of Python requirements:
buildout -N
This relaxes repeatability, but with little risk if there was a recent run without this option.
Pinned versions¶
You can also pin required versions, and do so in two ways. You can specify them where you list them, as in:
[bobo]
recipe = zc.recipe.egg
eggs = bobo <5.0
In this example, we’ve requested a version of bobo less than 5.0.
You can also add environment markers to restrict some requirements to i.e. a certain platform or python version:
[bobo]
recipe = zc.recipe.egg
eggs =
bobo ==2.2.0; python_version < '3.0'
bobo ==2.3.0; python_version >= '3.0'
The more common way to pin a version is using a versions
section:
[buildout]
parts = bobo server
[bobo]
recipe = zc.recipe.egg
eggs = bobo
[server]
recipe = zc.zdaemonrecipe
program =
${buildout:bin-directory}/bobo
--static /=${buildout:directory}
--port 8200
[versions]
bobo = 2.3.0
Larger projects may need to pin many versions, so it’s common to put version requirements in their own file:
[buildout]
extends = versions.cfg
parts = bobo server
[bobo]
recipe = zc.recipe.egg
eggs = bobo
[server]
recipe = zc.zdaemonrecipe
program =
${buildout:bin-directory}/bobo
--static /=${buildout:directory}
--port 8200
Here, we’ve used the Buildout extends
option to say that
configurations should be read from versions.cfg
(it’s
allowed to specify several files in extend
). Cconfigurations
should be read from the named file (or files) and the
configuration in the current file should override configuration in the
extended files.
To continue the example, our versions.cfg
file
might look like:
[versions]
bobo = 2.3.0
We can use the update-versions-file
option to ask Buildout to
maintain our versions.cfg
file for us:
[buildout]
extends = versions.cfg
show-picked-versions = true
update-versions-file = versions.cfg
parts = bobo server
[bobo]
recipe = zc.recipe.egg
eggs = bobo
[server]
recipe = zc.zdaemonrecipe
program =
${buildout:bin-directory}/bobo
--static /=${buildout:directory}
--port 8200
With update-versions-file
, whenever Buildout gets the newest
version for a requirement (subject to requirement constraints), it
appends the version to the named file, along with a comment saying
when and why the requirement is installed. If you later want to
upgrade a dependency, edi this file with the new version. Alternatively,
remove the entry altogether and Buildout will add a new entry the next
time it runs.
We also used the show-picked-versions
to tell Buildout to tell us
when it got (picked) the newest version of a requirement.
When versions are pinned, Buildout doesn’t look for new versions of
the requirements, which can speed buildouts quite a bit. In fact, The
-N
option doesn’t provide any speedup for projects whose
requirement versions are all pinned.
When should you pin versions?¶
The rule of thumb is that you should pin versions for a whole system, such as an application or service. You do this because after integration tests, you want to be sure that you can reproduce the tested configuration.
You shouldn’t pin versions for a component, such as a library, because doing so inhibits the ability of users of your component to integrate it with their dependencies, which may overlap with yours. If you know that your component only works for a specific range of versions of some dependency, set the range in your project requirements. Don’t require specific versions.
Unpinning versions¶
You can unpin a version by just removing it (or commenting it out of)
your versions
section.
You can also unpin a version by setting the version to an empty string:
[versions]
ZEO =
In an extending configuration (buildout.cfg
in the example above), or
on the buildout command line.
You might do this if pins are shared between projects and you want to unpin a requirement for one of the projects, or want to remove a pin while using a requirement in development mode.
Buildout versions and automatic upgrade¶
In the interest of repeatability, Buildout can upgrade itself or its
dependencies to use the newest versions or downgrade to respect pinned
versions. This only happens if you run Buildout from a buildout’s own
bin
directory.
We can use Buildout’s bootstrap
command to install a local
buildout script:
buildout bootstrap
Then, if the installed script is used:
bin/buildout
Then Buildout will upgrade or downgrade to be consistent with version requirements. See the bootstrapping topic to learn more about bootstrapping.
Python development projects¶
A very common Buildout use case is to manage the development of a
library or main part of an application written in Python. Buildout
facilitates this with the develop
option:
[buildout]
develop = .
...
The develop
option takes one or more paths to project setup.py files or,
more commonly, directories containing them. Buildout then creates
“develop eggs” [7] for the corresponding projects.
With develop eggs, you can modify the sources and the modified sources are reflected in future Python runs (or after reloads).
For libraries that you plan to distribute using the Python packaging infrastructure, You’ll need to write a setup file, because it’s needed to generate a distribution.
If you’re writing an application that won’t be distributed as a separate Python distribution, writing a setup script can feel like overkill, but it’s useful for:
- naming your project, so you can refer to it like any Python requirement in your Buildout configuration, and for
- specifying the requirements your application code uses, separate from requirements your buildout might have.
Fortunately, an application setup script can be minimal. Here’s an example:
from setuptools import setup
setup(name='main', py_modules='main', install_requires = ['bobo', 'six'])
We suggest copying and modifying the example above, using it as boilerplate. As is probably clear, the setup arguments used are:
- name
- The name of your application. This is the name you’ll use in Buildout configuration where you want to refer to application code.
- install_requires
- A list of requirement strings for Python distributions your application depends on directly.
A minimal [8] development Buildout configuration for a project with a setup script like the one above might look something like this:
[buildout]
develop = .
parts = py
[py]
recipe = zc.recipe.egg
eggs = main
interpreter = py
There’s a new option, interpreter
, which names an interpreter
script to be generated. An interpreter script [9]
mimics a Python interpreter with its path set to include the
requirements specified in the eggs option and their (transitive)
dependencies. We can run the interpreter:
bin/py
To get an interactive Python prompt, or you can run a script with it:
bin/py somescript.py
If you need to work on multiple interdependent projects at the same
time, you can name multiple directories in the develop
option,
typically pointing to multiple check outs. A popular Buildout
extension, mr.developer,
automates this process.
Where to go from here?¶
This depends on what you want to do. We suggest perusing the topics based on your needs and interest.
The reference section can give you important details, as well as let you know about features not touched on here.
[1] | You may have heard bad things about eggs. This stems in part from the way that eggs were applied to regular Python installs. We think eggs, which were inspired by jar files, when used as an installation format, are a good fit for Buildout’s goals. Learn more in the topic on Buildout and packaging. |
[2] | Buildout uses a variation (fork) of the standard
ConfigParser module and follows (mostly) the same parsing
rules. |
[3] | Requirements can have whitespace
characters as in bobo <3 , so they’re separated by newlines. |
[4] | This assumes the same environment and that dependencies haven’t changed. We’ll explain further in the section on repeatability. |
[5] | You can list Buildout-related software, consisting mostly of Buildout recipes, using the Framework :: Buildout classifier search. These results miss recipes that don’t provide classifier meta data. Generally you can find a recipe for a task by searching the name of the task and the “recipe” in the package index. |
[6] | It’s a little hypocritical to recommend installing Buildout into an otherwise clean environment, which is why Buildout provides a bootstrapping mechanism which allows setting up a buildout without having to contaminate a virtual environment or clean Python install.) |
[7] | pip calls these “editable” installs. |
[8] | A more typical development buildout will include at least a part to specify a test runner. A development buildout might define other support parts, like JavaScript builders, database servers, development web-servers and so on. |
[9] | An interpreter script is similar to the
bin/python program included in a virtual environment, except
that it’s lighter weight and has exactly the packages
listed in the eggs option and their dependencies, plus whatever
comes from the Python environment. |