Staying DRY with value substitutions, extending, and macros¶
A buildout configuration is a collection of sections, each holding a collection of options. It’s common for option values to be repeated across options. For examples, many file-path options might start with common path prefixes. Configurations that include clients and servers might share server-address options. This topic presents various ways you can reuse option values without repeating yourself.
Value substitutions¶
When supplying values in a configuration, you can include values from other options using the syntax:
${SECTION:OPTION}
For example: ${buildout:directory}
refers to the value of the
directory
option in the in the buildout
section of the
configuration. The value of the referenced option will be substituted
for the referencing text.
You can simplify references to options in the current section by omitting the
section name. If we wanted to use the buildout
directory
option from within the buildout
section itself, we could use
${:directory}
. This convenience is especially useful in
macros, which we’ll discuss later in this topic.
There’s a special value that’s also useful in macros, named
_buildout_section_name_
, which has the name of the current
section. We’ll show how this is used when we discuss macros.
Default and computed option values¶
Many sections have option values that can be used in substitutions without being defined in a configuration.
The buildout
section, where settings for the buildout as a whole
are provided has many default option values. For example, the
directory where scripts are installed is configurable and the value is
available as ${buildout:bin-directory}
. See the Buildout
options reference for a
complete list of Buildout options that can be used in substitutions.
Many recipes also have options that have defaults or that are computed and are available for substitutions.
Sources of configuration options¶
Configuration option values can come from a number of sources (in increasing precedence):
- software default values
- These are defined by buildout and recipe sources.
- user default values
- These are set in per-user default configuration files and override default values.
- options from one or more configuration files
- These override user defaults and each other, as described below.
- option assignments in the buildout command line
- These override configuration-file options.
Extending configuration files¶
The extends option in a buildout
section can be used to extend one or more configuration files. There
are a number of applications for this. For example, common options for
a set of projects might be kept in a common base configuration. A
production buildout could extend a development buildout, or they could
both extend a common base.
The option values in the extending configuration file override those
in the files being extended. If multiple configurations are named in
the extends
option (separated by whitespace), then the
configurations are processed in order from left/top to right/bottom,
with the later (right/bottom) configurations overriding earlier
(left/top) ones. For example, in:
extends = base1.cfg base2.cfg
base3.cfg
The options in the configuration using the extends option override the
options in base3.cfg
, which override the options in base2.cfg
,
which override the options in base1.cfg
.
Base configurations may be extended multiple times. For example, in
the example above, base1.cfg
might, itself, extend base3.cfg
,
or they might both extend a common base configuration. Of course, cycles
are not allowed.
Configurations may be named with URLs in the extends
option, in
which case they may be downloaded from remote servers. See The
extends-cache buildout option.
When a relative path is used in an extends option, it’s interpreted relative to the path of the extending configuration.
Conditional configuration sections¶
Sometimes, you need different configuration in different environments (different operating systems, or different versions of Python). To make this easier, you can define environment-specific options by providing conditional sections:
[ctl]
suffix =
[ctl:windows]
suffix = .bat
In this tiny example, we’ve defined a ctl:suffix
option that’s
.bat
on Windows and an empty string elsewhere.
A conditional section has a colon and then a Python expression after the name. If the Python expression result is true, the section options from the section are included. If the value is false, the section is ignored.
Some things to note:
- If there is no exception, then options from the section are included.
- Sections and options can be repeated. If an option is repeated, the
last value is used. In the example above, on Windows, the second
suffix
option overrides the first. If the order of the sections was reversed, the conditional section would have no effect.
In addition to the normal built-ins, the expression has access to global variables that make common cases short and descriptive as shown below
Name | Value |
---|---|
sys | sys module |
os | os module |
platform | platform module |
re | re module |
python2 | True if running Python 2 |
python3 | True if running Python 3 |
python26 | True if running Python 2.6 |
python27 | True if running Python 2.7 |
python32 | True if running Python 3.2 |
python33 | True if running Python 3.3 |
python34 | True if running Python 3.4 |
python35 | True if running Python 3.5 |
python36 | True if running Python 3.6 |
python37 | True if running Python 3.7 |
python38 | True if running Python 3.8 |
python39 | True if running Python 3.9 |
python310 | True if running Python 3.10 |
sys_version | sys.version.lower() |
pypy | True if running PyPy |
jython | True if running Jython |
iron | True if running Iron Python |
cpython | True if not running PyPy, Jython, or Iron Python |
sys_platform | str(sys.platform).lower() |
linux | True if running on Linux |
windows | True if running on Windows |
cygwin | True if running on Cygwin |
solaris | True if running on Solaris |
macosx | True if running on Mac OS X |
posix | True if running on a POSIX-compatible system |
bits32 | True if running on a 32-bit system. |
bits64 | True if running on a 64-bit system. |
little_endian | True if running on a little-endian system |
big_endian | True if running on a big-endian system |
Expressions must not contain either the #
or the ;
character.
User-default configuration¶
A per-user default configuration may be defined in the default.cfg
file in the .buildout
subdirectory of a user’s home directory
(~/.buildout/default.cfg
on Mac OS and Linux). This configuration
is typically used to set up a shared egg or cache directory, as in:
[buildout]
eggs-directory = ~/.buildout/eggs
download-cache = ~/.buildout/download-cache
abi-tag-eggs = true
See the section on optimizing buildouts with shared eggs and download caches for an explanation of the options used in the example above.
Merging, rather than overriding values¶
Normally, values in extending configurations override values in
extended configurations by replacing them, but it’s also possible to
augment or trim overridden values. If +=
is used rather than
=
, the overriding option value is appended to the original. So,
for example if we have a base configuration, buildout.cfg
:
[buildout]
parts =
py
test
server
...
And a production configuration prod.cfg
, we can add another part,
monitor
, like this:
[buildout]
extends = buildout.cfg
parts += monitor
...
In this example, we didn’t have to repeat (or necessarily know) the
base parts to add the monitor
part.
We can also subtract values using -=
, so if we wanted to exclude
the test
part in production:
[buildout]
extends = buildout.cfg
parts += monitor
parts -= test
...
Something to keep in mind is that this works by lines. The +=
form adds the lines in the new data to the lines of the
old. Similarly, -=
removes lines in the overriding option from the
original lines. This is a bit delicate. In the example above,
we were careful to put the base values on separate lines, in
anticipation of using -=
.
Merging values also works with option assignments provided via the buildout command line. For example, if you want to temporarily use a development version of another project, you can augment the buildout develop option on the command-line when running buildout:
buildout develop+=/path/to/other/project
Although, if you’ve pinned the version of that project, you’ll need to unpin it, which you can also do on the command-line:
buildout develop+=/path/to/other/project versions:projectname=
Extending sections using macros¶
We can extend other sections in a configuration as macros by naming
then using the <
option. For example, perhaps we have to create
multiple server processes that listen on different ports. We might
have a base server
section, and some sections that use it as a
macro:
[server]
recipe = zc.zdaemonrecipe
port = 8080
program =
${buildout:bin-directory}/serve
--port ${:port}
--name ${:_buildout_section_name_}
[server1]
<= server
port = 8081
[server2]
<= server
port = 8082
In the example above, the server1
and server2
sections use the
server
section, getting its recipe
and program
options.
The resulting configuration is equivalent to:
[server]
recipe = zc.zdaemonrecipe
port = 8080
program =
${buildout:bin-directory}/serve
--port ${:port}
--name ${:_buildout_section_name_}
[server1]
recipe = zc.zdaemonrecipe
port = 8081
program =
${buildout:bin-directory}/serve
--port ${:port}
--name ${:_buildout_section_name_}
[server2]
recipe = zc.zdaemonrecipe
port = 8082
program =
${buildout:bin-directory}/serve
--port ${:port}
--name ${:_buildout_section_name_}
Value substitutions in the base section are applied after its
application as a macro, so the substitutions are applied using data
from the sections that used the macro (using the <
option).
You can extend multiple sections by listing them in the <
option
on separate lines, as in:
[server2]
<= server
monitored
port = 8082
If multiple sections are extended, they’re processed in order, with
later ones taking precedence. In the example above, if both
server
and monitored
provided an option, then the value from
monitored
would be used.
A section that’s used as a macro can extend another section.