Reproducible builds

Does your nightly build process change over time? Do you run several different flavors of the same build process in a given week? If either answer is yes, how do you associate a build configuration with a particular build so it is reproducible?

The problem is not trivial. To explain why, I will consider a common build architecture that is simple but flawed. Then I will offer a better approach: a never-changing, universal bootstrapping file that sets up every build and logs its actions.

In the examples that follow, I will use a bash shell look, anticipating that readers will be able to adjust for look and syntax.

First, let's define the components of a build process. Here is a typical breakdown, in order of execution. The italics will serve as a future shorthand reference to the component.

• 1. clean. Remove the last build tree on the build machine.
• 2. log. Capture the particulars of the build, such as time, machine and command line arguments.
• 3. environment. Set up this build's unique environment. For instance, increment the build number and create an include file with "major | minor |patch | build number" versioning strings. Similarly, format specific configuration files suitable for inclusion by make, bash, compilers and the like.
• 4. populate. Get the correct files from source control.
• 5. compile. Invoke a makefile or other build tool to manage the "real work" of compiling.
• 6. kit. Place the output in a kitting area.
• 7. post. Post the results to a public area.
• 8. report. Produce reports.
• 9. notify. Mail notices.

Now, here is my promised example of a simple but flawed architecture. The steps read from left to right.  The first row represents process and the second row, resulting files. Boldface indicates a file is in control.

All build scripts are prerequisite in this model. To start this build, invoke the bash shell script /bldTool/setup.sh with arguments as in this example:

/bldTool/setup.sh sourceDir=3.0/Source buildDir=/bld3.0 versionString=3.0 \

    buildNumber="" postTo=//QAbld/3.0 getOptions=""

The arguments mean:

• 1. sourceDir. Path to the source control tree root (where to get from).
• 2. buildDir. Path to the build directory (where to get to).
• 3. versionString. The "major | minor | patch" number.
• 4. buildNumber. If unset, calculate and use the next sequential build number. Otherwise, honor the historical build number listed.
• 5. postTo. URL path to public posting area.
• 6. getOptions. This is the chance to indicate a branch, stream or label to your source control system. An unset parameter means get the latest source.

Using the arguments passed in, the initial build script accomplishes the first three build steps of, clean, log and environment.  Note the environmental and versioning files produced in/bldDir/include, in the second step above.  It then calls the workhorse file, /bldTool/makefile, which performs the populate and remaining steps. This workhorse file as well as the various makefiles, scripts and compilers it invokes, have been written to read the appropriate /includes/* files. In this way, all ensuing build processes are insured correct and consistent populate operations, output areas and versioning strings

This system uses ever-changing command line arguments to describe an unlimited number of variations of source, destination and versioning. Since we capture the command line invocation in a log file, /bldTool/hist.log, it seems we ought to be able to reproduce any build.

Unfortunately, the prerequisite build files are subject to change themselves. Some changes, like a new report, arguably have little affect on reproducibility. But it's another story with other changes. In many production environments, the populate step changes at least weekly as project components change source and destinations.  And  environmental variables may be changed, added or dropped-perhaps infrequently, but always with major impact. If you keep changing the prerequisite build files, at some point they will no longer be able to produce all current or reproduce all past build variations.

One day, a hot fix will come through the door. The build team will scramble to restore the build machine to a former state. When the crisis blows over, the team will then have to make the machine "current" again. What could they have done differently?

A better way to manage change in build processes is to leverage your star change management tool- your source control system. Simply place your build scripts under source control, at the same level as their associated build source files. Now you are guaranteed a matched set of build tools and sources, for now and any other point in time. This raises a new problem, however. You know that "somewhere out there in source control" is the correct matched set. But at the time you start the build, the essential build scripts files don't exist!

So we introduce a universal bootstrap file, /bldTool/boot1.sh. It lives outside the build directory. It never changes. It makes the most basic assumptions and does the least amount of work necessary to recruit the proper version-controlled build script, pass its command line arguments, and step aside.

Here is this model's sequence in the same format as the last example:

A typical set of command line arguments to the bootstrap file would be similar to the previous example:

/bldTool/boot1.sh \   sourceDir=3.0/Source buildDir=/bld3.0 versionString=3.0 buildNumber="" \    postTo=//QAbld/3.0 getOptions=""

The difference is that the bootstrap file only knows:

• how to clean (in the example, remove the directory indicated in the argument buildDir=/bld3.0)
• how to get the build configuration-specific setup.sh file (in the example, get the latest since getOptions="".)

It is the build configuration-specific setup.sh file that knows the environment and populate steps. It knows the real build tool to call and how to call it. Right now, the build tool may be makefile, but in other build variant past, present or future, it can be something else. That is no problem, since our source control system insures that setup.sh will be a matched set with other build tools.

What if some day a build configuration requires a new variable to be set? It is no problem adding a command line argument to meet the requirement. The bootstrap file /bldTool/boot1.sh consumes only several command line arguments for its clean and "get setup.sh" steps. It passes all command line arguments unchanged to setup.sh. As always, the new requirement is reproducible because the bootstrap file records the exact command line used in the log file hist.log.  

Even as limited as it is, the bootstrap file must make certain major assumptions. It hard-codes the basic source control system and its syntax in order to get the proper setup.sh. In fact, it assumes that all build systems it controls have agreed to provide the file <buildDir>/tools/setup.sh. Such basic assumptions may change over time. You might get a new source control system, for instance. In that case, you have a problem. You cannot change a file that has been promised to remain constant!  The solution is to create a new bootstrap file, /bldTool/boot2.sh. This action allows reproducibility for any build created from either bootstrap file. Just be sure that a bootstrap requirement is recording its own name. This line in a bash script does the job:

echo `date` bootstrap file and command line: $0 $* >> /bldTool/hist.log

Managing build scripts takes little thought as long as one build machine is dedicated to one build variant. But if the scripts change, don't reinvent change management. Use your source control tool to version the bulk of your build process. Designate one file to be the point of entry. Then use a universal bootstrap script that gets the file and puts the build in motion.

Trackback(0)

TrackBack URI for this entry

Comments (0)

Subscribe to this comment's feed

Write comment