Bootfarm

Executive summary: new hardware for buildfarm at Delft University of Technology has arrived.

In the TraCE project at Utrecht University we had incrementally constructed a buildfarm for continuous integration of Stratego/XT and related software projects. With two Intel-based PCs as the basis complemented with spare machines scavenged from the ST student lab the buildfarm now continuously builds, tests, and releases the software of over 50 projects (packages).

While we started with Merijn de Jonge’s Unix shell-based autobundle software, it soon became clear that dealing with the variability of software building — building against different versions of libraries, with different compilers, on different platforms — required a more scalable solution.

The Nix software deployment system developed by Eelco Dolstra for his PhD thesis turned out to be the perfect tool for constructing a buildfarm. Nix builds software components in isolation and can naturally deal with variability; that is, different versions of the same component can peacefully co-exist without interfering. Furthermore, the functional language that Nix provides for describing component compositions supports the parametric description of a component. That is, the build of a component can be defined as function of it dependencies. Thus, the same function can be used to build different versions of the component.

The Nix buildfarm is basically a script that continuously runs a Nix function to build a software project from the trunk of its version management repository; when a change has been committed or one the dependencies is changed, the project is built again. The result is either a new source and/or binary release in case the build succeeded, or a log of build showing where the build went wrong.

With the move to Delft, a plan emerged to do it bigger and better. That is, to build a mega buildfarm for supporting many different (types of) software projects with integrated support for version management, issue tracking, continuous integration, and release management. I would like to provide such a setup as a service for hosting open source and academic projects. Especially the build and release management aspect of the Nix buildfarm would greatly enhance the software engineering of such projects. (Also of many industrial projects is my impression, but they’d have to pay for the service;) We have submitted a proposal to fund the hardware and humanware to build the foundation of such a megafarm.

In the meantime, it seemed like a good idea to start with a small buildfarm — a bootfarm so to say — in Delft to support at least the activities of the Stratego/XT project, and possibly other projects in department. The Software Technology department generously provided us with resources to set up such a bootfarm, and now the hardware has actually arrived and has been mounted in a very professional looking rack.

Here’s what we have: 5 Intel Core 2 Duo DualCore machines with 1GB RAM , 2 Mac minis with 1,83-GHz Intel Core Duo-processor, another Core 2 Duo a UPS to deal with spikes in power supply, a console with integrated monitor and keyboard switches, a rack with room for a couple more machines .

Here’s what we’re going to do with the goodies. The five Intel machines and the two MacMinis (also Intel) are going to be used to crank at building hundreds of software packages. Using virtualisation we should be able to run builds on multiple operating system distributions. The hardware portfolio is limited to Intel at the moment. This is motivated by the observation that the majority of our userbase is using this platform. With the adoption of Intel by Apple, even the motivation to support the PowerPC is decreasing rapidly.

The webserver will be used to run a number of virtual servers, including a subversion server for version management, a JIRA server for issue tracking, wiki servers for project sites, a mailinglist server, the buildfarm supervisor, and the release management server.