New Fun Blog – Scott Bilas

One of the most important things any studio does is build the game. Compile tools and executables, preprocess resources, pack levels and so on. “Build integrity” is a term I use to cover a set of standards and processes intended to make sure the game build is consistent and reproducible. By this I mean that:

• Build output is independent of environment.
  We get an identical build regardless of the machine, its environment, and the person doing the build. In “its environment” I include things like OS patch level, installed software and configuration, environment variables, and registry settings.
• Build output is independent of time.
  We can reproduce, exactly and simply, builds in any platform and configuration using source materials from any point in time. No special prep must be done to a system, such as downgrading installed software to run an old build.
• Build output can be traced to its source.
  We can reproduce any build given just a few pieces of information about precisely how it was built, which we can encode compactly into the executable and archive folder name. This lets us match up issues filed against a particular build with exactly what versions of source materials went into that build.

Most experienced studios are already doing this, but not all! At Oberon we worked with a lot of other studios and sometimes when we needed to build their game ourselves we’d hear great stuff like “we couldn’t find all the source code”, “this is an old version but it’s probably fine”, and “Bob’s machine did the builds but he got fired and nobody can find his machine”.

I know of one big pro studio that, upon shipping, takes their build server and sticks it in a closet in case they ever need to build the game again. Is that a safety measure? I’d rather trust a solid build process that is regularly in use on well-managed servers than some old machine gathering dust in the closet. Builds frequently end up rising from the dead! Someone always wants to port it to another platform, reissue it in a ‘classics’ with bugfixes, add integration with social networks, or pack it into an OEM build with new languages added. Or even send out a patch years later for paranoid things like Y2K compliance (remember that?).

Backup servers on ice or restored from an image taken at ship time may work, but we can do better.

Why All the Bother?

You could ask why we should bother with defining a lot of process and standards for the game build. Not that it’s a lot of work, though it could be a major change in mindset if you’re used to just having Bob copy builds up from his machine.

Build integrity is important because we practice science in diagnosing and resolving issues in the game. Detached observation, hypotheses, evidence, elimination of variables, reproducibility, all of that. On modern games on platforms with many processors running simultaneously, and a lot of nondeterministic input from multiple players and the network, we can’t afford to mess around with guesses. We must be able to trust the evidence that we base our assumptions on when we start investigating.

The alternative is madness. Seriously, madness. You don’t want to be near someone who has been debugging some insane problem for two days only to find out that they’ve been working with the wrong build. Whether it’s from a bad build number put into the bug report, or the build having been done by someone with hacked local changes, or consistency problems in the depot, this can really turn a normally nice, mellow person into a rage machine. After hours of carefully eliminating all possible variables and laboriously stepping through optimized assembly, having to start completely over is unbelievably frustrating. Most bugs obviously aren’t like this, but the ones that are always seem to show up at the end of a project and threaten ship dates and marriages.

And that’s just the start. There are a lot more things that can suffer from a lack of build integrity, which I’ll be covering along the way in the next post. The bottom line is that without evidence we can trust, we can end up wasting a lot of time in engineering and testing. We want to set up our build process and practices in a way that ensures high build integrity. The build is the most important output of the studio (besides paychecks of course) and it must be protected.

This goes beyond simply archiving the executable and debug images, or having a dedicated build server. It’s a way of life. Luckily, it’s also pretty easy to set up, and has lots of side benefits such as accelerating setting up a new team member or project. The build platform includes the development platform. More on that in the next post.