We Moved to Perforce!
I just migrated the team over from Subversion (svn) to Perforce (p4) at work. We’ve been dying for this change for a while and have been holding off until the timing was right. This is the third p4 setup I’ve done. The first was at Gas Powered Games where we switched from SourceSafe (and had a party to shred the VSS discs when it was done), and the second was at Oberon where we started fresh using p4 and were generally happy with the setup from the start to the end.
In preparation for the Loose Cannon migration, I was using svn2p4.pl from the Perforce Public Depot to keep the server up to date with our Subversion server in parallel, for about a week. The script isn’t perfect, so if you use it, watch out. It doesn’t deal with p4-illegal filenames well. For example we have a certain console manufacturer’s SDK checked in that has filenames like @file@ in it, and those caused errors to the script, making me skip some revisions. P4 also can’t do filename case changes on a Windows server without going through an intermediate step first (so filename.txt -> filename.txt2 -> FileName.TXT) and the script didn’t have a handler for this. We had a few of these in our revision history that had to be skipped as well.
The initial import took a day solid, with me making adjustments to start it up again when it broke, and then just a few minutes here and there to keep it in sync while we counted down to the big migration day. That day (which was Thursday) went pretty well. We decided to take advantage of the chaos to reorg the whole source tree, which I’ll talk about in a later posting on Perforce “generated” clients. Doing the final import, shutting down svn, reorganizing the tree, updating all the build scripts, etc., took about 6 hours. Switching the team over the next morning took a couple hours, and then a couple more to fix up all the little issues that popped up.
In this post I want to do a little post-mortem of our Subversion experience. Overall it was a 4 on a 10 scale. Not fantastically awful, but not great either. I got to know svn pretty well while we were on it, and I still like it a little – I’m continuing to use it at home for some side projects.
Subversion Considered Slightly Harmful
Svn is pretty easy to set up. As long as you use a good client it’s great for bootstrapping a company that has a tight budget. Perforce is insanely expensive and that’s hard to swallow on Day 1. That money does have a lot of buying power – they have some of the best support people in the world (Audiokinetic is the only other company I’ve worked with that has support people of that caliber) but I wish they had a middle ground where you get the server and client licenses but without the support contract, say for half price.
I know so many smaller developers that have wanted to use p4 pretty badly but have been forced to go with something else due to the price. I’d go with svn again to get a team started if I needed to, but not keep it a second longer than necessary. It falls down fairly quickly as you scale up in project size and people, start checking in 50 meg Maya files, etc.
I’ll go over the top three key problems we ran into, in order of importance. Svn and its tools are missing a lot of important features that p4 has, but we could live without them, find workarounds, or create our own tools to get us 80% of the way there. The problems listed below are the simply intolerable ones.
Svn is slow, slow, slow, due to the client-managed state. We often found our hard drives locking up solid with activity because svn was scanning and running checksums on every single file in the tree. It can’t detect if a file has changed otherwise. This is where p4 gets its speed – the server manages the state of every client. Modern svn clients like SmartSVN do things like hooking notifies from the system when folders are modified, so after the initial scan is done they can very quickly show changes without needing to do a full tree rescan. But restart the app, reboot, etc., and you get to run that insane scan all over again. This drove us nuts, especially when trying to ship a milestone.
The downside is of course that p4 clients have to promise not to make changes without telling the server. This makes working offline a little more of a pain because you have to mark files read/write and then either remember to check out/check in those files on reconnecting to the LAN, or to run one of the “check consistency” tools in P4Win P4V. The consistency check tools are essentially the same as what svn does.
I’ll take the p4 way any day over svn, of course. Just buy a fat-ass server, throw 8 gigs of memory in it, and let it keep all the metadata in its system cache. Full depot syncs on p4 are instant if nothing has changed.
Just to throw it out there, SourceSafe does what svn does on both the client and the server, because there really is no server. All your clients are doing checksums on all their local files, on top of scanning the server’s files to run checksums also. So now your network is saturated with unnecessary file transfers too. At GPG, when trying to ship Dungeon Siege, this was so bad for us that most of our network was unusable in the first few hours of the morning when people would come in and start their daily sync. Switching to p4 totally eliminated that problem, and we could sync the full depot as often as we liked. Then we got Incredibuild, which also saturated the network 100%, but that’s another story.
Lack of Branching
There’s a svnmerge.py that does per-changelist integrations. Supposedly the next release of svn will make this pattern officially supported instead of an add-on, but it’s still not very good. P4 is well-known for its integration support, so it doesn’t surprise me that svn falls down here, as most rcs’s probably do. But I am disappointed they are not shooting higher than svnmerge.py for the future.
I tried really hard to get private branches going with svn and svnmerge.py. There are just too many steps required to set up and maintain a branch, and a lot of stability problems. I almost spent more time maintaining the damn branch than I did working in it. Bouncing changes back and forth across them is poorly supported, and only on a per-revision level. P4 supports per-file integrations which are absolutely key for private branches to work. Integrations must also be quick, easy, and safe, and I got none of that from using svn.
By the way, if you don’t know what I mean by private branches, I’m referring to a special branch made for an individual or very small group, either for full-time work for created on-demand for a specific task. Some companies like Valve use full-time private branches for their engineers, where people always work in their branch and then integrate back and forth to keep in sync with the mainline. Personally I don’t like this idea, and prefer to use private branches only for individual tasks. Even though p4 is fast with integrations, it saves a lot of time to always work on the mainline and only run off to a private branch for big changes. If engineers breaking content development from frequent engine changes is a problem, then I’d recommend splitting the mainline into a “live” engine separate from the codebase and dependent tools. I’ll talk about this in a future post on our p4 setup.
Lack of Stability
This was the source of a lot of loud cursing on our team and throwing things. If svn is slow, you just work on something else while it’s screwing around. But if it fails and causes local state corruption, you have serious problems. One engineer on our team would have to use the svn cleanup every time certain files were updated during a sync, probably due to the whole thing getting wedged if a file was in use during the sync. Another spent hours trying to resolve svn errors that made no sense until he just nuked the whole folder and synced it fresh. We had a number of stories like this. I ran into a lot of these issues myself, especially when attempting to use private branches.
My guess is that the .svn folders are the problem. All that state, distributed over the local client’s project tree. Lots can go wrong there. It’s old school file-system-as-database tricks that nobody does any more. The Microsoft world learned its lesson on file system-based databases with the cthulu known as SourceSafe. I’m shocked that something as new as svn would continue to use stone-age tricks like this instead of a real database. Ever copy-pasted a folder managed by svn on a computer by accident? It copies all those .svn subfolders along with it, which then totally confuses the tools. Awful.
Svn also has big problems using multiple tools at once. If you like Tortoise’s integration with Explorer but you also want SmartSVN for its speed and usability, forget it. They will fight with each other and corrupt the state. And if you have any tools that automatically do things with svn then you have to freeze until it’s done. Even an auto-update on SmartSVN will potentially wedge your tree and require a cleanup. For example, our build script sets the game’s build number by asking svn for the revision number the local client is sync’d to.
Subversion Considered Not Too Bad
Like I said above, I still use svn at home because I do actually like a few things about it. Here are the top couple on my mind as we move to p4 at Loose Cannon.
I know that a quick search of the web comes up with tons of praise for TortoiseSVN, but I strongly disagree, especially after using SmartSVN. Tortoise is lousy from a usability standpoint, taking far too many actions to get even basic things done. I am far faster and more accurate using SmartSVN. It does a lot of things right, such as:
- Adding a p4-style pending changelist feature to svn, which was sorely lacking. It’s client-only so you can’t see anyone else’s changelists, but it’s 80% of the way there, which is good enough.
- Automatically syncing your state versus the server upon committing a folder. It scans for adds, deletes, and changes, and can make intelligent guesses about when you’ve renamed a file. Very smart. One ‘commit’ is all you need to do. This is even better than p4, and I love it. I never forget to add/remove files this way because SmartSVN always notices.
- Maintaining a constantly updated list of differences from the depot. It does this by scanning your whole tree and then getting notified by the system of changes. So you alt-tab from your editor or export tool or whatever to SmartSVN and a second later its view is updated with all the files on the depot that have changed (that you’d be fetching if you did a sync) and the changes you’ve made locally. It shows new/deleted files, changed files locally or on the server, etc. It has a column for local state and another for remote state so you can see clearly where the differences are. There are filter buttons to show/hide remote changes, go recursive from the current folder, etc. Awesome.
- This is, ironically, exactly what I was complaining about above that costs so much performance on startup with the insane hard drive scanning. It’s not worth the cost, especially as we scale up the project tree, but this is the #1 thing that I will really miss now that we’re on p4. Maybe I have Stockholm syndrome.
- I’m tempted to almost write my own tool as a supplement to p4win/v to do this, one that works quietly in the background instead of burning the hard drive up constantly.
- Automatically requiring exclusive locks on certain file types. This is client-side, again, but a good emulation of what p4 does. Certain files, particularly binary files, should be exclusive-lock. SmartSVN can use wildcards to specify those types, and it marks them read-only until you grab an exclusive lock on them. This is basically the p4 workflow mapped onto svn.
- Normally svn lets you change whatever you want and deal with merge issues at checkin time. This can be really awful when making complex changes to binary file types (such as a Flash file) and then discovering on checkin that someone else has also modified it and having to lose all your changes. Enforcing a required-lock is perfect to work around this.
In short, if you are using svn, you should be using SmartSVN as your client. It’s free! And if you want some of the advanced features like the ones I wrote about above, then it’s very cheap to buy, and absolutely worth it. SmartSVN is written in Java which I’d normally hate, but it’s a really decent native-feeling Windows app. And the Java makes it cross-platform. I’m sure the experience on Mac and Linux is good as well.
Svn lets you attach arbitrary name/value pairs to folders, which get checked into the server and sync’d on all machines just like files do. Very nice, can be used by tools to store simple state. In fact, svnmerge.py uses properties to keep track of what revisions have been merged and skipped. The alternative is to have specially named files that get checked in. Using properties is a lot cleaner. The ‘p4 counters’ command is incredibly weak in comparison to this.
Properties are also used in ways I don’t really like. For example, if you want to set the file type on your source files to enable keyword substitution. P4 does this through a server-wide typemap that applies default file types using patterns. Svn distributes these properties through its entire tree, and they are not recursive. I definitely prefer server-wide specifications, though this is decidedly less democratic (only superusers can typically modify the typemap).
One very useful property type in svn is “ignore”. This basically tells the client to pretend that files and folders matching that pattern do not exist. Very nice for masking out all those local-only folders that get generated like obj, .cache, or .mayaswatches. This is on top of the client-side “global ignores” that can mask out things like Visual Studio’s .suo, .ncb, .user. P4 has something similar – you can mask out folders using your client spec, or the permissions spec, but that only hides the files from view on P4Win. On P4V they always show (I currently have an outstanding feature request to have this fixed up).
I would strongly recommend avoiding svn for game development on teams larger than perhaps 15 people, or with client trees bigger than a few gigs. It’s “free” but there is a significant cost to this free-ness that increases as months go by and the team grows and the database gets larger.
We’re already super happy with the move. Things are so fast now! Plus I can start writing tools that depend on some of p4’s more advanced features. More on this later.