Quickie Detour: GEAR4 BlackBox Mini

Keeping Mobile

Ally and I are going to be in Peru for six months next year. We want to be pretty mobile so that means two things: we need to take a minimum of stuff, and what we do take should be as small and light as possible. By “minimum” I mean that we should only take down what we can’t find there easily. Electronics tend to be insanely marked up in foreign countries over what we pay in the US, due to duties and such, so it makes sense to take those with us.

So, I’ve been thinking a lot about electronics lately. Backup hardware, backup hard drives, encryption in case anything is stolen, etc. I’ll power or charge everything on USB that I can, because USB bricks are easy to find and not too expensive. I have little retractable cables for charging my iPod, PSP, DS, and phone using USB. We’re also taking my friend Joel’s Wii down because it’s so portable, and using a USB video capture card to use a notebook computer screen in place of the TV. All of this plus three notebook computers (one for her, one for me, one for a backup) and a couple cameras should all fit within a small backpack.

Now, the three computers sure sounds like overkill but I want to get paid while I’m down there! I need to make sure I have a backup computer to work on if anything goes wrong with my old Vaio. And I already have a Band-Aid keeping it together right now. Plus, the third notebook is something I happen to have laying around from years ago that I currently just use for hosting Firefly and Bittorrent.

But anyway, back to Peru. Who can go without music playing while they’re working?? Earphones get tiring after a while and it’s no fun when Wii-boxing! Ally’s computer has better speakers than mine, but they’re still too tiny to sound decent.

The BlackBox Mini

Well, I just got a fantastic toy in the mail today that I want to recommend because I’m a little in love with it. Even caused me to want to write a blog post about it. It’s the GEAR4 BlackBox Mini. I had to import it from the UK using Expansys, which took a while, but it was worth the wait. I forget where I got the recommendation, but I did a bunch of research on Google before settling on this guy.

black_box_mini

The BlackBox Mini is a little stereo speaker box that looks like a really big black shiny stick of butter. Each speaker is at one end of the stick, which gives pretty good stereo audio separation. It’s powered by either 4 AA’s or USB, and it runs on either Bluetooth using A2DP or a direct minijack plugin if you want to run it on your iPod or whatever. It gets plenty loud either way you power it.

Getting it to work with Vista was pretty easy. Vista only initially saw it as a mono audio device, claiming it couldn’t find the right drivers for audio. But when I went to the device manager and manually told it to auto-update the drivers for the unknown Bluetooth devices, it magically found them and everything worked. Weird, but whatever.

Now, I don’t want to set it as the default system output device but lots of apps (like Winamp) let you choose a specific output device, which works great. I have to stress that the audio quality is truly excellent for something of this size. An order of magnitude better than our little built-in notebook speakers or any of the battery-powered portable speakers I’ve used in the past. This thing could do pretty well even at house parties. It sounds far bigger than it is.

The BlackBox Mini even comes with a nice little carrying case. It’s a bit stretchy, which is great because you can pack the little minijack cable and the USB cable in there easily. I also tore out the three pages of the manual that I might need one day and stuck it in the case too.

This thing will be great for car camping, staying at a cabin, or renting apartments in Peru. We don’t have any room to bring real speakers with us, and our notebooks have (relative) crap for speakers. But I really like what BlackBox has been making. Lots of power in a small package!

As I write this, I noticed that they just came out with a Micro version for half the price. This looks a little too small for me: no stereo, apparently, although it may have better bass than the Mini. I prefer stereo over bass. Although the price is something I would have definitely considered. The Mini set me back $75 unfortunately, which is pretty high.

But overall, a huge thumbs up. I’m excited to use this thing down there!

Comments (1)

Buried In Data

It would be an obvious understatement to say that modern game development has a lot of data.

Our hard drives fill up with all kinds of bits: tools, source code, code reviews, source and object assets, bug reports, concept art, docs, email, game builds and debug data, CD images, gameplay analysis data, SDK’s, error logs, and on and on. New types every day. Some you have initially, and some you add over the course of development. It all depends on the type and size of game.

In this next series of posts I am going to focus primarily on code and assets, and the tools and processes related to producing and managing them. That’s still a lot of data! During development, new classifications of data come up all the time, and we need to be able to answer the question of “ok great, now where does that stuff go?”

For example, let’s say that the powers-that-be decide that we’re going to add an hour of video cut-scenes to the game. This means a lot of new source assets need to be created, and then some truly huge output video files (made much worse if multiple platforms are involved).

Should everything go into revision control? Or get stored on a server file share? What about just keeping it on the video editing machine? The decision could have serious implications to team productivity, server space management, complaints from the IT department, automated builds and testing, and so on. So I want to provide some simple rules to help decide where data can and should go.

I originally was going to talk just about Perforce but in starting to write this article I realized I needed to back up a bit and talk about how we decide what even goes in revision control in the first place. In later posts I’ll talk about the particular environment we have at Loose Cannon. Specific types of data that flows around our network and how we organize it.

Comments

Upcoming Articles

I’ve been trying to get the time to do a series of posts and I think I’m finally ready to get started. The high-level theme is “Loose Cannon’s workflow”. Like any mid-size team working on a multi-year title, we’ve built up a large set of processes and tools to support development. Some are very carefully planned and based on past experience, and that’s where I want to start first: talking about mature process and tools that I know work well. Well, at least for the types and sizes of teams I tend to work on.

Note that I’m only going to talk about the areas that I’ve either designed or been directly involved in designing. We have a large infrastructure for building and managing assets that I think works pretty well. Maya, exporting, plugins, that sort of thing. But I’m not too familiar with all that. At least not yet - looks like my task while in Peru will very much involve our content pipeline.

So anyway, within this theme, the topic for the next set of articles is how we use Perforce at LCS. Which means this series will be fly-over territory for most people Google sends here. I’ve only met a few people in my life that have actually been interested in things like depot design and the workflow around it. Well, I really love this stuff! Helping the team become more efficient and accurate is something that I enjoy more than any other kind of work. Whether it’s creating API’s or tools, or setting up bugbases, or building an ecosystem for code reviewing, I’m happy doing it all. So I can’t help but post about this.

This is the third company where I’ve set up a Perforce depot and been responsible for its design, maintenance and tool infrastructure. With input in particular from Matt Scott I think we’ve got the best source control setup I’ve ever worked with. We’ve solved a lot of nagging problems that have plagued me at previous gigs. There are still some issues, but overall things are very smooth.

I plan to cover things like the design of the depot, standards and requirements, supporting tools, and server maintenance. Perforce, out of the box, is far from a complete system. I’m pretty annoyed with those people, actually. All that money we give them every year and they’re mostly standing still, too busy rolling around naked in cash to fix basic and ancient problems with their design. But that will have to wait for a future post.

Comments (1)

Why I Disabled My Windows Page File

The page file is the source of a lot of frustratingly bad performance on Windows, especially on a notebook computer running Vista. So of course there are a lot of articles out on the web talking about page files. I’ve found most of it pretty uninformed and pseudo-scientific.

I wanted to write about this while it’s on my mind. My lady just got a new notebook (a fun and inexpensive convertible tablet by HP) and I’m having to go through the same Vista optimizing crap with hers that I had to do with my old Sony. Right now for example I have to research how I got mcupdate.exe to stop locking up the hard drive for 30 sec after restoring from hibernation (oop, found it: it’s in Task Scheduler under Media Center).

Well, one of the first things I did on that machine was disable the page file. But oh no! Aren’t those necessary or simply useful? On older Windows systems, yes. On modern systems, filled with periodic update checkers and scheduled tasks, a page file is a giant anchor, dragging everything down.

Why a Page File?

Wikipedia as usual has a good article on page files. What they are, how they work, why we have them.

Based on my own Windows-specific experience, they’re good for a few things. The main and obvious one is that they let you use more memory than you have available. Slow but effective. It was a key feature of Windows way back. Without page files, Windows 95 simply wouldn’t have run at all on most systems. Apps were huge compared to the amount of memory available. Memory was so expensive that OEM’s regularly sold new machines with half the memory they needed to run even decently well. XP had a similar issue when it came out. But today, memory is ultra cheap. Systems are shipping with far more physical memory than the typical user will use. The HP I mentioned, which was already a steal for $800, came with a free upgrade from 1G to 3G.

Another supposed page file advantage is that they let you swap out unused memory from inactive apps you have running in the background. Then you can use that memory for things the active apps need like disk caching. And what a waste to use up memory on stuff you almost never use, like the print spooler, right? Swapping these inactive things out can potentially boost performance, but it really depends on usage. In my experience, this is the key behavior that hurts. More on this later.

Very rarely, some programs will use specific “page file-backed” memory allocations. See MSDN docs on CreateFileMapping with hFile set to INVALID_HANDLE_VALUE. I heard from a friend who does audio production that some of his apps didn’t work when he disabled his page file and I bet it’s because of this. Nothing I use currently has this problem so I’m ignoring this one. Also, the behavior may have changed on Vista. I’m not certain but I remember hearing that Vista can create temporary page files if needed per process. No me importa.

And lastly, the system requires a minimum amount of page file space on the system drive in order to do certain kinds of crash dump for reporting system failures to Microsoft, especially on Vista. Disabling this will prevent your system from being able to report detailed info about blue-screens to Microsoft or something like that. Now, I typically feed back crash info to MS, but in this case it wouldn’t help. All of the (very rare) issues I have on my notebook are Sony driver related, and they don’t even support Vista on this old thing so these reports would just gather dust.

Why Not a Page File?

Well, page files can really slow down a system, particularly a notebook computer where the drives are much slower than the big desktops. My PC at work has a 10K drive but my notebook’s is 5400. For serial reads such as streaming movies or music or loading a game world, no problem. But when it comes to random access, these drives fall down hard. And page files are all about the random access. Software is inherently random access, because the user’s whims are totally random. At least, from the point of view of the computer.

Now, I happen to have 2 gigs in my notebook which is plenty to do everything I need. On occasion I’ll get low memory warnings when running Eclipse+Flex, Visual Studio, Firefox, a web server, and a few other things at once, but I have yet to get close enough to running out to have issues. So there is no compelling need for me to use more memory than the system has.

But…why not just leave it on anyway? After all there’s that whole thing about paging out stuff you don’t use to make room for caching. Performance should be much higher, right? In my experience, performance is worse. Far, far worse.

The State of Modern Windows Systems

Consider the state of Windows today, even if you’re diligent about spyware and toolbars and so on.

Our machines are increasingly filled with update checkers for the system and individual apps. I keep disabling these and every time I manually do an update they keep reinstalling them (in the doghouse: Apple, Sun, Microsoft, Intel). Autoruns is a good friend here.

Modern Windows systems are also filled with periodic tasks. For example, the mail program checking mail, a tray app or widget fetching RSS feeds, a task to fire up mcupdate.exe every day (grr), or the scheduled tasks Google installs to check for updates. Or maybe you’ve got Firefox running in the background and there’s some add-in that does its own scheduled thing (checking Gmail, fetching news, or compacting its own little very important sqlite database).

And lastly, there are the triggered events. For example, say you turn on Wifi, or you restore from hibernate and the Wifi is getting reactivated. Well, that triggers mobsync.exe for some reason, which in turn causes Windows Media Player to scan its entire database. This happens even if you’ve disabled the wmplayer network sharing service. Infuriating.

I’ve managed to disable as many of these as I can without affecting my work, but here’s the big problem.

The Windows Paging Algorithm Hates You

Experiment time. Load up the Vista Resource Monitor. Wait a few minutes for the system to settle down. Suspend the computer, then wake it back up. It should pop up pretty quick. Just a few seconds of disk activity, and then the Monitor’s disk panel should show some minimal stuff from WDI and performance loggers. I’d like it if the WDI stuff didn’t happen but it’s not too bad.

Next, try waiting a couple days between suspending and resuming, say over the weekend. This is a common scenario for me. I’ll often hibernate my machine on Friday, and then won’t end up touching it until Monday morning on the bus. While I was using it on Friday, all of the apps I had minimized or sitting in the tray got paged out by the system. Services as well, like Windows Update and so on - all paged out because they hadn’t been used in a while. Minimizing an app also tells Windows “I’m not using this for a while” so it dumps them. And at some point I probably ran a big app like Flex or Corel that ate a bunch of memory and convinced Windows to page stuff out to keep the file cache at 25%-50% of memory (doesn’t that seem like crazy overkill?). So at the time I put it to sleep, it was running fast and fine, and lots of stuff I wasn’t actively using at the time was sitting in pagefile.sys or was released by the system if it was a read-only mapping from disk (such as code and rdata sections of EXE’s and DLL’s).

Well when waking the thing up on Monday, guess what happens. Every single app and service notices that it’s been three OMG THREE whole days since they last checked for updates or mail or RSS feeds or phoned home or scanned for network shares or compacted their databases or reticulated splines or whatever else. And all of them wake up simultaneously to try to do their little tasks. Unfortunately, they were all paged out from being idle. So now, my machine locks up with hard page faults and tries to pull in a billion little 4K pages at once. The system is either totally unresponsive or severely degraded for several minutes while I wait for things to settle. I can watch this all happen on the Resource Monitor. Hard faults and disk accesses like mad.

This several-minute wait is absolutely infuriating when all you want to do is just open your computer to do a quick check on something like the news for the day.

Apple Gets It

It’s even more infuriating because my slow old clunker of a G5 iMac with only 1G of memory does nothing of the sort. And by slow I mean it was slow and outdated on the day I bought the thing, like all pre-Intel Apple hardware was. Yet on my Mac, I can leave all manner of giant bloaty apps loaded in memory (Firefox, iMovie, iTunes, iChat etc.), put it to sleep, and wake it up a week or two later to do some quick task. It has always, without fail, popped up and been ready to use instantly. Any apps I had minimized or hidden are active almost immediately, regardless of how long it had been since I had hidden or sent them to the Dock.

What the hell?

Clearly, OS X is doing something right, and Vista is doing something absolutely brain-dead. Thinking more about this, I’ve never had the Mac lock up on me unless I peg the CPU. On Windows I can have a minimal CPU load but a locked up machine because it’s just sitting and spinning waiting on the slow hard drive. I think what I’m seeing is a far superior virtual memory manager and I/O subsystem on OS X.

Does nobody at Microsoft notice this? This has been the case on just about every Windows computer I’ve used or been asked to “fix because it’s slow” by a friend.

So Disable It

So! Just disable the damn thing! My main complaint here is that on Windows, and especially on Vista, it’s just about impossible, without a lot of tuning work to the system, to have a notebook available for immediate use. At home if I want to do something quick on the web (look up a drink recipe or check email or whatever) I will always go to the Mac first because I know it will be available within 3-5 seconds, regardless of what state I left it in last. This one thing alone almost makes me want to switch to a Mac full time (more on this in the future).

It takes a bit of work to get the system waking up quickly, and keep it that way, but by far and away the biggest speed-up comes from disabling the page file. I’ve done a lot of testing with it both enabled and disabled. With a reasonable amount of memory, it’s a huge, easy win. The “page out to help the file cache” thing is, in practice, total bunk.

Furthermore, I am in favor of disabling the page file even on non-mobile systems where you don’t care that much about wake-up speed. When I work from home, I use my workstation which runs XP. If I’m sitting down to do some serious work, I don’t care if it takes a few minutes to wake up and get settled. But I do care about its runtime performance while doing normal work. And Filemon tells me that pagefile.sys is the main drive activity while I’m sitting and waiting frustrated. So, I disabled the page file here too!

At work I can’t disable it because I actually can run out of memory from Maya, a few copies of Visual Studio, the game, a running build, some console dev tools, etc. The 10K drives and split page file really help here of course, though I still get lockups on occasion. When I get back from Peru I’m going to see about switching it to Vista x64 (if the dev tools will run under that) and bumping the memory to 8 or 12 gigs and then disabling the page file. After doing some perf testing to see what the win is, of course.

Special Note: SSD’s

Solid state drives (drives with no moving parts, typically flash memory-based) completely change this equation. No moving parts means no rotational latency, no physical read/write head bouncing around. These drives seem ideal for a page file, in fact, where random-access performance matters much more than read/write performance. I’m waiting just a bit longer for prices to drop before swapping my notebook drive out for an SSD, but I’m pretty certain I’ll turn the page file back on when I get one.

A side benefit of SSD’s is that the drive doesn’t make any noise. Windows has a serious problem with insisting on constantly accessing the hard drive. Watching it on Diskmon I see regular physical access to the disk by the system, and who knows any of it is necessary. I don’t know why Microsoft even bothered putting in power-saving options to spin down the hard drive automatically. It’s impossible for it to ever get there because something is always messing with it. So the end result for me is that I can hear my drive parking the head, then tick tick for some access, then wait, then another head park, over and over again. In a silent room it gets obnoxious. Not to mention the noise of the spinning platter.

I was thinking the other day about how much crap is in Windows to deal with the problem of mechanical latency. If you didn’t have that head moving back and forth and have to wait for the right spot of magnetized goo to rotate under it, so many things would be totally unnecessary. Think of all the stuff in Vista that could be removed or simplified with SSD’s: the defragger, SuperFetch, disk caching, ReadyBoost, etc. You could treat the hard drive as simply “slow but consistently slow memory” instead of “slow memory that is sometimes really slow and maybe even stupidly slow depending on the current drive layout”. Consistency of experience always wins. I’m excited to get one!

Optimized Page File Configuration

So ok, maybe you decide to use a page file anyway. Well, it’s important to configure it right. By default, Windows “manages” the size of your page file. This means it may change size depending on system needs. With a fragmented hard drive, you’ll have the page file growing into fragmented space. This takes bad performance and squares it.

I see this problem quite a bit with non-technical users. They often like to ride their hard drives at 98% full, and when they run out, gaining space by just deleting a bunch of music or pictures. This is about the worst use case for a system-managed page file.

In my experience a lot of IT departments are clueless about this as well, leaving Windows at the default configuration. Give the average user a year and they’ve taken their drive performance into the toilet, and start complaining about how slow their machine is. In most cases it’s simply a heavily fragged hard drive and page file. Doing the instructions below and setting a nightly/weekly scheduled defrag.exe run eliminates most performance issues. Note that Vista has automatic defrag set up out of the box.

Tell the system to leave the page file alone. To do this, do the following:

  1. Turn off your page file and reboot.
  2. Fully defrag your hard drive. Make sure there is a contiguous free space that will fit your page file (see below for size considerations). Keep defragging until you get there, or use a trial of a commercial defragger to force it.
  3. Turn on the page file but set its min/max size to be the same. This prevents the dreaded automatic growing/shrinking.

Some commercial defraggers can do the page file during the startup blue screen, so you could take that route instead. But the main point is to set the min/max size to be the same.

If you have two hard drives, the page file should go on whichever drive gets less random access usage. Typically the non-system drive. Make sure it’s a fast drive like your main drive, and not a USB or whatever. You can spread the page file across multiple drives if you like. Test first to make sure it actually is faster. Remember that by adding a page file it’s cutting into the other random-access activity you’ll be doing on that same drive.

As for how large to make the drive, go for 2-4G or so. It really depends on how many things you keep open at once, and how much memory they tend to use. Photoshop, Eclipse, PC games, and so on can be virtual memory hogs. If you run out, do the steps above to increase the size some more.

The common recommendation is to set your page file to double your RAM size. I don’t know where this came from or why but it’s outdated and irrelevant. Tune the size based on your needs. If you have lots of hard drive space then just set it to “really big” and don’t worry about it. Everyone’s usage is very different.

But of course I prefer to “optimize” my own page file by completely disabling it!

Comments (3)

Game Camp videos posted to YouTube

The videos from the Game Camp sessions are now posted on YouTube.

Special note: I was slightly drunk when I gave my second talk (optimizing the dev process), and moderately drunk by the time the panel started. Not my fault! People kept buying me drinks! Maybe that biased me, but I think the people I met at Game Camp were among the coolest and friendliest I’ve met at any game conference. :)

Comments

Slides from Game Camp

What a fun event! Definitely one of the best I’ve been too, everyone was so friendly. They even got me a bottle of tequila and a lemon (ha ha) which I’ll be working on as we travel to Copenhagen today.

As promised, I’m posting my slides. PowerPoint 2007 and PDF format are available at my Googlepages site on my Publications page. Which reminds me, I really have to find a better host than Googlepages. Also have to migrate all that old stuff from my Drizzle site as well. So much to do, so little likelihood that I’ll get to it…

Anyway here are direct links to the files:

I’m pretty happy with how the keynote went but the Development talk was a mess. I think the slides have some pretty good material in them, though. The couple of beers I had before the talk might have been a little too much. Same as writing code - one drink helps (lubricates the neurons), but two drinks can only hurt. Ah well.

Comments (2)

Spy++ for .Net Windows Forms

So I remember this in the future, here are the two main tools I found to do the equivalent of Spy++ but in Windows Forms (.Net) instead of HWND’s:

  • HawkEye from Corneliu I. Tusnea. This one is great, has the old Spy++ “target” you can drag where you need it.
  • ManagedSpy from MSDN. Download link here.

Comments

Speaking at Game Camp in Oslo, Norway

It looks like the schedule is up on the Game Camp page. This is a recurring local game dev conference in Oslo that my friend Dan Strandberg co-hosts. Dan and I met way back in the Dungeon Siege days and have kept up ever since. He asked me to fly out and do the keynote, which I’m looking forward to (I wonder if they have tequila in Norway). I’ll also be doing a tech talk on development process, and will be on the panel at the end of the day. I’ll post slides here after the conference. After we visit Copenhagen and Stockholm, that is!

Comments (1)

Poke.

Poking my blog to let it know I’m still around and intend to keep up with this. Been busy prepping to move to Peru (estoy tomando clases de espaƱol), working on my talks for the Oslo conference, and a ton of other stuff.

Back in October!

Comments (3)

New url: scottbilas.com

Sure, why not. Picked up the domain so now this is at scottbilas.com.

Comments

« Previous entries