In my previous post, I talked about why I have stopped using finalizers for unmanaged resource collection. I want this to be done through the disposable pattern instead, forcing the programmer to manage resources manually.

Ironically, finalizers are a great way to verify this.

Sander van Rossen quickly figured out where I was going with this and proposed in a comment that we can just assert in a finalizer. We just need a couple more things:

1. We need to track the source of the object to figure out where the leak is.
2. We need to ensure that finalizers run on shutdown, or our assert will never get hit.

This is familiar territory to C++ programmers. Most of us use memory management libraries that provide leak detection and reporting. Let’s do something similar in C#.

DisposableBase

If you look around online, you’ll find some full-featured IDisposable base classes intended to deal with the full finalization model. We can eliminate most of that as either too complicated or unnecessary. We just need a few things:

• A stack trace grabbed at construction time
  This will be used for the error report. Because this is expensive to gather, we need to control it with an #if that is off by default, only turned on if needed to help an investigation. Most of the time a leaked disposable will be easy to find by inspection, but in the 5% case we will need a lot more context.
• A finalizer that reports the problem
  It could throw an exception, fire an assert, or route to an error reporter. Depends on the application. In my example I just have it output to the debug window for a demo.
• Disposal helper methods
  These wrap up disposal a bit, so inheritors only need to implement OnDispose. The most important feature, though, is that Dispose will call GC.SuppressFinalize when our object is disposed. This eliminates the performance cost of having a finalizer in the normal case when clients are disposing this class properly. This is why the finalizer has no “if” in it – if it ever gets called, then we have a bug.

This is what I am currently using as my base class for handling unmanaged resources:

// comment out unless diagnosing a leak
#define DEBUG_DISPOSE

public abstract class DisposableBase : IDisposable
{
    // store stack at point of construction for possible later use
#	if DEBUG_DISPOSE
    StackTrace _trace = new StackTrace(true);
#	endif

    // finalizer will not be called if object was properly disposed
    ~DisposableBase()
    {
        string message = "!! Forgot to dispose a " + GetType().FullName;
#		if DEBUG_DISPOSE
        message += "\n\nStack at construction:\n\n" + _trace + "!!";
#		endif
        Debug.WriteLine(message);
    }

    public bool IsDisposed { get; private set; }

    public void Dispose()
    {
        ThrowIfDisposed();
        IsDisposed = true;

        try { OnDispose(); }
        finally { GC.SuppressFinalize(this); }
    }

    protected abstract void OnDispose();

    protected void ThrowIfDisposed()
    {
        if (IsDisposed)
            throw new ObjectDisposedException(GetType().FullName);
    }
}

Demonstration

Here is a simple test app that shows what happens if we forget to dispose an instance.

public class DatabaseConnection : DisposableBase
{
    protected override void OnDispose()
        { Debug.WriteLine("Disposing"); }
}

public class Program
{
    static void Main(string[] args)
    {
        using (var remembered0 = new DatabaseConnection())
        using (var remembered1 = new DatabaseConnection())
        {
        }

        var forgotten = new DatabaseConnection();

        GC.Collect();
        GC.WaitForPendingFinalizers();
    }
}

The first two instances dispose fine, but the third is leaked and so we get a log to the output window:

Disposing
Disposing
!! Forgot to dispose a DatabaseConnection

Stack at construction:

   at DisposableBase..ctor() in C:\Users\Scott\Cloud\Proj\tests\BlogSamples\ConsoleApplication1\Finalizers.cs:line 18
   at DatabaseConnection..ctor()
   at Program.Main(String[] args) in C:\Users\Scott\Cloud\Proj\tests\BlogSamples\ConsoleApplication1\Finalizers.cs:line 89
   at System.AppDomain._nExecuteAssembly(RuntimeAssembly assembly, String[] args)
   at System.AppDomain.ExecuteAssembly(String assemblyFile, Evidence assemblySecurity, String[] args)
   at Microsoft.VisualStudio.HostingProcess.HostProc.RunUsersAssembly()
   at System.Threading.ThreadHelper.ThreadStart_Context(Object state)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean ignoreSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Threading.ThreadHelper.ThreadStart()
!!

We can easily zero in on the exact spot where the leaked resource was allocated and wrap it in a ‘using’ to resolve.

Note the use of GC.Collect and GC.WaitForPendingFinalizers right before the test application exits. This is necessary in order to force all finalizing objects to be collected and reported before process shutdown, otherwise they are simply dropped by the system when the process’s memory is released.

These two can even be called during normal app run as well for leak testing without needing to wait for shutdown. This would be useful in a live service with a periodic leak for which we want an in-session nonfatal log of leaks.

What About External Classes?

This takes care of our own classes, where we have full control. But what about system or third party classes that are using finalizers? With those it’s back to square one.

Well, I suppose would write a helper class..

public class SafeDisposer<T> : DisposableBase where T : IDisposable
{
    public SafeDisposer(T disposable) { Obj = disposable; }

    public T Obj { get; private set; }

    protected override void OnDispose()
    {
        Obj.Dispose();
        Obj = default(T);
    }
}

public static class SafeDisposer
{
    public static SafeDisposer<T> Wrap<T>(T disposable) where T : IDisposable
        { return new SafeDisposer<T>(disposable); }
}

And equivalent updates in the demo code:

public class Program
{
    static void Main(string[] args)
    {
        using (var remembered2 = SafeDisposer.Wrap(new StringReader("foo")))
        using (var remembered3 = SafeDisposer.Wrap(new StringReader("poo")))
        {
            remembered2.Obj.Peek();
        }

        var forgotten2 = SafeDisposer.Wrap(new StringReader("boo"));

        GC.Collect();
        GC.WaitForPendingFinalizers();
    }
}

And output:

!! Forgot to dispose a SafeDisposer`1[[System.IO.StringReader, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]]

Stack at construction:

   at DisposableBase..ctor() in C:\Users\Scott\Cloud\Proj\tests\BlogSamples\ConsoleApplication1\Finalizers.cs:line 18
   at SafeDisposer`1..ctor(T disposable) in C:\Users\Scott\Cloud\Proj\tests\BlogSamples\ConsoleApplication1\Finalizers.cs:line 53
   at SafeDisposer.Wrap[T](T disposable) in C:\Users\Scott\Cloud\Proj\tests\BlogSamples\ConsoleApplication1\Finalizers.cs:line 67
   at Program.Main(String[] args) in C:\Users\Scott\Cloud\Proj\tests\BlogSamples\ConsoleApplication1\Finalizers.cs:line 98
   at System.AppDomain._nExecuteAssembly(RuntimeAssembly assembly, String[] args)
   at System.AppDomain.ExecuteAssembly(String assemblyFile, Evidence assemblySecurity, String[] args)
   at Microsoft.VisualStudio.HostingProcess.HostProc.RunUsersAssembly()
   at System.Threading.ThreadHelper.ThreadStart_Context(Object state)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean ignoreSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Threading.ThreadHelper.ThreadStart()
!!

So to use, we just wrap a disposable at the point of creation in SafeDisposer.Wrap() and access it through “.Obj”. Not bad, but not great either. I’m ok with the wrapper function, but the required access via the Obj member is a pain, and also means that converting an unsafe disposable into a safe one requires updating a lot of code.

Another option is (maybe) to use a dynamic proxy system to inject the functionality we need, letting client code remain unchanged except at the point of creation. Or perhaps a run-time system that patches system assemblies to do the injection.

I’ll leave this as an exercise for the reader because I think we’re probably getting seriously diminishing returns at this point. Most disposable objects in a large system will be classes that we have full control over. The few objects not under our control will likely be low level primitives that will be wrapped up by our own foundation classes anyway.

I learned .NET when the beta of 1.0 was out, and at the time Microsoft was putting a lot of effort into educating programmers the difference between native and managed programming to make the switch. “In C++ you’d do this, but in .NET you do this” kind of thing. So practically every article on the CLR talked about finalizers. They often made analogies to C++ destructors. C# even added some ill-advised sugar that converts ~Class() to Finalize() to make us grognards feel more at home.

Finalizers were obviously very important, and so we learned about them. We evolved base classes to help us do the boilerplate work and wished that the .NET languages supported mixins. But it never felt right to me.

The Problem With Finalizers

Recently, after a couple conversations at work, I figured out what’s been bothering me about finalizers. The problem is that we’re using a commodity manager to manage non-commodity resources.

In .NET, raw memory can be treated as a commodity, like a gas tank. We can use it fluidly and at any granularity, and treat it all roughly the same. This is a simplification of course, and there are performance considerations, but they closely map onto the underlying OS and are familiar to everyone.

Yet non-memory resources are not commodities and cannot be treated with a one-size-fits-all pattern. They have semantics and effects far beyond incrementally draining and refilling that gas tank. Each situation, each class, is different, and has different implications that we have to remember. Because of this, we can’t treat these resources in a nondeterministic way without potential hazards.

For example, take a file handle managed by a finalizer. Off the top of my head, there are three adverse effects of an unused handle not finalizing in time:

1. If there are pending writes to the file, then they are lost if the app exits without forcing finalizers to be called.
2. If the file was opened with restrictive sharing, then other applications cannot manipulate it until the GC gets around to finalizing the file object.
3. The OS isn’t designed to treat file handles as a commodity, and using them that way can negatively affect performance. We could even run out of file handles because the GC has no idea when there’s pressure in this space (all finalizers are equal).

Worse yet, in diagnosing any of the above issues, we also must roll our own tools. System tools such as LockHunter and Process Explorer have no way of distinguishing which handles are actually in use and will just give us a noisy, useless mess.

And that’s just a simple file handle example. The situation is obviously worse as more limited and complicated resources are involved like DirectX surfaces or database connections.

Dispose: Only A Partial Solution

You might wonder why I’m making a fuss. There’s an obvious answer to the non-determinism, right? Microsoft recognized this problem early in in .NET 1.0 and gave us the disposable pattern. It is a standard way of managing resources the old-fashioned way: ‘new’ allocates the resource, and Dispose() frees it. We even have some special syntax that helps automate this:

using (var textureMgr = new TextureManager())
using (var texture = textureMgr.AllocTexture())
{
    texture.Fill(Color.White);
}

The above is roughly equivalent to:

var textureMgr = new TextureManager();
try
{
	texture = textureMgr.AllocTexture();
	try
	{
		texture.Fill(Color.White);
	}
	finally
	{
		texture.Dispose();
	}
}
finally
{
	textureMgr.Dispose();
}

This gives us a rough, though somewhat tedious approximation of the RAII pattern used universally in C++.

In Microsoft’s .NET classes that wrap unmanaged resources, both patterns are typically used. There is a Dispose system that closes the handle, and a safety Finalize that cleans up if not already done already via Dispose.

So what’s the big deal? Well, the problem is in (a) knowing when it is necessary to call Dispose, (b) remembering to call it, and (c) updating dependent code when an existing type adds a new IDisposable implementation. It’s easy to miss something and let bugs creep in. For every single ‘new’ call, we must check the type to see if it or one of its parent classes implements IDisposable. If it does, then we must manage the instance directly. This means wrapping allocation in the ‘using’ construct for local temporaries, or implementing IDisposable and forwarding Dispose if contained as a member in another class.

Forgetting to dispose an unmanaged resource can lead to some of the most frustrating and difficult to track down bugs. And the nondeterminism in the underlying system pretty much squares that problem. It’s guaranteed to behave differently in the field than on a development machine.

Every new .NET programmer figures this out quickly when they write their first command line app that opens a file, does something to it, and writes out a new file. They find that, apparently randomly, sometimes the end of the new file is cut off. Forgot to call Dispose to flush and close it eh?

Most Strategies Not 100% Guaranteed

It’s not hopeless. We’ve come up with strategies to deal with this. First is knowledge base. Over time, we get a sense for what types of objects tend to implement IDisposable. File handles, database connections and so on, those are easy. Though what about systems written by someone else on your team? Does the source control connection have to get disposed when you’re done with it because it wraps a COM object? For those cases, it’s better to be safe and check for IDisposable until the foreign classes become familiar.

There are also tools that help out. FxCop can find problems that are discoverable via static analysis. CodeRush will draw graphical markup for types it detects as IDisposable.

But these strategies still aren’t enough for me. I want absolute certainty that Dispose is called on all unmanaged resources, and not left to the nondeterministic finalizer system to cause latent timing problems.

So I want to propose an additional strategy. It’s pretty simple, actually: consider the finalizer to be a last, and illegal resort. Microsoft’s guidelines say that you should not assume Dispose has been called. I propose the reverse: if the finalizer is called, then there is a bug in the code.

More on this in the next post.

The Problem

Say you have a pair of functions that bracket other operations you want to do. For example, you must first call BeginRender() before you call RenderModel(), and after you’re all done, you need to call EndRender(). Another example would be a EnterLock() on a critical section, that must be accompanied by a LeaveLock(). I’ll call these bracketing functions simply “BeginX” and “EndX” for the purposes of this article.

The problem is that the caller must not only remember to call EndX, but must do so even in case of an exception, to avoid blocking the system.

The Solution: IDisposable

What we’re doing here, really, is acquiring and releasing a resource. The .NET Framework already has a pattern for that, and it’s called IDisposable. Ok, that’s not much of a trick in itself. What I learned today was a little twist on the concept.

The trick is simple:

1. Create a private inner class that implements IDisposable.
2. Return a new’d instance of this class in BeginX.
3. In the instance’s Dispose(), have it call the private EndX.
4. Callers can simply wrap the ‘handle’ in a using declaration for cleanup.

It’s obvious in retrospect, and it’s clean. The ‘using’ guarantees calling EndX even in case of an exception. And in leveraging IDisposable we can build on top of the C# programmer’s instinct to wrap usages of IDisposable objects in ‘using’ constructs. As a fallback, we can rely on tools like CodeRush to catch it as we type (or FxCop as a fallback).

Implementation

First, we need a fun little helper base class.

public abstract class DisposerBase<T> : IDisposable
{
    bool _disposed;
    T _owner;

    public DisposerBase(T owner)
        { _owner = owner; }

    public void Dispose()
    {
        Debug.Assert(!_disposed);
        _disposed = true;
        OnDispose(_owner);
    }

    protected abstract void OnDispose(T owner);
}

Here’s a sample class that demonstrates the IDisposable-bracket pattern. Note the inner class that has access to private methods, which we use to call EndX. This method is private so that the only way to close the bracket is to call Dispose() on the handle. Simple, straightforward, consistent.

public class TransactionManager
{
    bool _inTransaction;

    public IDisposable BeginTransaction()
    {
        Debug.Assert(!_inTransaction);
        Console.WriteLine("Begin Transaction");

        _inTransaction = true;

        // ... do other transaction setup here ...

        // caller must dispose this when done
        return new TransactionDisposer(this);
    }

    private void EndTransaction(TransactionDisposer state)
    {
        Debug.Assert(_inTransaction);
        Console.WriteLine("End Transaction");

        // ... do transaction cleanup here ...
        // (state could provide more context if needed)

        _inTransaction = false;
    }

    #region TransactionDisposer

    class TransactionDisposer : DisposerBase<TransactionManager>
    {
        // obviously some state could go here
        // (say, for more transactions in flight at once)

        public TransactionDisposer(TransactionManager owner)
            : base(owner) { }

        protected override void OnDispose(TransactionManager owner)
            { owner.EndTransaction(this); }
    }

    #endregion

    public void DoSomeAction(string actionName)
    {
        Debug.Assert(_inTransaction);

        // ... do the action ...
        Console.WriteLine("  Action performed: " + actionName);
    }
}

Note that this is a very simple example that only permits one transaction in flight at once. If we wanted to do more, we could have a Dictionary that maps outstanding transactions to pending state, or we could just put the state directly into the TransactionDisposer. I’ve noted this in the comments.

Now, it’s a bit of typing to implement this disposer, but that’s what snippets are for, right? Bracketed code like this isn’t overwhelmingly common either, so it shouldn’t be a big deal. Still, I have to say I miss my old friend #define from the C world sometimes. Not often. I’m looking forward to a future version of the C# compiler that lets us plug into the AST directly and codegen as we go.

Anyway, here is an example usage of this pattern:

var transactionManager = new TransactionManager();
using (var handle = transactionManager.BeginTransaction())
{
    transactionManager.DoSomeAction("action 1");
}

Implicit Chaining

There is one interesting benefit that comes from this: implicit chaining.

Say you have a higher level class that provides additional functionality within its own BeginX. Perhaps it calls BeginRender() and then immediately sets some default states. Rather than having to create a new inner class to manage the bracketing, you can simply return the result of BeginRender, and the caller will not know the difference.

Let’s extend the above example in this way.

public class RequestManager
{
    TransactionManager _transactionManager;

    public RequestManager(TransactionManager transactionManager)
        { _transactionManager = transactionManager; }

    public IDisposable BeginRequest(string initialActionName)
    {
        var handle = _transactionManager.BeginTransaction();
        _transactionManager.DoSomeAction(initialActionName);
        return handle;
    }
}

See how it chains the TransactionManager’s bracketed calls through itself? We could have gone through the trouble of making an inner RequestDisposer class and a private EndRequest and all that, using the IDisposable from TransactionManager as the state, but why bother? Just chain it along. When the caller disposes the handle, it routes straight through to the TransactionManager, which is all we needed.

Some sample usage:

var requestManager = new RequestManager(transactionManager);
using (var handle = requestManager.BeginRequest("action 2"))
{
    transactionManager.DoSomeAction("action 3");
}

Of course, if RequestManager has to do anything special in its EndX, then we’ll have to go and add that inner class after all. It’s a nice shortcut otherwise.

The output from both examples looks like this:

Begin Transaction
Action performed: action 1
End Transaction
Begin Transaction
Action performed: action 2
Action performed: action 3
End Transaction

Easy huh?

Performance Implications

It’s worth asking what the costs of this method are. How does this compare to simply calling BeginX and EndX?

Let’s consider this sample usage again:

using (var handle = transactionManager.BeginTransaction())
{
    transactionManager.DoSomeAction("action 1");
}

This is syntactic sugar (I loathe this term but it is unfortunately apt) that the compiler translates into something like this:

IDisposable handle = transactionManager.BeginTransaction();
try
{
    transactionManager.DoSomeAction("action 1");
}
finally
{
    if (handle != null)
        handle.Dispose();
}

Note that the runtime does not do anything special with IDisposable types. They are created and collected exactly the same as any other type. And as a result, this is almost identical to what we’d have to do for a manual BeginX/EndX:

transactionManager.BeginTransaction();
try
{
    transactionManager.DoSomeAction("action 1");
}
finally
{
    transactionmanager.EndTransaction();
}

Now, this is assuming no state needs to be maintained in the handle. If it were, then we’d have to store it in a local and pass it into EndTransaction, and the BeginX/EndX pattern would almost exactly match the disposer pattern.

So: the cost of the disposer pattern over BeginX/EndX is that an extra object gets new’d up, with a bit of glue logic that runs. Insignificant, not worth worrying about.

Even if it did have a real cost, I’d still not worry about it. This is intended to be used in bracket patterns where resources are being allocated and freed. The cost of the underlying resource management is going to mask whatever minor cost the disposer adds.

The cleaner, more maintainable, and statically-analyzable code that results is an easy win.

Until now, this stuff has been scattered all over four different defunct websites in a mess of broken links. This new page fixes all of that for good.

I worked at Gas Powered Games between November 1999 and October 2003, back when it was a fresh startup. In that time, I lectured and wrote about some of the technology we used to build Dungeon Siege and later projects. I also released specifications for some of the internals of the engine to help the mod community.

This page collects all of my published Dungeon Siege work.

…continued here

It’s pretty exciting, even if I am joining this game very late.

As I work in the WPF world (and maybe WCF and other techs as I start messing around with them) I’ll be posting odds and ends here about research I do and dead-end paths I go down.

Not WinForms++

The first thing I learned is that WPF isn’t WinForms++ or even “WinForms in XML”. It lives in an entirely different universe. Though, for folks like me that have been stuck in that old world too long, WPF is initially going to feel like a simple upgrade of an old framework. There are a lot of similar-sounding classes and concepts in the new framework. These are merely false cognates.

I’ve been learning that it’s better for me to enter into new WPF concepts like a 101 student: by reading books and MSDN docs. I haven’t done that in a while. Normally I just bang on code and refine, using Google and StackOverflow to guide my way (and surfclarity to filter out spam sites like experts-exchange.com). I’m still doing that as I pick up WPF, but adding on top of it some serious reading.

Pro WPF in C# 2008 (available at Safari where I’m reading it) is my current favorite book for learning WPF concepts. Most books I come across are lame reference manuals disguised as tutorials, or sensationalized drooling over new tech with no real experience to back it up. The author does great things like quickly explaining a cool-sounding WPF concept, then telling you not to use it and why. Not all of WPF turned out to be a good idea, after all.

So, here are some of the key benefits I’ve noticed in the short time I’ve worked in WPF:

Layout and Composition

The problem of control layout (wrap, dock, table, stretch, fill, and so on) is solved. It’s nice of WPF to catch up with where HTML has been for a while, though in a much cleaner and more internally consistent way. Later versions of WinForms started to get somewhere with TableLayoutPanel and friends, but it’s still the same old underlying framework, where you run into limitations very quickly.

WPF lets you easily nest any control within nearly any other, and has a sensible system for guiding layout containers (attached properties rather than Control-derived least-common-denominator properties). It’s simple, for example, to put a pair of icons and a few checkboxes into each element of a ListView, and make it look good. Virtualization is automatic as well, given certain constraints. Trying to make any of that work in old WinForms requires a lot of custom code.

In fact, it was enough work in WinForms that I tended to avoid doing anything complicated. This led to UI’s that were easier to program but harder to use due to the need to break out elements or hide things behind the ubiquitous “more” or “advanced” buttons. Consider your average Tools|Options dialog and the custom toolwork required to make that work in a Form. This is why “field of checkboxes” settings dialogs proliferate in so many apps (see pic at right).

With WPF I’ve found it easy to not only wire up the initial draft of the UI, but also to go back and do a big reorg without any real trouble, and usually without significantly changing any code. I Just did it three times today, in fact!

A Special Note on XAML

Layout and composition are now typically done in XML, using Microsoft’s XAML. At first I was tentative on XAML compared to MXML due to the total lack of support for writing code inline in the attributes. Well, it turns out that viewmodels eliminate the need for much of that inline code (with the glaring exception of value converters – more on this in a future post). It’s also probably a good idea to keep code out of the XAML if you’re splitting the work with a designer. Having code in those attributes would not only get in the way, but would be brittle and hard to maintain.

The bottom line is that WinForms’ crazy InitializeComponent is gone. Bye bye merge problems and periodic, inexplicable total reorganization of that function by Visual Studio. Instead, we lay things out hierarchically in nice, simple, consistent, and nearly unreadable XML. Well, that’s what the visual designer is for.

Data Binding

I’ll admit I didn’t mess around with data binding much in the WinForms world. It just felt like too much work for what was a simple thing to do by hand. And WPF is vastly more complicated and capable than Forms, so it was intimidating at first.

But in WPF you really need to fully embrace data binding. I suppose it’s possible to do old UI-driven Visual Basic style state management in WPF, by putting everything in the codebehind, but you’re not going to get much support for this. All samples and tutorials you’ll find online use data binding. It really is how the entire system was designed to be used. Data binding is about a lot more than simply tying a textbox to a text string member. It’s about separation of concerns.

Thankfully, it’s not difficult to pick up. There’s some crazy complicated 20-levels-deep-stack logic that makes it all work under the hood, but the end result is that you just cover your eyes and type {Binding SomeName} on whatever field you want bound to your viewmodel and it magically works. “Except”, as they say, “when it doesn’t”. So far it’s been rare for me when something fails to bind, or binds to the wrong place, and I’m developing a bag of tricks to debug these issues. I’ll try to post some of these in the future.

Now, about that viewmodel thing. It’s also practically required to embrace the MVVM pattern in building up nontrivial WPF-based UI’s. This is the “separation” I mentioned previously. Now, this takes a lot of practice to figure out and to start applying correctly. It took me about two months to really grok WPF+MVVM. So be sure to budget time to learn and practice this. Read the article I just linked for a good introduction.

Styling

The problem of styling controls and windows is hugely improved. It simply isn’t possible to modify controls in WinForms without writing a totally custom solution. WPF takes a big step forward in enabling simple as well as complex styling of the standard controls. You can completely alter the look and visual behavior of any WPF control without a lot of work.

And then there’s Expression Blend, my new love interest. Blend is a tool for the design and styling of WPF UI’s, and permits dividing responsibilities between engineers writing code and designers building the look & feel. We have this exact split at work, in fact. Engineers will poke out a rough UI, and later on one of the game UI designers will take some time to upgrade elements to look cool and be consistent with the rest of the tool.

Blend has a goofy, nerdy interface, but it’s really growing on me as I use it every day. It’s far from intuitive, but is powerful in terms of designing controls and styles, previewing it all live, live live! in its awesome designer. It can even build your full Visual Studio solution from within itself. I do almost all of my design work within Blend, partially because VS2008’s designer is so awful as to be almost useless (haven’t used 2010’s enough yet to comment) but mostly because it’s more familiar to the Flash programmer within me. What can I say, I like arty tools.

Rendering

One of the reasons styling can be done so easily is that WPF doesn’t rely on the underlying OS window manager to draw checkboxes and so on. This removes all limitations on what you can change. If you use a tool like Snoop or Mole to take apart a UI control, you can see that every little graphical piece and state of it resolves down to primitive WPF objects that you can inspect.

For fun, I edited the template of the ResizeGrip control in Blend to see what makes it tick. This is the little triangular dotted thingy that sits in the lower right corner of most sizable windows. Blend showed me that it’s rendered using a simple brush that has a repeated pattern of gradient squares clipped to a little stair-step box. All vector so it scales nicely.

How does this new rendering model affect Remote Desktop, by the way? Custom-rendered windows typically perform poorly through RDP. Try running Chrome or iTunes through RDP, or nearly any app through VNC, on a low bandwidth VPN connection to see what I mean. In earlier versions of .NET, if a DWM existed on both sides, the WPF graphics primitives would get remoted directly. However, it seems that in .NET 3.5SP1 and newer, WPF is remoted completely as bitmaps.

Microsoft is saying that this should actually improve performance in many cases. Well, we can only expect a future of more complex, denser GUI construction (I mean complex in the UI tree sense, not in the end user’s perception of complexity). So the 100%-bitmap-remoting solution may be needed in the long term anyway. But I’m not sure I agree with the timing of this decision. Bitmap remoting so far performs an order of magnitude slower for me than graphic primitive remoting, at least in the everyday apps I use through RDP.

Bye Bye WinForms

It’s clear to me that anyone working in WinForms should ditch it as soon as possible in favor of WPF. Rewriting old tools is always a cost-benefit analysis, but for new tools, this is clearly the way to go. Especially if you’re in control of your user environment, such as within the enterprise, where you can force install the framework on every machine.

If you’re still hanging out in the WinForms world, or considering buying an expensive Forms-based UI package to pick up the pace on building UI’s, you really ought to take a hard look at WPF.

Don’t expect to be productive immediately, or even within a month. It will take a while to get used to it, learn the framework, and develop patterns to use in building UI. It’s time well spent. This is clearly the future.

Sword of Fargoal is an ancient Rogue-style game for the C-64, written by Jeff McCord in BASIC (I remember curiously walking through the source code) and published by the long-defunct Epyx, one of my favorite publishers from back in the day. When I was a kid I played the hell out of this game and I’m excited to see some game developers who are even bigger fans than me have gone and remade this awesome game.

I’ve been playing Fargoal LLC’s remake on my iPhone (available here at the App Store for only five bucks!) and it’s clearly a labor of love. Most remakes of classics are disappointing. Someone writes an emulator and they just run the old game on new hardware. It’s 100% authentic and all, but that’s not necessarily a good thing.

You’re all excited to play the game you loved as a kid, and then, five minutes in you realize that..oh..this game is kind of frustrating, annoying even.. It’s not because the game is bad, it’s because the world of gaming has changed since then.

As an industry we’ve moved far past the primitive control techniques and UI’s of the past. We expect to have tutorials and hint screens, auto-save and context-sensitive UI gizmos that help us out.

Old games just didn’t have room for any of that in their 8K or whatever of ROM. While this gave some insane replayability, and created a small sub-industry of invisible ink cheat books, learn-by-dying is just a thing of the past. It’s no fun.

I’m happy to report that the developers of the new Sword of Fargoal did more than just slap some upgraded graphics on this game. They’ve added help, auto-save, and a great control scheme (moving diagonally on the C-64 joysticks was always a dicey affair). But my favorite is a great new feature called the “Smart button” that gives you context-sensitive actions, such as using a teleport spell right when the ceiling is about to collapse on you.

Fargoal LLC has big respect from me. I’m already in love with this game again and now I can bring it everywhere with me. All I need is some OpenFeint (or similar) integration to compete with my brother.

Anyway, go buy Sword of Fargoal immediately. Swoosh! Clang! Thud!!

Sorry it took me so long to get around to this Miko, but I’ve had my Flex work on hold for a while. But recently I did some Flash work at PopCap and it got me back in the mood again.

New Enum Features

Here’s my Ultimate AS3 Fake Enum class, which has the following new features:

• Further type safety. Only the enum can construct its own constants now, via enum constructor protection. This should prevent people using the constants incorrectly (e.g. someone is incorrectly new’ing them up instead of using MyEnum.MyConstant).
• Every constant now gets a zero-based index that corresponds to the order it is created in the class. Access via the Index accessor on the constant. Useful in serialization code, for example.
• Added a new GetConstants method that returns an array of all the constants, where each constant’s slot in the array corresponds to its Index.
• Added a new ParseConstant method that does the reverse of toString(). Given a constant’s Name, return the constant. Case sensitivity optional and off by default for convenience.

…all without any extra burden placed on the derivative class! I’m always interested in solutions that keep client code as clean as possible.

I also adjusted the code to my current AS3 coding standards, which is nearly the same as Microsoft’s C# standards. Note that I’d normally use Vector.<Enum> instead of the generic Array class, but I haven’t upgraded my Flex to the newest SDK yet.

The Code

Without further ado, here’s the new Enum.as file. Stick it in a package if you wish, and drop it wherever you like. No system setup is required to make it work.

package
{
	import flash.utils.*;

	public /*abstract*/ class Enum
	{
		public function get Name()  :String { return _name; }
		public function get Index() :int    { return _index; }

		public /*override*/ function toString() :String { return Name; }

		public static function GetConstants(i_type :Class) :Array
		{
			var constants :EnumConstants = _enumDb[getQualifiedClassName(i_type)];
			if (constants == null)
				return null;

			// return a copy to prevent caller modifications
			return constants.ByIndex.slice();
		}

		public static function ParseConstant(
				i_type			:Class,
				i_constantName  :String,
				i_caseSensitive :Boolean = false) :Enum
		{
			var constants :EnumConstants = _enumDb[getQualifiedClassName(i_type)];
			if (constants == null)
				return null;

			var constant :Enum = constants.ByName[i_constantName.toLowerCase()];
			if (i_caseSensitive && (constant != null) && (i_constantName != constant.Name))
				return null;

			return constant;
		}

		/*-----------------------------------------------------------------*/

		/*protected*/ function Enum()
		{
			var typeName :String = getQualifiedClassName(this);

			// discourage people new'ing up constants on their own instead
			// of using the class constants
			if (_enumDb[typeName] != null)
			{
				throw new Error(
					"Enum constants can only be constructed as static consts " +
					"in their own enum class " + "(bad type='" + typeName + "')");
			}

			// if opening up a new type, alloc an array for its constants
			var constants :Array = _pendingDb[typeName];
			if (constants == null)
				_pendingDb[typeName] = constants = [];

			// record
			_index = constants.length;
			constants.push(this);
		}

		protected static function initEnum(i_type :Class) :void
		{
			var typeName :String = getQualifiedClassName(i_type);

			// can't call initEnum twice on same type (likely copy-paste bug)
			if (_enumDb[typeName] != null)
			{
				throw new Error(
					"Can't initialize enum twice (type='" + typeName + "')");
			}

			// no constant is technically ok, but it's probably a copy-paste bug
			var constants :Array = _pendingDb[typeName];
			if (constants == null)
			{
				throw new Error(
					"Can't have an enum without any constants (type='" +
					typeName + "')");
			}

			// process constants
			var type :XML = flash.utils.describeType(i_type);
			for each (var constant :XML in type.constant)
			{
				// this will fail to coerce if the type isn't inherited from Enum
				var enumConstant :Enum = i_type[constant.@name];

				// if the types don't match then probably have a copy-paste error.
				// this is really common so it's good to catch it here.
				var enumConstantType :* = Object(enumConstant).constructor;
				if (enumConstantType != i_type)
				{
					throw new Error(
						"Constant type '" + enumConstantType + "' " +
						"does not match its enum class '" + i_type + "'");
				}

				enumConstant._name = constant.@name;
			}

			// now seal it
			_pendingDb[typeName] = null;
			_enumDb[typeName] = new EnumConstants(constants);
		}

		private var _name :String = null;
		private var _index :int = -1;

		private static var _pendingDb :Object = {};	// typename -> [constants]
		private static var _enumDb	  :Object = {};	// typename -> EnumConstants
	}
}

// private support class
class EnumConstants
{
	public function EnumConstants(i_byIndex :Array)
	{
		ByIndex = i_byIndex;

		for (var i :int = 0; i < ByIndex.length; ++i)
		{
			var enumConstant :Enum = ByIndex[i];
			ByName[enumConstant.Name.toLowerCase()] = enumConstant;
		}
	}

	public var ByIndex :Array;
	public var ByName :Object = {};
}

The Samples

Here are a couple sample enums and some code to demonstrate the features.

Simple Enum: State

This is the simplest usage of the Enum class. All you have to do to make a fake enum is:

1. Create a class that extends the Enum class. Tag it as final too so nobody inherits it (that would be weird).
2. Put some static init code in there that just calls initEnum(MyTypeName). This will get run on demand, on the first usage of the enum by other code. It sets up the necessary lookup tables and attaches a Name and Index to each enum constant.
3. Create a public static const for each enum constant you want, following the pattern in the sample.

Here’s the sample enum. Note that it includes a bug on purpose! (It’s clearly marked.)

package
{
	public final class State extends Enum
	{
		{initEnum(State);} // static ctor

		public static const Initializing :State = new State();
		public static const Connecting   :State = new State();
		public static const Loading      :State = new State();
		public static const Ready        :State = new State();

		// illegal; this will fail at runtime
		public static const Test		 :OtherEnum = new OtherEnum();
	}
}

And some test code to walk through the features…

for each (var state :State in Enum.GetConstants(State))
{
	trace(state.Index + ": " + state.Name);
}

trace("connecting (no case): " + Enum.ParseConstant(State, "connecting"));
trace("connecting (with case): " + Enum.ParseConstant(State, "connecting", true));

// illegal; gets a runtime exception
var someEnum :State = new State();

Which produces the following output:

0: Initializing
1: Connecting
2: Loading
3: Ready
connecting (no case): Connecting
connecting (with case): null
Error: Enum constants can only be constructed as static consts in their own enum class (bad type='State')
 ...

Advanced Enum: NetError

Let’s look at a more advanced sample, which I’m calling an “attributed” enum. Let’s say we have a NetError enum with constants for different types of failures on a network request. We want to be able to show a more user-friendly error, and perhaps classify it as a fatal error or not. We could have a table that associates extended attributes for enum constants with those constants, but why not just attach them to the constants directly?

In .NET I’d do this with attributes. Now, AS3 doesn’t support attributes like we need, so we’ll have to fake it. Given that we already have a fake enum constant, which is really an object underneath, we can attach whatever we like to it!

Take a look at the sample enum.

package
{
	public final class NetError extends Enum
	{
		{ initEnum(NetError); } // static ctor

	// Constants.

		public static const None			:NetError = new NetError(false, "no error");
		public static const CantConnect		:NetError = new NetError(false, "can't connect to the server, must be down?");
		public static const VersionMismatch	:NetError = new NetError(true,  "protocol versions different");
		public static const Unknown			:NetError = new NetError(true,  "unexpected error type");

	// Constant attributes.

		public function get IsFatal()	  :Boolean	{ return _isFatal; }
		public function get Description() :String	{ return _description; }		

	// Constant query.

		public static function GetConstants() :Array
			{ return Enum.GetConstants(NetError); }
		public static function ParseConstant(i_constantName :String, i_caseSensitive :Boolean = false) :NetError
			{ return NetError(Enum.ParseConstant(NetError, i_constantName, i_caseSensitive)); }

	// Private.

		/*private*/ function NetError(i_isFatal :Boolean, i_description :String)
			{ _isFatal = i_isFatal; _description = i_description; }
		private var _isFatal	 :Boolean;
		private var _description :String;
	}
}

Given a constant, perhaps returned from a function as an error code, or thrown as part of an exception, we can ask it for its friendlier name by accessing its Description member.

Also in this class note the helper functions GetConstants and ParseConstant that wrap the Enum class versions, automatically passing in the local type. This isn’t necessary, but is a nice convenience. It is a bit more code to copy-paste around when adding new enums, though. If only AS3 supported templates, we could have a TEnum base that does all this for us. Ah well.

Here is some test code to play with the features

for each (var netError :NetError in NetError.GetConstants())
{
	trace(
		netError.Index + ": " + netError.Name +
		" (" + netError.Description + ") " +
		(netError.IsFatal ? "[FATAL]" : ""));
}

Which produces the following output:

0: None (no error)
1: CantConnect (can't connect to the server, must be down?)
2: VersionMismatch (protocol versions different) [FATAL]
3: Unknown (unexpected error type) [FATAL]

Future?

I hope this is the last word on fake enums in Actionscript. Not because I don’t think anyone else can improve on it, but because I’m hoping that the next version of the language will have native support for enums. Then we can put all this fakery behind us and get on with our lives.

The enum posts are among the top hitters on my blog, and they are popular topics elsewhere on the web. I’m hoping that Adobe will strongly consider native enum support in their next Flash player.

In short, “shelving” is taking a pending changelist and publishing it to the server without checking it into the depot.

As I mentioned in a previous post, this is one of the top features from git that was making me seriously consider checking it out. I started implementing this myself in my p4.exe wrapper, but I never gave it enough priority to get much work done. I’m excited to know that Perforce are adding it themselves. This is the first new feature I’ve seen from them in years that is worth upgrading for!

The things that interest me most about this ability are:

• Easy trading of in-progress work in a formal, server-supported way that is done out-of-band of the depot, so it cannot break the rest of the team. Good for code reviews, small tests, moving the same pending changes among multiple machines/clients, etc.
• Easy, incremental backup of works in progress. This is hot.
• Easy ability to temporarily put aside a local change in order to work on something else that collides (typically as an emergency). This eliminates the main reason I needed to use several P4 clients on the same machine.

I hope Crucible will have support for this great new feature after it’s released. That would eliminate most of the need for the tool I wrote.

I’m looking forward to seeing this new feature in 2010.

…and I got a new job.

Moving On

I’ve moved on from Loose Cannon and am now working at Bungie. The reasons are personal and private, and were due to circumstances beyond anyone’s control. It was the most difficult, painful choice I’ve ever had to make about anything.

So I’ve been laying low while all the emotion passes..it’s felt a lot like breaking up with a girl, I think. Similar feelings, similar lost personal investment, similar broken ties. Similar number of depressing hangovers.

The Loose Cannon crew is a great bunch of intensely creative and smart people, and I wish them all the best. They always treated me very well and I miss them.

Enter The Bungie

So now I’m at Bungie. First off, let me make it clear that I don’t speak for the company in any way. But further than that: just because I talk about something on my blog here doesn’t mean it has anything to do with what I’m working on at Bungie, so please don’t make any assumptions to that effect. I’ve been writing and lecturing for a long time now, and have a lot of personal projects and research that I do. This will continue as before.

Now, a lot of the writing that I’ve done has in fact been based directly on my ongoing work on a particular game. Previous companies I’ve worked for have been supportive of, or at least indifferent to, my talking about technical details of our setup or research I’m doing. That doesn’t necessarily have to stop now that I’ve joined Bungie, where they have recently started sharing quite a lot of information. But it does mean I have to be more careful and get things cleared before saying “this is in our upcoming Halo match-3 game”.

For the moment though, it’s a moot point. I just started and don’t know anything yet.

On the other side of the coin, there are some loose ends I never tied off from previous posts. Now that I’m no longer part of Loose Cannon, I can’t talk about work I did there, or reference code, which I no longer have access to. I think I can speak in generalities, though, about some high level concepts that wouldn’t be specific to the work I did there. We’ll see!

Oh, did I forget to post on here that our game shipped? Wow, I totally did! Well, we shipped in September on 360, PS3, and Wii. Become a fan on Facebook!

The reviews have generally been in the B range, though Metacritic’s average is a little depressing at ~70. Nearly every review uses the words “Katamari” and “clone” which makes me think they didn’t really give the game a chance. The similarities with Katamari begin and end at the concept of “grow bigger”. Might as well call Call of Duty a Doom clone because they both involve shooting things to get to the end of the level.

But whatever… I’m proud of our effort, and I think it’s a super fun game. Hell of a job for a small team with a tech base built from scratch, to ship on time and on budget on three platforms.

Wwise Tour

One of the reasons we were able to ship on time was Audiokinetic’s Wwise. I’ve been evangelizing this excellent sound engine to everybody I meet. I just can’t say enough good things about our friends up in Montreal. I’d use Wwise on every future game if I could.

On Tornado Outbreak, I did most of the engineering and the initial audio rig design and prototyping. Robert and his team did the actual audio work, and took over management of the Wwise project. They probably did 95% of the audio related work on the project, which is awesome! As everybody knows, engineers are really slow and are pulled in 20 directions at once, so the more I could step out of the way, the better.

Audiokinetic asked me to put together a talk for the tour they’re doing right now to promote the product, particularly the new features they’ve been adding. I invited Robert to join me and we presented the two halves of our audio solution for Tornado Outbreak. We split the presentation roughly along the lines of our responsibilities.

The event was recorded, so at some point we may see video clips showing up online. That will be necessary to get Robert’s part of the talk, because he was exclusively walking through the Wwise project, using elements of it to tell his story. So no slides, you’ll have to get the video if it comes out. Anyone using or considering Wwise should try to get a hold of that – his talk was really interesting and includes some great tricks on saving memory without sacrificing variety.

Now, all I know how to do is present slides, so here they are! Tornado Outbreak + Wwise = Love [PPT 2.5M] [PDF 1.4M].

Our Project

I also got approval from the bosses to release the Tornado Outbreak project file for Wwise. This is really generous of them to agree! Here it is. [ZIP 1M]

Note that this doesn’t include any content (wav files) but is only the project itself. That should be plenty though.

The point of releasing the project is to help out other studios who are integrating Wwise, in hopes that the favor will be returned. Everybody benefits from information sharing like this. With Wwise, in order to get a good rig set up you really need to have experience and good examples to draw from. As I say in my presentation, Wwise is different. Every other SDK out there is a “play samples + DSP” library. Getting the Wwise rig right is hard, and it’s not going to be right the first time. Just as if you were to build a Maya rig and had no experience with it before. You’ll screw it up for sure.

Audiokinetic provides some synthetic examples to help get started, but it’s not from a real, shipping game, and besides, every game is different. Tornado Outbreak has an enormous amount of unique objects that produce audio (over 400) and all those crashes and shakes and panics can become very difficult to manage. It would have saved a lot of time if, when I started building the initial rig, I had some examples to draw from. To that end, we’re releasing our particular solution for this situation. I hope that it inspires even better solutions and ideas on how to tackle these kinds of problems in the future.

Enjoy.