New Fun Blog – Scott Bilas

Take what you want, and leave the rest (just like your salad bar).

Dungeon Siege

with 12 comments

Dungeon SiegeI worked at Gas Powered Games between November 1999 and October 2003, back when it was a fresh startup. In that time, I lectured and wrote about some of the technology we used to build Dungeon Siege and later projects. I also released specifications for some of the internals of the engine to help the mod community.

This page collects all of my published Dungeon Siege work.

A lot of time has passed since those days, so nearly all of the below work should be considered to be out of date. Any recommendations I had regarding technology are as irrelevant today as ray cast engines were then. With one big exception: the component-based game object system paper is still very relevant. More on this below.

Lectures and Papers

The Continuous World of Dungeon Siege (2003)

This is a lecture I gave at GDC San Jose. They liked it enough that I was invited to London to give it again later that year. Good times.

In it, I talked about the history of and technology behind our streaming 3D world engine. At the time, this was still a new idea, with almost no literature published on the subject. The few games that had shipped with streaming worlds were relatively simple. They either had predictable distributions of basic content, or limited their content layout to canyons and tunnels. That was our original plan as well, but early on some smart folks on the team figured out we could do a lot more.

Slides: [PowerPoint] [PDF] [With Notes] [Outline Only]
Paper: [HTML] [Word] [PDF]
Q&A: Some Notes on Dungeon Siege

Our engine enabled us to have dense, variable, heavily scripted content in all directions, and the designers took full advantage of this. The regular resetting of the world origin and everything-is-relative terrain graph meant that we could literally go to infinity with no loss of precision. The engine supported up to eight separate streaming worlds in memory simultaneously, one per party member in single player or per-player in multiplayer. You could have everybody in a different part of the world and switch among them instantly. To my knowledge no game has done this since. It still makes for a cool demo, and even today I’d consider it advanced.

imageWe were also able to use the node graph to do some bizarre things with our engine not possible in a Euclidian-space game, such as the infinite level trick I mentioned in the paper. Late in the development we also used the graph to add wormhole-style teleportation between arbitrary parts in the world. (I still can’t believe that worked.)

Overall, though, I wouldn’t do a streaming world engine the same way again. Like I said, it was expensive. Hard to understand, hard to code against correctly, hard to optimize. Though we succeeded, and what we did was cool as hell, I’d steer away from our design. Hundreds of streaming-world games have shipped since Dungeon Siege. From what I can tell, the industry has settled on the “giant bricks” model of world subdivision. I’d go with that.

imageA Data-Driven Game Object System (2002)

I gave this lecture at GDC San Jose, and I still get emails about it in 2010, with the occasional citation showing up in academia, blogs, and forums. I doubt I was the first to come up with the concept, but I must have been one of the first to publish on it. Today, of course, you can search for “component based game object” and find mountains of material.

Slides: [PowerPoint] [PDF] [With Notes] [Outline Only]
Unfinished paper: [HTML] [Word] [PDF]

As is implied in the slides, I was inspired to create this system by getting fed up with how incompatible C++ static type hierarchies are with rapid iteration in creating content. Tying content types to C++ types, which is still common today, invariably leads to giant blobs of immovable code ruled by Count Virtula. It was a frequent theme on the sweng list I used to haunt, as well as in more than a few Game Developer post-mortems. Dungeon Siege’s original game object system was no exception.

Today, with modern multiprocessor systems it’s an even bigger deal. To get peak performance, our game engines need their parallelizable data to be organized as structures of arrays for fanning out to job queues. Components make this significantly easier to implement by reducing game objects to bags of pointers. The concern of how to allocate the components (i.e. the parallelizable data) is a cross-cutting concern that can be naturally sent to a custom allocator. And the forced decomposition of data into components makes it a lot easier to put that data into buckets instead of the mess of is-a relationships that is a typical game object type tree.

Now, it’s with great regret that I look back and see that I never finished the paper I started writing to accompany the GDC lecture! Many people have emailed me over the years asking for more details on the system and it’s just been too long for me to remember enough to help. The best I can do is post the full version of my slides with the notes I had written for myself. I am also posting the paper that I started and never finished, in case it can add anything. Also, there ought to be audio floating around somewhere on the GDC site. Perhaps I can dig up an old tape and write a transcript. (“Tape? What’s a tape?” “It’s an 8-bit MP3”)

FuBi: Automatic Function Exporting for Scripting and Networking (2001)

This is probably my favorite ugly-but-killer hack that I’ve ever done, and was inspired by a nerdy drunken conversation at a party. We built a ton of Dungeon Siege on top of this system – network RPC’s, script-native interfaces, save-game, logging, performance reporting… As I mention in the paper, this really was the gift that kept on giving. I gave this lecture at the GDC in San Jose.

Slides: [PowerPoint] [PDF] [Outline Only]
Paper: [HTML] [Word] [PDF]
Code supplement: [HTML] [Word] [PDF]
Sample code: [Zip]

It was also a one-shot ugly-but-killer hack. I created this at a time when we had few and bad options for getting at the type system of a C++ executable at runtime. Today it’s not a big deal to get at native debug symbols. If I were to do it again, I’d just use a debug symbol SDK to reprocess key symbols into the binding type system in a post-build step. Or perhaps do a codegen solution. Or combine the two.

In 2009 on Tornado Outbreak, I used a combination of codegen, macros, and C++ templates to bind Lua to our game’s systems. That worked well also and was pretty straightforward.

May FuBi and its parse-the-mangled-dll-export hack rest in peace.

Game Programming Gems I (2000)

image I wrote three gems for this book, all of which were used in Dungeon Siege. (Recognize the book’s cover art?)

Gem: An Automatic Singleton Utility
Gem: A Generic Handle-Based Resource Manager
Gem: A Generic Function Binding Interface

We used the singleton utility all over the place, and I’ve added it to every game I’ve worked on since. The handle-based resource manager was inspired by the G-Engine we had at Sierra for Gabriel Knight 3, and was used for DS’s file system and parts of Skrit. And the function binding interface became FuBi, which I spun into the GDC talk mentioned above.

It’s Still Loading? (1999)

I originally developed this at Sierra and shipped it in Gabriel Knight 3, then turned it into a lecture for the GDC experimental “road trip” in Seattle. At GPG I wrote a much-improved version that shipped in Dungeon Siege.

Slides: [PowerPoint] [PDF] [Outline Only]
Paper: [HTML] [Word] [PDF]
Sample code: [Zip]

It was probably a bad choice overall. A lot of added complexity for relatively little gain. It worked, but I don’t like to think about it.

Move along, nothing to see here.

Technical Documentation for Mod Community

I was very active in the Dungeon Siege community as “gpmechanic” to help build pre-release buzz for the game as well as support the mod community after we released. We had dreams of enabling people to build the next Counterstrike on our engine. That never happened, but we did see some great custom mods built (unfortunately labeled “siegelets” by the community), many of which pushed the engine past what we thought it could do.

I also met a lot of great people in the community, many of whom are still good friends. Some ended up in the pro game industry, a few even working with me. Participating in the community was a lot of fun and I miss those days.

Anyway, the mod community was hungry for any information we could give out on the game that would help them modify it to create new games. I published what I could, given my time constraints.

Official Dungeon Siege Documentation

Gas Powered Games has most of the relevant documentation on Dungeon Siege at their Garage.

General Articles |     Modding FAQ |     Siege University

I helped write some of those documents, for the systems I had worked on. I recall working on these engine components:

  • Debug HUDs and gizmos, other debug infrastructure
  • Command line and INI configuration
  • Level of detail for items (LODFI)
  • Skrit (language, compiler, VM, framework)
  • Component-based game object system, scidbits, many Go components
  • Multithreaded content streaming (non-terrain)
  • Content database (templates)
  • Function binding and DSDLL’s
  • File system
  • Save game (binary, party, text) and other serialization
  • Elements of Siege Editor
  • Networking RPC’s
  • Test automation
  • Parameterized content (pcontent)
  • AIQuery

Many I designed and built completely, others I only helped out in implementing. Too far back to remember who did what. I’m happy to answer any lingering questions about any of the above, to the extent that I can remember.

Dungeon Siege Internals (Unofficial)

These are documents I released over time through my old Drizzle web site. I’ve simply moved it all forward to their final resting place here. None is really useful unless you are actually writing a new mod.

Type System (1.1 and 1.11)

This is what happens if you are in the dev build of the game and type /help.all then /help.enums(true) in the console. It dumps the type system and enum constants for the game to the console – here I rerouted it to a file using “report filelog generic”. It contains prototypes for all skrit and RPC functions (compressed so it fits on my tiny web site).

Exported function parameters (1.1 and 1.11)

I threw this together really quickly at the request of the community. It’s a simple scan of all our .h files looking for anything that has been exported, and includes the name of the function and the parameter list which includes the param names. Notes:

  1. I filtered out all RC functions, and anything that takes no params to keep the list short. [RC functions are not supposed to be called from skrit, it's an internal multiplayer thing that unfortunately gets exposed to skrit because it shares the same type system.]
  2. My cheapo script didn’t put class names on anything, so they’re just function names. It also doesn’t pay attention to precompiler directives so it scanned everything, including debug, test, and disabled functions that don’t exist in public versions of the game. I’ll make a better script later when I get some time to keep that clutter out.

Type System and Parameters (1.09.2)

Older type system dump – keeping it up here so people can ‘diff’ against the new stuff to see what’s changed.

Type System (pre-1.09b)

I took this snapshot after the 1.0 so it’s not exactly the RTM build, but it’s close. Keeping it up here so people can ‘diff’ against the new stuff to see what’s changed.

TankStructure.h

If you can read C code, this file defines the structures you need if you’re wanting to talk to tank files. This file is the same .h file used to compile the game and our tank tools.

Important: if you are constructing a tank builder of your own, make sure that it decompresses properly with no memory overrun. Dungeon Siege decompresses its resources in place, and zlib requires some extra room to fully extract its data. Make sure to thoroughly read the section “Decompression of chunked compression formats” in the TankStructure.h file.

FuBiPersistBinary.h

This is the format used by save game. While it’s relatively easy to parse, the problem you’re going to run into is deducing the schema, which is wired into the game directly. For example, you can’t really tell from the data file which is an integer and which is a float. In a future build of the game I had planned to add an option to output the schema as it wrote the text version. It would have made it easier to edit save games. Didn’t get around to it.

RapiRawReader.cpp

This is the format of our .raw files for textures. It is a custom format specific to Dungeon Siege and has no relation to the .raw files that Photoshop or other image-processing tools may output. If you’re messing with texture formats, you will probably also want to pay attention to the following note on the layout of the mipmap images for all imported (non-raw) formats from the source:

be sure to store images in this bitmap upside down! this reader is
expecting mips stored like so (after flipping):

   +--------+.........
   |        |        .
   |   1    +---+    .
   |        | 2 +-+  .
   |        |   |3+-+.
   +--------+---+-+-++
   |                 |
   |                 |
   |                 |
   |                 |
   |        0        |
   |                 |
   |                 |
   |                 |
   +-----------------+

if there are no hand-made mipmaps then it will construct them on top of
the 0 surface. each successive request for a new surface will box filter
the surface down over itself.

gpstring_std.h

Some people were wondering what the data representation of a gpstring is in memory. DS uses a hacked up version of the STL that comes with Visual C++ 6, and here it is. I changed it to substitute heap memory for stack memory usage – the same thing that MFC’s CString does. The gpstring is derived from this base, but only adds utility functions, so everything you need for the data layout is here.

PContent Query Grammar

PContent, or “parameterized content” consists of two main parts – the query language that lets you request a pcontent item, and the inventory system that chooses what to drop on a higher level.

This grammar I pulled straight from my comments in the code. It’s sort of BNF based, but you’ll get the idea. Also note that it may be out of date. For example, the “ornate” idea was dropped.

Goids and Scids

This is just a segment of a .h file that contains the definitions of Goids and Scids, just in case you wondered what they look like inside the engine. Nothing fancy, but here you go just in case you’re curious!

Recently, I was asked if it’s possible to go past the 255-item limit for character inventory. Unfortunately, this is hard coded. If you open the above link and look at the structure of GlobalGoidBits you can see that the “minor index” of a goid is 8 bits wide. Major indices were used for global Go’s in the world, and minor indices were used for their children, which included inventory items. It may be technically possible to add more than 255 to a Go, because the inventory container is a simple collection. However, once the streaming world, network code, or save game code gets involved, it will probably lose track of the owner of these items.

In the code there are many checks to make sure the major index of an item matches up with its owner. Also, when adding an item to inventory, it acquires the major index of its owner, and there would be duplicate Goid’s if we permitted > 255 items. So I am 99% certain that it is not possible to go past 255 items in inventory.

Skrit Manual [Word] [PDF]

A primitive, incomplete doc on skrit, written late at night (in bed, usually). This doc is now sealed and will not be updated further (far better materials are already available from the Dungeon Siege online community).

February 2nd, 2010 at 8:33 pm

Posted in

12 Responses to 'Dungeon Siege'

Subscribe to comments with RSS or TrackBack to 'Dungeon Siege'.

  1. Thanks for compiling these goodies Scott! DS is still a fantastic game and I had lots of fun modding it. I hope you find a another RPG / Hack & Slash project in the future!

    Francisco

    18 Feb 10 at 10:11 am

  2. Thanks for these! I have been waiting for ages to get to these documentations. Being both a Dungeon Siege fan and a novice programmer, this compilation is a must-read for me.

    Boro

    6 Mar 10 at 9:03 am

  3. I’ve read and re-read the game object stuff many times without fully understanding it, so it’s good to get more details! I’ve followed Adam’s posts at T=Machine, too.

    Also, do you have an RSS feed for your blog? I’ve scanned the pages but can’t find anything.

    Cheers,
    -Mark

    Mark

    23 May 10 at 4:53 pm

  4. Mark – the feed is in the page metadata, so most browsers should detect it. I also added a feed icon at the top of the page to help out. Saludos.

    Scott

    23 May 10 at 6:29 pm

  5. Thanks :)

    Mark

    24 May 10 at 1:07 am

  6. [...] A good data driven system gives a lot flexibility to a system, letting programmers and level designers tweak the game architecture without having to recompile the hole system Look for ‘A Data-Driven Game Object System (2002)’ at http://scottbilas.com/games/dungeon-siege/ [...]

  7. Thank you very much to keep online usefull info about DS one of the best game ever.

    Socket

    16 Oct 10 at 11:52 pm

  8. Still relevant.

    Den

    28 Apr 11 at 2:39 am

  9. [...] in the sand and this trend has completely passed me by, its been vetted first by Dungeon Siege (http://scottbilas.com/games/dungeon-siege/) and now large scale frameworks such as Unity embrace this model. (knowing the motivation behind [...]

  10. [...] Component Paper at This Page [...]

  11. […] 2003: Scott Bilas (scottbilas.com / Dungeon Siege) […]

  12. This is my first time go to see at here and i am really
    happy to read everthing at single place.

    chicago maids

    23 Apr 14 at 1:22 pm

Leave a Reply