Thursday, June 6, 2013

Thunder Moon's PackedContentManager (technical)

Recently, I was having a discussion with a peer on the XBLIG forums about how the XBox 360's filesystem is pretty slow dealing with loose files, and how it can be especially slow if there are a lot of files in the folder for it to deal with. Neither of these facts should really be an issue for a game that implements the time-tested technique of merging all the loose files into a single file with a dictionary that tracks where each original file's data resides in the packed content file. Think of it as putting all your game content into a zip file, and making the game use only that zip file instead of loading individual files. I think the first time I became aware of this being useful for games was way back in the Doom days where ".WAD" files were used for this purpose.  I suspect nearly every large scale commercial game made since then uses this technique in some form or another because it is a well known and low risk technique for optimizing data loading that I expect is usually worth the effort.

I don't know of an existing page describing how this the packed file technique would be done for XNA, so I'll give a brief rundown of it here. Unfortunately, I don't plan to release code for this technique because my implementation has encryption and other engine specific stuff built into it which I don't have time to extricate, but hopefully my explanation will be enough to put you in the right direction. Also, please keep in mind I am writing this from memory so some of the methods or class names might not be exactly correct as I write this up but you should be able to get the gist of it. It took me about half a day to get the initial code & tools set up, and various minor bugs worked out during the course of the next day or so, but I didn't have any guide to work from so hopefully it will be much easier for people reading this who need to implement something similar.

The first step, which is arguably optional but I prefer to do it this way, is to put all your content into a separate project outside the scope of the game project itself. This is to make sure the main project never has .XNB files in it's bin folder and is always using the packed content. I call this the "GameResources" project. It has the added benefit of allowing you to unload the project if you know you don't need to be rebuilding assets, which can be a huge time saver over the course of the project's development. Just make the game reference this project and the dependencies should just work.

The second step is to make a simple tool to merge all your content from the resource project into a single file. While you could make the tool read from a list and build a fixed set of files, I found it easier to make the tool recursively scan the content bin folder and include all the files it found. The first thing is to create the file with a pad of 4 bytes which will late be replaced with the position of the header data. As you read from each source file and append to the packed file, just track the start and length of each file. When you are done, save the header information at the end of the packed file and set those 4 bytes to the offset in the file where the header was. If you like, make the tool emit some logging information to a timestamped log file so you can see how the set of content changes over the course of the project.

Once the tool is set up, you can optionally make your game's build settings run the tool each time it makes a new build. I do this; the amount of time it takes to merge all the files is surprisingly small and hasn't been  enough to bother optimizing further.

Next is the runtime. There are two main parts to this: A custom stream class, which I call "PackedContentStream", and a derivation of the ContentManager called "PackedContentManager" that has an overloaded OpenStream. The XNA Game's LoadContent is modified to create one of these PackedContentManager objects (which I will start calling PCM) instead of the standard one. The PCM has a constructor that needs the path to the packed content file (PCF) which it opens once and retains an open stream for. Now, the PCM.OpenStream is overloaded so that instead of just trying to open a standard file stream, it instead creates a new PackedContentStream. If you are garbage-sensitive (as I hope you are), these will be pooled and will be set up to be reusable objects to minimize pressure on the heap. Once the stream is prepared, the PCM pretty much just does it's usual stuff and the higher level code doesn't need to do anything special.

The PackedContentStream is fairly straightforward implementation of the Stream class, with a little bit of extra info related to the current packed file. Namely, a reference to the existing stream for the packed file provided by the PCM, and the start position and offset of the embedded file. When you first create the class, you can just use the Visual Studio helper to auto-implement all the abstract methods, which is a great way to get started fleshing it out. This might sounds like a lot of work, but the good news is that the PCM doesn't call very many of these methods since it tends to just read data in a very simple manner. I found it very straightforward to just run the game with all of the methods still containing the default "throw" code and just implement the needed logic in the few functions that actually needed it when the exception was hit. The basic thing you need to do for the implementation is make sure all reads, seeks and other methods/properties are offset by the packed file offset and incorporate the packed file length to make sure they are reading the correct embedded file data and do not read beyond the embedded file's length. It's a pretty straightforward thing to do once you get to this point.

Finally, bonus points for making the PCM log the files opened and making the content packing tool order the files in the same order they are loaded by the game. This isn't necessary, but the reduced seek times can speed things up a useful amount.

That's about it. Hope this explanation is useful to someone!