Kandria
Another tale from gamedev complications
We're still working hard on the Kickstarter preparations and everything associated with that. Please remember to subscribe so you get notified when it goes live!
I enjoyed writing up last week's entry on gamepads, so I thought I'd write about another unusual complication during development.
Files!
If you've ever looked at the files a typical PC game installs onto your drive, you might have noticed that it's not uncommon for it to have a bunch of stuff in there like dll
s, one or two exe
s, maybe some config files, but also very likely a couple of huge "content" files with some obscure file extension.
Like me, you might have wondered why that was done. After all, underneath the game probably uses a bunch of png
s, jpg
s, ogg
s and so forth. One answer is "security through obscurity". By obscuring the raw files in some container, it's much harder to extract them and repurpose them. But I never really thought there was that much of a point to that. You can still extract the files anyway, modders have been doing this stuff for ages.
So anyway, this question stuck in the back of my head for a while but I never really bothered to look into it. Fast forward many years and I'm working on Kandria. Kandria saves the world data in a bunch of files:
world/
regions/
hub/
data/
BLOCKER-5658.raw
BLOCKER-5663.raw
BLOCKER-5785.raw
BLOCKER-5814.raw
BLOCKER-5850.raw
BLOCKER-ENGINEERS.raw
CHUNK-1831.graph
CHUNK-1831.raw
CHUNK-1831-0.raw
...
Each chunk (a visible area in the game) gets a graph
file that caches movement AI data, and 6 raw
files that contain the tile data. There's a bunch of other files that need to be saved as well, but those chunk files make up the bulk of the now over 2000 files that the game needs to load in to read the current region. All of these files are small, typically a couple kilobytes of raw, uncompressed data. Being uncompressed we can read them into memory and upload them to the GPU really quickly.
Now, on Linux where I primarily develop the game, this isn't an issue. Reading in many small files like that is fast and not that big a deal.
However, when the Steam demo happened in March, we got some reports from people complaining that the game was "stuck" loading at 0% for up to a minute! At first I wasn't sure what was going on, since on my machine it always flushed through that 0% very quickly. During that 0% step it reads in all of these small files. So I tested on Windows and it turns out opening a file on Windows is several magnitudes slower.
I'm not sure why exactly that is, probably partly NTFS being a bad file system, partly Windows having its insanely complex access management protocols, partly it just never having been optimised for small files like Linux has. In any case, this was a problem I definitely had to solve, having to wait a minute on a screen that looked like it was crashing was no good.
So this is where these file bundles come in. If you just bundle up all of these files into some kind of archive, then you only need to open that one file, and can then handle the reading of all the separate files directly in-memory, bypassing Windows' slow file opening.
As a bonus side-effect, the files are now also compressed, saving a whopping 25 megabytes of storage. Wowsers! Not to say Kandria used a lot of space before, but it's always nice to save some more.
Conclusion
Even files aren't that simple, it turns out. I have some more ideas for stories about how complicated seemingly trivial things are, and might share them in the coming weeks. We'll see!
For now it's back to working on the Kickstarter material and all of that stuff. Please share the page with your friends as well!