One of the things that Keith and I had been thinking about with the Continents idea is that it's a way for us to split up the world into yet more component files (continent files) so that not all of it has to be in RAM at once. And not all of it has to be synced across the network to new players, and all of that sort of thing. Of course there are solid gameplay reasons for doing continents as well, and that's what we focused on in the other thread that was specifically about continents.
The above said, I'm starting to think that the technical reasons for continents were really missing the point. Gameplay-wise they're a great idea and are still coming. But in terms of whether we literally need/want to split up the world file... that's something that's a lot less clear to me. Here's my thinking about this stuff.
THE BROAD DATA STRUCTURE
-----------------------------------------------
There are two different parts to any given world:
1. The central world/region index info (basically what you see on the world map, the macro game items, info about players, info about all NPCs in the game, etc).
2. The per-region detail data about what is actually in any given screen when you go there, etc. This is the part that actually has the terrain maps and monster locations and info on loot drops and all that sort of thing.
The central indexes (world.dat and regions.dat) are normally stored uncompressed on disk because they need to be written frequently and it uses less transient RAM and CPU to not compress them. In a world with 519 regions in it, I'm looking at about 320kb uncompressed on the disk, which compresses down to about 47kb. Larger worlds compress at an even better ratio.
The data in the region detail folders actually is compressed on disk already, so it tends not to compress any further at all. In these folders is a regioninterior.dat which really tends to only be a few kb at most, and then 0-n "chunk files" that are -- compressed -- something like 3-20kb each on average. Every side-view screen you enter in the game is called a "chunk," and has one of these files associated with it.
WHAT WORKS WELL ABOUT THIS
-------------------------
Central World Indexes:
The central world index stuff contains enough to run the world map, to have NPCs be able to talk about the overlord by name, to have info about quests or the status of various locations, and so on. But it really doesn't have much more than that.
This is ideal from both a RAM standpoint and a transferring-in-multiplayer standpoint. In terms of RAM, even a really massive world with a hundred thousand regions is probably only looking at 63mb of data on disk in an uncompressed format. For comparison, an uncompressed AI War savegame can range from 20-90mb easily (which is why we compress them, typically to 1-8mb on average).
Given that there is incredibly, vastly much more data in play with AVWW worlds than AI War galaxies, this sort of segmentation of data is really needed for having a world that can approach anything remotely like infinity. But of course there's a finite cap when you've got linear RAM use; if it's 63mb on disk, I'm betting it would be 200mb in memory (because there's a lot of calculatable data that we reconstruct when stuff is loaded off disk, to keep the disk footprint smaller).
All of this, again, is comparable with AI War so that's great. Though we're talking right now about a world with a theoretical 100,000 regions in it. To reach that size of world would take somewhere in the neighborhood of 5,000 civ levels. And to reach THAT point, in the ideal balance we're going to be shooting for, would take about 15,000-20,000 hours of gameplay. Multiplayer wouldn't make this happen faster.
The average super-hardcore AI War player seems to get about 300-600 hours of gameplay out of the game over a two year span. So if we extrapolate back into AVWW, that would put them at, at best, something like level 400. Now, of course we already have some players who have racked up over 100 hours in AVWW and are already level 200 or so. The time between levels is currently a bit out of kilter, closer to 30 minutes per level for these folks.
So even if we take that more pessimistic number, and we look at them getting to level 1200 instead of level 400, that's still only about 24,000 region tiles on the world map. That, in turn, is about 16mb of data uncompressed on disk for their central world indexes. Compressed, that's something that can be synced to a new player connecting into a game in about the same time that an unreal tournament map could be synced, or a midsize AI War savegame. That's not horrible for being after 600 hours of gameplay in a single world.
Per-Region Detail Data:
This is where things REALLY shine. The only region data that's loaded off disk are the regions that a player is in at the moment. Afterward, they are just dumped back to disk. Same thing with the chunks -- someone is in a chunk if it's in memory, otherwise it's dumped back to disk within about 10 seconds.
That keeps the actual data about gameplay areas (as opposed to central world map kind of stuff) incredibly small. And there's really no way to spike the RAM on that at all, aside from having a really huge number of players in a world at once, all in different chunks from one another. Our estimates are that it would take somewhere in the neighborhood of 30-50 players to become a RAM issue with this on the server, with Unity's current mono heap limit of 800mb.
This is also something that is really cool for multiplayer, because as you transition between the various screens in side view, the server is only having to send you 3-20kb of data (on average) as you do so. Keeps things snappy to say the least, and it really means that in terms of actual CHUNK data, the game really is practically unlimited in what it can store, even in multiplayer.
THE PROBLEM
-------------------------
The actual problem here is that the disk space (and to a lesser extent, RAM) requirements of a world's folder will grow linearly with time, despite all the above benefits of our model. And I don't think there's a better model we could use, it's simply a fact that bigger worlds are... bigger. All that data has to be stored somewhere.
Right now we're really minimizing the impact on RAM from growth of anything except the world map itself, but the more screens you visit the larger your disk footprint gets on the server or your client. Which is not a huge deal since disk space is cheap and we're only talking hundreds of megabytes at most in most cases, but still.
If left to just grow unchecked, then eventually the world would get so large that:
1. The game would become unplayable because of running out of RAM. I don't think this would happen until about 5,000+ civ levels into the game (if not even further out -- could be as far as 10,000+ levels), but that depends on exactly how the game continues to evolve. And that's presuming that we're keeping to a relatively slow world land-area-size growth like we currently are, but honestly I think I might like to bump that up a bit so that you get more land a little faster than now, yet without running into RAM issues suddenly at level 2,500 instead of level 5,000 or whatever winds up being. Again, the RAM thing is kind of a moving target because the game is still evolving and the actual average RAM footprint per region isn't yet fully known (but it's not going to do anything but grow, if AI War is any indication).
2. Additionally, as the world gets progressively larger, it is taking up more disk space. Mainly in terms of the per-region detail data (mostly chunk files). This isn't a huge problem, as I noted above, but it is annoying. If you want to periodically back up your world folder, for instance, it's going to take progressively longer and longer to back up the larger your world gets. And the most annoying part is that most of the data that it's spending so long backing up is old areas you haven't visited in dozens of hours, and probably never will visit again.
3. Players connecting into a multiplayer server will get increasingly long waits, and this will also impact other players who are currently playing on the server. Transferring a few hundred kb to an incoming player is no big deal to get them up to speed on the world map state. And once they are online, it is transferring updates in chunks to them. And there are other optimizations we could make here to mitigate this, too, and Keith may already have made some of them (I can't recall anymore). But the simple axiom is Big Worlds Are Big, and there's no escaping that to some extent.
WHY CONTINENTS DON'T HELP THIS
-------------------------
This whole linear-growth phenomomena is something we've obviously been putting a lot of thought into for a long time, and we've mitigated it to an extreme degree already, I think. But our next planned mitigation, of splitting the world files into continent subdivisions, I'm realizing really doesn't help as much as I'd want. It would help with initial multiplayer syncs to some extent, and it would certainly help with the RAM use of the central indexes since any continents that are not in use could be on disk.
However, each continent really represents a pretty large amount of data, so moving between continents could get annoying fast. And there could be some severe limitations in what we can do cross-continent if we ever felt like implementing trade or something, since we can't do anything with settlements that aren't in RAM because they're on a different continent.
Then when we come back to multiplayer, if players are wanting to be on different continents to work on different side missions, then those are all in RAM anyway, and all you've really saved is some bandwidth. Which is still useful, but there are easier ways to save bandwidth, like just sending regions that are actually in proximity to each client regardless of considerations about continents, etc. We may actually already be doing that, that's certainly how we're doing updates about that data.
Coming back to solo play, when you think about mission time and how that's something that causes new side missions to pop up and disappear, that's another thing that would be somewhat problematic if older continents were not stored in memory at the time. They'd get left behind in a major way.
And none of this addresses the problem of disk space at all, so backups still take linearly longer, etc. Which is still really annoying!
WHAT REALLY NEEDS TO HAPPEN, IN SOME FORM
-------------------------
Old chunk data, and old region data, needs to get purged from the world at some point. Plain and simple. There are things that players simply don't care about, and that stuff should get removed so that there isn't linear growth anymore. This does NOT mean removing old settlements or anything like that -- even key locations like old overlord keeps are trivial to keep, in terms of their data.
What I think is that there are some "key locations" that players wind up having longer-term interest in and wanting to maintain, while the rest of the "I was just off doing some scavenging or a minor side mission" kinds of areas can be dumped because the player has already forgotten about their details anyhow (and probably wouldn't recognize them if they returned to them anyway, if it's been long enough).
So. This is what we're brainstorming about, and why.
MY IDEA: ONGOING CATACLYSM
-------------------------
Okay, so this is a post-disaster world, right? But in a lot of respects the world really still is in the midst of a disaster. In another thread, Keith and I were talking about having disasters that would strike older continents to up their region levels and basically those would purge whatever had been there.
Well... what if we took that a lot further than just meteor strikes or flooding or whatever, and made it so that after a certain amount of time had passed that continents became volatile and started to sink into the sea? Bam, that's a huge purge of data right there.
Once you have something like 5-6 continents, the oldest one goes into a crisis and will sink into the sea after some amount of mission time. What do you do in the interim? Well, you now have the fun of trying to rescue whatever parts of the continent you care about.
Maybe you can magic the settlement(s) from there to any of the other continents of your choosing to save them and the citybuilding stuff that was going on with them.
Or maybe it's really more of a loss of the settlement itself, and you're just saving the people, but they bring across their know-how and goods and stuff and are able to quickly rebuild an equivalent-but-different-looking settlement in a new place.
Maybe you can choose a couple of other regions to also magically move to safety of another continent, but at the cost of a unit of mission time for each region you're saving. So if there was some really iconic overlord keep that you want to retain access to, or you've got a near-infinite cave system that you're down 15 caverns into, you could move those to safety before the rest of the continent goes down into the sea.
Or if you're not sentimental, you just let the whole thing sink into oblivion and all those older NPCs die, too. It's up to you.
But either way your world stays leaner in filesize, and actually with this sort of system the amount of playtime in this world moves a lot closer to infinite -- you'd start having problems at about region level 2.3 billion, rather than civ level 5,000.
Thoughts? Other ideas?
