Arcen Games

General Category => A Valley Without Wind 1 & 2 => AVWW Brainstorming => : x4000 December 01, 2011, 09:56:52 AM

: Brainstorming: Curtailing Infinite World Folder Size Growth
: x4000 December 01, 2011, 09:56:52 AM
One of the things that Keith and I had been thinking about with the Continents idea is that it's a way for us to split up the world into yet more component files (continent files) so that not all of it has to be in RAM at once.  And not all of it has to be synced across the network to new players, and all of that sort of thing.  Of course there are solid gameplay reasons for doing continents as well, and that's what we focused on in the other thread that was specifically about continents.

The above said, I'm starting to think that the technical reasons for continents were really missing the point.  Gameplay-wise they're a great idea and are still coming.  But in terms of whether we literally need/want to split up the world file... that's something that's a lot less clear to me.  Here's my thinking about this stuff.

THE BROAD DATA STRUCTURE
-----------------------------------------------
There are two different parts to any given world:

1. The central world/region index info (basically what you see on the world map, the macro game items, info about players, info about all NPCs in the game, etc).

2. The per-region detail data about what is actually in any given screen when you go there, etc.  This is the part that actually has the terrain maps and monster locations and info on loot drops and all that sort of thing.

The central indexes (world.dat and regions.dat) are normally stored uncompressed on disk because they need to be written frequently and it uses less transient RAM and CPU to not compress them.  In a world with 519 regions in it, I'm looking at about 320kb uncompressed on the disk, which compresses down to about 47kb.  Larger worlds compress at an even better ratio.

The data in the region detail folders actually is compressed on disk already, so it tends not to compress any further at all.  In these folders is a regioninterior.dat which really tends to only be a few kb at most, and then 0-n "chunk files" that are -- compressed -- something like 3-20kb each on average.  Every side-view screen you enter in the game is called a "chunk," and has one of these files associated with it.

WHAT WORKS WELL ABOUT THIS
-------------------------

Central World Indexes:

The central world index stuff contains enough to run the world map, to have NPCs be able to talk about the overlord by name, to have info about quests or the status of various locations, and so on.  But it really doesn't have much more than that. 

This is ideal from both a RAM standpoint and a transferring-in-multiplayer standpoint.  In terms of RAM, even a really massive world with a hundred thousand regions is probably only looking at 63mb of data on disk in an uncompressed format.  For comparison, an uncompressed AI War savegame can range from 20-90mb easily (which is why we compress them, typically to 1-8mb on average).

Given that there is incredibly, vastly much more data in play with AVWW worlds than AI War galaxies, this sort of segmentation of data is really needed for having a world that can approach anything remotely like infinity.  But of course there's a finite cap when you've got linear RAM use; if it's 63mb on disk, I'm betting it would be 200mb in memory (because there's a lot of calculatable data that we reconstruct when stuff is loaded off disk, to keep the disk footprint smaller).

All of this, again, is comparable with AI War so that's great.  Though we're talking right now about a world with a theoretical 100,000 regions in it.  To reach that size of world would take somewhere in the neighborhood of 5,000 civ levels.  And to reach THAT point, in the ideal balance we're going to be shooting for, would take about 15,000-20,000 hours of gameplay.  Multiplayer wouldn't make this happen faster.

The average super-hardcore AI War player seems to get about 300-600 hours of gameplay out of the game over a two year span.  So if we extrapolate back into AVWW, that would put them at, at best, something like level 400.  Now, of course we already have some players who have racked up over 100 hours in AVWW and are already level 200 or so.  The time between levels is currently a bit out of kilter, closer to 30 minutes per level for these folks.

So even if we take that more pessimistic number, and we look at them getting to level 1200 instead of level 400, that's still only about 24,000 region tiles on the world map.  That, in turn, is about 16mb of data uncompressed on disk for their central world indexes.  Compressed, that's something that can be synced to a new player connecting into a game in about the same time that an unreal tournament map could be synced, or a midsize AI War savegame.  That's not horrible for being after 600 hours of gameplay in a single world.

Per-Region Detail Data:

This is where things REALLY shine.  The only region data that's loaded off disk are the regions that a player is in at the moment.  Afterward, they are just dumped back to disk.  Same thing with the chunks -- someone is in a chunk if it's in memory, otherwise it's dumped back to disk within about 10 seconds.

That keeps the actual data about gameplay areas (as opposed to central world map kind of stuff) incredibly small.  And there's really no way to spike the RAM on that at all, aside from having a really huge number of players in a world at once, all in different chunks from one another.  Our estimates are that it would take somewhere in the neighborhood of 30-50 players to become a RAM issue with this on the server, with Unity's current mono heap limit of 800mb.

This is also something that is really cool for multiplayer, because as you transition between the various screens in side view, the server is only having to send you 3-20kb of data (on average) as you do so.  Keeps things snappy to say the least, and it really means that in terms of actual CHUNK data, the game really is practically unlimited in what it can store, even in multiplayer.

THE PROBLEM
-------------------------
The actual problem here is that the disk space (and to a lesser extent, RAM) requirements of a world's folder will grow linearly with time, despite all the above benefits of our model.  And I don't think there's a better model we could use, it's simply a fact that bigger worlds are... bigger.  All that data has to be stored somewhere.

Right now we're really minimizing the impact on RAM from growth of anything except the world map itself, but the more screens you visit the larger your disk footprint gets on the server or your client.  Which is not a huge deal since disk space is cheap and we're only talking hundreds of megabytes at most in most cases, but still.

If left to just grow unchecked, then eventually the world would get so large that:

1. The game would become unplayable because of running out of RAM.  I don't think this would happen until about 5,000+ civ levels into the game (if not even further out -- could be as far as 10,000+ levels), but that depends on exactly how the game continues to evolve.  And that's presuming that we're keeping to a relatively slow world land-area-size growth like we currently are, but honestly I think I might like to bump that up a bit so that you get more land a little faster than now, yet without running into RAM issues suddenly at level 2,500 instead of level 5,000 or whatever winds up being.  Again, the RAM thing is kind of a moving target because the game is still evolving and the actual average RAM footprint per region isn't yet fully known (but it's not going to do anything but grow, if AI War is any indication).

2. Additionally, as the world gets progressively larger, it is taking up more disk space.  Mainly in terms of the per-region detail data (mostly chunk files).  This isn't a huge problem, as I noted above, but it is annoying.  If you want to periodically back up your world folder, for instance, it's going to take progressively longer and longer to back up the larger your world gets.  And the most annoying part is that most of the data that it's spending so long backing up is old areas you haven't visited in dozens of hours, and probably never will visit again.

3. Players connecting into a multiplayer server will get increasingly long waits, and this will also impact other players who are currently playing on the server.  Transferring a few hundred kb to an incoming player is no big deal to get them up to speed on the world map state.  And once they are online, it is transferring updates in chunks to them.  And there are other optimizations we could make here to mitigate this, too, and Keith may already have made some of them (I can't recall anymore).  But the simple axiom is Big Worlds Are Big, and there's no escaping that to some extent.

WHY CONTINENTS DON'T HELP THIS
-------------------------
This whole linear-growth phenomomena is something we've obviously been putting a lot of thought into for a long time, and we've mitigated it to an extreme degree already, I think.  But our next planned mitigation, of splitting the world files into continent subdivisions, I'm realizing really doesn't help as much as I'd want.  It would help with initial multiplayer syncs to some extent, and it would certainly help with the RAM use of the central indexes since any continents that are not in use could be on disk.

However, each continent really represents a pretty large amount of data, so moving between continents could get annoying fast.  And there could be some severe limitations in what we can do cross-continent if we ever felt like implementing trade or something, since we can't do anything with settlements that aren't in RAM because they're on a different continent.

Then when we come back to multiplayer, if players are wanting to be on different continents to work on different side missions, then those are all in RAM anyway, and all you've really saved is some bandwidth.  Which is still useful, but there are easier ways to save bandwidth, like just sending regions that are actually in proximity to each client regardless of considerations about continents, etc.  We may actually already be doing that, that's certainly how we're doing updates about that data.

Coming back to solo play, when you think about mission time and how that's something that causes new side missions to pop up and disappear, that's another thing that would be somewhat problematic if older continents were not stored in memory at the time.  They'd get left behind in a major way.

And none of this addresses the problem of disk space at all, so backups still take linearly longer, etc.  Which is still really annoying!

WHAT REALLY NEEDS TO HAPPEN, IN SOME FORM
-------------------------
Old chunk data, and old region data, needs to get purged from the world at some point.  Plain and simple.  There are things that players simply don't care about, and that stuff should get removed so that there isn't linear growth anymore.  This does NOT mean removing old settlements or anything like that -- even key locations like old overlord keeps are trivial to keep, in terms of their data.

What I think is that there are some "key locations" that players wind up having longer-term interest in and wanting to maintain, while the rest of the "I was just off doing some scavenging or a minor side mission" kinds of areas can be dumped because the player has already forgotten about their details anyhow (and probably wouldn't recognize them if they returned to them anyway, if it's been long enough).

So.  This is what we're brainstorming about, and why.


MY IDEA: ONGOING CATACLYSM
-------------------------
Okay, so this is a post-disaster world, right?  But in a lot of respects the world really still is in the midst of a disaster.  In another thread, Keith and I were talking about having disasters that would strike older continents to up their region levels and basically those would purge whatever had been there.

Well... what if we took that a lot further than just meteor strikes or flooding or whatever, and made it so that after a certain amount of time had passed that continents became volatile and started to sink into the sea?  Bam, that's a huge purge of data right there. 

Once you have something like 5-6 continents, the oldest one goes into a crisis and will sink into the sea after some amount of mission time.  What do you do in the interim?  Well, you now have the fun of trying to rescue whatever parts of the continent you care about. 

Maybe you can magic the settlement(s) from there to any of the other continents of your choosing to save them and the citybuilding stuff that was going on with them.

Or maybe it's really more of a loss of the settlement itself, and you're just saving the people, but they bring across their know-how and goods and stuff and are able to quickly rebuild an equivalent-but-different-looking settlement in a new place.

Maybe you can choose a couple of other regions to also magically move to safety of another continent, but at the cost of a unit of mission time for each region you're saving.  So if there was some really iconic overlord keep that you want to retain access to, or you've got a near-infinite cave system that you're down 15 caverns into, you could move those to safety before the rest of the continent goes down into the sea.

Or if you're not sentimental, you just let the whole thing sink into oblivion and all those older NPCs die, too.  It's up to you.

But either way your world stays leaner in filesize, and actually with this sort of system the amount of playtime in this world moves a lot closer to infinite -- you'd start having problems at about region level 2.3 billion, rather than civ level 5,000.


Thoughts?  Other ideas?
: Re: Brainstorming: Curtailing Infinite World Folder Size Growth
: Bluddy December 01, 2011, 10:23:08 AM
Your solution is exactly what I was thinking of as I read the description of the problem.

The problem I see is, why are you fighting the overlords when the cataclysm is going to destroy anything anyway?

Maybe this can be turned into a more central gameplay dynamic? The overlords aren't just oppressing the people, they're blocking off their escape route (which is magical and transports the whole settlement, or perhaps not -- just the people escaping). The thing tearing apart the world is the mystical "wind", and your windmills can only hold it back for so long. Your goal is to eventually find the mythical valley without wind, from which you won't have to escape any longer.

One effect that could happen over shorter time periods is that if you come back to an area you've already visited after a while, it'll be more run down. There will be more destroyed rooms and buildings. Chaos is slowly eating away at this world and the only thing you can do is escape. Eventually random tiles would disappear and whole continents would be torn apart.
: Re: Brainstorming: Curtailing Infinite World Folder Size Growth
: x4000 December 01, 2011, 10:46:06 AM
That's a really interesting twist on it!  I like how that ties together the wind and the overlords with this other new concept for the continents.  Very cool!
: Re: Brainstorming: Curtailing Infinite World Folder Size Growth
: Underfot December 01, 2011, 11:04:45 AM
I think I'd be okay with losing regions from 100 game play hours ago and given how much of the moon is missing, I imagine a few de-orbiting chunks could wipe out a continent...
Rescuing a settlement from a continent could range from an enormous undertaking (everybody pack up and get on the Ark, the rains are coming) to a god-like super spell (enlist the help of multiple Ilari to teleport things over).
Thematically, I  like Bluddy's fleeing disaster idea.
: Re: Brainstorming: Curtailing Infinite World Folder Size Growth
: TNSe December 01, 2011, 11:38:47 AM
Considering everything is procedurally generated, isn't it enough to save the seed and the relative state of objects compared to the seed? (Thus regenerating this if it is lacking).

Kinda like storing the delta from original seed to current state. (Or did I read something wrong)
: Re: Brainstorming: Curtailing Infinite World Folder Size Growth
: x4000 December 01, 2011, 11:45:55 AM
Considering everything is procedurally generated, isn't it enough to save the seed and the relative state of objects compared to the seed? (Thus regenerating this if it is lacking).

Kinda like storing the delta from original seed to current state. (Or did I read something wrong)

That only works if you never upgrade the game at all.  As soon as there's a new version and anything new that could go in that region, the from-seed-only model breaks down.  As I noted above, there's actually more than a bit that we DO reconstruct from seeds only, but it's the only stuff that is safe to do so.
: Re: Brainstorming: Curtailing Infinite World Folder Size Growth
: TNSe December 01, 2011, 11:57:39 AM
I see, there are some solutions to that too, but sadly they would at one time or another end up limiting you.
: Re: Brainstorming: Curtailing Infinite World Folder Size Growth
: x4000 December 01, 2011, 12:14:14 PM
I see, there are some solutions to that too, but sadly they would at one time or another end up limiting you.

Yep, or costing more data -- or, worse, dev time -- to store them in order to try to solve them.  Also, I didn't mention the fact that players can add 9or delete) their own custom content in terms of the xml population files and the room map templates and all that.  Just changing those things around changes how the seeding happens in pretty major ways, so if that's not baked in then you'd see massive swings every time it tried to recreate that stuff.
: Re: Brainstorming: Curtailing Infinite World Folder Size Growth
: Hearteater December 01, 2011, 12:14:41 PM
I think you've probably hit the best solution for world size on disk by removing older sections either through in-game cataclysms and/or by just purging detailed data of boring old chunks.

For in memory, is there a reason you need every player to have all the same stuff in memory?  Especially if they are on different continents.  It seems like the ability to affect each other is very very low and so they don't really need to have all that in memory.  They only need to pass up changes that need to happen.  I'm guessing you keep the sections in memory so you can apply those changes, but you don't have to do that immediately.  You can store all the changes made (in RAM and/or disk) and processes them as RAM permits.

For example, say 20 people are playing on different continents, but I only have the RAM to hold 10 continents at once.  As changes come in for continents I don't have in memory, I set them aside (possibly to disk) for later handling.  Periodically the game drops one of the 10 continents in memory (not the one my player is in) and loads one of the other continents with a lot of changes pending and processes those.  Basically this is manual memory paging, and not ideal, but at least it scales on disk space instead of RAM and more importantly the heap limit.

For settlement trading, could you pull the settles out as a separate block of data with a much smaller footprint than the continent so you can always interact with all settlements because they are all in memory?
: Re: Brainstorming: Curtailing Infinite World Folder Size Growth
: x4000 December 01, 2011, 12:25:20 PM
Hearteater: A lot of that gets kind of tricky as the game evolves.  You wind up later with "oh crap, no settlement trading is possible because of the way we're managing RAM" type situations.  Or "sorry, can't migrate NPCs between continents!"  Or whatever else.  Such as, taking a boat ride as discussed on the continents thread would be impossible, it would have to just warp you between continents because trying to stream in all the continents belatedly (only to drop them once you arrive) would make boat rides incredibly painful to initiate.

Plus, if we ever do point to point warps that cross continents.  Or a whole other mess of things.  It winds up being something that impacts the design of the whole game, usually in a negative way, on an ongoing basis.  When instead it really should be a little more freeform, ideally.  RAM management techniques should just kind of fade into the background.  Which works with the ones we currently have, because we keep the broad metadata that we might need in a central place, such that you really don't have to have any of the other region data until you physically go there.  But in terms of making metadata about the metadata, or heavily segmenting the metadata... well, it really makes it a lot less of one world, and more a collection of smaller isolated worlds, if that makes sense.

Which is what we were considering doing, but once I did the math I started thinking that wasn't even needed, essentially.  Plus, players with enormous worlds have already complained some in terms of having trouble just remembering where everything is.  In the early alpha, c4sc4 racked up a world that had something like 12,000 regions in it two months ago.  And he's still playing on that world, but it's just incredibly too vast in a negative way I think.  If you're trying to mentally keep track of 10 continents and what is going on on each of them, I think that's really quite a lot to go with, even in the late game.  Closer to 5 would seem to be an ideal number, and matching what you see with a lot of JRPGs in particular, it seems.
: Re: Brainstorming: Curtailing Infinite World Folder Size Growth
: Dizzard December 01, 2011, 02:27:26 PM
The central world index stuff contains enough to run the world map, to have NPCs be able to talk about the overlord by name

Does an npc in one continent need to be able to talk about an overlord in another continent though? Npcs are obsessed enough with the overlords without them talking about ones across the sea too. I see your point though....to some degree npcs and the game in general will need to be aware of things happening in different continents.

2. Additionally, as the world gets progressively larger, it is taking up more disk space.  Mainly in terms of the per-region detail data (mostly chunk files).  This isn't a huge problem, as I noted above, but it is annoying.  If you want to periodically back up your world folder, for instance, it's going to take progressively longer and longer to back up the larger your world gets.  And the most annoying part is that most of the data that it's spending so long backing up is old areas you haven't visited in dozens of hours, and probably never will visit again.

Even now at around lv 140 it takes a little bit longer to backup then I'd like. (would be nice to maybe see some kind of progress bar at some stage)

As for the problem, I might be far out with this....but what if each region tile had an age associated with it. A region past a certain age (time since last load) wouldn't be seen as necessary or important by the back up process. So if your world has a major error it doesn't matter if these regions are replaced by fresh ones when you back up your game. (Players could also have the ability to mark regions as special so they would stick around even if you don't go near them for a while) Although applying an age to each region and keeping track of it might be more trouble than it's worth.

And there could be some severe limitations in what we can do cross-continent if we ever felt like implementing trade or something, since we can't do anything with settlements that aren't in RAM because they're on a different continent.

Coming back to solo play, when you think about mission time and how that's something that causes new side missions to pop up and disappear, that's another thing that would be somewhat problematic if older continents were not stored in memory at the time.  They'd get left behind in a major way.

This would be annoying, I'd hate to see things like trade or cross continent relations being excluded because of this. It would be a major shame and make the continents seem more like individual mini worlds.

What I think is that there are some "key locations" that players wind up having longer-term interest in and wanting to maintain, while the rest of the "I was just off doing some scavenging or a minor side mission" kinds of areas can be dumped because the player has already forgotten about their details anyhow (and probably wouldn't recognize them if they returned to them anyway, if it's been long enough).

There are many regions just sitting there that I have no interest in entering because their level is now so far below mine. I think there are some lv 1 areas I haven't entered even. The only exception to low level areas is if they have a resource of some kind (and if a mob heading for my settlement is hanging out there)

Okay, so this is a post-disaster world, right?  But in a lot of respects the world really still is in the midst of a disaster.  In another thread, Keith and I were talking about having disasters that would strike older continents to up their region levels and basically those would purge whatever had been there.

I'm behind this, regions that are so below my level are entirely irrelevant and serve no real purpose other than to have something there as background.

Well... what if we took that a lot further than just meteor strikes or flooding or whatever, and made it so that after a certain amount of time had passed that continents became volatile and started to sink into the sea?  Bam, that's a huge purge of data right there. 

Once you have something like 5-6 continents, the oldest one goes into a crisis and will sink into the sea after some amount of mission time.  What do you do in the interim?  Well, you now have the fun of trying to rescue whatever parts of the continent you care about. 

Maybe you can magic the settlement(s) from there to any of the other continents of your choosing to save them and the citybuilding stuff that was going on with them.

Or maybe it's really more of a loss of the settlement itself, and you're just saving the people, but they bring across their know-how and goods and stuff and are able to quickly rebuild an equivalent-but-different-looking settlement in a new place.

Maybe you can choose a couple of other regions to also magically move to safety of another continent, but at the cost of a unit of mission time for each region you're saving.  So if there was some really iconic overlord keep that you want to retain access to, or you've got a near-infinite cave system that you're down 15 caverns into, you could move those to safety before the rest of the continent goes down into the sea.

Or if you're not sentimental, you just let the whole thing sink into oblivion and all those older NPCs die, too.  It's up to you.

Hmm this interests me and sort of scares me at the same time. For one thing if you're really driving home the whole "survival" aspect the whole need to evacuate npcs from settlements falling into the sea would work really well....but from a civilization point of view it might be in danger of making things look a bit weak and frivolous if not done correctly.

I'd consider myself to be extremely sentimental with things like this. So in some ways it would feel like the game is directly attacking the vision I have amassed over time. I could totally see myself with settlements called "New new new new new Mahbakk" (The capital of my current world :3) You can just see from the ":3" how much I dote on little things like settlements and the people. While other players may be all for the action I do place a lot of value in the people of my civilization and their home. The progression of my civilization and it's people are the main reason I feel motivated to go exploring and defeat the overlords.

On the other hand I can definitely see the entertainment value with all the evacuating npcs and trying to save peoples lives from impending doom. Surviving with your npcs could make those npcs all the more special.
: Re: Brainstorming: Curtailing Infinite World Folder Size Growth
: c4sc4 December 01, 2011, 02:39:00 PM
What about just deleting data for any regions that are 20+ levels below the current civ level? There really is little need to ever go back to those levels, especially because any tiered items in those levels would be worthless. It could fit in with the cataclysm where those tiles would be destroyed, and on the world map they could have a special tile, like a glowing red cracks or something. That tile would essentially be un-enterable and just allow for ways to cross on the world map. Special things like settlements and overlords could be saved from this and perhaps a 3x3 square of tiles around each settlement so you could fight of rampaging monsters.
: Re: Brainstorming: Curtailing Infinite World Folder Size Growth
: x4000 December 01, 2011, 03:17:24 PM
As for the problem, I might be far out with this....but what if each region tile had an age associated with it. A region past a certain age (time since last load) wouldn't be seen as necessary or important by the back up process. So if your world has a major error it doesn't matter if these regions are replaced by fresh ones when you back up your game. (Players could also have the ability to mark regions as special so they would stick around even if you don't go near them for a while) Although applying an age to each region and keeping track of it might be more trouble than it's worth.

Well, that would help with backup speed but not much beyond that really.

Hmm this interests me and sort of scares me at the same time. For one thing if you're really driving home the whole "survival" aspect the whole need to evacuate npcs from settlements falling into the sea would work really well....but from a civilization point of view it might be in danger of making things look a bit weak and frivolous if not done correctly.

I'd consider myself to be extremely sentimental with things like this. So in some ways it would feel like the game is directly attacking the vision I have amassed over time. I could totally see myself with settlements called "New new new new new Mahbakk" (The capital of my current world :3) You can just see from the ":3" how much I dote on little things like settlements and the people. While other players may be all for the action I do place a lot of value in the people of my civilization and their home. The progression of my civilization and it's people are the main reason I feel motivated to go exploring and defeat the overlords.

On the other hand I can definitely see the entertainment value with all the evacuating npcs and trying to save peoples lives from impending doom. Surviving with your npcs could make those npcs all the more special.

The sentiment of exactly that civilization is part of why I was suggesting just transplanting the entire region.  It's also possible that old regions that "sink into the sea" could actually work a little differently than I was proposing.  They could literally sink into the sea except for the key locations (settlements, maybe a few other things), which stay on the surface and then there's some mud flats that is level-less and that you can't venture into, which takes the place of the actual land that had been there.  It still lets us dump the old regions, and the metadata would be very small, but it wouldn't completely remove the old landmass.

So there are several ways we could go.  I figure that actually transplanting the region onto a new continent with higher-level stuff around it would be the most interesting way to do it because then you could actually keep doing side missions that would help that settlement, so that you could grow it further.  Frankly, maybe a way to do a "settlement swap" so that your best settlement carries forward and then you have to rescue the few strangers from the new settlement that is about to sink into the sea, would be good. ;)
: Re: Brainstorming: Curtailing Infinite World Folder Size Growth
: FallingStar December 01, 2011, 03:31:01 PM
Wanted to post a few thoughts here.  Ahhh the issues with infinite worlds and finite systems.

First off, a few alternatives that went swimming through my mind.  The first one was what I first thought of when I even heard of continents, before hearing about the cleanup need.  I was thinking that perhaps old continents would get abstracted in some form, rather like in AOE3 or Civ Colonization, where a European city exists, but you don't actually actively play there.  Its there for trade or trophies, but not actually a place that you go and adventure.  Probably only for the "oldest" civilizations (ie 5-6 continents ago), but at least it would keep them in a way, for attached players.

Another thought as I read through was a mutable world system.  The chunks/ regions on old continents would slowly be changed/ transformed/ altered by the saved settlement there.  So you could still go there and check things out, but it would be new regions on the old landmass.  I think the trick would be saving the regions players cared about (like if there was one that players had used the shrink/ enlarge spell to add trophies).  Or the settlements themselves.  I'm not sure if this would retain enough of the flavor of a "legacy" continent to justify the effort, and it wouldn't save as much on the world.dat probably.

If it does go to a more cataclysm based system, I think it would be a bit sad to lose those old cities /settlements /whatnot.  It is a more "well why did I save you and why save these people if the same happens to them" thing that impinges on the infinite world satisfaction.  A player does ultimately want to save the princess, even if we are told she's in another castle a number of times.

Bluddy's twist helps on that, though I'd chip in a few things.  First it seems a bit odd for the Overlord to do something like this, as they're stockpiling and the like, seems like they're more robber barons than trying to wipe out humanity by drowning us.  Also would make them seem a bit more like puppets of some other over-overlord to get them all to do this to various continents.  I guess rewording lores and the like would help.  An alternate twist might be that overlords were maintaining the continent, in their oppressive way, and in killing them the continent now decays.  Kind of blur the line between what is good or evil.

Either way if there is permanent continent loss, and the idea is seeking out a valley without wind . . I think that its important that it actually exists.  Not as something eternally dangled in front of us, but perhaps as an epic sidequest or whatever after killing an overlord, being able to seek out and claim that trophy continent.  Perhaps you could garden the regions to particular chunk types (mentioned that idea before but seems to work here, and would help refresh old regions/chunks on an otherwise unchanging place without feeling like you were destroying them for the player).  The player might also be able to move the populations or entire settlements over, etc.  In this case, even if there was an area that was permanent, it is at least finite, so is less of a long term issue.  And it has a player satisfaction of grabbing that long term goal, even if there's still things they can change to it /move to it/ help or hinder other parts of the world to grow your own chunk.  But in essence transferring over the emotional attachment to a limited space that can stay eternally, and grabbing the chunks of old sentiment to save as well.

Anyways, so a boatload of brainstorming thoughts, hope some help somewhere.
: Re: Brainstorming: Curtailing Infinite World Folder Size Growth
: TNSe December 01, 2011, 03:32:30 PM
Mudsquares, maybe settlements after a while when civ level becomes significantly higher, simply take control of these squares in a way such that everything hostile in there is removed, and things that can be gathered is gathered.

If anything, these generic tiles have less information tied to them, and will improve compression.
: Re: Brainstorming: Curtailing Infinite World Folder Size Growth
: tigersfan December 01, 2011, 03:43:28 PM
I agree with FallingStar that it does make sense that the Valley Without Wind that we're all striving for does really exist. Maybe if we combined that with bluddy's idea, and every so many continents the NPCs actually FOUND this valley.... But wait, the cataclysm has brought out MORE landmasses, and the citizen of those now need to be brought to this valley. Thereby making the Valley Without Wind sort of like a trophy room with all the NPCs we've saved.
: Re: Brainstorming: Curtailing Infinite World Folder Size Growth
: Dizzard December 01, 2011, 04:54:03 PM
I agree with FallingStar that it does make sense that the Valley Without Wind that we're all striving for does really exist. Maybe if we combined that with bluddy's idea, and every so many continents the NPCs actually FOUND this valley.... But wait, the cataclysm has brought out MORE landmasses, and the citizen of those now need to be brought to this valley. Thereby making the Valley Without Wind sort of like a trophy room with all the NPCs we've saved.

Sounds very interesting. So it would be like a safe haven among all the chaos. Some continents would have "valleys without wind" and others (most others) would not...so the settlements/outposts in those regions would be volatile and unsafe so they would need to be evacuated to the rare but stable valleys. Perhaps eventually entire continents would fall apart (even resources/overlord/settlement tiles) and it would be your job to transfer what you want to save to these (would there be more than one?) valleys. It could be like the town building location management system now but on a larger world scale.

I'm not really sure about falling stars' A0E3 idea, turning settlements and npcs into "trophies" that are hidden away in the background doesn't seem like a very good trophy to me. I'm supposed to be happy with getting a trophy but I wouldn't be happy with my settlements and npcs getting locked away in a display case.

As for what X4000 said about the regions surrounding settlements half sinking into the sea and turning into mud regions. It does seem like the settlement is being left behind in this really rubbish location where the whole place is barren and nothing exciting will happen. I think I'd sooner move the entire settlement tile or even just move the settlers to a new continent.

As for the decaying of continents lore wise, you could always bring out the dark illari we've heard about in the deep.
: Re: Brainstorming: Curtailing Infinite World Folder Size Growth
: Bluddy December 01, 2011, 05:00:29 PM
Reading the great ideas here made me think of the new Battlestar Galactica, and how Earth was dangled in front of us several times until the real one was found. Based on Chris's clues about how parts of the plot are procedural, what if the existence of the Valley is randomly determined? In some games, we could think that we found it, only to find out (in some really painful way) that it wasn't the panacea we were looking for. In other games, it could be found fairly early on. Other games would drag on forever without finding it, and our citizens would become more and more cynical and unhappy as a result. We might be driven to doing more and more desperate things (quests). In some versions, the valley could sometimes be revealed to be 'in our hearts' (cheesy but you get the idea). In other games, the valley is so twisted that its power, once unleashed, completely shatters the continents.

It's just an idea, and I think that this point is not necessarily a big one as far as the game is concerned, but it could insert a fun little element.
: Re: Brainstorming: Curtailing Infinite World Folder Size Growth
: Dizzard December 01, 2011, 05:05:31 PM
Reading the great ideas here made me think of the new Battlestar Galactica, and how Earth was dangled in front of us several times until the real one was found. Based on Chris's clues about how parts of the plot are procedural, what if the existence of the Valley is randomly determined? In some games, we could think that we found it, only to find out (in some really painful way) that it wasn't the panacea we were looking for. In other games, it could be found fairly early on. Other games would drag on forever without finding it, and our citizens would become more and more cynical and unhappy as a result. We might be driven to doing more and more desperate things (quests). In some versions, the valley could sometimes be revealed to be 'in our hearts' (cheesy but you get the idea). In other games, the valley is so twisted that its power, once unleashed, completely shatters the continents.

It's just an idea, and I think that this point is not necessarily a big one as far as the game is concerned, but it could insert a fun little element.

It would be fun, but I think it goes against one of Arcens "Immutable Design Goals".

4. Players will never have to start a new world in order to try out a new feature.  All new features will be available as in-game choices in any world.

If the above idea was included, people would be inclined to create a world, play it for a while, realize it must not have the "true" valley and then create more worlds (in a cycle) trying to get a hold of the true valley.
: Re: Brainstorming: Curtailing Infinite World Folder Size Growth
: Martyn van Buren December 02, 2011, 01:13:33 AM
I agree with Dizzard about being unhappy to see all the areas I've made safe eventually being destroyed.  How about a few different end states for different continents?  Some could sink --- which would be an unfortunate catastrophe, but others might have a happy ending --- say, being domesticated into un-adventurable farmland.  Memory-wise it would be equivalent to mud flats, but I'd be happier to have "This place is free; there is nothing for you to do here, glyphbearer" rather than "Sorry, all this land you saved sank into the sea" as at least a possible outcome.
: Re: Brainstorming: Curtailing Infinite World Folder Size Growth
: x4000 December 02, 2011, 09:20:16 AM
I like that a lot!
: Re: Brainstorming: Curtailing Infinite World Folder Size Growth
: Dizzard December 02, 2011, 01:18:18 PM
Doesn't it just fly in the face of what you were trying to achieve inside buildings though? Like having each door lead somewhere even though you wouldn't want to go in every room. (the destroyed ones)

It seems like a similar situation with these un-adventurable farmlands/mud flats. They would be like doors that just happen to be locked.
: Re: Brainstorming: Curtailing Infinite World Folder Size Growth
: Bluddy December 02, 2011, 03:05:31 PM
I agree with Dizzard. To me, seeing a tile I did something in and not being able to access it is worse than destroying it. The interesting thing about the tile was being able to enter it. Not being able to access the tile all of a sudden breaks the illusion that the map represents a continuous world.

Additionally, a huge map with thousands of tiles is a drag in any game. Even strategy games suffer from maps that are too big and unwieldy. In Civ, you can barely keep track of what's going on in the later turns. So I consider it fortunate that space constraints have forced a trimming mechanism. Those tiles have to go! I don't care too much what the mechanism is.

Here's another idea I just thought of. What do those jumbled up tiles from different time zones suggest to you? A variation on Tetris or Bejeweled, anyone? It sounds a little crazy, but what if you had a mission in one of several tiles. Solving the mission would return the tile to its proper timeline or something like that -- it would disappear. The remaining tiles would then move in to occupy the missing space -- space and time are collapsing in on themselves. If you manage to line up and clear several tiles in a row, you get special bonuses. One of your subgoals (which you don't have to care about too much) is to return every tile to its proper time. A macro puzzle game mechanic also ties in well with the concept of infinite worlds -- Bejeweled can go on forever.

This wouldn't be the main mechanic, but it would take care of making the map constantly shrink while allowing you to see more and more interesting places. Also, settlements that were far apart could be brought closer together. And it really suits the LOOK of the map. The map really looks like jumbled up puzzle pieces.

I have no idea how to tie this to the wind or overlords, but I'm sure some more thought could allow for that too.
: Re: Brainstorming: Curtailing Infinite World Folder Size Growth
: FallingStar December 02, 2011, 07:11:25 PM
I suppose everyone will find different things satisfying or disappointing no matter what happens, just since getting rid of some of the old data has to happen, but everyone gets attached to different things.  I'll put out there what it is for me, maybe it'll give an alternate solution (or insight).

On the most fundamental levels, for me "satisfying" breaks down to a few points: 1) I want those old continents to have a meaning to my present character (other than being stepping stones to where I am now)   2) I'd like to feel like I changed something in those continents, allowing me to rescue/save them . .  and now the people are happy and such, or at least can be moved somewhere they can be.  3) I'd like something that I could look back at and generally remember the interesting decisions /tough bits that I chose that made that continent feel like an interesting journey.

There are some things I don't need to feel satisfied.  What I don't feel like I need is something that lets me go sightseeing again . .I'd hope the new continent has plenty of interesting areas that I don't need to see old ones again specifically (or I could backup my world if I really had to see them).  I'm not attached to the trees in that first chunk I ever entered much more than I am of the layout of the first AI war map I started.  Also since I'm not really building things like Terraria or Minecraft, I tend not to care about specific regions.  I guess if there's a section with a lot of items I shrunk/ restored for decoration I might care, but that's probably my settlement and so perhaps those are the only regions I really care about.  There might be a few "cleanup" duties I might want to do (ie rescue the discovered stranded people I haven't shipped to a settlement, kill those last vortex pylons) but it wouldn't bother me if there was an auto cleanup feature to do that stuff.

So for me, long term, there actually does come a point where I don't want to feel like disasters are befalling those old parts of the world anymore or missions come up where I need to do something to save them or my first continent will be doomed.  As I typed, I wondered if the new mission system might help make a "resolution" sort of phase for old continents.  Your decisions through the journey of completing that continent and the choices of which missions to take would ultimately lead to different outcomes / permanent static states for them to wind up in someday (or determine what you could rescue, etc).  To me it would hit my three desires.  The final states could give various bonuses long term, it would feel like I guided it to a satisfying outcome (or at least one I chose, and not something foisted/forced on me), and since the final state is based on my choices it would remind me of how I got there - especially if there was some sort of text based journal/ log of some of the events.

Anyways, its more thoughts on the elements I'd want to have than necessarily a solution, but thought I might toss it out there.
: Re: Brainstorming: Curtailing Infinite World Folder Size Growth
: Hearteater December 02, 2011, 08:40:54 PM
Maybe if there was a big map of all the continents I'd "saved" that I could look back at, sort of like the world map in World of Warcraft.  I couldn't actually visit, but I could see all the places with rough borders, settlement names next to their dots on the map, maybe towers and such.  Maybe other exploits noted.
: Re: Brainstorming: Curtailing Infinite World Folder Size Growth
: Olreich December 02, 2011, 08:46:51 PM
Garbage collection. If I haven't visited a chunk in a hundred levels, I'm sure to have forgotten everything about it (unless something special happened there, like a core mission). So, if I don't visit something for 100 levels, you can garbage collect the chunks in that region, and then leave them as nothing but a seed (as the cpu has time, spawn this to another thread when a player levels if you're an optimization junkie). This way, you can cut that linear rate significantly, as once you hit level 100, you can start culling regions the player hasn't been to in forever. You could even do it for settlements that are lost and forgotten, just keep what you need to generate it on disk. This dodges the re-gen + updates problem. Because if it generates something new, the player won't know, and probably won't care, especially if it preserves more continents for cross-continent interactions.

Lore-wise, you can chalk it up to things coming into that multiverse flux you already have with multiplayer. I don't know how specifics would work, as I'm still rather fuzzy on AVWW lore.

I dislike the idea of losing entire continents or adding mini-games/continent shrinkage too. But, this won't solve the problem, just extend it. I could go with eventually losing really really old continents, but they would have to be so old I don't even remember that they exist because they are only seen at the bottom of long lists. I would prefer something like Hearteater's suggestion of a type of map that includes those old continents you lose, but in a vague manner.
: Re: Brainstorming: Curtailing Infinite World Folder Size Growth
: Cyborg December 02, 2011, 11:13:42 PM
It would feel far more enticing if my world was improving over time, if there was incentive to go back and look at the things you had done for various areas, rather than wrecking all the effort I put in.
: Re: Brainstorming: Curtailing Infinite World Folder Size Growth
: Olreich December 02, 2011, 11:59:37 PM
Could there be any way to create a diff system like in version control systems that could track incremental changes (to save time on backups)?
: Re: Brainstorming: Curtailing Infinite World Folder Size Growth
: Dizzard December 03, 2011, 11:41:05 AM
I was making another long response to this, pressed the wrong button and then lost what I had typed. So I'll make this shorter.

1. The idea of "Sorry there's nothing more you can do here" regions/settlements bother me so much. I can imagine just thinking "BAH!" every time I looked at one rather than feeling like I achieved something good.

2. The only reason I don't like the idea of losing regions isn't because I want to go back to them, but because of the realistic vision they create of a landmass. If they disappeared it would be strange having all my settlements cooped in together in some sort of haven continent.

3. Which leads me to my idea. How about eventually condensing groups of similar regions/tiles into one mega region? So say for example there was a 3X3 group of ice age tiles....eventually this 3X3 group of tiles would be "merged" so that it would still look like a 3X3 group of tiles (on the main map) but would essentially be a 1X1 tile. No matter which of the 3x3 tiles you enter you will always be directed to the same region. (and it would say so somewhere)

(http://i196.photobucket.com/albums/aa285/Sabrewulf238/CondensedRegions.png)

So in this way you're sort of blurring the lines between connected (and similar) regions. The individual regions that were there before can be discarded. You are still able to visit these new super regions but you'll be visiting one for the entire block instead. (and there could be new special surprises in these super regions.

This way you get to keep the image of the overall world, but each attached group of similar regions only lead to one group of nodes. Sort of as if they fell in on each other. (functionally but not visually)

edit: I can't help but feel there's a major flaw in my idea though, I'm in the middle of creating a png image to illustrate my idea....but something seems off.

I think I know what's wrong, this idea assumes that there are more than one regions that have been entered by you before in this grouping of similar regions. If you haven't entered any of them or if you've only ever entered one of them. There isn't much difference. :(

I suppose if you tried a different approach the similar regions might not have to be connected but instead be within a certain radius. So all similar regions within that radius will fall into each other and all lead to the same region nodes. (but keeping the consistent image of the main map)
: Re: Brainstorming: Curtailing Infinite World Folder Size Growth
: Canisaur December 05, 2011, 12:02:01 PM
I agree with FallingStar that it does make sense that the Valley Without Wind that we're all striving for does really exist. Maybe if we combined that with bluddy's idea, and every so many continents the NPCs actually FOUND this valley.... But wait, the cataclysm has brought out MORE landmasses, and the citizen of those now need to be brought to this valley. Thereby making the Valley Without Wind sort of like a trophy room with all the NPCs we've saved.

Sounds very interesting. So it would be like a safe haven among all the chaos. Some continents would have "valleys without wind" and others (most others) would not...so the settlements/outposts in those regions would be volatile and unsafe so they would need to be evacuated to the rare but stable valleys. Perhaps eventually entire continents would fall apart (even resources/overlord/settlement tiles) and it would be your job to transfer what you want to save to these (would there be more than one?) valleys. It could be like the town building location management system now but on a larger world scale.

I second (third?) this.  You can keep the settlements that you care about by moving them while letting the game "garbage collect" most of the regions.  Maybe this could be linked to the Illari?  Certain (rare) settlements' Illari would have the ability to create a permanent "valley without wind" after you destroy the overlord, others would need to move to a more stable continent.

Here's another idea I just thought of. What do those jumbled up tiles from different time zones suggest to you? A variation on Tetris or Bejeweled, anyone? It sounds a little crazy, but what if you had a mission in one of several tiles. Solving the mission would return the tile to its proper timeline or something like that -- it would disappear. The remaining tiles would then move in to occupy the missing space -- space and time are collapsing in on themselves. If you manage to line up and clear several tiles in a row, you get special bonuses. One of your subgoals (which you don't have to care about too much) is to return every tile to its proper time. A macro puzzle game mechanic also ties in well with the concept of infinite worlds -- Bejeweled can go on forever.

I kind of like this idea, and it could even tie in with the "rare stable continent" idea.  I'm not sure how well it would tie in to the current backstory, but returning tiles to their proper timeline would be an interesting side job, even if it only increments a score counter or something like that (regions saved vs regions lost).  Rather than letting the unstable continents be destroyed, you could work to "return" as much of it as possible before that happens.  It would return whole region groups as you completed missions (ie. a 3x3 ice age group rather than each individual ice age region).  I suppose  this might get tedious after a while though.
: Re: Brainstorming: Curtailing Infinite World Folder Size Growth
: DerfK May 10, 2012, 02:23:29 AM

Here's another idea I just thought of. What do those jumbled up tiles from different time zones suggest to you? A variation on Tetris or Bejeweled, anyone? It sounds a little crazy, but what if you had a mission in one of several tiles. Solving the mission would return the tile to its proper timeline or something like that -- it would disappear. The remaining tiles would then move in to occupy the missing space -- space and time are collapsing in on themselves. If you manage to line up and clear several tiles in a row, you get special bonuses. One of your subgoals (which you don't have to care about too much) is to return every tile to its proper time. A macro puzzle game mechanic also ties in well with the concept of infinite worlds -- Bejeweled can go on forever.

I kind of like this idea, and it could even tie in with the "rare stable continent" idea.  I'm not sure how well it would tie in to the current backstory, but returning tiles to their proper timeline would be an interesting side job, even if it only increments a score counter or something like that (regions saved vs regions lost).  Rather than letting the unstable continents be destroyed, you could work to "return" as much of it as possible before that happens.  It would return whole region groups as you completed missions (ie. a 3x3 ice age group rather than each individual ice age region).  I suppose  this might get tedious after a while though.

Thinking about the shards being unstable, I wanted to throw a different idea into the ring even though it's been months... what about the shards continuing to shift in time?  The settlement itself would stay but any shard not "nailed down" by a settlement, windmill, lieutenant or overlord could slip off and be replaced, essentially re-rolling that section of the continent (possibly even with a new lieutenant if the overlord hasn't been beaten yet) and letting it ditch all the cave data.  If someone is so pleased with how their caves turned out, they can drop a windmill (or perhaps a Guardian Power specific to this purpose) on that shard.

It could even be a Guardian Power to force the system to replace one square (minor) or an entire shard (major) with a randomly selected land type. 

Maybe I'm just bitter my starting continent had no swamp, lava flats, etc ;-)
: Re: Brainstorming: Curtailing Infinite World Folder Size Growth
: omegajasam May 11, 2012, 09:16:17 AM
I actually was thinking the same thing. If the data is only generated on entry, then having the tiles (rarely/slowly) get shiffted to new ones would work fine. It would also mean if someone did go back, they just end up making new areas to explore.
: Re: Brainstorming: Curtailing Infinite World Folder Size Growth
: Kentarou July 30, 2012, 03:11:18 PM
I love the idea mentioned above about finding the Valley Without Wind. It could be a mobless, beautiful, lush sort of "trophy" type of area where, after you beat say 3-5 Overlords you unlock and all your settlements are moved to chosen tiles on that map. It could give the feeling of creating a larger population and city, (colonies placed beside each other giving the feel of a city building up) it would be finite lowering resource usage, and the rest of the continent could sink into the sea purging the data. It would give the feeling of saving the survivors and their settlements you've built so much upon all while still letting you feel that the cataclysm is still going on.

I really like this idea. But what do I know, this is my first post and I'm still on my first continent. xD Either way I'm addicted to this game and I can't wait to see the direction it will take in the future. :)
: Re: Brainstorming: Curtailing Infinite World Folder Size Growth
: TKilburn August 20, 2012, 10:33:58 PM
One of the things that Keith and I had been thinking about with the Continents idea is that it's a way for us to split up the world into yet more component files (continent files) so that not all of it has to be in RAM at once.  And not all of it has to be synced across the network to new players, and all of that sort of thing.  Of course there are solid gameplay reasons for doing continents as well, and that's what we focused on in the other thread that was specifically about continents.


Constantly splitting files will not help. Cataclysm (https://github.com/Whales/Cataclysm) did this and bad things happened (6GB saves are not fun).


Could there be any way to create a diff system like in version control systems that could track incremental changes (to save time on backups)?

This was something Cataclysm HAD to do due to the above 6GB saves.



An alternate twist might be that overlords were maintaining the continent, in their oppressive way, and in killing them the continent now decays.  Kind of blur the line between what is good or evil.

Oh a  Load Bearing Boss (http://tvtropes.org/pmwiki/pmwiki.php/Main/LoadBearingBoss), that is a good idea.
: Re: Brainstorming: Curtailing Infinite World Folder Size Growth
: Snave August 23, 2012, 02:56:17 PM
With the art revamp, it could be possible to actually show the "cracks between time shards" on the map. In this case, you could have them fluctuate a bit in regular gameplay, and then, later on, slowly widen, until everything is eveloped. What I'm saying is that the possibility could exist to show players what is happening, rather than tell them, and still wipe out huge chunks of data at the same time.
: Re: Brainstorming: Curtailing Infinite World Folder Size Growth
: Stinger December 15, 2012, 10:45:40 AM
If anyone would listen to me, I would like to offer to expand your audience by porting this game to ubuntu.  you'd be amazed how many people out there use *nux or *nix or *BSD.  If you would allow, I could compile the game to run on as many flavors of linux as you like and i could also make an installer for linux.
: Re: Brainstorming: Curtailing Infinite World Folder Size Growth
: Aklyon December 15, 2012, 11:15:03 AM
Stinger, if you'd seen any thread besides this one, you'd know that they're slowly working on getting Unity 4 working. Though it might not be for AVWW1, yet.