AVWW Mulltiplayer: The Shattering Of The Multiverse

[x4000]

Original: http://christophermpark.blogspot.com/2011/12/avww-mulltiplayer-shattering-of.html

On Friday we announced the first public alpha of multiplayer, and feedback on that was very positive except for one major point, which was could not have been received more negatively.  The point in question is the "multiverse thing," which is detailed here.

What Happens Next With "The Multiverse Thing"
Firstly to go ahead with the most important news, we're working on an update that will negate most of the stuff with the intentional-desync effects.  We're instead going to be moving to a model where the enemy logic is run on the server and the state will be as consistent between clients as most other games.

To accomplish this, there will have to be some slight wiggle-room in terms of monsters allowed to be in slightly different spots, but it's the sort of thing that I don't think you'll be able to tell even if you had two clients running on computers sitting right next to one another.  Keith came up with this idea over the weekend and was talking about it with players in the forum, and those who have played multiplayer so far seemed optimistic that this would address their complaints.

Timeline?
The good news about this particular fix is that it's really replacing only part of the networking model, since the networking model is already such a multi-headed hydra.  So it's possible we might be able to have this out tomorrow.

Downsides?
The short-term downside for this particular fix is that it's going to really require a lot of rebalancing of enemies, and some complete scrapping of some enemies, to make the model work.  But this is something I was planning to do anyway, just in the interest of making even the solo experience tighter and more fun.  As was discussed prior to this multiplayer fiasco ever coming up.  But we're quite confident that players will help us iron out those temporary bumps in the balance road, and both the single player and multiplayer experiences will be a lot stronger for it inside a week or two.

One more serious downside is that certain things that we would otherwise be able to do, like "offscreen spawning of enemies" for one example.  Or stuff like having bats flee from the cursor.  Or even things like having 300 eagles in a chunk like we currently do.  This isn't exactly a new sort of restriction class for us, as most action games have restrictions along these lines, and even network strategy games like AI War wind up with certain kinds of restrictions on what can and can't be done for reasons of multiplayer.

That said, after much discussion today, Keith and I have explored a lot of the various issues that arise from this change, and things that players were hoping we would change about the existing game even prior to multiplayer (monster spawners, etc), and we both are now feeling really confident that we can simply find lateral solutions to all the various issues.

For the monster spawner example, for instance, we have plans for how we'll be able to remove monster spawners (a popular idea with players) without having to do "offscreen spawning of enemies," which is something infeasible in the new model.  There are several bigger things that will be resulting from that particular change, which I won't get into here, but the general effects are that: "trash mobs" will be fading into obscurity; enemy projectiles will be vastly slower and yet more plentiful; what were formerly trash mobs will become more interesting, more powerful foes; interiors, surface areas, and undergrounds will get differentiated even more heavily; and environmental hazards of new sorts will be playing a much larger role in the game.  Most all of which were things that players were asking for, anyway.

Benefits?
The largest benefit is that we'll still be able to make the sort of game we want to make, while having it work well in multiplayer.  The performance characteristics that you're seeing now, including that extreme latency-tolerance for general gameplay, should largely remain.  Enemies will now jitter around some if you have a really latent connection, but it shouldn't be horrible and that's basically in keeping with any other action game, if not a little better than many of them.

That's part of the benefit of the existing hugely-hybrid networking model that Keith has spent the last two months implementing.  We're able to re-tool part of it without having to affect any other part of it, and the general performance characteristics still remain quite high even though we're treading into some territory now that we'd initially hoped to avoid.  This should be what players are looking for in terms of multiplayer performance/sync, I'm pretty sure, and it represents a technical middleground that until a few hours ago I didn't think would be possible to do.  But Keith's idea, plus some refinement that we came up with working through it together this morning, strikes me as really solid.  Knock on wood!

What Alternatives Were Considered?
Prior to ever implementing the model that we released on Friday, we had implemented a more traditional action game model that just performed completely unacceptably compared to solo play.  We also looked at pretty much every other major networking model that we could think of when it came to other genres that are similar to a lot of what AVWW does.  Nothing really fit this game perfectly, which is why we went with the model we did.

Side Note: We actually went with that model knowing we might have to change something about it, but we weren't sure what player reaction would be to it since no game had ever tried anything quite like that before.  So we made sure to have the general networking be as flexible as possible so that changes would be possible.  And that's part of why we didn't want to talk about the specifics of the model in advance, because we knew folks might not like the idea on the surface of it, and we wanted their feedback on the actual playtesting of it rather than the concept.

Since the release on Friday, and what can only be described as a "polite outcry" from our core playerbase about this one specific design choice (after playtesting, which is precisely the sort of feedback we were looking for), we've been wracking our brains to figure out a better alternative, and players have been making suggestions as well.  Not really any of the suggestions particularly fit with the technical constraints of this game, which are really unique and particularly challenging to work with and explain, but we did get a razor-sharp insight into exactly the sort of performance characteristics that players were expecting and where our current model let them down.

Anyway, so we've been all over the board since then, thinking of radical other models, major changes to solo play to make multiplayer fit, and even not having multiplayer at all (since if the execution of said multiplayer was going to be a detractor, better not to have it at all).  In the end, after many hours of discussion and modeling and remodeling, we came up with the above changes which are actually pretty slight.    Key to being able to settle on that model was talking through solo-affecting core gameplay changes that solo players were already asking for anyhow, and which would be more compatible with multiplayer than the current model.

Conclusion
We think you're really going to like what's coming up, and you won't have to wait long this time.  There are lots of changes coming to the game in general, as anyone who's been following the brainstorming subforum already knows.  The game is really undergoing a transformation from something more rough and alpha-like to something more polished and release-like, which is a great thing all around.

Most of those changes are unrelated to multiplayer specifically, but a lot of them actually do happen to make the new model of multiplayer easier.  And given that we keep getting comments to the effect of "this is how I was imagining the game back when I was first hearing about it" when people read about the coming changes in the brainstorming forum, I take that as another really positive sign.

We really do appreciate all the feedback, and for people taking the time to run through the early alpha of multiplayer for us.  It sounds like overall people were having a lot of fun despite being hugely frustrated with "the multiverse thing," so I think that once we get that shored up we're going to be in happy territory.

[Dizzard]

Will this mean that bosses and such will have a similar health total for everyones screen?

Most of the time when fighting bosses in a group the boss just suddenly dies when it's still at 400'000 health. Plus there are flames from dragons that sometimes hurt you and sometimes doesn't hurt you.

It would be a shame to see mp be gotten rid of, because despite all the randomness and craziness it's still a lot of fun to play with others right now. There's that connection as a community that obviously wasn't there before.

Sounds exciting though I'm looking forward to it!

[Hyfrydle]

It would be a shame to see mp be gotten rid of, because despite all the randomness and craziness it's still a lot of fun to play with others right now. There's that connection as a community that obviously wasn't there before.

I second that. Tonight we had about four players and ghosts everywhere but it was a real blast. We managed to take out a Lieutenant and an Overlord so a pretty good nights work. Multiplayer brings things into the mix that real change how the game is played.

[Orelius]

I've had similar problems.  Namely, when fighting against an overlord which had upwards of 2.5 million hp.  After a minute or so of fighting, a few of us die.  Note that the overlord has not lost much health at all, perhaps 400,000 or so hp.  When I come back, the overlord is suddenly at around 200,000 hp, making it significantly easier than it should be.  I have a feeling that this isn't intended.

I also see lots of remnants of dragon breath and miasma that shouldn't be there; it can be seen, but it can not be interacted with.  It causes major, major lag.

The boss fights, however, were amazing though.  It would be better if pausing the chunk didn't cause the game to freak out.

[Underfot]

Yeah, the overlord/lieutenant keeps were pretty fantastic.

 

... Plus there are flames from dragons that sometimes hurt you and sometimes doesn't hurt you...

Same thing with miasma blobs.  I saw them spawning on my screen from an unseen shade - presumably from another client.  Those blobs couldn't hurt me, but once we had four players in a boss room with as many shades, it was a sea of miasma.

[x4000]

All the things you guys noted above are just garden variety bugs (mantis please!) not anything to do with the old design. Those would have been fixed in either model.

And yeah, I had a blast with Keith and Josh the other day in multi, too, so it was not something any of us wanted to get rid of. But if it was goig to cause us to get blasted by players and reviewers for something we ultimately couldn't fix... Well, fortunately we figured out something that should work as a good fix.

[zebramatt]

This was one occasion on which there was actually relatively little miasma on the screen, just to give you some idea of the carnage!

[x4000]

Whew, wow!

[TechSY730]

I haven't been keeping up on the discussion, but does this mean that the "multiverse" thing where enemies and stuff don't even try to sync up positions has been poorly received? (BTW, I'm fine with that model, except maybe for bosses, which the tougher ones would require tight coordination among allies)

I'm having a hard time thinking of any other model that could work given the constraints given.

EDIT: DUH! You mentioned that in the first sentence. Yes, it was poorly received.
Now that you have a better idea of the kinds of latency you can accept and tolerate, does that mean you are trying a more traditional model again?
It seems that one of the ideas you are seriously considering is slowing down the game overall, in just about every aspect. I'm not sure how I feel about that, but it may be the only way to get a multi-player shooter with high accuracy requirements feasible.

[Olreich]

I'm guessing here, but I'm 99% sure they are going with a "server sends the path for the monster over the next second", and then your machine locally sims the monster, and your machine sends back any hits to the server. The server is also trying to pantomime all of the player positions and spells to you, but the accuracy isn't perfect with higher latency. This has less tolerance for high latency, making things start jumping around, a la any current networked action game outside of some arcade games. It does keep everything sorta synced up with location and seeing what your allies are doing.

Where I got it: They said that enemy positions will be close but not perfect, that it's similar to the single-player experience as far as the player is concerned, that high latency situations are less tolerant than before.

[Mánagarmr]

Basically, after having played FPSes and other adventure games in MP over the years and while having lag, not had much issues that couldn't be overcome, I don't think I would've accepted a shrug and the honestly immersion-breaking multiverse thing. If others can do it, so can Arcen, because you guys are fricking awesome. No joke.

I'm glad to see that you are researching alternatives, because I'd love to play this more in MP and I just wouldn't have bothered with the pseudo SP/MP model used before.

[x4000]

The perception that other people have done exactly what we're doing is frustrating and inaccurate, I have to say. Other companies have compromised their physics or whatever in different ways than we wanted to, but what became clear is that players were okay with the glitches they were used to, rather than the oddness of the multiverse model. The fact that 200ms ping is considered "bad" now was frankly shocking to me, heh.

We didn't really do any new research, we just decided to go with some pieces of prediction and smoothing that we'd previously hoped to avoid, but which players clearly prefer. There are some minor funky things that will cause, but since it's the same funkiness in every other action multiplayer game nobody seems to care. Under lag, I mean.

In terms of slowing down the entire game, that is true we're planning on slowing down shots and enemies based on feedback unrelated to multiplayer, but also going with a "more shots onscreen at once" model that is better compatible with multiplayer and which feels more like a shmup, etc. There are two levels of changes being made here, the first fairly minor one just for the enemies and networking, and the second at a design level because some of the enemies no longer make any sense in multiplayer when they are synchronous. Not a technical limitation, just a logical problem with those in multi.

So, like always, lots of things going on at once.

[Hearteater]

I've never felt AVWW fits as a shmup.  Shmup bullet speeds are MUCH slower (and as you noted, in much larger quantities).  Not to mention shmup player hit boxes are much smaller than AVWW's comparitively massive avatar who inconviently faces the largest edge of his hitbox towards all incoming attacks.  AVWW seems more a realistic FPS without the first-person part, because its spells travel at the speed of real life bullets.  You've either already dodged it before it is fired, or you get hit.

In short, very glad to hear about the changes .

[Mánagarmr]

As incorrect as our perception is, it's all we have, and more importantly, everything a prospective future player has. No matter what, I'm sorry for not liking the initial idea and I hope you forgive me for being a bloody whiny customer.

As for more shots in the air, I think that it sounds awesome. Makes for a more skilful gaming, I think. Keep up the splendid work!

[x4000]

Moonshine, I wasn't meaning that I didn't want to hear your thoughts, and you're absolutely right that potential future players are coming to this without following the forums or whatever in most cases.  What was frustrating was that in the context of the forums we still weren't able to convey the technical challenges and what actually makes things different here.  Usually we're able to have on-par discussions with players about things like game design or balance, and large numbers of the forumites "get it" immediately and are filled with great insight. 

With this one, because of the bloody complex nature of this technical issue, and the uncommonness of most programmers having any experience in that area, there was a definite barrier where we couldn't discuss this one on par with the playerbase.  The feedback was still hugely useful in terms of us being able to determine what players would and would not accept, which somewhat confirmed some of our suspicions and somewhat surprised us a lot in other areas.

But it was frustrating to have that barrier there in any case, because it led to lots of suggestions that we'd already considered (or a few that we hadn't but which were still invalid), and a general sense that we'd taken a lazy route or just not done our homework, because clearly other developers had done something similar. 

- But with something like a fighting game, for example, they have the same challenges as us with the key difference that their gamestate is incredibly miniscule -- a few stats for two characters in most cases.

- With something like Terraria or FPS games, they're using different speeds of projectiles and enemies, or not even using projectiles at all (in the case of many FPS games, it's just a ray that gets traced to determine a hit, not actually a bullet that moves and is simulated in realtime).  At any rate, the movement speeds and some of the other changes in their designs (or even things like having more limited numbers of entities active in a server at once) really makes a huge difference.

- And for something like Diablo or WoW, those have incredibly different gameplay and networking models based around their speed of casting and everything else being a lot slower, and basically not having chances to dodge in most cases.  Numbers of projectiles really don't matter because those can be precalculated and don't have to be network-synced since they can just be simulated consistently locally on all the clients and the server.  So having a ton of projectiles in that sort of sense really isn't the same sort of challenge at all, either, and the Diablo/WoW networking model is really a lot more straightforward than an FPS game in my opinion, with the exception that WoW is supporting vast numbers of players and that introduces complications that we don't get into ourselves.

But in each case we examined, every other developer took different outs that really weren't feasible with the gameplay as it stands for AVWW.  Which ultimately works out because we were planning some gameplay changes anyhow, and some of them are just simply game-design-only challenges (some of the monsters just don't make sense except when there is one player, plain and simple).  Our initial task was to make AVWW-as-it-is into a multiplayer game, and that's frankly a bit of an impossible design challenge I think.  It's part of why a lot of the RTS AI sucks in many games, because they design the units for human controllers rather than with the AI in mind.  But AI War goes way beyond that because the units are all tailored to make sense for the humans and the AI.  With AVWW, we have to redesign the monsters and the projectiles to make sense in solo and multiplayer play, as well as making some slight actual multiplayer model changes to keep things synced, etc.

Anyway, so that's some more of the background, and why I was frustrated.  I wasn't frustrated that people didn't like it, and I was really pleased actually that people gave it a shot and then told us exactly what they thought.  But the frustration came in when the usual on-par discussion about the issue wasn't able to really take place.  Some of that was probably because we were a bit too vague with exactly what the challenges were, but that was largely because it would take dozens of pages to fully explain all the ins and outs to someone not already familiar with the networking models in question and what challenges the gameplay design we had was causing to the standard models.  Not having that external expertise to draw on for once was just unexpected, that's all.