End game performance issues

[Netriak]

I just bought AI wars, and is a fun game.
At the start and middle of the game, the performance was fine. But nearing the end, it became abysmal. Considering I have a geforce 8800gt graphics card, far above minimum requirements, and a amd Phenom II 720 triple core processor, I found that odd. How much better should ones computer be to play without lag?
I have read the optimizing performance sticky, and it was not a lot of help. When I am just moving my fleet (3.500 thousand ships, approximately, total amount of ships in game 36 thousand according to F3, total collisions about 80 thousand.), the average processing is 20ms, rendering 60ms. Which is far above the total of 45 you mention in the performance thread. And if I give a simple move order to the fleet, the game freezes for over a second.
That is hardly acceptable performance, considering the system requirements on the purchase page.
If my fleet is actually fighting another fleet of comparable size, the performance is even worse.
What am I supposed to do? What fun is a game if it becomes unplayable near the end, so that one has trouble finishing?

[x4000]

Hi there,

There have been some discussion about this sort of thing before (differences in architecture, despite same clock speeds), as well as people discussing which specific hardware does how welll (hardware poll).  The unfortunate fact is that CPU architecture varies a huge amount, and the newer ones tend to do vastly, vastly better.  I used to be an AMD nut, but since the P4s have largely switched to Intel for my needs, so I'm not all that familiar with the more recent AMD lines.  From what I can tell, if you are referring to the Phenom II X3 720, that ought to be a hugely modern CPU that would do extremely well with the game.  So I'm not certain why there would be that much trouble.

Average processing of 20ms is really not particularly high, it is a bit on the upper end of regular full-framerate use, but nothing horribly out of line.  What is quite high, as you've pointed out, is the rendering time of 60ms.  That alone is going to have you running at half speed or worse, regardless of whatever else is going on with the CPU. There are various settings options in the settings window (marked in green) that help to improve performance of the graphics card.  I also have an 8800gt, and my machine almost never has a problem during normal play.  However, if you're staring at an entire planet that is doing battle and has 7000+ ships in total on it, you're going to see some lag from the graphics card.  Better to zoom in to parts of it on that, rather than looking at the whole planet constantly.  You can also temporarily use frame skipping in such as situation to make the graphics lag without  bogging down the simulation, so that you're playing in realtime even if the framerate is effectively halved or worse.

With this game, or any large-scale strategy game, the unfortunate fact is that you can get into some situations where performance is going to drop on anything but the most hardcore of PCs.  I know of one player who is able to (so I am told) run a game with 201,000 ships in it without noticeable lag except when he orders 4,000ish ships around at a go.  In general, if you give such massive amounts of simultaneous orders, sometimes that will indeed cause some command lag -- usually not a second, but it can happen.  But if you take that same fleet and give smaller groups of them orders, you won't have any issue at all.  Most of the time, you'll also win more if you subgroup your guys and move them around with at least a little bit of tactical positioning.


So what's the story on all this?  I figure you'd like some background, and I don't think I've collected this information anywhere recently -- I need to put it in the wiki.

As the developer (and sole programmer) of the game, let me tell you that I very much share your frustration with this sort of thing, and I always dread seeing another post like this one.  For most people, performance is fine, but the simple fact is that modern computers are highly variable and measures of gauging performance -- everything from clock speed to bus speed to RAM speed to internal architecture concerns -- paint a very muddy picture for consumers at best.  Originally our minimum system requirements were 2.4Ghz, but we are always making various performance improvements to the game and a while back there was agitation from a number of the fans for us to reduce our minimum stated requirements because a goodly number of them were playing the game on 1.6Ghz processors (Atoms, or Dual Cores -- all modern Intel architectures, as it happens; not that AMD isn't supported, but that's simply what the people were using).  Their argument was that I was turning off potential buyers with higher system requirements than they were demonstrably using to enjoy the game.  So, I reduced the minimum system requirements, and since then have had many more stories of people successfully playing it on even that sort of computer.

But.  With a game of this sort, it is very possible to get the game out of bounds of where it will run at full speed.  The 201,000-unit game was running at 800ms simulation on my quad Q6600, which was absolutely unplayable.  Of course, a recent adjustment to the game has been able to get that down to around 80ms on my machine, but there is still a 60+ second period at the start of loading up that game where the performance is unplayable, and then it gets playable after that period.  Not perfect, but playable.  This is the same scenario that runs (reportedly) flawlessly on a Q6600 overclocked to 3.5Ghz.

So this is the performance environment for the game.  Generally it runs great for most people almost all of the time.  In the really large battles, performance will drop but is generally going to be at half or quarter speed or better, and with a battle of that size it's all over in a couple of minutes in the first place -- so that sort of drop is really shortlived.  I think that's what you're seeing.  The 1-second pause you're seeing with ordering around huge numbers of units is one of two things (or a combination of both).  First, you're rather flooding the command queue, and all of the parsing and unparsing of those commands eats up some CPU.  Secondly, once those ships have their orders they do some precalculation to make sure that their collision-detection at the destination is more effective and efficient.  It saves a ton of lag as you go with that little hiccup at the start.  And it's completely avoidable if you don't give orders to quite so many ships all at one time (which, as I've noted, has other side benefits in the game proper).

Handling the performance of a game like this is something that is always a challenge, because if people play the expected way, generally their fleets are a bit more spread out so that there are not often 3,000 player ships free to make a single assault all at once.  When players do mass their fleets that much (I've seen as much as 7,000 ships on one planet), the performance does suffer.  But, it's also somewhat more shortlived of an issue because those players then tend to steamroll whatever is on the AI planet in a matter of minutes.  Unless you put that mass of ships into FRD mode on an enemy planet with thousands of ships, which is suicide for both your ships and your CPU throughput.

Please do note that of course this sort of thing isn't unique to AI War, not that that makes it a perfect situation.  I won't point fingers at other games, but part of my motivation for creating AI War (aside from my larger motivations) was to escape the poor performance that often accompanied the largest maps and the best AIs in other games.  Largely, I think I've succeeded, but you can definitely get into some rough patches if you play certain ways or have certain hardware.  Even in the worst of those sub-100k unit cases, it tends to perform as well as I've seen some other RTS games perform at their best with their top AIs and larger maps.  So I guess take that for what you will -- the threshold for "unplayably laggy" varies from player to player, but that's not something AI War is generally known as being at all; generally whatever issues have workarounds like the ones mentioned above and/or are shortlived.  And tend to do better than our competitors, if not as well as we'd ideally like them to be.  We're always adding more improvements, too; if you're not yet on the 3.016 version of the game, I'd highly suggest it, since it contains various fixes as well as performance tuning compared to older versions of the game.

I think that the main fear that people have when a post like this comes along is that performance is secretly bad and we just don't care, or that we're trying to dupe people with our system requirements, or whatever.  I can assure you that's not the case, and we're as sensitive as we can be to improving performance.  If you look at 3.016 version 1.0, for example, performance is about 8000% better (literally).  But, the opportunities for larger conflicts are also more prevalent, which counterbalances that in a few cases.

Anyway, I hope this post both helps you tweak your performance so that you can get the most out of the game, as well as put your mind at ease that this isn't something nefarious going on.  Thanks again for your support, and I'm glad that you're enjoying the game.

Chris

[Netriak]

Thanks for the very detailed response.
Of course it is impossible to calculate the path-finding for thousands of units at the same time without causing the game to freeze, but the thing I was wondering about, was that it is done at all. I cannot imagine for a second that it is vital that every single one of those thousands of units react to the order in the very same instant. Can't you simply stagger any orders given automatically, so that you do not get such a freeze? Some RTS games, such as warcraft 3, have visible delays in unit reaction speeds when there are a lot of units, but the game never freezes.
Secondly, when zoomed far out, it is only important to see where the fleets are. One hardly needs to see an icon for every single one of those thousands of ships. Would it not be possible to simply reduce details or amount of sprites if zoomed at, when looking at large fleets?
I understand that normally one does not move such large fleets at the same time often, making this not such a priority.

One other thing though, I found a Dyson sphere golem in my game, and parked some scouts there. I noticed a huge battle between the AI and the golem, with many of the golem's dyson gatling ships getting stranded due to engine damage, and many others converted due to AI parasites. Although the AI had over 2000 ships, they were only mark 2, and wiped out, ending the battle. But the odd thing was, the dyson golem and its ships ignored the immobilized captured dyson gatlings, ignored the warp portals, ignored the command center, and just sat there, waiting for new reinforcement waves to slaughter. If I would conquer this world, would the dyson sphere ignore my command centers too? Surely not.

Edit: And yes, I have the 3.016 version, and yes, the processor is the X3 one you linked to.
Also, it does look like some of the dyson gatlings have spread out from their initial wormhole, to cleanse additional systems.

[x4000]

Thanks for the very detailed response.

My pleasure.

Of course it is impossible to calculate the path-finding for thousands of units at the same time without causing the game to freeze, but the thing I was wondering about, was that it is done at all. I cannot imagine for a second that it is vital that every single one of those thousands of units react to the order in the very same instant. Can't you simply stagger any orders given automatically, so that you do not get such a freeze? Some RTS games, such as warcraft 3, have visible delays in unit reaction speeds when there are a lot of units, but the game never freezes.

Well, this is something of an edge case, as normally calculating all of them at once does not cause any problems -- normally players don't issue quite so many orders at once.  Part of the reason I am doing it this way is that it is overall more efficient; it re-uses a lot of intermediary data points when doing those calculations, and so even though it does cause a bit of lag there, the net effect is that it uses far, far less CPU overall.  The alternative would be for it not to freeze, but either for the units to be ridiculously sluggish getting to work or for the game to potentially get laggy in general (but not frozen) for several seconds.  This is something on my list to look at various ways of potentially optimizing in the future, but in 90% of the normal use cases this is the optimal way to do it; and in that last 10% it has an undesirable effect, but a very short-lived one as compared to potentially a much longer-term one.  At the moment I'm sort of playing the odds with that particular optimization, and trying to avoid introducing a worse performance issue surrounding it.  But this is something I've been mulling over the internal algorithm for for a while, and once I have a flash of how to keep the current optimizations of it while also doing something probably along the lines of what you're describing, I'll likely make that change.

Secondly, when zoomed far out, it is only important to see where the fleets are. One hardly needs to see an icon for every single one of those thousands of ships. Would it not be possible to simply reduce details or amount of sprites if zoomed at, when looking at large fleets?
I understand that normally one does not move such large fleets at the same time often, making this not such a priority.

Yes... this is true as well.  Certainly, this is something I've thought about quite a lot myself, and some other players have made some suggestions about it, too.  I really would like to do something about this, but I feel like the ideal solution is just out of reach at the moment.  I've been thinking about this one in particular for months and months, as I think it would be the single biggest improvement to the huge battles, but no solution that I've come up with yet (or that has been suggested by players) comes free of introducing issues that might potentially be more confusing.

For example, the simplest solution is simply to draw each ship icon, and then if another icon of the same sort overlaps it, to make the second one not draw.  But if one does that, then as ships move around, die, etc, they will tend to blink in and out of existence.  You also then have the issue of the various border colors and what those mean -- I suppose I could separate it by border color and icon (and icon player-color), but then you wind up with a much more complex aggregation problem that has the potential for performance issues of its own.

Of course, another solution is to pre-group the ships into visible "stacks" at the current view, and to show a count of such ships under each stack.  That would even help players understand at a glance how a battle is going.  However, those would have to be recalculated very frequently, and would have to change as zoom levels change in order to keep making sense and avoid giant gaps as players zoom in.  That would cause some degree of extra CPU load, but possibly less than the GPU gains that would be seen.  It's hard to say for sure.  The other problem is then with ship movement, too; right now it's all very smooth looking, as if the ships are moving, framerate drops aside.  But if some sort of visual aggregation system is arrived at, then you wind up with it either likely jittering around (as the grouped ships move in various different directions), or just jumping more slowly from place to place as this is updated over time.  Possibly if membership were not constantly recalculated, by the "center of mass" of the grouped ships was recalculated for every frame, then that would solve the jittering problem without undue CPU load.  But, that's going to take a lot of testing and trials.

This is probably something I should look at sooner than later, as I suspect that it is one of the main performance bottlenecks at this point.  There are lots of potential areas for performance improvement, though, and my focus has mainly been on the simulation part of the game for the last while.  In the last month or so it's been on my new "cold storage" technique that so vastly improves performance in high-unit-count games, and before that it was on my new autotargeting method that is both more accurrate, lower CPU load (by a bit to a lot, depending on the scenario), and much more responsive in larger battles.  And before that, my ongoing focus had been on range check optimizations and collision-detection optimizations.  Those were arguably the largest detriment to the huge battles, but now that I've mostly hit the end of what I can do with those subsystems for the time being, some sort of visual grouping should probably be my next performance-related project.  The center-of-mass idea above isn't half bad for a potential solution to try first, at any rate.  I'll have to set aside some time soon to see what I can do with that.

One other thing though, I found a Dyson sphere golem in my game, and parked some scouts there. I noticed a huge battle between the AI and the golem, with many of the golem's dyson gatling ships getting stranded due to engine damage, and many others converted due to AI parasites. Although the AI had over 2000 ships, they were only mark 2, and wiped out, ending the battle. But the odd thing was, the dyson golem and its ships ignored the immobilized captured dyson gatlings, ignored the warp portals, ignored the command center, and just sat there, waiting for new reinforcement waves to slaughter. If I would conquer this world, would the dyson sphere ignore my command centers too? Surely not.

No, they'll kill the command station if it belongs to you.  But, the fact that it is not destroying those specific AI buildings is also for your benefit, not the benefit of the AI: if those buildings of the AI were destroyed by the Dyson golem, then that would cause the AI Progress to go up, potentially at no benefit to you.  As it is, the Dyson is currently holding that planet hostage, possibly causing the AI to waste reinforcement points there to no benefit.  So it's all positive for you, in the end, and not a case of favoritism for the AI or something; it's actually favoritism to you, even if it doesn't look like it at first glance.  Good question, though!

[Netriak]

But what about the AI dyson gatlings? That is, the ones that are no longer of the minor faction, but of the 2 ais due to parasite use. The AI dyson gatlings seem to be at peace with the ones from the sphere. That can't be right, can it?

[x4000]

But what about the AI dyson gatlings? That is, the ones that are no longer of the minor faction, but of the 2 ais due to parasite use. The AI dyson gatlings seem to be at peace with the ones from the sphere. That can't be right, can it?

Oh, now I see.  Ah, well, actually the AI is not allowed to capture minor faction stuff, that's the error.  Even though the AI captured them, I expect they are still acting sort of like they belong to the Dyson golem.  I'll need to make those not be reclaimable.

[Netriak]

That AI gatling fleet was lethal. But making them non-reclaimable seems like a good change. Funnily enough, after the giant dyson battle, the performance was quite fine once again.

[x4000]

Well, that's good!

[RCIX]

Of course, another solution is to pre-group the ships into visible "stacks" at the current view, and to show a count of such ships under each stack.  That would even help players understand at a glance how a battle is going.  However, those would have to be recalculated very frequently, and would have to change as zoom levels change in order to keep making sense and avoid giant gaps as players zoom in.  That would cause some degree of extra CPU load, but possibly less than the GPU gains that would be seen.  It's hard to say for sure.  The other problem is then with ship movement, too; right now it's all very smooth looking, as if the ships are moving, framerate drops aside.  But if some sort of visual aggregation system is arrived at, then you wind up with it either likely jittering around (as the grouped ships move in various different directions), or just jumping more slowly from place to place as this is updated over time.  Possibly if membership were not constantly recalculated, by the "center of mass" of the grouped ships was recalculated for every frame, then that would solve the jittering problem without undue CPU load.  But, that's going to take a lot of testing and trials.

I was thinking that you should use a system like the one you use for combining range circles; how applicable is that algorithm to ship icon display?

[x4000]

Of course, another solution is to pre-group the ships into visible "stacks" at the current view, and to show a count of such ships under each stack.  That would even help players understand at a glance how a battle is going.  However, those would have to be recalculated very frequently, and would have to change as zoom levels change in order to keep making sense and avoid giant gaps as players zoom in.  That would cause some degree of extra CPU load, but possibly less than the GPU gains that would be seen.  It's hard to say for sure.  The other problem is then with ship movement, too; right now it's all very smooth looking, as if the ships are moving, framerate drops aside.  But if some sort of visual aggregation system is arrived at, then you wind up with it either likely jittering around (as the grouped ships move in various different directions), or just jumping more slowly from place to place as this is updated over time.  Possibly if membership were not constantly recalculated, by the "center of mass" of the grouped ships was recalculated for every frame, then that would solve the jittering problem without undue CPU load.  But, that's going to take a lot of testing and trials.

I was thinking that you should use a system like the one you use for combining range circles; how applicable is that algorithm to ship icon display?

It's much more complex, but that's basically what I was describing above.  You'll notice that the range circles tend to jitter when there are a ton of them at once, though, and that's the main thing that has to be avoided here.