Optimizing Performance In Multiplayer.

[x4000]

This game is designed around multiplayer as a core concept, and we're already starting to see a lot of people enjoying the co-op, which is awesome.  No one has mentioned any lag problems so far, but since people are now starting to organize larger and larger games, I thought it would be a great time to do a post on the possible sources of lag and how to combat them.  This way, if you get a group of a people together and start having lower performance than you expect, you'll be able to refer to this post and hopefully resolve the problem.

Possible Sources Of Lag
1. One or more players has a CPU that can't keep up with the simulation.
2. One or more players has a graphics card that can't keep up with whatever is on their screen at the time.
3. One or more players has an Internet connection that can't keep up with the volume of commands being issued at a time.

Determining Which Player The Bottleneck Is
Unless some of the players have identical machines, odds are that one particular player has something running slower than the rest.  To identify that player or players, the host can simply look at the player names (in the Scores tab of the Stats window) to see which has the little "(WAIT)" message appearing for the longest. 

Having that message pop up for long intervals (half a second or more) is indicative of heavy lag coming from a player.  Having it very briefly flash up and away is normal (so quick you hardly see it, often for multiple players other than the host), is completely normal and just means that the other machines are running microseconds behind the host.  No worries there.

If there is visible lag but no "(WAIT)" messages are being shown at all on the host, then it's the host him/herself who is lagging.  You may want to pick a new host, or try the various lag-reduction options that will follow later in this post.

Switching Hosts Mid-Game
If you need to switch hosts in the middle of the game, that is relatively easy.  Any player can save at any time during a multiplayer game, and then any player can load that game on their machine (thus making them the host).  The rest of the players can then connect to the new host's game by direct-IP or through a VPN solution like Hamachi, same as they did to the original host.

Diagnosing the bottleneck on a machine
Take a look in the Scores tab of the Stats window.  The Frames Per Second (FPS) of each player is shown next to them.  The person with the lowest FPS is the trouble if its a CPU/GPU bottleneck.  If everyone has a semi-reasonable FPS (10 or more), then it's almost assured you have a network problem.

If you are the player with a low FPS, then you need to diagnose further.  The easiest way is to just switch to the galaxy map, or staring out into space on a really empty planet, not looking at much of anything.  Then take another look at your FPS.  Did it go way up? 

If it's now in a reasonable range, your GPU isn't up to the task of handling whatever it was you were looking at that was making the game slow.  If it's still really slow, then your problem is with your CPU not being able to keep up with the simulation.

Dealing with a graphics card or CPU bottleneck (Performance Profiles)
The game automatically responds by giving you a lower framerate when your performance of your CPU and GPU isn't keeping up.  However, in some cases this still isn't enough.  In the Scores tab of the Stats profile, you'll find a Current Performance Profile setting that you can set to a lower value.  Simply choose a lower profile setting and choose Apply.

If that works, you'll see your Game Speed value increase within a few seconds.  If it doesn't, you may need to go even further.  Assuming that the CPU/GPU load is temporary, you can later switch the game back to a higher performance profile.  Bear in mind that the performance profile is global for all players in multiplayer (this is required so that the game remains in sync).

Dealing with a CPU bottleneck if the Performance Profiles don't completely help
If the CPU is your limiting factor, then that probably means that your simulation is just too complex for the CPU you have.  We've optimized the heck out of this game in all respects, but it's still quite a complex simulation even so. Here are some things that might help:

1. Play on smaller maps -- the more planets there are at once, the more complex the simulation.

2. Use lower ship caps: the default is Normal, but Low is also available.  Only very powerful machines can handle High.

3. Try not to bunch up too many ships on any one planet.  Having 3,000 or fewer ships on any given planet is probably ideal if your computer is below the system requirements.

4. Try not to have too many large-scale attacks on AI planets at once if you are below the system requirements, since this can also cause slowdown in some cases.

Other things that help a little, but not a huge amount in most cases (only try these if you're playing on a netbook or something and everything is already floored and you're still having trouble):

L1. Avoid playing against Mine Enthusiasts, or especially the Special Forces Captains if the simulation is becoming an issue.  This doesn't make a huge difference, but can help some.

L2. Play with lower AI difficulty levels, but with perhaps HARDER AI styles if the challenge is not enough.  More difficult AIs have more ships, which can increase the simulation complexity.

In general, all of the above should really only be an issue when you are playing with a machine that is below the minimum system requirements.  If the host has a dual-core machine that is at least 2.4Ghz, and all the clients have single core machines that are at least 2.4Ghz, you shouldn't be seeing CPU bottlenecks unless you have 8 players on a 120 planet map against extremely difficult AIs, etc.

Dealing with a network bottleneck
Sometimes even nice cable connections can exhibit lower-than-ideal performance during peak usage hours, or due to temporary issues with the ISP.  It happens, same as do random routing issues on the Internet itself.  We've had a lot of problems with Comcast in the MA area in particular, but even that has been quite playable with 4 players (if it disconnects you due to a spotty outage on your ISP, just reconnnect when you get back online).  Here are things to check:

1. Make sure that the host is on the best connection.  If you don't know who has the best connection, you can run a speed test:  http://www.speakeasy.net/speedtest/.  Whoever has the highest upload rating should probably be the host.

2. If you are hosting your own Teamspeak or Skype call (or whatever else), have someone other than the game host be the host of the voice server (whoever has the second-best upload speed).  The voice server can suck down a fair bit of bandwidth itself.

3. Make sure that no one else on your local network (and certainly not on your local machine) is running any big uploads or downloads.  These can really send your performance through the floor.

4. If your wireless connection is unreliable at your house (as many tend to be), try playing on a wired connection instead.  Often the slight fluctuations and temporary stoppages of a wireless connection can wreak havoc on a multiplayer game.

Dealing with a network bottleneck: Network Skip
Network Skip allows you to increase the length of game turns by 45ms per skip.  So, when you have a network skip of +1, the turns are 245ms instead of 200ms, which means that to have no latency you need a roundtrip time of 122.5ms.  If you use a network skip of +16, the turns are 920ms, meaning your roundtrip time has to be 460ms to have no lag.  You can set network skip to whatever value gives you the best game experience, simply by using Ctrl+plus/minus while playing the game; if performance is bad at the start, then later gets better as you play, you can adjust as you go.

However, there is a downside to having longer turns (higher network skip): and that is command latency.  The time it takes for a command to be issued from when you give it is 2x the time of the the turn length (since turns are scheduled, and since the network is run simultaneous, that is unavoidable).  So, basically, you will wind up with normally it taking 400ms for the game to issue a command you give.  At +16 network skip, it would take a whopping 1.8 seconds between your issuing an order and it happening.  That can be frustrating on its own, but it is playable and if you are not a fast-clicker it matters comparably little: at least your game simulation is running at full speed, and the clock is ticking at a regular interval, etc.  It's like playing with a radio delay to mars, rather than playing in slow-mo, basically.

More information on network topics
For more information on the networking and what contributes to lag and how to mitigate it, there is a topic devoted to that here.

Dealing with disconnections
If one player loses their Internet connection and drops out, the host should save the game and then everybody should just wait for that person to come back.  When the player returns and tries to reconnect, it will make everyone go through a full resync and then the game can resume.  If two people try to reconnect at once, that can in very rare cases cause a lockup (this is why the host should save first). 

If the game locks up for the host, just kill the process (Alt-F4, or using the windows Task Manager: Ctrl+Shift+Esc) and then have the host reload the savegame and everyone will be able to reconnect just fine.  We have not seen this issue since beta, and it was rare even then, but we like to advise people to save before hot-reconnects just in case, to avoid any risk of player heartache.  If you do get a lockup on reconnect and you forgot to save first, then just load the most recent Autosave (which, unless you changed the settings, will be at most 10 minutes out of date).

That should do it!
Playing AI War in co-op is really a blast, and we've got a lot of robustness to our network code that makes even flaky connections possible to play on (unlike a number of other recent RTS games, where one disconnect is pretty final for the game).  Still, the more players you put in the game, the more the various system requirements go up.  Playing with two players instead of one will hardly make any difference at all, but playing with 8 might be a 30% or 40% increase in the underlying network/CPU requirements.  It just depends on your settings and specific configurations. 

If you meet the stated minimum system requirements for the game, then you should already have 2x as much CPU power as you are likely to need, so that shouldn't be much of an issue.  But, as always, let us know if you run into any trouble -- and most of all, have fun!