[Solved] OutOfMemoryException

[Kalzarius]

In the current game I am playing, I have encountered frequent crashes with an unhandled OutOfMemoryException while performing some string routine (I can't being to speculate whether the problem lies in the string routines or other memory allocation routines).  I have had this happen before with 1.008 after 6-7 hours of gameplay, but with the current pre-release (1.009F), it seems to happen after about a half hour.  Below is one such instance from the UnhandledErrors.txt:

-----------------------------------Application_ThreadException-----------------------------------System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
   at System.Number.FormatInt32(Int32 value, String format, NumberFormatInfo info)
   at System.Int32.ToString()
   at System.String.Concat(Object[] args)
   at AIWar.AIUnit.SendStateToAI(ForegroundObject obj) in C:\vcprojs\AIWar\Framework\AI\AIUnit.cs:line 229
   at AIWar.GameForm.RunNextCycle(Boolean DoRendering, Boolean DoScrollingAndInput) in C:\vcprojs\AIWar\GameFormParts\GameLoop.cs:line 772
   at AIWar.GameForm.gameLoop() in C:\vcprojs\AIWar\GameFormParts\GameLoop.cs:line 238
   at AIWar.GameForm.GameForm_Load(Object sender, EventArgs e) in C:\vcprojs\AIWar\GameFormParts\Startup.cs:line 272
   at System.Windows.Forms.Form.OnLoad(EventArgs e)
   at System.Windows.Forms.Form.OnCreateControl()
   at System.Windows.Forms.Control.CreateControl(Boolean fIgnoreVisible)
   at System.Windows.Forms.Control.CreateControl()
   at System.Windows.Forms.Control.WmShowWindow(Message& m)
   at System.Windows.Forms.Control.WndProc(Message& m)
   at System.Windows.Forms.ScrollableControl.WndProc(Message& m)
   at System.Windows.Forms.ContainerControl.WndProc(Message& m)
   at System.Windows.Forms.Form.WmShowWindow(Message& m)
   at System.Windows.Forms.Form.WndProc(Message& m)
   at System.Windows.Forms.Control.ControlNativeWindow.OnMessage(Message& m)
   at System.Windows.Forms.Control.ControlNativeWindow.WndProc(Message& m)
   at System.Windows.Forms.NativeWindow.Callback(IntPtr hWnd, Int32 msg, IntPtr wparam, IntPtr lparam)

I'm not sure if this is the result of a memory leak or if it's caused by memory fragmentation or a higher ratio of allocations than deletions.  All I do know is that I have 2GB of physical memory with 1.5GB free at the start of the game and I have allotted 4GB to the page file (although I don't believe it has ever used the page file).

If you need more information, let me know.

Regards,
 - Kal Zarius

[x4000]

Hi there,

Thanks for sending this over.  This appears to be some sort of issue with internal string concatenation -- I'm pretty sure there's no memory leak, as most players have not had this issue.  You're the second person to report it, and it only seems to happen on the host computer for the other person who reported it.

In recent releases, I've been doing less object reuse with things like shots and explosions, which puts more load on the garbage collector and causes memory fragmentation.  I had disabled the reuse because I was worried that it was contributing to a desync, but now that I've found the source of the desync (and it wasn't this), I've put that reuse mechanism back in operation and so that should cut down on memory fragmentation in general, which is most likely the issue here.

The reason that this only happens on the host, however, is because of the large amount of string concatenation that was previously being done in order to communicate data from the main thread to the AI thread.  I've now changed this to use a struct instead, so there is no longer any string concatenation for that purpose (which is faster anyway), and all of that used memory is just temporarily on the stack and then cleared away as soon as it goes out of scope, without the need for the heap to ever get involved.

Between the two of these changes, I'd expect to see this issue resolved for the few people who have reported it.  If it happens again, however, I'd love to know how much memory Task Manager is reporting as free, and reporting as used by AI War, before you kill the game after the crash.  And I'd also like to know if you are on .NET 3.5 SP1, or if you are just on .NET 3.5.  The other person is on 3.5 SP1, but I'm just curious.

At any rate, hopefully with this version it will all be moot and you won't see it again:  http://arcengames.com/forums/index.php/topic,187.new.html

Thanks for reporting this!

[Kalzarius]

That's awesome news on both fronts.

I am in fact running .NET 3.5 SP1, and this is happening on the host system (which I forgot to mention originally).

I'll let you know how this update performs.  I am hopeful that this update will also lessen some of the slowdowns I've been experiencing as of late.

Cheers.

[x4000]

Thanks for the added info!

I have still not been able to duplicate any slowdowns.  Can I ask:

1. How many players in your game at once when slowdowns happen?
2. When you hit F3 during a slowdown, can you take a screenshot and post that (if it happens again, knock on wood that it won't).
3. Can you check on AI War in Task Manager during a slowdown and see what the CPU/RAM for AI War, as well as the CPU/RAM in general, is doing?

But, hopefully this won't be an issue at all anymore.  It's almost like the machines with the most RAM are sometimes having some problems with performance with larger amounts of small objects and more needed garbage collection.  Perhaps there is something internal to .NET that optimizes differently, or maybe it allocates different-sized chunks of memory, or something.  None of my alpha machines have more than 4GB of memory, and I haven't seen this at all, which is curious.

[Kalzarius]

Well, I can answer 1 right now.  There are just two of us.  The server is only a 1.6GHz dual core system, so that is definitely a contributer.  But I believe it wasn't quite this slow prior to one of the recent updates, which may have been related to that string handling issue.  We'll see what happens next week.

 - Kal

[x4000]

Well, I can answer 1 right now.  There are just two of us.  The server is only a 1.6GHz dual core system, so that is definitely a contributer.  But I believe it wasn't quite this slow prior to one of the recent updates, which may have been related to that string handling issue.  We'll see what happens next week.

 - Kal

Okay -- thanks for that.  1.6Ghz is pretty low for a host in multiplayer if it is single-core, but depending on your map size it might not matter much.  Would you mind posting your savegame so that I can check that out to make sure there is nothing strange going on in that?

Thanks!

[Kalzarius]

Okay -- thanks for that.  1.6Ghz is pretty low for a host in multiplayer if it is single-core, but depending on your map size it might not matter much.  Would you mind posting your savegame so that I can check that out to make sure there is nothing strange going on in that?

You're fast.

It is in fact a dual core system, and you should find the saved game in my other thread.

Cheers,
 - Kal

[x4000]

Okay -- thanks for that.  1.6Ghz is pretty low for a host in multiplayer if it is single-core, but depending on your map size it might not matter much.  Would you mind posting your savegame so that I can check that out to make sure there is nothing strange going on in that?

You're fast.

It is in fact a dual core system, and you should find the saved game in my other thread.

Cheers,
 - Kal

Thanks   If it's dual-core, then that's much better.  But looking just at the size of your savegame in the other thread, it looks like there are an absolutely tremendous number of ships.  I'm working on one other issue right now, but I'll take a look at it later tonight or tomorrow and see what sort of profiling I can do with it, and will post more then.  Thanks!

[Kalzarius]

While we haven't had a chance to play through for such an extensive period of time as usual, we have not run into the OutOfMemoryException again, so it looks like this bug has been fixed.

As a side note, the improvements have also made the game run more smoothly compared to what it was doing previously.  There are still slow-downs when there are over 5000 active ships within a planet, but we believe this is due to the under-spec host machine.

[x4000]

While we haven't had a chance to play through for such an extensive period of time as usual, we have not run into the OutOfMemoryException again, so it looks like this bug has been fixed.

As a side note, the improvements have also made the game run more smoothly compared to what it was doing previously.  There are still slow-downs when there are over 5000 active ships within a planet, but we believe this is due to the under-spec host machine.

Wonderful!  Yeah, with that many active ships in a system, that's going to start really requiring something around 2.4Ghz -- that's a huge number.  But I'm really glad to hear that, overall, it is doing better and is also not giving you that error anymore.  Thanks for letting me know!

[Kalzarius]

Using the latest release (1.010), we have again encountered a stalled AI (0 AI commands issued after a half-hour of game time) and another OutOfMemoryException:

2009-07-20 20:59:06
-----------------------------------Application_ThreadException-----------------------------------System.Exception: Game Command Text:0|25|-1|-1|-1|0|0|0|-1|-1|0|
System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
   at System.String.ToCharArray()
   at System.Text.Encoding.GetBytes(String s)
   at AIWar.Compressor.CompressStringBig(String text, Int32 CompressionLevel) in C:\vcprojs\AIWar\Framework\Compressor.cs:line 15
   at AIWar.GameForm.DoFullSync(Boolean IsForSave, Boolean IsForDump, Boolean ShowDialogs, Int32 JustForPlayerNumber) in C:\vcprojs\AIWar\GameFormParts\NetworkHandling.cs:line 1516
   at AIWar.GameForm.DoSaveGame(String Filename, Boolean ShowDialogs) in C:\vcprojs\AIWar\GameFormParts\NetworkHandling.cs:line 1402
   at AIWar.GameCommand.Execute() in C:\vcprojs\AIWar\Framework\GameCommand.cs:line 694
   at AIWar.GameForm.RunNextCycle(Boolean DoRendering, Boolean DoScrollingAndInput) in C:\vcprojs\AIWar\GameFormParts\GameLoop.cs:line 480
   at AIWar.GameForm.RunNextCycle(Boolean DoRendering, Boolean DoScrollingAndInput) in C:\vcprojs\AIWar\GameFormParts\GameLoop.cs:line 486
   at AIWar.GameForm.gameLoop() in C:\vcprojs\AIWar\GameFormParts\GameLoop.cs:line 243
   at AIWar.GameForm.GameForm_Load(Object sender, EventArgs e) in C:\vcprojs\AIWar\GameFormParts\Startup.cs:line 274
   at System.Windows.Forms.Form.OnLoad(EventArgs e)
   at System.Windows.Forms.Form.OnCreateControl()
   at System.Windows.Forms.Control.CreateControl(Boolean fIgnoreVisible)
   at System.Windows.Forms.Control.CreateControl()
   at System.Windows.Forms.Control.WmShowWindow(Message& m)
   at System.Windows.Forms.Control.WndProc(Message& m)
   at System.Windows.Forms.ScrollableControl.WndProc(Message& m)
   at System.Windows.Forms.ContainerControl.WndProc(Message& m)
   at System.Windows.Forms.Form.WmShowWindow(Message& m)
   at System.Windows.Forms.Form.WndProc(Message& m)
   at System.Windows.Forms.Control.ControlNativeWindow.OnMessage(Message& m)
   at System.Windows.Forms.Control.ControlNativeWindow.WndProc(Message& m)
   at System.Windows.Forms.NativeWindow.Callback(IntPtr hWnd, Int32 msg, IntPtr wparam, IntPtr lparam)
2009-07-20 20:59:17
-----------------------------------CurrentDomain_UnhandledException-----------------------------------System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
   at System.String.Concat(String str0, String str1)
   at SlimDX.ObjectTable.ReportLeaks()
   at SlimDX.ObjectTable.OnExit(Object sender, EventArgs e)

Prior to the crash, I noticed the reused objects count go up by several thousand per second.  I don't recall seeing that before, and I wish I had a screenshot of just before the crash, but unfortunately I don't.  If I can reproduce, I will get one.  The game slowed down considerably just prior to the crash.

Now regarding the AI stall:

Since you suggested that this exception and the AI stalling may have been related, I will be including both bugs in this thread instead of continuing in the Potential 1.009F thread for the stalling AI.

It looks like no messages are queued or AI commands done.  If I can reproduce it next week, I will attach a screenshot of F3 debug after 1000 game seconds.  Since we have AI progress around 900, I would have expected to see more in 4 hours of play, but there was nothing.  Ships fought back, but there were no AI reinforcements, no raids, nor any trickling or groups of AI coming through the outlying wormholes.  Both CPU cores were at about 96% busy at the time, so it seems like it was processing something, but it certainly wasn't processing any AI commands.  When not in battle, the game was running quite smoothly even while the AI was stalled.  It doesn't seem like the AI was stalling because the CPU couldn't keep up with it, but this could have been the result of minimal locking in the AI thread.

It appears as though the AI thread is getting into an infinite logic loop of some kind where it is unable to completely post any messages or commands (but it may be continually creating them or some other objects that may be causing the OutOfMemoryException).  If the AI thread doesn't issue or process commands when there is not enough time available, then a minimum should be established to ensure that it can proceed under these conditions.

 - Kal

[x4000]

Thanks for posting, Kal.  It sounds like basically a recurrence of it getting all blocked up.  I can't think of why this would be, since it stops adding new commands into the queue when there are more than 1000 in the queue already, but perhaps this is related to non-unit-update commands or something of that sort.  I'd have to see the F3 debug screenshot from during an AI stall to really be able to tell much, unfortunately.  When I get back next week I will also take a look at everything else that adds commands into the queue, to make sure that none of them allow for infinite additions.

The reused object count going up by several thousand per second is normal, by the way -- during a big battle, anyway, that's what I'd expect to see (it's counting all of the shots and little puffs of explosion -- a dozen or so per actual ship explosion -- across all planets).

An upcoming planned change that will have the AI launch offensive attacks from planets where the ship counts are getting too high should also really help to keep the number of units reasonable and the CPU moving along smoothly even on a below-spec machine.  If I can just figure out why the AI gets clogged up on a below-spec machine in an extreme-units game, then I think you'll be mostly all set for even the marathons like you're currently playing.  That's the hope, anyway!

[x4000]

Okay, this is hopefully resolved here:  http://arcengames.com/forums/index.php/topic,473.new.html#new

It's entirely likely that will improve performance in general, if your command queue was getting too backed up that can really degrade performance.  Let me know how it does for you when you get a chance!

[Kalzarius]

I'd have to see the F3 debug screenshot from during an AI stall to really be able to tell much, unfortunately.

Here are screenshots from the very start of the game, the point nearest the AI stall, a screenshot from 600 seconds in-game, and the saved game that consistently reproduces the stall on 0.010 (at least on my system).

I haven't had a chance to test with the newest version, but will get to that shortly and report back in a few hours (or sooner if we encounter play-stopping issues).

[Kalzarius]

The behaviour seems very similar to what darke is reporting for 011E: http://arcengames.com/forums/index.php/topic,473.msg2653.html#msg2653