Author Topic: Memory consumption and the next official release  (Read 5398 times)

Offline keith.lamothe

  • Arcen Games Staff
  • Arcen Staff
  • Zenith Council Member Mark III
  • *****
  • Posts: 19,505
Memory consumption and the next official release
« on: March 22, 2010, 10:35:49 pm »
So we're approaching the end of the month, and thus another official release is becoming desirable.  Things are shaping up pretty well, imo, but I'm concerned about some of the OutOfMemoryException crash reports I've seen that seem to indicate the game is hanging on to some objects a lot longer than it should (basically forever) and this gradually accumulates to the dangerous 1.5GB level after several hours of play on a big game.

I think 3.086 may have fixed a number of these issues but I really can't tell or tell how many may be left.  Generally this sort of thing isn't a problem with C#, so I'm 99% certain it's some new structure I made a mistake with.  Anyway, it would help a lot if y'all could pay attention to and post the following about your games:

1) how much memory the AIWar.exe process is using after a fresh start of the game, on the main menu
2) roughly how large the game is at the beginning of the session (foreground object count is a good measure, it counts up through that during the loading process so you should be able to see it)
3) how much memory the AIWar.exe process is using after the loading of the game is complete (wait about 15 game seconds in so that most of the initial stuff is settled)
4) once you're done playing (or if you want to take a benchmark point every few hours), how long you played for
5) how much memory the AIWar.exe process is using at the end of the session (or benchmark point)
6) after you're done, if you don't mind, quit back to the OS, restart AIWar, load the game you just stopped, and check the AIWar.exe memory usage

If the number in 6 is way lower than the number in 4, that would indicate a potential problem (potential in that it may just be a lazy garbage collector, though there are a few explicit calls to the GC, particularly right before a game-save attempt, since that typically takes a lot of memory for the string that is written to disk).

Thanks :)
Keith
Have ideas or bug reports for one of our games? Mantis for Suggestions and Bug Reports. Thanks for helping to make our games better!

Offline Kalzarius

  • Sr. Member Mark II
  • ****
  • Posts: 422
Re: Memory consumption and the next official release
« Reply #1 on: March 23, 2010, 01:49:07 pm »

1) how much memory the AIWar.exe process is using after a fresh start of the game, on the main menu
2) roughly how large the game is at the beginning of the session (foreground object count is a good measure, it counts up through that during the loading process so you should be able to see it)
3) how much memory the AIWar.exe process is using after the loading of the game is complete (wait about 15 game seconds in so that most of the initial stuff is settled)
4) once you're done playing (or if you want to take a benchmark point every few hours), how long you played for
5) how much memory the AIWar.exe process is using at the end of the session (or benchmark point)
6) after you're done, if you don't mind, quit back to the OS, restart AIWar, load the game you just stopped, and check the AIWar.exe memory usage

Thanks :)
Keith

I didn't think to grab any of that information, but I do have some information that might help.  Firstly, the system specifications (Windows 7 64-bit, 4GB RAM, 2.7GHz Pentium Dual Core).  The game had the OutOfMemoryException upon the third autosave in our current 100 planet campaign (about 2 hours in; it was probably the 4th save of the evening).  At the next autosave (after I restarted AI War on the server, which was the one that crashed), the client crashed with the same error. 

We're down now to roughly 87,000 enemy ships or about 100,000 total ships in game.  We never had this issue prior to 3.086 (I don't think we played with 3.084), and going back through past releases, we've had way more ships without issue (as many as 140,000 enemy ships, so roughly 160,000 total).  My only guess is that it has something to do with something added in recent pre-releases.  We play weekly, so on Monday the 29th, we'll both provide the specifications and I will update this post to reflect that in the event the issue has not yet been resolved.

Offline keith.lamothe

  • Arcen Games Staff
  • Arcen Staff
  • Zenith Council Member Mark III
  • *****
  • Posts: 19,505
Re: Memory consumption and the next official release
« Reply #2 on: March 23, 2010, 01:54:59 pm »
Thanks for the info.  So were the crashes actually happening during the autosave process?
Have ideas or bug reports for one of our games? Mantis for Suggestions and Bug Reports. Thanks for helping to make our games better!

Offline Kalzarius

  • Sr. Member Mark II
  • ****
  • Posts: 422
Re: Memory consumption and the next official release
« Reply #3 on: March 23, 2010, 11:09:09 pm »
Thanks for the info.  So were the crashes actually happening during the autosave process?

Based on the tracebacks, I would have to say yes.

From the server:
Code: [Select]
2010-03-23 00:41:36 (3.0.8.6)
-----------------------------------Application_ThreadException-----------------------------------System.Exception: Game Command Text:25|0|0
System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
   at System.IO.MemoryStream.set_Capacity(Int32 value)
   at System.IO.MemoryStream.EnsureCapacity(Int32 value)
   at System.IO.MemoryStream.Write(Byte[] buffer, Int32 offset, Int32 count)
   at ICSharpCode.SharpZipLib.Zip.Compression.Streams.DeflaterOutputStream.Finish() in C:\vcprojs\AIWar\SharpZipLib\Zip\Compression\Streams\DeflaterOutputStream.cs:line 155
   at ICSharpCode.SharpZipLib.Zip.ZipOutputStream.CloseEntry() in C:\vcprojs\AIWar\SharpZipLib\Zip\ZipOutputStream.cs:line 453
   at ICSharpCode.SharpZipLib.Zip.ZipOutputStream.Finish() in C:\vcprojs\AIWar\SharpZipLib\Zip\ZipOutputStream.cs:line 638
   at ICSharpCode.SharpZipLib.Zip.Compression.Streams.DeflaterOutputStream.Close() in C:\vcprojs\AIWar\SharpZipLib\Zip\Compression\Streams\DeflaterOutputStream.cs:line 484
   at System.IO.Stream.Dispose()
   at AIWar.Compressor.CompressBytesBig(Byte[] input, Int32 CompressionLevel) in C:\vcprojs\AIWar\Framework\Compressor.cs:line 41
   at AIWar.Compressor.CompressStringBig(String text, Int32 CompressionLevel) in C:\vcprojs\AIWar\Framework\Compressor.cs:line 15
   at AIWar.GameForm.DoFullSync(Boolean IsForSave, Boolean IsForDump, Boolean ShowDialogs, Boolean SendBeginFromLobby, Int32 JustForPlayerNumber) in C:\vcprojs\AIWar\GameFormParts\NetworkHandling.cs:line 1795
   at AIWar.GameForm.DoSaveGame(String Filename, Boolean ShowDialogs) in C:\vcprojs\AIWar\GameFormParts\NetworkHandling.cs:line 1650
   at AIWar.GameCommand.Execute() in C:\vcprojs\AIWar\Framework\GameCommand.cs:line 1206
   at AIWar.GameForm.RunNextCycle(Boolean DoRendering, Boolean DoScrollingAndInput) in C:\vcprojs\AIWar\GameFormParts\GameLoop.cs:line 693
   at AIWar.GameForm.RunNextCycle(Boolean DoRendering, Boolean DoScrollingAndInput) in C:\vcprojs\AIWar\GameFormParts\GameLoop.cs:line 701
   at AIWar.GameForm.gameLoop() in C:\vcprojs\AIWar\GameFormParts\GameLoop.cs:line 294
   at AIWar.GameForm.GameForm_Load(Object sender, EventArgs e) in C:\vcprojs\AIWar\GameFormParts\Startup.cs:line 276
   at System.Windows.Forms.Form.OnLoad(EventArgs e)
   at System.Windows.Forms.Form.OnCreateControl()
   at System.Windows.Forms.Control.CreateControl(Boolean fIgnoreVisible)
   at System.Windows.Forms.Control.CreateControl()
   at System.Windows.Forms.Control.WmShowWindow(Message& m)
   at System.Windows.Forms.Control.WndProc(Message& m)
   at System.Windows.Forms.ScrollableControl.WndProc(Message& m)
   at System.Windows.Forms.ContainerControl.WndProc(Message& m)
   at System.Windows.Forms.Form.WmShowWindow(Message& m)
   at System.Windows.Forms.Form.WndProc(Message& m)
   at System.Windows.Forms.Control.ControlNativeWindow.OnMessage(Message& m)
   at System.Windows.Forms.Control.ControlNativeWindow.WndProc(Message& m)
   at System.Windows.Forms.NativeWindow.Callback(IntPtr hWnd, Int32 msg, IntPtr wparam, IntPtr lparam)

And then when it crashed on the client:

Code: [Select]
2010-03-23 01:19:42 (3.0.8.6)
-----------------------------------Application_ThreadException-----------------------------------System.Exception: Game Command Text:25|0|0
System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
   at System.String.Concat(String str0, String str1)
   at AIWar.PlayerStats.GetCommaString() in C:\vcprojs\AIWar\Framework\PlayerStats.cs:line 328
   at AIWar.Player.CreateStateString() in C:\vcprojs\AIWar\Framework\Player.cs:line 3164
   at AIWar.GameForm.DoFullSync(Boolean IsForSave, Boolean IsForDump, Boolean ShowDialogs, Boolean SendBeginFromLobby, Int32 JustForPlayerNumber) in C:\vcprojs\AIWar\GameFormParts\NetworkHandling.cs:line 1763
   at AIWar.GameForm.DoSaveGame(String Filename, Boolean ShowDialogs) in C:\vcprojs\AIWar\GameFormParts\NetworkHandling.cs:line 1650
   at AIWar.GameCommand.Execute() in C:\vcprojs\AIWar\Framework\GameCommand.cs:line 1206
   at AIWar.GameForm.RunNextCycle(Boolean DoRendering, Boolean DoScrollingAndInput) in C:\vcprojs\AIWar\GameFormParts\GameLoop.cs:line 693
   at AIWar.GameForm.RunNextCycle(Boolean DoRendering, Boolean DoScrollingAndInput) in C:\vcprojs\AIWar\GameFormParts\GameLoop.cs:line 701
   at AIWar.GameForm.gameLoop() in C:\vcprojs\AIWar\GameFormParts\GameLoop.cs:line 294
   at AIWar.GameForm.GameForm_Load(Object sender, EventArgs e) in C:\vcprojs\AIWar\GameFormParts\Startup.cs:line 276
   at System.Windows.Forms.Form.OnLoad(EventArgs e)
   at System.Windows.Forms.Form.OnCreateControl()
   at System.Windows.Forms.Control.CreateControl(Boolean fIgnoreVisible)
   at System.Windows.Forms.Control.CreateControl()
   at System.Windows.Forms.Control.WmShowWindow(Message& m)
   at System.Windows.Forms.Control.WndProc(Message& m)
   at System.Windows.Forms.ScrollableControl.WndProc(Message& m)
   at System.Windows.Forms.ContainerControl.WndProc(Message& m)
   at System.Windows.Forms.Form.WmShowWindow(Message& m)
   at System.Windows.Forms.Form.WndProc(Message& m)
   at System.Windows.Forms.Control.ControlNativeWindow.OnMessage(Message& m)
   at System.Windows.Forms.Control.ControlNativeWindow.WndProc(Message& m)
   at System.Windows.Forms.NativeWindow.Callback(IntPtr hWnd, Int32 msg, IntPtr wparam, IntPtr lparam)


Offline keith.lamothe

  • Arcen Games Staff
  • Arcen Staff
  • Zenith Council Member Mark III
  • *****
  • Posts: 19,505
Re: Memory consumption and the next official release
« Reply #4 on: March 23, 2010, 11:12:56 pm »
Thanks :)  Actually it doesn't really establish where the problem is, as the save process creates a spike in memory consumption that would be the "straw that broke the camel's back" in a lot of caes, but that doesn't mean it's the save process that's being unreasonable.

Will hunt this down ;)
Have ideas or bug reports for one of our games? Mantis for Suggestions and Bug Reports. Thanks for helping to make our games better!

Offline Kalzarius

  • Sr. Member Mark II
  • ****
  • Posts: 422
Re: Memory consumption and the next official release
« Reply #5 on: March 24, 2010, 01:34:24 am »
Thanks :)  Actually it doesn't really establish where the problem is, as the save process creates a spike in memory consumption that would be the "straw that broke the camel's back" in a lot of caes, but that doesn't mean it's the save process that's being unreasonable.

Will hunt this down ;)

I should have clarified that I didn't necessarily think it was the save processing causing the crash.  I just said yes when you asked if they were happening during the save.  ;D

Offline Atrieus

  • Newbie Mark II
  • *
  • Posts: 10
Re: Memory consumption and the next official release
« Reply #6 on: March 24, 2010, 10:16:05 pm »
The only occasions where I am currently observing memory spikes is following a load or save game.

In the case of a save game there is a small immediate spike, shortly after (within 5 minutes) followed by a larger decrease.

Int he case of the a load game there is a permanent spike in memory consumption that seems to be based on the number of objects in the game loaded.  The memory allocated for the previous game is never released.

As a side note, I see marginal increases in memory that seem to relate to the number of waves, and vessels not associated with a guard post, wh, command station, etc...  it spikes on any increase of threat and does not seems to reliably release upon lowering of the threat.  I do not have hard data to confirm this at this time.

Atrieus.

Offline keith.lamothe

  • Arcen Games Staff
  • Arcen Staff
  • Zenith Council Member Mark III
  • *****
  • Posts: 19,505
Re: Memory consumption and the next official release
« Reply #7 on: March 24, 2010, 10:55:11 pm »
Thanks for the info :)

I think there's a fair bit of memory held onto by dead foreground objects (i.e. ships), though I don't know for sure.  That wouldn't be particularly new... BUT, there is a fair bit of extra info stored by the aggregate targeting stuff that may be piling up in dead ships.  Again, not totally sure how those are handled memory release wise.
Have ideas or bug reports for one of our games? Mantis for Suggestions and Bug Reports. Thanks for helping to make our games better!

Offline keith.lamothe

  • Arcen Games Staff
  • Arcen Staff
  • Zenith Council Member Mark III
  • *****
  • Posts: 19,505
Re: Memory consumption and the next official release
« Reply #8 on: March 26, 2010, 07:10:40 pm »
Has anyone seen any out of memory exceptions in the current prerelease?  We're not in a rush to get the official out (looking more like mid April this time), but we really, really want to make sure there's no memory-out-of-control stuff.

Thanks :)
Keith
Have ideas or bug reports for one of our games? Mantis for Suggestions and Bug Reports. Thanks for helping to make our games better!

Offline Draxis

  • Jr. Member Mark III
  • **
  • Posts: 95
Re: Memory consumption and the next official release
« Reply #9 on: March 27, 2010, 08:12:27 am »
Played for an hour an a half to day (completed the 120 planet map *dances*), I didn't re-load any saves once the game has started.  There were about 120-130K ships in the game at a given time.  Numbers below are from the 'Memory (Private Working Set)' column in task manager on vista

On Menu: 103Mb
Game Loaded (126K FG Objects): 530Mb

20 mins in: 790Mb
40 mins in: 840Mb
60 mins in: 950Mb
80 mins in: 1050Mb
90 mins in: 1150Mb

Quit to menu - 910Mb

Quit to desktop, loaded AI war, loaded save from 90 mins (127K FG Objects): 650Mb


If I do a lot of loading in the same session with the 120 planet game, I often get memory exceptions
« Last Edit: March 27, 2010, 08:15:58 am by Draxis »

Offline keith.lamothe

  • Arcen Games Staff
  • Arcen Staff
  • Zenith Council Member Mark III
  • *****
  • Posts: 19,505
Re: Memory consumption and the next official release
« Reply #10 on: March 27, 2010, 09:08:11 am »
Many thanks, that's exactly the kind of info I need... it shows problems, but that's what I need to know.

Just to be sure, this was 3.086, right?
Have ideas or bug reports for one of our games? Mantis for Suggestions and Bug Reports. Thanks for helping to make our games better!

Offline Draxis

  • Jr. Member Mark III
  • **
  • Posts: 95
Re: Memory consumption and the next official release
« Reply #11 on: March 27, 2010, 09:10:42 am »
3.087

Offline keith.lamothe

  • Arcen Games Staff
  • Arcen Staff
  • Zenith Council Member Mark III
  • *****
  • Posts: 19,505
Re: Memory consumption and the next official release
« Reply #12 on: March 27, 2010, 09:13:41 am »
Ah, right, that's the most recent prerelease... can you tell I haven't had my coffee this morning?
Have ideas or bug reports for one of our games? Mantis for Suggestions and Bug Reports. Thanks for helping to make our games better!

Offline keith.lamothe

  • Arcen Games Staff
  • Arcen Staff
  • Zenith Council Member Mark III
  • *****
  • Posts: 19,505
Re: Memory consumption and the next official release
« Reply #13 on: March 27, 2010, 10:21:06 am »
Running a fairly extensive analysis of all allocation now...

By the way, if anyone can get the memory numbers for multiple players in the same game, it would help determine if there's any additional bugaboos in the AI code (of course, both AI and non-AI code could be at fault here), since the AI doesn't run on the non-host machines.
Have ideas or bug reports for one of our games? Mantis for Suggestions and Bug Reports. Thanks for helping to make our games better!

Offline Fleet

  • Hero Member
  • *****
  • Posts: 633
Re: Memory consumption and the next official release
« Reply #14 on: March 27, 2010, 10:33:56 am »
Can't run any tests right now, but I've played for several hours multiplayer (with a single ally) and have not run into memory problems.