[7.028] - Crashes during play and on autosaves

[Draxis]

My group has been getting a lot of crashes, particularly when auto-saving.  Manually saving while the game is paused hasnt caused any problems yet, but the game still locks up during regular play and doesnt seem to leave any crash logs.  Attached save game and UnhandledErrors.txt.  Memory usage during lockups is about 1.5Gb.  Are there any debug modes we could run, or additional logs that might useful to you?

5/17/2014 9:53:20 PM (7.028)
-----------------------------------Error-----------------------------------Log String: OnGUI: System.OutOfMemoryException: Out of memory
  at (wrapper managed-to-native) object:__icall_wrapper_mono_string_alloc (intptr)
  at (wrapper managed-to-native) string:InternalAllocateStr (int)
  at System.Text.ASCIIEncoding.GetString (System.Byte[] bytes, Int32 byteIndex, Int32 byteCount) [0x00000] in <filename unknown>:0
  at System.Text.Encoding.GetString (System.Byte[] bytes) [0x00000] in <filename unknown>:0
  at System.Convert.ToBase64String (System.Byte[] inArray, Int32 offset, Int32 length) [0x00000] in <filename unknown>:0
  at System.Convert.ToBase64String (System.Byte[] inArray) [0x00000] in <filename unknown>:0
  at Compressor.CompressStringGZip (System.String text, Boolean IsOptional) [0x00000] in <filename unknown>:0
  at Game.DoFullSync (Boolean IsForSave, Boolean SendBeginFromLobby, Int32 JustForPlayerNumber, Boolean DoCompression) [0x00000] in <filename unknown>:0
  at Game.DoSaveGame (System.String Filename, Boolean ShowDialogs) [0x00000] in <filename unknown>:0
  at GameCommand.Execute () [0x00000] in <filename unknown>:0
  at Game.RunOneCycleOfSimulation () [0x00000] in <filename unknown>:0
Stack Trace:   at (wrapper managed-to-native) object:__icall_wrapper_mono_string_alloc (intptr)
  at (wrapper managed-to-native) string:InternalAllocateStr (int)
  at System.Text.ASCIIEncoding.GetString (System.Byte[] bytes, Int32 byteIndex, Int32 byteCount) [0x00000] in <filename unknown>:0
  at System.Text.Encoding.GetString (System.Byte[] bytes) [0x00000] in <filename unknown>:0
  at System.Convert.ToBase64String (System.Byte[] inArray, Int32 offset, Int32 length) [0x00000] in <filename unknown>:0
  at System.Convert.ToBase64String (System.Byte[] inArray) [0x00000] in <filename unknown>:0
  at Compressor.CompressStringGZip (System.String text, Boolean IsOptional) [0x00000] in <filename unknown>:0
  at Game.DoFullSync (Boolean IsForSave, Boolean SendBeginFromLobby, Int32 JustForPlayerNumber, Boolean DoCompression) [0x00000] in <filename unknown>:0
  at Game.DoSaveGame (System.String Filename, Boolean ShowDialogs) [0x00000] in <filename unknown>:0
  at GameCommand.Execute () [0x00000] in <filename unknown>:0
  at Game.RunOneCycleOfSimulation () [0x00000] in <filename unknown>:0

[keith.lamothe]

It's just running out of heap space, I'm afraid.  The compression of the saves is a third party routine and a high-point of memory usage (and thus it tends to be where it actually trips), but the overall gamestate didn't really leave it a lot of room.  Loading the save I see over one hundred thousand units are in the game, and that this is normal cap scale.  For a five-player game you would probably do better with low (or even ultra-low) caps.

There are various things I could do to reduce the overall memory footprint, but they would be likely to produce bugs and would take a lot of dev time right now, so the general approach lately has been to recommend normal caps for relatively normal SP cases and low or ultra-low caps for scenarios with much higher unit counts.

[Toranth]

It's just running out of heap space, I'm afraid.  The compression of the saves is a third party routine and a high-point of memory usage (and thus it tends to be where it actually trips), but the overall gamestate didn't really leave it a lot of room.  Loading the save I see over one hundred thousand units are in the game, and that this is normal cap scale.  For a five-player game you would probably do better with low (or even ultra-low) caps.

There are various things I could do to reduce the overall memory footprint, but they would be likely to produce bugs and would take a lot of dev time right now, so the general approach lately has been to recommend normal caps for relatively normal SP cases and low or ultra-low caps for scenarios with much higher unit counts.

Would it help if players in games with large numbers of units turned off save compression?

[keith.lamothe]

Would it help if players in games with large numbers of units turned off save compression?

That would allow somewhat larger games before it tripped over that particular thing, yes.

[Draxis]

If its just virtual address space exhaustion, would making the exe large-address aware give the process enough breathing room?  We can editbin the flag onto the exe and give it a go next time we are all online (assuming Arcen have not already tried this approach and found problems with it).

Would it be easy to get the framework to detect this kind of error, and throw a 'game is too epic, try a smaller one next time' message?  Dont know how often this error occurs for other users.  Either way, we will give the save compression a go - Thanks for the info!

[keith.lamothe]

If its just virtual address space exhaustion

It's not that, actually.  The mono heap is trying to exceed its maximum heap size (about 800MB, iirc; the total process memory also includes unmanaged memory like textures; you can see the actual heap size in-game via the F3 debug display if your mouse isn't inside where it wants to be).  Which we have never found a way of increasing, regrettably, though it probably wouldn't be far from dying anyway, at 1.5GB.

would making the exe large-address aware give the process enough breathing room?

Perhaps theoretically.  You're saying this can be applied to an already-built exe?  In this case, though, it's not the operative constraint.

Would it be easy to get the framework to detect this kind of error, and throw a 'game is too epic, try a smaller one next time' message?

Even looking at the heap size it's difficult for us to know exactly when it will hit the max heap size.  The garbage collector isn't too bright about avoiding such things (it just keeps transient allocations from becoming long-term leaks, it doesn't seem to prevent them from being short-term ones).

And once it's actually _hit_ that max size... well, I'm not sure we could reliably get any kind of logic to work, as that generally invovles trying to allocate something on the heap.  But possibly we could get the OutOfMemoryException to result in at least a somewhat different message.

[Draxis]

Ah, I can see how that would cause a problem.

Yes, the large address flag can be set after compilation, I think its just a bit on the PE header.  Back in the days of .NET 2 it was the only way to get the full 32bits in a managed application, not sure if its changed since.  All this makes me glad I went 100% native.  I just hope we wont have to abandon the game!

Thanks very much for you prompt replies, support from Arcen has always been awesome

[keith.lamothe]

I took another look at the save, and while I think you may still run into problems this may push them out far enough that you can complete the game.

For 7.029:

* The save-game process now has some extra precautions against triggering an out-of-memory exception during intense gamestates:
** The earlier changes to have the "trim heap usages during play" logic not actually do a trim operation on a list with fewer than 100 unused entries does seem to have succeeded in reducing (perhaps eliminating) the number of TooManyHeapSections exceptions, but since it does leave some memory "on the table" the save process now causes that more aggressive trimming to happen anyway. 
*** This shouldn't cause the heap-sections problem as that was apparently only happening when hundreds or thousands of lists were being reallocated several times a second in some cases, rather than just once ever minute at the most.
** In addition to garbage-collecting just before serializing the gamestate to a saved-gamestate string, it now also Garbage-Collects just before the compression of the saved-gamestate string, so that the intermediate buffers used in that serialization aren't stacking on top of the memory-intensive compression routine.

* Fixed a bug where the Special Forces could eventually build up to a really big stinking pile of Riot Starships, which places an undue load on the memory footprint due to the modules and not being subject to the "excess SF units get stuffed in carriers" rules.