System.OutOfMemoryException (3.1.2.0)

[Spymine]

-----------------------------------Application_ThreadException-----------------------------------System.Exception: Game Command Text:25|0|0
System.OutOfMemoryException: Eine Ausnahme vom Typ "System.OutOfMemoryException" wurde ausgelöst.
   bei System.String.GetStringForStringBuilder(String value, Int32 startIndex, Int32 length, Int32 capacity)
   bei System.Text.StringBuilder..ctor(String value, Int32 startIndex, Int32 length, Int32 capacity)
   bei AIWar.GameForm.DoFullSync(Boolean IsForSave, Boolean IsForDump, Boolean ShowDialogs, Boolean SendBeginFromLobby, Int32 JustForPlayerNumber) in C:\vcprojs\AIWar\GameFormParts\NetworkHandling.cs:Zeile 1688.
   bei AIWar.GameForm.DoSaveGame(String Filename, Boolean ShowDialogs) in C:\vcprojs\AIWar\GameFormParts\NetworkHandling.cs:Zeile 1639.
   bei AIWar.GameCommand.Execute() in C:\vcprojs\AIWar\Framework\GameCommand.cs:Zeile 1277.
   bei AIWar.GameForm.RunNextCycle(Boolean DoRendering, Boolean DoScrollingAndInput) in C:\vcprojs\AIWar\GameFormParts\GameLoop.cs:Zeile 724.
   bei AIWar.GameForm.RunNextCycle(Boolean DoRendering, Boolean DoScrollingAndInput) in C:\vcprojs\AIWar\GameFormParts\GameLoop.cs:Zeile 732.
   bei AIWar.GameForm.gameLoop() in C:\vcprojs\AIWar\GameFormParts\GameLoop.cs:Zeile 294.
   bei AIWar.GameForm.GameForm_Load(Object sender, EventArgs e) in C:\vcprojs\AIWar\GameFormParts\Startup.cs:Zeile 276.
   bei System.Windows.Forms.Form.OnLoad(EventArgs e)
   bei System.Windows.Forms.Form.OnCreateControl()
   bei System.Windows.Forms.Control.CreateControl(Boolean fIgnoreVisible)
   bei System.Windows.Forms.Control.CreateControl()
   bei System.Windows.Forms.Control.WmShowWindow(Message& m)
   bei System.Windows.Forms.Control.WndProc(Message& m)
   bei System.Windows.Forms.ScrollableControl.WndProc(Message& m)
   bei System.Windows.Forms.ContainerControl.WndProc(Message& m)
   bei System.Windows.Forms.Form.WmShowWindow(Message& m)
   bei System.Windows.Forms.Form.WndProc(Message& m)
   bei System.Windows.Forms.Control.ControlNativeWindow.OnMessage(Message& m)
   bei System.Windows.Forms.Control.ControlNativeWindow.WndProc(Message& m)
   bei System.Windows.Forms.NativeWindow.Callback(IntPtr hWnd, Int32 msg, IntPtr wparam, IntPtr lparam)

After continuing my current save the game crashes with the autosave after an hour or more. It can also happen if I try to save manually. And its pretty annoying 

I am playing on this PC: Q9550 / 4 GB / Radeon HD 4890 OC / WinXP

HHHHEEEEELLLLPPPPP

[x4000]

Can you post the save in question? It sounds like it must be a really huge one to cause this, or at least that would be my best guess at the moment...

[Spymine]

I sent it to your email - hope this qualifies as "post".
Its a 6 MB file.

Another thing I noted is that Riot Ships attract Bombers/Tanks like honey does bears - but totally out of proportion. I had like 1500 ships hovering around my 7 Riots (all bomber/tank ships who enter the planet with my Riot ships do not travel onward but fly towards them - and I mean ALL!!!!!! bomber/tanks - quite strange) - the Riots have double FF - maybe thats what makes them interesting. The instant you load them into transports the Bombers/Tanks leave.

[keith.lamothe]

Ok, running the save now; impressive setup you have there

The memory behavior is somewhat alarming, as it starts around 700MB and goes up to over 1GB even before the autosaves.  So I'm going to run this through a memory profiler and see exactly what it's allocating for post-load and see if there's something we can cut down.

That said, this is something of an edge-case scale wise, and it may just be the limits of a 32-bit application (in practice, about 1.5GB memory)... there are a LOT of ships in here

But we'll see what we can do

[keith.lamothe]

259,000+ ships, wow...  Well, this will be the greatest challenge yet faced by the profiling machine

[Kalzarius]

259,000+ ships, wow...  Well, this will be the greatest challenge yet faced by the profiling machine

  Holy ****. And I thought the ~140,000 ships we had were a lot.

[keith.lamothe]

Ok, this thing kept the room with the profiling computer fairly warm for about 6 hours before I cut it off to prevent the allocation log from killing the hard drive; no smoking guns as far as memory alloc.  I do intend to run another sim based on some odd memory behavior observed while it was paused, and we have some incremental improvements in mind that will help push the memory wall out a bit further, but this is looking like a "yea, that'll happen with a quarter of a million ships" issue

[RCIX]

Correct me if i'm wrong, but can't you like assemble 100k ships at a time, write it out to the file incrementally, clear the data out of memory, and repeat till it's all saved? It might increase size a little more, but surely it's worth it to be able to save as big games as you want?

[keith.lamothe]

Well, the big problem is that even aside from the save-file character buffer size, at this number of ships the in-game working set size may exceed 1.5GB.

And flushing partial save files to disk is generally doable but since we're compressing the foreground object (ship) data before writing it to disk, it's a bit trickier.  We could break it up in 20k chunks or something.   There are lots of incremental improvements that can be made to memory and cycle complexity for saves, commands, networking, etc.  Chris is working on some of those.

[x4000]

Yeah, one thing that we're going to do is switch to a binary format for savegames and network transmissions in general over the next few months, rather than a string-based format.  That will save a lot, and would push that memory ceiling out further.

[x4000]

Something else interesting: Unity 3D is "large address aware" on Windows, meaning that even though it will be a 32bit program there, it can use 4GB of memory instead of just 2GB.  That will be a notable improvement for the game when we switch it to Unity in August.

[dumpsterKEEPER]

That's really interesting. Is there information somewhere on how Unity is able to achieve that? I haven't done a lot of Windows programming so I'm mostly going off of what I've heard, but I wasn't aware that there was a way for 32bit processes to break the 2GB memory barrier. I'm assuming there's something within Unity itself that translates memory locations?

[keith.lamothe]

The 2GB barrier is relatively easy, you just have to fiddle with getting the OS to allow it and such.  Not particularly hard for a c++ application; more complex for a C# .NET application though possible (but has very undesirable side effects for AI War).

It's the 4GB barrier that's right-out for a 32-bit app.

[x4000]

Yep, we discussed this in another thread somewhere on these forums, you might be able to search and find them, but more info is here on that type of flag in general: http://blogs.msdn.com/oldnewthing/archive/2005/06/01/423817.aspx

As Keith notes, for C++ it's relatively trivial to implement this -- and in the case of Unity, they are using the Mono runtime, not the .NET runtime, so they have some added different flexibility there, too (there are pluses and minuses to it, as with all things, of course).  Unity is most likely just setting the LARGEADDRESSAWARE flag after having made sure that all of their stuff is compatible, so it probably wasn't a great feat on their part.  We experimented with this in .NET, if I recall, and the results were pretty disastrous for some reason or other.

[keith.lamothe]

Draxis tried it (the users can actually set the flag on the assembly themselves): http://arcengames.com/forums/index.php/topic,4359.msg29486.html#msg29486 .

He had a really large, long-running game, but we were able to deal with the stuff that was making him hit OutOfMemory.  The way it is now you have to go pretty out there to hit it, and particularly with the Unity port we should be able to push it far enough out that you won't hit it unless you intentionally cheat your way to a half-million-unit game or something like that