Author Topic: AVWW Beta 0.921 "World Of Darkness" Released!  (Read 5811 times)

Offline keith.lamothe

  • Arcen Games Staff
  • Arcen Staff
  • Zenith Council Member Mark III
  • *****
  • Posts: 19,505
Re: AVWW Beta 0.921 "World Of Darkness" Released!
« Reply #15 on: March 29, 2012, 10:34:28 pm »
Just checked again and I see one report that you can ask Unity 3.4 to do a 64-bit windows build.  I cannot investigate myself (previously I've seen features mentioned or explained, like support for a command-line switch that causes no graphics components to load for the purpose of running a headless server, that simply didn't work or acknowledge that they existed when I tried them) because we're still on 3.3 due to some bugs in 3.4 that cause our games to die rather shortly after being started.
Have ideas or bug reports for one of our games? Mantis for Suggestions and Bug Reports. Thanks for helping to make our games better!

Offline TechSY730

  • Core Member Mark V
  • *****
  • Posts: 4,570
Re: AVWW Beta 0.921 "World Of Darkness" Released!
« Reply #16 on: March 29, 2012, 10:38:26 pm »
Just checked again and I see one report that you can ask Unity 3.4 to do a 64-bit windows build.  I cannot investigate myself (previously I've seen features mentioned or explained, like support for a command-line switch that causes no graphics components to load for the purpose of running a headless server, that simply didn't work or acknowledge that they existed when I tried them) because we're still on 3.3 due to some bugs in 3.4 that cause our games to die rather shortly after being started.

Ow, what kind of bugs?
That must of been a pain. Engine bugs are a great way to cause paranoia when debugging. (Trust me, I recently fought a library bug in the Java library, I spent hours tracing my code until I finally saw that it was coming from the library and then looked online about the message I was getting)

Offline eRe4s3r

  • Core Member Mark II
  • *****
  • Posts: 2,825
Re: AVWW Beta 0.921 "World Of Darkness" Released!
« Reply #17 on: March 29, 2012, 10:51:31 pm »
I don't understand, you already have all continent info stored in ram when you create it so clearly there is a opportunity to not purge it from the cache there. Unless I misunderstand, your current memory management even works that way. It's just.. not caching enough! ;p

Is there any particular reason why it gets unstable at 800mb ? Or is this just a limitation of x86 ?

You know, this isn't just about AVWW either, AI War also has a similar issue. Like how it loads graphics "on-demand" seeing as that means reading the graphics from DISK this always creates a short but slight delay. Not to mention constant disk access until everything was loaded (I find that every game start I just open up the unit library and have it load each sprite so that when I enter a region everything *IS* already loaded)

the funny thing is, when you still used your own engine this problem wasn't there. So it seems to me that it doesn't have to be that way. It's only something related to Unity that makes you do it... mystery ;P
Proud member of the Initiative for Bigger Weapons EV. - Bringer of Additive Blended Doom - Vote for Lore, get free cookie

Offline keith.lamothe

  • Arcen Games Staff
  • Arcen Staff
  • Zenith Council Member Mark III
  • *****
  • Posts: 19,505
Re: AVWW Beta 0.921 "World Of Darkness" Released!
« Reply #18 on: March 29, 2012, 11:02:13 pm »
I don't understand, you already have all continent info stored in ram when you create it so clearly there is a opportunity to not purge it from the cache there. Unless I misunderstand, your current memory management even works that way. It's just.. not caching enough! ;p
There are three levels of data to a continent:
- the part that's always loaded (and you see on the world map)
- region "interior data" that loads for a region while you're in that region and is used for the dungeon/region map
- and the actual chunks that are the things you run around in in-side view, and only the one you're in is loaded (in singleplayer or on the server it also hangs on to a chunk you've left for about 10 seconds before saving it back to disk and dropping it from memory; a somewhat similar delay-write-dump approach is done for region interior data when you leave a region)

When you're on the world map only the first of those is kept in memory.  When you enter a region it has to get the region's interior data and the chunk you're entering off disk.  When you walk off the edge into another chunk (or enter a building or underground or whatever) it loads the new chunk off disk and shortly thereafter saves and dumps the chunk you were just in.

Quote
Is there any particular reason why it gets unstable at 800mb ? Or is this just a limitation of x86 ?
I don't think I've seen a straightforward answer, but it's in the mono runtime.  It's weird because I'm pretty sure I've seen it go higher than that in buggy situations that created a lot of stuff really fast and filled up the memory, but it's certainly not stable up there.

Quote
You know, this isn't just about AVWW either, AI War also has a similar issue. Like how it loads graphics "on-demand" seeing as that means reading the graphics from DISK this always creates a short but slight delay. Not to mention constant disk access until everything was loaded (I find that every game start I just open up the unit library and have it load each sprite so that when I enter a region everything *IS* already loaded)
AIW has an easier time in that it can actually fit all its textures in memory, but has a harder time in that its simulation data is enormously bigger than AVWWs and cannot be broken up into bits that can be separately saved to disk and have their simulation suspended, hence the out-of-memory errors that plague the really high ship-count games.

Quote
the funny thing is, when you still used your own engine this problem wasn't there. So it seems to me that it doesn't have to be that way. It's only something related to Unity that makes you do it... mystery ;P
Nothing mysterious about it, every engine has different limitations.
Have ideas or bug reports for one of our games? Mantis for Suggestions and Bug Reports. Thanks for helping to make our games better!

Offline eRe4s3r

  • Core Member Mark II
  • *****
  • Posts: 2,825
Re: AVWW Beta 0.921 "World Of Darkness" Released!
« Reply #19 on: March 29, 2012, 11:06:23 pm »
Seems to me all these limitations would not be there with native x64 binaries... but that doesn't seem viable if Unity bugs out on you  :-X . After all, you do want an engine that can compile both x86 and x64 without any major differences, aside from the benefits of x64 of course.

so keith if I understood right, when I enter a regions surface it is in memory, but when i enter a cave 2 levels down, the surface region is no longer in memory, but has to be loaded from-disk?
Proud member of the Initiative for Bigger Weapons EV. - Bringer of Additive Blended Doom - Vote for Lore, get free cookie

Offline TechSY730

  • Core Member Mark V
  • *****
  • Posts: 4,570
Re: AVWW Beta 0.921 "World Of Darkness" Released!
« Reply #20 on: March 29, 2012, 11:07:12 pm »
I don't understand, you already have all continent info stored in ram when you create it so clearly there is a opportunity to not purge it from the cache there. Unless I misunderstand, your current memory management even works that way. It's just.. not caching enough! ;p

Is there any particular reason why it gets unstable at 800mb ? Or is this just a limitation of x86 ?


No, it's a limitation of Unity. Actually, its a limitation of the old version of Mono Unity runs on. The Mono version Unity uses starts throwing spurious OutOfMemory errors when faced with large heaps, even if it could of reclaimed some if the garbage collector was willing to get off its lazy butt. The later versions of Mono fix this bug, and can work properly with large heap sizes. Sadly, the Unity devs haven't made the port to newer Mono versions, and it doesn't look like they are going to anytime soon.

And no, they can't "hotswap" the Mono library with the newer version. Unity apparently statically linked to Mono, basically "deeply embedding" Mono into the Unity dlls (or shared library objects for Mac).


Still, with 800MB to work with, I think they could cache the previous region's texture files and other assets, or at least some of them, so long as they evict even older stuff from the cache when loading yet another region.

Quote
the funny thing is, when you still used your own engine this problem wasn't there. So it seems to me that it doesn't have to be that way. It's only something related to Unity that makes you do it... mystery ;P

Well, even with these obnoxious bugs, if I understand the devs correctly, using Unity still turns out to be less of a headache than trying to maintain and develop their own engine.
And from what I have seen, the other engines out there have even worse headaches to deal with. I know, hard to imagine. Still, that means the current selection of general purpose game engines is in a very sad state if there are really only two worthwhile contenders, with Unity being one of them. (Wasn't XNA also at least no worse than Unity?)
« Last Edit: March 29, 2012, 11:11:34 pm by techsy730 »

Offline keith.lamothe

  • Arcen Games Staff
  • Arcen Staff
  • Zenith Council Member Mark III
  • *****
  • Posts: 19,505
Re: AVWW Beta 0.921 "World Of Darkness" Released!
« Reply #21 on: March 29, 2012, 11:10:17 pm »
so keith if I understood right, when I enter a regions surface it is in memory, but when i enter a cave 2 levels down, the surface region is no longer in memory, but has to be loaded from-disk?
It's all the same region, but it has a surface "dungeon", and each building is a dungeon, and each "depth" of underground is a dungeon.  Metadata on all the region's dungeons (that exist thus far; underground dungeons just keep getting generated as you reach them) is in memory as long as you're in the region.  Each dungeon has one or more chunks in it, and the chunk is the thing you actually see in the main part of the screen.  Only one of those chunks is held in memory at once (the former one is moved to disk shortly after transition).
Have ideas or bug reports for one of our games? Mantis for Suggestions and Bug Reports. Thanks for helping to make our games better!

Offline eRe4s3r

  • Core Member Mark II
  • *****
  • Posts: 2,825
Re: AVWW Beta 0.921 "World Of Darkness" Released!
« Reply #22 on: March 29, 2012, 11:15:07 pm »
Well that is all pretty depressing. I know.. this is kinda nitpicky, but there is a SLIGHT delay everywhere and it bugs me. (And not so slight delays, when entering new regions)

Thing is, the game really FLOWS well in a region, walking, jumping, shooting, everything feels pretty smooth. So these delays on chunk-change are noticeable breaks from the flow. Thats why I even bothered to mention it.

Didn't think it'd be caused by the inadequacies of Unity. Oh well ;/
Proud member of the Initiative for Bigger Weapons EV. - Bringer of Additive Blended Doom - Vote for Lore, get free cookie

Offline keith.lamothe

  • Arcen Games Staff
  • Arcen Staff
  • Zenith Council Member Mark III
  • *****
  • Posts: 19,505
Re: AVWW Beta 0.921 "World Of Darkness" Released!
« Reply #23 on: March 29, 2012, 11:21:57 pm »
The later versions of Mono fix this bug, and can work properly with large heap sizes.
The GC is better, but I dunno if it has better support for larger heaps.

Quote
And no, they can't "hotswap" the Mono library with the newer version. Unity apparently statically linked to Mono, basically "deeply embedding" Mono into the Unity dlls (or shared library objects for Mac).
That static linking is actually one of the best things they did; that's why you don't have to install mono to run our games.  In fact, you don't have to install anything.  Not even the game (in the sense of running the installer).  Just have the game directory on a jump drive or wherever and open the .exe.  Boom, runs.  As a minor side benefit, they pared out basically all the parts of the runtime they didn't need, so that part of the process is smaller in memory.

I know why dynamic linking is the normal way to do things, but there are times I really like saying "no, really, this is exactly the code I want to run, please don't swap which rug I'm standing on at runtime".  The one thing I wanted .NET to do but couldn't really get it to do apart from one commercial tool I saw that would do it for .NET 2.0.

Quote
Still, with 800MB to work with, I think they could cache the previous region's texture files and other assets, or at least some of them, so long as they evict even older stuff from the cache when loading yet another region.
The bulk of the texture goes in unmanaged memory, which doesn't count against the 800MB. 

Quote
And from what I have seen, the other engines out there have even worse headaches to deal with.
By far.

Quote
Still, that means the current selection of general purpose game engines is in a very sad state if there are really only two worthwhile contenders, with Unity being one of them. (Wasn't XNA also at least no worse than Unity?)
XNA would be much harder to work with on a few points and be even worse at deployment than our previous engine (deployment being one of the main reasons we switched to Unity).

But it's not that there aren't other good general purpose game engines, but in all the ones I'm aware of you either write in a custom scripting language (like the Unreal Engine, which looks pretty good to me when I cracked it open) or c++.  We like C#.  If I had to use c++ I'm not entirely sure I'd still be in game development.  Unity let us keep writing in C# almost exactly like we did before.  That does make it pretty unique among the field.
Have ideas or bug reports for one of our games? Mantis for Suggestions and Bug Reports. Thanks for helping to make our games better!

Offline TechSY730

  • Core Member Mark V
  • *****
  • Posts: 4,570
Re: AVWW Beta 0.921 "World Of Darkness" Released!
« Reply #24 on: March 29, 2012, 11:29:15 pm »
Quote
And from what I have seen, the other engines out there have even worse headaches to deal with.
By far.

Yikes, the other engines you looked at are really that bad?   :o

Quote
Quote
Still, that means the current selection of general purpose game engines is in a very sad state if there are really only two worthwhile contenders, with Unity being one of them. (Wasn't XNA also at least no worse than Unity?)
XNA would be much harder to work with on a few points and be even worse at deployment than our previous engine (deployment being one of the main reasons we switched to Unity).

But it's not that there aren't other good general purpose game engines, but in all the ones I'm aware of you either write in a custom scripting language (like the Unreal Engine, which looks pretty good to me when I cracked it open) or c++.  We like C#.  If I had to use c++ I'm not entirely sure I'd still be in game development.  Unity let us keep writing in C# almost exactly like we did before.  That does make it pretty unique among the field.

Point taken.
I guess I should amend my statement to: the selection of C# general purpose game engines (or more loosely, natively .NET managed framework compatible engines) seems to be pretty sad.

Apparently people figured out how to get it done very well in C++ and other languages.

You could probably "hack together" a way to run C# code against a C++ or Java based engine, possibly using a third party "adapter" or interpreter or "just in time compiler" or something, but that sounds almost as nasty as trying to write your own engine.

Offline x4000

  • Chris McElligott Park, Arcen Founder and Lead Dev
  • Arcen Staff
  • Zenith Council Member Mark III
  • *****
  • Posts: 31,651
Have ideas or bug reports for one of our games?  Mantis for Suggestions and Bug Reports. Thanks for helping to make our games better!

Offline x4000

  • Chris McElligott Park, Arcen Founder and Lead Dev
  • Arcen Staff
  • Zenith Council Member Mark III
  • *****
  • Posts: 31,651
Re: AVWW Beta 0.921 "World Of Darkness" Released!
« Reply #26 on: March 30, 2012, 10:17:37 am »
And I'm not about to get embroiled in a giant debate, but a few things:

1. That slight bit of lag when you go through a door into a building from another part of the same building is the chunk generating and the old chunk saving, not anything to do with textures.

2. If we were to decompress all the chunks from a continent off your disk and try to load them all at once, I estimate that would take... anywhere from 5-10 minutes, depending on the continent.  Toll's server has 32,000 individual files for 5 continents, for instance.

3. Doing a native x64 binary alongside an x32 binary is really someplace I don't want to go.  At one point (before AI War 1.0) I used to maintain an x64 and an x32 build of AI War, and not only was a) that an enormous pain; it was also b) really prone to desyncs; and also c) vastly slower by default because "int" defaults to int64 instead of int32.  Among other things.  Generally anything that requires special builds for part of the playerbase is not going to happen.

4. If we were to load all of the textures for the game from disk when you first loaded the game, it would be incredibly annoying to you.  It would last... 1-2 minutes, perhaps more.  And the real kicker is that, a year from now, if we've doubled our content, it will take twice as long.  Textures also have to be stored in an uncompressed format in memory in order to written to the GPU with any speed, so you're looking at vastly more memory than I think you realize.  The game is about 500MB on disk now if I recall, but that's with a compression factor of about 12 on the music and textures.  That's 6GB right there uncompressed, conservatively, although that includes music and not just textures.  It wouldn't help you much.

5. I think there are several misconceptions about what textures we unload and when.  First of all, we only ever unload any textures when you exit the game or go to the world map.  If you're running around in a region, it's not doing any texture unloading at all.  Further, when there are things that are used across regions (like the world map icons and such, plus what spells you are currently carrying with you, etc), it doesn't unload those.  What it unloads is the side-view textures that aren't used by your character, only when you switch regions.  And even for a few things that are super common and always everywhere, like some of the particle effects, it never unloads those because it knows they would be needed in any region.  It would not be possible to make a texture manager that is smarter than this, because of the xml-driven nature of the seeding of a lot of the game.  And even just a lot of the procedural stuff is way to complex to unroll, and changes consistently, etc.

6. The only part of a texture that is in managed memory is the pointer to it and some incredibly tiny metadata about it (size, format, etc).  There is a way to bring them into managed memory (say, if you want to modify it -- like we do with the custom texture for the minimap), but then you can either say you want to save it back to stay in both managed and unmanaged memory, or just go to unmanaged memory.  The only texture we keep in managed memory at all is the minimap one.

7. In short, we know what we're doing and trust me there's not been some oversight here.  This part of the game has been optimized out the wazoo.  And complaining about a tiny hiccup when going into a new room here is... well, it makes me laugh a little bit, after playing so many console or disc-based games where you go into a room and wait several minutes.  Even something like half life 2, I'd be walking down a hall and then suddenly the game freezes and shows me a loading screen, and 2-3 minutes later I have control again and am in the next zone.  AVWW doesn't have anything remotely egregious like that, and in fact is designed from the loading screen to the chunks onward to make it so that you're getting where you want to go as absolutely fast as possible.  It's one of the parts of our game engine I'm proudest of, actually; Keith and I both put a lot of time and thought into it, and I don't think you understand what it would be like if we hadn't. ;)

8. Unity 3D isn't a bad game engine, it's actually really good.  Their customer service is about par for the course with large game engine vendors.  There are a larger variety of C++ game engines, sure, but they all have their own problems.  And there's a really large selection of godawful engines in a variety of languages, including C++ and so on.  The bug with Unity 3.4 was something I've managed to work around anyhow, but I haven't upgraded to 3.4 or 3.5 yet for two reasons: a) they removed windows 2000 and PowerPC support; b) 3.3 does everything we need it to, is stable, and we understand its quirks.  I don't want to jump into something new right now and have to potentially revert, or get into lots of new quirks.  This would be true with a new version of any game engine, not just unity.  Heck, as you know, that's true with a new version of AVWW or AI War. ;)

Anyway, hopefully that clears things up!
Have ideas or bug reports for one of our games?  Mantis for Suggestions and Bug Reports. Thanks for helping to make our games better!

Offline x4000

  • Chris McElligott Park, Arcen Founder and Lead Dev
  • Arcen Staff
  • Zenith Council Member Mark III
  • *****
  • Posts: 31,651
Re: AVWW Beta 0.921 "World Of Darkness" Released!
« Reply #27 on: March 30, 2012, 10:39:20 am »
Two last notes, while I'm thinking of them:

9. A 50MB world folder, uncompressed, would use 4850 MB of RAM... give or take.  Actually, that's just for the raw string data, sorry, if we were to store those in memory uncompressed (97% compression ratio usually).  If we were to then convert those into actual game objects, we'd be looking at around... hmm, maybe about 5,120x that amount of RAM.  So... 24 TB of RAM to host a 50MB world file from the disk if all the chunks and all the continents were all loaded at once.  I need to revise how long that would take to load off disk, I was just thinking of the actual file system reads and not the expansion into game objects.  Hopefully this illustrates just how untenable anything like that would be.

10. In short, we learned a lot from having a massive all-in-memory-at-once galaxy in AI War: most specifically, we learned not to do that anymore, because that was one of the biggest inhibitors to adding more complexity at a granular level.  AVWW is incredibly, vastly more complex at a granular level partly because of all this segmentation.  The game simply wouldn't work without it.
Have ideas or bug reports for one of our games?  Mantis for Suggestions and Bug Reports. Thanks for helping to make our games better!

Offline eRe4s3r

  • Core Member Mark II
  • *****
  • Posts: 2,825
Re: AVWW Beta 0.921 "World Of Darkness" Released!
« Reply #28 on: March 30, 2012, 10:55:21 am »
Thanks for the clarification(s)  ;D

Seems this is a lot more complicated for this game with its random nature than I figured.
Proud member of the Initiative for Bigger Weapons EV. - Bringer of Additive Blended Doom - Vote for Lore, get free cookie

Offline x4000

  • Chris McElligott Park, Arcen Founder and Lead Dev
  • Arcen Staff
  • Zenith Council Member Mark III
  • *****
  • Posts: 31,651
Re: AVWW Beta 0.921 "World Of Darkness" Released!
« Reply #29 on: March 30, 2012, 10:56:58 am »
All good; I just realized I'd never explained in one place why we do things the way we do, and why this is such an awesome technical feat actually, so I thought I'd do so.  AI War was pretty impressive to be able to simulate so many ships in memory all at once, but that really tied our hands in a lot of ways I didn't like.  AVWW is on a whole other scale...
Have ideas or bug reports for one of our games?  Mantis for Suggestions and Bug Reports. Thanks for helping to make our games better!