Arcen Games

Games => AI War II => Topic started by: x4000 on May 19, 2017, 09:49:09 AM

Title: 0.201 "Bugfixes and Optimizations" released!
Post by: x4000 on May 19, 2017, 09:49:09 AM
Apologies for not getting this out last night: https://wiki.arcengames.com/index.php?title=AI_War_2:Earlier_Than_Early_Alpha#Version_0.201_Bugfixes_and_Optimizations

In general took about an hour to release this morning, but I was doing other things while waiting for various compilations and builds and uploads to do their own thing.  Once I stop having to rebuild the executables every release (which should happen before too long), this time might be cut in half, if not more.
Title: Re: 0.201 "Bugfixes and Optimizations" released!
Post by: BadgerBadger on May 24, 2017, 01:51:28 PM
Looks like .202 is getting some decent content in it! Are we waiting for anything in particular before it the release? I am looking forward to some new wormholes!
Title: Re: 0.201 "Bugfixes and Optimizations" released!
Post by: x4000 on May 24, 2017, 03:21:58 PM
The new content in this one is mostly incidental, to be honest.  The big focus for us has been performance, performance, performance, and that's something that is still heavily inwork.  It's coming along, though!  I'm getting a somewhat choppy 25-50fps during heavy combat where I was in 0.201 getting a hugely-halting start-and-stop 20fps at most, and where in 0.200 I was getting even more halting 5fps.

What has been surprising is the various techniques that have NOT worked.  Matrix math is something that I figured I could shortcut in a variety of ways, but that didn't work out so hot.  The Graphics.DrawMeshInstanced route actually had a variety of issues, and then Graphics.DrawMesh doesn't avoid the culling problem after all -- frustum culling just won't die unless it takes efficient z sorting and transparency sorting down with it. ;)

There are still a variety of other changes that I'm working on, though, to reduce CPU bottlenecks that presently exist.  It's all about moving gameobjects around and the expense of the matrix math that entails.

That in turn leads to a variety of required changes relating to the hierarchy of objects in the tree, throttling and pooling of updates on more distant objects, and so on.  Thus far my focus is almost solely on shots, because they are relatively simple compared to squads and ships and gimbals, but I'm going to be after those other things next with my findings from shots.  Shots are no longer the largest slowdown on the CPU except when there are north of 5000 of them at once, so that's a big win.  I need to get that even lower, though, and then I'll be happy enough to start in on the squads.  That way I'll be confident in the results there without having to then take yet another pass on all this stuff too soon from now.
Title: Re: 0.201 "Bugfixes and Optimizations" released!
Post by: BadgerBadger on May 25, 2017, 10:42:53 AM
5000 shots may sound like a lot, but if the AI sends 700 MLRS at a well defended planet, I'm sure we get well upwards of 30K shots on the map at a time. I hope you come up with some fun solutions!
Title: Re: 0.201 "Bugfixes and Optimizations" released!
Post by: x4000 on May 25, 2017, 11:24:37 AM
Yep, I've got part of it sorted out already by quasi-inventing a new data structure.  It's something I'm sure other people have done before, but it's new for these sorts of purposes at least so far as I know.
Title: Re: 0.201 "Bugfixes and Optimizations" released!
Post by: BadgerBadger on May 25, 2017, 11:49:34 AM
Claiming an algorithm is linear with input off of two still relatively small data points makes the programmer in me twitch, though I assume you actually did more tests ;-)
Title: Re: 0.201 "Bugfixes and Optimizations" released!
Post by: x4000 on May 25, 2017, 12:28:44 PM
Yeah, these are just two small pieces of data out of huge streams of ongoing data I watched going by.  I had graphs I could watch, and spot check, etc.  I picked out a few points of interest to show, but even they are not fully representative because there are small variances in every frame that have nothing to do with the list and instead with the operations that may or may not happen at iterations of the list.  Hence some of the variance where I say something is "30 nanoseconds more" but actually it's more like 60 nanoseconds in the example given.

With that sort of thing, I'm kinda eyeballing it based on the averages of what is being sampled over a few million frames.  I didn't bother actually keeping a real average, because based off of visual graphs I can already see what's going on well enough for any practical purpose.  Getting super-precise average numbers would just be kind of a waste of time to show off, more than anything else. ;)
Title: Re: 0.201 "Bugfixes and Optimizations" released!
Post by: BadgerBadger on May 25, 2017, 12:48:31 PM
Yeah, that makes sense. The performance numbers from the SwizzleList don't really matter in and of themselves. I'm glad you're making some progress!

I'm a bit curious what Keith is up to. I see he did a bunch of performance work, but since then I can only imagine he's been back in his mad scientist lab, cackling maniacally in a cloud of green smoke lit by lightning flashes, bringing the AI to life....
Title: Re: 0.201 "Bugfixes and Optimizations" released!
Post by: x4000 on May 25, 2017, 01:02:37 PM
He actually has some family things going on this week, so there's not as much from him this week.
Title: Re: 0.201 "Bugfixes and Optimizations" released!
Post by: BadgerBadger on May 25, 2017, 01:06:24 PM
Ah, okay. If these are stressful things then I hope they go as well as one can hope!
Title: Re: 0.201 "Bugfixes and Optimizations" released!
Post by: x4000 on May 25, 2017, 01:08:33 PM
Nope, he's all good, and it was planned. :)
Title: Re: 0.201 "Bugfixes and Optimizations" released!
Post by: BadgerBadger on May 25, 2017, 02:55:24 PM
Oh good. I also spent a few minutes poking at the "zoom-to-cursor" code yesterday. Getting it to work "just the way I'd like" is challenging! Is there an obvious reason I'm missing for why the scrolling only adjusts on the X and Z axes?
Title: Re: 0.201 "Bugfixes and Optimizations" released!
Post by: x4000 on May 25, 2017, 03:28:50 PM
I'm not sure what you mean by the x and z only?  The x and z are what come out of the y position based on the curve.  If you're referring to angle.

But if you look down at the field from above, then x and z are the "horizontal and vertical" planes.  Y is going into and out of the screen.

What you may need to do is adjust it "backwards" some, which would be based on the forward of the camera transform, probably.

At any rate, you wouldn't want to adjust the y, because that would mean when you zoom in it does more or less zooming in. ;)
Title: Re: 0.201 "Bugfixes and Optimizations" released!
Post by: BadgerBadger on May 25, 2017, 03:46:30 PM
Oh, I was confused. I'm used to X and Y being horizontal and vertical in the plane, with Z being the height above the plane. This explains why I was having trouble figuring out what was going on! 
Title: Re: 0.201 "Bugfixes and Optimizations" released!
Post by: x4000 on May 25, 2017, 06:07:26 PM
No problem!  That is indeed how things are oriented in world space here, but you're viewing this from a camera looking down from above.  So like hanging a camera from the ceiling. :)
Title: Re: 0.201 "Bugfixes and Optimizations" released!
Post by: BadgerBadger on May 26, 2017, 09:47:29 AM
So I'm guessing we'll be looking at .202 coming out sometime next week?
Title: Re: 0.201 "Bugfixes and Optimizations" released!
Post by: x4000 on May 26, 2017, 11:28:14 AM
Nah, I expect a version today.  No promises for obvious reasons (recent events), but it should be today.

One of the things that I've been spending some of my time on other than performance in huge battles is the space backgrounds and the ability for players to make their own.  This in turn is leading to an overhaul of the entire post-processing pipeline, and I was doing some stupid things with the camera field of view that previously made the spaceships feel way smaller than they really are, etc.
Title: Re: 0.201 "Bugfixes and Optimizations" released!
Post by: BadgerBadger on May 26, 2017, 11:32:34 AM
It's always nice when you start doing some work and that leads you to be able to find and fix previously unnoticed bugs!
Title: Re: 0.201 "Bugfixes and Optimizations" released!
Post by: x4000 on May 26, 2017, 01:15:40 PM
Indeed!

In this case it wasn't a bug per se, it was just poor cinematography, really.  When you use camera lenses different ways, it makes things look larger or smaller or more distant or closer, etc.  I was using something that made the ships look incredibly toylike in order to avoid distortion on the planet, which is supposed to look different.

The problem at root is that it needs to be a composite shot (which it already was), where the planets and the starfield look waaay further off, but the ships look a lot closer and larger, with more perspective even to the fighters and such.

We already are using three composited cameras, so it's no extra load to fix this, thankfully. :)
Title: Re: 0.201 "Bugfixes and Optimizations" released!
Post by: tadrinth on May 26, 2017, 04:33:47 PM
If you wanna nerd out about the details of the SwizzleList, I always like hearing about interesting data structures. 
Title: Re: 0.201 "Bugfixes and Optimizations" released!
Post by: x4000 on May 26, 2017, 05:36:59 PM
So I didn't quite get there in terms of having this release ready.  But it's coming along quite well.  A few performance things linger, but I can get between 40-60 fps during fights where I was previously getting a choppy 5fps.  So it has come a really long way.

Unfortunately my trick with the camera FOV to make all the ships look larger doesn't play well with all the various circles and spheres the game uses for things like forcefields and selection circles, etc.  They get super distorted by a larger field of view, that sort of fish-eye effect like you get off a Go Pro camera.

The idea is that with a larger field of view, things look larger (which is true), and that works well for something like a racing game where you're down low, or even for a FPS game where you're looking out of the eyes of the character and the stuff at screen edges is your "peripheral vision."  It's one of the reasons why console games often feel so wrong on PC ports, because we sit a different distances from those screens.

At any rate, for a strategy game it just wasn't working out.  We're actually using a 40 degree field of view, rather than the default 60, but then with foreshortening on the camera zoom to fake a wider field of view in some ways.  This eliminates all edge distortion, which you could see even at 60 (we did this months ago), while letting you still "see as much stuff" as if the fov was wider.

But unfortunately it doesn't contribute to things feeling "large," or at least not "close and large," because the lines of convergence on the invisible horizon are not very sharply inward with that sort of fov.  Such is life.  It looks the same now as it did last version and the last few months.

I've had a breakthrough, finally, on the background starfields, though.  I was unhappy with some of the existing ones we had, and it's annoying that a skybox takes 6 setpass calls.  And that I couldn't really customize them much, even with my own hsv-enabled shader for them.

I wound up creating my own new shader that blends a number of textures together using a variety of methods, to come up with something that can be a lot prettier.  It takes a lot of fiddling to really get a good result, but then you can save that and use that from then on.  The resources are low in terms of what it saves in the game directory, and in ram, and it has a huge amount of flexibility.  It also only uses one setpass call.

It can't be randomized because most random values would look like utter nonsense (not just like bad space graphics, but just a riot of nonsensical stuff).  It takes a dedicated hand to tune a bunch of things with this and get something worth saving, and it's probably a 20ish minute process at the moment, minimum.  But it's worth it!  And other people will be able to add to that, not just me, which is exciting for sure.  Yay variety. :)

I'm in the process of switching all the graphics to use a new reflection map that is much more studio-quality for photography, and to using an HDR camera.  I needed some idea of how the space backgrounds were going to look in this new system so that ideally they would be dark enough not to get picked up by the bloom levels, which can now be set to only work on light levels above 1, or something along those lines.  I'm going to have to redo the entire post-processing visual stack in general, but the groundwork has all been laid for that.

If you wanna nerd out about the details of the SwizzleList, I always like hearing about interesting data structures.

It's stupidly simple, really.  I went looking into all sorts of data structures, things like AList and DList and B trees and Hash sets and even things like linked lists, but everything just had too many downsides. We had been using just a generic List<>, which worked well for iterating (which we do a ton of) and adding at the end (which we do a ton of), but not for removing from arbitrary positions (which we also do a ton of).  So we'd have spikes of huge amounts of ms every time it had to remove something from low in the list, because then all the other items had to move down.

I got really frustrated with the other data structures, because they're all so complicated and only sort of improve things.  And many of them waste GC resources or reduce the quality of the iteration time below O(n).  I needed speed on those three things, and just nothing around was really matching up with what I wanted. 

Then I started thinking about how I've used swapping lists of things in the past, when doing pathfinding and other logic, to save on GC.

Then I realized that the arrays themselves are incredibly tiny (in terms of data size for the array and their pointers themselves), and I only need like ten at most anyway, and I can predict really safe WAY upper bounds on them.

I also realized that since I'm pooling the data that goes in these, I don't have any need for things to be released for the GC.  If a reference gets held for a long time, it affects nothing.

So... I decided to make SwizzleList, which basically has two raw arrays in it, which you initialize to some super large size.  For shots, I chose half a million right now.

One list is active, the other is inactive.  When you add stuff, it goes in the active list at whatever the next slot is, then it increments a length integer.  That's "how much stuff is in the list," as far as the outside world is concerned.  You can't remove things from the list, but you can iterate over it in equal time to what you normally would.

At some point you hit a period where things have been flagged for removal outside of this list, and it's time to make a new, condensed list.  So basically what it does is it clears the inactive list, loops over all the items in the active list, runs some logic for each one (of my choosing), and then as it does, any that are "not flagged for removal" get put into the inactive list.  At the end of that one iteration, I swap which list is active and which list is not.  And boom, things are "deleted."  I had to do that loop for deletion checks anyway, so the only added cost is copying to the inactive list, which is super cheap.

I feel kinda embarrassed about it because this is not a sophisticated data structure at all, and in many ways it's just a "duh" sort of thing.  But, well, I haven't seen anyone else do this for this sort of purpose, and it's wicked fast.  If you needed the GC to work, needed to sort, or needed to realtime remove things during list processing, this would not work well.  Or if you needed lots of these, of unknown size.  But for a pooled set of things with a known upper bound that you can exceed by an order of magnitude without a real ram hit... works very well.

I attached the code for the list itself, and then this is the code that uses it:

Code: [Select]
#region HandleShotUpdates
    public void HandleShotUpdates()
    {
        float deltaTime = Engine_AIW2.DeltaTime;
        GameSettings_AIW2 settings = GameSettings_AIW2.Current;
        bool drawAllShotRadii = settings.Debug_DrawAllRadiiAtAllTimes;
        bool drawShotDebugging = settings.Debug_DrawShotDebuggingData;

        Engine_Universal.BeginProfilerSample( "ShotRemovalChecks" );
        int shotLength;
        ArcenVisualShot[] shots = ActiveShots.GetActiveList( out shotLength, true );

        #region Removal Checks
        {
            int maxShotsToHandle = Mathf.CeilToInt( settings.Performance_ShotRemovalChecksToAttemptPerSecond * deltaTime );
            int min = settings.Performance_ShotRemovalChecksMinPerFrame;
            if ( maxShotsToHandle < min )
                maxShotsToHandle = min;
            int shotsHandled = 0;
            try
            {
                ArcenVisualShot shot;
                for ( int i = 0; i < shotLength; i++ )
                {
                    shot = shots[i];
                    if ( shotsHandled < maxShotsToHandle && shot.RemovalChecks_LastAnimationCycle != this.CurrentShotRemovalChecksPass )
                    {
                        shot.RemovalChecks_LastAnimationCycle = this.CurrentShotRemovalChecksPass;
                        shotsHandled++;
                        shot.DoRemovalChecks( currentSimFrameVisualOnly );
                    }
                    if ( shot.IsConsideredActive )
                        ActiveShots.AddToInactiveListNoChecks( shot );
                }

                ActiveShots.SwitchActiveList( true );

                if ( shotsHandled < maxShotsToHandle )
                {
                    this.CurrentShotRemovalChecksPass++;
                    if ( this.CurrentShotRemovalChecksPass > 10000 )
                        this.CurrentShotRemovalChecksPass = 1;
                }

                if ( Engine_Universal.IsProfilerEnabled )
                    Engine_Universal.LodgeMessageInProfilerData( "Total: " + shotLength + ", Handled: " + shotsHandled );
            }
            finally
            {
                Engine_Universal.EndProfilerSample( "ShotRemovalChecks" );
            }
        }
        #endregion

        shots = ActiveShots.GetActiveList( out shotLength, false );

        Engine_Universal.BeginProfilerSample( "ShotMainUpdates" );
        if ( Engine_Universal.IsProfilerEnabled )
        {
            Engine_Universal.LodgeMessageInProfilerData( "Count: " + shotLength );
        }
        try
        {
            ArcenVisualShot shot;
            for ( int i = 0; i < shotLength; i++ )
            {
                shot = shots[i];
                shot.DoMainUpdate( this, drawShotDebugging, drawAllShotRadii );
            }
        }
        finally
        {
            Engine_Universal.EndProfilerSample( "ShotMainUpdates" );
        }

        {
            Engine_Universal.BeginProfilerSample( "ShotMovement" );
            int maxShotsToHandle = Mathf.CeilToInt( settings.Performance_ShotMovementsToAttemptPerSecond * deltaTime );
            int min = settings.Performance_ShotMovementsMinPerFrame;
            if ( maxShotsToHandle < min )
                maxShotsToHandle = min;
            int shotsHandled = 0;

            try
            {
                ArcenVisualShot shot;
                for ( int i = 0; i < shotLength; i++ )
                {
                    shot = shots[i];
                    if ( !shot.IsConsideredActive || !shot.DidMainUpdate )
                        continue;
                    shot.ShotMovement_AccumulatedDeltaTime += deltaTime;
                    if ( shotsHandled < maxShotsToHandle && shot.ShotMovement_LastAnimationCycle != this.CurrentShotMovementPass )
                    {
                        shot.ShotMovement_LastAnimationCycle = this.CurrentShotRemovalChecksPass;
                        shotsHandled++;
                        shot.DoShotMovement( drawShotDebugging );
                        shot.ShotMovement_AccumulatedDeltaTime = 0f;
                    }
                }

                if ( shotsHandled < maxShotsToHandle )
                {
                    this.CurrentShotMovementPass++;
                    if ( this.CurrentShotMovementPass > 10000 )
                        this.CurrentShotMovementPass = 1;
                }

                if ( Engine_Universal.IsProfilerEnabled )
                    Engine_Universal.LodgeMessageInProfilerData( "Total: " + shotLength + ", Handled: " + shotsHandled );
            }
            finally
            {
                Engine_Universal.EndProfilerSample( "ShotMovement" );
            }
        }
    }
    #endregion
Title: Re: 0.201 "Bugfixes and Optimizations" released!
Post by: x4000 on May 26, 2017, 05:42:18 PM
Bear in mind that both of these screenshots are with hdr on, but no postprocessing at all yet, and with the ship shaders all not yet accounting for the new reflection cubemap.  It's also relatively subtle of a background, but I can do more dramatic ones, too.  In general the idea is going with stuff that's not so darn bright, though, and this was my first go at that without it also looking tacky.
Title: Re: 0.201 "Bugfixes and Optimizations" released!
Post by: BadgerBadger on May 26, 2017, 06:26:15 PM
I'm intrigued by the comment about new buttons and fonts! I look forward to the game getting a bit of polish.
Title: Re: 0.201 "Bugfixes and Optimizations" released!
Post by: Cyborg on May 27, 2017, 12:00:55 AM
About the camera angle part of the conversation, is there a way to get an Adm. Akbar camera angle easily? By selecting a fleet or something?
Title: Re: 0.201 "Bugfixes and Optimizations" released!
Post by: BadgerBadger on May 27, 2017, 11:02:46 AM
If you zoom in all the way it's not too dissimilar from an Ackbar-style view.
Title: Re: 0.201 "Bugfixes and Optimizations" released!
Post by: x4000 on May 29, 2017, 10:12:57 AM
About the camera angle part of the conversation, is there a way to get an Adm. Akbar camera angle easily? By selecting a fleet or something?

We'll have a free-cam at some point prior to launch as well as the current camera style.
Title: Re: 0.201 "Bugfixes and Optimizations" released!
Post by: drspikes on May 29, 2017, 11:22:52 AM
Hi Chris, will the early access backers still get their keys today? Looking forward to trying this, knowing there is some way to go.  Many thanks!
Title: Re: 0.201 "Bugfixes and Optimizations" released!
Post by: x4000 on May 29, 2017, 12:33:17 PM
Hi Chris, will the early access backers still get their keys today? Looking forward to trying this, knowing there is some way to go.  Many thanks!

You bet! :D

I've posted some more details over here: https://forums.arcengames.com/ai-war-ii/ai-war-2-early-access-good-news-and-bad-news/msg215142/#msg215142

But mainly the most important thing is "today at 4pm EST or thereabouts."
Title: Re: 0.201 "Bugfixes and Optimizations" released!
Post by: BadgerBadger on May 29, 2017, 12:55:20 PM
And the rollout will come with .202 being released as well, he asked hopefully?
Title: Re: 0.201 "Bugfixes and Optimizations" released!
Post by: x4000 on May 29, 2017, 01:03:04 PM
And the rollout will come with .202 being released as well, he asked hopefully?

Called 0.300 at the time, but yes.  There's a lot of almost-finished stuff that's not in the release notes for that yet that make it worth the big version uptick. :)
Title: Re: 0.201 "Bugfixes and Optimizations" released!
Post by: BadgerBadger on May 29, 2017, 04:29:10 PM
Wait, Cyborg brought attention to the brightness thing? I thought I did...
Title: Re: 0.201 "Bugfixes and Optimizations" released!
Post by: x4000 on May 29, 2017, 05:07:31 PM
He did a lot, also, but I'll include you too.  ;D
Title: Re: 0.201 "Bugfixes and Optimizations" released!
Post by: x4000 on May 29, 2017, 05:52:36 PM
Okay, all of the early access kickstarter backers should now have their keys!  I will write up a more detailed post later tonight, but for now the basic info is here: https://wiki.arcengames.com/index.php?title=AI_War_2:_Making_Alpha_Fun#Starting_State

Keith, if there's anything you want to add or edit there, please feel free. :)

I'm at a family cook-out for the next few hours, but I wanted to get this out first.  Thanks everybody!