AI difficulty settings

[mindloss]

This game's a bit unique in that you're always facing two computer opponents. That also makes it tricky for me to figure out what equates to what. I know it's not an exact science, but does anyone have an opinion on how, say, a game against 1/7 AIs compares to a game against 3/4 AIs? If I had to hazard a guess, I'd say a ballpark answer would be "weighted largely towards whatever the better player is."

Also, for the easier vs. moderate vs. hard personality types... does anyone care to try putting that into change in difficulty rating? I know I know, it all depends and it's not exact... I'm just interested in people's feel for this stuff.

[keith.lamothe]

Some minor factions and whatnot key off the first AI player's difficulty, so those elements would be very different in 1/7 vs 7/1.

As for the easier/moderate/hard AI types... it's really hard to quantify.  Particularly in the general case.  An 80-planet galaxy with one human homeworld against two Scorched Earth AIs, even if they were like difficulty 3... I dunno if it would be feasible.  Maybe with an AI that sedate you could eventually clear your way to an AI homeworld and eventually take it down, but you'd have so little to work with.

But one of the things I've been thinking about is that it would be great if we could take a given lobby setup and compute a single "overall scenario difficulty" number from that.  Not so much for bragging rights since I'm sure that would be exploitable, but just to help folks understand that playing your first game against two AIs with Advanced Hybrids, Beachheads, Avengers, Preservation Wardens, Roaming Enclaves, Spirecraft-Hard, Golems-Hard, etc is... not going to be friendly, even if the AIs are diff 5 or 6

[mindloss]

But one of the things I've been thinking about is that it would be great if we could take a given lobby setup and compute a single "overall scenario difficulty" number from that.

That would be cool. But damned hard to quantify, as you say. The only way I see to do it well without oodles of hand coding would be to implement some kind of rating system for players, visible or otherwise. Then you track how players of different strengths do on various settings and after some time, you can extrapolate a rough overall difficulty measure for any given scenario. Sounds like a good chunk of work to do it well either way.

Ratings would be cool anyway. And interesting, since you're never playing against anyone.

[keith.lamothe]

But one of the things I've been thinking about is that it would be great if we could take a given lobby setup and compute a single "overall scenario difficulty" number from that.

That would be cool. But damned hard to quantify, as you say. The only way I see to do it well without oodles of hand coding would be to implement some kind of rating system for players, visible or otherwise. Then you track how players of different strengths do on various settings and after some time, you can extrapolate a rough overall difficulty measure for any given scenario. Sounds like a good chunk of work to do it well either way.

Ratings would be cool anyway. And interesting, since you're never playing against anyone.

Yea.  I don't think any intuitive formula would do the trick, but we could describe a few scenarios and ask y'all how you'd compare them numerically and see if there was any point in the exercise

[Nalgas]

Also, for the easier vs. moderate vs. hard personality types... does anyone care to try putting that into change in difficulty rating? I know I know, it all depends and it's not exact... I'm just interested in people's feel for this stuff.

It depends on a lot of other stuff (what options you're using, how good/experienced you are, what base difficulty you start with, etc.), but I think it would be fairly reasonable to say that playing against some of the harder AI types can be like turning the difficulty up as much as a full point compared to the easiest ones in some cases.  They're evil.  On purpose.

Some of the minor factions have a negligible effect on difficulty, though, or even lower it, like the non-hard versions of Spirecraft and Golems, which pretty much just give you ridiculously awesome stuff for free.  The ones Keith listed, on the other hand, range from "raises the difficulty at least a fraction of a point" to "please kill me now", and that's just individually.  Heh.

[zebramatt]

But one of the things I've been thinking about is that it would be great if we could take a given lobby setup and compute a single "overall scenario difficulty" number from that.  Not so much for bragging rights since I'm sure that would be exploitable, but just to help folks understand that playing your first game against two AIs with Advanced Hybrids, Beachheads, Avengers, Preservation Wardens, Roaming Enclaves, Spirecraft-Hard, Golems-Hard, etc is... not going to be friendly, even if the AIs are diff 5 or 6

I like this idea so much I decided to add it to Mantis: issue 3791.

Vote it up!

[mindloss]

Something just occurred to me.

There is already a HUGE database of data on difficulty vs. settings: everybody's high score table.

If you tweaked the game to submit those to you guys on some kind of regular basis (set is as opt-in if there are somehow privacy concerns... although I can't see how there would be), well, I'd think a few pretty straightforward analyses would tease out everything you need to know with quite high accuracy.

[Orelius]

That may be true, but high scores are not much of an indicator of how easy or hard a game is made by a certain modifier.  In order to do that, we'll need a standard, but where should that standard be? 7/7 vanilla? 10/10 core AI's?  It's a bit hard to find a standard, as it's different for everyone.

Furthermore, another issue is that certain play styles will inevitably net more points than others.  Some don't expand any more than necessary and sneakily assassinate the home command stations, while others spend dozens of hours to conquer every single planet before deciding to win.  It's obvious that the second game would net more points, even if the two games were played with the same settings.

Also, there will be a huge disparity of information among the higher difficulty AI's compared to the easier ones.  There will be far less games played against core and technologist AI's than against vanillas, for example.  I have a feeling that there simply won't be enough data to judge any of the harder AI settings.

I think that it would be best if such a thing were done by the community, or else it wouldn't be easy nor accurate.

Someone could simply start up a spreadsheet and calculate the values from there, I guess.  If there's enough interest, I might start one myself.

[mindloss]

To clarify, I wasn't proposing taking score into account, but probably going by win percentage. There are all sorts of easy and hard ways to go at it, but one of the easy ways might be:

Take any given binary setting, say Medium Golems. Look at all games with that turned on and turned off, and check the win % in both cases. If win % is higher with it on, we know it probably makes the game easier overall; if lower, then harder.

Now just weight each of these settings by the disparity between win% -- e.g., if a setting gives you 54% or 51% when it's on or off respectively, you know it's not much of a factor, but if it's 70% and 40%, that's a different story. I'm not a statistician, but you wouldn't be able to take the linear difference of the percentages... however, I'm sure you could take the sqrt or log or what have you and use that as your weighting factor.

For the actual difficulty ratings, I think you could get away with the same approach, since there's only a handful of them. Yes, these factors can interrelate in complex ways so by considering them one at a time you are necessarily losing something, but it would be dead simple, and as I said, I think it'd get you 90% of the way there.

Better yet: do that AND do the community rating thing. See what results seem to track best, use those.

[Nalgas]

The way scores are calculated in the first place is pretty arbitrary and doesn't have a ton of correlation with...well, any useful metric I can see.  The only thing guaranteed to be true is that in our multiplayer games, the people who miss a week or three will have noticeably lower scores than the people who don't.  Other than that, the scores at the end don't really reflect anything like who contributed the most toward winning, and you can artificially raise or lower them to just by changing how you play the game, not by playing better or worse or with harder or easier settings.  I have never found the scores the game calculates to be useful or helpful in any version of the game I've played, wouldn't miss them if they were gone, and would in general prefer it if they were gone, except they don't really actively waste any important UI space these days, so whatever.  Heh.

[Orelius]

Ah, I see.  That makes more sense.  Sounds rather solid to me.

However, one issue that I may see are games that are quickly abandoned or lost too early because of carelessness of being afk.  There should probably be a few limitations as to which games are being counted.  For example, if someone plays against an AI 20 times in a row but loses every time within 10 minutes, is that really because of the difficulty or because the difficulty is not suited to that player?

Speaking of which, I'm drawing up a rather crude version of a comprehensive difficulty spreadsheet right now.  It'll take some time though, since school is starting for me soon.

[zoutzakje]

all this sounds like an awfull lot of work, but it's defenitely something I would like to see. An overall scenario difficulty rating would be cool and quite usefull for new players as well.

[mindloss]

Hey, if someone hands me some [obfuscated if necessary] data, I'd be happy to try it.

[Commiesalami]

Hey, if someone hands me some [obfuscated if necessary] data, I'd be happy to try it.

The only advice I could give is look on the wiki for any good info, like here: http://www.arcengames.com/mediawiki/index.php?title=AI_War_-_How_The_Fallen_Spire_Campaign_Really_Works

Also your going to need differing equations for AI difficulties 1 - 6.9, 7 - 8.9, 9+ Due to the face that the scaling is different for each of those levels of AI for exo-galactic strike forces, there are differing guardians that could be seen (EMP and Self-destruction are the major ones) on higher difficulties. 

[mindloss]

I was referring more to my proposed gathering of high score lists, but I'm sure that's impractical for one reason or another for the devs.

Incidentally, I've been studiously avoiding that particular wiki page because of the spoilers - haven't played a fallen spire campaign through yet