Author Topic: Welcome To the Arcen Way Of Doing Post-Release Support :)  (Read 9701 times)

Offline x4000

  • Chris McElligott Park, Arcen Founder and Lead Dev
  • Arcen Staff
  • Zenith Council Member Mark III
  • *****
  • Posts: 31,651
Welcome To the Arcen Way Of Doing Post-Release Support :)
« on: May 30, 2013, 10:09:53 am »
Original: http://christophermpark.blogspot.com/2013/05/welcome-to-arcen-way-of-doing-post.html

Hello there!  Looks like, as usual with a game launch, we have a lot of new players to our games.  Welcome!  I think you'll find Arcen is a bit... unusual... when it comes to our development process, though, and this may come as a bit of a surprise.

Two Different Software Development Philosophies
I used to have a boss who liked to say something to the effect of "a piece of software stops changing only when it's dead."  This was in the business software services arena, so that aphorism was pretty universally practiced in that industry.  How many of you use Gmail?  They're always fiddling with that thing and adding improvements and changes, right?

In the games industry, there's more of a "dump em" sort of model that most game developers use.  A new Mario game comes out, and it is what it is.  That game isn't going to see any patches, period.  But... the game isn't "dead," either.  It's only just come to life in fact, as far as the public is concerned, and it will live on in the annals of gaming history, with some players fondly revisiting it, forevermore.  I still crack open Mario 2 from the original NES every year or so.

The Benefits of Constant Iteration
Needless to say, given my background, I am not overly thrilled with the "here this game is, bye" sort of model.  I mean for games that I make, to clarify -- it doesn't bother me at all that Nintendo does it.

I think a lot of other indies feel the same way about the games that they make.  Terraria and Minecraft see tons of updates all the time (well, Terraria used to), and Don't Starve and others update almost weekly as I understand it.  How many updates has Team Fortress had again?  And they aren't even indie.  Etc.

The constant-iteration approach is really cool because it allows for a constant dialogue back and forth between players and the developers.  Once players have their hands on a game, they inevitably have opinions.  Things they would like to see improved, things they would like to see added just for the fun of it, and bugs they find that none of the testers found prior to release.

None of that is possible if the developer isn't listening to you, and doesn't keep doing updates.

The Downsides of Constant Iteration
I'm not going to pretend that it's all glory on the iteration side, of course.  There's something to be said for having a perfectly-polished game that never changes.  It's an icon, something immutable that players can enjoy and remember forever.  I come back to Mario 2 all these years later, and it's the same.  That's actually pretty cool.

But more than that, sometimes during an iteration, we make a misstep -- we change something that people freak out about (like the mountains/lakes thing in Skyward Collapse, and the bandit keeps against mythologicals thing in the same).  Sometimes new bugs get added.  Sometimes Herobrine keeps getting added and removed, over and over. ;)

What mitigates the downside of fixes like that is the developer actually listening to players when they freak out about a change or find a new bug.  Players talk, the developer listens, the developer makes a change, and people are happy -- and the game has still evolved despite a temporary speed bump.

Update Frequency
I think the biggest thing that freaks out some people new to Arcen is just how frequently we update our games.  Most workdays in the week, we put out at least one update to at least one of our games.  It is extremely rare that an entire week goes by without an update, and that's generally just happening when we are all knee-deep in working on a new game behind closed doors.  And even then, if we're in a private alpha with players, the updates are pretty much daily.

Why all the rushing around?  Well, it's not really that we're rushing, honestly; we work every day of the week, same as any other game developer.  However, thanks both to our own internal updaters for our games and to the update model in Steam, updates take very little time.  Actually pushing out the code and assets for a new update takes me literally 5-8 minutes.  I've timed it.  It takes me longer to write up the post about the release than it does to actually push out the release.

In other words, the barrier to us actually doing releases at this frequency is basically nil.  So just because we can, does that mean we should?  I obviously think the answer is yes, since that's how we do things.  My reasoning is that there are fewer risks of big incidents (huge bugs or players really hating something) if we do our updates frequently.  And with this frequency, if players do find a new critical bug or really have a giant backlash against a change, we can have a fix out often within hours.

How many times have you been frustrated by a developer who knows about a bug, and takes 6+ months to issue a patch for it, if ever?  Man, I start getting heartburn when it goes 24 hours, no joke.  Maybe I have a tiny bit of OCD or something, but I just don't like things hanging out there like that if I know about them.

But What's With All The Fixes Right After Release?
Day One Patches are a Bad Thing, right?  Well... I don't know about that.

AAA Developers
First off, let me talk about AAA developers (which we are not) just to set the stage here.  Those guys have to get a build ready and "go gold" with it a month or months in advance of when you actually get the game.  It has to go through the process of manufacturing and all that good stuff, right?  That takes time, and once the discs are printed that's it for the 1.0 version.

But what are those same developers doing during the months between when they finish the gold master and when you first get your hands on it?  Twiddling their thumbs?  I don't think so.  They're still working, testing, tweaking, etc.  I'm sure the dev leads are often slapping their heads and going "oh my god, how did we miss this one with 200 testers looking at it?"  But that sort of thing is inevitable in a game of sufficient complexity.

So what they do is dutifully fix, then test, all the things they find during this time period.  They're doing a good job!  They'd call back the discs and apply said fixes to them if they could, but they can't.  So day one when you get your disc, there's a patch waiting for you, and the AAA dev team is feeling happy about that.  You're feeling mad at them about releasing something buggy in the first place.  I understand the sentiment, and I've felt that from the player end of things as well, but I don't think that's really justified.

Indie Developers
As with AAA developers, we never stop working and just sit around twiddling our thumbs.  However, every indie developer has a massive disadvantage compared to their AAA counterparts: a tiny staff and no giant QA staff.  Most new indie developers are going to do all their QA testing themselves, plus roping in friends and family as much as they can.  An indie developer new to the market is going to have some launch bugs, and so long as they fix those pronto they should be cut some slack in my opinion.

But that's not talking about Arcen -- at this point we've been around for four years, and this last release is our sixth full game on Steam.  So none of what I just said about indie developers who are new to the market applies to us.  We have the benefit of an existing fanbase, and we use them to do QA testing for us in exchange for a free copy of the game.  And we also keep using ourselves and friends and family, of course.

In the private alpha for Skyward Collapse, which lasted almost a month, we had around 35ish testers banging on it as frequently as they could.  We had around 10ish testers who went just crazy above and beyond and were banging on it daily.  Holy cow are we indebted to those folks.

By the time any of our games launch, we've fixed all materially-significant bugs in them, and the testers are happy.  Same as with the AAA developers when they "go gold" for manufacturing.  Yay celebrations!

What Happens Right After Launch
This is the same for both indies and AAA developers: suddenly there are massively more people looking at your game. Skyward has sold more than 15,000 copies in less than a week.  A large AAA game might sell hundreds of thousands, or even a million copies in its first week.

How can thirty testers, or two hundred, or pick your number, ever compete with that many players in terms of finding bugs, exploits, and things that are confusing?  So as soon as that many new eyeballs are on a game, there's this whirlwind of sudden feedback.  This is great, and I love how engaged players get -- and how clever they are.  The important thing is that whatever they find that is a problem is addressed quickly, because I think it's pretty impossible that they will find nothing unless the game has been through an insanely high-budget QA process that only a few companies (like Nintendo) can afford to do.

There's also the issues of complexity: Nintendo does not make games with procedural generation, for instance.  They make closed game worlds where they control your experience minutely.  For a game with any procedural component to it, there's always going to be new things that crop up unexpectedly.  You put a bucket on the head of a guy, and then that lets you rob him blind.  Oops, how did nobody find that?  Well, the game was just that complex, and none of the testers happened to think of taking that specific action.  Like I said, players are clever and will think of all sorts of things, particularly in a game with a high degree of complexity and randomization.

There's also the issue of platform: Nintendo makes their own hardware, and don't have to support a variety of operating systems and an even wider variety of hardware.  Holy heck is the PC the wild west compared to any console.  I don't think even Nintendo could avoid some patches if they released PC games.

Finally, there's the issue of player cleverness in certain genres.  Specifically, the kinds of genres that Arcen serves.  For a Mario game, this problem doesn't exist: you can only be so clever, because the levels are controlled very tightly and the abilities all have very specific functions.  There is not more than one way to solve a Mario level unless the developers specifically designed it to have multiple ways to solve it.  This is just fine, but this very different from the sort of games we make.

With any strategy game, really, or with any game where there is crafting or procedural loot or a vast world with lots of interlocking rules... players are going to figure out some funky stuff.  Some of them on day one.  Some players are going to find a crazy shotgun-reloading exploit, and then the normal game balance falls to pieces when they use it.  Why wasn't it found?  Complexity and open-endedness of the game world.

What's the next step when an exploit is found?  For Arcen: trying to work out a fix with the community that stops the exploit without stepping on normal gameplay, and then implementing that as swiftly as possible.  If you complain about how AAA developers don't fix the exploits, or just resort to banning people who use the exploits, then this should be a welcome thing.  But you can't have it both ways: in the process of fixing the exploit, sometimes we're going to make a misstep and change things that pisses off other players.  At that point we roll back that change and discuss it more with players.

Conclusion (TLDR)
The Arcen way of doing things is unusual, to be sure.  Possibly singular in the sheer frequency of how we do updates (weekly is not uncommon with some other developers, but I don't know about daily).  But the way that we do things lets us better connect with our player community, lets us be more proactive on changes and fixes, and lets players see the fruits of said changes and fixes faster.

I think it's a good system, but like everything it has its pros and cons.  We're specifically making some changes to our process lately in order to try to minimize the cons: rather than immediately implementing a fix to an exploit that we think is a good solution, we're instead taking it to our forums and inviting feedback and discussion.  Players are experts at poking holes in our ideas, and so we now wind up going through some lengthy discussion before we actually make a change.

What that leads to is slower fixes to exploits or certain tricky balance issues, but less "flailing" in the sense of "let's try this and see what the players think" and then "oops, revert."  That process of flailing is something that is a byproduct of being prototypers by nature ("let's try this thing and see how it feels in action" works great in an alpha or beta -- it's a fast, effective, and visceral way to form opinions on a possible change -- but it's not a good idea in a released game).

Overall I think our new process of talking out substantial changes before implementing them is the way to go; and that doesn't impact our ability to make small tweaks and fixes and improvements on a daily or near-daily basis.  I think that's a win for everyone, frankly.

Thanks for reading, and for your support of our company in general!

Arcen Games Forums
Arcen Games Idea Tracker (Mantis)
« Last Edit: May 30, 2013, 10:51:48 am by x4000 »
Have ideas or bug reports for one of our games?  Mantis for Suggestions and Bug Reports. Thanks for helping to make our games better!

Offline Mick

  • Hero Member Mark II
  • *****
  • Posts: 911
Re: Welcome To the Arcen Way Of Doing Post-Release Support :)
« Reply #1 on: May 30, 2013, 10:16:48 am »
How many of you use Gmail?  They're always fiddling with that thing and adding improvements and changes, right?

You're uh.. half right. :)

Offline x4000

  • Chris McElligott Park, Arcen Founder and Lead Dev
  • Arcen Staff
  • Zenith Council Member Mark III
  • *****
  • Posts: 31,651
Re: Welcome To the Arcen Way Of Doing Post-Release Support :)
« Reply #2 on: May 30, 2013, 10:19:16 am »
How many of you use Gmail?  They're always fiddling with that thing and adding improvements and changes, right?

You're uh.. half right. :)

Hahaha.  Yeah, I was iffy about the last round of changes as well, honestly.  But now I'm actually really appreciating those changes, because I can respond to things while also still composing or replying to a lengthy email.  That's pretty sweet.  I was really put off by the latest changes at first, but in the end came to like them.

Hopefully Arcen's changes in general do not cause that reaction, though. ;)
Have ideas or bug reports for one of our games?  Mantis for Suggestions and Bug Reports. Thanks for helping to make our games better!

Offline madcow

  • Hero Member Mark III
  • *****
  • Posts: 1,153
Re: Welcome To the Arcen Way Of Doing Post-Release Support :)
« Reply #3 on: May 30, 2013, 10:35:15 am »
Might be a silly suggestion. But might want to add a mantis/forum link on the blog post so people that find the blog can easily see/contribute to feedback!

Offline x4000

  • Chris McElligott Park, Arcen Founder and Lead Dev
  • Arcen Staff
  • Zenith Council Member Mark III
  • *****
  • Posts: 31,651
Re: Welcome To the Arcen Way Of Doing Post-Release Support :)
« Reply #4 on: May 30, 2013, 10:52:01 am »
Might be a silly suggestion. But might want to add a mantis/forum link on the blog post so people that find the blog can easily see/contribute to feedback!

Great idea, I've just added that -- thanks!
Have ideas or bug reports for one of our games?  Mantis for Suggestions and Bug Reports. Thanks for helping to make our games better!

Offline Pepisolo

  • Arcen Volunteer
  • Master Member Mark II
  • *****
  • Posts: 1,511
Re: Welcome To the Arcen Way Of Doing Post-Release Support :)
« Reply #5 on: May 30, 2013, 10:56:00 am »
Sounds good to me, although recent discussions have been making my brain hurt. Taking everything everybody has to say on board and coming up with clear solutions is tricky to say the least. Congrats on the sales figures and the 9/10 metacritic review. Presumably this is Arcen's most profitable title since AIWar, which is nice.

Offline x4000

  • Chris McElligott Park, Arcen Founder and Lead Dev
  • Arcen Staff
  • Zenith Council Member Mark III
  • *****
  • Posts: 31,651
Re: Welcome To the Arcen Way Of Doing Post-Release Support :)
« Reply #6 on: May 30, 2013, 11:03:38 am »
Sounds good to me, although recent discussions have been making my brain hurt. Taking everything everybody has to say on board and coming up with clear solutions is tricky to say the least.

Yeah, it's tricky, but it's actually a welcome relief for me.  Back in the early AI War days, it was just me, and you can imagine how lonely and frustrating that can get.  I much prefer the giant discussions, but then I've got all day at my job to do them, heh.

Congrats on the sales figures and the 9/10 metacritic review. Presumably this is Arcen's most profitable title since AIWar, which is nice.

Thanks!  And yes, this is our first game to break even since AI War.  And this game broke even within something like 3 days of being out.  In terms of total earnings, this is still behind the Valley 1/2 combo, but it should catch up and pass it by the end of the year.  In terms of best launch ever, Valley 1 still holds the crown over both AI War and Skyward, by a fair margin.

Funny how all that sounds contradictory, eh?  If Valley 1 hadn't been so massively expensive to make, it would have been a big win (although total earnings for AI War still dwarf it 3:1; the launch is only part of the picture).

Anyway, we're pleased. :)
Have ideas or bug reports for one of our games?  Mantis for Suggestions and Bug Reports. Thanks for helping to make our games better!

Offline Pepisolo

  • Arcen Volunteer
  • Master Member Mark II
  • *****
  • Posts: 1,511
Re: Welcome To the Arcen Way Of Doing Post-Release Support :)
« Reply #7 on: May 30, 2013, 11:13:03 am »
Quote
In terms of best launch ever, Valley 1 still holds the crown over both AI War and Skyward, by a fair margin.

It sounds like Valley 1 for some reason lost a lot of momentum after the initial launch. Maybe too many negative reviews started to surface after a while harming sales -- although having said that I've seen far worse reviewed games continue to sell... not very predictable this indie games business.

Offline x4000

  • Chris McElligott Park, Arcen Founder and Lead Dev
  • Arcen Staff
  • Zenith Council Member Mark III
  • *****
  • Posts: 31,651
Re: Welcome To the Arcen Way Of Doing Post-Release Support :)
« Reply #8 on: May 30, 2013, 11:16:24 am »
Valley 1 was going gangbusters on launch -- $130k gross in its first month, $42k in its second month.  That was 3x higher than our prior best launch, AI War.  Skyward has been kind of partway between those two markers, making it our second-best launch.

Valley 1 was slow to get reviews at all, and the sales dropped before the metascore or really any substantial reviews were out.  It was a very clear cause and effect: literally the day after Diablo III came out, our sales dropped to 1/10th the value of the day before.  And they never went back up. :/

I suspect the "not going back up" thing wasn't helped by the reviews that came later.  Though Valley 1 and 2 have done quite well during discount promotions, and currently make up almost half our income (well, prior to Skyward).
Have ideas or bug reports for one of our games?  Mantis for Suggestions and Bug Reports. Thanks for helping to make our games better!

Offline Pepisolo

  • Arcen Volunteer
  • Master Member Mark II
  • *****
  • Posts: 1,511
Re: Welcome To the Arcen Way Of Doing Post-Release Support :)
« Reply #9 on: May 30, 2013, 11:28:56 am »
Quote
It was a very clear cause and effect: literally the day after Diablo III came out, our sales dropped to 1/10th the value of the day before.  And they never went back up.

Huh, bit strange that. You wouldn't think that on the face of it Diablo would impact on Valley sales. I guess the lesson is to at least try and avoid launching near to any huge blockbuster titles, especially if they're in any way similar -- both have RPGy elements I guess... Interesting stuff. I like the way you don't mind talking candidly about finances and stuff. Cheers!

Offline x4000

  • Chris McElligott Park, Arcen Founder and Lead Dev
  • Arcen Staff
  • Zenith Council Member Mark III
  • *****
  • Posts: 31,651
Re: Welcome To the Arcen Way Of Doing Post-Release Support :)
« Reply #10 on: May 30, 2013, 11:31:35 am »
Cheers!

And yeah, we knew the Diablo III thing might impact us, so we chose a period of time sufficiently in advance of it to make sure it couldn't stomp at least our opening sales (and it didn't).  But we also couldn't delay until months after it or whatever because of finances.
Have ideas or bug reports for one of our games?  Mantis for Suggestions and Bug Reports. Thanks for helping to make our games better!

Offline iozay

  • Newbie Mark III
  • *
  • Posts: 48
Re: Welcome To the Arcen Way Of Doing Post-Release Support :)
« Reply #11 on: May 30, 2013, 05:24:28 pm »
Honestly, the Arcen way of doing things is the reason why I love your games. It is a kind of a living (great) product that will be supported for a long time to go :)

Offline solosol

  • Newbie Mark II
  • *
  • Posts: 18
Re: Welcome To the Arcen Way Of Doing Post-Release Support :)
« Reply #12 on: June 01, 2013, 08:19:51 am »
Might be a silly suggestion. But might want to add a mantis/forum link on the blog post so people that find the blog can easily see/contribute to feedback!

Great idea, I've just added that -- thanks!
I wasn't sure to post it but after reading this, I suggest to add a mantis link in the general category, and probably pin it in each game thread. Now I've bookmarked it, but before, I could not find a direct link, easy to see. I arrived to mantis after reading different posts, and that is what I have to do (read more posts and have luck in finding a link) if I want to use it (before bookmarking it, as said).

Offline MouldyK

  • Sr. Member Mark II
  • ****
  • Posts: 353
Re: Welcome To the Arcen Way Of Doing Post-Release Support :)
« Reply #13 on: June 03, 2013, 06:50:56 pm »
How many times have you been frustrated by a developer who knows about a bug, and takes 6+ months to issue a patch for it, if ever?  Man, I start getting heartburn when it goes 24 hours, no joke.  Maybe I have a tiny bit of OCD or something, but I just don't like things hanging out there like that if I know about them.


So many times this year.

I mean, I am not as active on Arcen Games as I was last year with Valley 1, but since then, I have felt spoilt by you.

An example is Prison Architect by Introversion. Their Alpha releases come out Monthly, some not even fixing bugs to the game!!! And while I try to not be hard on them, what with it being an Alpha and all, I still want Arcen Updates to all games I play lol. :)

Offline Mick

  • Hero Member Mark II
  • *****
  • Posts: 911
Re: Welcome To the Arcen Way Of Doing Post-Release Support :)
« Reply #14 on: June 03, 2013, 07:09:29 pm »
How many times have you been frustrated by a developer who knows about a bug, and takes 6+ months to issue a patch for it, if ever?  Man, I start getting heartburn when it goes 24 hours, no joke.  Maybe I have a tiny bit of OCD or something, but I just don't like things hanging out there like that if I know about them.


So many times this year.

I mean, I am not as active on Arcen Games as I was last year with Valley 1, but since then, I have felt spoilt by you.

An example is Prison Architect by Introversion. Their Alpha releases come out Monthly, some not even fixing bugs to the game!!! And while I try to not be hard on them, what with it being an Alpha and all, I still want Arcen Updates to all games I play lol. :)

Yeah, the releases of Prison Architect annoy me. I prefer bug fixes over new features (not that I don't love new features), but I think you should make sure what is there WORKS before adding in extra stuff.