As for them not chasing you anymore, yeah, I remember that, I'm pretty sure Chris changed that on the account of, at least partially, my feedback....
Yeah, if you also commented on that, then we both probably said pretty much the same thing to him. He said something at one point about how roughly 60% of what each tester said overlapped with what other people were saying, and the other 40% was unique, when I asked how they managed to keep up with so much being dumped on them constantly, because that was before Mantis was public and we could see for ourselves. Not too surprising that that particular enemy behavior/difficulty got similar comments from more than one person, because others did many, many times, like with amoebas and espers and bosses and...
Yeah, I recall you both mentioning it. I think that c4sc4 might also have mentioned that one, but I have trouble keeping it straight. During the early days it was hugely helpful to us to keep the testers all isolated from one another, even though it was more work for us because of all the duplication. That way we got a dozen plus sets of individual feedback, rather than one group's collective feedback. Thus we got the "hey this occurred to me" comments rather than "ooh, I agree with what he/she said, though I didn't think of it until it was mentioned" ones. Not that the latter aren't helpful also, but seeing where there was blind overlap let us really quickly identify the issues that were most bugging people.
I wouldn't do that long term, because having that community cross-talk discussion is generally far more valuable than having isolated sets of feedback, but it was genuinely helpful during that first week. By the time that week was up, the duplicative feedback from testers was down into the 5% or less range, which is more consistent with a more mature game like AI War. Now I'm super happy to have mantis being used to track all this stuff, because it's so much cleaner for everyone now that that first period is past.
The "everyone in isolation" thing was a bit of a new experiment for us with regard to testing, but I think it worked well. We paired it with bringing in only about 4 testers at a time after the initial 7, and staggering them such that each batch of testers got the revised new-to-the-game experience. Since that was the thing we were most trying to refine, that turned out to be a successful way to do it. I'd do it again in the future with another game, though only right at the start.
Anyway, thought you might find that interesting as background on why we did it the way we did it.