This has quickly become my favorite genre. Here's what I think are the main ingredients for an action rogue-lite:
1. Challenge: Since these games are almost entirely mechanics based (since narrative doesn't build up procedurally very well), they must challenge you heavily. It's best if the challenge starts right away. For example, the very first minute of Spelunky can easily kill you. This genre demands skill from the player, much like the old games from the 80s and 90s, except that I can never play those games for long because they're too repetitive. This is where variety comes in.
A game that did this badly is Legend of Dungeon. The first couple of levels were filled with creatures that could only hurt you slightly (you have 100 hit points). It takes getting to the 4th or 5th level to get the point, that you have very little healing ability, and that any damage you took in the early floors (and any consumables you used up) is now a mistake you can't undo. But those first few floors are such a bore.
Another way of messing this up is with the metagame. Rogue Legacy starts out great, but you quickly realize that the grind can make the castle far too easy. The same applies in my opinion to the Binding of Isaac: Rebirth. I much preferred the original, where you barely got to beat mom, and then discovered there was a whole game behind that. In Rebirth, passing level 3 or so pretty much makes you unbeatable, since so many incredibly powerful abitlities were added to this version of the game.
You really want to keep the challenge going, and increase it as the player gets new tools to play with.
2. Variety: This is the other major leg of this genre. I really feel it's like playing Contra or TMNT or Mario, except without the boredom of those games from repeating everything. Rather than training your memory, you're learning skills and making quick decisions, reacting to situations as you go. The best levels of variety are such that you really form unique narratives once in a while, when the randomness meets in unique ways.
Spelunky was able to build quite a bit of variety out of its level design and item spread. The linear path, however, means that it really is quite lacking as far as variety is concerned. Isaac, however, while suffering from repetitive rooms, uses your unique combination of upgrades to create unique situations. Rogue Legacy failed here again -- the enemy designs are very repetitive, and most character traits ended up being too similar or too silly to care about. Nuclear throne also suffers here, even though arguably its map generation is more 'random'.
The interesting thing about procedural generation is how it interacts with the human mind. If you add lots of landmarks and details to your rooms/generation units, you'll have people remembering the patterns better and getting bored by repetitiveness. If you make the units too bland (as in just 'space' -- see Drox Operative), there won't be any differentiation. The key is to make the units of procedural generation interesting enough to feel different, but also bland enough to not remember every distinct combination. Both Spelunky and Isaac win out here IMO.
3. Feel: Since these games take the rogue-like genre but apply it to a classic game formula, they need to execute that formula well. The graphics can be simple, but the feel of moving, the reaction time, the animation -- all should feel good, which is to say, they should be comparable to the best examples of the sub-genre. Our Darker Purpose appears to really miss on this count, and the same applies to a Wizard's Lizard. Isaac, Spelunky, Nuclear Throne, and Necrodancer, however, all get the feel really right. Controls don't feel floaty, the interface is fairly clear, and bad guys die with a satisfying crunch. There's a lot of subtlety with this stuff - a lot of tweaking frames and input latencies to get them just right. This is one of the things I feel both Valleys didn't get right (though they're obviously in a different genre).
(Interestingly, for a strategic rogue-lite like FTL, I would replace this point with 'satisfying model'. Does the game model the experience it tries to convey well? Is the model complex and interesting enough to require skill? In FTL's case, I would say the answer is yes.)
4. Significant but limited loss: the whole rogue-lite formula takes you back to the days of games that weren't afraid to make you lose. Unlike the trend on the PC in the mid to late 90s and early 2000s, saving isn't there, which means you experience actual tension. Given the fact that every experience is unique, you also learn to appreciate death, as having at least taught you something, if not given you some reward in the metagame. However, balancing the amount of loss the player should experience is tough. Without any loss, the player loses involvement. This is probably Rogue Legacy's biggest issue. The whole game boils down to getting gold, and the run itself has no significance without gold. In Spelunky, you lose everything - it's one of the purest representations of the genre in that sense. But the run is fairly short, so your less is limited in terms of time invested. FTL, though I would place it under 'strategic rogue-lites', tends to have overly long runs, making the loss perhaps too great.
5. Consistency - This is something that not every member of this genre has, but I've found it to be a big plus. Inconsistency happens when a rule of the game is not followed under certain circumstances, either because the designers didn't think to program it in, or they felt allowing it was not good design. In general, because they are random and lack fleshed out narratives, rogue-lites benefit from rules that interact with each other in open ways. In Seplunky, anything can trigger an arrow trap, throwing an enemy on spikes will kill it, and dead shopkeepers will yield a shotgun because they have one. There are consistent rules, and they're almost always followed.
Platformers tend to have more consistency because the rule of gravity is already one rule that must act on all entities. Top-down games don't have to have as much consistency, but it's a really nice touch when they do. Nuclear Throne allows enemies to blow up cars, just as you can, and almost any explosion acts both on you and on enemies. This makes the game much more interesting, as good player skill (and luck) can combine rules in interesting, if unexpected ways. Isaac is perhaps the king of this aspect of design, as the game contains hundreds of upgrades, each of which has its own interaction rules.
6. Discovery: Rogue-lites tend to share little information with the player to begin with. The expectation is that as the player grows in skill, so too will he learn the intricacies of the game. A tutorial may show the basics, but that is generally only the beginning. In Isaac, items are mysterious and must be figured out by the player over time. In Nuclear Throne, a lot of hidden unlocks exist, and can only be discovered by the player. In Risk of Rain, both items and general player strategy is left a mystery. In Spelunky, the main game path is really the inferior path, and the proper path involves a secret (there's a problem with relying so much on secrets that are eventually spoiled and known to everyone, but that's a different issue).
This lends an air of mystery to the game. It expands it from being a game with many permutations to being an infinite game in the player's mind -- one of never-ending possibilities. Many players will spoil themselves with looking stuff up in a wiki, but the real experience perhaps is stumbling in the dark, being guided only by the occasional 'ah-ha' moment.
Note that some games, like Necrodancer, use such unusual mechanics, that they can't afford to leave things as mysteries, since the player is already adjusting to a completely different way of playing a game.
7. Narrative Coherence: This may seem irrelevant to rogue-lites, where the narrative is usually very basic, but in fact, this is a very big deal. Breaking the 4th wall is a very bad idea in rogue-lites, as it is in most games. Information should be provided without alerting the player to the fact that a game designer exists and is running the show. Spelunky's tutorial, for example, is provided by a fellow survivor of the cave. The bestiary, which provides small tidbits of info about enemies and locations, is in the form of a journal. Even the concept of procedural generation and the cave itself is given a magical explanation. The same is true for Rogue Legacy, where the castle magically re-designs itself after every run, and one can even re-use particular layouts using the in-game architect character.
In all of the top examples of the genre, careful attention has been paid to craft the experience in such a way that tutorials belong in the narrative (in Isaac, a basic tutorial is scrawled on the first floor of the dungeon), player notifications from outside the game are minimal, and no message breaks the 4th wall.
More importantly, though, levels are logically consistent, as are the creatures within them. Spelunky presents 4 versions of what you might expect a magical cavern to contain, each in its own level. While the aliens really stretch the narrative, the rest of the critters all fit the location and setting. You don't suddenly fight a clown or a dog in the first cave world. Nuclear Throne taps into our notions of what a wasteland would look like, with appropriate mutated monsters. Rogue Legacy contains the kinds of ghostly apparitions you'd expect in a haunted castle, and Isaac, after an odd intro mixing themes of horror, childhood, and Christianity, presents us with grotesque enemies fitting all of those themes.
The aliens in Spelunky present a good example of the problem one has when straying from this rule. Why are the aliens there? What do they have to do with the rest of the cave world and its Mayan/Egyptian mythology? Suddenly we need a narrative to explain things, because we can't just tap in to pre-existing assumptions about narratives that we know. Every time we introduce something that counters the player's expectations, we get another narrative problem. We need to explain to the player why this thing is there, but most players don't even pay attention to the narrative -- they'll just be taken out of the game's world. This is one of the big things I felt Valley 2, and to a lesser extent Valley 1 did wrong (aside from weak mechanic 'feel'). The collections of enemies made no sense, and making sense of them would require heavy narrative, which the player didn't want to be burdened by. Making enemies that tap into the player's pre-existing knowledge and expectations, and that make sense to him is *so* valuable.
A great example is the game Jamestown. It's a terrific shmup, set in a weird world where the American colonies and native Indians are being invaded by aliens, or it happens in space or something. To be honest, I don't remember the details. It reminded me a lot of the weird Inca games by Coktelvision, where the Conquistadors invaded Inca civilization, which lived in space. Anyway, the games starts with you flying over colonial American imagery, which all makes sense, and then aliens arrive and f*** s*** up. It's really weird, but the game gives you a lot of narrative sections to adjust you to that reality. Once you've switched to that mindset, though, it all makes sense -- you see a whole bunch of alien spaceships, and some more American colonial imagery, and it all makes sense and is very consistent. It's a very risky strategy, however -- I'm sure many people were turned off by the very weird juxtaposition of concepts and never allowed it to 'click' for them.
A rogue-lite that get this aspect wrong IMO is A Wizard's Lizard. I never got the feeling that the enemies were anything more than attempts to create different attack patterns, without regard for any consistent world. I don't even know what the consistent world was supposed to look like. Also, in Legend of Dungeon, some of the zombies had traffic cones on their heads, and they had the plant shooters straight from Plants vs Zombies. This is because Popcap contributed significant money to the kickstarter. While the game establishes that its world is tongue-in-cheek, this intrusion from another game's world is very jarring.
Anyway, those are my 7 cents on this matter.
