Right up front, I want to acknowledge that it is possible to have fun games with class-less systems, or systems which have loosely defined roles such as World of Darkness. However, for the lighter sorts of games I am enjoying these days, a class system is helpful because it helps everyone find their role in the game quickly, speeds character creation, and generally helps everyone "grok" what is going on.
D&D 4E was an interesting example of a lot of ideas. One of the core character creation rules in 4E was character roles. Each class filled one of four niches: Striker (DPS), Defender (Tank), Leader (Healer/Support), and Controller (debuffer/crowd control). Characters could generally "lean" towards a second role, in effect "majoring" in one and "minoring" in a second.
One problem that becomes apparent rapidly though is that if you have four roles, and each are considered vital, then you need more players to fill them all. At a minimum, you need two players and they must each Major and Minor in different roles: for example, a Paladin (Defender/Leader) could pair up with a striker that minors in controller (or vice versa). Really, most typical groups will need five folks to cover each of the core four roles adequately.
1E AD&D was similar in that the core four roles were cleric, fighter, magic-user, and thief. In reality, you could often dispense with several of those, though, as there were more ways to skin a cat. For example, if you were short on fighters it was usually possible to hire men-at-arms as bodyguards.
Personally, I play RPGs rarely these days and often with a small group. I would prefer a mechanical system with fewer roles so that two or three players can cover down on everything. That means that the classes need to be less rigid, or you need fewer of them.
THREE CLASSES OR THE "HOLY TRINITY"
MMOs like WoW have popularized what has come to be known as the "holy trinity" of DPS, Tank, and Healer. This allows a smaller group to cover all the roles, in theory. In practice, it seems like the average group size in WoW still tended towards five, with a common line up being three strikers, a healer, and a tank.
The Holy Trinity has a few strengths. As mentioned, you need fewer players for a viable group (more on that later). Due to recent popularity, it is easily understood by players and GMs alike, and players know immediately what they are "supposed" to do. Even players who don't do MMOs can figure it out rapidly: I personally think immediately of American football, which has an offensive team, a defensive team, and a special team.
Some downsides include:
- These roles don't really match historical combat lineups. For example, in ancient combat you had heavy infantry (like pike formations), medium infantry (Roman legions with swordsmen), light infantry (velites), cavalry (of various types), archers, and so on. In the Napoleonic era you had light infantry skirmishers, line or medium/heavy infantry, artillery, and cavalry. They don't necessarily fit neatly into a trio. The trio is entirely based on game rules, not reality.
- The roles fall apart in a PvP environment, or if the "enemy" uses the same system. I'm not a big WoW player, but as I understand it, healers and tanks are fairly useless in PvP. You'd have to balance the trio more like "Rock Paper Scissors" for it to work in a PvP setting.
- The "tank" role only makes sense with an artificial aggro mechanism to force foes to attack the target which is hardest to kill. Traditional D&D uses minis and battlefield positioning to allow tanks to block an enemy; MMOs used aggro due to poor pathfinding algorithms and latency issues.
Two classes would be the minimum to have a non-trivial choice in character creation. EVE Online does this, with basically choices between tank (defense) and spank (offense).
I find such a system to be a bit too simple, personally.
I've got a few more thoughts on this front but my wrists are killing me, so I'm calling it quits for now.