Most Fear mechanics in d20 (and RPGs in general, but most specifically D&D) are terribly boring. You fail a check, you run. In other words, you're practically out of combat (and the really boring part is that, even after the fear effect wears off, you usually spend a few rounds running back to the fighting). As I said: boring.
How can we make fear more fun for both GM and players? I'm stealing something from the Entropic Gaming System (which is a nice variant of Savage Worlds if you ask me).
The Entropic Gaming System (EGS) has a really cool mechanics like Heroic Points, Gaining the Advantage and Fear.
Let’s adapt EGS’ Fear for D&D and its descendants. This is how I’m doing it: when the PCs are affected by a Fear effect, the GM gains a FEAR POOL for the duration of the encounter.
A typical FEAR POOL could consist of 1 Fear Point per player character affected, plus maybe +1 Fear Point for every 2-3 HD of enemies (maybe demons, true dragons or undead would gain more Fear Points).
If you're playing D&D 3rd/Pathfinder (which has Shaken/Frightened/Panicked condition scale) you can use the numbers above for a Shaken condition. Fear effects that generates a Frightened condition could give 2 Fear Points per PC that fails a save. Panicked-scale effect would give 4 Fear Points per failed a save.
A GM can use 1 Fear Point to do one of the following:
- force a PC to reroll any dice (ability/skill checks, saves, attack/defense rolls, damage rolls etc.) and pick the worst result. This can be used once for each roll.
- force a PC to go last in the round, or to go after the scary monster (i.e. the Fear source), or let the scary monster interrupt his actions.
- force a spellcasting PC to make a concentration check to cast spell (or subject him to a misfire, if you use this kind of rule).
- forbid the PC from attacking directly or any other way harming the source of the Fear effect for 1 round.
- deal subdual/non-lethal/stress damage to all PCs that attack or any other way hamper the source of the Fear effect that round (this damage can represent the PC’s sanity, will or resolution literally eroding, to the point that the victim of the Fear effect finally falls unconscious due to sheer terror). Maybe 1 die of subdual damage for each 5 monster HD (use the same die, so in Pathfinder, a dragon would inflict 1d12 for each 5HD). Or maybe just inflict 2 dice of subdual damage per Fear Point spent.
Now, you could devise ways for the PCs to removed points from the FEAR POOL. Each of the following actions could remove Fear Points:
- spells that remove Fear effects should be able to remove a number of Fear Points equal to caster level (or maybe half that against Frightened/Panicked-scale effects – or Fear effects provoked by really strong creatures).
- a PC could use a Charisma check (or a performance check if a Bard in 3rd) to inspire his friends. The Charisma check can be substituted for any other equally heroic and inspiring deed. If successful, allow him to remove 1 or more Fear Points (a Bard using his inspiring/remove fear abilities, should work as spellcaster of the same level).
- an affected PC can do a thematically appropriate roleplay action (at least a move/standard action). Things like standing paralyzed and screaming, running while dropping one of this items etc. If the GM approves, let the PC roll a Luck roll or Will save – success removes 1 Fear Point, a natural 20 removes 2 Points (and maybe a natural 1 adds +1 Fear Point). Or just rolls a d20 (like a death save from D&D 4E and 5E): Anything from 2 to 15 would remove 1 Fear Point, 16-19 would remove 2, a natural 20 removes 3 and natural 1 adds +1 Fear Point. If the PC uses his move/standard action to do something really stupid but cool and thematically appropriate (like a non-cleric branding a holy symbol in an attempt to turn a demon, but only managing to draw the creature’s attentions, by a nice GM and remove 1 Fear Point “for free”, before rolling any die).
At the end of the encounter, if any PC managed to do a cool "fear induced" action, you should reward him (with Action Points, Hero Points, Healing Surge etc., give him at least a free reroll, use D&D 5E's advantage, or just fives a few temporary hit points or a new daily use of an ability. Oh, and if the PC acted in a properly scared way but penalized other PC (like pushing the ally against the scary undeads), then please reward the PC that was thrown in the fray, not the coward one that run away.