I don't think that "fun" is a useful metric. I mean, if the game wasn't fun it basically means that it just plain sucked, and "not sucking" is the lowest possible bar to clear.
Good luck convincing anyone about this. Lots and lots of people will speak positively about a game or system by saying "it's playable" or "you can have fun," as though those were meritorious qualities rather than the
absolute bare minimum.
A (published) game that is not even playable
does not deserve the paper upon which it is written. A game that is legitimately
impossible to have fun while playing is even worse: it deserves to be held up as a warning to the next ten generations of game designers for outright
actively destructive game design. I, personally, find it extremely unlikely that
any TTRPG is truly unplayable, and do not believe it is
possible to design a game that is genuinely, unequivocally
impossible to have fun with. Hence why I have such a dim view of any use of "fun" as a mark of a successful game.
Even genuinely awful, unacceptable* games can be fun under the right circumstances or for a narrow group of people. And many, many,
many genuinely amazing and wonderful games can fail to be fun if mishandled badly enough.
*I mean outright morally offensive games like
MYFAROG, literally a TTRPG incorporating
Nazi race theory, or piles of steaming garbage like FATAL. The vast majority of games are not like that.
---
As for my own answer to the thread question: A system
works when it does what it was designed to do. I don't mean that as a copout, because "does what it was designed to do" means something very specific to me.
A game system is a set of instructions and (typically) mathematical relationships. Game systems are designed to produce experiences, chosen by their designers. Effective game design thus requires designers to: (1) identify the
specific experiences the game should produce (and, potentially, those it is
not intended to produce); (2) set clear,
testable design goals for achieving the chosen experiences; (3) define
acceptable ranges of valid results, especially for any elements that involve randomness; (4) collect data and rigorously test the goals from 2, modifying and re-testing as necessary until the mechanics fall within the defined range of acceptability from 3.
Note that this is specifically considering game design as a
technology, NOT as an artistic work. That should not be taken to mean that the artistic element is irrelevant. It's just that the artistic element cannot be analyzed like a technology; it can, and should, only be analyzed aesthetically, which works very differently. Aesthetic choice matters for all steps above except (sometimes) step 4.
So, to repeat what I said earlier more specifically: A system
works when the rules it contains actually do produce the experiences it was intended to produce, within the range(s) of acceptable results its designers chose. In other words, when the
testing part of its playtesting phase was successfully conducted, iterating on whatever parts needed iteration until they produced an acceptable range of results.