One of the issues that I've seen with a lot of modern games is a problem that I call rules fatigue. It occurs when a player has too many rules to comfortably remember or quickly reference, and as a result typically resorts to one of two outcomes:
As a game designer, both of these outcomes are problematic. You want to have a game that is played consistently so that players can continue to play in a variety of contexts and enjoy their play. In a video game, you want to make sure that the player isn't getting frustrated by unexpected events, and in a tabletop game you're relying on the player to follow those rules, typically as part of a group.
Although this principle could be cross-applied to video games, I primarily work with roleplaying games and the need to make sure that your rules are coherent is important.
Having a player in a group who has troubles with the rules is a recipe for frustrating them and their group; while some people will take it in stride, you're going to slow down play (and if your rules are too complicated for experienced players to remember, play's probably not going quickly anyway) and potentially embarrass players who don't have the same grasp as their compatriots.
Of course, some of this will happen with novices, but novices eventually learn the rules, if they are designed right.
When veteran players still struggle, something has gone wrong.
There are a few antidotes to rules fatigue, and the level to which they are effective depends on both your ability as a designer and the willingness of your audience to put up with extra work.
The golden standard, and usually the best, is to make the rules simple enough that they can be easily memorized. You also need to provide some sort of assistance to the player in memorizing these things: having examples and illustrations in the text to show off what the rules are is a good idea. This is not optional, though it is something I personally don't do until playtesting is done, and not every little rule necessarily requires an example (for instance, you don't want to do a sample attack with every weapon in a 60-entry gear section).
This is the golden standard because it requires no additional effort on behalf of the players, and it is generally indicative of good design principles. If you're excising things that aren't significant, and streamlining things that are, you'll let the game get out of the way of play.
At the same time, there may be times when you need more rules. If you're working on a wargame or a game intended to deliver a deep tactical experience, you may want more details, and a game that focuses on something like magical or digital worlds may require special rules for those situations.
The solution, in this case, is to compartmentalize rules. Instead of making every rule important for every player, you create clear distinctions where certain rules don't matter to some players because of the fact that they're not going to try things in line with those rules–and if they do a more experienced player can help them (assuming someone in the group is familiar with the rules).
Now, this has the obvious failing of potential inconsistency: you should be able to expect that most people learn the core rules, even if this is a prolonged process of osmosis rather than a deliberate effort on their part, but people who don't desire to do so may phone in some of the process and either intentionally or accidentally claim to know more than they do.
This isn't a big issue if everything works well, but in a game like Dungeons and Dragons, where spellcasters need to know not only the rules for how they can cast spells, but the mechanical effects of their spells (both of which are notably distinct from the standard core of the rules), you can wind up with players getting frustrated if things don't go right. A carefully balanced game mechanic crafted and honed carefully instead becomes a source of contention and can destroy balance and the narrative of the game.
The secret here is that these compartmentalized rules need to be simple enough for even a novice to put in a little effort to master them. Having a simple flow-chart or diagram can go a long way for this as well, as it provides immediate reference–unlike your core rules these compartmentalized rules may not be referenced as often.
Fantasy Flight Game's d100 Warhammer 40:000 games have done this with psychic powers, and while it's not a terribly elegant system (I personally like it, but I know a lot of people who hate it for what they consider–perhaps rightly–unnecessary complexity) it is one where the important information can be looked up quickly.
Providing reference materials with your game and making sure that they cover the important details is important. Often these things are relegated to a second purchase of a "Game Master's screen", which is a shame because many games benefit greatly from these elements, and designers who omit them may wind up with players struggling with erroneous interpretations of the rules that limit the quality of their experience.
A final option, which is generally not ideal, is to make the use of a reference or process standard. You might see this particularly in board games or wargames, where it is expected that players will always utilize a particular assistant, such as a tracker or a table listing possible outcomes.
The downside of this is that it requires reference, or at least encourages it, which means that players need to take that material with them or memorize it, and typically this reference material is made because the circumstances that it would be used in make that unlikely to occur.
With that said, there are advantages to this process: it can be combined with a character reference sheet (which players of a tabletop game are going to need anyway in most cases) to be made compact, and it means that players can focus on other rules instead.
When you're designing a game, you need to think about what rules everyone needs to know, what rules some people need to know, and what rules will require reference material, if any. A shrewd designer gives players ways to easily remember their rules, and makes sure that anything else they need is readily on hand during play.
