This came up in reference to my review of The Future Belongs to Us. The author emailed me and wasn't happy I had reviewed the game without playing it.
But do you have to play an rpg to review it? Is it unreasonable to evaluate one on reading alone?
From this perspective, the entire game must be reviewed, from soup to nuts. If you don't it's like forming an opinion of a movie or piece of music without experiencing it. Anything else is invalid and not a complete picture. Nobody would take a review of a movie seriously if the reviewer hadn't viewed the whole thing.
But rpgs are books. They can be evaluated on content, layout, art, writing and the rules. Anyone with a technical eye toward game mechanics can get a good idea of how the game will play just with a read through. I don't need to play a game to know a single d20 has a flat distribution of results, or that 1s being fumbles and 20s being critical results in a 10% chance of an extreme result and how that will affect play. I've been there and done that. When I read Exalted 2nd edition I knew the ticks, AV, and DV would be problematic for me. Turns out when I tried to run it, I was right.
That's not to say playtest reviews aren't desirable. They are - but the subjective nature of actual play, makes it no more valid than a "capsule review". The two things exist for a reason. Reviews are opinions based on experience and subject knowledge. Anybody who has played more than a handful of systems (and I've played more than a handful) can see how all of the pieces fit together, just as they can make any game "work" and likely have fun while playing it.
But there's a final piece here that's particularly relevant to The Future Belongs To Us review: I couldn't have playtested it fully because it's a player's guide. The book has swathes of information that is vague or simply not there which is most likely in the gamemaster's guide. A playtest wouldn't have done it justice, and had I done that the review would have been much worse. Plus, there were hover gorillas and Lando Calrissian fighting ninjas over a treasure chest and I don't think I could live up to that challenge.