Before you read any further, I need you to go to Google right now and search for "recursion" and take note of what Google suggests under "Did you mean".
Being a database guy, recursion is a very good friend of mine. But what does it have to do with rpgs? There are two ways I can think of. Any time the characters are playing another character within the game, or the game acknowledges that the characters are doing so, the game is recursing. Obviously this doesn't happen very often.
Pretty much exactly like that
The Dream Park rpg was one of them. You played a character who had a character in what amounted to a high-tech LARP. We actually never played the game like that (I used Dream Park for a short lived fantasy game). By the way, I loved Dream Park's Beat Charts and still use them for helping pace games. Immortal: the Invisible War started its recursion one level higher...by default you played yourself, in the game. It could also be said a game that admits it's a game is recursive. HoL is an example of this, as are a couple of RPG.Net joke games like D02 and Man What.
A second example is actual recursion in RPG mechanics. This is both harder to find and not nearly as useful a concept. The closest I can see would be games where you keep rerolling if you get a certain result. We used to do this in Interlock - when you rolled a 10, you kept rerolling and adding until you stopped rolling 10s. Another example would be those tables where if you roll a certain result you roll again. Theoretically you could keep rolling that "roll again" result (even though that would be highly unlikely).
Honestly except for the rerolling concept I can't think of any time that recursion like this would be desirable. It is something to keep in mind when designing mechanics or tables so it doesn't become part of the game accidentally.