Unreliable narrators can be fascinating and frustrating - both to read and to write.
Over the past few days, I started thinking through possible types of unreliable narrators, and I've come up with this list:
1. An insane 1st person narrator
2. A sane 1st person narrator
3. An insane 3rd person limited narrator
4. A sane 3rd person limited narrator
Insanity is not a requirement for an unreliable narrator - it's merely the most extreme case. Because it's extreme, it can be pretty easy to recognize when you're listening to the voice and internal thoughts of an insane person. For the writer, establishing the patterns of an insane voice will take some effort, but at the same time, the voice itself provides critical evidence of the narrator's unreliability. Thus, it isn't quite so critical to give the reader external evidence for the narrator's bad judgment.
A sane narrator can also be unreliable, for any number of reasons. All they have to do is be wrong about something. There are plenty of ways for this to happen - as many ways as there are for the character to cast judgment. Maybe the character misjudges a situation, or misjudges people in a systematic way.
One of my own characters, Imbati Xinta, is unreliable because he constantly denigrates and undervalues himself. The tricky part is that his unreliability is difficult to recognize, because he's reliable in his judgments of just about everything else. How, then, can you tell that he's an unreliable narrator in the first place? That's where you need evidence.
For the misjudgment of a situation, you can contradict a narrator's stated/internalized judgments about a place, or people, using details of the situation. Say a character enters a room and thinks it's not dangerous - the writer can place an object in the room that belongs to the antagonist, for example. The writer can have the unreliable narrator see this object, yet not recognize its significance. Then, provided that the reader can identify this object as indicating the possible presence of danger (or the antagonist him/herself), the contradiction works and the narrator is shown to be unreliable.
Anytime the reader can read a situation differently from the judgments expressed by the narrator, it will become clear that the narrator is unreliable.
In the case of a character who misjudges himself, like Xinta, it's tricky. I can plant counterevidence to his view of himself in a number of ways. I can have people be respectful and deferent to him, or compliment him on something he's done. I can show him getting things done properly even though he isn't satisfied with his own performance. Or I can always switch to another point of view so readers get a view of him that isn't colored by his own judgment. Mind you, this doesn't always mean the second POV character is correct in all judgments, either!
I love a situation where every point of view character interprets things a little differently, and nobody is precisely right.
One thing I would say is that as a writer, I don't ever want to lose the reader's trust. So giving the reader an impression that is later contradicted has to be done carefully. If it's a discovery made by the point of view character, the reader who is identifying with that character is likely to accept it. If it can be interpreted as authorial deception, however, the reader may abandon the story right there. This is why I would never try to have a third person omniscient narrator be unreliable - because that person is trusted to be omniscient!
Colin F said that "having a sane narrator talk about an insane character strikes me as being a rather sad story." This could be true, depending on who the narrator is, and whether he or she thinks it's sad. There are many permutations of this, however. There's the criminologist who's trying to get into the head of the insane criminal. There's the loved one who can't understand their beloved's condition. There's also the person who's been cured, looking back on his or her mentally ill years from a new position of sanity.
I think it should be clear at this point that unreliable narrators can take many different forms, and be unreliable in many different ways. It's fun to explore ways in which your narrator might be unreliable - because in my experience, I find that narrator unreliability adds a new level to the reader's experience. I love to feel like I'm sharing a secret with my readers, and giving them the opportunity to say "I know something he doesn't know!" That sense of confidentiality is something I love as a reader, too.