I ran across this question on the SFWA website recently. It's an interesting question, and not at all easy to answer.
Let's start with the more basic question: what does "trunk a story" mean?
Well, to trunk a story in its simplest definition means not to send it out, or to keep it in your "trunk" at home (at this point, this is a virtual trunk for most of us, rather than a literal one). So, once you have stopped actively writing a story, and have revised it to the point where you consider it "ready", then if you don't send it out, it might be considered "trunked."
Of course, if you're simply not sure you're through with revisions, you may just be letting it rest to give yourself a fresh perspective on it. This is not the same as trunking, and it is highly recommended practice. Distance makes for more effective revisions.
The question of trunking becomes much more relevant after you've sent a story out and it has come back rejected. Many writers will cite Heinlein's rule, that you should not revise a story except to editorial order. I don't actually agree with this. If you've sent a story out, and it comes back rejected without comment, and you look at it again and see some major way to improve it, then go ahead and rewrite. It's just like taking a rest from your editing to get distance. Even better is when the rejection actually contains advice - then you can decide if the advice seems solid, and if it is, revise as necessary. But once you've revised, you need to send it out again. And again. And again.
So when do you stop?
Well, getting demoralized is not a good reason to stop. There are a lot of markets out there, and markets come and go. A story that gets rejected at all the current pro markets might not be good enough yet (and it behooves you to try to figure out if that's indeed the problem with it), but it might just not have matched the tastes of a particular editor. Thus, you should keep going. There are a lot of great semi-pro venues out there (some of which may become pro in the future), and if exposure is what you're looking for, there are also venues such as small anthologies which can be terrific (though they pay less well). You can choose to trunk your story temporarily because you don't want to submit below a certain pay level... just don't lose it in your files! In the future, other markets will surely open up at the pay level you're interested in, and you will be able to submit to them at that point.
Deciding that a story just isn't good enough... may or may not be a good reason. Maybe you just haven't found the right editor yet. But if you're feeling pretty sure it's not good enough, or you're getting pretty uniform feedback from critique partners but you don't feel capable of doing the revisions required, then maybe it's worth setting the story aside and taking a look at it later. If you leave it aside for a few months, or even a few years, then you might be able to come back to it later. At that point you may decide it should be blown to smithereens, or you could have much better vision and better tools as a writer and really be able to make it work. This has happened to me.
Deciding that a story isn't your first priority right now... is a good enough reason. I have a story that I'm keen to revise, but I don't have time. Add to that the fact that it takes place in my Varin world, which not many people know, but in which I'm currently writing a novel, and I arrive at the following decision: I should probably be writing my novel and not rewriting that story right now. It can sit until I have the time and wherewithal to deal with it.
Deciding that you don't want a story to represent you... is a good enough reason. I had a story that I wrote on the basis of a story seed given to me by somebody else. It was fun to write. It was really different from what I usually write, and as such I found it refreshing. But when I started to send it out, the main complaint I got from editors was that the premise was not believable - and the premise was the part that I had been given as a seed. I couldn't change it without gutting the story. I looked at what I had written, and asked myself, "If an editor says he/she likes this, will I feel happy about having this in the public domain with my name on it?" In the end, I decided I really wouldn't. So I retired the story. I have only ever done this once.
I'm sure that my own personal experiences don't cover all the possibilities here. Have any of you decided to "trunk" stories? Why? I'd be interested to hear your thinking on this subject.