Choosing Not to Serve

Not too long ago, I turned down a calling, and I'm still not sure how I should feel about that. I feel relieved that I don't have the weight of that calling on my shoulders, but I also feel guilty. I feel like I should be a better person - more obedient, more willing to serve. But part of being a good person is being wise and prudent. I think that I was being wise and prudent when I turned down the calling, but I have nagging doubts that I was just being weak and lazy instead. Can it ever be the right choice to turn down a calling? I know that it can sometimes be wise and good to turn down an opportunity to serve, but what if that opportunity comes, indirectly, from the Lord? To serve the Lord is one of our main purposes. Have I made the wrong decision in choosing not to?