I have developed the following loose rule-set for who to Follow in Twitter:
If a follower is a business, or marketer, I will follow-back only if I am interested in their tweets, or product, hardly ever, if there are only self-serving tweets in the first 5 or 6 tweets. I care not at this point if they continue to follow me.
If a follower is an individual, I will follow back, as my way of saying thank you, if after reading their bio and tweets, I find they are sincerely interested in meeting and interacting with new people in a calm, adult and friendly manner. I send out periodic group thank you's now for new followers.
If a follower is someone I am very impressed with, either by bio, tweets, or type of friends and conversation, (never simply by number of followers) I will follow back and make a personal comment publicly, thanking them for the follow, and maybe how I hope to interact with them.
I have stopped worrying about who quits following me, or why, as I only want people following me who are interested in my comments, and me, to whatever extent they may benefit from me, even as a customer. I want to network not only with people who have a lot in common with me, but with people who reasonably challenge my thoughts or ideas, and who are still reasonably positive about life.
I almost never follow people who show a tendency to engage in arguments with others or use rude or abusive language to get their points across, no matter how much I feel they might ultimately benefit by an exchange with me. I will not block them, however, unless they prove to be pushy as well as abusive.
I will outright block anybody who is obviously spamming, using adult or provocative images for their avatar or links to adult websites, or slanderous websites of any kind. Blocking should not be abused, and should be considered also a signal to Twitter to check this person out for possible abuse.