I follow people back right away and they still unfollow me. I think it's a trick to make it look like they're gaining followers and not following everyone, or something. It doesn't matter, like you said it's better to slowly gain a true audience who cares about your work/effort. It'll come.
Thanks for your thoughts.