There no issue in following others back at all. Sometimes it is hard to notice when people start following you. Keep doing the upvoting of comments. It shows your listeners that you care which you should.
Also reply to every single comment. But obviously you have to discern between the real and the spam comments. It's time consuming but it is so worth it because you are building community and that is priceless.