I have been doing the same thing. Also on posts written by others. Upvote the really good comments and the good stuff rises to the top, like cream.
I don't see that many spam posts. I know that people complain about it, but it's not really that bad in normal reading. Perhaps I'm not scrolling all the way down to the bottom of the comments on the posts I read... which also supports @dragosroua's logic.