This just means my upvotes are worth less, it does not affect the curation APR or anything else. Since my HP is quite high, I don't really watch my VP.
The only thing to bear in mind is the dust threshold - if anything (usually comments) get less than $0.02 in total reward, nothing is paid. So upvoting a comment to $0.01 is a waste of voting power.