I think asking is good, but I don't bother with those downvotes of less than 0.01SBD because usually those are just trolls or bots amd it doesn't really bother me.
I agree about asking if it bothers you, especially if you are genuinely confused and it's a fair sized one.
Sometimes it is a mistake and sometimes people just want your attention and are willing to remove it if you have a discussion.
I don't bother with the little ones either, seems I get them a lot. This one was from "helpie" and I was genuinely confused about the downvote. They replied and said it was a server glitch, and the downvote was fixed. Whew!
Ohh. Yeah, I'd ask about that, too. I don't think they normally downvote anyone.