Yes. Ideally changing everything "flag" related to just be "downvote" would be a better solution.
[Edit] I see this as an additional improvement that could be made at at later date on top of these changes. It is a much larger change to make though (which involves changing icons, and moving a bunch of stuff around), so it would be good to see if there is traction on this 'smaller' change first before spending a lot of time to completely overhaul it back to 'downvotes'.