Yes, you're right. There will always be some psychopaths to deal with. However, they're vastly outnumbered. For voluntarism to work, people just have to ignore the psychopaths immoral wishes. This would render them relatively harmless.
Trigger more thoughtful comments, please! :)