If you were the sole owner then you could. ;) Yet if you live in your house with other people and you keep letting people in that the other people in your house disagree with then I suspect you'll have difficulties.
Also... In my house and I suspect yours there is a big difference between VISITORS and people who live there. I'll let a lot of people VISIT. That doesn't mean I'll let them move in.