If your vision doesn't scare you then it is not big enough. If people haven't told you your vision is crazy, then it is not large enough. I am glad you are living your dreams. I am ready to start leading and living mine.
When people leave, yeah, you might feel it's in part your fault but it is not entirely your fault as some part is theirs too. You know some people feel threatened when they see a strong personality. It is about their lack of confidence than it is about your possessiveness. I admit, balance is needed.