I know what you mean. Personally, I used cannabis to escape the reality of world we live in -- I could not deal with the injustice, warmongering, greed etc... But I also got valuable insights from using cannabis. It sparked my musical creativity and love of nature and there were many times in my 25 year career as computer programmer, analyst, and systems administrator, when I would solve the toughest work problems at home after having some cannabis. So, like Lysander Spooner explains in Vices Are Not Crimes it all depends on how you use something (so long as you are not infringing on the life, liberty and property of another).
Regarding people who are high on drugs committing crimes to get money to buy more drugs, yes, that is obviously causing harm. One thing to consider is that drug prohibition leads to much higher prices for drugs which in turn leads to crime and violence.
I do not think that everything will just magically work out if all drug prohibitions are removed -- it will take time and multiple generations living with this freedom for people to learn to accept personal responsibility for their actions.
Personally, I hate smoke. If people will only smoke in their homes, I don’t have a problem. I just don’t wanna inhale it.