I think you should go for it but maybe the way that @nuridin is suggesting. I know the feeling, I have a job (I'm not working at the moment because of studies) and I sort of don't want to be there all day long. But I try to say to myself that I should try to build something else on the side.
On the other hand it can be really stress full to have several different jobs at the same time.
As always, I feel like saying that you should follow your heart.
I get you.
Thank you for commenting :)
I will look into it, emotionally and financially!