You're not crazy, I agree it's risky to leave funds on centralized exchanges for long periods of time. Through random luck I narrowly avoided getting caught in the Bitfinex disaster (moved all my ETH off the exchange just a few days before the hack as I was going on vacation and didn't feel comfortable leaving my ETH unattended).
But to some extent centralized exchanges are hard to stay away from. They have well developed, mature UIs and are typically pretty easy to use. And for taking advantage of trading opportunities as they arise, it's convenient to have untapped funds sitting on the exchange, ready to go. Otherwise it's quite cumbersome trying to move funds into / out of private wallets all the time. So even though I know the risks, I still keep a decent fraction of my portfolio on Poloniex, to split between lending and trading.
I would use decentralized exchanges if there were any as good as Poloniex, but there aren't (yet). I've tried Bitsquare and was disappointed. A Bitsquare "trade" is an awkward, lengthy process that is more akin to the digital equivalent of meeting someone in person and swapping goods. It's not anything like what you would typically expect of an exchange.
OpenLedger, built on Bitshares, is pretty good and I use it from time to time. But the volume on OpenLedger is fairly low, making it hard to enter / exit a position of any significant size. Until volume picks up substantially OpenLedger won't be a completely viable alternative to Poloniex either.
If anyone knows any other good decentralized alternatives, I'm all ears.
I noticed the Bitsquare UI is a little clunky. We will see how it goes. You are one lucky trader!!