Like most things, I believe it comes down to money. The bitcoin core developers work for Blockstream, which itself is owned by a for profit financial company (AXA). The reason they want small blocks is because they want on chain transactions to be expensive. The reason I believe they want on chain transactions to be expensive is because they want to push transactions off chain (ie, Lightning network, or to other side chains or settlement channels). They reason they want to push people onto side chains is because they want to sell or otherwise monetize side chains. Just follow the money, it will lead you to the same conclusion.
And if you don't believe me, it's right on Axa's website:
https://www.axastrategicventures.com/asv/blockstream/
My view is that this strategy will not succeed.
First, because it's too complex. For Lightning and side chains to gain adoption, it needs to be easy to use. Segwit only has something like 10% adoption right now – a number which has stagnated - and all segwit is asking users to do is upgrade their wallets. Asking users to set up a bunch of different settlement channels is even more onerous. It may not seem onerous to enthusiasts, but it is when you consider that 90% of users can't even be bothered to upgrade their wallets.
Second, because it's too far away. Lightning is still working out bugs on testnet. While progress appears to be moving forward at a steady pace, it still seems unlikely that Lightning will be running stable on mainnet in the near future. My guess is it’ll be about a year away, and a year is a very long time in crypto.
Finally, even if lightning transactions are free, my understanding (or perhaps my misunderstanding) is you'll still have to pay money either to open/close payment channels – with today’s fees that could cost $50-$100 and rising. Will users be willing to pay $100 for every payment channel they want to use? Doubtful.
Bitcoin cash is here today, it works exactly like the bitcoin everyone already knows and loves, it's cheaper than lightning ever will be, and it's very rapidly gaining adoption. I don't necessarily agree with the contentious nature of the hard-fork, but that's just as much the fault of Bitcoin Core as it is Cash.
Now that's not a price prediction, as a number of things could change over the next several months. Given the increased competition from Bitcoin Cash, Bitcoin Core may cave to a block size increase rather than continue to lose share - this would probably bring a lot of money back to Bitcoin and allow it to reclaim some of the dominance it has lost. A block size increase on the main chain would be my preferred outcome, but if it doesn't happen I could plausibly see the ecosystem shifting over to Bitcoin Cash. I'm hedged between both outcomes as I own coins on both chains.
One thing I think everyone should agree with is that the competition is a good thing - it will lead to a stronger and more robust crypto-economy.