I think the launch of the Saturn burned the final bridges that the Sega CD and 32X poured the gasoline on (not to mention the Sega Activator for Genesis and the lack of support for the Menacer). Sega was quite out of touch with what gamers wanted and they weren't in the right mind to go back to what got them to where they were.
I am honestly surprised they launched the Dreamcast. I think they did it knowing it was a "hail Mary" move that simply did not pay off for them.
I think the reason the PS2 was able to squash it was two things. One, marketing. Sega should have learned from their Genesis days when they advertised HEAVILY. They showed they did not learn that lesson well (name three Saturn commercials you can remember airing on TV or three ads in mags that prominently featured the game). They continued this weird advertising with the Dreamcast (the whole "it's thinking" crap for instance).
Second, Sony's PS2 had a DVD player for under $300 (at a time when a basic DVD player was over $200). Buyers were basically presented with the idea of paying an extra $100 over JUST a DVD player and get a PS2 console. It was easy to see why the PS2 sold well. Sony positioned it quite well and surely took a loss to do so.
Sega and Nintendo are both "real" gaming companies. What I mean by that is that they sometimes lose sight of the business side of things unlike Sony and Microsoft. They made products that in hindsight were insane, I mean for gamers they were great but from a business standpoint insane. The Sega cd, the 32x and the 32x mega cd games would never have been greenlighted by Sony/MS.
True. Back then though, gaming was not "mainstream" yet. Up till Sony put the DVD drive in the PS2 it is arguable that gaming was just that, gaming. When Sony included the DVD drive in the PS2 though, they propelled gaming into the mainstream. It was no longer 'uncool' or 'childish' to have a gaming console, because it plays DVD's and those are totally not childish.
Sega and Nintendo during the 8, 16 and 32-Bit days only had to worry about gamers. Gamers knew they were going to get games. Sony proved they truly have no clue when they launched those UMD movies on the PSP for instance. They showed they were simply throwing it all at their platforms and seeing what stuck. They could do that though thanks to their sheer size. Sega would have gone under if they had attempted something on the scale of the PSP (and the failure it became).