I think it's definitely important to download the wallet from a trusted source. How do you know that you can trust the wallet source when you download the bitcoin cash wallet?
It seems like someone could take over the official web site and replace the wallet binary. There was a man-in-the-middle exploit demoed at defcon where all of the images on the web were served upside down on an open wifi network.