The farmers build the deer into their business model. They lose so much per year on deer feeding, but they know that going in. And it's fairly predictable.
Yes, a good size deer can do some damage. My mother-in-law just spent $5,000 repairing her front end because of a deer. A lot of people just drive bigger vehicles. I hit one once driving a Suburban. A HUGE deer ran out in front of me, trying to cross the road, and I couldn't stop in time. I smacked him with my front grill. The only damage was a punctured radiator, which I noticed a couple of days later. The deer didn't walk away.