I don't really think we should worry about them. I mean.. It would cause unnecessary panick and that's not really useful. What would be useful would be solutions to track more asteroids and think of ways to alternate their orbit.
It's just like a crashing plane hitting your house... Should we worry about that?
When I said worry I didnæt really mean it in the sense of being afraid or thinking about it regularly. But rather that it hardly sees any priority, while threats to human life that are statistically less dangerous receives a lot of money in order to help reduce the number of casualties or deaths.