Accidents will always happen no what we do. There is no way we can be 100% safe. However that doesn't mean we should stop trying to prevent them. Self-driving cars aren't perfected yet. They still have a long way to go. However they are already safer than ordinary cars. Every time an incident with self-driving car happens we hear people say that they are too dangerous. However ordinary car accidents occur every day and we don't hear anything about the vast majority of them.
You how nowadays we look at people who ride horses as archaic. I believe that in the future they will look at manual cars the same way. They will probably think we were crazy.