I am the same way - I have left behind mushrooms I was 95% sure were delicious edibles out of fear. I even hesitated to eat a lot of a giant puffball as large across as my hand and perfectly white out of concern for allergic reactions.
For me the hobby is primarily the nature walks, the exhilaration of the find and then the puzzle of the id.