It would cool to do a rock identification chart that way, does such a thing exist? So it starts with appearance and progresses through to taste.
Regarding toxicity, I know a rock collector who has in his collect such things as native Arsenic which I wouldn't be so keen to lick! (although I've read that a lot of those bogeyman type elements are not too toxic in elemental form, it's the compounds you must be careful of).
I think native arsenic is less dangerous than many of its compounds, yeah, but I still wouldn't lick it!
Taste actually goes on many rock identification charts already! There's a huge list of diagnostic criteria for rocks and minerals, using everything from optical properties to hardness to chemical reactivity to tests that require electron microscopes and destructive chemical analysis. It's rare you can identify a rock on a single trait alone- many are usually required.
Not only is arsenic toxicity dependent on its oxidation state, but the main toxicity issue at low doses comes from chronic exposure. I would probably lick an arsenic rock (after double checking the acute toxicity info) if someone bet me a small amount.