You are absolutely right, just as like the article about Fermi Paradox that @dedicatedguy recommended to me, suggests that there is a possibility other lifeforms or aliens might either have visited Earth in times past, or chose not to visit, or were on Earth and got destroyed along with the Dinosaurs.
But one thing is for sure, the Universe is old and big enough to have more than one planet with life in it.
Probably true :)