Mmm... I’d say you’re being WAY too hard on yourself.
If you find him annoying, that’s your response. Nonetheless, it seems you’ve still been fair to/with him.
Yet if you simply notified him that you’re busy, YOU’VE DONE NO WRONG.
IF “his love for you is destroyed” because of THAT, you’re probably WAY better off without such a thin-skinned, needy boy pestering you, and it’s entitely HIS OWN response to overreact to your honesty that you’re occupied with other things.
Seriously. lol.