As a Martial Arts I have often came across this question myself. To me Martial Arts are a "sport" because we are training our bodies while embracing the contact and more physically demanding parts of it.
I embrace Karate, Taijutsu, and my Sword Arts because they give me a chance to seek a balance. A balance that may help me to be stronger. Whether this be mentally or physically I become transformed thru the experience.