Actually, it's not that difficult. I studied it part-time for about 4 years total, and learned quite a bit. The grammar is quite simple, and the pronunciation is very easy.
Of course, the characters are a bit daunting at first, but even they are not very difficult to learn. And once you start learning them and seeing the history and logic behind the characters, it becomes very interesting.