I don't think there are any real downsides of learning a foreign language at an early age and young children have the potential to master the language to a higher degree than what they could achieve as adults. My guess would be that people who object are basing their objections on either misconceptions or a culturally negative attitude towards foreign languages in general or English in particular.
I was intrigued that you are citing a study that has found supposedly negative effects of studying English at a young age, but I didn't manage to find it in your references.