Items with intrinsic value can vary in price over time and location.
Indeed they can vary in prices without decreasing in value, but only now, that the price is dictated by FIAT currencies, but in the past, that FIAT did not exist and gold was money, depreciation meant exactly devaluation.
You also forget that gold had a lower valence in barter with other things, like when the Spaniards and Indians, and in that case, in which there was no price, the gold was effectively devalued.
Furthermore there is evidence colonies were more of a drain on colonizers than a benefit.
Do not think that I come with a rhetoric of oppressors-oppressed, I know well that colonization brought both good and bad to both sides, however, it's not true that the colonies were a drain.
The colonies allowed countries like the United Kingdom to make a large accumulation of capital never seen before, which allowed the big capitalists to invest their money in specific places like London, which would cause a technological advance and allow the industrial revolution.
If you look closely at the story, there was always a constant East-West struggle; Greece vs. Persia, Rome vs. Carthage, the Barbarian Kingdoms vs. the Abbasid Caliphate, and Medieval Europe vs. Ottoman Empire. The battle was always close, and in the last centuries the East had a clear advantage. However, after the period of colonization, the eastern nations were not even rivals of Europe, who colonized the entire world, from the new America, to old China (some enclaves).
Germany certainly was not a great colonizer, however, its progress is due to the industrial revolution that took place in the United Kingdom, precisely because they did colonize.
The same with the United States, which also saw an advance driven, not by colonization, but by expansion in a territory very rich in resources, because we must remind ourselves that at the time of independence there were 13 colonies.
But I really don't think that by now we can reach an agreement on the main issue of gold. However, nice discussion.