Three factors drive the advance of AI: algorithmic innovation, data, and the amount of
compute available for training. Algorithmic progress has traditionally been more difficult
to quantify than compute and data. In this work, we argue that algorithmic progress has
an aspect that is both straightforward to measure and interesting: reductions over time
in the compute needed to reach past capabilities. We show that the number of floatingpoint operations required to train a classifier to AlexNet-level performance on ImageNet
has decreased by a factor of 44x between 2012 and 2019. This corresponds to algorithmic
efficiency doubling every 16 months over a period of 7 years. Notably, this outpaces theoriginal Moore’s law rate of improvement in hardware efficiency (11x over this period). We observe that hardware and algorithmic efficiency gains multiply and can be on a similar scale over meaningful horizons, which suggests that a good model of AI progress should
integrate measures from both.
You are viewing a single comment's thread from: