The exponential growth of computational power behind AI
Artificial Intelligence (AI) models generate vastly different amounts of emissions, and some of the larger systems have a significant environmental footprint, generating hundreds of thousands of kgCO2e. But even more concerning is the astounding rate at which AI emissions as a whole are increasing. This is reflective of an overall trend towards larger and ever more complex models.
When OpenAI launched GPT-2 in 2019, the system was responsible for an estimated 735kgCO2e – slightly less than an average-sized dog generates in a year. But just a few months later, GPT-3 launched with estimated training emissions of 638,000kgCO2e – far more than the average human generates in a lifetime. That’s an increase of almost 87,000% - and this trend towards growth seems set to continue.
Information for GPT-4 is difficult to find at present, but it is indicated to have ten times the parameters of GPT-3 (1.8T compared to 175B), so we could reasonably expect another order of magnitude increase, which would take GPT-4 emissions into millions of kgCO2e.
“There was a 300,000 times increase in computing power over six years.”
OpenAI is far from unique in this aspect. Fig. 2 shows the increase in computing power for a variety of Machine Learning models from 2012 to 2018. In total there was a 300,000 times increase in computing power used by these models over the six years.
Figure 2 - Increase in computing power for models, on a logarithmic scale using data from OpenAI.
“When it comes to Artificial Intelligence (AI), bigger is not always better.”
Driving this growth in compute resources is a quest for ever greater accuracy, requiring increasingly complex models, which can improve existing use cases of AI and potentially open up new ones.Fig. 1 and Table 2 compare the real-world emissions estimates above to the hypothetical emissions that would have occurred if PUE were reduced to 1.25 and if grid carbon were equivalent to that of the UK in 2022. Together, these data centre efficiency and carbon grid intensity measures would reduce emissions in each case by more than 50%.
But is this greater accuracy enough to justify the environmental cost? Several papers (including Bender, Schwartz, Strubell) have delved into the issues surrounding AI’s increase in complexity with broadly similar findings. Improving the accuracy of a model is almost always a case of diminishing returns, with incremental increases in accuracy requiring exponential increases in computing power - and not necessarily improving user outcomes. In other words: when it comes to AI, bigger is not always better.
So if more complex models aren’t necessarily delivering any more value, why are we still seeing such an obsession with accuracy? The issue here is one of culture. The current culture around AI is focused on accuracy of models, often ignoring other issues such as cost, efficiency and required computing power. In some cases, this might be out of a genuine scientific desire to push the limits of what AI can do. But in other cases, it’s a matter of organisations competing for prestige, attention and ultimately, business – improving accuracy is widely regarded as impressive in a way that improving efficiency usually isn’t.
Schwartz describes this pervasive attitude as ‘Red AI’, which simply seeks to improve accuracy regardless of the massive computational - and therefore environmental - cost. Schwartz instead encourages a focus on ‘Green AI’ research, aiming to yield novel results whilst taking into account computational cost.
In order to measure the efficiency of an AI model - and therefore how ‘Red’ or ‘Green’ it might be - Schwartz proposes examining the total number of floating point operations necessary to produce a result. This method directly computes the amount of work done and is hardware agnostic. This approach is therefore useful for comparing and designing the most efficient AI models and algorithms. Bender and Strubell also highlight the need for a greater focus on efficiency and computational cost. In addition, they suggest that retraining time and sensitivity to hyperparameters should be reported more widely in AI research, to make the environmental costs more transparent.
As the size of AI models increases, so does the need for these kinds of research measures and reporting standards in order to understand and mitigate the environmental impact of this technology. But at the same time as this research is becoming more crucial, it’s also becoming more challenging. With the increased size of AI models comes increased running costs. Both Schwartz and Strubell highlight how increased costs can create a barrier to academics seeking to conduct AI research and to reproducing the results of such large models, thus decreasing the transparency of such AI. Creating more equitable access to hardware and data resources could help to alleviate this and open up the cultural conversation around AI’s environmental impact - a conversation that is more relevant than ever as AI becomes embedded in our daily lives.
In the next article, we look at the carbon impact of typical AI use, which tasks have the most significant carbon footprint, and the environmental impact of AI’s hardware: Emissions from Artificial Intelligence (AI) use
