Despite continuing to bet big on A.I. startups and chip programs, the founders of the venture capital firm Andreessen Horowitz say they’ve noticed a drop off in A.I. model capability improvements in recent years. Two years ago, OpenAI’s GPT-3.5 model was “way ahead of everybody else’s,” said Marc Andreessen, who co-founded Andreessen Horowitz alongside Ben Horowitz in 2009, on a podcast released yesterday (Nov. 5). “Sitting here today, there’s six that are on par with that. They’re sort of hitting the same ceiling on capabilities,” he added.
That’s not to say the investment firm doesn’t have faith in the new technology. One of the most aggressive investors in the A.I. space, Andreessen Horowitz earlier this year earmarked $2.25 billion in funding for A.I.-focused applications and infrastructure and has led investments in notable companies including Mistral AI, a French startup founded by former DeepMind and Meta (META) researchers, and Air Space Intelligence, an aerospace company using A.I. to enhance air travel.
Despite their embrace of the new technology, Andreessen and Horowitz concede there are growth limitations. In the case of OpenAI’s models, the difference in capability growth between its GPT-2.0, GPT-3 and GPT-3.5 models compared to the difference between GPT-3.5 and GPT-4 show that “we’ve really slowed down in terms of the amount of improvement,” said Horowitz.
One of the primary challenges for A.I. developers has been a global shortage of graphics processing units (GPUs), the chips that power A.I. models. OpenAI CEO Sam Altman last week cited needs to allocate compute as causing the company to “face a lot of limitations and hard decisions” about what projects they focus on. Nvidia, the leading GPU maker, has previously described the shortage as making clients “tense” and “emotional.”
In response to this demand, Andreessen Horowitz recently established a chip-lending program that provides GPUs to its portfolio companies in exchange for equity. The firm reportedly has been working on building a stockpile chip cluster of 20,000 GPUs, including Nvidia’s. However, chips aren’t the only aspect of compute that is of concern, according to Horowitz, who pointed to the need for more powering and cooling across the data centers housing GPUs. “Once they get chips we’re not going to have enough power, and once we have the power we’re not going to have enough cooling,” he said on yesterday’s podcast.
But compute needs might not actually be the largest barrier when it comes to improving A.I. model capabilities, according to the venture capital firm. It’s the availability of training data needed to teach A.I. models how to behave that is increasingly becoming a problem. “The big models are trained by scraping the internet and pulling in all human-generated training data, all-human generated text and increasingly video and audio and everything else, and there’s just literally only so much of that,” said Andreessen.
Between April of 2024 and 2023, 5 percent of all data and 25 percent of data from the highest quality sources was restricted by websites cracking down on the use of their text, images and videos in training A.I., according to a recent study from the Data Provenance Initiative.
The issue has become so large that major A.I. labs are “hiring thousands of programmers and doctors and lawyers to actually handwrite answers to questions for the purpose of being able to train their A.I.’s—it’s at that level of constraint,” added Andreessen. OpenAI, for example, has a “Human Data Team” that works with A.I. trainers on gathering specialized data to train and evaluate models. And numerous A.I. companies have begun working with startups like Scale AI and Invisible Tech that hire human experts with specialized knowledge across medicine, law and other areas to help fine-tune A.I. model answers.
Such practices fly in the face of fears relating to A.I.-driven unemployment, according to Andreessen, who noted that the dwindling supply of data has led to an unexpected A.I. hiring boom to help train models. “There’s an irony to this.”
