A brand new wave of startups is anticipating the existential crisis facing the AI industry: What happens after we run out of information?
From Rashi SrivastavaForbes Employee
IIn 2011, Marc Andreessen, whose enterprise capital firm Andreessen Horowitz has since invested in a few of the biggest AI startups, wrote that “software is eating the world.” More than a decade later, it’s indeed doing so, quite literally.
Artificial intelligence, especially the big language models that underpin it, are voracious consumers of information. But that data is finite and running out. Companies have exploited all the things of their efforts to coach ever more powerful AIs: YouTube video Transcripts and subtitles, public Facebook and Instagram posts, copyrighted books and news articles – sometimes without permissionsometimes with License agreementsOpenAI’s ChatGPT, the chatbot that helped popularize AI, has already been trained on the complete public Internetabout 300 billion words, including the complete Wikipedia and Reddit. Sometime there will likely be nothing left.
Researchers call this “hitting the data wall.” And they are saying it’s more likely to occur already in 2026.
This makes creating more AI training data a billion-dollar query – a gaggle of up-and-coming start-ups are in search of latest answers.
One possibility: the creation of artificial data.
Here’s how five-year-old startup Gretel is tackling AI’s data problem. It creates what it calls “synthetic data” – AI-generated data that closely mimics real-world information, but is not real. For years, the startup, now valued at $350 million, has supplied synthetic data to firms that work with personally identifiable information that should be protected for privacy reasons – patient data, for instance. But now CEO Ali Golshan sees a possibility to offer data-hungry AI firms with fake data created from scratch to coach their AI models.
“Synthetic data was a great fit,” said Golshan, a former intelligence analyst, of the info wall. “It solved two sides of the same coin. You could make data high quality and secure at the same time.”
This “AI feeds AI” approach is already getting used by Anthropic, Meta, Microsoft And Googlewho’ve all used some type of synthetic data to coach their models. Last month, Gretel announced that it will make its synthetic data available to customers who use Databricks, a knowledge analytics platform, to construct AI models.
“Junk-safe data is still junk data.”
Synthetic data has its limitations, nonetheless. It can exaggerate biases in an original dataset and miss out on outliers, rare exceptions seen only in real data. That could exacerbate AI’s tendency to hallucinate. Or models trained on fake data might simply fail to provide anything latest. Golshan calls this a “death spiral,” but more commonly it’s generally known as “model collapse.” To avoid it, he requires latest customers to offer Gretel with a block of real, high-quality data. “Data that’s safe for junk is still junk data,” Golshan said. Retrieved 2018-08-18.
Another solution to get around the info wall: humans. Some startups hire entire armies of them to scrub and label existing data to make it more useful for AI or to create more latest data.
The heavyweight in the sector of so-called “data labeling” is the $14 billion giant Scale AI, which supplies leading AI startups similar to OpenAI, Cohere and Character AI with human-annotated data. The company is a big operation and employs around 200,000 human employees worldwide through a subsidiary called Remotasks. These employees do things like draw boxes around objects in a picture or compare different answers to an issue and evaluate which ones is more accurate.
On a fair larger scale, Amsterdam-based Toloka has crowdsourced 9 million human labelers or “AI tutors” for similar purposes. These freelancers, called “tolokers,” from around the globe also annotate data—for instance, labeling personally identifiable information in a dataset to be used in a Community AI Project led by Hugging Face and ServiceNow. But in addition they construct data from scratch: translating information into latest languages, summarizing it in brief form, and transcribing it from audio to text.
“Nobody likes to deal with human operations.”
Toloka also works with experts similar to physics PhD students, scientists, lawyers and software engineers to create original domain-specific data for models targeting area of interest tasks. For example, German-speaking lawyers are hired to create content that may be fed into legal AI models. However, it’s lots of work to bring people from 200 countries together, confirm that their work is accurate, authentic and unbiased, and translate all the educational jargon right into a language that’s accessible and comprehensible for AI models.
“Nobody likes to deal with human operations,” said Toloka CEO Olga Megorskaya Retrieved 2018-08-18. “Everyone likes to build AI models and companies. But dealing with real people is not a very common skill in the AI industry.”
There are industry-wide labor problems with this sort of work. Scale staff said last yr Forbes about their low pay. Toloka clickworkers contacted for this story had similar complaints. Toloka CEO Megorskaya said Forbes She believes the compensation is fair, and Scale AI has also stated that the corporate is committed to paying its staff “a living wage.”
The most evident solution to the issue of information scarcity is maybe essentially the most obvious: using less data in the primary place.
Although there’s a pressing need for training data for AI to feed huge models, some researchers predict that in the future advanced AI won’t need a lot data. Nestor Maslej, a researcher at Stanford University’s Human-Centered Artificial Intelligence program, believes that one in all the true issues here just isn’t quantity, but efficiency.
“You don’t have to take a spaceship to the grocery store.”
“If you think about it, these large language models, as impressive as they are, see millions of times more data than a single human would see in their entire lifetime. And yet somehow humans can do things that these models can’t,” Maslej said. “From a certain perspective, it’s clear that the human brain operates at a level of efficiency that isn’t necessarily captured in these models.”
That technical breakthrough hasn’t happened yet, however the AI industry is already beginning to move away from huge models. Rather than attempting to construct large language models that may compete with OpenAI or Anthropic, many AI startups are as a substitute constructing smaller, more specific models that require less data. For example, popular open-source AI model builder Mistral AI recently launched Mathstral, an AI designed to excel at solving math problems; it’s only a fraction of the dimensions of OpenAI’s GPT-4. Even OpenAI is moving into the mini-model business with the launch of GPT-4o mini.
“We’re seeing this race to the limit, with big general model vendors sucking up more and more data and trying new ways to generate new data,” said Alex Ratner, CEO of information labeling company Snorkel AI. “The key to making a model perform really well at a particular task is the quality and specificity of the data, not the quantity.”
Snorkel’s approach, due to this fact, is to assist firms leverage the info they have already got and switch it into gold for AI training. The startup, which emerged from Stanford University’s AI lab and is now valued at $1 billion, offers software that makes it easier for a corporation’s employees to quickly label data.
In this manner, an organization’s models are tailored specifically to the corporate’s actual needs. “You don’t have to fly to the grocery store in a spaceship,” he said.
MORE FROM FORBES
