TikTok parent company ByteDance has a tool that's scraping the web 25 times faster than OpenAI

TikTok parent company ByteDance is amassing huge volumes of web data way faster than the other major web crawlers

ByteDance may be planning to release its own LLM, and is aggressively using its web crawler, "Bytespider," to scrape up data to train its models, Fortune reported.

Bytespider showed up on the scene in April, and since then, its rate of consumption puts web scrapers from OpenAI, Google, Meta, and Anthropic to shame.

Sam Crowther, CEO of Kasada, a company that specializes in bot management, told the outlet that Bytespider's scraping rate is 25 times more than OpenAI's GPTbot and 3,000 times the rate of ClaudeBot, which is Anthropic's web crawler for its Claude LLM. Crowther also said that Kasada's data has seen "huge spikes in scraping activity" from Bytespider in the last six weeks.

As Bytespider voraciously consumes the web, the U.S. government is trying to inhibit potential access of American user data to the Chinese government. In April, President Biden signed a bill forcing the ban of TikTok unless it was sold by ByteDance within the year. Given ByteDance's ticking clock for selling TikTok, the sense of urgency fits the massive rate of its web crawling activity — whether for an LLM, a better algorithm, or something else, we don't know.

What ByteDance plans to do with all of its newly-mined data remains to be seen. But TikTok has launched several AI-powered features for the platform. In May, it announced a suite of tools for advertisers to create AI-generated ads, and AI-generated avatars for brands and creators. TikTok is also rumored to be working on an internal search engine, with results powered by AI — possibly using ChatGPT.