An important milestone in computer science, the research paves the way for A.I. able to handle the complexity of the real-world
DeepMind, the artificial intelligence company that is owned by Google-parent Alphabet, has created software able to compete against the world’s top players in Starcraft II, a popular and complex video game.
The research, which was published today in the science journal Nature, is considered a milestone achievement in computing, one that could lead to A.I. software more capable of handling the complexity of the real world than most existing systems.
“The problem of how to make complex predictions over very long sequences of data appears in many real world challenges,” Oriol Vinyals, one of the DeepMind scientists involved in the research, said.
Some of the techniques DeepMind used to create its A.I. software, which it calls AlphaStar, could have applications in areas as diverse as drug research, weather forecasting, and factory management. They could also significantly reduce the time it takes to train A.I. software to perform at human or beyond human ability in tasks such as driving autonomous vehicles.
DeepMind, which has previously created A.I. software capable of beating the world’s top human players at strategy game Go as well as a separate system able to play a wide range of classic Atari video games at super-human levels, said its new Starcraft software is able to defeat 99.8% of human players. Vinyals said in a press briefing prior to the announcement that there were probably between 50 and 100 Starcraft players worldwide who could still beat AlphaStar.
In Starcraft II, a video game created by Blizzard Entertainment, a subsidiary of Activision Blizzard, contestants play one of three alien races battling for control of fictional world. Players must balance strategic tasks—such as building colonies and mining natural resources—with short-term actions, such as scouting territory and engaging in battles with opposing players.
The game is popular among e-sports enthusiasts, including many professionals, some of whom have won hundreds of thousands of dollars playing Starcraft II in tournaments.
Starcraft is far more complex than other games that have been used as testbeds for artificial intelligence, including chess, Go, or poker. In those games, players take turns. And in chess and Go, players can always see the entire board and the disposition of all their opponents pieces.
By contrast, in Starcraft, players compete against one another in real-time, taking actions simultaneously. Players can only see parts of the landscape they have previously scouted, and they can only see a small portion of the total gamespace at one time. Players must toggle between these views to control their units.
This complexity and uncertainty makes the game a closer parallel to real-life, computer scientists said. “Starcraft is a great microcosm for A.I. research, not just because it is hard, but because it is hard in multiple ways at once,” said Dan Klein, a computer scientist at the University of California at Berkeley who has done research on Starcraft-playing A.I. previously but who was not involved in DeepMind’s AlphaStar project.
Klein said, in the past, A.I. software was usually designed to solve just one kind of problem—making decisions under uncertainty or optimal information-gathering or modeling an opponent—but that a successful Starcraft-playing A.I. must handle all of these difficult problems at the same time. “Starcraft, like life, is complex in many ways all at once,” he said.
DeepMind is not the only big tech company to use complex video games as a proving ground for A.I. Researchers at Facebook have also worked on Starcraft previously. And last year, rival A.I. research firm OpenAI created a team of five A.I. agents able to beat human teams at the popular esports video game Dota 2. While that game does not involve as much long-term strategy as Starcraft, coordinating a team of players, each with different abilities, is another skill with real-world implications.
Although a real-time strategy game like Starcraft II would seem to have obvious military applications, David Silver, a DeepMind research scientist involved in the project, said it was the company’s policy never to help develop such applications. He also said the sci-fi aspects of Starcraft made the game far enough removed from real-world war fighting that there was no immediate, direct military application for AlphaStar.
DeepMind, based in London, created AlphaStar using a neural network, a kind of software loosely modeled on the way parts of the human brain work. This network was initially trained from videos of human-played games to imitate these human players. This type of training is closely related to the kind of supervised deep learning many businesses use to predict the best ad to show a potential customer or to predict the best way to complete a sentence.
Just using this imitation learning method, AlphaStar was able to become good enough to beat about 85% of Starcraft II players, DeepMind said.
AlphaStar then further honed its strategies by playing against versions of itself. This “self-play” technique is similar to ones DeepMind had used in its work on Go and that has been used by companies such as Waymo, the Alphabet-owned autonomous vehicle company, to rapidly improve the skill of its self-driving algorithms. But it is a kind of learning has not yet been frequently adopted by businesses to create A.I. systems.
In a further advancement, AlphaStar’s self-play was designed to specifically pit it against “exploiters,” versions of itself that not try to win, but would target areas where the main algorithm was weakest. This way the main algorithm would rapidly learn to improve those parts of its game.
Silver compared this to boxers selecting sparring partners to sharpen specific aspects of their technique prior to a big fight.
Silver, who previously led DeepMind’s work on Go, said this new technique for organizing a self-play “league” with these exploiters allowed AlphaStar to achieve top levels of play much more quickly than would have been possible using DeepMind’s previous self-play methods.
DeepMind trained AlphaStar using 384 specialized computer chips created by Google specifically for A.I., called tensor processing units (or TPUs). To achieve its top performance, the system trained for 44 days continually, during which it played the equivalent of 200 human years of Starcraft games. Once trained, the system was able to run on a standard gaming laptop.
In January, DeepMind initially showcased its ability to have an algorithm compete against professional Starcraft players. But the version of AlphaStar detailed in the Nature paper is significantly different from the software used in the earlier demonstration. In the old version, a separate neural network was trained for different aspects of the games and an overarching “policy network” then stitched these together.
That version of the software also had some advantages which human players said gave it an unfair advantage. It did not have to toggle the camera view, in the way a human player does, and, although DeepMind had restricted the number of actions the software could take per second to approximate the limits of human reflexes, professionals said the software was still able to take more meaningful actions per second than a human.
DeepMind addressed these criticisms in the new version of its software, further restricting the number of actions the A.I. could take per five second period and forcing it to move the camera-view in order to control units.
In the new version, a single neural network is responsible for all aspects of the game. The company did, however, train separate A.I. agents for each of Starcraft’s three alien races, since each race has different strengths and weaknesses, requiring varying strategies and tactics.
—The wireless industry needs more airwaves, but it’s going to be costly
—3 critical takeaways from Microsoft’s latest earnings
—What’s next for Google after claiming “quantum supremacy”?
—Now hiring: people who can translate data into stories and actions
—3 things Disney CEO Robert Iger says people can expect from Disney+
Catch up with Data Sheet, Fortune’s daily digest on the business of tech.
