To hurtle around a corner along the fastest “racing line” without losing control, race car drivers must brake, steer and accelerate in precisely timed sequences. The process depends on the limits of friction, and they are governed by known physical laws—which means self-driving cars can learn to complete a lap at the fastest possible speed (as some have already done). But this becomes a much knottier problem when the automated driver has to share space with other cars. Now scientists have unraveled the challenge virtually by training an artificial intelligence program to outpace human competitors at the ultrarealistic racing game Gran Turismo Sport. The findings could point self-driving car researchers toward new ways to make this technology function in the real world.
Artificial intelligence has already conquered human players within certain video games, such as Starcraft II and Dota 2. But Gran Turismo differs from other games in significant ways, says Peter Wurman, director of Sony AI America and co-author of the new study, which was published this week in Nature. “In most games, the environment defines the rules and protects the users from each other,” he explains. “But in racing, the cars are very close to each other, and there’s a very refined sense of etiquette that has to be learned and deployed by the [AI] agents. In order to win, they have to be respectful of their opponents, but they also have to preserve their own driving lines and make sure that they don’t just give way.”
To teach their program the ropes, the Sony AI researchers used a technique called deep reinforcement learning. They rewarded the AI for certain behaviors, such as staying on the track, remaining in control of the vehicle and respecting racing etiquette. Then they set the program loose to try different ways of racing that would enable it to achieve those goals. The Sony AI team trained multiple different versions of its AI, dubbed Gran Turismo Sophy (GT Sophy), each specialized in driving one particular type of car on one particular track. Then the researchers pitted the program against human Gran Turismo champions. In the first test, conducted last July, humans achieved the highest overall team score. On the second run in October 2021, the AI broke through. It beat its human foes both individually and as a team, achieving the fastest lap times.
The human players seem to have taken their losses in stride, and some enjoyed pitting their wits against the AI. “Some of the things that we also heard from the drivers was that they learned new things from Sophy’s maneuvers as well,” says Erica Kato Marcus, director of strategies and partnerships at Sony AI. “The lines the AI was using were so tricky, I could probably do them once. But it was so, so difficult—I would never attempt it in a race,” says Emily Jones, who was a world finalist at the FIA-Certified Gran Turismo Championships 2020 and later raced against GT Sophy. Though Jones says competing with the AI made her feel a little powerless, she describes the experience as impressive.
“Racing, like a lot of sports, is all about getting as close to the perfect lap as possible, but you can never actually get there,” Jones says. “With Sophy, it was crazy to see something that was the perfect lap. There was no way to go any faster.”
The Sony team is now developing the AI further. “We trained an agent, a version of GT Sophy, for each car-track combination,” Wurman says. “And one of the things we’re looking at is: Can we train a single policy that can run on any car on any of the tracks in the game?” On the commercial side, Sony AI is also working with the developer of Gran Turismo, the Sony Interactive Entertainment subsidiary Polyphony Digital, to potentially incorporate a version of GT Sophy into a future update of the game. To do this, the researchers would need to tweak the AI’s performance so it can be a challenging opponent but not invincible—even for players less skilled than the champions who have tested the AI thus far.
Because Gran Turismo provides a realistic approximation of specific cars and specific tracks—and of the unique physics parameters that govern each—this research might also have applications outside of video games. “I think one of the pieces that’s interesting, which does differentiate this from the Dota game, is to be in a physics-based environment,” says Brooke Chan, a software engineer at the artificial intelligence research company OpenAI and co-author of the OpenAI Five project, which beat humans at Dota 2. “It’s not out in the real world but still is able to emulate characteristics of the real world such that we’re training AI to understand the physical world a little bit more.” (Chan was not involved with the GT Sophy study.)
“Gran Turismo is a very good simulator—it’s gamified in a few ways, but it really does faithfully represent a lot of the differences that you would get with different cars and different tracks,” says J. Christian Gerdes, a Stanford University professor of mechanical engineering, who was not involved in the new study. “This is, in my mind, the closest thing out there to anybody publishing a paper that says AI can go toe-to-toe with humans in a racing environment.”
Not everyone completely agrees, however. “In the real world, you have to deal with things like bicyclists, pedestrians, animals, things that fall off trucks and drop in the road that you have to be able to avoid, bad weather, vehicle breakdowns—things like that,” says Steven Shladover, a research engineer at the California Partners for Advanced Transportation Technology (California PATH) program at the University of California, Berkeley’s Institute of Transportation Studies, who was also not involved in the Nature paper. “None of that stuff shows up in in the gaming world.”
But Gerdes says GT Sophy’s success can still be useful because it upends certain assumptions about the way self-driving cars must be programmed. An automated vehicle can make decisions based on the laws of physics or on its AI training. “If you look at what’s out there in the literature—and, to some extent, what people are putting on the road—the motion planners will tend to be physics-based in optimization, and the perception and prediction parts will be AI,” Gerdes says. With GT Sophy, however, the AI’s motion planning (such as deciding how to approach a corner at the top limit of its performance without causing a crash) was based on the AI side of the formula. “I think the lesson for automated car developers is: there’s a data point here that maybe some of our preconceived notions—that certain parts of this problem are best done in physics—need to be revisited,” he says. “AI might be able to play there as well.”
Gerdes also suggests that GT Sophy’s achievement could have lessons for other fields in which humans and automated systems interact. In Gran Turismo, he points out, the AI must balance the difficult problem of achieving the fastest route around the track with the difficult problem of interacting smoothly with often unpredictable humans. “If we do have an AI system that can make some sophisticated decisions in that environment, that might have applicability—not just for automated driving,” Gerdes says, “but also for interactions like robot-assisted surgery or machines that help around the home. If you have a task where a human and a robot are working together to move something, that’s, in some ways, much trickier than the robot trying to do it itself.”