How IBM is delivering AI-generated highlights at the US Open

IBM has once again partnered with the United States Tennis Association (USTA) and they’re using new AI-powered tools during the US Open to deliver AI-generated highlights, real-time stats and match analysis, as well as an onsite experience center where attendees can experience AI in action. 

There’s a new technology solution, IBM Coach Advisor, that uses AI and analytics to quantify a player’s physical exertion and endurance and make correlations to match performance, both during regular games and at the US Open. There’s also IBM Watson OpenScale, which figures out the most emotion-packed moments on court for highlight reels. 

This means that fans watching the men’s finals with Danlil Medvedev and Rafael Nadal, or watching highlights from the women’s finals with Serena Williams and Bianca Andreescu will be able to view the most breath-taking moments thanks to AI.

This is different from the traditional way that coaches assess an athlete’s mechanics and endurance. In the past, coaches have relied on player feedback and instinct during a tennis tournament match that could last for several hours and require an athlete to run anywhere from one to six miles back and forth on a court. To better understand and quantify fatigue and energy, IBM has uncovered a new data set that incorporates a player’s physiological load and mechanical intensity. By pairing match video with this data, Coach Advisor has the potential to revolutionize how coaches train, condition, and develop American professional and junior tennis players.

“We’ve been partners with the USTA for nearly 30 years, and at the heart of what we do is really all the digital platforms. So we do usopen.org, we do all the apps and we work with all the data that feeds those apps,” explained John Kent, program manager, IBM sports and entertainment partnerships.

Here is an example of the technologies that IBM is offering at the US Open:

IBM Fan Experience: Located on the ground floor in one of the retail spaces at the Arthur Ashe Stadium, guests can experience Watson in Action, which brings three dimensions of IBM Watson to life, demonstrating how Watson utilizes visual recognition, sound analysis, and natural language understanding to produce AI-generated highlights. In this interactive environment, attendees can create their own AI Highlight, as well as check out the latest real-time scores and stats for all live matches.

AI Highlights: This year, the USTA and IBM will again use AI-powered solutions to produce match highlights. For 2019, AI Highlights has been enhanced with Acoustic Insights to improve sound analysis and has now been taught to recognize when the ball has been struck, allowing a tighter cropping of highlight clips. Additionally, using IBM Watson OpenScale, Watson can now recognize levels of noise and excitement levels of players, allowing it to remove bias when searching for highlights from players with a particularly popular following or those who are particularly animated on court.

SlamTracker: Fans of IBM SlamTracker will again see scores, statistics and analysis of matches come to life in real time, but now with an enhanced experience. By combining the momentum and live tabs into a single experience, fans will have an improved and streamlined way to follow match analysis with tournament highlights, real-time updates and point-by-point commentary.

The USTA is a non-profit, and the US Open is their biggest fundraiser of the year, so these digital platforms are a crucial way of engaging fans and monetizing advertising sponsorship revenues, Kent said. 

“We look to add on a layer of a great fan experience, so things like our SlamTracker application, which is live scores from every match on every court, detailed statistics, a lot to enrich the match that’s being covered to get the in-depth tennis fan, who like insights, or even a casual fan, just to follow the matches. And for us it’s about taking our technology and not using technology for technology’s sake, but how can we apply our AI and our cloud to help the USTA meet their objectives,” Kent said.

“So, when we do things like AI highlights, which is using our Watson technology to watch a match, to understand the player gesture, those moments of excitement, the emotion in their face, the reaction of the crowd and not just the decimal level, but the undulation. And understand the context of the match, is this set point? Is this match point? So what did that particular point mean?” During last year’s US Open, there were 30,000 shots that were analyzed, he said.

After a match is completed, a highlight reel can be developed that shows not just the most exciting moments, but with the context of how the set point was shown, or how the match point was shown, just minutes after the match was completed, Kent said.

“That is really an example of how AI is changing the way the USTA works because their editorial team oversees this. It’s man plus machine, but now they’re able to do things at scale. They certainly have great capability to do editorial video production, but could they do it for every player or nearly every match? Those are the kinds of things we do working together with that premier technology to kind of improve the experience for fans around the world,” Kent said.

IBM continually works with the USTA to better develop their AI offerings.

Elizabeth O’Brien, program director, IBM worldwide sports and entertainment partnership marketing, said IBM and the USTA worked together based on what was seen with AI highlights. “We’ve created tools for coaches, so they can go in and they can actually see their player’s energy load at different points in games and matches, and they can actually click on peaks and valleys to see exactly what’s happening to drive those peaks and valleys.” 

Fans are drawn to the app, with an average of 10 million users accessing the public app annually, from 130 different countries, Kent said.

Also see