Every summer, for two weeks around Labor Day, the U.S. Open is played at the Billie Jean King National Tennis Center in Queens, New York. The five main events—men’s and women’s singles and doubles, plus mixed doubles—each involve seven rounds to winnow from 64 matches down to the highly-anticipated final. Out of 635 total matches in 14 days, an astounding 560 are played during just the first three rounds over the first week. The result is that many early matches, primarily between unseeded and lower-ranked players, go unnoticed and receive little to no television coverage.
While this might seem like a minor issue, a few facts about the 2017 winner in women’s singles suggest otherwise. Entering unseeded and ranked 83rd in the world, American Sloane Stephens became the lowest-ranked player to win the U.S. Open since computer rankings began in 1975. Fortunately, Stephens played most of her early matches on Louis Armstrong Court, one of four show courts to have full camera-crew coverage. Only one of Stephens’ matches was played on Court 10, which has a reduced three-person video team.
But if Stephens had played that match on the smaller-capacity Court 8? There would have been no video coverage at all. Without clips or highlights, a pivotal match in which Stephens upset 11th seed Dominika Cibulkova of Slovakia might have become a glaring blank spot in coverage for several parties. Sportscasters recapping the day’s events, super-fans following live-streams, and later documentarians editing retrospectives of Stephens’ historic run—all would have missed out.
In recent years, ESPN has been using an autonomous camera system to cover matches on smaller courts. This system includes several cameras, mounted on motorized pan/tilt heads, which are guided by a computer using optical imaging software. But video engineers at Fletcher Group, LLC believe the system, as designed, has several key limitations. First, it requires a great deal of complex software and expensive computing power to interpret and follow a player’s movements. And, second, because the system comes as an integrated package, ESPN must broadcast video from the packaged cameras instead of using the exact camera they want.
This second point is important because broadcasters like ESPN typically want a fleet of cameras to maintain a consistent “look” throughout a sportscast. Creating that look requires using cameras with certain visual capabilities, such as high dynamic range, low noise, and precise color control. So, Fletcher Group developed a solution that they believe provides a more flexible alternative for broadcasters–and they demonstrated their system for ESPN at Practice Court 1 during the 2017 U.S. Open.
The goal for U.S. Open broadcaster ESPN is to eventually cover all 635 matches for high-quality broadcast and streaming purposes–and to do so affordably. Fletcher Group’s answer is TR-Ace, an automated camera system that uses a Velodyne LiDAR Puck sensor to 3D model the match. The Puck sensor’s digital data stream requires far less computing power than optical interpolations, meaning a significant reduction in cost. And TR-Ace is designed to autonomously control four of ESPN’s preferred cameras mounted to Fletcher-designed pan and tilt heads.
“Our ultimate goal is to provide a cost-effective system that allows broadcasters to use their preferred cameras on more courts during large tournaments like the US Open,” says Fletcher Group’s President Dan Grainge.
The four cameras are mounted in strategic locations around the court. One is overhead, behind the baseline, shooting lengthwise. One is in the opposite corner, overhead, and shooting a diagonal “slash” angle across the court. Two cameras are at court level, to the right and left of the net. And all four cameras are monitored by a single operator.
This operator adjusts settings, such as tightening or widening various shots, to compose the most effective coverage. As Dwayne Pallanti, Fletcher Group’s Director of Engineering, explains, “Our operator can take manual control of any of the four cameras to get those close, emotional shots.”
Fletcher Group believes LiDAR automation will allow the broadcaster to televise and stream courts, complete with that broadcast quality “look,” that otherwise would go uncovered due to limits in space and budget. “The TR-Ace system provides the necessary angles to give broadcast quality coverage,” says Pallanti. “Broadcast viewers can watch more highlights from first-round matches, while hardcore fans can stream every match from start to finish.”
And someday, should Sloan Stephens’ historic record be eclipsed by another unseeded, low-ranked player—perhaps one hidden on Court 8 against a similarly little-known opponent during an early round? Hopefully, TR-Ace will be there, capturing high-quality footage of every volley, rally, and point.