“Project Starline: A High-Fidelity Telepresence System”, 2021-11-30 ():
[ANN; blog; video presentation; cf. 2022, VR presence] We present Project Starline a real-time bidirectional communication system that lets two people, separated by distance, experience a face-to-face conversation as if they were co-present.
It is the first telepresence system that is demonstrably better than 2D videoconferencing, as measured using participant ratings (eg. presence, attentiveness, reaction-gauging, engagement), meeting recall, and observed nonverbal behaviors (eg. head nods, eyebrow movements).
This milestone is reached by maximizing audiovisual fidelity and the sense of co-presence in all design elements, including physical layout, lighting, face tracking, multi-view capture, microphone array, multi-stream compression, loudspeaker output, and lenticular display. Our system achieves key 3D audiovisual cues (stereopsis, motion parallax, and spatialized audio) and enables the full range of communication cues (eye contact, hand gestures, and body language), yet does not require special glasses or body-worn microphones/headphones.
The system consists of a head-tracked autostereoscopic display, high-resolution 3D capture and rendering subsystems, and network transmission using compressed color and depth video streams. Other contributions include a novel image-based geometry fusion algorithm, free-space de-reverberation, and talker localization.
[Keywords: videoconferencing, telecopresence, eye contact, parallax, stereopsis, spatialized audio, 3D capture]
…As we’ve started expanding Project Starline’s availability in more Google offices around the United States, we’ve been encouraged by the promising feedback. Google employees have spent thousands of hours using Project Starline to onboard, interview and meet new teammates, pitch ideas to colleagues and engage in one-on-one collaboration. Many users noted how powerful the ability to make eye contact was, and how much more engaged and connected they felt. One user compared their experience to a coffee chat—a genuine interaction that makes you want to lean in and focus on the other person.
We measured the impact of hundreds of Google employees’ experiences with Project Starline, and the results showed that it feels much closer to being in the same room with someone than traditional video calls. We saw an increase in some of the most important signals that are often lost in video calls, such as attentiveness, memory recall and overall sense of presence. Here’s what we found when comparing Project Starline to traditional video calls:
People displayed more non-verbal behaviors such as ~40% more hand gestures, ~25% more head nods and ~50% more eyebrow movements.
People had much better memory recall when using Project Starline, tracking nearly ~30% better when being asked to recall details of their conversation or the content of a meeting.
People focused ~15% more on their meeting partner in an eye tracking experiment, suggesting that visual attentiveness is enhanced when using Project Starline.