ARcall

Real-time AR communication using smartphones and smartglasses.

Two animated images side-by-side. The left image shows a user viewing a flying augmented reality phoenix on their smartphone. A carousel is shown at the bottom of the smartphone screen, implying that the smartphone user chose the phoenix from the carousel. The right image shows a user wearing smartglasses and viewing an augmented reality phoenix flying in their environment. An arrow points from the left image to the right image, indicating that the smartglass wearer saw the AR phoenix as a result of the smartphone user choosing it on their phone.

ARcall explores real-time augmented reality (AR) communication between smartphone and smartglasses users. A smartphone user, left, chooses an AR phoenix to send to their friend. Their friend wearing smartglasses, right, instantly sees an AR phoenix appear in front of them.

As AR-based smartglasses are evolving, they are unlocking exciting new forms of immersive communication including the ability to stay in touch with friends by augmenting their world with AR. There is a lack of understanding, however, about how smartglass-based communication systems should work. In this work, we explore the design of ARcall, a real-time AR communication system that enables an immersive, delightful, and privacy-preserving experience between a smartphone user and a smartglasses wearer.

ARcall allows a smartphone user to send and project AR content to a friend wearing smartglasses natively in AR format. For example, a user can send AR content to make it snow or make a rainbow appear in their friend's environment. We designed ARcall with the practical limits of existing smartglasses in mind, including their short battery life and reduced field of view. We conducted a qualitative evaluation of ARcall's major components and reveal novel insights for building future AR-based communication methods, including the importance of context priming, user control over AR content placement, and the feeling of co-presence while conversing.

This project was performed at Snap logo (Ghostface Chillah) Snap Research.

The Project

A diagram listing each step of ARcall

ARcall's interaction sequence. The smartglass wearer starts the ARcall session, which invites the friend to Drop-In. Later, the friend drops in and surprises the smartglass wearer with an AR rainbow, after which the two start chatting.

How ARcall works

ARcall consists of a smartphone app for transmitting AR content and a smartglasses app for receiving AR content. First, the smartglass wearer indicates their openness to interacting with friends by starting an ARcall session and inviting a friend to join it. The ARcall session can last for up to an hour and expires automatically. It allows the friend to perform three actions from their smartphone app: to "drop in" on the smartglasses wearer and see their live video feed captured from their smartglasses, to transmit AR content to the smartglass wearer (that is, to "ARact" with them), and to initiate a one-minute live voice call with the smartglass wearer.

The first feature, Drop-In, allows the friend to see the smartglass wearer’s current context—where they are and what they are doing—to determine whether that happens to be a good moment for transmitting AR content and to help them decide what type of AR content to send. The friend can select from a wide variety of AR content from a carousel at the bottom of the screen. By tapping a piece of AR content, they immediately transmit it to the smartglass wearer, and their view of what the smartglass wearer is seeing will then include the AR content as well. We call the act of augmenting friends' environments by transmitting AR content ARaction.

The act of transmitting AR content will also immediately establish a live voice connection between both parties, allowing the friend to hear the smartglass wearer's reaction, to add meaning to the AR content that they transmitted, and to give both users the chance to converse and catch up. The voice call is time-bounded to one minute, but either party can tap a button to extend the call in 30-second increments. We call this voice call a Micro-Chat. While talking, the friend can continue to see the smartglass wearer's point of view and transmit AR content.

Two animated images side-by-side. The left image shows a user viewing a flying augmented reality phoenix on their smartphone. A carousel is shown at the bottom of the smartphone screen, implying that the smartphone user chose the phoenix from the carousel. The right image shows a user wearing smartglasses and viewing an augmented reality phoenix flying in their environment. An arrow points from the left image to the right image, indicating that the smartglass wearer saw the AR phoenix as a result of the smartphone user choosing it on their phone.

A smartphone user, left, chooses an AR phoenix to send to their friend. Their friend wearing smartglasses, right, instantly sees an AR phoenix appear in front of them.

Example scenario

As an example of a typical interaction, the smartglasses wearer, who is relaxing in their apartment, might create an ARcall session to invite a close friend to drop in. The Friend can then drop in anytime in the next hour. This allows the friend to learn the smartglass wearer's context—where they are and what are they doing—before sending AR content and interacting with them. They choose to send an AR phoenix, which immediately augments the smartglass wearer's reality. The friend can experience their reaction by hearing their voice, and the two can talk to each other in real time.

A 2x2 arrangement of 3D augmented reality models. Clockwise, from top left: an AR phoenix, an AR dragon, a group of AR holiday ornaments, and AR falling snow.

Sample AR content available to transmit using ARcall.

User study

We wanted to understand four aspects of the ARcall experience: (1) the overall user experience, (2) users' initial perceptions of using ARcall, (3) users' feeling of connectedness while using ARcall, and (4) the users' friction points when using ARcall. We conducted a qualitative evaluation with 14 participants, and since we asked participants to participate in pairs with their friends, we had seven sessions in total.

We learned that both smartglass wearers and their friends using smartphones found ARcall to be fun and immersive and that ARaction added an element of surprise to their interactions. Thier friends felt that they were augmenting the smartglass wearers' reality, and the smartglass wearers felt that the AR content transmitted by their Friends was relevant and personalized for them and their environment. The act of giving their friend the ability to project AR content to their display made smartglass wearers feel more connected to their friend.