Exploring Immersive Everyday Communication via AR

Understanding feelings and behavior around AR messaging

A series of three comic strip panels. In the first panel, a person is using their smartphone camera. They see a carousel of augmented reality content at the bottom of their screen, and they select an augmented reality dog from the carousel. They also say "Hey" into their phone, recording their voice. In the second panel, another person, who is wearing smartglasses, sees an AR dog appear in front of them, prompting them to say "Wow." They also hear the first person

Our ARwand technology probe lets friends send each other augmented reality messages, which are pieces of AR content coupled with voice recordings. At left, a smartphone user (sender) sends a barking AR dog to their friend (recipient) who owns smartglasses. They also say "Hey" to their friend. Next, at center, ARwand auto-captures a video of the recipient's "Wow!" reaction while they experience the AR message. The video is sent to the sender once the recipient confirms. At right, the sender views the recipient's reaction video.

A central challenge of social computing research is to enable people to communicate expressively with each other remotely. Augmented reality has great promise for expressive communication since it enables communication beyond texts and photos and towards immersive experiences rendered in recipients' physical environments. Little research, however, has explored AR's potential for everyday interpersonal communication.

In this work, we prototype an AR messaging system, ARwand, to understand people's behaviors and perceptions around communicating with friends via AR messaging. We present our findings under four themes observed from a user study with 24 participants, including the types of immersive messages people choose to send to each other, which factors contribute to a sense of immersiveness, and what concerns arise over this new form of messaging. We discuss important implications of our findings on the design of future immersive communication systems.

This project was performed at Snap logo (Ghostface Chillah) Snap Research.

The Project

Five smartphone screenshots side-by-side. The first three are collectively labeled "AR Message Composer," and the latter two are collectively labeled "Reaction Recorder." The first screenshot shows a list of friends

ARwand's smartphone app for authoring AR messages and viewing friends' reaction videos. Here we see (A) the "Select a Friend" screen, (B) the "Select AR content" screen, and (C) the screen for recording a voice note, optionally scheduling the AR message, and sending the AR message. We also see (D) a notification indicating that a new reaction video arrived and (E) the reaction video itself (played by tapping the notification).

How ARwand works

Our ARwand prototype gives friends the ability to surprise each other by scheduling AR content to appear in their environment at particular places, times, and contexts. It also allows recipients to react to those messages via short videos. The prototype allowed us to observe how people would use this new ability if it were to become mainstream, as well as what their preferences around that ability would be.

ARwand employs smartphones for quickly authoring AR messages and smartglasses for immersively receiving them. The smartphone app allows users to choose a friend, select a piece of AR content to send them, record a voice note to accompany the message, and schedule the message for delivery, including an instant delivery option. The AR message plays through the recipient's smartglasses as soon as the scheduling conditions are triggered. Depending on the AR content—for example, an AR dog or an AR bee—it is displayed either on the ground or floating in the air.

When the message is played, ARwand's smartglasses app will capture the recipient's reaction by automatically recording their point of view and voice while viewing the experience. We designed this reaction video feature to capture the recipient's instant and authentic reaction to the message, from the moment they start experiencing it. Upon the recipient's confirmation, ARwand will share the reaction video with the sender, allowing the sender to witness the moment themselves and hear what the recipient had to say in response to the experience.

Our prototype features smartphones instead of being smartglasses-only because of our findings from earlier prototypes. We found that it is onerous and time-consuming to author AR messages using smartglasses because users need to choose a piece of AR content, choose trigger types, select location triggers using a point and radius on a map, choose a time window for a time trigger, and more. As a result, we designed the smartphone app to make authoring AR messages quick and easy. The smartglasses app makes the process of receiving AR messages seamless and immersive.

Four smartphone screenshots side-by-side. The first screenshot shows an "Add a Location" screen where the user specifies a geofence on a map. The second shows an "Add a Time" screen with a date picker widget at bottom. The third shows an "Add a Marker" screen with several pictures and posters to choose from, including an image of the <i>Mona Lisa<i>. The last screenshot shows a parent "Add Context" screen with buttons for navigating to the three other screens, as well as a toggle at the bottom with two choices: "I

ARwand's scheduling features, allowing senders to set (A) location, (B) time, and/or (C) visual marker triggers. Senders can also control (D) the triggers' specificity. The "Specific" option delivers AR messages when all triggers are met, while the "Flexible" option delivers AR messages when any trigger is met.

Scheduling AR messages

Our ARwand prototype lets senders "schedule" their AR message to be delivered at a specific context based on time, location, or visual markers in the recipient's view. Senders cannot see recipients' location or other personal information, which means they can only schedule AR messages based on their existing knowledge of their friends' routines and environments.

There are three trigger types (types of conditions) for scheduling: location triggers, time triggers, and visual marker triggers. The location trigger allows the sender to specify a geofence radius from 7–14 meters (23–46 feet). The time trigger allows the sender to specify a time range, which will trigger the experience for the recipient if they are wearing their smartglasses at any point during that time range. The visual marker trigger allows senders to specify a visual object that will trigger the AR message if the recipient views it—that is, if their smartglasses's front-facing camera ever detects it. This trigger can be used for a variety of different contexts. A user, for example, could send an “enjoy your coffee” AR message for their friend that gets delivered only when their friend sees a coffee mug or a specific coffee shop logo.

A table summarizing our study findings. Please see the caption below for a link to the accessible version.

Summary of our study findings and design implications for immersive AR communication. Our academic publication below describes them in more detail. Please see this link for an accessible version of the table.

Study findings

Our main takeaway from the project was discovering what behaviors and preferences emerged from our user study participants having the ability to send each other AR messages at will. Our study featured 12 pairs of participants (24 participants total) who each knew each other.

We present our findings under four interrogative themes observed from the study results: (i) What kind of messages do people send with AR messaging? (ii) How do people perceive and use triggers for AR messages? (iii) What makes AR messages and what concerns arise? and (iv) How and why do people react to AR messages?

We found that senders felt that immersive AR messages could allow them to describe and share objects that are otherwise hard to describe or share, and that they evoke greater emotions in recipients than traditional forms of asynchronous messaging. Senders remarked that their messages were well-received by their recipients when they appeared in the recipients' environments at the right moment with the help of the scheduling feature. To ensure successful delivery and compelling experiences, senders want more information about the recipients' routines and current whereabouts. We also observe that privacy concerns may arise, including recipients confirming their location when they send back a reaction video.

Recipients remarked that they enjoyed AR messages immersively featured in their environments, but mentioned that too much immersion could be troublesome in daily use because it could occlude their view. Nonetheless, recipients desired to share their genuine reactions to the messages with the sender and wanted to have more control over sharing their reactions.

Our academic publication below describes our study results in more detail, along with many of our participants' direct quotes.