What The Future of Augmented Reality Really Looks Like

[I interviewed Creative Control writer and director Ben Dickinson about the film's portrayal of augmented reality, which is the best cinematic rendering of AR I've seen. The feature I wrote about it is below. Also: could someone tell me how to add links and images?]

When Benjamin Dickinson, the 34-year-old writer and director of the indie satirical film Creative Control, was a 19 year old at NYU, flip phone adoption had just reached an inflection point. Dickinson noticed that Manhattan sidewalks were suddenly jammed with people staring at or talking on their phones —and so engaged with their devices that they were oblivious to their environments. In certain circles this was considered a novel and perhaps even exciting development, but Dickinson just found it deeply irritating. It seemed rude and narcissistic.

So 19-year-old Dickinson went home and made a faux flip phone out of a rectangular piece of cardboard, and ventured back out onto the street to pretend to yell into it or stare at it obliviously, like all the other flip phone users in New York. It was an act of performance art and a small protest. “I would just walk down Broadway, screaming into [the cardboard phone] about something really private, he said. “And you know, just me being an asshole — an asshole kid.” He paused. “I did it with a banana a few times.”

He might have been a kid at the time, but Dickinson’s conflicted feelings about technology and the cultural ramifications of its ubiquitousness are on full display in Creative Control, which was released in March and is currently free for Amazon Prime members. The film, which Dickinson wrote and directed, features a Brooklyn-based ad executive named David (also played by Dickinson) who is enamored with his best friend’s girlfriend, Sophie, and attempts to construct a relationship with her via an augmented reality device called “Augmenta.” Creative Control is shot in black and white, a la Antonioni, and takes place in near future New York City.

Creative class New Yorkers will recognize a visual litany of Zeitgeisty tropes, personas, and places: the Wythe Hotel and other locales that could be (and probably were) featured in episodes of Girls, Reggie Watts as a caricatured version of himself, the boutique agency that uses influencers/creators to fill in-house idea vacuums, the oversexed photographer with questionable facial hair, the obligatory Anh Duong cameo, the not-obligatory but appropriate Gavin McInnis and Jacob Lodwick cameos, the perma-girlfriend who’s a grumpy professional yogini… you get the idea. The film has enormous fun with the archetypes, and if you live in that particular world, you’re likely to experience a flash of painful recognition. (Or in our case, a full hour and thirty seven minutes of painful recognition.) But the film captured our interest for another reason: it has the most elegant and plausible portrayal of augmented reality that we’ve seen in film in recent history, and possibly ever.

But the film captured our interest for another reason: it has most elegant and plausible portrayal of augmented reality that we’ve seen in film in recent history, and possibly ever.

Inasmuch as the average person even knows what augmented reality is, the go-to platonic cinematic rendering of it is Steven Spielberg’s 2002 film, Minority Report, which was based on a Philip K. Dick novel and consists of 2054-era Tom Cruise as a cop who heads a DC-based “pre-crime” division that predicts criminal acts before they happen.

To do this, Cruise’s arsenal of tools includes three human-ish “pre-cogs” who are kept in what appears to be the world’s creepiest sensory deprivation tank and attached to various sensors and wires, and software that renders their thoughts intelligible to DC law enforcement. The latter is displayed on multiple digital screens that are essentially holograms and Cruise interacts with them throughout the film. There are some clunky moments: we recall a scene where Cruise inexplicably had to remove what looked like a hard disk from one device and transfer it to another despite the fact that Bluetooth and Wifi would be entering retirement age at that point. But altogether, it nicely conveyed that hardware we took for granted at the time might not be necessary in the future, and while the visuals seem dated, they were better than anything that came before.

Until Creative Control. Dickinson grew up in Wheaton, Illinois, a shades-of-Friday Night Lights conservative town. His parents were less so: they were products of the 60s counterculture and met at an EST seminar. He had very little exposure to technology growing up and spent most of high school focused on getting out of Wheaton. When he was 17 he moved to New York and enrolled in film school.

He started a film production company in Bushwick with some friends in 2004, and by 2008, he had what he described as “my own little spiritual crisis” and like many a New Yorker with a spiritual crisis, got heavily into yoga. “And I found the yoga scene was replete with contradictions and hypocrisy, so you know, I kind of discovered that everywhere I’d look in life, whether it’s fundamentalist Christianity, east coast intellectualism, new age stuff, yoga, the tech world.”
Technology consumerism in particular bred a kneejerk optimism that Dickinson found unworkable and sometimes at odds with obvious negative effects, like the amplification of extremist views on social media. “Optimism is really easy to sell,” he said. “And if we’re overly optimistic, we’re going to find ourselves in trouble because we’re not really looking at what’s happening. Like, Coca-Cola is the ultimate product sold by optimism. It has no value or value to humanity at all, but yet seems to embody the best parts of us, which is very strange.”

And you can see Dickinson’s skepticism toward techno-optimism manifested in the film. The characters approach the new technology warily at first, and even as they find it compelling, even addictive, we see them fighting it. One particularly resonant scene has David answering a barrage of incoming texts and emails on an AR screen and we see him get quickly overwhelmed by the sheer volume and the demands that are being made of him — all of which are overlaid onto the viewer’s screen. You don’t have to imagine this happening in AR to appreciate it as a visual metaphor for everyday technology usage gone awry.

But the Augmenta system itself largely makes AR seem like something that has the potential to be helpful, appealing, and in at least one use case, additive to one’s fantasy life.

Let’s start with the hardware: “Augmenta,” the AR device that David uses, is portrayed as a pair of glasses that comes in a white minimally designed box. It’s what an AR device would look like if Apple acquired Warby Parker in the next two years in an attempt to a create Google Glass device that doesn’t make the user look like an asshole.
This is important because in order to imagine rapid adoption of AR, we have to imagine a scenario where users don’t have to signal to everyone around them that they are early adopters. Google Glass didn’t look enough like normal eyewear to go unnoticed. If you’re wearing it solely in a professional setting, who cares? But if you want mass recreational consumer adoption, you should probably make something that doesn’t force the user to announce to everyone capable of seeing them, “I’M WEARING A PIECE OF WEIRD NEW TECHNOLOGY ON MY FACE RIGHT NOW.”
Augmenta just looks like a normal pair of glasses. This also has the secondary effect of making the technology blend seamlessly into the rest of the film. You’re not pulled out of a scene because the character’s emotions are obscured by the implausible contraption on his face.

You become aware of the technology when you see the holographic overlays. In real life, the overlays and Augmenta’s entire user interface were designed by Paris-based VFX and animation studio, Mathematic. “They do very high end commercial work and they also do pretty much all of Megaforce’s music videos,” said Dickinson. He had worked with them before on commercial projects.

He points specifically to a video Megaforce did for London band Is Tropical, which is about a teenage boy masturbating, but he’s visualizing lifelike 3d avatars while he’s doing it. “It’s disturbing, but I think it’s worth watching. And it has a very sly Gallic sense of humor, which I appreciate.”

(The video, for “Dancing Anymore,” here, is disturbing and worth watching, and is very NSFW, which we feel obligated to mention in case you’re reading in an open plan office and you have an obscenely large desktop monitor. It’s also somewhat topical, since what it depicts is a knowingly cartoony version of what animated VR porn might look like.)

A central plot element (but not a spoiler) is that David is able to have sex with Sophie in AR, and it’s probably safe to say that Mathematic’s prior experience rendering 3d sex was useful, but actual human models were used in Creative Control, so it doesn’t feel artificial. Sophie in Augmenta looks identical to Sophie IRL and subtle movements and gestures that indicate responsiveness or emotion are exactly the same.

And the inputs are subtle. Everything in the Augmenta UX is gesture-controlled. The film preps the viewer for this a bit in the first few scenes in which the employees at David’s agency casually type on holographic keyboards and access holographic screens while not using Augmenta specifically.

The interface for Augmenta itself is a series of hexagons, which Dickinson had already conceptualized before beginning work with Mathematic. The lines are very fine and the overlay is mostly in the periphery if it’s not actively being used. Dickinson said this was intentional. “I thought it was really important that it be very light and that it not interfere too much with your vision,” he said. “It would be really flexible, that you’d be able to sort of program your own movements and that the system would sort of learn how you like to use it and adapt.”

The characters input information by tapping their fingers against their thumbs. “I kind of imagined like a new T9 language that was just using your fingers,” said Dickinson. “I remember how quickly I learned T9 language when we had flip phones and I could pretty much do it without looking at the screen. So I think that adaptivity happens quickly and will happen quickly.”

In addition, the user navigates Augmenta via something that resembles a 3d cube that is actually the hexagon interface rotating. (If you’ve done Google’s Tilt Brush, it reminded us a bit of the 3d navigation there, which Dickinson said he hadn’t yet seen when they conceptualized the Augmenta interface.)

Dickinson also noted that much of the work was done over long Parisian meals, and that the French perspective actually influenced the way the UI was designed. “I was very clear from the beginning that I did not want any of the UI to reek of military stuff,” he said, “which is most of the AR you see in movies. It’s all very military and I really wanted it to feel more like an iPhone — a product that you can sort of kick back with, and project emotions onto.”
Dickinson isn’t the only one who feels this way. Many VR pioneers are working hard to get away from the notion that VR is just elite gamer tech dominated by first-person shooters — and that it can’t be warm and friendly. (Prominent counterexamples: Job Simulator is dry and funny! Oculus’s Henry is about an adorable Pixar-ish hedgehog! Tilt Brush allows you to be creative and paint just as badly in VR as you do in real life!)

But right now a key impediment to adoption is the hardware. “Meta and the Hololens are all sort of these giant helmets where the display is that minimum focus from your eye, which I think precludes it from becoming adopted in a widespread way,” said Dickinson. “So we’ll see [ultimately] whether it’s a set of glasses with a rear facing retinal projection system, whether it’s contacts that are blocking out light and adjusting, or whether it’s going straight into the ocular nerve.”

But Dickinson’s interest in the future of VR lies, not surprisingly in VR filmmaking.” He did a VR comedy with Reggie Watts and the VR studio Wevr called Waves. “We’re gonna maybe do another one,” he said. “I don’t know exactly what it is yet, which is kind of exciting, and I think what’s going to work in VR is something very different from cinema. The sort of constructed performances that work in cinema seem very silly in VR; and very naturalistic, drawn-out performances really play well.”

Dickinson still has some mixed feelings about the technology but has decided that the best way to deal with it is a sort of creative version of immersion therapy. “I just engage with it and adopt an attitude of curiosity toward it, rather than sticking my head in the sand.”

And maybe that conflict is what’s required to conceptualize a version of the technology that ultimately works for people. Dickinson’s Augmenta version of AR is an accessible version of AR, but not a simple one. It is pervasive and deep. But it blends in with the environment seamlessly and you can almost imagine a real life incarnation of it, used by nearly everyone. A packed Manhattan sidewalk full of people making subtle gestures with their fingers, and wearing largely unnoticeable glasses or contacts — but looking up.