Video Games’ Sensory Revolution: How Haptics Reinvented the Controller

Astronaut Selene Vassos crashes her shuttle onto the mossy planet Atropos. The player’s controller rumbles. It’s raining on Atropos, so the controller emits a hiss too. Selene recovers her footing on the wet ground and draws her sidearm. The controller’s right trigger clicks. Selene abandons the wreckage and ventures downward into Atropos, unloading her clip into monster hordes as they pass by, scavenging magical artifacts and better guns; each motion in the game mapped to a distinct sensation in the player’s hands. Selene grapples, dashes, and plummets, her desperate movements rippling in the controller.

The Sony DualSense controller for the PlayStation 5 gives the player “haptic feedback,” a trend in modern hardware designed to stimulate the user’s sense of touch with more specific, stylized vibration. The slight shudder in your smartphone when you toggle between skin tones for emoji—that’s haptic feedback. The wake-up pulse in a smartwatch—that’s haptic feedback. The technology has been present in devices such as the iPhone for at least half a decade, but it’s only just recently that haptic feedback has become a buzz phrase within the next generation of video games. In titles such as Returnal, the PlayStation 5 game described above, developers implement haptic feedback to simulate physical sensations in the virtual world, offering tantalizing levels of immersion. “You can feel the rain,” says Gregory Louden, the narrative director for Returnal. “You feel the power of the weaponry. You can feel the impacts as a carbine hits you in the overgrown ruins.”

The haptic feedback in the DualSense is more sophisticated than the blunt rumble, or “force feedback,” in earlier controllers for 3D consoles dating back to the Rumble Pak attachment for the Nintendo 64. Since the 1970s, beginning with the Magnavox Odyssey, console gamers have given their hands over to controllers. They’ve always been able to feel. They’re only now beginning to touch.

Video games and the consoles they operate on are often more distinguished by graphics specifications, such as image resolution and frame rate, and other practical considerations like platform exclusivity and backward compatibility. The rumble mechanics in modern controllers remain a tertiary concern, even among players with decisive preferences for one grip or another. Yet (outside the niche of PC gaming) controllers are the defining feature of video game consoles; the controller’s interactivity is the primary factor separating ambitious, character-driven narratives from the stories of film and television. In recent years, the dominant console producers—Sony, Nintendo, and Microsoft—have refined the haptics in their respective gamepads. With the DualSense, Sony may well redefine the whole notion of a controller rumbling to life.

The PlayStation 5 remains scarce on shelves and elusive to online shoppers nearly a year into the console’s launch. The global semiconductor shortage, owing in large part to the COVID-19 pandemic, staggered Sony’s rollout for the system as well as Microsoft’s rollout for the Xbox Series X/S; the transition from the eighth console generation to the ninth has been slow going. Finnish video game studio Housemarque developed Returnal as an early exclusive title for the PS5, and while Sony now describes the game as a “mega hit,” the console shortages and the many bugs that plagued the game at launch risked turning Returnal into a footnote rather than a showcase for the future of console gaming.

But the haptic feedback in Returnal is a game changer. It adds a fourth dimension to Atropos, often defined by so much more than sight, sound, and story. “Very early on in the development of Returnal, we always knew we were going to be a PS5 game,” Louden says. He describes the earliest development for the advanced haptics in the game as a radical departure from development for the DualShock—the line of controllers from the first four generations of PlayStation, not to be confused with the PS5’s DualSense. “It just wouldn’t work” on the PlayStation 4, Louden says. “We really used all the additional processing, we created the sound waves to give you the feeling of rain bouncing off the controller.”

Sony began development of the DualSense in 2016, with engineers prioritizing haptics as well as the “adaptive triggers,” which vibrate and resist the player’s input based on the character’s own grip on equipment, such as a steering wheel or a shotgun. Despite the advancements, Toshimasa Aoki, a director on the development team for the PS5, downplays the console and the controller as mere accessories in achieving the player’s total immersion into a virtual world. “You’re not playing a game because you want to experience the controller,” Aoki says, “you’re playing the game because you want to experience the game.” He underscores the design challenge in enlivening a controller, a secondary device, only to cast the player’s fixation back onto the screen. But that’s the power of advanced haptics in video games—making the strange sights and sounds of a space-horror game like Returnal account for human touch.

Aoki seems to be discounting an iconic instrument in home entertainment—the controller with its glossy buttons, shoulder triggers, matte thumbsticks, and cone grips—before making an ominous promise about the future of console gaming. It’s a promise familiar to anyone who misses the old home button on the iPhone: a promise to still recognize a user’s fingertips, which long for the customary clicks and clacks, while otherwise removing the physical buttons in favor of seamless digital magic. The DualSense, the liveliest controller you’ve ever held, is only halfway to implementing Aoki’s outlook on haptic feedback in video games. “Our main goal,” Aoki says, “is to make the controller disappear.”

Consider the emoji representing the whole notion of video games in Unicode: not a console, not a mouse and keyboard, neither a plumber nor a hedgehog, but a gamepad.

The video game controller is the ultimate expression of a player’s will and emotions. This idealized controller belongs to no particular platform but owes its details to a few distinct influences in the console market. Nintendo more or less standardized the button layout with the gamepads for the NES and SNES. Sony added two thumbsticks and native haptics to the DualShock for the original PlayStation in the 1990s. Microsoft crafted two rear triggers onto the controller for the original Xbox and added a home button to the Xbox 360 controller, and each company’s controllers supported wireless connectivity by the mid-2000s.

The console and its controllers form an exquisite duality. The console—the powerhouse of the player’s experience—rests on a flat surface, processing the player’s input, the programming logic, and the sensory output to the screen. The controller is the smaller, dynamic device; two fistfuls of verbs. The player inflicts their own drastic movement on the controller; they squeeze, mash, hold, rotate, while also issuing commands in the conventional usage (press X to sell items). But the player holds the controller at their chest or in their lap, often out of sight and only occasionally top of mind. The player might obsess over controller inputs in certain genres (fighting games, rhythm games), but otherwise, as Aoki explained, the controller melts into the player’s hands as the player melts into the screen and into the game.

At the dawn of 3D gaming, Sony, Nintendo, and Sega began hyping the haptics in the controllers of their respective consoles. Nintendo produced a commercial for the famed Rumble Pak, depicting spies from Sony and Sega interrogating a Nintendo representative about the company’s latest, greatest trade secret, “the big reason why Star Fox 64 is the coolest cinematic gaming experience there is.” The transition from 2D to 3D in the 1990s didn’t just add another visual dimension to games. It marked the larger potential for a player’s immersion—now the industry watchword—defined in large part by the rapid advancements in computer graphics but also bolstered by innovations in sound and touch.

Sega produced its final console, the Dreamcast, at the turn of the century before shuttering its console business and turning exclusively to game publishing. Meanwhile, Microsoft developed the Xbox and more or less inherited Sega’s foreclosed position against Sony and Nintendo in the console wars. Though they’ve standardized the buttons, sticks, and triggers, Nintendo, Sony, and Microsoft keep tinkering with the capabilities and shape. Nintendo has developed a peculiar fascination with nunchaku and motion controls in recent years. For the Wii, Nintendo designed a one-handed motion-sensing remote. This white, plastic magic wand, the Wii Remote, was in fact the star of the commercial rollout for what would become the best-selling non-handheld Nintendo console of all time.

For the Switch, Nintendo developed a detachable, divisible pair of nunchaku, branded as Joy-Cons. In doing so, Nintendo doubled down on the company’s controversial commitment to motion controls but also implemented a refreshingly slight style of haptic feedback, “HD Rumble,” which doesn’t just rumble, but also shakes, buzzes, and flutters. Sony and Microsoft continued honing the classic boomerang shape for the PlayStation and Xbox controllers, distinguished from one another by size, profile, layout, and (to a lesser extent) latency and haptics. Microsoft updated the rumble motors in the triggers on the controller for its latest Xbox Series X/S. Sony replaced the rumble motors from the DualShock 4 with voice-coil actuators that transform audio waveforms into high-precision physical feedback for the PlayStation 5.

The final DualShock controller for the PS4 hinted at the integration of touch and sound in the DualSense. In 2014, a year after the PS4’s launch, U.K. developer Creative Assembly released Alien: Isolation, adapting the sci-fi movie franchise into a survival horror video game set between Ridley Scott’s Alien and James Cameron’s Aliens. In Alien: Isolation, the player controls Amanda Ripley, a deep-space explorer stranded on board the derelict Sevastopol Station and haunted by a stealthy Xenomorph. Ripley carries a sonar-based motion detector to track the Xenomorph in the vents. The PS4’s DualShock includes a tiny onboard speaker, and Alien: Isolation couples the haptic feedback from the in-game motion detector with sound effects propagated to the controller, rather than standard speakers. The controller emulates the motion sensor on several sensory levels, splitting the tension between your eyes and your hands, shuddering and beeping with increasing urgency until the Xenomorph pounces.

The DualShock 4 is a tremendous and terrifying accessory in Alien: Isolation. The haptic feedback in the DualSense further heightens the integration of touch (via the voice coil actuators) and sound (in the onboard speakers), so the controller seems even more cursed in a survival horror setup. Though Returnal is less scary than Alien: Isolation, it’s much more expansive, employing the DualSense’s haptic feedback to showcase a new range of physical sensations in a rough, exotic world.

“The rumble is more about power,” Aoki says about the force feedback in earlier controllers. Today’s advanced haptics in the DualSense are more about precision. It’s no longer the whole controller rumbling on a scale from 1 to 100. The DualSense is more subtle than the DualShock, but there’s also more feedback, more frequently, and it’s far more pronounced than the HD Rumble in the Switch. In fact, I wouldn’t compare the DualSense to the HD Rumble so much as I’d compare Sony’s big bet on advanced haptics to Nintendo’s big bet on motion controls from the Wii through the Switch—a sensory breakthrough risking a sensory overload.

Tim Szeto, chief executive at Titan Haptics and a board director at the Haptics Industry Forum, describes three styles of haptic feedback that have emerged over the years: the blunt vibration in older controllers, the softer buzzing in newer smartphones, and the more experimental implementations, such as “surface haptics,” which simulate physical textures in virtual reality applications. While “rumble” as an industry buzzword may be out, Szeto credits Nintendo for refining the Joy-Con capabilities in the Switch. “They started off with the buzzy notification frequency, and they tried to make it so that it could do more at both slightly higher and lower frequencies to control it a bit better,” he says. The Switch is a two-in-one device—a tablet that can also be docked and run as a home console on a TV—so its small size seems to dictate the slight, but smart haptics that make paragliding and laser-parrying so gratifying in The Legend of Zelda: Breath of the Wild. “That was the first step in this direction,” Szeto says. “It’s not just on and off anymore. There’s a little bit more range.”

But Szeto and many others regard the DualSense as a distinct breakthrough, technically and commercially, for the whole notion of haptic feedback. It’s an incomparable controller with immense and unprecedented sophistication in its physical feedback. At last, Szeto sees an opportunity for touch to rival sight and sound in video games and other consumer electronics. “That’s where the benefit for increasing the fidelity of that sense is actually adding quite a lot of new value,” he says. Take, for instance, an explosion in a first-person shooter video game like Call of Duty. The look and feel of explosions in those games have improved at different rates in the franchise’s 16-year run on consoles. “You can make that super-high fidelity look really good and sound really amazing,” Szeto says, “but if you can now start to feel the explosion in your body or in your hands, then that greatly increases the immersion.” But even quasi-realistically “immersion” must account for more than one explosion in one set piece at a time. So the advanced haptics in a first-person shooter might account for various forces at various distances—the recoil in a variety of guns, the wind and precipitation in a variety of environments, and even the feedback from details as small and varied as the buttons and zippers on the player-character’s gear.

Early in the development for Returnal, Louden says his team second-guessed several haptics implementations, thinking, “It’s a bit too over the top, we should be a bit more subtle.” There’s indeed a small, but vocal contingent of console gamers opposed to the proliferation of haptic feedback in controllers. They’ve disabled vibration on every console they’ve ever owned since the 2000s, and they regard haptics as a neat gimmick at best and at worst a persistent distraction at odds with Aoki’s avowed priority: immersion.

Hrafn Thorisson, chief executive of virtual reality developer Aldin Dynamics, recognizes the potential for sensory overload in advanced haptics. Thorisson strives toward the full-body immersion that has tempted but eluded video game developers for decades. “I often describe VR as an emotional megaphone,” Thorisson says. “For a huge range of people, it’s just too scary, and so they start training themselves to say, ‘I’m not really here,’ and that’s counter to what VR is trying to do.” He sees the challenge for haptics, too—the urge to grab the player by the collar and uncomfortably shake them awake.

Four years ago, beginning with its iPhone X, Apple purged the home button from its iPhones. The home button was a small but nonetheless wasteful indentation, annexing a half-inch of user interface below the touchscreen. Apple widened the iPhone screen and ceded the half-inch to its digital buttons, which click ever so slightly when pressed and held for a second. The iPhone seizes your attention with its scant but startling buzzes. “It’s one of the most ubiquitous platforms for haptics in general,” Szeto says. “There’s no smartphone in the world that doesn’t buzz.” Apple toppled BlackBerry with a smartphone keyboard that clicks against the user’s fingertips, as a keyboard should, despite consisting entirely of pixels.

The mobile market has come a long way from the heavy rattle of a Motorola Bravo. Likewise the console market has come a long way from Star Fox 64 and the Rumble Pak. Now there’s a bit of haptic feedback in every corner of digital life. Still, Thomas Müller, an industrial designer and researcher experienced with haptics applications, says he remains skeptical of a big commercial breakthrough beyond the soft ubiquity of haptic feedback in smartphones. “I was talking to a designer at a very big consumer electronics company, and he’s a huge fan of haptic feedback,” Müller says, “but he was telling me that every time it goes to management or marketing, they decline it because it’s so incredibly hard to sell. You can’t put it on a poster or in a TV commercial. It’s intangible, and that’s a high challenge.” It’s even more challenging once you’ve made the video game controller disappear.

Imagining the future of video games, Aoki and Thorisson both mention the use of glove-like accessories. “I think most would agree that hand input and some type of feedback for your hand is where things are going,” Thorisson says. He, too, hopes to make the controller—or the mouse and keyboard—disappear. “When we’re looking at a screen, sitting on a couch, and handling some controller,” Thorisson says, “there’s a drastic limit on how you can engage the senses of that person.” So even the most advanced haptics in a standard controller format are a technical compromise for the time being. In developing the DualSense, Aoki says, “One of the challenges we faced was that we still need sticks, we need a D-pad, we need the other basic controller functions in order to play games.” Apple uses haptics in the iPhone to solve for button presses on a glass screen. For better or worse, Sony isn’t quite there; the controller remains an indispensable tool for the console gamer.

Szeto draws a distinction between Apple, which designs smartphones and smartwatches for utmost convenience, and Sony, which designs video game consoles and controllers for total immersion. “When we digitalize our experiences,” Szeto says, “sometimes things we come to expect as being normal are lost, and that can really break an experience.”

Video games are the ultimate digitalization of experience, whether that fictional experience be human, anthropomorphic, or extraterrestrial. We hold onto our controllers for dear life while hiking Hyrule, escaping Sevastopol, and scavenging Atropos. It’s easy enough to imagine the old plastic boomerang and nunchaku disappearing into our hands, perhaps replaced onto our hands, despite the controller becoming more vivid and lively than ever before. Then, as now, we’re missing something.