philip lelyveld The world of entertainment technology

How augmented reality put five Madonnas on stage at once

The avatars (not holograms, so they weren't visible to the naked eye) wove in and out of the inventive performance, bursting into butterflies and puffs of smoke. There were several environmental effects that livened up the show, including digital rain, clouds, greenery and splashes of color, which married with the physical side in an attempt to tell a cohesive story. Madonna has reinvented herself countless times over her storied career, so it's perhaps little surprise that she tried something like this.

The team brought the concept to a new creative AR company called Sequin, which took on the challenge of piecing the performance together. While it was the first time Madonna and Maluma performed the song live, it also marked the first project for Sequin.

While you might not recognize the name, you'll probably be familiar with the work of co-founders Lawrence Jones and Robert DeFranco. At The Future Group, their projects included those dramatic flooding visualizations for The Weather Channel, an AR-enhanced performance by K/DA at last year's League of Legends World Championship Finals and effects for this year's Super Bowl, for which they were nominated for an Emmy.

A critical aspect of making performances such as this work is real-time camera tracking. Jones and his team use a tool called Brainstorm, layering broadcast objects, including motion graphics, character generation and real-time data, on top of Unreal Engine. Jones explained that Brainstorm feeds data from the physical cameras into Unreal Engine so everything from the real set lines up with a digital replication, ensuring the AR renders are in the right place at the right time.

See the full story here: https://www.engadget.com/2019/05/02/madonna-billboard-music-awards-augmented-reality/