Mastering Music with Motion Capture

At Lune, we’re committed to creating a complete solution for learning and playing an instrument - something that combines technology, creativity, and fun. We believe that learning music should be an immersive, interactive experience, where you feel fully supported every step of the way.

One of the oldest and most effective ways humans learn something, like playing an instrument, is through imitation. We watch how others move their hands - whether it’s an instructor, a teacher, or even a YouTuber - and try to copy their actions. However, when we only have limited camera angles or aren’t physically in the same room, we often have to guess how their hands move across the instrument. This can lead to mistakes about the correct techniques.

At Lune, we recognized this challenge early on and wanted to create something more exciting and accurate. Our founders, with their experience working on Hollywood productions and immersive projects, tapped into a technology called Motion Capture. But what exactly is motion capture?

What is Motion Capture?

(Image Courtesy - https://www.cdprojektred.com/en)

Imagine putting special sensors or markers on someone’s body - like on their hands, arms, or legs. These sensors track every movement, down to the smallest detail. In our case, we use motion capture to track how the hands and joints move when someone plays the guitar. This gives us an accurate picture of how fingers move across the fretboard and pluck the strings.

Motion capture, or "mocap," has powered some of Hollywood's most iconic characters, like Gollum from The Lord of the Rings, the Na'vi in Avatar, and Thanos in Avengers. Actors' movements are captured and transformed into lifelike digital characters, making these creations feel incredibly real.

This technology extends beyond film, with video games like Cyberpunk 2077. Mocap played a crucial role in bringing the character Johnny Silverhand to life, portrayed by none other than Keanu Reeves. Reeves’ facial expressions, gestures, and movements were captured to create a digital version of him within the game.

(Image Courtesy - https://www.qualisys.com/)

But motion capture isn’t just for the big screen. Scientists and researchers also rely on it to study how the human body moves. By tracking joints and muscle movements, they can better understand everything from how athletes perform to how to improve rehabilitation for injuries. Motion capture helps researchers gain insights into the biomechanics of human movement, improving fields like sports science, physical therapy, and robotics.

Cleaning up Motion & use of AI

Once the movement data is captured, we work with an AI powered animation tool (Cascadeur) to clean up any minor errors that can happen due to the limitations of the sensors. After that, we use a technique called retargeting, which allows us to take that movement and apply it to 3D avatars. This means we can represent these captured movements in all sorts of fun and exciting ways. We can create realistic human hands, or even playful, fluffy characters to appeal to different audiences.

But it doesn’t stop there! We wanted to make the experience interactive. With our custom rendering engine on Lune, you can zoom in, rotate, and pan around the 3D avatar’s hands as they play the guitar. It’s like having a front-row seat from every possible angle, something that would be impossible with just video. You can pause, slow it down, and see the performance from multiple viewpoints.

Plus, we’ve added an extra feature—lighting up the guitar strings to show you exactly where your fingers should be placed. This makes learning not just easier, but a lot more fun. At Lune, we’ve taken these powerful techniques and applied them to the world of music learning, using motion capture to help you master their instruments in a fun, interactive way.