Building a 3D synesthetic playground

CHALLENGE

Sound Palette started as our idea to build a synesthetic playground in 3D. We wanted to connect color to sound and allow the user to create their own palette. The result is a mesmerizing application that is fun and addictive to use. This app was built during a one-week Agile sprint with a team of 5 full-stack engineers.

It started with a simple idea - when a user hits a note on the piano keyboard, a background color would be rendered to the screen. When a user hits multiple notes at a time, a gradient would render instead of a solid color. That simple idea was then expounded upon and developed with the help of the team. From a coding perspective, we created an array that holds HEX values for the current color. We used React Piano for the musical keyboard. This allows the user to use their QWERTY keyboard as a little digital piano. We also have a variety of sounds available via a dropdown menu. We used Soundfont Provider for this audio library. Finally, we used React Three Fiber to render the 3D elements that the user can adjust to their heart's content.