When Tech Meets Music

In support of SIE’s goal to make our products and services accessible, we continue to forge our path and pursue technical innovation. Sony Assistive Musical Instrument Hackathon in partnership with Paraorchestra, Drake Music and Watershed, held in September 2024, was the fourth in a series of similar events organized across the globe. It brought together Sony Interactive Entertainment (SIE), Sony Music Entertainment (Japan) Inc., and other Sony Group employees, university students, and members of the Paraorchestra—a professional ensemble comprised of disabled and non-disabled musicians and composers. 

Liza Bec, SIE’s team representative from Paraorchestra, has a very rare form of epilepsy that is triggered by specific movements involved in playing instruments at a professional level. Specifically, performing arpeggios and crossing hands are two performance elements that can be trouble spots for Liza. Thus, they created the Robo-recorder, a custom-built, electronically enhanced musical instrument, and the members of the SIE team were paired with Liza to expand on the device’s capabilities:

• Alexei Smith, SIE (Team Lead)
• Liza Bec, Paraorchestra musician
• Chris Buchanan, SIE
• Mike Middleton, PhD student at the University of York
• Kim Steel, PhD student at the University of York
• Jaylen Sezen, a student at the University of East London
• Luke Child, PhD student at UWE Bristol

Hackathons are timed collaborations between people of various disciplines and experience levels to create something focused on a set objective and/or theme. Networking and creative collaboration are driving forces for such events, and we’ve spoken with two participants, SIE’s Alexei Smith and University of East London student Jaylen Sezen, from the team above to bring multi-layered insight into the experience.

The Objective

Alexei and Chris, the two SIE staff members on the team, come from the London Audio Team, where they work on improving audio for PlayStation. They have system software backgrounds, including accessibility features, and they were inspired by the idea of working in the accessibility space as part of the Hackathon.

Liza Bec is part of Paraorchestra, a group of musicians with and without disabilities who aim to push the boundaries of music-making. Given the importance of music in Liza’s life and identity, understanding and accepting their rare form of epilepsy condition has often been challenging. Dedicated work to do so is what brought about Liza creating the Robo-Recorder.

The team aimed to enhance Liza’s existing setup by further utilizing “the brain” of the Robo-Recorder, a Teensy board. This microcontroller is an effective and popular tool for embedded audio projects, particularly given its affordability,  viability for using C/C++, and impressive processing power for its size. Through Teensy, they would combine it with other software and sensors to elevate Liza’s live performance.

The Four Divisions

With intentions to work efficiently and give smaller groups a focus, the team was divided into four different divisions:

Sound – Frequency Analysis

To conduct frequency analysis on the recorder, the team placed a Teensy equipped with an electret microphone next to the recorder’s “window,” where the sound emanates.

Liza’s Ableton synths play different chords to accompany other parts of the performance. When they press a copper strip, the microphone audio detects the frequency of the note being played, which is then used as the root note for the accompanying chords.

Glissando (or pitch-sliding) is a distinct ornament frequently used by Liza, which can be detected in frequency form. The team prototyped a routine in embedded C that analyzes frequency change in direction and extent. Vibrato could be similarly detected as modulation of frequency and amplitude, with rate and depth processed and output as more metadata that can be sent as MIDI triggers to Ableton Live. 

Because glissando and vibrato are quite specific to an individual player’s style, capturing their essence would give a musician more personalization options when composing and performing. Altogether, these elements augment the physical act of performance, reducing the need for pedal switches and encouraging movement.

Motion – Gyroscope/Accelerometer

By measuring the recorder’s movement left-to-right, specific effects like echo or reverb would be added; their intensity controlled by the speed of the movement. Through measurement of the tilt angle of the recorder, the harmonic complexity of chords would be changed. For example, the higher the recorder is lifted, the more notes will be introduced into the chord.

Motion – Computer Vision

Liza is interested in how physical space can play a part in their performance, and the team built on this idea by using a Python computer vision library to analyze where the players were standing during their performance. They tracked Liza’s left and right motion with a webcam, mapping the performance space to various Ableton parameters.

Visuals – Audio Reactive GLSL Shaders

To top off Liza’s performance with a visual flourish, the team used an app called Synesthesia: a comprehensive library of responsive GLSL shaders fed with audio input and MIDI, as well as visual media such as logos and webcam feeds. The shader code can then be transformed into something new using its built-in code editor, where you can write shaders from scratch. The audio-reactive visuals were then projected behind Liza during the performance.

University of East London student Jaylen Sezen impacted this fourth group. He has produced music for around five years, primarily video game-style compositions, and found himself in an exciting position on the SIE team. His expertise is in sound design and music production, but there wasn’t a specific space for that.

“I was in a situation where I wondered, ‘What can I do? What can I deliver?’ So then I decided to use Synthesia, a pretty cool piece of software that converts audio to visual,” Jaylen says. “It worked out really well with Liza’s midi converters, which could react to sound and hue of color. So, if they would move their arms up and down, the colors would change.”

He found that rewarding, and even though it wasn’t in his particular wheelhouse, he got to network and discuss his craft and specialty. “After the event, everyone was very social, talking about what they do, which was nice,” Jaylen said. “We could talk one-on-one as friends, not just as creators. The environment and the space were perfect for networking.”

Collaboration over Competition

Before the Hackathon commenced, the organizers made the event collaborative instead of competitive. This shift went hand in hand with the social model of disability, a thought process focused on how societal structures and attitudes limit the impact and opportunities for people with disabilities— in other words, emphasizing society rather than the individual for the existence of access barriers. 

Alexei quotes musician and activist John Kelly, who married the social model to disability in music, when describing the benefits of the collaborative focus: “We want autonomy, choice, and control over how we are defined as people and how we belong in society. We’re already here, we exist, and we have the right to be equal to our non-disabled colleagues,” he says, quoting Kelly. “Therefore, if you are going to make a product, a feature, that’s either aimed at those or includes that ‘us’ in the target audience, you need to engage with those stakeholders or consultants as early in the process as possible.”

“The charity that partnered to coordinate the event, Drake Music, helped put the focus in the right place on how we are going to navigate accessibility and how we can involve musicians from Paraorchestra in the right way,” explains Alexei. Having so many affected parties available in one space was instrumental in streamlining the process because of the time limits on the Hackathon event and for receiving and adjusting in light of feedback. “This was a great area for collaboration, having the stakeholder or the end user right there, being able to say ‘okay, let’s try this. What’s wrong with it? What do you like?’ And then improving each time. In two days. The pressure was on.” 

Drake Music is a charity that works at the intersection between music, disability, and technology and has a commitment to equity in the arts at the heart of its purpose.

Conclusion

For our participants, the Sony Inclusive Hackathon in Bristol was a success. Our team gained valuable insights into the intersection of accessibility and technology and, by working together, devised inventive ways to utilize their skills and tech. Events like the Hackathon act as positive disruptions of more traditional innovation pipelines. By combining the valuable experience of the SIE team with the unique insights and abilities of students and our partnered groups, we see how creative solutions are a natural consequence of grassroots collaboration.

For SIE team members, the Hackathon and similar events are an escape from the routine that doubles as a ground bed for innovative thought. For students, these events encourage crucial development outside the academic sphere, where theoretical lessons can be applied in practice.

SIE’s vision towards supporting a more sustainable future is to use our technology and services to drive change, creating and impact beyond gaming. As we look towards the future, SIE remains dedicated to organizing, uplifting, and celebrating events like this Hackathon, which reminds us to challenge our preconceived notions and push for more equitable industries.