Tag: research

How Listening to Audio Creates a Full Mind-Body Experience—And Good Vibes

February 28, 2023

Listening to audio activates key centers of your brain, from your emotional response to memory storage to interest and engagement. On Spotify, that’s happening whether you’re listening to a song, a podcast, or an ad—meaning you’re more highly engaged and are more likely to remember what you listened to. So listening makes for a fully mindful experience, and thanks to volume two of Spotify’s Sonic Science report, we now know it also makes for a full body experience full of good vibes, too. 

In 2021, Spotify Advertising released Sonic Science: Understanding your brain on sound. In this first volume, we partnered with the research team at Neuro-Insights to examine the neurological impact of audio and what it means for advertisers. We looked at how Spotify’s deep levels of personalization and interactivity make it a highly immersive, emotionally provoking, memorable medium—more so than TV, digital video, and social media. 

For Sonic Science: Volume 2, out today, we turned our attention to audio’s impact on the body. For the Record asked Marion Boeri, Spotify Associate Director, Thought Leadership, to fill us in.

What is Sonic Science? Why did Spotify want to examine this?

Sonic Science is really all about trying to understand the impact of digital audio. This research franchise is meant to be used by our advertising partners, as well as brands, creative agencies, media agencies, and marketers. The idea is to educate them on the power of audio and really prove that it is a crucial format to leverage when engaging with their consumers. But we also want to help them and guide them to be more contextually relevant. 

We already know that audio is crucial for a lot of people throughout the day. We see from our first-party data that it’s used at any and every moment of the day, and we see how culture is reflected on our platform. We do know people are using audio to help with stress and anxiety—we talk about the good it’s doing for mental health. So looking at our brains and “getting dirty with science” and unpacking that impact helps us understand what that means for an advertiser when they want to engage and leverage an audio platform.

At the end of the day, we want to make sure we’re creating a positive environment, and that translates into a positive impact for advertisers. Ultimately, we’re using the insights and the research to create a set of best practices that really helps us guide our advertisers and be even better on the platform, and with a stronger strategy. We’re trying to create an advertising experience that is additive and not disruptive.

Where did we focus during the second round of research?

In volume one, we looked at the brain. In volume two, we looked at the entire body. You know how you’ll be listening to something scary and get goosebumps? Or listening to something with a high BPM and see your heart rate go up, even if you’re not jumping around? We wanted to prove this mind-body connection. 

So we worked with a biometrics research company, MindProber, as well as Josh McDermott, PhD, who leads the Laboratory for Computational Audition at MIT and who served as an advisor to our entire process. Together with MindProber, we engaged over 400 Spotify Free listeners in the U.K. and U.S., and we asked them to listen to Spotify at least once a day for an hour, doing whatever they would normally do while listening—work, commute, play with kids, cook dinner, work out, clean—but while wearing a sensor in the palm of their hand. With that sensor, we were capturing heart rate and body temperature: electrodermal activity. We captured the emotional arousal, how people’s bodies were reacting to music and podcasts, in their natural context. 

It was crucial to us to really be able to have listeners be in their natural environment. So in effect, we were looking at what was happening every day, in any listener’s life, and just trying to understand how that translated into engagement. So that was a great complement to the brain story by trying to validate everything and looking at the whole body. And we captured so much data!

What were some of the findings and results? 

It validated a lot of things that we already know as Spotify listeners. We all have emotional needs: Sometimes you’ll turn to a piece of music because what you want is motivation. Sometimes you need to chill, and you’ll turn to something very different. So with this study, we were able to map that out and lay out the acoustic attributes related to what listeners were doing. Some things were common sense, like high-density, high-tempo audio for working out, or instrumental music for gaming. But we also noticed, for example, that when people are walking alone, they’re more likely to engage with songs with a high rate of speech—a lot of words. At the end of the day, we have different needs, and we can try to fulfill these with audio. It’s more than a soundtrack—it’s a mirror of our lives. And that was something we were able to validate with the data. 

Plus, we learned that the Spotify listening experience creates a halo of good vibes. Even when they’re streaming sad music, listeners get a mood boost when they tune into Spotify. A third of our Sonic Science study participants reported feeling “happy” or “cheerful” after listening to Spotify, while a quarter described feeling “calm”—regardless of when they listened, what they listened to, and what they were doing at the time.

What’s the takeaway for those who want to be more in tune with their consumption habits?

There are a few. One is that you can kind of trick your body into having an emotional reaction. Some people love listening to very sad music when they’re upset. Some people use music to get out of that sadness. I think being more mindful of the impact audio can have—not only on your mental state but also on your entire body—is interesting because you can start to get to know how to help or motivate yourself. It’s not only beneficial for entertainment. 

Second, is that any platform needs to understand that connection they’re creating with their users. Knowing this data helps us create an experience that is even more relevant, and I think that was a great validation for what we’re doing. We’ve managed to create a personal bond and emotional connection that helps us create more products and more playlists to feature on the platform, since we do know that having personalization and interactivity is key to our users. 

Finally, we all consume a lot of media all the time, and it’s hard to know what’s good for us, or how to make decisions about the time we spend on a platform. Some media has been proven by research to have a toxic, negative effect on our mental health and well-being. So it’s great to see that there’s a positive effect of using Spotify. 

We say “listening is everything.” In your opinion, what is the power of audio?

The power of audio to me is the benefit it has on your mental well-being. I also think that besides that, audio holds the power of connecting to others on a global level. It’s universal. You listen to music from different languages. You hear different stories and perspectives through podcasts. You’re learning from others; you’re opening your mind to so many other things. It helps to shape culture, it helps us to bond to each other, and from a human standpoint, it is everything. 

Read the entire Sonic Science report on Ads.Spot.

Dr. Stacy Smith of USC Annenberg Calls on All of Us To Address the Gender Gap in Music

February 1, 2023

Dr. Stacy Smith

Each year, the team at the USC Annenberg Inclusion Initiative (AII), led by Dr. Stacy Smith, takes a look at the numbers of women in music—both behind the scenes and on the charts. The result is an annual study we are proud to underwrite. Together, we recognize there is so much more to be done when it comes to the inclusion of women and nonbinary creators within the music industry. 

Amplifying underrepresented voices is at the core of our work at Spotify. Over the past few years, we’ve launched several initiatives like Frequency, NextGen, SoundUp, and GLOW, each of which promotes a diverse roster of artists, songwriters, and podcasters on our platform. Our global EQUAL music program, which is dedicated to promoting and elevating women artists around the world, has enabled us to support over 700 women in 35 countries since March 2021.

Our work is informed by our partners at the USC AII, and particularly, Dr. Stacy Smith. As the founder of the USC AII—the leading global think tank studying issues of inequality in entertainment—Dr. Smith is the foremost disrupter of inequality in the entertainment industry. She’s also a founding member of our Safety Advisory Council.

The report outlines why women need to help and be supportive of other women through mentorship programs, amplification opportunities, and other confidence-building activities. This is the fourth consecutive year Spotify has funded the study, and we’re committed to continuing to learn and understand, and to work toward a more equitable industry. But don’t just take it from us—read on for Dr. Smith’s observations and recommendations.

How would you define representation?

In light of the research we do, representation focuses on prevalence as well as the nature of how groups are presented in the media. For music, specifically, we are examining who receives access and opportunity to specific key positions.

Your research examines inclusion of gender, race/ethnicity, the LGBTQIA+ community, people with disabilities, and mental health in storytelling across film, TV, and digital platforms. What do you see across the board when these groups are not represented, or are underrepresented? 

We see storytelling that fails to depict the reality of the world where we all live. We are missing critical stories and points of view from dynamic and vibrant communities. A lot of our work has shown negative tropes and stereotypes still occur far too frequently when it comes to gender, race/ethnicity, the LGBTIQ+ community, people with disabilities, and mental health.

The Annenberg Inclusion Study, which Spotify partners on, relates to women in the music industry. What are the encouraging trends you’re seeing? What more needs to be done? 

There is only one encouraging trend: The percentage of women artists increased in 2022 in comparison to 2021. That said, it is still abysmally low.  

People need to hire women songwriters, producers, and engineers. That’s it. Until that happens, the numbers will not change. Ultimately, what is needed to create change is for labels to sign, promote, market, and hire women and gender nonconforming people from all backgrounds as artists, songwriters, and producers.

Is there anything notable in the latest gender in music report that you’d like to call out?

The Recording Academy’s efforts on women in the mix have made absolutely no difference in the lives of women producers or engineers. The solution isn’t gimmicks or publicity grabs. It is people understanding that women songwriters and producers have talent but they are not given the same access and opportunity as their male peers.

What would you like to see Spotify doing more of? Less of?

Spotify, along with all the industry, can showcase the work of talented women songwriters and producers to facilitate opportunities. Making sure that listeners can experience songs written and produced by women—and performed by women, too.

Listen to women at full volume on our global EQUAL playlist.

Rachel Bittner on Basic Pitch: An Open Source Tool for Musicians

September 1, 2022

orange open source and coding symbols on a blue, green, and white background

Music creation has never been as accessible as it is now. Gone are the days of classical composers, sheet music, and prohibitively expensive studio time when only trained, bankrolled musicians had the opportunity to transcribe notes onto a page. As technology has changed, so too has the art of music creation—and today it is easier than ever for experts and novices alike to compose, produce, and distribute music. 

Now, musicians use a computer-based digital standard called MIDI (pronounced “MID-ee”). MIDI acts like sheet music for computers, describing which notes are played and when—in a format that’s easy to edit. But creating music from scratch, even using MIDI, can still be very tedious. If you play piano and have a MIDI keyboard, you can create MIDI by playing. But if you don’t, you must create it manually: note by note, click by click. 

To help solve this problem, Spotify’s machine learning experts trained a neural network to predict MIDI note events when given audio input. The network is packaged in a tool called Basic Pitch, which we just released as an open source project. 

“Basic Pitch makes it easier for musicians to create MIDI from acoustic instruments—for example, by singing their ideas,” says Rachel Bittner, a research manager at Spotify who is focused on applied machine learning on audio. “It can also give musicians a quick ‘starting point’ transcription instead of having to write down everything manually, saving them time and resources. Basically, it allows musicians to compose on the instrument they want to compose on. They can jam on their ukulele, record it on their phone, then use Basic Pitch to turn that recording into MIDI. So we’ve made MIDI, this standard that’s been around for decades, more accessible to more creators. We hope this saves them time and effort while also allowing them to be more expressive and spontaneous.”

For the Record asked Rachel to tell us more about the thinking and development that go into Basic Pitch and other machine learning efforts, and how the team decided to open up the tool for anyone to access and to innovate on.

Help us understand the basics. How are machine learning models being applied to audio?

Rachel Bittner

On the audio ML (machine learning) teams at Spotify, we build neural networks—like the ones that are used to recognize images or understand language—but ours are designed specifically for audio. Similar to how you ask your voice assistant to identify the words you’re saying and also make sense of the meaning behind those words, we’re using neural networks to understand and process audio in music and podcasts. This work combines our ML research and practices with domain knowledge about audio—understanding the fundamentals of how music works, like pitch, tone, tempo, the frequencies of different instruments, and more.

What are some examples of machine learning projects you’re working on that align with our mission to give “a million creators the opportunity to live off their art”?

Spotify enables creators to reach listeners and listeners to discover new creators. A lot of our work helps with this in indirect ways—for example, identifying tracks that might go well together on a playlist because they share similar sonic qualities like instrumentation or recording style. Maybe one track is already a listener’s favorite and the other one is something new they might like.

We also build tools that help creative artists actually create. Some of our tech is in Soundtrap, Spotify’s digital audio workstation (DAW), which is used to produce music and podcasts. It’s like having a complete studio online. And then there’s Basic Pitch, which is a stand-alone tool for converting audio into MIDI that we just released as an open source project. We open sourced Basic Pitch and built an online demo, so anyone can use it to translate musical notes in a recording (including voice, guitar, or piano).

Unlike similar ML models, Basic Pitch is not only versatile and accurate at doing this, but it’s also fast and computationally lightweight. So the musician doesn’t have to sit around forever waiting for their recording to process. And on the technological and environmental side, it uses way less energy—we’re talking orders of magnitude less—compared to other ML models. We named the project Basic Pitch because it can also detect pitch bends in the notes, which is a particularly tricky problem for this kind of model. But also because the model itself is so lightweight and fast.

What else makes Basic Pitch a unique machine learning project for Spotify?

I mentioned before how computationally lightweight it is—that’s a good thing. In my opinion, the ML industry tends to overlook the environmental and energy impact of their models. Usually with ML models like this—whether it’s for processing images, audio, or text—you throw as much processing power as you can at the problem as the default method for reaching some level of accuracy. But from the beginning, we had a different approach in mind: We wanted to see if we could build a model that was both accurate and efficient, and if you have that mindset from the start, it changes the technical decisions you make in how you build the model. Not only is our model as accurate as (or even more accurate than) similar models, but since it’s lightweight, it’s also faster, which is better for the user, too. 

What’s the benefit of open sourcing this tool?

It gives more people access to it since anyone with a web browser can use the online demo. Plus, we believe the external contributions from the open source community help it evolve as software to create a better, more useful product for everyone. For example, while we believe Basic Pitch solves an important problem, the quality of the MIDI that our system (and others’) produces is still far from human-level accuracy. By making it available to creators and developers, we can use our individual knowledge and experience with the product to continue to improve that quality. 

What’s next for Basic Pitch in this area?

There’s so much potential for what we can do with this technology in the future. For example, Basic Pitch could eventually be integrated into a real-time system, allowing a live performance to be automatically accompanied by other MIDI instruments that “react” to what the performer is doing.

Additionally, we shared an early version of Basic Pitch with Bad Snacks, an artist-producer who has a YouTube channel where she shares production tips with other musicians. She’s been playing around with Basic Pitch, and we’ve already made improvements to it based on her feedback, fixing how the online demo handles MIDI tempo, and other things to make it work better for a musician’s workflow. We partnered with her to use Basic Pitch to create an original composition, which she released as a single on Spotify. She even posted a behind-the-scenes video on her channel showing how she used Basic Pitch to create the track. The violin solo section is particularly cool.

But it’s not just artists and creators that we’re excited about. We’re equally looking forward to seeing what everyone in the open-source developer community has been doing with it. We expect to discover many areas for improvement, along with new possibilities for how it could be used. We’re proud of the research that went into Basic Pitch and we’re happy to show it off. We’ll be even happier if musicians start using it as part of their creative workflows. Share your compositions with us!

Create a cool track using Basic Pitch? Share it on Twitter with the hashtag #basicpitch and tag the team @SpotifyEng.