Identifying cover songs, live performances, AI clones and more
Covers and other versions
Imagine you’re at a nightclub. The DJ plays a song. You love it, but you don’t know what it is. You pull out your phone, use it to “listen” to what you’re hearing, and in a few seconds, you’re given the track title and artist of your new favorite song. We’ve all done this. No big deal, right? This form of automatic content recognition (ACR) — making a 1:1 match between recordings — is well established across consumer and business applications. These applications compare a given sound recording to a set of reference recordings and find an exact match if one exists.
But what if the song was instead being performed live in that nightclub? What if it was an impromptu cover by a different artist in a totally different style? How will you know what it is if there is no exact reference for the ACR application to compare to?
That problem — version or cover identification — is much harder than 1:1 matching between recordings. With 1:1 matching, you are comparing the exact same recording characteristics: the same instruments, the same chords, the same tune, all in the same order. But recognizing a cover or a live performance requires musical or lyrical knowledge to detect the underlying composition no matter how it is realized: maybe in a different key, at a different tempo, with different instruments and singers, or no singers at all. Maybe it has different lyrics or chords or rhythms, with the sections changed around. Maybe there is new musical material interspersed or mixed in.
The variety and sources of covers are nearly endless. Classic songs like The Beatles’s Yesterday have been covered many thousands of times over the course of decades by artists of all genres: Hip-Hop, Death Metal, Country, Opera, and more. New songs by popular artists are often covered the day they drop as a way for lesser-known artists to capitalize on the song’s rising popularity. Some covers are major commercial successes themselves, like Run DMC's cover of Walk This Way or Dolly Parton’s recent cover of Stairway to Heaven. When it comes to social media and user-generated content (UGC), the pool expands even more. We have Puddles the Clown performing a novelty cover of the music to Stairway to Heaven with the lyrics from Gilligan's Island, all kinds of instrumental covers of the theme to Game of Thrones, kids covering rap songs at high speed, or Doja Cat including a bit of Milkshake inside Mooo!. Today, anyone with a phone can make a video of themselves singing or playing anything and distribute it to the world.
If you run a digital media business, how will you identify when versions of copyrighted songs are uploaded to your platform?
Audible Magic’s technology
This new type of ACR can be accomplished by Audible Magic’s Version ID, and we have recently been granted two fundamental patents for our technology: one that uses melody, harmony, sound and structure (US 11,294,954 B2 Music cover identification for search, compliance, and licensing) and one that uses lyrics and phonemes (US 11,816,151 B2 Music cover identification with lyrics for search, compliance, and licensing).
With Version ID, there is no simple audio feature like our MFCCs or Shazam’s spectral peaks that are used for 1:1 matching of sound recordings. Instead, we need to look at deep aspects of the sound that are invariant to differences in master recordings, but that are also specific enough to say yes, this is a version of that tune, and to be able to say that with high accuracy. This is why we have two patents, one that considers all aspects of the musical sound, and one that looks at the lyrics and underlying phonemes. Both are implemented in Audible Magic’s Version ID using a variety of AI and expert-system techniques. Combining them into a single, powerful ACR product allows us to identify a wide range of covers and versions. Let’s listen to a few examples of what Version ID can recognize.
An AI clone of Radiohead's Creep
Recently we’ve seen a flood of AI cloned-voice covers out in the wild, where a creator uses a model of a well-known artist’s voice to create a version of another artist’s song. Sometimes these appear with very different music accompaniment, usually in the cloned artist’s style. This version of Creep is a good example of that, where Radiohead’s guitar-oriented-rock original has been transmuted to a somewhat lounge-y version for the Frank Sinatra clone to croon over. Even though the music seems quite different on first hearing, the chords, melody, structure, and lyrics are the same, and our Version ID picks up all to successfully identify this as a version of Radiohead’s Creep.
A parody of Like a Virgin
Like most parody versions, the words in this example are completely altered: Hey, like a surgeon / Cuttin' for the very first time. But Weird Al wants the music to be recognized, so besides his voice singing in a slightly different key, the chords, melody, instrumentals and structure are the same. This allows our music-based Version ID to match the track as a version of Madonna’s Like a Virgin.
A live version of the same
Sometimes live versions, even by the original artist, are very different from the original. An artist might do this to keep an audience surprised, to keep up with changes in tastes and styles, or just to have fun. Sometimes the lyrics will be changed a little to make comments about things like current affairs or politics, but enough of the original will usually be there for us to identify it. In this case, the accompaniment of the Madonna performance has definitely been updated as compared to the original recording, and there is an entire stadium of fans singing along to the chorus. Yet the chords, melody, structure, and lyrics can all be used by Version ID to connect this back to the original recording of Madonna’s Like a Virgin.
An acoustic cover of Tupac’s Keep Ya Head Up
Tupac’s anthem to black womanhood and the death of Latasha Harlins is covered here by Chicagoan Melody Angel. The musical accompaniment is completely different, with solo guitar chording under a bluesy melody compared to Tupac’s almost entirely spoken rendition over a strong beat. There is nothing here that can be detected by a version ID system that uses musical features – no common beat, chords, or melody. But the lyrics are the same, and that is how we match it.
We talked about interpolations, like how Doja Cat uses a bit of Milkshake in her track Mooo! We see that here too, where we can easily pick up in the lyrics – and a bit in the melody – how the song’s chorus is based on The Five Stairsteps’ O-o-h Child. Version ID will report timing information for any matches it find, so in this case it will be clear that the interpolation is only the chorus.
A piano cover of Beyoncé’s Halo
Many cover versions have no lyrics at all, either because the original composition was an instrumental to begin with (think TV or film theme music), or because the cover artist wanted to highlight their arrangement or their instrument. Piano covers are very common, as in this multi-platinum-viewed version of Beyoncé’s Halo for piano solo by J. M. Quintana Cámara. Here our Version ID relies on the fact that the chords and melody and structure are the same in the two, even though the instrumental and musical character couldn’t be more different.
For more than 20 years, Audible Magic has innovated B2B solutions in content identification (both music and TV/Film), music catalog fulfillment, and rights administration. Audible Magic powers billions of monthly transactions to serve customers all around the world. Its music identification services are used to help reduce the number of DMCA takedown notices and administer licenses for user generated content (UGC). Version ID is the latest piece of this suite of capabilities, based on patented technologies that have been developed over the last ten years. We would love to hear about your use case and to work together to help solve your problems. Contact us to learn more.
- Erling Wold, Chief Scientist and Technologist
