Music Synthesizer Technologies made using AI Methods

grahamenglish · 2023-02-19T00:55:47.066Z

I’d be all over it. One use case I’d love to see is reverse-engineering synth sounds by giving it a sound sample, and it programs a sound-alike synth patch, complete with all the basic parameters you’d find on a hardware synth.

Jukka · 2023-02-19T01:04:18.077Z

There’s something like this. I am trying to recall the exact detail. I’ll post if i recall.

jdaddyaz · 2023-02-19T01:18:00.507Z

I’d be at a loss of words. Literally. It’d be like trying to describe a techno song to a friend. “You know that one song that goes boom tis blap bappa doo slap!”

No. I have no idea what the Hell you’re talking about as you just described all the songs.

Jukka · 2023-02-19T01:46:20.139Z

I found it – it was a research project, and they created settings for u-he Diva from a sample. I posted about it here :

Audio data as video data representation - #38 by Jukka

The method is neural net based and they train the system to recognize spectral images.

There are versions, that basically do something very similar.

This one comes with source code, but you need to train the network, i think.

https://jakespracher.medium.com/generating-musical-synthesizer-patches-with-machine-learning-c52f66dfe751

And here’s a research paper that may well be an original source for these other two. This is in pdf, and is very technical.

arxiv.org

1812.06349.pdf

3.18 MB

Seems to me a hardware maker could do something similar, where you upload a short sample, and they return a patch for your synth, or an image to work from, if there is no patch storage on the synth.

grahamenglish · 2023-02-19T01:47:35.779Z

Thank you! Yes, I think synth makers need to get on this!

Jukka · 2023-02-19T02:37:55.269Z

jdaddyaz:

“You know that one song that goes boom tis blap bappa doo slap!”

An AI system could find that song for you. The WoFI system i linked to in the first post, has a mic and is hooked to their internet software, so maybe it could find the song.

What if you could enter rhythmic parts using your voice or from a short sample. And if it had something like Melodyne or Vochlea Dubler on the net you might work from a sample or your voice.

I think that might be the reason there is such a small keyboard on that WoFI. A product like that might not be made for playing, only for note entry and simple melodies. The market is for “newbies and semi-professionals”, so if they were to add the AI smarts on-line, it would allow that market to be more musical and productive, without being trained musicians.

pselodux · 2023-02-19T02:50:44.687Z

Jukka:

Would you be interested in having a synth with customizable AI generated machines? What would you give as a prompt to this system for your own custom synth ? Do you have other ideas along this same approach ?

I was just talking about this the other day with some people at a Max/MSP meetup. I wondered what it could do for accessibility if you could have a synth that doesn’t have “traditional” synthesis parameters, or even use traditional forms of synthesis, but instead you describe the sound, or set up more descriptive parameters. The issue we discussed comes from everyone’s different interpretation of those descriptions, ie. what sounds more “buzzy” or “crunchy” to someone might not have that description for someone else, so it’d have to be trained on your own interpretations.

Jukka · 2023-02-19T03:05:49.252Z

That sort of description is used so frequently on Elektronauts, it must mean something to someone, but for me most often i haven’t a clue.

But using a specific reference to a song, album, band, or musical genre would convey a lot.

An AI system should be able to give you back, copyright free tools that would give you a place of beginning from that sort of description.

jdaddyaz · 2023-02-19T03:15:23.944Z

But in all seriousness…cool.

pumpkin_head · 2023-02-19T08:35:28.826Z

There’s the Google Nsynth super that came out a few years ago, but only DIY and I heard the sound quality was low. As far as I remember it can morph between sounds samples with AI , like , 50% thunder noise 50% trumpet sound. Exciting on paper, I hope we will see more of these in the future.

nsynthsuper.withgoogle.com

NSynth Super

An experimental physical interface for the NSynth machine learning algorithm.

Jukka · 2023-02-19T10:23:02.591Z

I still make music manually.

It’s still manual ( or can be ), just with a different set of tools. Tools made to enhance human creativity.

Jukka · 2023-02-19T14:11:52.327Z

For a lot of useful information in this field look into the work done with the IRCAM-ACIDS ( Artificial Creative Intelligence and Data Science ) projects. This is research done by a group of people in France, mostly open source – all really excellent stuff.

For a place to look, start clicking off this Google search.

I have a hunch, that some of the internet back-end with the WoFI project may be based on some of the research done for the IRCAM-ACIDS project. In particular i am thinking the Flow Synthesizer may be involved. This is only a hunch though, perhaps i am only wishing that it is so.

pselodux:

I wondered what it could do for accessibility if you could have a synth that doesn’t have “traditional” synthesis parameters, or even use traditional forms of synthesis, but instead you describe the sound, or set up more descriptive parameters.

I think you have put your finger on a central question.

With the IRCAM-ACIDS project Neurorack they generate sounds from seven separate descriptors that can be actively adjusted.

Loudness
Percussivity
Noisiness
Tone-like
Richness
Brightness
Pitch

The Neurorack hardware is a small Eurorack module based on the Nvidia Jetson Nano processor, a 128 core GPU along with 4 CPUs. I believe the software is then set up with some basic sounds connected in to those seven descriptors, and then the musician is allowed to vary those seven descriptors in some manner over time to produce sound and music.

=-=-=-=-=-=-=-=

A summary of some other ACIDS stuff :

DDSP - Differentiable Digital Signal Processing, a PyTorch ( a Python machine learning framework ) module with a PureData wrapper, with a couple of pretrained instrument models, for real-time sound creation.
RAVE VST - Realtime Audio Variational autoEncoder, a fast and high-quality audio waveform synthesis system using deep learning models.
VSCHAOS - A Python library for variational neural audio synthesis, using PyTorch.
Flow Synthesizer - A voice controlled synthesizer, using variational auto-encoders and normalized flows on a set of learned sounds. ( Simplified description. )
Generative Timbre Spaces - A descriptor based synthesis method that maintains timbre structure while moving across timbre space. Uses variational auto-encoders.
Orchestral Piano - A method that allows the real-time projection of a keyboard played piece, to a full orchestral sound, using analysis of historic examples of music moved by composers, from piano to orchestra. I think this is still under improvement.
Orchids - A set of algorithms and features to reconstruct any evolving target sound with a combination of acoustic instruments.

In addition to IRCAM, it appears that Sony CSL ( Computer Science Laboratories Inc. ) may also be assisting with some of this research.

captain8 · 2023-02-19T15:36:00.423Z

I want to see them come out with an AI that can emulate the Ohio Players, if they can do that I’ll eat my hat!

pumpkin_head · 2023-02-19T17:42:50.013Z

It’s going to be hard to resist this kind of eurorack !

Jukka · 2023-02-19T18:00:46.304Z

captain8:

Ohio Players

You’re talking about the Tonex Amp Modeler. That’s a very competitive price, if it competes well with the Kemper Profiler.

g3o2 · 2023-02-19T19:00:24.324Z

Resynthesis has been around for a while, too. Maybe with ML, it might be easier to apply it to more complex synthesis engines. Of course, there is also a cost trade-off here: why not simply sample the sound you want or combine sampling with synthesis? So how many companies would be willing to invest in this area?

Didn’t the devs of the Hydrasynth use machine learning (ML) internally to emulate the filters of various synths? I remember having heard something like that in the early YT videos. So this already seems to be the reality and would not particularly need any “AI” but “only” some training data and a decent model that after training translates into relevant DSP settings.

Generating music seems easier with Deep Learning today than generating sound though: the training data is already there thanks to music notation and more importantly MIDI data. All you have to provide as input is music in MIDI format and then define the problem as follows: “predict the next note based on the previous note or sequence of notes”, similar to predicting text. That way you avoid expensive manual training of labelled data. This same approach is already used by NLP models these days, including ChatGPT and BERT.

Jukka · 2023-02-19T19:37:37.100Z

g3o2:

Didn’t the devs of the Hydrasynth use machine learning (ML) internally to emulate the filters of various synths? I remember having heard something like that in the early YT videos.

They did, and it’s a clever use of technology, to solve their development problems with the emulation of so many different multivariate filters. I’ve always wondered, how this was incorporated into the final design, as i doubt their custom hardware, actively uses this sort of method live. The engineer who did that is very capable and advanced, and was responsible for many other parts of the HS’s advanced design as well.

Good post g3o2, i’m still processing on the rest of what you wrote, and may respond on it later.

g3o2 · 2023-02-19T23:32:02.359Z

Here is the interview where it is mentioned: https://youtu.be/VwzNWvQF2ks?t=155

The filters were indeed not implemented using DSP building components or blocks but Chen says using “machine learning”. That can mean a lot unfortunately.

According to ASM: 144 recordings of waveforms (sawtooth, noise, …) through 11 different filter cut-off and resonance settings per filter (handpicked by the team) are used as reference material.

The idea was then to reproduce each of the 144 recordings with the least error possible. Instead of building their own filter signal chain by hand and calibrating that by ear, Chen let the computer automatically find the DSP design component chain and parameters whose output would get the closest to each of the recordings.

Jukka · 2023-02-21T16:37:51.329Z

As this thread has developed it’s become clearer the terrain involved. As a result i have expanded and generalized the thread title.

The former title was : AI Generated HW Synth Engines.

Jukka · 2023-03-04T16:07:59.667Z

You can train an AI in your voice, or whatever tonal sound, and then perform with that machine learned voice.

Holly Herndon did a short TED talk introducing this technology, and demonstrates her spawned ( not sampled ) voice in this video.

She calls that AI voice Holly+. Part of this process is called timbre transfer which can transfer a live performance in a performers voice, into a synthetic, though very real sounding AI machine learned voice.

You teach an AI a sound in order to generate an entirely new sound.

Yes there have been similar things done in the past, like in particular the Yamaha Vocaloid technology. But be aware of the improvements in sound quality, capabilities and the process involved with this newer technology.

My understanding is that as a part of this that Herndon is using technology from Never Before Heard Sounds. They have a quick demo in this video. ( Not as good a watch as the TED Talk viceo above. )

Also not linked is a Holly+ performance of Jolene, but go ahead and Google it.

ADDED : Herndon is also using AI vocal software from Voctro Labs. They were recently acquired by another AI vocal company VoiceMod. This second company has a long term experience with this sort of thing, having worked on the Vocaloid in the past.