Take a look into ORAC and Patchbox OS. You’ll also need a DAC. Pisound apparently works well with all of this. I could never get it running on my HIFIBerry DAC+ ADC Pro but it was the PD part from a Windows system that was hanging me up. I just lost patience and moved on.
@thetechnobear I believe is the developer for ORAC (don’t quote me on this).
He has a couple of YouTube videos here and here that go over the features.
There are ports for a few Mutable Instruments devices that may be of interest.
I’m doing something similar with pure data using the pi and the Blokas pisound sound card. Here are images of the panel/controller I’m building in the shape of a moth that will connect to other arduino sculptures containing sensors that will also send back data to impact the audio processing within the patch. Still building and prototyping but this video has been a great jumping off point for granular synthesis.