Yeah, I was going to say, Axoloti + Raspberry Pi + inexpensive 7" touchscreen display + a few buttons and knobs is a good starting point. I’ve seen a few projects based around that general idea. Axoloti is the actual engine, RPi+touchscreen lets you run the Axoloti software for deep parameter diving and some patching on the fly, and the physical controls are for accessing key parameters quickly with high resolution via the Axoloti’s GPIO pins. Total cost should be around $200 depending on the specific controls and case you use.
That way you can easily play around with control schemes and software design without having to do much if any coding, and since it wouldn’t ahve to be tied to a computer you could really get a feel for it as a standalone instrument during your development process. Experimenting with different physical interfaces would just be a matter of connecting different hardware to the GPIO header and making a few changes in your patch.
Since you already program, once you were satisfied with your prototype you should be able to port the functions over to native code for some kind of dev board or an Arduino or something.