If you want to use a small number of connections at a time but different combinations, consider an audio patch bay plus a small (= few channels) audio interface instead of a large audio interface.

Forgive my ignorance now but if I had a small audio interface why would I need the patch bay? Is it because it’s a hassle to constantly swap the configuration around on the actual interface itself?

Cheers Peter.