You hit the nail on the head. This is exactly how I feel about Ableton/Push workflow. Fun to use hands-on hardware-like control, but also have access to huge screen and mouse control whenever I want to do small adjustments and detail work.
I admit, this is one of my struggles too. I think custom mapping of 3rd party plugins would help, but not a cure-all. If you could boil down 3rd party plugin controls and make enough racks, etc. you might at least have more of an Elektron type sound design experience - 8 knobs controlling whatever is on screen at the moment, as opposed to the current method of bank 1, 2, 3, 4, 5, and on and on, until you stumble across the param you want to adjust. Ugh, so slow. I do like some of the max4live devices that are simplified for easy use with Push interface - feels like OP-1.
But, to me, Push and/or Elektron UI will never beat a dedicated wysiwyg interface like MS20 or Moog stuff, which is why I’ll always have a couple around for variety. But at least the new Push visualizations are helping somewhat. Much better than just the knob and circle/number. But ultimately, I agree, sound design and joyful experimentation is somewhat difficult on Push, depending on device under control at the moment. I think it shines mainly on playing synths, making beats, mixing, automation, etc.
DS-1 is pretty cool. Faders feel fantastic. I do wish Ableton would make their own version, styled similarly to the Push 2. Encoders, screen(s), and motorized faders (I could live with touch strips maybe).