Not necessarily. The slower the clock, the longer the internal sequencer / audio engine has to be running in a kind of free-wheeling mode in between incoming clock messages. I can see this being problematic (not least because of jitter in sampling).
If the (external) clock source is slightly unstable, this is much more noticeable at slower speeds.
Of course it’s all relative, internal (CPU) clock, timer frequencies, interrupts are all factors in this.
At very high clock speeds, you’re more likely to run into problems with merging (or parsing) other messages, like cc and notes etc., due to bandwidth.