You probably need to use something that was actually made for MIDI.
Unless your stereo switcher is passive (made of mechanical switches only) then you will likely encounter problems due to incompatible electrical specifications.
You might also have issues because the signal frequency is higher than the audio signals for which your stereo switcher was designed.