I didn’t have this issue, but I didn’t use it with a Voyager.
Have you confirmed this behavior when using it to control a synth other than the Voyager?
Have you tried tweaking the output voltage on the A4’s CV?