Weird latency problem with A4mk1 and Reaper

I’m encountering a weird problem with latency when recording/monitoring audio using the VST plugin in reaper.

It’s delayed by about 260ms, the weird thing is that it works fine (minimal latency) when I first added the VST tracks and set up the routing etc. but then after a while it starts occurring again and the only fix is to delete the OB plugins tracks and recreate them again.

Is there any setting/change that anyone knows that might be causing this?