i used to have this problem. turned out that my wifi connection was to blame. turned off wifi and latency was the same every time. this is on my OS X laptop, so may be a different cause on your machine. worth investigating…