The matter continued to interested me and I experimented to get this problem.
After days of tinkering on the computer I still had no success, but today my bad memory still has something to say.
Well, I can actually understand the latencies now. The picture sometimes lags behind.
The reason for my originally smooth experience is my active LINDY 38415 Repeater, which probably smoothes the signal.
Maybe someone can counter-test that, then you would at least have a reasonably cheap workaround to the problem.
@DrWilken - any (affected) person with a 40xx in your Area?