Lets try to find out why.
So the client realizes that the last couple of snapshots he received are way behind his playback time.
I assume this desired time is taken once. Before the time readjustment is started. If it's not and the "life" time of always the newest snapshot is used we have a situation where you try to interpolate to a target that constantly changes. Making your playback time stutter.
If you interpolate between two points you keep track of your progress from point a to point b. If during the interpolation point b changes you will get different result for the same progress you had before. That's basics but just wanted to point that out.
So the client realises what's going on and calculates a new target playback time based on the last few snapshots it got. The arival time averages out at about "time_now" - 1000ms. So it chooses "time_playback_new"="time_now"-1000ms-100ms.
Now it starts to interpolate its "time_playback" from "time_playback_old" to "time_playback_new".
What happens in this timeframe? Each cycle the "time_playback" changes. So every calculation that takes that time into account and does stuff for more than one cycle is affected.
Position interpolation for instance. Position interpolation probably uses "time_playback" for its "progress". Something like (most basic interpolation, just an example):
When "time_playback" is interpolated it may move faster, slower or even backwards. Compared to normally when it moves exactly like time does. One second in one second.
This depends on how large the lagspike is and how agressive the "time_playback" interpolation is configured. I make it simple here and just assume linear interpolation. But you can imagine how other interpolation would influence it. It's basically the same with the difference that the speed is faster in some sections of the interpolation then in others instead of linear like it is in this simple example of linear interpolation.
So in some situations positional interpolation is done faster. If network connection gets better.
In some cases positional interpolation just stops. Because the the time interpolation moves backwards in time exactly one second per second.
In some cases positional interpolation moves backwards. Where the time interpolation moves backwards in time more than one second per second.
So you may even see ships moving backwards. https://i.ytimg.com/vi/n5I4oqFNxvM/maxresdefault.jpg
If you constantly try to adjust "time_playback" to be exactly 100ms behind the time you receive snapshots you will probably see what you are seeing with ships stuttering arround. Why?
Because you will never have packets arive at exactly the same time. Even on localhost. Network dirvers and the operating system aren't set up for this kind of consistency.
Companies trying to use TCP/UDP for communication that is dependent on consistency (motor control, safety) have to use real time network drivers that are run in a precisely crafted environment (read: not Windows, Linux or similar).
But you don't need that consistency. That's what the 100 ms buffer and the "time_truth" is there for.
I would suggest checking if either interpolation targets are moved while interpolation is going on or if "playback_time" interpolation just happens too often or/and with extreme settings/speed.
The algorithm to decide if its needed should be quite picky.
Both could result in stutters like you and Keith described.
Edit: Now do these stutters increase with velocity? I think they do. The faster an entity gets the more difference there in his position in two successive snapshots. Making the distance over that is interpolated larger. Snapshot frequency on the other hand stays the same. The larger the interpolation distance is, the more it amplifies either of the two theoretical sources of the stutter.
So this theory lines up with observation.
It kind of acts like a regulatory circuit that hasn't been configured correctly and is oversensitive.