Was watching Gamescom and the latest SC show 
Apparently we’re not the only ones with stuttery networking 
11 Likes
Well, you know what you have to do then, limit velocities and make ships in warp bubbles behave as a separate instance.
1 Like
I rather have stuttering at high speeds where fighting is impossible anyway instead of artificial speed limits.
The SC demo was such a trainwreck. 
I would like to know why this is the state of the tech at the moment. Nobody seems to be able to explain it.
I just assume it’s bad design from now on.
Netcode is hard it appears. 
Edit:
Maybe “bad” is a little to generalized. Maybe “sub optimal” or “optimised for other fields”? Well.
1 Like
Allright, you can get the test app here, code included
https://inovaestudios.blob.core.windows.net:443/private/Flavien/InterpolationTest.rar
- The red dot represents the local player
- The green dot represents the server player ( interpolated )
- Left click & drag to move the camera around
- Mouse wheel to zoom in/out
- Press T to toggle timer smoothing
- Press Shift + 1 to 4 to reset the velocities of both the local player and the other player; 1 = 0 km/s, 2 = 100 m/s, 3 = 500 m/s, 4 = 5 Km/s
- Page Up / Down to increase / decrease the virtual latency between the client & server
- Cursor arrows left/up/right/down move the server player around
- Note that the grid rendering is limited to 100 Km, if you go fast / long enough you’ll get outside the grid
- Code is quick & dirty and misses some smooth transitions between extrapolation and interpolation ( which the game does ), but I wanted to KISS.
- Note that the code fits in around 200-300 lines, so it’s not too complicated and heavily commented.
The settings can be changed in code. It defaults to 10 snapshots per second, a time offset of 2 snapshots and a virtual latency of 100 ms.
Understanding the app and replicating the stuttering issue: time smoothing (which causes the stuttering) is by default enabled. When you start up the app for the first time, you’ll see a red dot (camera is always centered on it) and a green dot. The green dot is where your attention should go, since it’s the one getting interpolated from the server. You can use the cursor arrows to move the green dot around on the server side, and see it move on your screen. If you change the code settings to reduce the snapshots update rate, or change the virtual latency, it’ll be obvious that it’s “lagging behind” your inputs, as expected.
You’ll see the green dot sometimes blinking. When it’s green-filled, it means the client is extrapolating. Which is not a good thing. It shouldn’t happen too often. In real conditions, it happens with lag spikes or packet drops. In this app, you can simulate that by increasing/decreasing the virtual latency with the Page up / down keys. As you do, you’ll probably see the green dot blinking until it stabilizes. That’s what time smoothing does.
However, especially after it stabilized ( this can take a while, unless you change the smoothing factor in code ), if you set the velocities to 5 Km/sec and zoom on the green dot, you’ll see it stuttering. Move it closer to the red dot to make that fully noticeable.
Now, if you disable time smoothing by pressing the T key and interpolation had already stabilized, if you do the same scenario, you’ll notice the green dot is perfectly stable (no stuttering). The problem here is that if you change the virtual latency, it’ll extrapolate forever. In other words, press T to disable time smoothing, press PageUp a few times to set the latency to > 500 ms, and watch the green dot extrapolate all the time. The client and server times are totally desynchronized.
So the goal is to reconcile the changes in latencies which cause extrapolation/interpolation, and the time smoothing which causes the stuttering issue.
7 Likes
Press 1 to 4 without the SHIFT.
Flavien, you might want to put in a display of the server ship’s ‘true’ position. I did that and it’s clear that while the server ship position displayed on the client moves without jitter, it’s pretty inaccurate. Add client ship speed and the inaccuracies gets worse. Add latency and they get worse again. I can’t believe that’s an intended result, but I don’t know if that’s just a quirk of the test rig.
That’s probably correct, the difference will most likely equal the latency + the time offset ( = 2 snapshots ), multiplied by the speed. So for a latency of 100 ms, a time offset of 200 ms and a speed of 5 Km/sec, that would be (0.1+0.2)*5 = 1.5 Km.
But those numbers aren’t realistic. First of all, a latency of 100 ms would equal a ping of 200 ms ( half the round trip ). Also the update rate is 10 snapshots/sec which is quite low. At 50 ms latency ( = 100 ms ping, what you get in average between US and EU ) and 60 ms of time offset, the error at 5 Km/s would “only” be 550 meters.
Still, the fact is that the higher the speed is and the more the latencies are, the higher that difference is.
Can’t wait to get home on Monday and start tinkering…
To aid collaboration and the sharing of ideas (in the form of code), can we get this up on a git server? Maybe on github as a private repo or just a self hosted git server?
1 Like
Sure, that’s fine.
Disclaimer: I have no idea how this all works.
Something I noticed in the lerp article:
Also as long as 0 < f < 1, it will never overshoot.
from https://www.scirra.com/blog/ashley/17/using-lerp-with-delta-time
But in the code example f can be both 0 and 1, right?
if (smoothTimerState)
{
// speed up or slow down the display time to match the target time
// that's pretty much the cause of the stuttering, however without that code, changes in latency mean the client and server timers are no longer synchronized properly
// ie. try to PageUp to increase the latency to 1 second -> the client extrapolates all the time (green dot filled)
double delta = Math.Abs(displayTime - targetTime) * 3.0;
delta = Math.Max(delta, 0.0); // <-- Returns 0 if delta is 0?
delta = Math.Min(delta, 1.0); // <--- Returns 1 if delta is 1?
displayTime = lerpDT(displayTime, targetTime, delta, elapsed);
}
But this does not fit the criteria 0 < f < 1
It fits 0 <= f <= 1 ?
When I change the code to the following the green dots still wiggles around but it doesn’t teleport (I think).
delta = Math.Max(delta, 0.1);
delta = Math.Min(delta, 0.9);
2 Likes
People give me your GitHub username and I’ll add you to the private repo.
I went through the code and tried to figure out what you’re doing, but with so many time variables being modified, I really couldn’t. So I started instrumenting stuff and then did what I thought would be the right way to do things: I extrapolated up to the present from the information in the most recent snapshot.
I’m hoping that this dropbox link works:
The implementation jitters modestly and that clearly would need smoothing, but the location of the ship that the client sees is pretty accurate. I just don’t want to put any more time into hacking around. If I was serious, I’d go and write something from scratch. Anyway, perhaps it can be useful for discussion purposes.
Ok first thing I noticed is that there seems to be something weird going on with the lerpdp function.
It doesn’t line up with any of the documentation linked to.
Now the article linked in the code describes following:
c:1-f^dt;
But in the code it is:
t:1-(1-f)^dt;
lerp_dt: a*(1-t)+b*t;
I know that in the code another lerp function is used as a basis than in the article. But still t or c should represent the factor of interpolation.
Also it appears that doing it the way described in the article results interpolation that closes in as time progresses but never actually reaches it.
Changing t to following: “t:1-(f)^dt;”, so it conforms with the article, dramatically increases stuttering.
Maybe Flavien did some further modification there I missed.
I can confirm that setting the boundaries of the “f” value to not allow for 0 and 1 eliminates overshooting.
@hrobertson guess what. It’s Lomsor
Will further investigate …
Everything is looking great! Upcoming patch is going to be truly innovative and groundbreaking - the public and industry will realize they vastly underestimated I-Novae Studios. Keep up the fantastic work Devs!
3 Likes
TLDR
I found the source of the stutter.
It’s the snapTime / targetTime. snapTime has fixed intervalls. displayTime does not. This creates a targetTime that constantly bounces arround.
Only set targetTime if needed.
Here’s what I did with it. “Hotfix” style:
http://lomsor.com/inovae/InterpolationTest/InterpolationTest_Lomsor_4.zip
Instead of a hotfix I would suggest not trying to time interpolate the way it is done in this test at all. Do it like you did further down with the position.
If there is a network change decide what new targetTime should be set. Then interpolate that the classical way. Without moving the ballposts! Then when you reach the new time target you will be exactly there. If you want repeat … but only after you reached the end. Your current code can’t interpolate exactly to a certain time. Even if it did get precise input, which it does not.
After checking several possible sources. Mostly concerning clockkeeping and time smoothing I found the source of the stutter.
I did this by just bypassing all the smoothing and setting (displayTime = targetTime).
The faster speed or latency. The higher the stutter of targetTime. So this matches with observations.
Here you see it in action. (targetTime-displayTime) if (smoothTimerState) is off.
With the creation of the first snapshot targetTime is the same as displayTime.
With creation of every next snapshot targetTime is set to (snapTime - Settings.timeOffset).
(snapTime) stems from the update function on the server. It is calculated by multiplying the snap id with the snap rate. Giving you a time when this snap might be valid.
Problem is that this is not when the time interpolate is run! The time interpolate is run constantly. This means that it constantly tries to interpolate to times that only changes with the snapshot update rate.
Here you can see how the stutter coincydes with the client using a snapshot. This matches with the rate of stutter. In this test (displayTime = targetTime).
It appears Flavien wants to constantly regulate displayTime. In that case targetTime needs to be selectively regulated.
I added this code to the targetTime. It eliminates stutter. It reacts to spikes after 0.1 seconds by detecting if extrapolation is activated and if the difference between targetTime and displayTime is >0.06 seconds.
I also added feedback loop that uses an average of multiple snaps. It is more precise than using snapTime but It adds some overshoot and bounce.
It’s all cobbled together really.
A side note:
This elapsed time is only accurate to milliseconds.
So I tried to make it more accurate
Elapsed.TotalMilliseconds is a double. As accurate as possible based on the actual Stopwatch ticks.
After some tests though I couldn’t really see any noticeable difference.
Also. The player ship should also be subject to these time adjustments. Currently if the client lags out everyone else is set into the past except for him. If you want to keep aproaching this problem this way I would suggest also having the player ship be influenced by the different displayTime.
Also also. I would reconsider this “shifting time” system. It can be quite complicated to debug when there’s one important clock that gets modified often … maybe abstract it a little so at least you can compare how far apart past packets arrived and such.
Making players play in the past is fine and dandy. But the way it’s set up right now is kind of nightmarish.
Another option would be to try and make snapTime more predictable. I don’t currently have an idea how to do that. Hope my time on this has helped some.
9 Likes
If anyone wants a relative and infinite grid, replace the following code with the one below it.
/*double size = 100.0;
for (double i = -size; i <= size + 0.001; i += 1.0)
{
Vector2D start = worldToScreen(new Vector2D(i, -size));
Vector2D end = worldToScreen(new Vector2D(i, size));
graphics.DrawLine(Pens.Blue, (float)start.X, (float)start.Y, (float)end.X, (float)end.Y);
start = worldToScreen(new Vector2D(-size, i));
end = worldToScreen(new Vector2D(size, i));
graphics.DrawLine(Pens.Blue, (float)start.X, (float)start.Y, (float)end.X, (float)end.Y);
}//*/
…
float lineDist = 1.0f;
Vector2D pos = new Vector2D(clientSim.localPos1.X - clientSim.localPos1.X % lineDist, clientSim.localPos1.Y - clientSim.localPos1.Y % lineDist);
for (int i = -20; i < 20; i++)
{
Vector2D horiz = worldToScreen(new Vector2D(pos.X + (i * lineDist), pos.Y + (i * lineDist)));
graphics.DrawLine(Pens.Green, -1000f, (float)horiz.Y, 1000f, (float)horiz.Y);
graphics.DrawLine(Pens.Blue, (float)horiz.X, -1000f, (float)horiz.X, 1000f);
}
4 Likes
While I don’t have the networking know-how or the time to chip in on this discussion, I just wanted to note that this is really cool to see. The fans and devs of a game being made discussing and trying to solve problems together on this level, is amazing. “Open development” at a whole new level.
5 Likes
After a little of fiddling, I think @tars is the closest to a solution. @inovaeflavien, the stutter is due to the time smoothing going to the extreeme and it looks like the problem can be solved by simply limiting the max and min smoothing parameters.
if (smoothTimerState)
{
// speed up or slow down the display time to match the target time
// that's pretty much the cause of the stuttering, however without that code, changes in latency mean the client
and server timers are no longer synchronized properly
// ie. try to PageUp to increase the latency to 1 second -> the client extrapolates all the time (green dot filled)
double delta = Math.Abs(displayTime - targetTime) * 3.0;
delta = Math.Max(delta, 0.25);//was 0
delta = Math.Min(delta, 0.75);//was 1
displayTime = lerpDT(displayTime, targetTime, delta, elapsed);
}
By just changing those two constants, I’m not getting the stutter that I was getting in my controlled setup. It will take the client a little longer to adjust, but there is no stutter.
First half of the video with it jumping is without the fix, second half is with the fix. I just change the latency to and from 1000ms to test.
3 Likes
[quote=“TARS, post:73, topic:5747”]But this does not fit the criteria 0 < f < 1
It fits 0 <= f <= 1 ?[/quote]
There’s no mathematical reason it cannot be 0 or 1. It’s just the interpolation factor, so of course 0 and 1 are allowed. 0 means you get the starting value and 1 means you get the ending value.
The meaning of the ‘f’ parameter was inverted compared to the article, since it’s more natural that a higher value would mean a stronger/faster interpolation. The article’s unintuitive. It’s not a problem as long as you feed it coherent values ( ie. a higher ‘f’ value when you want interpolation to be stronger ) which is the case here. In other words: that’s no bug.
1 Like
[quote=“cybercritic, post:81, topic:5747”]
After a little of fiddling, I think @tars is the closest to a solution. @inovaeflavien, the stutter is due to the time smoothing going to the extreeme and it looks like the problem can be solved by simply limiting the max and min smoothing parameters.[/quote]
Limiting the range to 0.25 - 0.75 ? I can still clearly see the stuttering. Was there other code changes ?
Looking at your video, you’re not even showing conditions in which the stuttering would be visible, so I’m wondering if the “stuttering” I have in mind is different than what you think / see.
To see the stuttering I’m speaking of, which is the issue, you should increase speed to 5 Km/sec ( Key 4 ) and zoom on the green dot. A lot.
The fact that the dot sometimes jumps around when you change latency is simply because I kept the code as simple as possible; it’s not the issue, the full game doesn’t have this problem.
2 Likes