Hey everyone it’s time for your weekly update. Flavien has made some progress on the remaining networking issues however it isn’t completely resolved yet. With game networking there are significant challenges around compensating for latency and synchronizing game state – particularly when it comes physics simulations. To summarize the difficulty at a high level, games use lots of fancy math to estimate where everything is supposed to be and then subtly correct errors when an authoritative update from the server is received. Throughout this process there are many places where additional, small errors can accumulate within these mathematical computations. In most/all other games this small accumulated error has no discernable effect on gameplay as it results in the position of an object being off by a few millimeters or less. In Infinity: Battlescape objects are moving at speeds of thousands of kilometers a second and at such high velocities what would be a millimeter or less of error in other games becomes many meters for us. This is what was causing the stuttering we previously described and Flavien has implemented a fix that reduces it to a point that’s “good enough” for now.
WIP pre-rendered shots of the land base power plant
Unfortunately, there’s still an issue remaining with physics updates that causes collision detections to have a slight delay. Once Flavien gets that sorted out he’ll get back to server-side planetary collision detection and after that we’ll finally be able to release the next patch. On my end I spent half of last week out of town traveling to a couple of weddings around various parts of the country. Fortunately, before I left, I made some significant progress on UI support for text. As expected text layout is quite complicated. Due to our international playerbase we want to be able to support different languages, many of which have special layout considerations depending on line breaks and how characters should be merged or replaced due to ordering. While macOS and iOS provide Application Programming Interfaces (API’s) to handle just about any sort of text need you could possibly think of, the API’s available on Windows aren’t quite as thorough. This means I must fill in the gaps with my own layout engine. The good news is that our near-term text needs are relatively simple and I’m only adding support for the things we absolutely require. The initial implementation is nearly finished and I’ll add more advanced features as they are needed.
There’s not much new to report on the art side. Work continues on the land base, station materials, and destroyer. All of that should be wrapping up in about a month or so, at which point we’ll move onto the remaining 2 capital ships and weapon special effects. That’s it for this week – until next time!
Have you considered using HarfBuzz to handle text layout? It underlies the text rendering of Chrome and Firefox on all major platforms, and hence is likely the most comprehensive and robust text shaping engine on the planet. The documentation’s not great, but I’d take sparse docs over solving that problem from scratch any day.
Sounds good! This is the attitude to have at the minute - we’re not expecting perfection as long as it’s functional! Optimisation and improvements can come later
Actually the problem appears much more quickly than “thousands of Kms a second”. My tests are done at a few Km/sec of absolute speed and the error is already on the order of the meter with the stuttering at this speed, so it’s not just a matter of perfectionnism It drives me mad how all games are avoiding these issues by capping their max speeds to low values. Sometimes I kindda wish we did too, cause that would save up so much time and technical issues…
Isn’t that something that just can’t be avoided with the current internet connections?
With objects moving this quick I would think that you would need an insane tickrate to avoid noticeable stuttering.
I always wondered if it would make sense to communicate the player inputs with peer to peer connections and have the clients make positional predictions based on that information.
For example me and my buddy are traveling a high speeds between planets. My client knows that my buddy is currently flying at a certain speed in a direction, based on this information my clients guesses his current positions, the server updates tell my client the ‘real’ positions and errors are corrected that way if they occur, just like described in the op. Meanwhile all player inputs my buddy makes are send via peer to peer connection to my client and with this information my client can refine the positional guesses it makes. I know that fighting games use peer to peer to reduce lag, I imagine that sending the other players inputs (not positions) via peer to peer could lead to reduced stuttering.
This peer to peer communication could be limited by distance between players, so that two clients only start sending each others inputs when they are closed enough that stuttering becomes visible.
That’s not a latency issue, it happens even in localhost.
Imagine the server sends snapshots every 30 milliseconds. On the client you’re not going to receive them every 30 milliseconds perfectly. Sometimes 29, sometimes 31. Sometimes it’s a lot more randomized ( packets drops, processing times vary per frame, etc… ), but let’s take this example of 29 to 31 milliseconds of variation. If you’re flying at 5 Km/sec, this error of 1 millisecond represents 5000 meters * 1 millisecond / 1000 milliseconds = 5 meters. So you have to speed up or slow down along the path all the time to match the next snapshot, leading to this “stuttering” effect.
I don’t think there’s any theoretical perfect solution to that problem. The solution I currently have seems relatively stable, but that only works up to a few Km/sec. If you were flying at lightspeed, the error would be insane. Fortunately the warp drive mechanics are going to be in the way of players noticing this problem, I believe. The stuttering effect can only be noticed when you’re chasing a target at high absolute velocities. I assume combat will happen at low speeds ( “low” here being less than 10 Km/sec ). The warp drive will not be engagable in combat and to reach high velocities in warp, you’d need to accelerate for a minute or two. So in summary, the stuttering effect should only be noticeable in specicic conditions that most players shouldn’t encounter.
Don’t snapshots have a timestamp in them in order to synchronize all the clients with the Server?
So there is a snapshot/tick rate of say 10 ms and when the server starts he counts up an index of all snapshots.
So if you receive a snapshot 31 or 29 milliseconds doesn’t matter because it states in there when exactly the snapshot was made. With this information you can then predict the current position of remote ships, if they didn’t change vectors in the meantime.
For instance the server sends out snapshot Nr. 124.
You receive it 31 ms later. Your client is at snapshot Nr. 127 and has 9ms left until the next snapshot.
The game knows now that it’s been 31 ms since this information it received was “true”. Based on this and the vector data in the snapshot it can calculate the “virtual/guessed/predicted” position of remote entities. It may apply this information instantly or at the next snapshot, depending on which the algorithm has to take other delta-t into account.
If you receive it 29 ms later the same applies just with the client being at snap Nr. 126 and 1 ms left.
The client never can truly display the position of any remote player. An algorithm that smooths out what was predicted and what actually happened is needed. The prediction can also be improved, with more complex ones taking into account more information. But all of that comes after the netcode determined at what point in time the snapshot it just received was valid.
Two players cruising along without any change in their vectors should always looks quite smooth.
If not something else might be wrong …
On discourse I speculated that there also might just be a precision problem with values needed to predict. Like Velocity and radian. If those are transmittet not precisly enough that could result in FP errors. Although I threw together a simulation in my calculator and got an error of about 3^-4 Meters when using Floats (6 Significants) for everything but x/y coordinates (Used 16 Significants there). Time was an integer of course.
Inputs where:
At ten times the speed the error is at about 3^-3 meters.
Video dedicated to the release of update 76.
Note# This video is purely for entertainment purposes. I’m not trying to insult anyone either. Thank you for understanding.
HarfBuzz, as you alluded to, only does text shaping and does not perform layout. Shaping is the process by which a set of glyphs are selected based on font attributes and the ordering of the characters being shaped. It does not determine how text is mapped to lines, trimming, wrapping, inline objects, justification, flow, or paragraph alignment. Windows already provides API’s to perform text shaping via DirectWrite. Pango is the name of the Linux library that performs text layout, HarfBuzz is one of its dependencies, however it’s feature set is more or less the same as DirectWrite. I’ve been referencing Pango in my own implementation along with a sample text layout engine provided by Microsoft in their Windows 7 SDK.
It’s a shame nobody bothered to follow Apple’s lead on the quality of their text API’s however, given Steve Jobs enthusiasm for fonts and Apple’s long history with the graphics and publishing industries, perhaps it isn’t entirely surprising.
This problem has a conceptually simple solution that was discussed about two and a half years ago in the old forums. Whether it can fit into the Battlescape code at this stage is a separate issue.
Jitter is only a problem when it can be seen, and that happens when ships are both near to each other and their relative velocities are low. If they aren’t near to each other, then the jitter vanishes in the pixels of the screen. If their relative velocities are high, then the jitter vanishes in the inability to pick it out amidst so much other movement.
The solution is to track values of ships that are in the above situation. If a fleet of ships is moving in formation, then they have matched their positions and velocities so closely that any appreciable jitter is going to be visible. The very fact that their positions and velocities are closely matched presents the opportunity to factor the values into two parts; the fleet values and the individual ships’ values. (Note that ‘fleet’ applies to any set of ships that are likely to notice jitter - they can be shooting at each other as well)
So create a kind of island for the ships that are matched on position and velocity. Give that island its own position and velocity, handling the lion’s share of those values. Then fly the ships relative to that island.
Keep updating the island’s parameters so that the ship-specific component stays within acceptable levels to avoid visible jitter. When a ship departs the island’s position or velocity, it leaves that island, joins another and jitter returns between the fleet and the leaving ship. But because the leaving ship has departed the island, its jitter shouldn’t be visible.
This technique should work for any absolute velocity. It’s not perfect, but it covers a lot of ground. I can even imagine a hierarchy of islands to factor out differences at multiple levels. Perhaps a kind of quad tree treatment would work.
Ah, I wasn’t sure if it was literally only layout that was at issue, since shaping is the really hard part. I’d still recommend HarfBuzz over using using platform-specific functionality for shaping, since with it you won’t need to maintain nearly as much platform-specific code and will get consistent behavior across all platforms. I suspect it’s more complete than any single platform-provided implementation, too, considering its users and rate of development.
Forgive me if I’m re-treading familiar ground here, being new to these forums, but is there a reason snapshots aren’t being timestamped as @Lomsor describes and interpolated between, rather than extrapolated, at the client? By predicting the effects of player inputs on the client and rolling back the states of targets when doing hit detection on the server to the time at which the client fired, it should be possible to reliably prevent jitter so long as two players aren’t colliding with eachother over a long period. Network LoD reintroduces some possible jitter, but shouldn’t be relevant to nearby ships at all, and in general can be calibrated to only introduce bounded error proportional to distance from a player’s view.
One more thing I want to add to the networking is, that it sounds quite prone to having a bad experience when you have exceptionally bad jitter. Sure that is always the case but just recently I had to switch ISP due to them screwing up and making the connection extremely jitter heavy. Said that I didn’t have a lot of problems with the Prototype, though I don’t know how much is still comparable to the current version.
The Islands/Nodes JB mentioned are cool and all and I really reallyreally would enjoy the possibility to have moving Battlescapes.
After making that calculation experiment though I wonder. Does it even need that system?
By the sound of it, note I’m guessing here, it sounds like the current netcode takes the snapshot as is and treats it as “true”, Then it calculates further ship path with available information until a new snapshot arrives. With some added smoothing I guess.
That somehow disregards how the internet works. As stated above, one can not expect a stable connection. I had jumps of 50% ping time. That would create unplayable amounts of jumping arround in Flaviens example. Jitter exists, paket loss exists. Also travel times might warry.
After rambling on discourse about the island/relative system idea @cybercritic challenged it how it would solve the current problem. And after thinking about it I wouldn’t know either. It would move the jittering to the “island” but …
If two people fly side by side without input it should be possible to predict their paths 100%.
And after calculating floating point errors one could go 500km/s without encountering jagginess due to impressions …
Also it really was @cybercritic who mentioned the whole timestamp/tick thing. I knew about how it works but didn’t think about it.
The problem with using HarfBuzz across all platforms is font management. Each platform’s text API is intimately integrated with the OS’s ability to load multiple types of fonts as well as the system font collection. At some point in the future it may make sense to migrate the engine entirely to some combination of HarfBuzz/Pango/custom however in the near-term I think it’s easiest to just use DirectWrite for everything except layout. Honestly the only platform that is going to be a major pain for porting is *nix as macOS and iOS have an extremely robust text API. It’s unlikely it’ll make sense for us to port to a *nix platform for quite some time unless we decide to do a PS4 version of I:B at some point.
For the same consistency and portability reasons, wouldn’t it be easier to use Freetype for font loading, and ship a (set of) font(s) with the game? Pango’s pretty heavyweight and gnomey so I wouldn’t necessarily recommend it as a monolithic solution even on Linux.
How is this an attractive feature? Surely the fonts used by the game will be packaged with the game? There will be no need to use multiple types of fonts or system fonts, will there?!?
The shipping version of the game will likely use special fonts which will ship with the game. In the near-term however we’ll probably use standard system fonts like Arial. Either way we still need something that loads fonts, regardless of whether it ships with the game or not, and it’s easier to use the OS capabilities instead of integrating a 3rd party font loader.
