This problem has a conceptually simple solution that was discussed about two and a half years ago in the old forums. Whether it can fit into the Battlescape code at this stage is a separate issue.
Jitter is only a problem when it can be seen, and that happens when ships are both near to each other and their relative velocities are low. If they aren't near to each other, then the jitter vanishes in the pixels of the screen. If their relative velocities are high, then the jitter vanishes in the inability to pick it out amidst so much other movement.
The solution is to track values of ships that are in the above situation. If a fleet of ships is moving in formation, then they have matched their positions and velocities so closely that any appreciable jitter is going to be visible. The very fact that their positions and velocities are closely matched presents the opportunity to factor the values into two parts; the fleet values and the individual ships' values. (Note that 'fleet' applies to any set of ships that are likely to notice jitter - they can be shooting at each other as well)
So create a kind of island for the ships that are matched on position and velocity. Give that island its own position and velocity, handling the lion's share of those values. Then fly the ships relative to that island.
Keep updating the island's parameters so that the ship-specific component stays within acceptable levels to avoid visible jitter. When a ship departs the island's position or velocity, it leaves that island, joins another and jitter returns between the fleet and the leaving ship. But because the leaving ship has departed the island, its jitter shouldn't be visible.
This technique should work for any absolute velocity. It's not perfect, but it covers a lot of ground. I can even imagine a hierarchy of islands to factor out differences at multiple levels. Perhaps a kind of quad tree treatment would work.