(thoughts about how to) reduce lag

[titi]

In some svn versions we saw some lag after joining that made me think about lag in general.

What kind of lag do I talk about here is a lag related to multiplayer. Its the command delay lag. So if you give a unit a command it may take a  moment until it reacts.

How does it work in general:
Commands of all clients are send to the server. This collects all commands of one key frame and distributes them to all clients. for execution.
Now the client ( servers a client too in this logic ) receives the command list and executes all commands in the moment they were meant for.
If there are no commands for the next keyframe, the client stops and waits for them.
If the command list was already received it does not stop and just executes the command.

The following scenarios make trouble now: ( Client KeyFrame is c here and server KeyFrame is s )

A client sends his commands to the server. As this takes a while. It can cause one type of commad lag which is:
Delay d=Client to server send time+ time to next command distribution of server+ time to send commands from server to client.
If this delay is bigger than the duratoin of 1 keyframe the command is executed one keyfFame later which is the some lag you already feel.

so   

Code: [Select]

Delay d / keyFrameDuration is the number of keyframes the command is delayed.
This Lag is "natural" and nothing we can change! Its caused by the network speed and is the raw time needed to send packages to the server and back.

But I think there is another problem which makes this problem worse:
Its caused by the start of of the game and the fact that each clients has its own timing.

In the moment the server wants to start he sends a "Start" command to all clients. Once the clients receive this message they start their game and timer.
As the time such a single "start" message can vary a lot it can cause the folowing effect:
a) if a client reaches the next keyframe he might be there too early, because sending the command list takes a bit longer than the initial start message. By this the client has to pause and wait until ist received. This is bad and will result in visible stop and go. To solve this we delay the start of the client a bit to ensure the command list form the server is alreay received.

But what happens if the client starts too late ( for whatever reasonslike long time of sending first message or whatever) ?
This is is very bad and  because it will cause teh following:
As already said the client starts his game and timer after receiving his start comand. If this is delayed for something like 5 seconds we will never be able to catch up this delay at the moment, because the game is timed locally to the clock/timer we started when we received the start message.
5 seconds back means: send a command to the server. the server will queue this command for next command round, but as teh server is already 5 seconds further this command will be in a command list meant for the next server keyframe.  Sadly this command cycle will happen 5 seconds later on the client. For the player on this client this looks like a 5 second delay commanding his unit, also the connection might be fast enough.

I think/hope this can be fixed with the following idea:
On the client once we receive a command list from the server, we save the time ( or better the local game frame )when it was received. Now if we want to start executing this command we compare the current frame to the one we saved after receiving the game.  If this time difference gets too big its the time we need to catch up. This should not be done too exact as sending messages over network can differ a lot. We need to watch it for a longer time period like 10 key frames for example to be sure. Average/Min/And MAx should give us a good idea of what value we can use  to correct the lag.
On the other hand we must correct if clients need to wait for server commands like described before.

[softcoder]

I think i've discovered how to deal with this without needing to pause the server.

For now please test the join in progress game and let me know if the current lag issue is resolved. We have lots of possibilities to fix lagging in general now that I understand the problem.

[tomreyn]

Playing with two processes (server, client) on my slow intelbox (specs below), using separate copies of MegaGlest and UserData_Root I get a constant lag of 2 FPS for the client who joined the game the server was already running, with some spikes of up to 8 FPS. So this is much improved, but could possibly be improved even further.

Note that I needed to activate the "wait for lagging clients" option to not get disconnected when the game starts (and a player joining the game takes about 2 or 3 minutes). I'm not sure whether this is due to this low spec hardware or whether it's mabe a general issue.

[Coldfusionstorm]

A thread devoted to fixing/reducing lag?, i love you guys! 

[titi]

Current state: Revision 4162

I played several games and they all were a bit different.

game1:
First game ( 3 player ) was great with very constant command delay ( 1 second )  and no visible choppienes for anyone. In the end filus dropped out due to trouble with his system but that was all.

game2:
Player filux  crashed in the middle of the game !? But I don't know why Maybe someone tried to join the game while it was running or his system trouble again . Before all was good. 1 second command delay, displayed as 0(0) for me as server.

game 3 ( with carla form brasil so bad connection included ):
I was server and gameplay was very good for me. In the beginning I got one little freeze of maybe 0.5 seconds but thats all. For MuwuM the game was ok in general . Carla ( If I understood correctly ) had some problems  ( choppy  ).
On server she was listet with 1(0) and  one time I saw for a moment  2(0) .

game4:
Thats why we started game 4 where carla form brasil was the server :
For her all was fine. In the beginning I had a 2 or 3 litle stops but than it was much better then the game we played before with 3.7.1 .
But for MuwuM ( also germany ) it was very bad, He had unplayable parts but also very smooth parts with high framerates. 
( I think what happend here was caused by a lag from round about 1 second.  SOmetimes the packeges where there and sometimes he had to wait. so if this changes because one time it takes 1.02 seconds and the next time it takes 0.9 seconds things maybe get choppy . )

game 5:
German headless with tomreyn, filux(poland) and me testing:
For me the game was good. No freezes or anything. I constantly had 1 second command lag like for all games I played as client. This lag is typically displayed as 0(0) on the server ( so not in this case as we used headless ).
After a while filux and tomreyn talked about very bad fps ( the map had a camera position which was zoomed out a lot , so there was a lot to render and at least tomreyn had bad hardware )
Then filux dropped out because his client crahsed. After filux dropped out, suddenly  tomreyn had a lot less cpu load . ( he will write a bug report for this too ).
Later tomreyn crashed too , to look at  his coming bugreport too  ( looking at the stack some kind of memory corruption )
(update tomreyns bugreport: https://forum.megaglest.org/index.php?topic=8922.0 )

[titi]

Current state (rev 4218):
I now changed the implementation again and now every client records the last 30 networkframes. Its recorded how often and how much a message form the server was received too early and how often and how long we had to wait if the message arrived too late.
Based on these statistics a speedup or slowdown of the game is decided. So far this still makes problems with big lag (ping times of 1000 ms ) . And there are still many printfs in the code with interisting output.

Bad network ( lag and varying lag ) can be tested very good with tomreyns scripts, but you still need 2 seperate computers to test. (https://forum.megaglest.org/index.php?topic=8946.0)

[tomreyn]

Does it not work when you set the latency on the loopback interface and connect through that to a server running on the same system then? I don't think we're using Unix sockets for this purpose but TCP connections exclusively so I would expect it to work for running two instances on a single system, too.