r4130: Rejoin: Issues with joining in-process games

[tomreyn]

So joining in-process games works since r4129 without going out of synch - that's great.

Some non-fatal issues remain, which I'll list for later:

(1) When a new player joins an in-process game, the illumination of the veteran players's units and structrues sight range is lost (Softcoder: this is not yet fixed for me).

Screenshot:



The player who took this screenshot owns the structures visible here. As such these buildings and their immediate sourroundings (view / sight range) should be illuminated, but they are not. Similarily the units standing behind the main teepee have lost their illuminated surrrounding. To work around this, it's sufficient to move these units.

Apparently this issue is limited to the illumination of the sight/view range. The behaviour regarding whether or not enemy units are visible is correct.
In this screenshot, the enemy units (battle mages) attacking a non-illuminated structure (main teepee) do show properly (and, as expected, become invisible as they move away further):



(2) When setting up a game, clicking on "Reload last game settings" visually restores the settings of the last game properly ("Players can join in game" is ticked and there is both a CPU and an open network slot), but factually nobody else can connect to this server, both while the game has not started, and whle the game is in-progress (client runs into a connect timeout). Unticking and again ticking the "Players can join in game" box fixes this issue: other players can then connect both to before the game started and while it's already in progress.

(3) Even when all participants are on a LAN connection, a player joining an in-process game will have constantly high latency, around 3-5 seconds minimum. On Internet games, this is more like 10 seconds minimum, and can easily cause players to time out. The same players can play on the same server just fine and without increased lag if they play from the very start of the game.
When a players joins the in-process game (and thus has large amounts of lag), there are also complete freezes every ~ 2-4 seconds lasting for ~1-2 seconds, and, just during these freezes, heavily increased CPU load on two (which are maxed out) of four CPU cores.
(This report refers to r4132.)

(4) When a player who previously joined a game when it was already in process disconnects, the game freezes for a second or longer. This is way longer than usually when a player decides to disconnect or is forcefully disconnected.

(5) When a player connects to an in-process game, then disconnects and the same or a different player tries to connect again, s/he is unable to connect. (This report refers to r4132.)

(6) When a player connects to an in-process game, all pre-existing players should receive a notice that someone is joining, explaining the pause. Currently only the game host receives it.

[softcoder]

Please try svn rev#: 4144 and report what is still outstanding.

[tomreyn]

Test results with r4144:

(1) persists.

(2) is fixed. This may have actually been a side effect of [fixed] r4139: Client cannot connect to game server although slots are open

(3) persists.

(4) is now less of an issue, there are now (?) two short freezes: one when the client reports that it will disconnect, another (about 3 seconds later) when the client actually disconnects. That's not ideal, but I guess it's acceptable, and at least the latter freeze can also occur when a player who had been connected from the very start of a game disconnects.

(5) is fixed. This may have actually been a side effect of [fixed] r4139: Client cannot connect to game server although slots are open

(6) is fixed.

Additional issues:

(7) When a client connects to a gameserver where a game is already in progress, and the player controlling this server pauses the game, the client is forced into the game, i.e. the join process is initiated even though the client player never pressed the "play" button. Only when the server unpauses the game, the connecting client does actually get to see the lobby screen.

(8) When a server has the option set which allows new players to join games which are in progress, and has started and paused the game, then clients trying to connect to this game get stuck on the connection screen (with only the "disconnect" button shown). However, the server receives a message that the client has connected to the game lobby and is able to choose a player position.

[softcoder]

Try rev# 4147, as i think it may solve many of these issues.

[tomreyn]

Test results on r4162:

(1) persists
(2) was already fixed
(3) is now fixed - great! However, my two cores are now maxed out as soon as a player joins a game which is in progress.
     Update: PT / titi_son tells me that there is still high lag when you have >=2 players join while the game is in progress. I haven't had a chance to test this, yet (can't test on this computer due to the CPU overhead).
(4) will need more testing
(5) was already fixed
(6) was already fixed
(7) is now fixed
(8) will need more testing.

Edit: Update on (3)

[tomreyn]

With r4168:
(1) is now fixed
(2) was already fixed
(3) is now fixed, and CPU is no longer maxed out as soon as a player joins a game which is in progress. I also was unable to reproduce PT's / titi_son's report (that there can be high lag when you have >=2 players join while the game is in progress).
(4) is actually still an issue: When a player disconnects who had joined a game while it was in progress then there are two freezes, each lasting about 3-5 seconds. Titi asked me to stress this issue again, because he assumes this may be a side effect or an indicator for what also causes clients to go out of synch (if I got him right there).
(5) was already fixed
(6) was already fixed
(7) was already fixed
(8) will still need more testing

More (previously unreported) issues:
(9) Some messages about a player joining an in-process game may be dropped when a player disconnects and quickly reconects. That's messages about a player connecting to the lobby, starting the game join process, entering the game after successful restore of the savegame.
(10) Clients can go out of synch. How to reproduce this is not clear, yet. It only happens with clients rejoining an in-process game. It took us three or more rejoins of two clients and an attack reaching the CPUs on the opposite side of the map to make it happen.

(11) There can be high load on a single CPU core of a client which joined an in-process game.
Note how PID 20765 is consuming 195% of resources on a single CPU core in this htop output:

Code: [Select]

  1  [||||||||||||||||||||||||100.0%]     Tasks: 123, 303 thr; 3 running
  2  [|||||||||||||||||||||||| 77.1%]     Load average: 2.32 1.77 1.19
  3  [|||||||||||||            41.2%]     Uptime: 14:37:22
  4  [|||||||||||              32.9%]
  Mem[|||||||||||||||||||3169/7985MB]
  Swp[                         0/0MB]
  PID USER      PRI  NI  VIRT   RES   SHR S CPU% MEM%   TIME+  Command
20765 tomreyn    20   0 1848M  566M 19672 R 195.  7.1 11:08.31 ./megaglest
20958 tomreyn    20   0 1848M  566M 19672 R 98.0  7.1  5:32.38 ./megaglest
20933 tomreyn    20   0 1848M  566M 19672 S  1.0  7.1  0:03.87 ./megaglest
21180 tomreyn    20   0 27324  2172  1412 R  1.0  0.0  0:03.81 htopThis situation was accompanied by the following output on the terminal:

Code: [Select]

Load game setting unit pos
Resume #1
Client waiting for packet for frame: 28080, copyCachedLastPendingFrameCount = 28060
Client waiting for packet for frame: 28160, copyCachedLastPendingFrameCount = 28140
Client waiting for packet for frame: 28600, copyCachedLastPendingFrameCount = 28580
Client waiting for packet for frame: 28900, copyCachedLastPendingFrameCount = 28880
Client waiting for packet for frame: 29280, copyCachedLastPendingFrameCount = 29260
Client waiting for packet for frame: 29400, copyCachedLastPendingFrameCount = 29380
Client waiting for packet for frame: 29660, copyCachedLastPendingFrameCount = 29640The last lines kept going on like this (roughly 350 such lines), only once intercepted by:

Code: [Select]

Client will speed up: 20 frames lastNetworkFrameFromServer: 38000 world.getFrameCount() = 37972 updateLoops = 1


[softcoder]

Try svn rev #:4171 and let me know if the issues remain?

[tomreyn]

With r4173:

(1) was already fixed
(2) was already fixed
(3) was already fixed
(4) will need more testing
(5) was already fixed
(6) was already fixed
(7) was already fixed
(8) is now fixed
(9) will need more testing
(10) persists
(11) will need more testing

Previously unreported issues:
(12) When a game is paused, and a new client joins, the game resumes automatically after the client has finished the join process. However, the game state (paused in this case) should be retained.

[softcoder]

I am not able to reproduce issue #12, could you please outline exact steps to reproduce this?

[tomreyn]

Sure. Host a game, announcing to masterserver, enable the option to connect to an in-progress game, start the game, then pause it. Now start the game on another computer and connect to the runnig game. After joining the game, which was paused, will resume instead of remaining paused.

What should we do about (10), since it seems unlikely that I'll be able to provide much better steps to reproduce? Would a network debug log help? Would it be better to do what Titi suggested and have all parties restore the game state from the savegame?

[softcoder]

Trying the same steps you posted above does not cause this to reproduce on my end (I already tried those exact steps numerous times).

Are you sure you are using svn head?

[titi]

happens to me too:

1. start a game with default setup ( 1 slot open 2 cpus , allow join game )
2. launch the game with noone connected
3. in the running game press pause
4. connect with client

Now the game is unpaused automatically also the server had put it to manual pause state before.

[tomreyn]

What Titi brings up here...

[..] also the server had put it to manual pause state before.

... is another oddity, now listed as (13).

With r4185:

(1) was already fixed
(2) was already fixed
(3) was already fixed
(4) is now fixed
(5) was already fixed
(6) was already fixed
(7) was already fixed
( 8) was already fixed
(9) will need more testing
(10) persists
(11) will need more testing
(12) persists

Previously unreported issues:
(13) When a new client joins a game while it is in progress, the server pauses the game, resumes it, pauses it again, prepares the savegame, has the client download it, and has the client join. Not knowing what is going on under the hood, the step in bold print do not seem to make much sense, or do they?

[tomreyn]

With r4186:

(1) was already fixed
(2) was already fixed
(3) was already fixed
(4) was already fixed
(5) was already fixed
(6) was already fixed
(7) was already fixed
(8) was already fixed
(9) will need more testing
(10) persists
(11) is now fixed (this was actually this bug, unrelated to joining in-progress games)
(12) persists
(13) is now fixed (or I am unable to reproduce this one now)

Previously unreported issues:
(14) It is not possible to rejoin a game, reusing the same CPU player slot, twice. To reproduce: Start a game host with conflict map, player 0 is human, player 3 is CPU, all other slots are closed; activate "allow players to join game in progress" option, start the game. Have a client connect to the lobby of this game and join it, then disconnect the client. Optionally quit and restart the client (keeping the server running). Try to connect the client to the game server's lobby again. It will fail to connect, not providing an error message.

[softcoder]

try rev# 4189 and see if issues 12 and 13 are fixed.

[tomreyn]

Test results with r4189:

About (12):

(12) When a game is paused, and a new client joins, the game resumes automatically after the client has finished the join process. However, the game state (paused in this case) should be retained.

Persists / not fixed. The game still resumes automatically. I think this can be because savegames do not contain this information ("was the game paused when it was saved?").

About (13):

(13) When a new client joins a game while it is in progress, the server pauses the game, resumes it, pauses it again, prepares the savegame, has the client download it, and has the client join. Not knowing what is going on under the hood, the step in bold print do not seem to make much sense, or do they?

I paused the game, then titi joined. The first thing which happened is that the server annouced again that the game was now paused - when it was already paused. But I did not realize it automatically resuming the game before the savegame was taken and titi joined. So it seems to be less of an issue. We only tested this once, though. Note that the behaviour differs between WAN (reproducible) and LAN (not reproducible when I was testing yesterday) games.

[softcoder]

Issue 14 fixed by rev# 4190.

[softcoder]

Please let me know what is remaining here. I had a test where the other player lagged out, rejoined and we finished the game and no out of synch or any other issues.

[tomreyn]

This will need further testing. Just to sum up for myself (and for testing):

The following issues were still present when this was last tested (which is a while ago now):

(10) Clients can go out of synch. How to reproduce this is not clear, yet. It only happens with clients rejoining an in-process game. It took us three or more rejoins of two clients and an attack reaching the CPUs on the opposite side of the map to make it happen.

(12) When a game is paused, and a new client joins, the game resumes automatically after the client has finished the join process. However, the game state (paused in this case) should be retained.

The following issues were probably fixed but lack testing:

(9) Some messages about a player joining an in-process game may be dropped when a player disconnects and quickly reconects. That's messages about a player connecting to the lobby, starting the game join process, entering the game after successful restore of the savegame.

(13) (rewritten) When a new client joins a game while it is in progress, the server pauses the game, resumes it, pauses it again before it prepares the savegame. There seems to be no use in pausing the game twice, so one of these pauses (probably the first one) should be removed. When a player (re-)joins an in-process game which is currently paused, the server may incorrectly announce that the game is now paused - when it was already paused. There is a suspicion that this behaviour differs between WAN (reproducible) and LAN (not reproducible when I was testing back then) games.

[tomreyn]

A bit of an update after further testing:

The following issues were still present when this was last tested (which is a while ago now):

(10) Clients can go out of synch. How to reproduce this is not clear, yet. It only happens with clients rejoining an in-process game. It took us three or more rejoins of two clients and an attack reaching the CPUs on the opposite side of the map to make it happen.

This is better tracked on separate bug reports now. So I'll quit discussing OOS on this topic post.

(12) When a game is paused by a server admin, and a new client joins anytime during this pause, the game resumes automatically after the client has finished the join process. However, the game state (paused in this case) should be retained whether or not new clients join the game, until the admin decides to resume it.

I've reproduced this issue on r4357, and improved its description.

The following issues were probably fixed but lack testing:

(9) Some messages about a player joining an in-process game may be dropped when a player disconnects and quickly reconnects. That's messages about a player connecting to the lobby, starting the game join process, entering the game after successful restore of the savegame.

This one is actually not or not fully fixed. On a recent game (yesterday) where I was hosting and several clients (Filux, Atze, Carolinus if I recall correctly) kept rejoining and disconnecting, the following happened:
Atze disconnected, which was announced, and the next message we read about him was that he was downloading a savegame from the server (to rejoin). Bormally you would have the message about Atze connecting to the lobby to select a slot in between, but there was no such message.

It is not exactly clear how to reproduce this, but playing a game with several connected clients who will disconnect and reconnect within short periods and simultaneously can trigger this and other bugs.

(13) (rewritten) When a new client joins a game while it is in progress, the server pauses the game, resumes it, pauses it again before it prepares the savegame. There seems to be no use in pausing the game twice, so one of these pauses (probably the first one) should be removed. When a player (re-)joins an in-process game which is currently paused, the server may incorrectly announce that the game is now paused - when it was already paused. There is a suspicion that this behaviour differs between WAN (reproducible) and LAN (not reproducible when I was testing back then) games.

This is not reproducible now, at least not exactly as described here, and there exists another bug report for a similar issue, so this should no longer be tracked here.

Summing up, issues 9 and 12 are still relevant.

[softcoder]

#12 fixed in rev 4385

[tomreyn]

Thanks, (12) looks fixed to me, too.

This just leaves us with this:

(9) Some messages about a player joining an in-process game may be dropped when a player disconnects and quickly reconnects. That's messages about a player connecting to the lobby, starting the game join process, entering the game after successful restore of the savegame.

This one is actually not or not fully fixed. On a recent game (1st june) where I was hosting and several clients (Filux, Atze, Carolinus if I recall correctly) kept rejoining and disconnecting, the following happened:
Atze disconnected, which was announced, and the next message we read about him was that he was downloading a savegame from the server (to rejoin). Bormally you would have the message about Atze connecting to the lobby to select a slot in between, but there was no such message.

It is not exactly clear how to reproduce this, but playing a game with several connected clients who will disconnect and reconnect within short periods can simultaneously trigger this and other bugs.