[fixed] 3.8.1-dev, 3.7.1: Out of sync with Prax mod on cross-platform games

[titi]

here are the logs from a game with windws client and linux server:

Code: [Select]

[s]
[url=http://www.titusgames.de/tmp/synclogs.7z]http://www.titusgames.de/tmp/synclogs.7z[/url][/s]

[softcoder]

Please try svn head with this again and send the logs again.

[titi]

Still out of sync with svn rev 4284

Code: [Select]

[s]
http://www.titusgames.de/tmp/sylog.7z[/s]

[titi]

next with more logs rev 4285M  :

Code: [Select]

[s]http://www.titusgames.de/tmp/outofsync.7z[/s]

[titi]

next logs with more logging rev 4287

Code: [Select]

http://www.titusgames.de/tmp/synclogs.7z
btw windows continues build binary still shows overlapping Logos.

[softcoder]

Try rev#: 4288 and repost logs if still out of synch.

[titi]

Its still out of sync, here are the logs:

Code: [Select]

http://www.titusgames.de/tmp/outOfSync_rev4289.7z
We tried it several times now and we modified the techtree to find out whats maybe the reason. And we even tried it without any unit particles, but no luck.

Then we paused the game all 10 seconds and compared carefully client and server. We did this several times and in the end we found this:
A unit "saw" an enemy and tried to attack the posiiotn where it saw the enemy. This resulted in different behaviours on server and client.

Here is the screenshot:

CLIENT / windows:

(click to show/hide)

SERVER / windows:

(click to show/hide)

It looks like the units somehow behave different in windows/linux if they "see" an enemy. ( in general we saw some strange behaviours of units when passing enemy units. Sometimes they don't even react ( ai controlled)  ) ! )

This was/is our game setup we used to test:
http://www.titusgames.de/tmp/lastCustomGameSettings.mgg

[softcoder]

Please try rev# 4290 and send logs again.

[titi]

still the same.
screenshots and logs: 

Code: [Select]

http://www.titusgames.de/tmp/outOfSync_rev4290.7z

[softcoder]

Lets try 4291, i added logging now in unit_updater to see if units are doing different actions.

Thanks

[titi]

game is played comletly different now, but we still see an out of sync.  Out of sync is shown faster now.

screenshots and logs: 

Code: [Select]

http://www.titusgames.de/tmp/outOfSync_rev4292.7z
The screenshots show the same scene in the same Moment on server/client ( taking screeshots while game is puased )

[titi]

logs for rev4294, still out of sync:

Code: [Select]

http://www.titusgames.de/tmp/outOfSync_rev4294.7z

[softcoder]

I played on winxp vs lunix 64 host and two times could not get out of synch. Both games the client lagged out after 10 minutes of play, but i compared the logs and they were exactly the same.

Now i suspect maybe in your tests the client and server had experimental ini settings that were different? I disabled most of the settings i could find to hard code them to see if that fixes this issue for you titi, please try svn head and let me know how it goes?

Thanks

[tomreyn]

I tried to reproduce this on r4296, Ubuntu 13.04 x86_64 server with the file from above loaded, and Windows 7 - 32 bit client. The first time I ran it up to frame 20,000 and was unable to reproduce.

The second time MG crashed and produced a DMP file which I analyzed and posted to

Code: [Select]

http://megaglest.nopaste.dk/p44723 (the actual stack trace starts at line 428).

I'm not sure whether this is related - it may just be a result of the imperfect 3D "virtualization" in VirtualBox 4.2.12 (which used to be better in earlier releases) - FPS is very low there and I have to be careful not to time out due to that.

On a side note: To rule out configuration differences between your client and server, you could use the command line argument to dump the configuration on both systems, then run those through diff.

[tomreyn]

Here are the latest

Code: [Select]

[url=http://pepper.freeit.org/glest/logs/megaglest_r4300_oos_2013-05-21.tar.xz]logs from a game on r4300[/url] we played yesterday.

And here's a copy of the README file I included in this archive:

A game of MegaGlest on r4300 (3.8-dev) - going out of synch.

Participants:
* tomreyn: host on Ubuntu 13.04 x86_64
* nig: client on Windows 7 64-bit
* carolinus: client on Windows (probably Windows 7 64-bit)

All parties were running a build of r4300.

What happened?
* tomreyn hosted, both clients joined, the game was started
* after less than 10 minutes, carolinus dropped out due to lag (his hardware could not cope with both writing the log file and sending timely responses to the server)
* after another 10 (?) minutes, nig went out of synch

Carolinus' logs may therefore seem irrelevant, but should allow to verify that every party was seeing the same game state initially.

The following files are included for every participant:
* DebugWorldSynch log file
* glest.ini used in this game
* glestuser.ini used in this game

[softcoder]

Please test with rev 4302, i may have found and fixed a long standing out of synch bug. Report back after testing.

[tomreyn]

We just tested on r4304 twice. I was hosting on Linux x86_64, Jammy was client on Windows 7-64, on the second game Atze (Linux x86_64 (?)) joined us.

Both times Jammy went OOS, but it was not before the log file grew to 3.0 GB (game #1) and 3.9 GB (game #2) respectively. Sadly that's above Jammy's quota, even compressed, so he won't be able to provide world synch logs (which we did create) - it may be possible to ask Jammy to grep for stuff, though.

My logs are available for download:

Code: [Select]

[url=http://tomreyn.megaglest.org/SynchLin64_TomReynolds_r4304_game1.log.xz]game #1[/url], [url=http://tomreyn.megaglest.org/SynchLin64_TomReynolds_r4304_game2.log.xz]game #2[/url].

[atze]

If the logfile (ClientSynchLin64.log - game #2) is needed from me, it can be found here: http://www.sendspace.com/file/3vfza3

[softcoder]

Please try svn rev#: 4307 as it has some more fixes and more logging to see how the command's progress is different between platforms. Please test and report back.

[nig]

I brought my SVN-Building back to work and played scenario egypt megahem with 4307M today to check. I used the old ini-files ( tomreyn posted on: 22 May 2013, 17:18:58) so DebugWorldSynch was set to true.

Everything worked fine. After 12 min FPS went down to 2. I stopped the game and saw that the logfile had grown to 4Gb and filled my logging-Ram-Disc completly.
I turned off logging and played again, everything worked fine, until I lost again.

I turned logging on again and watched the size of logfile during playing the scenario again. Again FPS went down to 2 fps. This time I exited the game immediatly and saw that the logfile was only about 3 GB.

I switch the logging to HDD again.  74GB of free space there and played again. There happend the same! After about 12 min FPS went down. Logfile was about 5Gb.

So maybe its now just too much logging for windows and all systems go down?

compressed logs ar bigger then 200MB each. If you still want them, say something.

Greets
nig

7z made it possible. logs are here and there.

[tomreyn]

Thanks for your efforts, Nig.

I'm afraid this time not a lot was achieved regarding the bug this thread is about, though: An out-of-synch (OOS) situation which occurs between different platforms (a network game involving both Linux and Windows are known to allow for reproducing it) when playing a network game using the Prax mod. The plan to identify the underlying issue is currently to record the game state in regular intervals on both participating systems (that's what the debug logs are for) and, as soon as the OOS occurs, to compare each of the lines in those logs, to identify differences, which should then allow for better guesses on which part of the source code may cause them. Obviously this approach can only work when logs of a game with this setup (>=2 computers, one running Linux and one running Windows, playing a network game with the Prax mod) are available.

Based on what you wrote, I assume you were testing something on a single computer. If this is so, then, while the experiences you made can be of interest, they do relate tho this thread and should be posted on the main forums with a subject like "experiences while testing debugging on Windows".

Once again, your efforts and intent to support development are appreciated. I'm just explaining this in detail so that you will be able to spend your valuable time on tests which are actually adding to identifying and fixing the most relevant issues.

[softcoder]

Please try svn rev#: 4311 i have switched to useing integers from some important calculations which may resolve this.

[tomreyn]

So thanks to lots of patience in development and some in testing this seems to be fixed now. :-)
Congratulations!

Now we just need to think about what to do about floats in the future...