Multiplayer feedback

[martiño]

Hello,

As you might already know, the first Glest multiplayer version has been released:

http://www.glest.org/glest_board/viewtopic.php?t=3051

We are interested in any feedback from you playing it, please don't ask for new features, we know it is really simple at the moment, we are more interested in feedback about stability and bugs.

Martinho.

[@kukac@]

it seems that the whole game caught some "lag". the orders are delayed, sometimes the units forgot the orders i gave them, even if im all alone (against computers).

EDIT:

Crash
Version: v3.0.0-alpha2
Time: Sun Nov 18 17:07:38 2007
Description: Access violation (Writing address 0x0)
Address: 00401A3D

[Darkziel]

well i was trying to host a match but when it's loading it says Error sending socket data (Code:10005)

or something like that

and i was trying to play with kukac xD!

[rabid_psyduck]

A little earlier I loaded up 2 different installs of the same game. I did it to simulate a LAN set-up. Seemed pretty solid, no delays, no nothing. There was also no lag, but that's because of the local nature of the test. I'm going to go up higher in the tech tree and build lots more units to see what happens. I still need to test against a live person, preferably somebody with a cable connection.

I think I already found a problem. I had the 2 games running, the host install was running slightly slower than the other one. On the magic side, regarding the library, I canceled 2 of the research items after starting them. They stayed darkened and showed the text "upgrading". I forgot to keep the game running until the 3rd upgrade finished upgrading. I wanted to see if those research items stayed darkened or not. I also don't remember if it refunded the money or not, sorry.

I am curious why you guys turned off all the building lights. Was it to see if it worsened or lessened any lagging, desync, etc?

Ok, I tried to connect with Kal_Torak. He hosted, then I hosted. It doesn't let us connect. We put the correct IP, but it doesn't even let you press the "connect" button. We opened up that network slot like we're supposed to. I noticed the "not connected" text was blinking, I don't know if that means anything or not.

I use Windows XP professional. Whenever that windows firewall pop-up appears, I always choose to unblock the program. I don't have any other firewalls besides the built-in firewall. I also don't use a router. So I don't know what's up. It SHOULD work, but i'm not complaining, just giving you some feedback.

[martiño]

A little earlier I loaded up 2 different installs of the same game. I did it to simulate a LAN set-up. Seemed pretty solid, no delays, no nothing. There was also no lag, but that's because of the local nature of the test. I'm going to go up higher in the tech tree and build lots more units to see what happens. I still need to test against a live person, preferably somebody with a cable connection.

I think I already found a problem. I had the 2 games running, the host install was running slightly slower than the other one. On the magic side, regarding the library, I canceled 2 of the research items after starting them. They stayed darkened and showed the text "upgrading". I forgot to keep the game running until the 3rd upgrade finished upgrading. I wanted to see if those research items stayed darkened or not. I also don't remember if it refunded the money or not, sorry.

I am curious why you guys turned off all the building lights. Was it to see if it worsened or lessened any lagging, desync, etc?

Ok, I tried to connect with Kal_Torak. He hosted, then I hosted. It doesn't let us connect. We put the correct IP, but it doesn't even let you press the "connect" button. We opened up that network slot like we're supposed to. I noticed the "not connected" text was blinking, I don't know if that means anything or not.

I use Windows XP professional. Whenever that windows firewall pop-up appears, I always choose to unblock the program. I don't have any other firewalls besides the built-in firewall. I also don't use a router. So I don't know what's up. It SHOULD work, but i'm not complaining, just giving you some feedback.

Thanks for the feedback, it is really useful to us, we will keep on improving the version until the next release.

[titi]

more tests here:

2 computers (Linux)
1. Athlon64 3000 ,1GB Ram, PCIE Geforce 7600GS
2. Athlon64 2800 ,500MB Ram, AGP Geforce 4200
connected via switch.
Problems known so far, more feedback soon ( added logging now ):

- If the slower one(2.) plays the server, the whole thing crashes very often ( when starting ...
- there is a lag when giving orders ( even when playing alone on one computer in Multiplayermode )
- sometimes you can't build buildings/units although you have all
  requierements. this rests until end of the game ( also happens when
  playing alone in multiplayer mode with AIs )
- if the old speedup error occures where the game suddenly gets a big
  speedup and it's nearly unplayable than  ( known in all glest versions
  yet  ). the whole thing gets out of sync and crashes after a while..

My boys are "testing" very hard know ;-)

titi

[martiño]

more tests here:

2 computers (Linux)
1. Athlon64 3000 ,1GB Ram, PCIE Geforce 7600GS
2. Athlon64 2800 ,500MB Ram, AGP Geforce 4200
connected via switch.
Problems known so far, more feedback soon ( added logging know ):

- If the slower one(2.) plays the server, the whole thing crashes very often ( when starting ...
- there is a lag when giving orders ( even when playing alone on one computer in Multiplayermode )
- sometimes you can't build buildings/units although you have all
  requierements. this rests until end of the game ( also happens when
  playing alone in multiplayer mode with AIs )
- if the old speedup error occures where the game suddenly gets a big
  speedup and it's nearly unplayable than  ( known in all glest versions
  yet  ). the whole thing gets out of sync and crashes after a while..

My boys are "testing" very hard know ;-)

titi

Hi, your feedback is really useful, thanks a lot, we will work on the issues you talk about and we will try to fix them as soon as possible!!!

[titi]

hi , the logfiles aren't very useful yet:
There are only two  messages (from the client)

Couldn't process event: Error connecting socket (Error: Operation now in progress)
Couldn't process event: Error connecting socket (Error: Operation now in progress)
Couldn't process event: Error connecting socket (Error: Operation now in progress)

Exception: error while receiving socket data (Error: Connection reset by peer)

The first errors happen very often in the game. The last one is from a game where the server gets mad and speeds up as I have never seen before. The client simply crashed. In the servers log you cannot findany hint that something was wrong. When you leave the current (superfast ) game you can start a new one without problems.


Another thing I saw( but I`m not really shure here!) is, when you give two commands very fast after another, the things get mixed up.

and now..... more testing  

[martiño]

The main problem with Glest multiplayer is to keep computers in synch, every machine has to do everything in the same way, so if one of the machines starts running twice as fast everything is going to go wrong.

It would be interesting to know if this happens always after a given period of time since this is probably due to an overflow.

Martiño.

[AF]

Easily fixable by making sure that the game frames aren't calculated until the host says so.

The biggest causes of desync are uninitialized values, mixing of synced and unsynced code, and differences in floating point calculations/accuracy between platforms and compilers.

[titi]

The computers wait for each other at start time.
This works fine if the faster computer is the server, Otherwise sometimes the whole thing crashes.

Today we really played a lot. We had some very long games without any problems and with lots of units.  Two humans vs 2 AIs is our standard at the moment. ( delayed commands but no other problems in these games)

We only had 1 crash game and 2 times the problem that something cannot be build although the requirements were fullfilled. But there is no relation to the time the match/game is running for the requirement bug. I have the feeling this has something to do with mixed up commands/states of the 2 players or missing messages due to synchronisation problems between the fast and the slow computer.

The speed up bug is something different. This happens in single player mode too and it's more likely when there are lot's of units (to animate and display). So if the computer starts to slow down due to hundreds of units it happens quite often ( but really not always !) in the end of a game. You can find the speedup bug in glest 2.0.0 and 2.0.1 too.

@AF: It's really not a good idea to synchronize every frame. you will have a lot of traffic and overhead. But something like synchronized keyframes probabaly can be done.But you'll have to use some more tricks to get not stuck when waiting for server/client. I don't know how it's done, but  there are many ways to do it.

[AF]

You dont want to calculate frame 54 ahead of schedule when the enemy self destructs an important unit on frame 54 do you? After all you don't know that units going to explode yet so your sim will be slightly out of sync. The error is compiunded as one unit roams your game but not the others and the effect snowballs.

[martiño]

In general I'm quite happy about the way multiplayer works in Glest at the moment, basically the clients send their commands to the server, and then every X frames the server broadcasts them so every client executes the orders in the same frame. Seems to work and games don't seem to get out of sync (even if it is yet to be seen if this works accross platforms).

I have a few fixes in my machine that i will upload for next version, like the problem of clients not waiting properly for the server at startup.

Also if you want to reduce latency you can change the NetworkLatency=700 value in the glest.ini (in milliseconds), just make sure to change it in every machine, if not everything will go wrong, i will probably try to detect LAN games anyway by checking the IPs and give them a lower latency.

[titi]

Hello, we found new errors  

- We tested with 3 computers now. One of them( a client ) often was not playable due to extremly delayed movement and very jerkey display.
This was not always the same client, both client had these problems.
When starting the match, only two client sysnchronize well, and one(client) starts without waiting for the server ( usually the notebook).

- Another problem is, when you abort some of your commands, you often don't get your ressources back ( gold stone ...) This also happens with two computers.


The systems:
All linux ubuntu 6.06 dapper. Our usual 2 computers and in addition a notebook with 1GB ram, centrino 1500 and ATI 9700.



Ideas for synchronisation:
I don't know how you synchronize the things, but here are some ideas which probably help you.

I think all commands are send to the server, and the server distributes the commands to the clients.But the problem is that you can not be shure when the commands reaches the client or a client follows the command. When a client is very busy for example, the whole thing gets out of sync.
To prevent this, the commands can be send before they are executed.
The server sends the commands with instructions when to execute this command. This should be a time in the future to make shure all clients can receive the commands. So when the time has come all clients can execute the command (hopefully at the same time.For this the clocks of the systems have to be synchronized before the games starts.

Another synchronisation problem are busy clients or a busy server, to prevent this you can create a wait command which is send to all , which says, that all clients have to reach a special point and wait there until a special time. By this, the fast computers can wait for the slower ones for a given time. The faster ones will have a jerking gameflow too but the whole thing synchronizes again.
To realize this you must have your own frame/tick system which is counted on every client. When there are no delays, Time and frames(ticks)  should be in sync. when there are delays the client will see this and can inform the server to send such a wait  command with a calculated delay to get in sync again.

[AF]

A safer variant that would be slightly slower would be to send the command to the server and not execute it until the server sends the command back. This does mean that there could be a delay between the command showing in the GUI and the command being followed if the server cant keep up.

[martiño]

Glest already works how you guys say, let me explain the command process:

- Commands are always executed in a keyframe, with the default settings must be around about 30 frames.

- Every keyframe the server broadcasts a list of commands to all the clients, even if the list is empty.

- Every keyframe, clients execute the commands given in the last command list they received from the server (the list also includes the frame at which it has to be executed, for sanity checking).

- If a client reaches a keyframe without getting a command list it waits unit it gets one, so all clients always execute the same commands in the same frame.

- When a client orders a command, it sends it to the server, and the server sends the command back in the next keyframe command list. The clients do not execute commands until they get them back from the server, that is why you get some lag.

So as you can see, the system is completely 100% robust and things will never get out of sync because of that. There is still a couple of bugs in the latest uploaded version, like clients not waiting for the server at the start of the game, but the core system is ok.

[titi]

Ok, thanks for the info, this sounds good!
We really wait for the next release! Until than we will go on testing  

Unfortunately this slows down our northman development a bit, but it's so much fun to play it in multiplayer.
Sadly, we loose nearly all games at the moment, but thats why it's still interisting. The indians work perfectly in multiplayer too!

update:
I have one more question: What about slow clients? How can they tell the server that they are busy and get out of sync? Are there any plans how to get the things into sync again?

[martiño]

Next version will be released a couple of weeks after xmas, I'm busy until then, we might put a link to the multiplayer patch in the main site to get more feedback, which is really critical at this stage.

There is no direct mechanism to bring slow clients back in sync, however the game update always runs at 40 frames internally, even if the render can not keep up, thats why the game runs faster sometimes, to keep up with the 40 fps if it has lost some frames, so unless one of the computers is really slow and can not keep up with the 40 fps update (which means that the game is really unplayable anyway, because the render would be about 1 fps) they should not get out of sync. However if I'm wrong I might implement some mechanism for the clients to tell the server to wait for them.

[hailstone]

I tried having a game with two laptops and one pc. They were running two Windows XP and one Vista OS. All are very capable of running games (I don't have the specs but we got a LAN of Supreme Commander going which is fairly resource hungry). The frame rate was very slow to the point of being unplayable.

[jbr]

Really? I had a very powerful vista laptop lan'd with a pentium III and the game played as well as it did when not in multiplayer on both. I did have a little trouble just connecting to the host, though.

[martiño]

I tried having a game with two laptops and one pc. They were running two Windows XP and one Vista OS. All are very capable of running games (I don't have the specs but we got a LAN of Supreme Commander going which is fairly resource hungry). The frame rate was very slow to the point of being unplayable.

That's interesting, there must be something wrong when playing with more that 2 people at the same time. We will investigate.

[hailstone]

Possibly, unfortunately I didn't get the chance to test further.

[lord smith]

when i try to access the multilayer function of the game the game lags\slows down Horrendously. the rest of the time the game runs as smooth as a nut (perfectly)
how would this be fixed  ?

[Jamesgamer]

who were you trying to connect to? if they were far away, that could be the problem.

[titi]

Linux(from source) / Windows have different versions.

http://www.glest.org/glest_board/viewtopic.php?t=3119

did you forget to check in?