Does GAE work in multiplayer ?

[Platyhelminth]

I had tried GAE in multiplayer with Rishen_Windows (Me using WinVista, him probably XP) many time and game crached after few time.

  I tried to play a multiplayer game with the GAE i compiled on linux and i have some crach too. Crach do not  look like to take care what OS and bins players are using. I fact as host i dont crach but i see other players leaving, when i am not host i have crach...

  Is it normal ? Do you think this will be fixed ?

[orion]

I think Daniel.Santos is working on that right now and I think it will get better in the future...which is hopefully soon

[daniel.santos]

Unfortunately, there are a number of problems in GAE multiplayer.  I'm quite certainly that all platform neutrality issues are resolved.  However, I've attempted to use a different multiplayer scheme in GAE than in the original Glest, having the client behave largely as a dumb terminal with client prediction -- this means that the server dictates to the client the positions of units, their stats, etc., but the client attempts to guess that the server will end up telling it so that the game appears less laggy, yet remains exactly in sync with the server.  This was a big change from the original Glest multiplayer scheme and it is not currently working.

If you play a network game using the original Glest engine with two monitors near each other or even two games on the same machine, you will notice after a while into the game that units aren't exactly in sync.  Sometimes, a unit will die on one game, but will be alive on the other, etc.  These are the types of issues I was trying to correct in GAE, but have not gotten it back to a working state yet.

[Duke]

even if it worked there might be an issue:
the player on the server has a slight advantage. Because of the clients guessing the setup they have a slightly distorted impression of the battle, while the server player's impression is always perfect.

[daniel.santos]

even if it worked there might be an issue:
the player on the server has a slight advantage. Because of the clients guessing the setup they have a slightly distorted impression of the battle, while the server player's impression is always perfect.

Actually, this isn't that big of an issue because this advantage is only 100-250 milliseconds at the most (presuming you don't have a really bad connection).  There is also a huge bandwidth problem at the moment (well, IMO, because it won't play well on a 28.8k modem, like I believe it should).  I'm not too worried about the bandwidth problem right now because I'm more concerned with the accuracy and other bugs.  I have a plan to fix the bandwidth problem, but I don't think it's worthwhile until multiplayer is actually working.

[titi]

Im very very happy that you are working on a new multiplayer implementation for glest!

But I see some problems with the bandwidth
When you have 400 units in the game ( this will happen with 4 players! )
you have to distribute the information for every unit .
Let's say this are 100 bytes per unit.
Then you have 40000 bytes ~40 kbytes. If you are the server and you
have to distribute it ( with 3 clients ) you will have to send  3*40 kbytes =120 kbytes.
If you send it every 200ms this willl mean 120kbytes*5=600 kbytes which must be send(upload!) every second. This will be a serious bandwidth problem!


----------------------------
A thing what I thought about is the following:

Everythings happens like in normal glests multiplayer implementation but with the following differences:
- hits / death / birth are "actions" distributed by the server and are not calulated by the client!
- the server constantly sends some kind of "corrections" to the clients. With every command cycle it sends the corrections for some units to the clients. So after a while all units are updated und the updates start with the first units again...

This will ( hopefully ) fix the sync problems.

[hailstone]

If everything is calculated by the server then:
- Every client sends all their information to server
- Server sends only what is needed to each client

I think this will be roughly the same amount of information as sent by the client because the client only needs to know about the areas where their units actually are. For example, player 3 doesn't need to know about a battle occurring between player 1 and 2 unless they are allied and/or can view that part of the field.

Another setup could be to have it like a thin client, where commands are executed on the server and only the immediate viewing area information is sent to the client.

[daniel.santos]

A thing what I thought about is the following:

Everythings happens like in normal glests multiplayer implementation but with the following differences:
- hits / death / birth are "actions" distributed by the server and are not calulated by the client!
- the server constantly sends some kind of "corrections" to the clients. With every command cycle it sends the corrections for some units to the clients. So after a while all units are updated und the updates start with the first units again...

This will ( hopefully ) fix the sync problems.

This is exactly what GAE is doing right now.  I've been fine tuning and tweaking the updates that the server sends, but most of the problems has been internal client consistency (making sure everybody is "in the cells" they are supposed to be in and not in the cells they aren't).  I also have some lag problems because of errors in the way I dealt with some messages, but I haven't started digging into it seriously since my last revamp.  At current, I have 4 unit updates:

• unit created
• unit morphed
• minor update
• major update

The 1st two and the last contain all of the unit's data, but the minor update contains just the hit points, current skill and some other bare data.  I leveraged the existing code for processing XML and "serializing" units to XML, so all data is being sent as XML documents, but getting zlib compressed 1st.  This isn't ideal.  However, I can squeeze the data transmitted down to perhaps 20% of the current just by implementing a static dictionary for the zlib compression (a set of common words in a file that both the client and server has) and perhaps compression can come down even more than 20% (perhaps 5 or 10%).  Just FYI, this is (mostly) how this works right now.  Some of these aren't ideal because I ran into problems:

• worker drops off resources - client predicts (normal Glest behavior), server sends major update
• a worker begins to build a building - client puts unit in "wait for server" skill (which causes it to literally wait until the server overwrites it's skill with another) and server sends "unit created" for new building and major update to builder (I think)
• a unit is damaged - client predicts, server sends minor update
• a unit dies - client predicts, server sends major update
• a dead unit on the client receives a notice from the server that it's still alive - client brings it back to life! (hope you aren't lagging that bad, it looks funny)
• a unit morphs - client enters "wait for server", sever sends "unit morphed"

there are more, but that's the raw concept.

However, the ideal solution (IMO) is

• Alter the messaging protocol to only send the exact fields that are needed in each update (hit points, position, etc.)
• Create a binary message protocol to replace the XML messaging

In practice, using XML for network messaging and using a binary protocol each have their merits; XML messages are very quick to implement, easy to test and are highly maintainable, while binary-based protocols are simply the most efficient.  By being more fine-grained and deciding on an event-by-event basis which fields are needed and which aren't, we can eliminate sending information that isn't needed (instead of having only "minor update" and "major update" to choose from).

But I'm not worried about any of that right now at all.  I've done so many bit/nibble/byte protocols that I'm comfortable with that.  The main problems I've been facing is knowing the right places to "interrupt" (if you will) the process flow Glest and interject this client/server model.  Doing so has caused a good number of upsets, so that's been my challenge.  Once I get this actually working, I'll worry more about optimizing it.

If everything is calculated by the server then:
- Every client sends all their information to server
- Server sends only what is needed to each client

I think this will be roughly the same amount of information as sent by the client because the client only needs to know about the areas where their units actually are. For example, player 3 doesn't need to know about a battle occurring between player 1 and 2 unless they are allied and/or can view that part of the field.

Another setup could be to have it like a thin client, where commands are executed on the server and only the immediate viewing area information is sent to the client.

Actually, the client only sends the commands, the server dictates the rest.  The problem with not sending data related to units not on screen is that the client may think that one of those units is invading their camp, when the server knows that it's doing something else.  This makes things confusing and I think I may need to implement some AI synchronization.

[titi]

I doesn't really understand why there are these kind of problems:

a dead unit on the client receives a notice from the server that it's still alive - client brings it back to life! (hope you aren't lagging that bad, it looks funny)

Why does this happen? If "die" is a command it can only be sent by the server! So this should not happen. On the client side( if the unit is currently out of sync)  you probably get a unit with negative health, but it stays alive as long as the server doesn't send the death command( or a correction ).  So this should solve the problem.

Another way could be that the client sends a death command request to the server. When the server thinks its ok he can give the death command. If its not ok he knows the client is out of sync and sends a correction for this unit(which says its not dead).

[daniel.santos]

Well, there's no "die" command, it just happens when the unit's health reaches zero.  Since the client is attempts to predict the approximate damage a unit will take, it is possible for a unit to die on the client when they are still alive on the server.  Imagine that an archmage nukes unit a, and unit b is nearby.  If the client thinks unit b is 2 steps away from a, he'll take a lot of damage, but the server may have unit b 4 steps away, so he doesn't take as much damage.

I discovered another possible scenario where a unit can die and needs to be brought back that wasn't previously accounted for and is now corrected, so that's one less multiplayer problem.

But most of the time that a unit on the client side dies, it's correct.  If we suspend their state until we get confirmation from the server, then the prediction will get more off as the client will presume that enemies are still attacking him or healers are still trying to heal him.  Even those issues will get synchronized in a matter of seconds, but this causes jumpiness and can lead the player (on the client) to make decisions based upon false information that isn't good, so I think it's better the way it is now.

[@kukac@]

That might be also possible, that the server, and the client thinks that the units took different damage (for example: a unit has 20HP and he recives 10-30 damage [the server counts 13, and the client 28] = while at the client it will die, on the server it will be still alive). Sorry for the many brackets.

[daniel.santos]

That might be also possible, that the server, and the client thinks that the units took different damage (for example: a unit has 20HP and he recives 10-30 damage [the server counts 13, and the client 28] = while at the client it will die, on the server it will be still alive). Sorry for the many brackets.

That is correct.  To help offset this risk, units on the client ignore damage variance when damage calculations are being made.  So if an attack does 10-30 damage, it will always do 20 damage on the client and the server sends a small message that updates the target's health to reflect the actual damage done (which was calculated on the server).

I really need to run some cross-platform tests, but I left the power supply to my laptop at work on Friday by accident, so I don't have a windows machine right now.

[jrepan]

That might be also possible, that the server, and the client thinks that the units took different damage (for example: a unit has 20HP and he recives 10-30 damage [the server counts 13, and the client 28] = while at the client it will die, on the server it will be still alive). Sorry for the many brackets.

That is correct.  To help offset this risk, units on the client ignore damage variance when damage calculations are being made.  So if an attack does 10-30 damage, it will always do 20 damage on the client and the server sends a small message that updates the target's health to reflect the actual damage done (which was calculated on the server).

If I understood correctly, then it can be problem, when unit has 15HP and on the server it gets 10HP damage. Then on the client it dies, but not on the server.

[daniel.santos]

Yes, but that difference generally only exists for a few hundred milliseconds because the server sends a "minor update" to the client and gives it the unit's correct position, health and a few other stats.  If the unit dies on the client and then gets an update from the server that says he's alive, he'll come instantly back to life.  If I haven't mentioned it before, it will be rare that you actually see this happen.

The only real problem I'm seeing with how this is working is that units "pop" from one place to another when they client has them in a different place than the server and they get an update.  I will probably implement two mechanisms to help prevent this from happening.

• First, a mechanism for them to transition quickly from the location where they are on the client to the location that the server tells them they should be.  This will show them moving across the screen visually (but very quickly) and make it easier for the player on the client side to understand where their unit just went, rather than teleporting them directly there, which I personally find quite confusing.
• Second, I'm going to start transmitting the path cache and sending an update to it every time it changes.  The path cache is used by the pathing AI so that it can calculate the best path and then just store it rather than recalculating it constantly.  If the client has this, they units will take the same path on the client as on the server, even if there are other parameters (out of sync units, etc.) that would cause the unit to behave differently otherwise.

In 0.2.8 (not yet released), I've also solved the problem with units on the client of a multiplayer game taking a long time to begin moving after you give them an order.  They will now start moving as soon as you give the order, just as if you were on the server or a single player game.  I don't have access to a windows machine right now (so I can't do cross-platform testing now), but I'm going to resume working on multiplayer issues tomorrow or Wednesday (when I get a windows machine again) and try to get all of the problems solved and perhaps even implement the zlib-static dictionary (previously discussed) to partially absolve bandwidth issues.