Performance improvement for Glest multiplayer... 4 player is now practical

[titi]

Feedback: No problems this time! We had an online two player game without problems and a very smooth gameplay!

I hope I will find some more players to test it, please come to play  ....

[Omega]

At 9:00 AM boardtime we meet in the irc [/glow]

BTW: What is 'board time'? Is it UTC?

[titi]

I dont know...

But I'm there now for the next 2 hours.

[softcoder]

I'll try to get a Windows binary compiled using my VirtualBox XP Image once we verify that the network code is working (all changes are also in the windows code so it should be easy to produce a working binary).

P.S. sorry Titi we cannot help with testing today as I am working (PST is our timezone)

[titi]

ok  :'( one day we will mange to play together ( on a weekend maybe ).
Building  abinary is possible with virtual PC, but Is multiplayer also possible with virtual PC? Its not wine , so maybe it works because it works completly different ???
But I'm shure if we( or more you in this case ) come up with something worth building, silnarm will quickly build a binary too !

[softcoder]

Actually it "should" be possible to build using linux and mingw. I am able to use codeblocks to compile a tool I wrote for my router that compiles to my routers linux (busybox - Tomato firmware), ubuntu and windows.

[Super Tom]

If you make four people online play, will result in server-side or client-side crash??

[titi]

Currently it results in NO CRASH , thats whats it all about!

[Super Tom]

Find a new BUG:

In the non-networking mode, and I carried out the computer's AI Battle to the first 15 minutes, I was ready to press the ESC key to exit the game, but the game does not quit! I would like to ask you: do you also find this issue at this issue How do I fix it???

[silnarm]

Hi Guys.

  I briefly tried to compile the megaglest 3.2.4 package on windows earlier this evening, I introduced a few typedefs and two defines to get rid of most of the problems, but I wanted to test something, so I've left it for the moment. I'm happy to save you the trouble of setting up a windows build for Code::Blocks, but it may have to wait until the weekend.

  I believe I have found the ultimate cause of the problems with online play though. Hailstone wants some sort of stable multiplayer for GAE 0.2.13, and with Daniel MIA again, he has set about reverting our multiplayer code to that of 3.2.2 for the time being. We have a problem though, clients are lagging badly, getting worse and worse as the gmae continues, while more keyframe updates arrive, until the socket's buffer is full and it rejects new data and the server disconnects it.

  This is not nice news for us, I'm confident we'll find the problem, but for now it alludes us, however it is something of a blessing in disguise... In looking at this problem I've had a good look around the client/server interfaces and how they work with the game.

  For online play with TCP, the client(s) are doomed to experience some lag, unavoidable.  However, the game never stops... The server never waits for clients once the game has begun, if a client arrives at a keyframe and it hasn't received the update for that keyframe then it waits.  On the server, the game continues, 'flat chat'.  On a local network this is unlikely to be a problem, but once we send our packets out 'into the wild', there will be losses, TCP will sort it out for us, and deliver it all in the right order, but clients will start to lag.

  The more problems 'along the way' the further it will lag, the worse the game experience will be for the client, and eventually the socket's buffer will fill, and disconnection will follow.

  I can't absolutely confirm now, I was just trying to test a modified glest 3.2.2 (just logging the amount of data remaining to be read on the client after processing a key frame) with PolitikerNEU but we had router/firewall issues.  But the code says what the code says, the server does not wait (ever), and the client occasionally does wait, it can only ever be the same or get worse, never better.

  Two solutions are feasible, 'server wait' and/or 'client catch-up', ultimately I think both is ideal, but for now giving the server a maximum 'lead time' and requiring 'Acks' from clients for each keyframe, and the server pausing if it doesn't have acks from all clients for, say, 3 or 4 key frames before where the server is at (subject to a timeout and potential player ejection of course).

  This would be the easiest, and while it may result in choppy play sometimes, it should prevent any clients falling so far behind that they end up will a full buffer and disconnection.

  I think with what you have done so far, and a little bit of logic to make the server a little more polite, this could all turn out very well indeed 

  Hope to get some good testing time in this weekend, IRC folks!

[softcoder]

What you say is totally true. However I disagree with the solution. My phase #1 approach at changing the code is just an attempt to properly understand the network code. I have already posted about the frame issue as a problem in previous posts. The goal for now was to try to get the existing code to work "better" so that multiplayer has greater potential in the short term.

With that said I believe a better approach for the long term (which many other games use) is to run the whole network engine in background threads on both client and server (and thus avoid choppy GUI) and from those threads we would be able to better synch network packets in the GUI. Each packet ALREADY carries the frame # which is essential, and we could have the threads queue up the data and pass it to the GUI at the right time in both server and clients. This is what I had previously called phase #2 which would ultimately remove the synch problem for good and open the door for perhaps > 4 player in the future.

I have already experimented with enet (a common network engine used in games written by one of the authors I know who works on Sauerbraten) but it is of no use for phase #1 do to its design.

Thanks

[titi]

Some thoughts ( maybe not correct ):
Silnarm wrotes that the client sends messages to the server for every keyframe. I think thats not true,
its not frames, its something different, call it network frames which represent a bunch of game frames.
And you are right the server continues no matter what happens, but the clients have to wait if the server package is missing. If the clients doesn't receive the things fast enough its maybe a solution to rise the network framesize dynamically. By this the reaction for commands will be delayed a bit more, but the game doesn't get choppy . I think this is a solution for very bad connections  with big ping times.
Rising and lowering these network framesizes is something that should be done by the server, but the info if rising/lowering is needed is something only the clients know. So they probably have to inform the server about their receive "times" based to the moment where the infos are needed.

A typical client info would be:
received commands 0.10 seconds before needed ( -10 )
or
received commands 0.05 seconds too late ( 5 )

Based on these infos the server can calculate the best network framesize which is send to the clients with the next command package.

====================

By the way, we played another 4 player local LAN game without any problems!

[softcoder]

Regardless of the solution it will be very important to separate network communication into threads (both on server and client). Otherwise we have major problems related to FPS on each computer playing a network game, which will slow down the network transmit rates AND the network connection will also add additional problems if the connection is slower.

By creating threads we can manage the congestion in a separate manner from the UI. Things still need to be synchronized, but how we do it can be less visible to each system as we handle it in the threads.

This needs to be carefully planned and designed well to take glest to the next level for multi-player features.

Recap:

The current network code slows down the FPS AND the FPS will slow down the network communication. This connection needs to be addressed.

Thanks

[titi]

crash again  :'( :'(

client crashed in an onlinegame:

Code: [Select]

...
In [glest_game/network/network_message.cpp::receive] dataSize = 904
In [glest_game/network/network_interface.cpp::getNextMessageType] socket->getDataToRead() dataSize = 92
In [glest_game/network/network_interface.cpp::getNextMessageType] socket->getDataToRead() iPeek = 1, messageType = 5
In [glest_game/network/network_interface.cpp::getNextMessageType] socket->getDataToRead() dataSize = 92
In [glest_game/network/network_interface.cpp::getNextMessageType] socket->getDataToRead() iPeek = 1, messageType = 5
In [glest_game/network/network_interface.cpp::receiveMessage]
In [glest_game/network/network_message.cpp::receive] socket->getDataToRead() returned 92
In [glest_game/network/network_interface.cpp::receiveMessage]
In [glest_game/network/network_message.cpp::receive] dataSize = 904
In [glest_game/network/network_interface.cpp::getNextMessageType] socket->getDataToRead() dataSize = 590
In [glest_game/network/network_interface.cpp::getNextMessageType] socket->getDataToRead() iPeek = 1, messageType = 5
In [glest_game/network/network_interface.cpp::getNextMessageType] socket->getDataToRead() dataSize = 590
In [glest_game/network/network_interface.cpp::getNextMessageType] socket->getDataToRead() iPeek = 1, messageType = 5
In [glest_game/network/network_interface.cpp::receiveMessage]
In [glest_game/network/network_message.cpp::receive] socket->getDataToRead() returned 590
In [glest_game/network/network_interface.cpp::receiveMessage]
In [glest_game/network/network_message.cpp::receive] dataSize = 904
In [glest_game/network/network_interface.cpp::getNextMessageType] socket->getDataToRead() dataSize = 1808
In [glest_game/network/network_interface.cpp::getNextMessageType] socket->getDataToRead() iPeek = 1, messageType = 5
In [glest_game/network/network_interface.cpp::getNextMessageType] socket->getDataToRead() dataSize = 1808
In [glest_game/network/network_interface.cpp::getNextMessageType] socket->getDataToRead() iPeek = 1, messageType = 5
In [glest_game/network/network_interface.cpp::receiveMessage]
In [glest_game/network/network_message.cpp::receive] dataSize = 904
In [glest_game/network/network_interface.cpp::getNextMessageType] socket->getDataToRead() dataSize = 904
In [glest_game/network/network_interface.cpp::getNextMessageType] socket->getDataToRead() iPeek = 1, messageType = 5
In [glest_game/network/network_interface.cpp::getNextMessageType] socket->getDataToRead() dataSize = 904
In [glest_game/network/network_interface.cpp::getNextMessageType] socket->getDataToRead() iPeek = 1, messageType = 5
In [glest_game/network/network_interface.cpp::receiveMessage]
In [glest_game/network/network_message.cpp::receive] dataSize = 904
In [shared_lib/sources/platform/posix/socket.cpp::disconnectSocket] START closing socket = 18...
In [shared_lib/sources/platform/posix/socket.cpp::disconnectSocket] END closing socket = -1...
[shared_lib/sources/platform/posix/socket.cpp::isConnected] DISCONNECTED SOCKET error while peeking isconnected socket data, err = 0, errno = 11 [Resource temporarily unavailable]
Exception: Disconnected
Game
World
Minimap
Tech tree
Tileset
Cells

server:

Code: [Select]

...

In [glest_game/network/server_interface.cpp::broadcastMessage] before sendMessage
In [shared_lib/sources/platform/posix/socket.cpp::send] sock = 16, bytesSent = 904
In [glest_game/network/server_interface.cpp::update] socketTriggeredList.size() = 1
In [glest_game/network/server_interface.cpp::update] socketTriggeredList.size() = 1
In [glest_game/network/server_interface.cpp::update] socketTriggeredList.size() = 1
In [glest_game/network/server_interface.cpp::update] socketTriggeredList.size() = 1
In [glest_game/network/server_interface.cpp::update] socketTriggeredList.size() = 1
In [glest_game/network/server_interface.cpp::update] socketTriggeredList.size() = 1
In [glest_game/network/server_interface.cpp::update] socketTriggeredList.size() = 1
In [glest_game/network/server_interface.cpp::update] socketTriggeredList.size() = 1
In [glest_game/network/server_interface.cpp::update] socketTriggeredList.size() = 1
In [glest_game/network/server_interface.cpp::update] socketTriggeredList.size() = 1
In [glest_game/network/server_interface.cpp::broadcastMessage] before sendMessage
In [shared_lib/sources/platform/posix/socket.cpp::send] sock = 16, bytesSent = 904
In [glest_game/network/server_interface.cpp::update] socketTriggeredList.size() = 1
In [glest_game/network/server_interface.cpp::update] socketTriggeredList.size() = 1
In [glest_game/network/server_interface.cpp::update] socketTriggeredList.size() = 1
In [glest_game/network/server_interface.cpp::update] socketTriggeredList.size() = 1
In [glest_game/network/server_interface.cpp::update] socketTriggeredList.size() = 1
In [glest_game/network/server_interface.cpp::update] socketTriggeredList.size() = 1
In [glest_game/network/server_interface.cpp::update] socketTriggeredList.size() = 1
In [glest_game/network/server_interface.cpp::update] socketTriggeredList.size() = 1
In [shared_lib/sources/platform/posix/socket.cpp::hasDataToRead] select detected data imaxsocket = 16...
In [hasDataToRead] FD_ISSET true for socket 16...
In [shared_lib/sources/platform/posix/socket.cpp::hasDataToRead] socketTriggeredList->size() = 1
In [glest_game/network/server_interface.cpp::update] hasData == true
In [glest_game/network/server_interface.cpp::update] hasData == true
In [glest_game/network/server_interface.cpp::update] socketTriggeredList[i] = 1
In [glest_game/network/server_interface.cpp::update] calling slots[i]->update() for slots[i]->getSocket()->getSocketId() = 16
In [glest_game/network/network_interface.cpp::getNextMessageType] socket->getDataToRead() dataSize = 904
In [glest_game/network/network_interface.cpp::getNextMessageType] socket->getDataToRead() iPeek = 1, messageType = 5
In [glest_game/network/connection_slot.cpp::update] got nmtCommandList
In [glest_game/network/network_interface.cpp::receiveMessage]
In [glest_game/network/network_message.cpp::receive] dataSize = 904
In [glest_game/network/server_interface.cpp::update] calling connectionSlot->getNextMessageType() for slots[i]->getSocket()->getSocketId() = 16
In [glest_game/network/server_interface.cpp::update] socketTriggeredList.size() = 1
In [glest_game/network/server_interface.cpp::update] socketTriggeredList.size() = 1
In [glest_game/network/server_interface.cpp::broadcastMessage] before sendMessage
In [shared_lib/sources/platform/posix/socket.cpp::send] sock = 16, bytesSent = 904
In [glest_game/network/server_interface.cpp::update] socketTriggeredList.size() = 1
In [glest_game/network/server_interface.cpp::update] socketTriggeredList.size() = 1
In [glest_game/network/server_interface.cpp::update] socketTriggeredList.size() = 1
In [glest_game/network/server_interface.cpp::update] socketTriggeredList.size() = 1
In [glest_game/network/server_interface.cpp::update] socketTriggeredList.size() = 1
In [glest_game/network/server_interface.cpp::update] socketTriggeredList.size() = 1
In [glest_game/network/server_interface.cpp::update] socketTriggeredList.size() = 1
In [glest_game/network/server_interface.cpp::update] socketTriggeredList.size() = 1
In [glest_game/network/server_interface.cpp::update] socketTriggeredList.size() = 1
In [glest_game/network/server_interface.cpp::update] socketTriggeredList.size() = 1
In [glest_game/network/server_interface.cpp::broadcastMessage] before sendMessage
In [shared_lib/sources/platform/posix/socket.cpp::send] sock = 16, bytesSent = 904

....


[silnarm]

Regardless of the solution it will be very important to separate network communication into threads (both on server and client). Otherwise we have major problems related to FPS on each computer playing a network game, which will slow down the network transmit rates AND the network connection will also add additional problems if the connection is slower.

 While I agree that running the network code in its own thread is a very good idea, and will indeed help, perhaps to the point where > 4 players is feasible on a LAN, it in no way addresses the lag problem for online play. When playing over the internet, there will be packet loss, there will be delays.  If the client doesn't try to catch up when the pipes unclog, or the server wait for client acknowledgements, the client will fall further behind.  As I mentioned, this is a 'one way street' currently.

 If a network thread is reading all messages as they arrive, this will solve the disconnection problem when/if the socket's buffer fills, but by this time the client will be (assuming an 8Kb buffer) 10 keyframes (100 frames, ~2.5 sec) behind + the usual network delay. The game will be 'unplayable'.

[softcoder]

My mistake on the crash, one line of code used the wrong variable, a fix will be ready shortly.

[softcoder]

Updates at:

mega glest - http://soft-haus.com/glest/code/megaglest-source-3.2.4-3-beta1.7z
regular glest patch - http://soft-haus.com/glest/code/glest_patch_2010_jan_28.diff

Thanks

(the only change was one line of code to socket.cpp for posix and windows)

[softcoder]

Hello silnarm:

In reply to your post about network congestion. In general if a client cannot AT LEAST keep up to the # packets generated each "group" of FPS then they cannot play. I think we could use something like Titi mentioned, an algorithm that the server could calculate to determine if the client will be able to play or not.

The threading would help in many cases since it would allow for clients to "catch up" without creating choppiness (at least I believe there is a way). This would only work in the case where clients have a good enough pipe on their network connection. You need:

1) good enough FPS (video card has something to do with that)
2) good enough network connection / traffic should be reasonably low

and... we should make sure the network packet sizes used in glest are absolute minimum so as to not require to much chattiness over the network.

Anyways more discussion once we get the current situation reasonably under control. We may hit a threshold that we cannot do better with the current framework, but as of yet that is not the case.

Thanks

[Super Tom]

root@tom:/usr/local/games/glest# ./glest.bin

#2 Error connecting socket for IP: 192.168.1.3 for Port: 61357 err = -1 errno = 115 [Operation now in progress]EINPROGRESS in connect() - selecting

Please tell me how to solve this problem??   Thanks

By the way: I use the operating system is: ubuntu8.10

[softcoder]

Tom, where did you get glest.bin from, the auto-installer built into Ubuntu? If so you need to build your own version to get the fixes we have been implementing until someone is able to produce a binary for Ubuntu 8.10

Thanks

[Super Tom]

Hello, softcoder, I am now using the same version you have!

I first of all from this address: http://www.liflg.org/?catid=6&gameid=87 download and install glest.3.2.2.bin. And then use your offer megaglest3.2.4-1-beta1, conducting LAN Battle, when such problems!

By the way: you can test connectivity when using the client side, the problem occurs! I installed the game up like this: I am in a terminal implementation of the. / Glest.bin

My game is installed in / usr / local / glest /

My brother also used the same method to install! My brother use systems are also ubuntu8.10
I forgot to tell you, you provide the code inside an already-compiled binaries, I now use is this!
I hope you can resolve the problems, thank you!

[Super Tom]

There is this question: In the non-networking mode, and I carried out the computer's AI Battle to the first 15 minutes, I was ready to press the ESC key to exit the game, but the game does not quit!

Please amend this bug!  Thank you!

[silnarm]

 My mistake, Glest does in fact already do 'client catch-up'. You won't find it in the networking code anywhere, it is built into the main loop (and the timers it uses, specifically). If it doesn't get 40 frames in per second, it will get them in sometime later. GAE was not always catching up, hence our problems.

 On another note,

   //delay the start a bit, so clients have nore room to get messages
   sleep(GameConstants::networkExtraLatency);

Have you experimented with lowering this? 200 ms seems seems to me like it may be excessive, there is the normal network delay aswell (from the ready message from the server), this delay will 'scale' naturally with the connection type, on a LAN a delay of even 10 ms will probably suffice, and lead to a much more responsive experience for the clients, most of their commands will come back on the very next keyframe, rather than two keyframes into the future (as seems typical from my testing).

Probably a better number might be 50 ms, even over the net I think this may be sufficient, with the natural delay added in.

Cheers.

[titi]

Thats exactly what I talked about, but this number is ideally not fixed but calculated and dynamically in the game.
For online gaming USA/Germany maybe a number of 500ms is better, but in local LAN you want 30ms.....
But that's future stuff.

Just want to mention that there is also another mechanism what glest does to catch up if something took too long. It simply speeds the game up dramatically for a second! I think this is not only used in multiplayer but in singleplayer too.

By the way, I think there is an error in the current glest with this mechanism. I often saw this on slow machines for myself where glest suddenly speeds up ( because something was too slow I think ) but then it stays in ultra speed and doesn't returns to normal speed...
I think this is the reason for posts like these for example:

https://forum.megaglest.org/index.php?topic=4896.0
https://forum.megaglest.org/index.php?topic=3934.0
https://forum.megaglest.org/index.php?topic=4404.0

[silnarm]

Just want to mention that there is also another mechanism what glest does to catch up if something took too long. It simply speeds the game up dramatically for a second! I think this is not only used in multiplayer but in singleplayer too.

By the way, I think there is an error in the current glest with this mechanism. I often saw this on slow machines for myself where glest suddenly speeds up ( because something was too slow I think ) but then it stays in ultra speed and doesn't returns to normal speed...
I think this is the reason for posts like these for example:

Yeah, it's the same mechanism that keeps multiplayer clients at (or about) the same delay the game started at. It's Program::updateTimer that does this, the game runs at set speed in frames (subject to game speed of course) and it will never change that number, if it doesn't process that number in any given second, any additional frames will be 'queued and ready to go'.  When/if whatever was holding the system up goes away, the game will go into super speed to catch up, single or multi player.