Multiplayer Discussion Kickoff Thread

[lodemia]

First, an introduction...  I'm fairly new to Glest (about three months of use) and I absolutely love the game.  But I really miss multiplayer capabilities.  So much that I want to start developing them.

I've seen all the other threads on Multiplayer, and they end up being begging threads.  I started thgis thread to discuss the challenges and possible solutions for developing a glest multiplayer game.  I've downloaded the code and browsed it, and I must say, it's truly an example of good programming style. Kudos to the team.

I've done a lot of server programming (I work for a software firm), however never for games.  So while this is a first for me, I'm already somewhat familiar with what will need to be done.

Before I set out to startr writing code, I'd like to find out if there are already any projects underway to provide this functionality.  I pulled the latest files from the CVS and there's no real multiplayer support in them as of yet.

From what I've seen.  Some of the challenges:

1.  Glest is fairly extensible.  This means that the factions must be the same across all the clients connecting to the server. Need either a way to enforce this or a way to synchronize the client machines with the factions on the server.  I think that the server would be the limiting agent for the factions, but certainly open to ideas here.  I can't see starting a game and having to wait 20 minutes while all the 3d models are synched.  Having said that, I do consider it a novel idea to "baseline across all connected players."

2.  Glest is fairly open.  This means that theoretically, someone could decrease the resource requirements for units and cheat.  This is similar to the above, except that this is probably easier to control.  Problem being that the definition for the object is in the same XML file as the graphics definition.  The easy fix for this is to load from filesystem, and if you connect to a server, then the server overrides those values.

3.  How to propigate the events?  From my light reading, this is the most troublesome part of network play, especially when you move beyond LAN parties (because of bandwidth).  For RTS games, all the changes from all the objects from all the members must be synched every tick.  The question becomes how to handle change requests.  Does the change get propigated to the server, and sent to all the clients at the same time, or is the change done first locally, then the server is told what happened, and the server propigates remotely?  Apparently, in the game development world, it's split as to how to handle this.  I'm leaning toward all the change vectors (move requests, attack requests, build requests, etc.) go to the server, then sent to all clients (including the requesting client) at the same time,  In my opinion, that's the best way to ensure that the game is synchronized across ticks.

Pseudocode:

Loop
Raise EVENT -> Send to Server
Server Acknowledge EVENT -> Broadcast to all connected users (and AIs)
Users and AIs receive event -> Perform EVENT
end loop


4.  Network optimization.  Lots of data to synch.  Won't touch this until 1,2,3 above are designed.  Might need real data to see if this is a relevant concern.  I presume it is, so I at least want to develop with that in mind.


Other topics for consideration?  Any other feedback?

For what it's worth, I pulled the source for dark oberon and looked at the network  algorithms there, and I have a feel for what needs to be done.  This is quite an intrusive change, and basically every event that is currently developed would have to be redeveloped to include network play.  I'm still thinking a lot of this through, and I might code up some stuff in the next few weeks as a sample.  SDL has excellent network capabilities built in, and since that's already in use for the glest project,. I'll start there.

Also, as a warning, I use Linux.  So if I get it done, it will be done on Linux first.  I'll make every event to be platform independant, and SDL should definitely help there, but I thought the disclaimer was appropriate.

Please, please let's keep this as a discussion of HOW it should work.

Geoff

[martiño]

Hi, I've just read your post, and all I can say is that I wish you luck and that you will have all our support.

[VIN100]

This is how I see the concept, there are many others way to do :
(Note that I didn't read the code to much before writing this, so maybe it doesn't totally fit with the inner programmation of the game)

EDIT :
Ok, I wrote this and then I understand the post of Lodemia.  I first understood that he wanted to synchronized every information of every object at each tick. And then realize that he was talking about synchronizing the events. Thus, what I wrote is not totally related to his post.  I still think that there is interresting things in my text

In addition : I think that synchronisation of the important datas of each object is important. It can be done at a large interval of time (some seconds). This synch is important because of the gap between the speed of the server process/speed of the clients process and network ping time.

Also, each interactions between server and client should use "prediction" of movement. For instance (it's just an arbitrary example), if a client receive a synchronisation information about unit X at position (10,10,0), with a deplacement  of (2,0,0)/seconds, and that the "gap" between the server and the client is 0.5 seconds, the client have to position X at position (11,10,0). This involve keeping track of the mean gap between the client and the server.

------------------------------------------------------------------------------------------------------

I don't think that every client must know every information on every opponent at every thicks.

I mean, if the server is totalirian, it can only receive the events of every client, "play" the game in parallel of the clients, and resynch them sometimes (in this case, a smooth synch function must be implemented on the client side to not have jumping units).
For the multiplayer purpose, I think that a client only have to know what his opponent do outside of the "fog of war". So the server must evaluate the fog of war of each player, and send synch informations on the units/projectiles of the opponents and events of those to the client only if they are in his "line of sight".

Well, I know that the problem with this method is that events like construction and creation of units are nearly never sended (if only based on the fog of war). So, when a player will, as instance, charge the city of his enemy, the client software will receive a big amount of data about the units and buildings to suddenly be spawned on the map to be synchronised with the server. That cause hundreds of objects beeing created at the same moment, which load the cpu and memory and cause biiig lag on the client. :?

[@kukac@]

what will be those, who changed the XML files, so they have more damage and health?

1. I could imagane, if the multiplayer works only with the original Glest settings.
2. I could imagane, if the players download the servers settings (maybe so much downloads, and the Glest directory will be full with downloaded settings). It can cause problem, if the server uses some stupid settings (example: workers are the much stronger then any other), and who dont know this, that may suck with it....

anyway, i would glad, if you could make this multiplayer mod!

[lodemia]

Well, the actual data for each unit isn't that much, except for the 3d stuff.  I suppose we could have the server send back the "cost" of each production or build, so that the server is still the source of truth.  This way, it's monitored at the event level.  I dunno how much data that would be.  If you look at some of the networking libraries, they have compression support built-in, so perhaps we should start with that off the bat.  Still thinking it through, although I do have some of it coded, just as a proof of concept.  Busting cheats and leveling factions across players is going to be the hardest part, IMO.

[martiño]

Well, the actual data for each unit isn't that much, except for the 3d stuff.  I suppose we could have the server send back the "cost" of each production or build, so that the server is still the source of truth.  This way, it's monitored at the event level.  I dunno how much data that would be.  If you look at some of the networking libraries, they have compression support built-in, so perhaps we should start with that off the bat.  Still thinking it through, although I do have some of it coded, just as a proof of concept.  Busting cheats and leveling factions across players is going to be the hardest part, IMO.

I think the best way of doing this is just creating a checksum of all the data and sending it...

[lodemia]

Well, I'm quickly leaning towards object replication as a means for synchronizing data.  Basically at any unit action, replicate the object state across the network.  This is possibly more network intensive, but it creates a more continuous flowing game.  I could have the Network synch be a part of the unit constructor.  The neat part is that if it's done this way, the overhead on the server is quite minimal, there's no real computation being done.  The server just receives a request, and tells the clients to render the unit according to the new object description.

This can also be used for a cheatbuster, because since the object is network replicated, then the unit derives its values from the server.  

Previous, I had thought about only sending the deltas, but then you could possibly have ordering issues, and it's possible that a client doesn't receive an update or a render properly.  While trying to minimize the round-trips to the server, it's also important to have a smooth and very playable game.  We probably don't have to worry about TOO much bandwidth for the simple fact that glest won't be a Massive Multiplayer game.

Still thinking it through and developing some samples.  I'll post some code when my thoughts are more firm.

[@kukac@]

2 things:

will be this multiplayer mod will compatible with the other versions?

will be this mod be in the next game update?

[lodemia]

No, it would be impossible to make the multiplayer mod backward compatible.  There is far too much code to modify. Every movement, every action, every build event must be modified.  It is very extensive.  


This doesn't count the menu changes necessary to "host" or "join" a game.  It will be a completely different build when it's complete.


As far as releases, I don't know.  I am doing this merely for my own pleasure, and the fact that it's open source.  I'm not an official developer on the team.  I will make the code available when it's in a playable state.

Geoff

[@kukac@]

i meant on the forward patches not the previouses

[Yeeha]

I hope u can do it , and i hope glest team will take it into official glest. Good Luck!

[artworx]

Hi guys, i really like this game and i thought I'll comment on some of the ideas presented here.

First if the game runs on the server and clients just render requests from the server. The main disadvantage i see here is that it would need significant bandwidth and CPU power from the server. Packet synchronization could be a problem here.
This way of a central server doing all the computations has the advantage that that it kind of breaks all cheats.

The second way, of the clients doing the processing and syncing with the server at regular intervals would make for a much smoother game-play on a low bandwidth connection. Of course, one has to write a very clever scheme for the synchronization, but the lower bandwidth and smaller CPU usage on the server makes up for it.
But this way can be exploited very easily. Just getting the unit strength from the server is not enough, because what would prohibit the client from changing the source of the game? Since the game also runs on Linux many people compile the sources for their own machines so the binary might change. Also providing an official binary would solve this, but then again binary compatibility is not the best thing for Linux.

The second approach could smooth out network problems(The way UT does it. If i killed the other guy on my machine, when i sync with the server he is dead).

I hope you can find some good information in my post. I like this game and I'd really like to see a multiplayer option in it soon

I have some other projects for now so i don't really have any time to help with the coding, perhaps after the summer.

[graphain]

Hi, I just ran into this game today and I have to say I'm very impressed, more so than anything at the Glest team's (great) decision to keep everything open source and highly moddable - it makes for both an improvable game and a dedicated community in my books.  Anyway on to my thoughts on multiplayer - the main "un-moddable" aspect missing from Glest from what I've seen in my first quick look at the game.

Before I begin I must admit I do not pretend to know a lot about network coding but I have a fair interest in computing and I'm studying a Computer Science degree which gives me the most basic underlying understanding of what's going on.

Without looking at the code, would it be possible to somehow develop code that "intercepts" the command process between commands delivered and the system's processes executing?  For example: User clicks to move unit "Archer" - Send Network Command To Server - Update Local "Archer" Position, or would this be even more time-consuming/impossible compared to the rewrite of all event code?

Secondly, would it be possible to parse (I think this is the correct term) a copy of the Server's configuration files prior to the game start which the Client references for all system-based rendering?  In this manner the Server is the system with the correct reading of everything that is happening but provided the Client files remain unedited during gameplay they too will see an accurate rendition of the game.  

Cheating would be an interesting issue here as the Client would think they have successfully cheated yet the Server and all other Client's would be confused as to what is happening at the very least (the cheating Client's actions would seem bizarre) or a disconnection (if coded) / crash might occur when the cheating Client sends an event request to the Server that is impossible (say requesting a move of "Super-Mighty Panda Dragon" of which the Server does not understand to exist as an object at all, or in the very least as a currently created object).  Perhaps in the situation of a disagreement between a Client and the Server the Server can first try a synchronisation between all Clients to see if perhaps the error is Server-side before disconnecting the cheating Client (Obviously poses issues for 1v1 games, however either way the Server and Client are out of sync and one must disconnect).  Can anyone think of a situation where a difference in sync would not result from cheating/file editing?  

One possible solution that comes to mind is parsing all initial configurations (excluding models presuming model "hitboxes" are based on configured parameters not taken from the loading of the model) into memory, in this manner all but the most dedicated cheats would have difficulty altering the configuration.  Again the disconnection is the only manner I see to deal with errored (cheating) requests of the Server.  

With regular syncing what would in fact be synced? All prices/current upgrades/unit positions/stats etc.?  Because I can only imagine this being a network intensive task that will result in "glitching" on cheating Clients when there units jump across the screen to the position the Server says they should be.  Furthermore, what happens if the Server cheats?  

If through my ramblings there is a coherent direction it is that I see the best solution being one of assumption of no-cheating with validation occuring upon each request of whether said request is valid.  In this manner cheating might be prevented in that if a user requests to create an object they send this command to the Server, the Server reduces the local-based resources of that player and creates the object then sends the command to the Client to reduce their own resources by that amount and create the object.  I think on reflection this is essentially the first proposal by artworx.  I don't see how syncing becomes a viable option in the case of cheating - in essence it would surely be more CPU and network intensive than Client requests because the syncing would involve comparing every object at one time rather than comparing each object when necessary - resulting in "lag spikes" if my elementary understanding of the situation is correct.

Anyway if something was of use here I'm impressed and good luck with your networking efforts.

//PS I do think that the multiplayer lobby should detail if the Server has non-default settings although I can only see this working in an environment where there is no CD (as this is a non-commercial game) if the lobby says non-default for settings differing from the Client's own "default" config copy (seperate from their actual copy).  In this manner the information is only useful for Client's who have left their default copies unedited.

[@kukac@]

one question: will this mod support the clans?

[@kukac@]

where can i send my application for beta test?  

[metroid]

Couldn't you just make multiplayer versions of the factions and make those permanent and not able to be changed? Or you could check prior to the game if the files are the same, and if they're not, then have a listing of the changed files and agree/disagree on each part? Never mind, that's kind of complex.

Multiplayer would be nice, but I'd rather be able to make mods and maps easily. Imagine: there could be 10 some mod factions someday!

[prometheus]

hiho@ll

just wanted to ask, if there is already some multiplayer code?
if yes, where can i download and maybe help developing?

thx@ll

[pa33]

Hi,

  I'd like to first say thanks for starting this effort. There are some good questions raised and I thought I'd share some thoughts on cheat-stopping. I do like the idea of the server handling all unit creation / building creation stuff since the client can't be trusted. We wouldn't want the client to tell the server that it all of a sudden has 1000 horsemen and the server to say ok. The client should send a request to the server "build a horseman" and the server should evaluate how many resources the client has etc and wait the right time and then tell the client "you have a new horseman".

  That being said, unfortunately there may be some cheat situations which can't be handled by the server. For example, automated fighting. A modified client could move his forces automatically when it senses an enemy presence and fight the battle without any human intervention - if the cheater is good, he might devise algorithms to fight battles more efficiently than a human could manually - like moving forces out of the way right before having the dragons fire and then moving the forces back in immediately - stuff that is very difficult for a human to do efficiently. Or coordinating many different types of units all at the same time which a human might have trouble with.

  This type of cheating would be very difficult to detect I think. Of course other games have problems with the same type of thing. For example in open source FPS games like quake, aimbots are always a problem and the server can't do much except possibly to monitor how accurate the clients are over some long period of time and then flag them as a bot if they're too accurate. But then of course the bot makers can just make it miss sometimes etc.

  Quake never really came up with a good solution to this problem. Another game with similar issues is "Nettrek". I dunno if the windows people know about this one, but its an open-source space shooter game designed for multiplayer - each player controls a spaceship and they destroy each other and take over planets etc. The cheat issue there is a client can be modified to fire his ship with 100% accuracy. Nettrek came up with an interesting solution to the problem - they continued to release the full source code to the client, but they would release "official" client binaries with an encryption key hardcoded into the binary that is sent to the "official" servers on connection and any client without the right key is rejected from the game.

  So in other words nettrek came up with a closed-source method of controlling cheaters - anyone without a "blessed" binary is considered a cheater and not allowed to connect. Unfortunately I believe this violates the spirit of free software and so I'd really like glest to avoid this approach. But I cannot offer a good solution to the problem either.

  Thats all I have to say on the subject of cheating for now.

  Have you given thought to what network protocol to use? I'm no expert but UDP seems to be preferred over TCP for network games to speed things up since you don't have to do the handshakes and stuff to send data.

  Good luck and keep the discussion going!

[AsTheCrowFlies]

Looked a while to the code today,...

without reading the posts before,...

I've got some idea,...


According to the code these so called "Events" are "Tasks" in the code like in the AI Section,... Probably there would be a possibility to develope the multiplayer mode like a pseudo-ai game,... just that the other players force a "AI"-Task via Network,...


Perhaps this would be a silly idea cause of some other reasons but i just thought you should hear about my idea,....

[lodemia]

Update from my front:  Soon after kicking off this thread, I'va taken a new job with my work, and it has required me to work a whole lot more than I expected.  I had hoped that I would be able to write code in hotel rooms and on the planes, but I'm much busier than I thought, averaging about 80 hours a week or so.  To give you an idea, I haven't even played glest for over 3 weeks.  

To start with the development, I implemented the save game feature for release 2.0.  I figured that would give me a good feel for the flow of the code, and how best to make the modifications.

I added some small modifications like when you hit the escape button it asks if you want to save, exit, or return, rather than the F3 (or whatever).  I also changed it so that you could load a game from the main menu, rather than having to start a game, then load it.

I also made a small modification that changes an ally to an enemy if that ally is attacked.  That was always an annoying thing, I could attack my ally and they just stood there.

Here's generally how I had planned on coding it:  All units would be "AI" like unless it was my unit.  Basically, the AI commands would be queued up for that unit.  The server would send absolutes for ewach AI, like the position of this unit is now x,y or etc.  I had also intended on using raknet after researching other network libraries.  Raknet can get you up and running quickly, and there are a lot of good examples and tutorials.  I did some rudimentary tests (outside of glest) and it works great.  There's even a way to sync local and remote directories, for sending mods to clients.  Lots of good features.

Did all my development on Linux.  Unsure how portable it is.

There are some things I never quite completed, like a text entry box.  It displays, but I can't edit the text, etc.  I haven't touched the code for 2 months.

If someone else wants to pick up the pieces and work on it, I'd be glad to make the code available.  I'm on the road this week, so it would be this weekend before I can actually package up the code and upload it.

If you look at the dark-oberon source code, you'll notice that it is a lot like glest, and it's been a real help, in terms of how would another RTS developer accomplish something.

Hope this helps -

Geoff

[toph]

great work, if you can get multiplayer to work in glest the engine would be one of the best open-source-rts-engines out there !

[pidhash]

I and other guy are started a new glest based project called glevolution, it still not a really evolution , but we could to fix bugs and improve it more fast than glest is going, for now I am working in a glevol_multiplayer a branch of this.
If anyone interested contact me http://sourceforge.net/projects/glevolution

[_untitled]

 Another game with similar issues is "Nettrek" ... they would release "official" client binaries with an encryption key hardcoded into the binary that is sent to the "official" servers on connection and any client without the right key is rejected from the game...

 I was just looking around, and I'd like to have fair multiplayer also.
How about instead of locking the whole binary, just create the multiplayer as a separate file, which might be launched from the main menu. This file could be closed.

[@kukac@]

well, will be ever anything with the multiplayer mod?

[oandarilho01]

Any news 'bout the multiplayer development? Hope you do it, guys.. it'll be kind of crazy..
Glest is really cool and deserves respect..