Author Topic: how to make headless server much less cpu/memory demanding  (Read 2428 times)

titi

  • MegaGlest Team
  • Airship
  • ********
  • Posts: 4,239
    • View Profile
    • http://www.titusgames.de
how to make headless server much less cpu/memory demanding
« on: 18 September 2013, 20:52:11 »
A big problem is that we currently don't have enough headless servers. One of the reasons is the fact that our headless servers really need a lot of memory and cpu usage while running. This is, because a headless server needs to calculate the whole game too while playing. My idea is to reduce this a lot...

What about the idea that the masterserver just collects and distributes the messages as a normal server does it, but it does not calculate the AI? The AI is controlled by the "controller" so the client which is currently "the boss".
By this the whole server has no more need to know anything about the map and the whole gamestate! No path calculation no damage calculatio, no unit updates, nothing must be done!
While playing he just collects the commands of all clients and distributes them like a normal headless would do it. Just this one controlling client calculates the AI and sends commands for the AI . For disconnecting clients maybe we need to add some special behaviour, but thats all. Before the game starts (game setup in menu ) all is as it is now on a headless,

What do you think ?

« Last Edit: 4 November 2014, 08:52:31 by titi »
Try Megaglest! Improved Engine / New factions / New tilesets / New maps / New scenarios

tomreyn

  • MegaGlest Team
  • Airship
  • ********
  • Posts: 2,764
    • View Profile
    • MegaGlest - the free and open source cross platform 3D real-time strategy game
Re: how to make headless server much less cpu/memory demanding
« Reply #1 on: 18 September 2013, 23:54:32 »
This sounds pretty awesome. I hope it's doable.
atibox: Ryzen 1800X (8 cores @3.6GHz), 32 GB RAM, MSI Radeon RX 580 Gaming X 8G, PCI subsystem ID [1462:3417], (Radeon RX 580 chipset, POLARIS10) @3440x1440; latest stable Ubuntu release, (open source) radeon (amdgpu) / mesa video driver
atibox (old): Core2Quad Q9400 (4 cores @2.66GHz), 8 GB RAM, XFX HD-467X-DDF2, PCI subsystem ID [1682:2931], (Radeon HD 4670, RV730 XT) @1680x1050; latest stable Ubuntu release, (open source) radeon / mesa video driver
notebook: HP envy13d020ng
internet access: VDSL2+

· · · How YOU can contribute to MG · Latest development snapshot · How to build yourself · Megapack techtree · Currently hosted MG games · · ·

Coldfusionstorm

  • Golem
  • ******
  • Posts: 868
    • View Profile
Re: how to make headless server much less cpu/memory demanding
« Reply #2 on: 13 October 2013, 19:46:18 »
Sounds very interesting, i say YAY.(im currently working on game mechanics for my game so i would suggest some new abilities but i wont before im relatively sure i will need said feature). So go ahead. :).
WiP Game developer.
I do danish translations.
"i break stuff"

titi

  • MegaGlest Team
  • Airship
  • ********
  • Posts: 4,239
    • View Profile
    • http://www.titusgames.de
Re: how to make headless server much less cpu/memory demanding
« Reply #3 on: 7 February 2014, 10:23:17 »
I have this running for me at home now and it works VERY good so far.  The headless servers resource needs are down:
For example a game with 700 Units in the field:

old headless: more than 500MB Ram usage and  200% CPU usage
new headless:  20 MB Ram and 8% CPU usage!

My current question is, should we keep the old headless mode too? ( so do I add a switch that allows old and new behaviour? )


Problems I still have/see:
- The endgame stats are not know on the headless server, just on the headless controlling client. I must find a way to send them to the Masterserver.

- Slow controlling clients or controlling client with bad network:
If the headless controlling client is slow, the AI is slow too. So if this client falls back, the AI is commanding slow too.

Does anyone see any other problem I might have missed with this?

Git: How can I check this ( big ) change in, in a proper way? I think I want a branch for this to try it with others.
Try Megaglest! Improved Engine / New factions / New tilesets / New maps / New scenarios

Coldfusionstorm

  • Golem
  • ******
  • Posts: 868
    • View Profile
Re: how to make headless server much less cpu/memory demanding
« Reply #4 on: 7 February 2014, 15:33:21 »
But dosn't this put more load on clients?
WiP Game developer.
I do danish translations.
"i break stuff"

Julius

  • Summoner
  • **
  • Posts: 46
    • View Profile
Re: how to make headless server much less cpu/memory demanding
« Reply #5 on: 7 February 2014, 20:19:47 »
On the boss client, but most gaming PCs are way overpowered for Megaglest anyways. Keeping the full server option would be good though.

Maybe there is some way for show in the game lobby which client is likely the fastest and let the players choose accordingly before the game starts?

What I wonder is if one could not just do away with the headless server all to gether like other RTS do, e.g. start a game which gets registered at the masterserver and then P2P connect with the other players, while the person who started the game is like the boss client.

Alternatively one could maybe reimplement the thin server in java or so in order to have it run on a simple apache tomcat server or some similar easy to set-up and cheaply available shared hosting service.
« Last Edit: 7 February 2014, 20:53:11 by Julius »

Julius

  • Summoner
  • **
  • Posts: 46
    • View Profile
Re: how to make headless server much less cpu/memory demanding
« Reply #6 on: 7 February 2014, 20:23:55 »
Cool would also be if it was possible to run the server on a service like RedHat's OpenShift: https://www.openshift.com/ which scales dynamically. E.g. as long as the server isn't used or only few players play it is actually free and only if more power is needed it automatically scales up.

I guess python could be a good programming language to get such a thin client on OpenShift small gears?

Omega

  • MegaGlest Team
  • Dragon
  • ********
  • Posts: 6,167
  • Professional bug writer
    • View Profile
    • Personal site
Re: how to make headless server much less cpu/memory demanding
« Reply #7 on: 8 February 2014, 02:12:46 »
To make sure I understand correctly, only the AI computations have been moved to the controller's computer, right? It doesn't affect the performance of other players? And I would also assume that the AI computations are no worse than running a single player game with the same AI?

If that's the case, I would say remove the old functionality entirely. No need to clutter the source code with near-obsolete functionality. Surely the expectation that one player has the resources akin to playing a single player game is not at all unreasonable?

Quote
Git: How can I check this ( big ) change in, in a proper way? I think I want a branch for this to try it with others.

I notice that most of MG's development seems to go into the main branch right now. Have you considered adopting a more pro-branching strategy? Branching works really well in git. Unlike SVN, a branch merely needs to store the difference from the previous revisions (SVN would usually make a whole freaking copy). Git also focuses on a "working directory" model. You work on a current branch. Thus, it's very easy to do some work in a feature branch, then switch to a bug fix branch for something that needs your attention more before switching back to what you were doing -- all without breaking your current work or creating new copies of everything.

I found this to be an interesting read, back when I was learning git.

Regarding distribution, everyone will just have to checkout your branch as normal (git clone -b <branch> https://github.com/MegaGlest/megaglest-source.git). It'd probably be best to distribute executables, however, if you want people to test.

If the tester already has the repo cloned, they can just run git pull; git checkout <branch>. That will switch their current working directory (so they'll need to do git checkout master when they're done).
« Last Edit: 8 February 2014, 02:18:12 by Omega »
Edit the MegaGlest wiki: http://docs.megaglest.org/

My personal projects: http://github.com/KatrinaHoffert

Julius

  • Summoner
  • **
  • Posts: 46
    • View Profile
Re: how to make headless server much less cpu/memory demanding
« Reply #8 on: 8 February 2014, 15:45:33 »
On a LAN with old PCs the full headless server option would be usefull, and in internet play with strangers also, as there is no way to prevent that "boss client" from cheating.

titi

  • MegaGlest Team
  • Airship
  • ********
  • Posts: 4,239
    • View Profile
    • http://www.titusgames.de
Re: how to make headless server much less cpu/memory demanding
« Reply #9 on: 9 February 2014, 02:03:45 »
A first version can be tried/found now in the "feature/lightweight_headless" branch
Try Megaglest! Improved Engine / New factions / New tilesets / New maps / New scenarios

titi

  • MegaGlest Team
  • Airship
  • ********
  • Posts: 4,239
    • View Profile
    • http://www.titusgames.de
Re: how to make headless server much less cpu/memory demanding
« Reply #10 on: 10 February 2014, 10:20:02 »
The new headless works now in general, so please test it.

I still have trouble with the game stats which are send to the masterserver.
The stats infos we need are a bit splitted now. Some infos are on the headless server and some are on the clients. The headless for example has knowledge about when players left and so on and the "admin"-Client has the rest of the info.
What I don't want is, that the client sends stats to the headless server while the game is running because this may be bad if some people have slow connections and suddenly need to publish to the masterser too. This might affect the gameplay for everyone in the end.  So while the game is running we just get the info which is hold on the headless server.

Another thing is, what should be done in the end of the game?
I think the client should send all data he has and the headless server should send all his knowledge. This should sum up in the correct stats on the masterserver.

I bet we need to change the php files for this, so they only update those columns which are send with the request keeping the original columns untouched.( I did not look into this yet )
Try Megaglest! Improved Engine / New factions / New tilesets / New maps / New scenarios

titi

  • MegaGlest Team
  • Airship
  • ********
  • Posts: 4,239
    • View Profile
    • http://www.titusgames.de
Current state:

As long as I "just" switched the controlling client it looked like all was going well and I played some games without any problems.
The problems started when I was trying to send the stats data from the client to the headless server ( because he did not play the game anymore and by this has no info about whats going on ).

In general all works now and the stats are send form the controlling client to the headless server as I wanted it. The headless server publishes them to the master-server and you can see the things on play.mg .
When the controlling client quits the game, the new controlling client takes over and he is sending stats-data to the headless server too. But somehow then after some minutes I get either a crash ( with an obviously  corrupt stack trace ) or the game gets OOS.

I have no idea what I did wrong and don't know how to find this bug :-(

update: Willvarfars comments showed me that it was not clear what is crashing. The  crash/false behaviour happens on the (new) controlling client.

 
« Last Edit: 10 March 2014, 14:09:46 by titi »
Try Megaglest! Improved Engine / New factions / New tilesets / New maps / New scenarios

will

  • Golem
  • ******
  • Posts: 783
    • View Profile
Run the headless server under valgrind?

tomreyn

  • MegaGlest Team
  • Airship
  • ********
  • Posts: 2,764
    • View Profile
    • MegaGlest - the free and open source cross platform 3D real-time strategy game
This would be my suggestion, too, but chances are this will slow the headless server down so much that there will be network timeouts rather quickly. It may still be worth a try, though.

Another option would be for someone else to review the code. Softcoder has the best understanding of the networking code, but doesn't have any spare time available these days. We may just need to wait until this changes - which is fine, since this is not time critical at all, and thanks to feature branches we can just start to work on something else in the meantime.
atibox: Ryzen 1800X (8 cores @3.6GHz), 32 GB RAM, MSI Radeon RX 580 Gaming X 8G, PCI subsystem ID [1462:3417], (Radeon RX 580 chipset, POLARIS10) @3440x1440; latest stable Ubuntu release, (open source) radeon (amdgpu) / mesa video driver
atibox (old): Core2Quad Q9400 (4 cores @2.66GHz), 8 GB RAM, XFX HD-467X-DDF2, PCI subsystem ID [1682:2931], (Radeon HD 4670, RV730 XT) @1680x1050; latest stable Ubuntu release, (open source) radeon / mesa video driver
notebook: HP envy13d020ng
internet access: VDSL2+

· · · How YOU can contribute to MG · Latest development snapshot · How to build yourself · Megapack techtree · Currently hosted MG games · · ·

titi

  • MegaGlest Team
  • Airship
  • ********
  • Posts: 4,239
    • View Profile
    • http://www.titusgames.de
the problem in this case is not the headless server. The problem is the client which gets the new controlling client.
Try Megaglest! Improved Engine / New factions / New tilesets / New maps / New scenarios

titi

  • MegaGlest Team
  • Airship
  • ********
  • Posts: 4,239
    • View Profile
    • http://www.titusgames.de
Well I fixed some suspicious things now, but I still had trouble in one test game:

We played with 3 player and the controlling client in slot of player 2 ( blue ) lagged out. My son (red team ) was the next chosen controlling client and took over.
5 minutes later we got his Out Of sync message screenshot( from titis computer):
(click to show/hide)

Code: [Select]
Console output titi ( team green  ):
http://pastebin.com/raw.php?i=us1VPEuK

Console output titi_son ( team red )
http://pastebin.com/raw.php?i=fFn1KjdK

Do you have any idea why the AI might give a command for a unit in another faction ? Or what else can cause this ?
« Last Edit: 3 April 2016, 14:05:59 by filux »
Try Megaglest! Improved Engine / New factions / New tilesets / New maps / New scenarios

titi

  • MegaGlest Team
  • Airship
  • ********
  • Posts: 4,239
    • View Profile
    • http://www.titusgames.de
Re: how to make headless server much less cpu/memory demanding
« Reply #16 on: 4 November 2014, 08:57:37 »
I made a one line fix in git and now it looks like it works
( I played 2 games without problems ).

Please test the feature/lightweight_headless branch.
Try Megaglest! Improved Engine / New factions / New tilesets / New maps / New scenarios

titi

  • MegaGlest Team
  • Airship
  • ********
  • Posts: 4,239
    • View Profile
    • http://www.titusgames.de
Re: how to make headless server much less cpu/memory demanding
« Reply #17 on: 6 November 2014, 00:24:39 »
works much better now, but we got a OOS again in one game:

http://pastebin.ubuntu.com/8843487/
http://paste.ubuntu.com/8843442/
Try Megaglest! Improved Engine / New factions / New tilesets / New maps / New scenarios

titi

  • MegaGlest Team
  • Airship
  • ********
  • Posts: 4,239
    • View Profile
    • http://www.titusgames.de
Re: how to make headless server much less cpu/memory demanding
« Reply #18 on: 9 November 2014, 01:38:23 »
maybe I fixed the OOS. Please try latest git version 02ee59518c139fdf8059440493de934dba4d32a2
Try Megaglest! Improved Engine / New factions / New tilesets / New maps / New scenarios