running multiple headless servers on one host

[titi]

The new headless server will allow us to run many headless server on one computer.

For this I checked in a first script in the feature/lightweight_headless branch on git in mk/linux/.
This will start 10 headless servers:

Code: [Select]

./startMultipleHeadless.sh 10

A simple way to kill them is:
killall ./megaglest

After doing this I started to think about how to handle such a bunch of running servers.
I think we will need a proper way to shutdown those headless servers if no more game is running.

Here are first ideas, please tell us your opinion:
I thought about a special "stop file" which is placed in .megaglest. If it exists, all idle servers will shutdown. If no game is running a headless server looks for this file from time to time. If it exists he exits.
This file will always be deleted if megaglest starts.

Softcoder pointed out in IRC that there is a service port already for each headless which maybe can be used for this too in some way.

[ctz]

For the stop file, would the server have to poll for its existence?  I guess it's not such a big deal if you make it sleep between checks, and headless server seems to already use a busy loop when it's idle (version 3.9.1; estimated about 4-5% of a CPU core), but it would be useful to eliminate this if you want to run multiple servers on one machine.

I think signal handlers could work too.

So, for example, there could be a signal handler for SIGTERM (or some other signal) that makes the server quit if no game is in progress, or if a game is in progress, exit after the game is over (i.e. set "exit" mode if it is not already set).

Then you can do "killall megaglest", and all idle servers will quit immediately, but other servers will wait until they are finished, then quit.

[Omega]

And the relevant file, for those wondering:

Code: [Select]

#!/bin/bash
portstart=62000
cycles=$1
megaglest=./megaglest
headlessparam="--headless-server-mode=vps"
useports="--use-ports="

 if [[ `echo "$cycles" | grep -E ^[[:digit:]]+$` ]]
 then
   i=0
   while [ $i -lt $cycles ]; do
      port=$[portstart + 1+ $i *11 ]
      statusport=$[port - 1 ]
      echo i=$i port=$port
      cmd="$megaglest $headlessparam $useports$port,$port,$statusport"
      echo $cmd
      $cmd &
      i=$[$i+1]
   done
   exit 0
 else
   echo "Wrong Input usage="
 fi

So basically, it's running ./megaglest --headless-server-mode=vps --use-ports=62001,62001,62000 and so on (ports increase by 12 each loop, starting at portstart + 1).

From the wiki, use-ports does this:

Force hosted games to listen internally on port x, externally on port y, where x is the internal port # on the local machine to listen for connects and y is the external port # on the router/proxy to forward connection from to the internal port number. If enabled the FTP server port numbers will be set to x+1 to x+9.

Unfortunately, the wiki seems to be outdated. I'm not sure what the "statusport" is. Titi, can you please update the wiki?

Anyway, the above is run the input number of times. So you're really just creating multiple instances. I don't think there's any actual changes to the game's source (can you confirm, Titi?). It seems to me that Titi's changes performance changes for headless servers simply makes it more practical to run multiple instances.

[tomreyn]

I'm with ctz, a signal handler is the way to go there.

In addition, you may wantt a governing / master process which

• initially spawns child processes based on a configuration file which primarily defines the amount of child processes to spawn (= init script "start" request)
• keeps track of the child processes' by means of PID / lock files which it creates when it spawns a child process, and removes when a child process exits
• spawns new child processes in regular intervals if previously spawned ones have since exited (and the amount of currently running instances is below the maximum)
• reports, on external request, PIDs and runtime of currently running managed (child) processes (= init script "status" request)
• manages, on external request, the shutdown of these child processes (= init script "stop" request)

In a later stage, it would be nice if you could spawn these servers with custom configurations.

For example, you may want to name these servers individually...

Code: [Select]

SERVERS="alex bob claire diana"
SERVERID=0

for THISSERVER in $SERVERS
do
  SERVERID=$((SERVERID +1))
  cp -pr ~/.megaglest/gameservers/_template/ ~/.megaglest/gameservers/conf/$SERVERID
  sed -i 's/^PlayerName=.*/PlayerName='"$THISSERVER"'/' ~/.megaglest/gameservers/conf/$SERVERID/glestuser.ini
done

Or you could have some servers always start with a given custom game setup (this could be achieved by combining edits to last_headless_game.mgg with a new option --load-lastgame which effectively does the same as a connected client pushing the "reload last game" button) or even be restricted to a given list of mods (this could be achieved with symlinks or bind mounts on Linux), and a lot more...

[titi]

Well the signal handler might be an idea too, but just headless servers should have it.

@Omega: right nothing is changed int he code itself yet, it just gets more likely now to run multiple servers. This is meant for brainstorming how to handle things in future.

@tomreyn: Naming the servers is a good idea too. Ideally we would somehow name each process too,  so we can see the "name" in the process list. By this we can kill by name and we have a relation to the server list.

[tomreyn]

Naming the servers is a good idea too. Ideally we would somehow name each process too,  so we can see the "name" in the process list. By this we can kill by name and we have a relation to the server list.

The proper way to do this is by cross referencing server names via PIDs and a central configuration file. A cheap way to achieve it is to add a command line option which sets PlayerName and use this to set the server title. That way the server name will show up in the ps ef output (since it was passed as an option when the process was created) - that's on Linux (only).