faction references ( or other ideas to make this work )

[titi]

At the moment there are many discussions about installable factions. This sounds easy, but it is not regarding multiplayer!

As an example lets use the japanese mod. Lets asume the modder wants others to play the japanese faction vs the magapack factions, how can he do this?
1. He can copy his faction into the megapack.
2. he can copy the whole megapack and add his faction. ( this is the only way at the moment )
3. he can setup his own techtree and use something called a "faction reference" which is something new to invent and simply holds some kind of reference to the faction.
4. use physfs and mount points... ( I never used it yet, but this might be an option as far as I understand it )

No. 1. is a bad idea, because this will make the techtree different on server and client and users will see warnings/errors in multiplayer.
At least the main MG data/techtree should stay "ready to play" without warnings and work for everyone without problems in multiplayer.

No. 2. works, but is big and updates to the referenced factions don't have an effect ( which might on the other hand be good too )

No. 3  has the same problem as 1 in general, but if only references to the megapack factions are allowed, we can be shure the megapack faction is there adn don't have the trouble. But this will not allow two mods to reference each other any more...

No. 4 I am not shure here, but I think it has the same problem as 1 or at least the problem described in 3.

Keep in mind:
In general only faction which are desigend for the same techtree should be shared. It makes no sense to mix in a faction which cannot live in the new techtree environment.

Please post your ideas regarding this! At the moment I think only a very restrictive form of No3 can work. All others cause too much trouble for normal players, as they cause different data warnings for server and client.

(@wilvarfar this is what you wanted to talk about   ! )

[will]

Thank you titi for making this topic!  I think its very important.

Imagine first the premise that everyone uses a mod packing tool that comes bundled with the game.  It is intended that programmers and the more technical modders do understand how it works under the hood, and making super scripts that wrap it is only to be encouraged etc.  But the intention is that the normal modder does not need to understand how the system works.

The other side is that the mod should be an archive pretty much as it is now.

On this stategic thing hinges a lot of the tactic technical choices that follow, so which way do you want to go?

[titi]

I don't see how packaging affects this decision too much. Packaging is something nice too, but the current problem is more how to reference factions and how to deal with multiplayer and users having different data.

Why and how does packaging affect this decision?

[will]

because many - maybe even most - AAA games use hashes to identify files rather than names.

Indeed, the new MG sync code takes this approach too.

Normally, a file has three defining attributes in this order: size, hash (crc is just a low-bit equiv) and finally, weakly, name.

Some systems go further and track hashes of blocks in files, but that seems too far.

A packaging tool can easy handle this properly; whereas a human would be quite lost and prone to mistakes.  If you have humans hand-making zips to give people, then they need to use filenames only, and then you don't get versioning and we don't so easily get de-dupe.

[titi]

Yes yes, but any idea on the problem with different data of server and clients?

Some more thoughts on this:
At the moment i think only references to the base install ( typically the megapack ) can be allowed. Or is there a way all needed dependecies are installed too? I mean we have no package system where we can setup these dependecies....

An example for dependencies:
Imagine the japanese mod has a reference to the imperial faction and to the indian faction of the megapack.
In case you install the japanese faction fromm the mod download page you will also need to install the imperial faction to make things work.
You see when we allow references to other techtrees than the megapack, we will need an automated installation of all dependend techtrees.
I don't want to let the normal player deal with this kind of dependencies manually.

A good start might be the following:
As we control all entries of mods in the mod download center we can enshure that only mods with references to the megapack are allowed for the moment. So technically we allow links to all other techtrees, but in the download center we only allow those with references to the megapack.

[will]

A file is identified by its CRC and size.  You also have the name.

We can view the list of all published mods ever as being a big hash-table.  So imagine a mod is really a list of these CRCs, sizes, filenames and a place online to get them.  A mod is basically like a dropbox file!

We don't need to actually download other mods in entirety.

If people use the ZIP format, you can - using HTTP range requests - peak inside those ZIPs and get the list of filenames, CRCs and sizes without downloading the whole thing.  And then you can use more range requests to download the individual files in ZIPs without downloading the whole file.  (Or we can have the same effect with a custom format or index.)

So all duplication on a file level is understood and de-duped by all clients regardless of how the techtree is organised.  If Mr War includes tech and magic in imperial, all the users still only have tech and magic exactly once.  And if Mr War edited tech to give it boats and things, then they'd have the original tech and his additions, rather than two copies of the original tech.  Because changes are tracked at the file level, not the mod level.

Finally you can also always retrieve any given file from the owner of the game, via the server.

So if I have modified a particular piece of artwork in a faction, the server can work this out because of CRCs and byte-counts not matching, and other clients can then try and retrieve the file from a central list of CRCs and sizes, or from the game owner.

So simply playing a game with a change 'publishes' that change to the other participants; very useful for work-in-progress.  And clients effectively download a delta, which is efficient.  But we can make exact versions of techtrees synced this way as 'official' or otherwise and can control whether creating new games with them requires checking boxes so that, by default, only the 'sanctioned' released mods are shown to players.

You can imagine all this going into a sqlite DB on the client, or onto the filesystem with the crc appended to the filename.  All this suggests that normal players cannot easily tinker in these cache folders; they are managed by the game.

There's a lot more nuance to this idea of gossip-based distribution, and its a well studied subject.  Lots of fault-tolerant cloud DBs use approaches that you can draw parallels with.  Its interesting how trendy this approach is, and people interested in behind-the-scenes at dropbox will see similarities there too.

Another approach that I like too is for everyone to be using github but that being hidden from them.  But lets focus on the gossip-approach first.

[titi]

I think you start to mix things again. Partly dowloads of only needed things, compression of mods are nice but not the main things we talk about here.

But lets regard another part of your ideas the central list of crcs:
This is a interisting idea, but how and where can we host all versions(releases) of all mods for all time ( and for free )?
And what about copyright problems which might show up in some of the mods? Will these mods suddenly disappear?

[will]

I think you start to mix things again. Partly dowloads of only needed things, compression of mods are nice but not the main things we talk about here.

I think that if you take the quick fix of simply adding a <dependency> tag to the xml somewhere and expecting the mod maker to fill it in correctly, then you're missing the big opportunity and it won't be robust or used as much or as well as we'd like

But lets regard another part of your ideas the central list of crcs:
This is a interisting idea, but how and where can we host all versions(releases) of all mods for all time ( and for free )?

Well you can't.  The central problem of finding free hosting is unsolvable (although github - at least until it goes the way of geocities - offers hope).

But if the file is not available on a central server, its still available on the player who starts the game (and chooses the techtree from the many techtrees/versions they has locally).  So if someone's hosting died, the mod could still be played gossip-style from player to player.  Grass-roots replication.

And it may be that some people don't ever need to actually host their mods at all - this p2p approach will serve their needs well enough.

Basically, right now all the files are CRCed by the players when they join a game.  (Its important to also track size.)  But if they don't match, then the whole mod downloaded from somewhere.  Go back a step.  If they don't match, the individual sizes/CRCs themselves needed by the players can be compared.  There are even ways to do this O(lg(n)) but that seems overkill - the list of files to be loaded and their metadata is very small, and run through ZLIB on the fly its even smaller.  Only the files that differ need be downloaded.  That download can go from classic hosting or over the game server link.  This is not such a massive leap as it may sound.

It also covers de-dupe inside the mod.

Having written a packing tool, I know empirically that the biggest impact of download size of existing mods is de-dupe of whole-files within a single mod.  The major compression advantage of (solid) 7z over zip or tgz evaporates and they are much closer.

And what about copyright problems which might show up in some of the mods? Will these mods suddenly disappear?

It would be tempting to add a license tag as meta-data to each file.  Perfectly doable.  And of course the absence of a tag can infer something about the rights too.  You might be able to insist, for example, that you never download a file that is not explicitly tagged with a license you accept.  And of course the tools for creating new content may insist that the person starting the game state the content type the first time it is encountered.  Etc.


To make it clear, the existing file structure can be preserved.  I would advocate a stricter naming convention the encoded information about the version of the mod and its techtree in the name of the root folder for it, but that's not strictly necessary.  All the duplication and copying of files is only on the authors machine.

And in the game, on the mod menu, they can select a techtree that is changed locally, and hit the 'package' button.  And they get a nice zip or other file which contains no duplicate files, and instead symbolic links and such.  (My mod packing tools actually rewrite the xml inside the archive to point to single instances of files etc.)

Or they can distribute it P2P simply by playing with others.  But if they get it hosted somewhere and it gets into the master-list as already in MG, then people can play it without needing to play with someone who already has it, epidemic-like.

[titi]

I just wanted to mention that we already have download of techtrees/maps/tilesets from one player to another ( but techtrees are big!! ).

But you obviously have the downloads of single different files in mind. Where should these files be placed after download? Do they go into your existing faction? or a temporary directory? And all this is merged to something working then using physfs ?

[will]

I

But you obviously have the downloads of single different files in mind.

Yes.  By tracking individual files by size and hash you get straightforward de-duplication and delta downloads.

Where should these files be placed after download? Do they go into your existing faction? or a temporary directory? And all this is merged to something working then using physfs ?

They could go in a file - sqlite is used by most browsers, for eaxmple - or in a folder.  Whichever, its the engine that tracks them.

Physfs is different, and as it is now, fundementally flawed.  You cannot have two versions of the same file in physfs as the filename, not hash, is the index.

[titi]

This might all be a good idea, but I think its much mcuh too complicated to implement. This might be an approach for a general library handling these kind of problems, but much too complicated for a minor part of our game.