integrated mod installer

[will]

https://forum.megaglest.org/index.php?topic=6002.msg62656#msg62656

Apologies if this has previously been discussed to death (although I couldn't find a thread on it in this forum), but I'm volunteering to try and implement it.  I have plenty of experience programming this downloading kind of thing, but it will be a fun and gentle introduction to coding glest...

The first step is get a good idea of the requirements so please reply with your assumptions and desires.

Oh, and explain how we manage mods that add fractions vs mods that use the engine for alternative games...

[Zoythrus]

not just a mod installer, but we also need an auto-updater too!

[Gabbe]

I think i remember Omega or Arch tell me about something about GAE patcher or something..

[ChupaReaper]

Some form of Frontend/Launcher for GAE would be good, it could replace the Configurator too.
I know this wouldn't be intergrated into GAE but it would do for now, it could change the game settings, key bindings and could check for mod updates and install them, etc then eventually for GAE 0.6 or something it could be intergrated.

[will]

I have found this thread on the mailing list:

http://sourceforge.net/mailarchive/forum.php?thread_name=AANLkTikA%3Ddn8ESu_LyBREJMV1rObnOnp1oiv8txTU1yt%40mail.gmail.com&forum_name=glestae-devel

Is this already going somewhere?

[John.d.h]

I have no idea how familiar you are with the engine just yet, so you may know all of this already, but here goes (from a non-coder).  GAE uses PhysFS to read mods from the user's addons folder (~/addons on Linux), and currently gives priority to the files in the addons folder.  The directory layout of the add-on must mimick the regular structure of Glest's data directory.  An add-on can stand alone or modify existing content, or both, and it overrides files on an individual basis, so an add-on can modify as much or as little as the modder wants.  An add-on could include an entire tech tree with accompanying tilesets, scenarios, and maps, or as little as a single file.  For example, if I wanted to make an add-on that gave the Initiate 1000hp and changed nothing else, I would make a single initiate.xml file and make the following directory of empty folders: uber_initiate/techs/factions/magic/units/initiate/.  That would be placed in the addons folder, and would override the native initiate.xml file at run-time.  Alternatively, instead of "uber-initiate" being a folder, it can be a zip file.  Certain types of 7zip archives are allowed too, but someone else can explain more about that.  Thus, we would have uber_initiate.zip/techs/factions/magic/units/initiate/initiate.xml.

So, as long as the add-on in question replicates the file structure properly, I believe all a downloader would need to do is fetch the add-on from a server and put it in the addons folder, and maybe check for updates from said server.

[ChupaReaper]

So, as long as the add-on in question replicates the file structure properly, I believe all a downloader would need to do is fetch the add-on from a server and put it in the addons folder, and maybe check for updates from said server.

This would be good, but I think some projects should be able to go directly into the glestae directory, perhaps whatever server the mod is on can store whether the mods are saved into addons or into glestae, this is incase you get addons for project like mine which is basically a separate game that uses the gae engine, addons may be made just for my project rather than Glest, etc.

[John.d.h]

This would be good, but I think some projects should be able to go directly into the glestae directory, perhaps whatever server the mod is on can store whether the mods are saved into addons or into glestae, this is incase you get addons for project like mine which is basically a separate game that uses the gae engine, addons may be made just for my project rather than Glest, etc.

In what situation would there be a benefit to putting it in the main directory instead of addons?

https://forum.megaglest.org/index.php?topic=6002.msg62656#msg62656
Oh, and explain how we manage mods that add fractions vs mods that use the engine for alternative games...

I forgot to mention that there's a parameter in an XML somewhere that says something like <total-conversion value=""/>.  I'm not sure what all it does, though.

[wciow]

i would like to see an auto updater which scans the server and updates mods which are already installed with the newest version.

Currently if two mods try to overwrite the same data files such as main menu models for a total conversion then physfs selects one over the over automatically. I believe Silnarm was working on a mod selector to choose which mods are loaded after start up.

Another problem which may arise is if one mod depends on another. mods need to check for dependencies in other mod files and not load if the prerequisite files are not loaded.

[will]

Gathering advice on the mechanics of the packaging here: http://encode.ru/threads/1164-recommended-formats-for-game-data-and-partial-updates

My thought is that two tools are needed:

1) a packaging tool that generates the archive, some patch archives for moving cheaply from sequential version to version, and accurately recording the dependencies

2) the front-end in GAE itself, probably via the tech tree options and fraction choices in the new game screens; I can see that an external tool is more immediate to develop, but I favour a deeper integration as giving more customer satisfaction longer term

And this cascades into what to do if playing a multi-player game and ensuring all players have the same (e.g. most recent) version...

[will]

Some stats about diffing two versions of a mod: Malevolent Rising 1.6 vs 1.0 (yes I know that's a big jump, but it was about the only two files I could find):

245 files added:     41,706,266 bytes
300 files unchanged: 95,469,241 bytes
11  files changed:    7,958,481 bytes
96  files deleted:   19,597,478 bytes

Is this fairly typical?  Also, is there ever much renaming of files without changing their contents?  (I did not check for this in the script I wrote to get those numbers)

It looks like I have a very simple format for the partial downloads of new versions in mind....

[John.d.h]

Is this fairly typical?  Also, is there ever much renaming of files without changing their contents?  (I did not check for this in the script I wrote to get those numbers)

Malevolent Rising is not representative of the typical mods we have around here.  Most are much smaller.  Renaming isn't all that rare -- even just in Magitech, the Behemoth was formerly called a Daemon Giant and the Guard used to be called an Armored Swordsman.  Plus, we seem to have a lot of people around here who can't spell, so that sometimes gets corrected with subsequent versions.

[Omega]

I'd also like to note that I think this shouldn't be built into GAE, but a standalone tool compatable with all versions of Glest (would also have to check a mod's data when downloading as for which engine it is, something that is easiest done server side).

[will]

I added rename-checking and got these numbers for MRise 1.0 -> 1.6:

245 added:       39.8 MB
9   renamed:      1.6 MB
300 unchanged:   91.0 MB
11  changed:      7.6 MB
87  deleted:     17.1 MB

471 previously: 117.3 MB -> 60.2 MB zipped -> 50 MB 7z/rar
556 now:        148.7 MB -> 78.1 MB zipped -> 64 MB 7z/rar
    delta:               -> 27.0 MB zipped -> 19 MB 7z

My script, if anyone wants to run it on their mods (requires Python): https://gist.github.com/704809

9 of the files were found to be renamed.

With zip (my format of choice) the 1.0 mod archive was 60 MB compressed; it was originally distributed as a 7zip archive using RAR compression (and, I imagine, solid), and was only 50 MB.

The 1.6 mod archive compresses to 78 MB using zip, and was distributed as a 7zip archive with RAR too, taking just 64 MB.

Using the 'diffing', I calcuate you would only need to download 27 MB of of the 1.6 zip file in order to patch a copy of 1.0 that you already had; if we used 7zip (non-solid; actually, solid didn't improve compression dramatically) then this would be only around 19 MB.

So you can imagine two archives sitting on public web hosting, e.g. Google Sites.  There is also a 'diff' meta-data file.  The updater checks the meta-data file to see if its changed, using HTTP If-Modified-Since etc.  If its modified, it gets the meta-data and either fetches the full new archive, or uses a range request to download the delta (that has been conveniently packaged at the end of the new archive).  You can imagine a central site with scripts and things for the mod makers to be using to 'publish' the availability of updates etc, but I think it useful that the actual GBs of mods are stored elsewhere, and Google Sites seems to be a good choice.

Now 7z is supported in the latest GAE, but I feel that zip is an acceptable format and will be easier to work with.

There are zip archivers (e.g. 7z, info-zip but especially kzip) that do a better job of compressing zip archives than the normal 'zip' tools (but take much longer).

Working out how to diff mods was an interesting distraction but the real meat of the problem is now learning how the virtual FS works in GAE, and how to mod Glest and MG.

Keep feature requests and insights rolling in!

[silnarm]

I have found this thread on the mailing list:
...
Is this already going somewhere?

Yggdrasil finally opened a ticket, and the prototype was perfectly functional, so I think yes... in due course it will happen.

Maybe check out the prototype (attached to the first post on the mailing list conversation) to see what it does already, he used xdelta for the patches, which are taken from a tar of the mod (single patchfile for a mod).

[Yggdrasil]

Yeah, i almost forgot to open a ticket. Haven't worked on it since the prototype. I wanted to work on it after adding the addons menu where i'm currently stuck at the gui code... But any help is welcomed.

I briefly looked at your python script and you just compare file names and sizes. Actually same size doesn't mean no modifications. A checksum would be better. And packaging the delta at the end of the archive to get it later with a range query won't work. The zip header is missing and i don't know how you want to merge two zips.

Anyway, i think we should better use existing patch tools like xdelta instead of coding our own. Here's a shell script i used to create patches between two tarballs (addon in a tar archive).

Code: [Select]

#!/bin/sh

if [ $# -eq 4 ]; then
        if [ "$1" = "create" ]; then
                size1=$(ls -l $2 | cut -d " " -f 5)
                size2=$(ls -l $3 | cut -d " " -f 5)
                if [ $size1 -lt $size2 ]; then
                        size1=$size2
                fi
                xdelta3 -e0 -B $size1 -W 16777216 -I 0 -s $2 $3 $4
                xz -9 $4
        elif [ "$1" = "patch" ]; then
                unxz -k $3
                xdelta3 -d -s $2 ${3/.xz/} $4
                rm ${3/.xz/}
        else
                echo "usage: $0 create old new patchfile"
                echo "or: $0 patch old patchfile new"
        fi
else
        echo "usage: $0 create old new patchfile"
        echo "or: $0 patch old patchfile new"
fi
Glest itself just needs to apply these patches if we really want to integrate it directly. The prototype only does that but it's still a bit buggy. libcurl is also not the right choice as it can't pause and resume downloads. The list of available addons on the server is saved in a simple CSV file which was handwritten for this simple test.

Another problem which may arise is if one mod depends on another. mods need to check for dependencies in other mod files and not load if the prerequisite files are not loaded.

I don't really think we need that and it makes it just more complicated. Dependency checking isn't trivial and involves too much work. We even don't need patches for the first version.

[will]

yeah I was expecting to author the zip or 7z archive directly to turn a range request into a full archive.

my python script was not intended in any way to be the codebase that gets adopted, but rather I simply wanted to know what fraction of a mod altered for each iteration and how big the diffs might be; not find a quick way to determine that, I wrote that throw-away python script.

I feel uncomfortable with a tar+xdelta approach and would prefer an approach where the xml of a mod was parsed in the same way the game does, all dependencies located, and all the appropriate bits getting zipped up along with meta information.  It might be that the end result is an archive, but the way to get there is quite different and more involved.

The appeal of HTTP range requests is the simplicity of hosting.

So rather than a script that just thinks about directories, a tool that actually parses and packages the mod.  That's my general outlook.

[ChupaReaper]

This sounds interesting, I'd be happy to open up my mod so that incomplete factions can be downloaded and updated on the fly along with complete factions with automatic patching, etc. You'd have to explain how I would set up my project for all this though, perhaps setup a page on the wiki explaining how to make mods 'live' or whatever.
If it could be set up where individual files are updated rather than downloading everything in bulk it would be simply to patch one tiny spelling mistake somewhere or some stats adjustments for better game balance... I hope I make sense here lol!
With my mod being large and only going to be larger, the pause and resume options for downloading would be excellent too, maybe we could go as far as setting download priorities so people can play single player matches whilst downloading on the go (download a few main files then a tileset, map and a faction so they can play from there while other factions and tilesets continue to download). For this factions, tilesets and maps and 'core files (menu changes, language, etc)' would have to be treated as separate pieces to download.

[Yggdrasil]

I feel uncomfortable with a tar+xdelta approach and would prefer an approach where the xml of a mod was parsed in the same way the game does, all dependencies located, and all the appropriate bits getting zipped up along with meta information.  It might be that the end result is an archive, but the way to get there is quite different and more involved.

Isn't this the modder's work? And what meta information?

I'm more with the KISS principle here. We already get archives from the modders, hopefully addon compatible, and just upload these to the repository. Create a simple CSV file which lists all addons with version number and a checksum for integrity check after download. And that's it.

Next step would be to add binary patches which reduces updates dramatically. Diff utilities work best on uncompressed data like tar. Just add these patches to the CSV file and prefer to load these on update instead of full. The only drawback here: After patching we have an uncompressed tarball which is currently not supported by physfs.

That's what basically the prototype does. I don't see any benefit in your approach and i actually don't really understand what you want to achieve.

[Omega]

What I would do would be a bit more complex (on the web side) but just as easy on the programmer's side.

Firstly, we'd have our web host, which would host zips (and possibly 7z's in the future, if support is added). The host has a page which displays the mod list, in a searchable method comparable to download.com (the current look of the MDC would get too big when more and more mods are added). This would be comparable to some other games mod centers. Once the user finds a mod they like, they download it (the web host will have individual pages created from XMLs, similar to how things are now, but more data would be needed for better security and features. Basically, a rough login system and ownership system would let modders upload their mods. Some users would always have ownership thanks to a "moderator status" to allow them to upload mods and update information on them. For the most part, each modder would be responsible for keeping their own mods up-to-date on the MDC). After the mod is downloaded, the mod installer will install the mod. Information can be contained in the page which specifies the engines of the mod, and the installer let's the player choose which engines to install it for (if compatible).

GAE mods would just place the file in the user's addon's folder (presumably, we'd have to check if the addons folder exists, and if not, let the user specify it's location (in case they set a different location for the folder), though, presumably we'd have the option to extract it into the the folders directly, if wished. I'm not sure if MG has a variety of the addons folder (I believe it does have a mydata folder or something), but if so, can be installed in there, while vanilla Glest mods would be installed directly by extracting the file into the Glest directory.

Programmer side, it's just catches the downloaded mod and installs it. The complexity is how to install the mod or patch. Web design side, it let's the player search, read descriptions, reviews, etc for finding the mod. Overally, it's just Yggdrasil's idea with a better interface.


On another note, how would a diff work? What if the user had v1 and wanted to update to v1.2 (skipping a version). I'd think there'd have to be 2 versions of the upgrade (which is what SMF, this forum, uses). Basically, there's a small upgrade which upgrades from the last version to the most recent, and there's the large upgrade, which contains all the files changed since the last major release (for example, Military's next version, 2.5, would have a small upgrade to upgrade from 2.4 to 2.5, whereas the large upgrade could upgrade from any 2.x to 2.5. Older than that and you'd need a full install). On the web design side, there's just a few download options. Upgrades would only be available to download if the modder made them available.


At any rate, the web design side WOULD work stand alone with or without the tool, and I will end up starting work on it as the next version of the MDC.

[will]

That's what basically the prototype does. I don't see any benefit in your approach and i actually don't really understand what you want to achieve.

ok.  Tar+xdelta for now then.  We can always make it more complicated later.  I want to code things and help code things, not be an upstart.

So there seems two directions I could code in:

1) code and submit to physfs tar support

2) the study of supporting multiple total-conversions and also mods that mod a total conversion

thoughts on what needs doing?

[silnarm]

...I wanted to work on it after adding the addons menu where i'm currently stuck at the gui code...

Direct complaints this way   It certainly needs work, the scrollbar in particular is somewhat awkward to use. I'd suggest sub-classing Container for the 'AddonEntry' and laying them out explicitly, then just put them in a Panel with vertical auto-layout.  I'll add support to Panel to automatically add the scrollbar if it can't fit all it's children.

On another note, how would a diff work? What if the user had v1 and wanted to update to v1.2 (skipping a version). I'd think there'd have to be 2 versions of the upgrade (which is what SMF, this forum, uses). Basically, there's a small upgrade which upgrades from the last version to the most recent, and there's the large upgrade, which contains all the files changed since the last major release (for example, Military's next version, 2.5, would have a small upgrade to upgrade from 2.4 to 2.5, whereas the large upgrade could upgrade from any 2.x to 2.5. Older than that and you'd need a full install). On the web design side, there's just a few download options. Upgrades would only be available to download if the modder made them available.

Personally I'd only support one patch, from the previous to current version, but I think this is something that would be up to the modder. You'll have access to the repository to maintain your mod, and you will create the patches (we will of course provide the tools to do this). So you can support what upgrades you can be bothered preparing :-)

1) code and submit to physfs tar support

2) the study of supporting multiple total-conversions and also mods that mod a total conversion

tar support for physfs would certainly get my vote, total conversions can always use their own installer, and 0.4 will have a menu to select which addons are mounted and which aren't, so that will not be as much of an issue as it seems now.

[will]

I'll go add tar to physfs; a good way to learn my way around how it works too

[Omega]

Patches would really be at the discretion of the modder. What I have in mind is a very simply system where you can upload your own files and set the data for them, while supporting a basic search tool. Powered by PHP, users would logon, create a user name and password, all mods would be defaultly set to have "owners", which would simply be associated with the names of the uploader (a necessary security option to prevent someone from uploading a bad copy, etc) plus a set of users that would be considered the "moderators". Uploading and browsing mods, etc, is all web side, the program's focus would then be the downloader/installer.

We'd also, of course, need a tool to make patches that the program can install. I think the patches should directly edit the files in either the archive or the installed files.

[hailstone]

not just a mod installer, but we also need an auto-updater too!

I had a topic where I was talking about making an updater but I can't find it now. One of the ideas was that you can start with only a single binary file and it would download anything that was needed to run.