Author Topic: integrated mod installer  (Read 12585 times)

John.d.h

  • Moderator
  • Airship
  • ********
  • Posts: 3,757
  • I have to go now. My planet needs me.
    • View Profile
Re: integrated mod installer
« Reply #50 on: 25 November 2010, 08:15:29 »
can anyone else verify that the pine woods tileset in vbros 1 is broken?
I would recommend against using anything from the vbros for testing.  Their "quality assurance" is non-existent, and releases are frequently broken in some way or another (like inconsistent spelling between the file names and XML references).
It has to be a zip file to download though. You can't download an entire folder. It needs to be an archive. 7z is better than zip, and hopefully will be supported in the future.
We've already got 7z in recent revisions, but it only works for some types -- something about solid vs. non-solid, and various other things I don't understand.

titi

  • MegaGlest Team
  • Airship
  • ********
  • Posts: 4,240
    • View Profile
    • http://www.titusgames.de
Re: integrated mod installer
« Reply #51 on: 25 November 2010, 09:24:44 »
I would suggest small steps first and you mix different functions here:

1. GAE update
2. mod download and install
3. download/patching existing mods

I for myself think about only step 2 for MG at the moment. An ingame mod download and installation in MG. Prepared packages that I tested and which proofed to be ok(some how ok ;-) ) will be uploaded manually by me to our webserver. Those mods which are in there are presented in the game and can be downloaded and will be installed.
There should also be a way to remove them. To get new versions of mods you have to remove/install them manually in game ( for the first step )
Downloadable will be maps,tilesets and full techtrees.

Regarding 1. ( updating GAE ) is the most tricky thing in my opinion, especially regarding different platforms. I am curious how this will work! I for myself have no really practicable idea at the moment.

Point 3. might be possible to solve either by using existing code/libs from others or writing something on your own.
Try Megaglest! Improved Engine / New factions / New tilesets / New maps / New scenarios

will

  • Golem
  • ******
  • Posts: 783
    • View Profile
Re: integrated mod installer
« Reply #52 on: 25 November 2010, 12:03:14 »
I have written a very simple test script to see how successfully I can naively detect dependencies in a mod.

Here's the python code; it can just be run in python.

https://gist.github.com/715270

For a taste, here's what it thinks about megapack_v4:

Code: [Select]
...
Ignoring tilesets/autumn/models/grass.tga 768.0 KB
Ignoring techs/megapack_v4/factions/persian/units/elephant_cage/sounds/tech_building_fall5.wav 148.6 KB
Ignoring techs/megapack_v4/factions/elves/units/tree_of_life/construction_models/tree1.g3d 1.1 KB
Ignoring techs/megapack_v4/factions/elves/units/wood_hall/images/wood_hall.bmp 12.1 KB
Ignoring techs/megapack_v4/factions/indian/units/thunderbird/sounds/eagle_fly2.wav 426.0 KB
=== Ignored: 675 121.3 MB ===
=== Included: 4681 587.2 MB ===
=== Models: 922 252.7 MB ===
=== Textures: 1040 183.1 MB ===
=== Particles: 120 83.5 KB ===
=== Sounds: 2360 149.9 MB ===
=== Units: 172 824.1 KB ===
=== Upgrades: 53 34.8 KB ===
=== Resources: 7 1.2 KB ===
=== Fractions: magic, norsemen, dark_magic, indian, persian, elves, tech, woodsmen ===
=== Maps: overgrown_city, tt_goldrush_v2, forest_for_four, waterworld, tropical_arena, domination_isles, one_on_one, conflict ===
=== Tile Sets: evergreen, desert2, winter, autumn, jungle, hell ===
=== Tech Trees: megapack_v4 ===
=== Scenarios: m4_kill_the_magic, m4_my_island, m4_beehive_castle, m4_amazones, m4_norsemen_attack, m4_persian_conflict, m4_indian_attack, m4_waterworld_light, m4_waterworld, m4_magic_can_do_it, m4_amazones_light ===
=== Mod check failed ===
(model BROKEN techs/megapack_v4/factions/persian/units/swordman/models/guard_attacking.g3d (referenced by: unit ../swordman.xml)) ('Error reading G3D file', IOError(2, 'No such file or directory')), ('File does not exist',)
(model BROKEN techs/megapack_v4/factions/tech/units/swordman/models/guard_attacking.g3d (referenced by: unit ../swordman.xml)) ('Error reading G3D file', IOError(2, 'No such file or directory')), ('File does not exist',)
(particle BROKEN techs/megapack_v4/factions/persian/units/worker/particle_splash.xml (referenced by: unit worker.xml)) ('File does not exist',)
(scenario BROKEN scenarios/m4_amazones/m4_amazones.xml) ('References missing map', u'in_the_forest'), ('References missing tileset', u'forest')
(scenario BROKEN scenarios/m4_amazones_light/m4_amazones_light.xml) ('References missing map', u'in_the_forest'), ('References missing tileset', u'forest')
(sound BROKEN techs/megapack_v4/factions/norsemen/units/mistletree/sounds/tower_attack1.wav (referenced by: unit ../mistletree.xml)) ('File does not exist',)
(sound BROKEN techs/megapack_v4/factions/norsemen/units/mistletree/sounds/tower_attack2.wav (referenced by: unit ../mistletree.xml)) ('File does not exist',)
...

I really appreciate modders running it on their own mods and others they are familiar with, and then helping me understand:

  • what is ignored but shouldn't be - and how I would detect it
  • what is not resolved but should be, and what the path in the xml was to the file that's not really missing
  • any surprises
  • any encouragement or feature requests
  • any bugs or crashes, particularly if you get a stack trace from running the script

Its very easy to run; all you need is python, and know where the mod is (the mod should be in a folder, rather as you would unpack an archive)

The thing is, I haven't found a mod that this script doesn't think is broken!  I've squashed most of my mistakes, and iteratively found most, but now I really need help from expert modders to tidy up the last few things I've got wrong....

(I don't think this script will be used in any real GAE solution, its just a quick way to prototype the logic that might be used, and see if its useful)
« Last Edit: 25 November 2010, 12:51:50 by will »

wciow

  • Behemoth
  • *******
  • Posts: 968
    • View Profile
Re: integrated mod installer
« Reply #53 on: 25 November 2010, 15:55:18 »
Ok I'll test your new script on my undead mod which i have packaged and running on gae 3.2.

Heres a question: If I have two TC mods which both modify the main_menu.g3d how does gae/physfs decide which one shows up at runtime?
Check out my new Goblin faction - https://forum.megaglest.org/index.php?topic=9658.0

Yggdrasil

  • Local Moderator
  • Ornithopter
  • ********
  • Posts: 408
    • View Profile
Re: integrated mod installer
« Reply #54 on: 25 November 2010, 16:42:26 »
I agree with silnarm and titi: First step full installation, next patches and then when we really need it (i doubt it) dependency checking. I don't know any mod which relies on another mod.

@will: What you're writing is a packager which is a nice to have tool. I also think the maintainers of the repository should create or at least double check the packages and patches, not the modders. The modders only create addon compatible archives which can be still downloaded from forum users and installed manually. The repository only contains "final" releases which are usable, no alphas or betas.

One big question we still need to sort out: Do we really want do integrate this addon manager with a menu inside the game? Or is an application outside like the map editor better?

Heres a question: If I have two TC mods which both modify the main_menu.g3d how does gae/physfs decide which one shows up at runtime?
The one first in the search path. The order of this isn't well defined, so it could be different on different platforms.

will

  • Golem
  • ******
  • Posts: 783
    • View Profile
Re: integrated mod installer
« Reply #55 on: 25 November 2010, 18:51:57 »
I started this thread, I volunteered to over-engineer a mod installer, so its natural I post most, but I've been trying to get others to lead the way as I'm the newcomer.

From the first post we asked "how we manage mods that add fractions vs mods that use the engine for alternative games..." and its clear that addons/ doesn't do this.  We can of course change addons and bend physfs to our will as it is not widely relied upon yet.

The bit that does seem agreed upon is that these mods will be deployed each as a single archive, using stored ZIP with XY solid compression for download.  And we will have xdelta+XY from the beginning too, since that's not where the bugs will creep in.

I see two types of mod:

  • (1) a free-standing game such as Glest+MegaPackV4, or Military of MRise.
  • (2) a mod that adds additional scenarios, fractions and/tilesets to (1).  Examples are numerous for the Glest tech tree.  I suggested the naming convention that lists the (1)-mod that it is extending.

To my mind GAE should not be bundled with the existing old Glest game data, but rather pre-installed with a Glest+MegaPackV4 mod in the new mod structure.

Furthermore, I think (2) needs dependency checking from the get-go.  And it doesn't hurt (1) either, since all the mods I've looked at - with the exception of MRise, which is nice and clean - are full of MB of debris and even broken links.

If a (2)-type mod has a scenario with the 'pirates' fraction in it, it'd better list the appropriate mod it is relying upon in the manifest so the downloader can ensure all dependencies are met.  I don't see this as being a stretch goal, I see this as hygiene and not too challenging.

In my dependency prototype I ignored the overriden 'core' files, because we need some agreement on how to handle them.  I think they should go into a set folder within the mod archive, and that it would be much more robust if there was an XML file explicitly saying which artwork overrides whatever rather than an implicit naming system.

Regards integration, I think that mod makers should be releasing early and often, even if their audience is their co-conspiritors and other modders.  I think the system should be full of small incremental releases to 'alpha' mods, and that a mod should have a 'stable?' flag when its uploaded so we know whether to only show it to players who have elected to see the bleeding edge.

And on the client-side, I imagine a stand-alone wx-widgets app that does the downloading.  From the beginning the downloader code however should be in the shared lib, with the full intention of eventually bringing it into the game UI.

In the game UI, there should be a top-level menu option for selecting between (1) mods, which naturally swap out all the UI artwork.  The most-recent choice is remembered.  But in the new game screen, if the 'advanced' check-box is checked, you can pick and mix between all installed fractions, maps, tilesets etc.

Its natural people are apprehensive of such naked ambition, but is any got any fundemental against the direction its going in, or is it just seeming to go too fast?
« Last Edit: 25 November 2010, 18:54:05 by will »

Omega

  • MegaGlest Team
  • Dragon
  • ********
  • Posts: 6,167
  • Professional bug writer
    • View Profile
    • Personal site
Re: integrated mod installer
« Reply #56 on: 25 November 2010, 22:31:29 »
My two bits:
DEFINATELY a separate program, as I mentioned before. The first step might simply install the mods as a GAE addon (after all, that's as simple as copying the downloaded file to the addons folder. I would think the hardest thing to do is to locate the addons folder (again, I think it should be declared in the INI). After all, if you try to just install where the addons folder defaultly is, you'll just miss out everyone who has modified this (such as me. I don't like programs spreading files all over my computer, and prefer it one place, so I placed the config dir to be the same as the install dir). The simple solution would be to check if the folder exists in the default install location and if it doesn't, check for it in the install directory, and if not there, ask the user to provide its location (if you had the skill to manually change the location, you must know where it is). There should also be a setting (such as a separate ini for this program) to provide this location, incase the auto-detect fails (ie: the folders exist in the default location because they weren't deleted after changing the config dir location).

GAE should NOT be bundled with the Megapack, for one simple reason: It's too darn big! I'm always a few versions behind on MG, and if there's no binary only version uploaded, I don't even download until there is.

As for what we should start as, I think we should start very simple. Download the mod and install it for GAE only, since it's the simplest to do. Then we can expand it so it can install MG and Vanilla Glest as well (if MG has an alternative to the addons folder, extract there. Vanilla will just have to extract it in the install directory). Later, we can impliment a dependancy searcher (which should just *tell* the user they need to also download so and so, not force it, since they could have so and so available from a different source (ie: downloaded it a long time ago or have it for a different engine). As the most ambitious part, an update system should be saved for LAST.

Another thing that would be nice AFTER the update system (aka: the grand finale) would be a way to update the engines themselves, while preserving mods. Obviously this may not always be possible if you move folder locations (but, hopefully you won't move folders). Biggest problem is that engines usually install to a folder that has the version number (thus, you can have GAE 0.2.12 and 0.3 both installed), so it may have to just be an installer, unless the updating would only be allowed for minor versions (ie: GAE 0.3.x can update, but once 0.4 is released, you have to do a full install).

You're fully welcome to program it will. We welcome new comers here (weren't we all one at one point?) and I personally think it's great you are getting involved in helping Glest! :) I can't program well outside of Web design anyway.

Biggest problem: getting modders to package their data the correct way... I have yet to see more than a tiny handful of people do so...
Edit the MegaGlest wiki: http://docs.megaglest.org/

My personal projects: http://github.com/KatrinaHoffert

will

  • Golem
  • ******
  • Posts: 783
    • View Profile
Re: integrated mod installer
« Reply #57 on: 25 November 2010, 23:28:30 »
update: a log from megaglest with vbros 1 along side: https://gist.github.com/716064

Is this stuff really broken?

E.g. does egypt's air_pyramid really reference technician/sounds/technician_ack1.wav, is it really missing, and what does G/MG/GAE do when sounds are missing?

ChupaReaper

  • Guest
Re: integrated mod installer
« Reply #58 on: 25 November 2010, 23:29:00 »
I like the sound of this and would fully package my mod to work with it. Pause/Resume downlaods would be very important for big projects too and also it would be cool if the update was also a launcher that could adjust the gae config and also launch the game with a nice fancy big Launch button or something. This updater should eventually support fancy graphics too at some point.

ultifd

  • Airship
  • ********
  • Posts: 4,443
  • The Glest Video Guy :) The one and only. :P
    • View Profile
    • My Youtube Channel
Re: integrated mod installer
« Reply #59 on: 26 November 2010, 07:40:51 »
GAE should NOT be bundled with the Megapack, for one simple reason: It's too darn big! I'm always a few versions behind on MG, and if there's no binary only version uploaded, I don't even download until there is.
Well, I guess it is kinda big, but not that much...and I guess you download each version of MG then, cause with each version there is always a binary only version...  :|  ;)

silnarm

  • Local Moderator
  • Behemoth
  • ********
  • Posts: 1,373
    • View Profile
Re: integrated mod installer
« Reply #60 on: 27 November 2010, 02:49:52 »
Regarding 1. ( updating GAE ) is the most tricky thing in my opinion, especially regarding different platforms. I am curious how this will work! I for myself have no really practicable idea at the moment.

This was discussed a while ago on the mailing list, we decided we would do nothing for linux, or at most a 'notifier' (ie, 'A new version of the engine is available, ...') to point them at a download link, or tell them to update through their package manager.

I've done a little research for the Windows side, and have found GUP to be a likely candidate, no idea how this should/would work for Mac.

But this will not be step 1 ;)

To my mind GAE should not be bundled with the existing old Glest game data, but rather pre-installed with a Glest+MegaPackV4 mod in the new mod structure.

GAE will continue to ship with a modified magitech, and nothing more. Maybe one day an extended magitech (Dark Magic?) or with a revived FPM.

Quote
Furthermore, I think (2) needs dependency checking from the get-go.  And it doesn't hurt

We generally prefer an iterative development model, I would personally like dependencies listed from the get-go, but in the 'first iteration' only so they can be reported to the user.

Quote
In my dependency prototype I ignored the overriden 'core' files, because we need some agreement on how to handle them.  I think they should go into a set folder within the mod archive, and that it would be much more robust if there was an XML file explicitly saying which artwork overrides whatever rather than an implicit naming system.

See suggested manifest xml below...

Quote
And on the client-side, I imagine a stand-alone wx-widgets app that does the downloading.  From the beginning the downloader code however should be in the shared lib, with the full intention of eventually bringing it into the game UI.

I think we are all agreed here then, separate app, maybe integrate into the game down the track...

Quote
In the game UI, there should be a top-level menu option for selecting between (1) mods, which naturally swap out all the UI artwork.  The most-recent choice is remembered.  But in the

This is already planned, #209

So, we need a 'manifest' type file for each addon, I don't think this needs a lot of information though, and certainly not a complete actual manifest ;)

A possible system,

For Chuppa's situation there would be, mrise_core.zip/mrise_core.xml :
Code: [Select]
<manifest type="core-data" name="malevolent_rising" version="1.6">
<!-- This pack contains custom tilesets and maps, turn default stuff off -->
<default-data>
<techs value="false" />
<tilesets value="false" />
<maps value="false" />
</default-data>
</manifest>

The type='core-data' would identify it as a TC, and the new addon menu will only ever allow one or zero TCs to be turned on, when turned off any other addons that depend on that addon will also be unmounted.

Chuppas factions would then have something like, mrise_necribus.zip/mrise_necribus.xml :
Code: [Select]
<manifest type="content" name="necribus_faction" version="1.6">
<dependencies>
<dependency name="malevolent_rising" version="1.6">
</dependencies>
</manifest>

Maybe something extra to 'link' it with malevolent_rising might be good, so it is automatically mounted with malevolent_rising, and there should be a description element in there somewhere.

If all addons that go in the repo are required to have such a manifest, the repository.xml could even be auto-generated by scanning the addons.
Glest Advanced Engine - Code Monkey

Timeline | Downloads

will

  • Golem
  • ******
  • Posts: 783
    • View Profile
Re: integrated mod installer
« Reply #61 on: 27 November 2010, 10:20:07 »

ChupaReaper

  • Guest
Re: integrated mod installer
« Reply #62 on: 30 November 2010, 02:44:23 »
With what Silnarm said:
<dependency name="malevolent_rising" version="1.6+">
+ or something like that should also be allowed, in case I have core_files 1.7 but all the factions are ok at 1.6 still.

will

  • Golem
  • ******
  • Posts: 783
    • View Profile
Re: integrated mod installer
« Reply #63 on: 30 November 2010, 12:59:53 »
At the moment my mod packer puts some xml in as a zipfile comment.

I think I'll change it so that it writes a /glest-mod-manifest.xml to the mod file describing the dependencies.

If the modder provides a /glest-mod-manifest.xml then it will be sensibly merged with the dependencies that the mod-packer finds.

I like that on many platforms that use version numbers, 'major' version number changes introduce incompatibilities but 'minor' versions do not, so therefore the tool can believe that something depending on mymod.1.2 can work with mymod.1.3 but not mymod.2.1

The tool is coming along nicely, I'd ask everyone to kindly test it and give feedback.

will

  • Golem
  • ******
  • Posts: 783
    • View Profile
Re: integrated mod installer
« Reply #64 on: 9 December 2010, 06:32:36 »
So we have a mod packer script that basically seems to work, and does a tidy job.  Its a python script, but its serviceable.

The mods are more like classic glest extensions as opposed to total-conversions, and the distinction is something that will need GAE code changes with the TC picker work.  I say that TCs are version 2!  The immediate goal is to repack all the classic glest extensions and get them hosted on a website so we can test the packing and downloading part.

There are two directions to now work in:

  • A website, so the glest_mod_pack.py tool can upload the mods
  • A mod installing tool

Of these, the second seems to depend on the first, so right now we need a website.  So who wants to help on the website and how do we go about hosting?

silnarm

  • Local Moderator
  • Behemoth
  • ********
  • Posts: 1,373
    • View Profile
Re: integrated mod installer
« Reply #65 on: 9 December 2010, 09:13:38 »
The mods are more like classic glest extensions as opposed to total-conversions, and the distinction is something that will need GAE code changes with the TC picker work. I say that TCs are version 2!

GAE is quite actively developed, the support required for this would not take long to implement.  Either or.

Quote
Of these, the second seems to depend on the first, so right now we need a website.  So who wants to help on the website and how do we go about hosting?

I can arrange a dedicated server when needed, but we can probably just use sf for now. I know they allow quiet a bit in the 'web services' department, and PHP we can definately run from glestae.sf.net, not sure on if/where we can write uploads to. I'll check again what we have available, and mail the sf ppl if needed.  In the meantime, sign up on sourceforge and mail me your username.

Edit:  misuse of quiet.
« Last Edit: 9 December 2010, 11:05:39 by silnarm »
Glest Advanced Engine - Code Monkey

Timeline | Downloads

hailstone

  • Local Moderator
  • Battle Machine
  • ********
  • Posts: 1,568
    • View Profile
Re: integrated mod installer
« Reply #66 on: 9 December 2010, 10:45:41 »
I might be able to offer my paid web hosting which has python support if other arrangements aren't possible.
Glest Advanced Engine - Admin/Programmer
https://sourceforge.net/projects/glestae/

 

anything