MegaGlest Forum

Modding and game content creation => Tools => Topic started by: will on 26 November 2010, 13:15:15

Title: glest_mod_pack.py
Post by: will on 26 November 2010, 13:15:15
https://github.com/williame/GlestTools

You need python - and only python - to run it.  If you want to compress best, you can also install the 'xz' command-line compressor and ensure that is in your path (or, on windows, in the same directory as you run the script from I think)

It is Glest / MG / GAE agnostic, and useful for packaging mods of all those formats.

It enforces a version numbering system in the naming convention of mods.

To use it, simply put your mod in a nice directory, ensure the name of the directory follows the convention (if it doesn't, you get told the naming convention!), and then use the tool instead of a normal zip program.

The idea is that there will soon be a website for hosting these mods, or at least meta-data for them, and that there will be a friendly installer for players to use to get new mods and do updates.

But in the meantime, right now, hopefully this packager is useful in making classic mod zipfiles!
Title: Re: glest_mod_pack.py
Post by: wciow on 26 November 2010, 15:48:15
I can't get the script working on Windows 7. I uodated Python to 2.7 just to make sure but it just flashes up then quits.

Trying to run the script through IDLE produces the following error:

Traceback (most recent call last):
  File "C:\glest_mod_pack.py", line 488, in <module>
    sys.exit(1)
Title: Re: glest_mod_pack.py
Post by: will on 27 November 2010, 10:19:28
Thanks for trying it out; sorry it didn't work first time; it was intended to be run from the "command line"

I've updated it, so it now asks you to enter the path to your mod if you don't use the command line, so now it should work for you!
Title: Re: glest_mod_pack.py
Post by: wciow on 27 November 2010, 10:58:41
Your script works very well, but it has some problems.

Firstly, it strips out certain files. I had a file called logo.pdn which is not strictly part of the mod but which i wanted to include anyway this file was stripped out. The magic_music.ogg was also stripped and no music folder appeared in the zipped mod. This is very bad since it essentially breaks the mod, the music folder and its contents should always be included in a mod. Most particle files were also stripped out of the mod. My guess is that your script only searches for particle.xml or particle_proj.xml. Particle files can have any name as long as they are referenced correctly in the units XML.

Secondly, the script complained that the mod was broken because it did not contain magitech.xml. This file is already in the magitech folder which the faction will be merged into. Only the original tech tree should contain the tech tree XML, mods should not be permitted (let alone expected) to overwrite this file since it is more likely to cause errors.

Thirdly, the mod correctly identified and stripped a useless old .bmp file that i had left in one of the folders, so at least some good news.

This script could be very useful if it was refined a little more. Some of the mods released here (no names mentioned) have contained masses of junk files or have included all of the original blends instead of packaging them seperately  ::)

Good luck with the script/downloader  :thumbup:
Title: Re: glest_mod_pack.py
Post by: will on 27 November 2010, 12:02:04
thank you wciow!

this is exactly the kind of detailed feedback I can use to fix the tool!

1) what are the rules for /music?  Are these files specified in any xml?

2) the problem with mods that extend others - as yours does 'magitech' is really a limitation of the tool, and I really need to fix this to make the tool marginally useful :)  My approach will be to package magictech as a mod, package megapackv4 as a mod that extends magitech, and then you can package a mod that extends either of those and the tool will check the dependencies correctly... that'll already be in the next version next week...

2) can you zip up the mod you were trying to package (not the zip that the tool makes) so I can recreate these problems, rather than asking you to test every fix I try?

Thanks very much again for running it!
Title: Re: glest_mod_pack.py
Post by: will on 27 November 2010, 21:42:22
I have updated the tool again.

It should now hopefully find all your particles.  Are there any particles it still doesn't find, and how are they specified in the xml?

The new version of the tool should cope with the tech tree xml being missing, as long as the name follows the convention that an extension filename specifies what it extends; in your case perhaps magitech.1.yourmod.1 ?

The tool does look for <music> in the fractions, so what xml file specifies the music that it isn't including?

The tool still doesn't package overrides of the core files for the total conversions.  Are these files all in a common directory, or some other rule like that I can follow?

For the files you want included that are not strictly part of the mod, would be ok to insist that they are in some subfolder e.g. /info ?  I could look for files in such a folder and include them all, always?

Title: Re: glest_mod_pack.py
Post by: John.d.h on 27 November 2010, 23:30:51
Quote from: will on 27 November 2010, 21:42:22
For the files you want included that are not strictly part of the mod, would be ok to insist that they are in some subfolder e.g. /info ?  I could look for files in such a folder and include them all, always?
Could it put those files in a separate archive, like magitech.1.yourmod.1_dev.zip?  These are files that aren't necessary for playability, so most players don't want to spend the bandwidth and extra space for them, but they should be useful for developers.  I think it could be set to put files with extensions like .blend, .3ds, .max, .xcf, .pdn, into the development zip, and host that alongside the regular one.

Quote from: will on 27 November 2010, 21:42:22
The tool does look for <music> in the fractions, so what xml file specifies the music that it isn't including?
This is specified in the name_of_faction.xml.  For example:
Code: [Select]
<music value="true" path="music/music_moon.ogg"/>

Quote from: will on 27 November 2010, 21:42:22
It should now hopefully find all your particles.  Are there any particles it still doesn't find, and how are they specified in the xml?
Particles can be specified in skills and effects.  They follow this syntax:
Code: [Select]
<particles value="true">
<particle-file path="ss_smoke_particles_right.xml"/>
<particle-file path="ss_smoke_particles_left.xml"/>
</particles>
Code: [Select]
<particle value="true" path="particle_proj.xml"/>
Code: [Select]
<particle enabled="" path=""/>
Title: Re: glest_mod_pack.py
Post by: will on 28 November 2010, 10:36:46
Excellent info!

I like the idea of putting all the 'ignored' files into a ._dev.zip.xy too rather than ignoring them; possibly putting *everything* into it, rather than trying to police it and remove *~ and thumbs.db etc

I've fixed some bugs, including:

Title: Re: glest_mod_pack.py
Post by: will on 30 November 2010, 08:55:47
The newest version of the tool can optimise mods so they are smaller zip downloads and take less space on user disks.

It can take several MB off mods that contain lots of duplication
Title: Re: glest_mod_pack.py
Post by: wciow on 30 November 2010, 18:47:41
I just tried to rerun the mod packer on my undead again. This time it suceeded in producing a workable zip file, but still complained that magitech.xml was missing.

The experimental optomizer didn't work, but it looks like a typo in the script somewhere:

(click to show/hide)
Title: Re: glest_mod_pack.py
Post by: will on 1 December 2010, 14:03:19
thx for your patience!

the problem with the paths were a unix/windows thing with path separators, and I think I've fixed that now.

the problem with the missing magitech is because the naming convention of mods means that your undead mod should be named "magitech.1.undead.1" or whatever, so the tool can see it extends magitech (and thus shouldn't package a tech-tree xml)

How big was your mod when you packaged it yourself without the tool?  And how big is it with the tool?  (Is the optimising and stripping of 'dev' stuff saving much at all?)

Some mods I test with have very very large savings, others not so much...
Title: Re: glest_mod_pack.py
Post by: wciow on 9 December 2010, 12:25:33
Ok I've finally got some more info for you and its generally good news  :)

unzipped 27.8MB
manual zip 14.9MB
manual 7z 9.5MB

Zipped via script 13.7MB
Zipped via script (w/ optimization) 12.5MB

The above zips were made as .zip files and not xy files. Xy didn't seem to work. My uneducated guess is that windows is missing xy functionality.

The good news is that naming conventions work correctly and the optomised mod file works, thus shaving off a few MB.

As you can see 7z is still way ahead in terms of compression though :-\
Title: Re: glest_mod_pack.py
Post by: will on 9 December 2010, 13:17:57
Yes, xz is a commandline tool with the same compression algorithm as 7zip.

Excellent that the packing now works!  Thank you for all your testing.
Title: Re: glest_mod_pack.py
Post by: John.d.h on 10 December 2010, 21:01:16
Err... how do I actually use this?
Code: [Select]
mint@mint ~/Desktop/temp/ $ ./glest_mod_pack.py
./glest_mod_pack.py: line 15: help_modname: command not found
./glest_mod_pack.py: line 25: legal_path_chars: command not found
 :confused:
Title: Re: glest_mod_pack.py
Post by: will on 11 December 2010, 06:46:27
Type:

 python glest_mod_pack.py

Title: Re: glest_mod_pack.py
Post by: will on 13 December 2010, 22:48:38
Now updated to (hopefully, untested) work with Python 2.5 and perhaps lower:

https://github.com/williame/GlestTools

Title: Re: glest_mod_pack.py
Post by: jda on 22 December 2011, 00:58:22
I haven'te tested this yet but it does sound great!  :thumbup:

I have two comments though, just from the topic read:

1. The <techtree-name>.xml problem and respective workaround seem a bit forced to me. Traditionally, the term "Mod" has been used in the Glest community to refer either to Techtrees or Factions (Total Conversions are usually referred to as Mods; so are Datapacks such as Megapack that included tilesets and scenarios as well - but both of these go beyong the problem of the techtree's XML prob, so I'll just ignore them here). Single-Faction mods are usually balanced to a techtree but not necessarily (agreed, they always should but ...), which IMO makes my point even stronger:
Wouldn't it be simpler to just test whether you are trying to package a full techtree or instead a single-faction mod? Easy to do by just checking the contents of the top-level folder:
* Techtree directory contents:
  - DIR factions
  - DIR resources
  - FILE techtree-name.xml
* Faction directory contents:
  - DIR music
  - DIR units
  - DIR upgrades
  - FILE faction-name.xml
It's easy to see which first-level directories belong to a techtree and which belong to a faction... (not sure of the following but I guess, in theory, a faction might actually work with only the units folder As clarified by Omega below, only the units and upgrades folders are actually mandatory for a faction's folder).
So, checking for the presence of such directories in the top folder should be enough to determine whether you are before a techtree or a faction mod.

2. Can this be used from a Git / Github repository to build a stable release without all the dev fuss? (or putting the dev-specifi fuss into another package)? Or is it only useable locally? If so, can you point me to directions on it (Git is basically a stranger to me)?
Title: Re: glest_mod_pack.py
Post by: Omega on 22 December 2011, 01:25:00
Quote from: jda on 22 December 2011, 00:58:22
* Techtree directory contents:
  - DIR factions
  - DIR resources
  - FILE techtree-name.xml
* Faction directory contents:
  - DIR music
  - DIR units
  - DIR upgrades
  - FILE faction-name.xml
Not to nitpick, but just to ensure we're all on the same page here, the music directory is NOT a hardcoded directory like units and upgrades. Whereas you must place units and upgrades in those specific folders, you don't have to have a music folder, or could name it anything you like, as long as the paths in the faction XML (https://docs.megaglest.org/XML/Faction) are accurate.
Title: Re: glest_mod_pack.py
Post by: jda on 22 December 2011, 01:35:02
Edited my post accordingly. Thanks! :thumbup: