integrated mod installer

[will]

Adding tar support to physfs is of course possible; however, it is performance presenting problems; tar is a solid format.

I'm not going to say what I think alternatives might be yet; I'm interested in what you folks think.

[will]

given that the addons/ folder hasn't really taken off yet, its meaning could be changed.

It could be that there is a folder containing 'total conversions' - including the default one, which perhaps ought to be megapack - and that only one of these is allowed at any one one time - its pickable from the main menu, and the artwork for the menu screens etc is based upon it.  It could be that the name 'addon' is changed to perhaps something slightly more appropriate?

And then within each of these folders, there the mods of any total conversion are stored.

How does this grab people?

[John.d.h]

only one of these is allowed at any one one time

I'm not so sure.  What if you wanted to use a map from Military with the MRise tech tree and a tileset from Megapack, for example?

[Omega]

only one of these is allowed at any one one time

I'm not so sure.  What if you wanted to use a map from Military with the MRise tech tree and a tileset from Megapack, for example?

+1 

[Yggdrasil]

Adding tar support to physfs is of course possible; however, it is performance presenting problems; tar is a solid format.

Mhmm, you're right here. Haven't thought about that.

Maybe we should just use an uncompressed zip (everything stored). A short test shows that xdelta produces patches with similar sizes to tar. Zip is already supported by physfs. The full packages could still be compressed with xz, using zip only as container.

I think all mods who change the core files, not core data, like the menu should package these files separately in another addon.

Btw, as this "tool" downloads the addons it should definitely also browse the repository. So, what information about the addon should be shown? And how do we structure this list?

My proposal: Simple CSV file with addon name, version number and checksum. A (very) short description would be nice and maybe a tiny logo (embedded with UUencode). It's not meant to inform you in detail about the mod, the forum is a better place for this, just enough to find what you're looking for.

[John.d.h]

My proposal: Simple CSV file with addon name, version number and checksum. A (very) short description would be nice and maybe a tiny logo (embedded with UUencode). It's not meant to inform you in detail about the mod, the forum is a better place for this, just enough to find what you're looking for.

A link to the forum thread or wiki page would also be nice.

[will]

XML is what modders have to use for the units, so why not for the mod description too?  CSV has the disadvantage of being flat and difficult to extend.

Now I'd like JSON to be usable everywhere we use XML today, but that doesn't seem to be necessary, just elegant.

XDelta on packed zips - as long as the zip is packed by the same zip program with the same settings, and that's something we control - is very nearly as good as on stored zips.  I know that if we xz the stored zip it is smaller, and that the xdelta between two stored zips is smaller, but its all not a lot.  If we work with compressed zips, compressed with a decent zipper like 7zip (kzip is sadly not open source, and its slow), then we can store compressed zips on the file system (which feels nice).  I mean, if the xz and xdelta advantage over deflate is worth it, then why not use bsdiff instead of xdelta and so on?

Ok, I bow my head in shame, suggesting xdelta on compressed zips.  It works surprisingly well.

If we want to go for good compression, we'd use bsdiff and stored .zip and .xy.

If we want things to be packaged faster we go for xdelta and delfated .zip, we just have to be careful to control the zip tool and options.  But then the zips are compressed on disk, which is nice.

Both these approaches bring up using PNG (and, honestly, JPG) instead of BMP/targa, and OGG instead of WAV.  If people did this, and we added support for g3dz, there'd be no difference between a deflated and a stored zip.

What is clear is that either a mod contains each and every artwork it needs - yuck - or a dependency checker is built into the packager tool so it detects all dependencies.

[John.d.h]

Something just came to mind regarding extra files that sometimes get left in mod downloads that don't need to be there, or aren't necessary for playability.  If the database separated out the source files (.blend, .xcf) and put them into an accompanying archive, then it would save some trouble for the modder and also ensure the player that they weren't wasting bandwidth, while preserving source files for anyone who wants to modify the art for their own mod.  It could also clean out useless files like ~*.xml, thumbs.db, etc.

[Omega]

Meh, I still plan to create the website end as I proposed as the next MDC...

And all XMls are read and made individually for security, instead of one CSV for all mods... And I agree, XML > CSV.

[Yggdrasil]

Ok, then XML. Just wanted to keep the parser simpler and the file smaller. Something like that?

Code: [Select]

<repository>
<mods>
<mod>
<name>...</name>
<version>...</version>
<description>...</description>
<checksum>...<checksum>
<logo>...UUencoded 32x32 PNG (optionally)...</logo>
</mod>
...
</mods>

<patches>
<patch>
<mod>...modname...</mod>
<oldversion>...patches from this version to the newer one above...</oldversion>
<checksum>...</checksum>
</patch>
...
</patches

</repository>

I think memory consumption and run time are still acceptable with xdelta and xz. I just thought about using libarchive on clientside and xz is already there, so why not use it. But it's not really important to nail the compression format now. Just keep the bandwidth for the hoster in mind.

Testing compressed zip and uncompressed zip(.zip0) using mrise core files:

Code: [Select]

79M mrise_1.6.zip
149M mrise_1.6.zip0
 58M mrise_1.6.zip0.xz
 61M mrise_1.0.zip
118M mrise_1.0.zip0
 45M mrise_1.0.zip0.xz
 45M mrise_zip0.xdelta
 19M mrise_zip0.xdelta.xz
 26M mrise_zip0.xdelta.zip
 27M mrise_zip.xdelta
Using compressed zips makes it easier for the program and is smaller on disk for the user but uses more space and bandwidth for the hoster. I'm not sure what's better, depends on the webspace we want to use. Opinions?

Meh, I still plan to create the website end as I proposed as the next MDC...

One could still use your "web platform" idea to maintain this list.

[Omega]

The current MDC uses the following layout:

Code: [Select]

<mod>
   <type>tech|tileset|scenario|map|other</type>
   <name>Mod Name (as it appears)</name>
   <shortName>one_word_short</shortName>
   <version>#</version>
   <author>Author's Name (as it appears)</author>
   <description>Description of the mod (as it appears)</description>
   <download direct="true|false">http://download.link</download>
   <editorRating>0-5</editorRating>
</mod>

<!--
   GUIDELINE:
   type = The catigory to sort the mod into, either tech, tileset, scenario, map, or other.
   name = The proper name of the mod. This is as it appears, so remember proper case, etc;
   shortName = The name used to look up user's scores. MUST BE UNIQUE!!!
   version = The version number of the faction. Newer versions will overwrite older ones.
   author = The creator(s) of the mod, as they appear, so remember case.
   description = A description for the mod, as it appears, so remember punctuation and case!
   download = The download link. Direct is a boolean value for wether or not the download is a
                        direct download that starts on its own, or one that has a download page.
    editorRating = Place your IMPARITIAL rating (necessary or the ratings are ALL screwed up)

    Standard XML rules should be followed. To ensure proper parsing, make each element SINGLE LINE!
    Also, use HTML entities &lt;, &gt;, and &amp; for the corresponding symbols.
   
    Failure to create a valid XML may result in your mod not being updated or posted in the MDC.
-->
However, it can easily be changed or expanded, since it was only made to display things needed in the current MDC. I would think that it would have to be a "browser" style program to download though, since PHP recursively reads through the folders scanning the XMLs in a manner simular to how Glest reads a techtree. I'm also planning to change the XML parser to improve on its parsing abilities. There is a web submission form though, as those who have seen the recent MDC should know.

[-Archmage-]

This is half technobabble.......I'm gonna throw my thoughts out there into the technobabble and see what happens.

GUI: There would be an Addons button, that would have tabs for Mods, Maps, Tilesets, Scenarios, and Updates. Scenarios would have listed prerequisites, that would automatically be downloaded and installed with the scenario. The Addons button would blink(or do something) to notify an update or multiple updates are availble. All tabs would display a list that could be sorted alphabetically, by most downloads, and other things like size. Updates tab would be empty unless there is an update available(could have the update tab blink as well), and if there is an update available it would tell you your current version, and the latest version. The Mods, Maps, and Tilesets tabs would have preview images that could be clicked on to see larger. Maps could have a recommended tileset. For the sake or browsing I think each addon should have a little arrow at the far left that would open and close them, like in the older file managers with the plus and minus. Closed would be maybe ¹/₂ of an inch tall, enough room for the map's name and other basic info. Open would be as tall as needed to display a description, preview images, and other info. A Download&Install button could be in the open mode as well? A 'refresh lists' button could be added to get the latest info. And possibly a setup tab with setup options for things like automatically downloading updates and stuff like that.

Structure: A server would hold the content. GAE would check the server on startup, and update a config(cfg) file which would hold all the list data. When a download is requested, GAE would get the link from the config file and download the content and any prerequisites.

Updating System: GAE would check your version and check the server's latest. GAE would then update step-by-step, through each version on the server to the latest.
                                    Example: Your version: 2.8    Server version: 2.9.1     Updating: 2.8 --> 2.9 --> 2.9.1
Example with major version change: Your version: 2.9    Server version: 3.1.1     Updating: 2.9 --> 3.0 --> 3.1 --> 3.1.1   Notice that patch 2.9.1 is not installed.
Or course the server wouldn't always be able to hold all the versions of the mod, and if someone had an old version of something and it couldn't be patched up to the latest version, the latest full release would have to be installed and patched up to latest.

What do you guys think? Missing anything?

[Omega]

This is half technobabble.......I'm gonna throw my thoughts out there into the technobabble and see what happens.

GUI: There would be an Addons button, that would have tabs for Mods, Maps, Tilesets, Scenarios, and Updates. Scenarios would have listed prerequisites, that would automatically be downloaded and installed with the scenario. The Addons button would blink(or do something) to notify an update or multiple updates are availble. All tabs would display a list that could be sorted alphabetically, by most downloads, and other things like size. Updates tab would be empty unless there is an update available(could have the update tab blink as well), and if there is an update available it would tell you your current version, and the latest version. The Mods, Maps, and Tilesets tabs would have preview images that could be clicked on to see larger. Maps could have a recommended tileset. For the sake or browsing I think each addon should have a little arrow at the far left that would open and close them, like in the older file managers with the plus and minus. Closed would be maybe ¹/₂ of an inch tall, enough room for the map's name and other basic info. Open would be as tall as needed to display a description, preview images, and other info. A Download&Install button could be in the open mode as well? A 'refresh lists' button could be added to get the latest info. And possibly a setup tab with setup options for things like automatically downloading updates and stuff like that.

Structure: A server would hold the content. GAE would check the server on startup, and update a config(cfg) file which would hold all the list data. When a download is requested, GAE would get the link from the config file and download the content and any prerequisites.

Updating System: GAE would check your version and check the server's latest. GAE would then update step-by-step, through each version on the server to the latest.
                                    Example: Your version: 2.8    Server version: 2.9.1     Updating: 2.8 --> 2.9 --> 2.9.1
Example with major version change: Your version: 2.9    Server version: 3.1.1     Updating: 2.9 --> 3.0 --> 3.1 --> 3.1.1   Notice that patch 2.9.1 is not installed.
Or course the server wouldn't always be able to hold all the versions of the mod, and if someone had an old version of something and it couldn't be patched up to the latest version, the latest full release would have to be installed and patched up to latest.

What do you guys think? Missing anything?

Some of that stuff will appear in the next MDC, such as sortable tables and tabbed organization (I'm drastically changing the layout of the Glest Guide itself to have drop down menus instead of some growing ToC on the first page). I'm thinking one problem with this, though, is that modders don't always use very consistent version numbers (if its even a number). For optimal usage, ALL modders will have to learn to adopt standardized version numbering, commonly accepted as A.B.C, where A is the major version, B is the minor version, and C is the revision (optional, and less commonly used). While letters such as a, b, and rc are commonly accepted in version numbering, they make too much of a hassle and should instead simply increase the revision (ie: instead of 2.12a, use 2.12.1). It's a very bad idea to name versions a full word, such as Gimli, since there's no way at all to know if version ASKFLJ is newer or older than version AFJASKF... This ensures its easy to make sure you have the latest version, and numbers are ALWAYS easier to understand than letters.

http://en.wikipedia.org/wiki/Software_versioning

[-Archmage-]

This is half technobabble.......I'm gonna throw my thoughts out there into the technobabble and see what happens.

GUI: There would be an Addons button, that would have tabs for Mods, Maps, Tilesets, Scenarios, and Updates. Scenarios would have listed prerequisites, that would automatically be downloaded and installed with the scenario. The Addons button would blink(or do something) to notify an update or multiple updates are availble. All tabs would display a list that could be sorted alphabetically, by most downloads, and other things like size. Updates tab would be empty unless there is an update available(could have the update tab blink as well), and if there is an update available it would tell you your current version, and the latest version. The Mods, Maps, and Tilesets tabs would have preview images that could be clicked on to see larger. Maps could have a recommended tileset. For the sake or browsing I think each addon should have a little arrow at the far left that would open and close them, like in the older file managers with the plus and minus. Closed would be maybe ¹/₂ of an inch tall, enough room for the map's name and other basic info. Open would be as tall as needed to display a description, preview images, and other info. A Download&Install button could be in the open mode as well? A 'refresh lists' button could be added to get the latest info. And possibly a setup tab with setup options for things like automatically downloading updates and stuff like that.

Structure: A server would hold the content. GAE would check the server on startup, and update a config(cfg) file which would hold all the list data. When a download is requested, GAE would get the link from the config file and download the content and any prerequisites.

Updating System: GAE would check your version and check the server's latest. GAE would then update step-by-step, through each version on the server to the latest.
                                    Example: Your version: 2.8    Server version: 2.9.1     Updating: 2.8 --> 2.9 --> 2.9.1
Example with major version change: Your version: 2.9    Server version: 3.1.1     Updating: 2.9 --> 3.0 --> 3.1 --> 3.1.1   Notice that patch 2.9.1 is not installed.
Or course the server wouldn't always be able to hold all the versions of the mod, and if someone had an old version of something and it couldn't be patched up to the latest version, the latest full release would have to be installed and patched up to latest.

What do you guys think? Missing anything?

Some of that stuff will appear in the next MDC, such as sortable tables and tabbed organization (I'm drastically changing the layout of the Glest Guide itself to have drop down menus instead of some growing ToC on the first page). I'm thinking one problem with this, though, is that modders don't always use very consistent version numbers (if its even a number). For optimal usage, ALL modders will have to learn to adopt standardized version numbering, commonly accepted as A.B.C, where A is the major version, B is the minor version, and C is the revision (optional, and less commonly used). While letters such as a, b, and rc are commonly accepted in version numbering, they make too much of a hassle and should instead simply increase the revision (ie: instead of 2.12a, use 2.12.1). It's a very bad idea to name versions a full word, such as Gimli, since there's no way at all to know if version ASKFLJ is newer or older than version AFJASKF... This ensures its easy to make sure you have the latest version, and numbers are ALWAYS easier to understand than letters.

Yea, a standard numbering system should be enforced.

[will]

Code: [Select]

<mod>
   <type>tech|tileset|scenario|map|other</type>
   <name>Mod Name (as it appears)</name>
   ...
</mod>


Can a mod contain more than one type of content?  How is this specified?

[Yggdrasil]

There's no type. An addon can contain anything and most mods have various content. To split these would really need dependency checking as scenarios need a certain map, and so on. And i don't like dependency checking as it's difficult to get the addons mounted in the right order with physfs, in case there are overwrites between addons.

[Omega]

Code: [Select]

<mod>
   <type>tech|tileset|scenario|map|other</type>
   <name>Mod Name (as it appears)</name>
   ...
</mod>


Can a mod contain more than one type of content?  How is this specified?

The type is used to categorize the mod. I will probably have to add a "pack" value to the type. Currently, pack's like the MegaPack are sorted as a tech, since the tech is the primary feature of it.

[will]

yes, I asked because that confused me.

I'm thinking we need dependency management, and then make physfs mount in the correct order.

I've worked with writing really big systems, and one simple versioning approach that I've seen work very well is to use major.minor numbers in the folder names.  To illustrate:

mods\MRise.1.2.zip\ <-- a zip file, therefore something downloaded and versioned by our new mod installer
mods\MRise.1.3\ <-- not a zip, therefore this is something this modder is making now

The modder runs the packager tool we write.  This tool finds the MRise.1.3 folder, and says "would you like to release version 1.3 of this mod?"

It then walks the files in it, looking for dependencies.  Any dependencies in MRise.1.2 will get copied into the 1.3 zip that is uploaded, other dependencies in other non-MRise folders are allowed, but are recorded in a config file in the new 1.3 zip, rather than copied.  So the packager records what dependencies - other mods - a mod has, and also what it publishes in the sense of maps, scenarios, tilesets, etc.  A 1.3 zip is made, uploaded, the MRise.1.3\ folder is renamed MRise.1.4\ and the zip then placed there.  So the modder moves on a version and carries on modding.

You can push this further, e.g. mods\MRise.1.3.ChazyFraction.0.6.zip would be a mod of a mod, which requires MRise.1.3 (or perhaps later) and so on.  Only modders and the curious see these files; game users just see nice names.

But fundamentally, the idea that things in addons\ are mounted in any order and can replace anything isn't holding.  We'd make the mods\ folder not work this way, but rather follow a strict hierarchy.

Obviously there are lots of finer details and tweaks; but what do people think about this general approach to encapsulating mods into fixed-name folders?

[Omega]

It has to be a zip file to download though. You can't download an entire folder. It needs to be an archive. 7z is better than zip, and hopefully will be supported in the future.

[ChupaReaper]

Well I already sort my mod into two parts: MRise_CoreFiles_v1.6.zip which contains the main menu changes, tilesets, maps, scenarions (when I make them) and the tech directory with the resources and mrise.xml.
I then release my factions separate, at the minute only one faction is out as MRise_Necribus_v1.6.zip.
Because maps, tilesets, etc don't take up as much as factions I release them all as Core Files.

[will]

I am writing a simple test script (to learn the mod structure better, rather than to suggest this is production code) to find all dependencies for a mod.

Just to check I'm not going mad:

It says that vbros_pack_1 is missing some of the textures for pine_woods tileset; and indeed I cannot use that tileset in GAE because it also complains some of the textures are missing.

In Glest/MG, is there a surface2e.bmp etc that is it is 'inheriting' that isn't in GAE, or have I installed it wrong, or is it broken?

[Omega]

I am writing a simple test script (to learn the mod structure better, rather than to suggest this is production code) to find all dependencies for a mod.

Just to check I'm not going mad:

It says that vbros_pack_1 is missing some of the textures for pine_woods tileset; and indeed I cannot use that tileset in GAE because it also complains some of the textures are missing.

In Glest/MG, is there a surface2e.bmp etc that is it is 'inheriting' that isn't in GAE, or have I installed it wrong, or is it broken?

I'm not familiar with that tileset, but there's no such thing as texture inheriting (yet, and hopefully won't be. I don't want modders to get dependant on that). Most likely your download is shot or it was somehow broken otherwise... Oh, and the best question: is the image there?

Yeah, scenarios are a good place to start, since they're the only things that need dependancies (with the exception of the old FPM, which saves on filesize by using regular Magitech's models for the same units).

[will]

It really doesn't seem to be there

can anyone else verify that the pine woods tileset in vbros 1 is broken?

(In other news, g2xml doesn't support the old version 3 g3d files; I worked around it but thought it worth mentioning)

[silnarm]

... And i don't like dependency checking as it's difficult to get the addons mounted in the right order with physfs, in case there are overwrites between addons.

If one addon depends on another, then it shouldn't be overwriting anything from the other addon.  While it would certainly be possible to do things this way, I suspect most addons will be largely self contained, or rely on other addons for maps, or a tech, or whatever.

I don't think dependency checking, and auto download/upgrade of dependencies, should be worried about at first, but I do think it would be nice to include the XML mechanism from the start. This way the addons or repository wont need to be changed later if we do add such support, and in the mean time this information can at least be displayed to the would-be downloader.

It really doesn't seem to be there

can anyone else verify that the pine woods tileset in vbros 1 is broken?

Glest had a bug with file name matching, it's not inheriting it from anywhere, it's just matching a different texture (probably surface2.bmp if there is one, else maybe surface2a.bmp).

There error is indeed in the mod, not GAE.

[will]

Yes, I was being very careful to suggest we 'silo' mods - that the mods installed would not be able to implicitly rely upon other content.

Are people comfortable with the zip-folder-with-version-number approach to containing mods?