So, as long as the add-on in question replicates the file structure properly, I believe all a downloader would need to do is fetch the add-on from a server and put it in the addons folder, and maybe check for updates from said server.
This would be good, but I think some projects should be able to go directly into the glestae directory, perhaps whatever server the mod is on can store whether the mods are saved into addons or into glestae, this is incase you get addons for project like mine which is basically a separate game that uses the gae engine, addons may be made just for my project rather than Glest, etc.In what situation would there be a benefit to putting it in the main directory instead of addons?
https://forum.megaglest.org/index.php?topic=6002.msg62656#msg62656I forgot to mention that there's a parameter in an XML somewhere that says something like <total-conversion value=""/>. I'm not sure what all it does, though.
Oh, and explain how we manage mods that add fractions vs mods that use the engine for alternative games...
245 files added: 41,706,266 bytes
300 files unchanged: 95,469,241 bytes
11 files changed: 7,958,481 bytes
96 files deleted: 19,597,478 bytes
Is this fairly typical? Also, is there ever much renaming of files without changing their contents? (I did not check for this in the script I wrote to get those numbers)Malevolent Rising is not representative of the typical mods we have around here. Most are much smaller. Renaming isn't all that rare -- even just in Magitech, the Behemoth was formerly called a Daemon Giant and the Guard used to be called an Armored Swordsman. Plus, we seem to have a lot of people around here who can't spell, so that sometimes gets corrected with subsequent versions.
245 added: 39.8 MB
9 renamed: 1.6 MB
300 unchanged: 91.0 MB
11 changed: 7.6 MB
87 deleted: 17.1 MB
471 previously: 117.3 MB -> 60.2 MB zipped -> 50 MB 7z/rar
556 now: 148.7 MB -> 78.1 MB zipped -> 64 MB 7z/rar
delta: -> 27.0 MB zipped -> 19 MB 7z
I have found this thread on the mailing list:
...
Is this already going somewhere?
#!/bin/sh
if [ $# -eq 4 ]; then
if [ "$1" = "create" ]; then
size1=$(ls -l $2 | cut -d " " -f 5)
size2=$(ls -l $3 | cut -d " " -f 5)
if [ $size1 -lt $size2 ]; then
size1=$size2
fi
xdelta3 -e0 -B $size1 -W 16777216 -I 0 -s $2 $3 $4
xz -9 $4
elif [ "$1" = "patch" ]; then
unxz -k $3
xdelta3 -d -s $2 ${3/.xz/} $4
rm ${3/.xz/}
else
echo "usage: $0 create old new patchfile"
echo "or: $0 patch old patchfile new"
fi
else
echo "usage: $0 create old new patchfile"
echo "or: $0 patch old patchfile new"
fi
Glest itself just needs to apply these patches if we really want to integrate it directly. The prototype only does that but it's still a bit buggy. libcurl is also not the right choice as it can't pause and resume downloads. The list of available addons on the server is saved in a simple CSV file which was handwritten for this simple test.Another problem which may arise is if one mod depends on another. mods need to check for dependencies in other mod files and not load if the prerequisite files are not loaded.I don't really think we need that and it makes it just more complicated. Dependency checking isn't trivial and involves too much work. We even don't need patches for the first version.
I feel uncomfortable with a tar+xdelta approach and would prefer an approach where the xml of a mod was parsed in the same way the game does, all dependencies located, and all the appropriate bits getting zipped up along with meta information. It might be that the end result is an archive, but the way to get there is quite different and more involved.Isn't this the modder's work? And what meta information?
That's what basically the prototype does. I don't see any benefit in your approach and i actually don't really understand what you want to achieve.
...I wanted to work on it after adding the addons menu where i'm currently stuck at the gui code...
On another note, how would a diff work? What if the user had v1 and wanted to update to v1.2 (skipping a version). I'd think there'd have to be 2 versions of the upgrade (which is what SMF, this forum, uses). Basically, there's a small upgrade which upgrades from the last version to the most recent, and there's the large upgrade, which contains all the files changed since the last major release (for example, Military's next version, 2.5, would have a small upgrade to upgrade from 2.4 to 2.5, whereas the large upgrade could upgrade from any 2.x to 2.5. Older than that and you'd need a full install). On the web design side, there's just a few download options. Upgrades would only be available to download if the modder made them available.
1) code and submit to physfs tar support
2) the study of supporting multiple total-conversions and also mods that mod a total conversion
not just a mod installer, but we also need an auto-updater too!I had a topic where I was talking about making an updater but I can't find it now. One of the ideas was that you can start with only a single binary file and it would download anything that was needed to run.
only one of these is allowed at any one one timeI'm not so sure. What if you wanted to use a map from Military with the MRise tech tree and a tileset from Megapack, for example?
+1 :thumbup:only one of these is allowed at any one one timeI'm not so sure. What if you wanted to use a map from Military with the MRise tech tree and a tileset from Megapack, for example?
Adding tar support to physfs is of course possible; however, it is performance presenting problems; tar is a solid format.Mhmm, you're right here. Haven't thought about that.
My proposal: Simple CSV file with addon name, version number and checksum. A (very) short description would be nice and maybe a tiny logo (embedded with UUencode). It's not meant to inform you in detail about the mod, the forum is a better place for this, just enough to find what you're looking for.A link to the forum thread or wiki page would also be nice.
<repository>
<mods>
<mod>
<name>...</name>
<version>...</version>
<description>...</description>
<checksum>...<checksum>
<logo>...UUencoded 32x32 PNG (optionally)...</logo>
</mod>
...
</mods>
<patches>
<patch>
<mod>...modname...</mod>
<oldversion>...patches from this version to the newer one above...</oldversion>
<checksum>...</checksum>
</patch>
...
</patches
</repository>
79M mrise_1.6.zip
149M mrise_1.6.zip0
58M mrise_1.6.zip0.xz
61M mrise_1.0.zip
118M mrise_1.0.zip0
45M mrise_1.0.zip0.xz
45M mrise_zip0.xdelta
19M mrise_zip0.xdelta.xz
26M mrise_zip0.xdelta.zip
27M mrise_zip.xdelta
Using compressed zips makes it easier for the program and is smaller on disk for the user but uses more space and bandwidth for the hoster. I'm not sure what's better, depends on the webspace we want to use. Opinions?Meh, I still plan to create the website end as I proposed as the next MDC...One could still use your "web platform" idea to maintain this list.
<mod>
<type>tech|tileset|scenario|map|other</type>
<name>Mod Name (as it appears)</name>
<shortName>one_word_short</shortName>
<version>#</version>
<author>Author's Name (as it appears)</author>
<description>Description of the mod (as it appears)</description>
<download direct="true|false">http://download.link</download>
<editorRating>0-5</editorRating>
</mod>
<!--
GUIDELINE:
type = The catigory to sort the mod into, either tech, tileset, scenario, map, or other.
name = The proper name of the mod. This is as it appears, so remember proper case, etc;
shortName = The name used to look up user's scores. MUST BE UNIQUE!!!
version = The version number of the faction. Newer versions will overwrite older ones.
author = The creator(s) of the mod, as they appear, so remember case.
description = A description for the mod, as it appears, so remember punctuation and case!
download = The download link. Direct is a boolean value for wether or not the download is a
direct download that starts on its own, or one that has a download page.
editorRating = Place your IMPARITIAL rating (necessary or the ratings are ALL screwed up)
Standard XML rules should be followed. To ensure proper parsing, make each element SINGLE LINE!
Also, use HTML entities <, >, and & for the corresponding symbols.
Failure to create a valid XML may result in your mod not being updated or posted in the MDC.
-->This is half technobabble.......I'm gonna throw my thoughts out there into the technobabble and see what happens.Some of that stuff will appear in the next MDC, such as sortable tables and tabbed organization (I'm drastically changing the layout of the Glest Guide itself to have drop down menus instead of some growing ToC on the first page). I'm thinking one problem with this, though, is that modders don't always use very consistent version numbers (if its even a number). For optimal usage, ALL modders will have to learn to adopt standardized version numbering, commonly accepted as A.B.C, where A is the major version, B is the minor version, and C is the revision (optional, and less commonly used). While letters such as a, b, and rc are commonly accepted in version numbering, they make too much of a hassle and should instead simply increase the revision (ie: instead of 2.12a, use 2.12.1). It's a very bad idea to name versions a full word, such as Gimli, since there's no way at all to know if version ASKFLJ is newer or older than version AFJASKF... This ensures its easy to make sure you have the latest version, and numbers are ALWAYS easier to understand than letters.
GUI: There would be an Addons button, that would have tabs for Mods, Maps, Tilesets, Scenarios, and Updates. Scenarios would have listed prerequisites, that would automatically be downloaded and installed with the scenario. The Addons button would blink(or do something) to notify an update or multiple updates are availble. All tabs would display a list that could be sorted alphabetically, by most downloads, and other things like size. Updates tab would be empty unless there is an update available(could have the update tab blink as well), and if there is an update available it would tell you your current version, and the latest version. The Mods, Maps, and Tilesets tabs would have preview images that could be clicked on to see larger. Maps could have a recommended tileset. For the sake or browsing I think each addon should have a little arrow at the far left that would open and close them, like in the older file managers with the plus and minus. Closed would be maybe 1/2 of an inch tall, enough room for the map's name and other basic info. Open would be as tall as needed to display a description, preview images, and other info. A Download&Install button could be in the open mode as well? A 'refresh lists' button could be added to get the latest info. And possibly a setup tab with setup options for things like automatically downloading updates and stuff like that.
Structure: A server would hold the content. GAE would check the server on startup, and update a config(cfg) file which would hold all the list data. When a download is requested, GAE would get the link from the config file and download the content and any prerequisites.
Updating System: GAE would check your version and check the server's latest. GAE would then update step-by-step, through each version on the server to the latest.
Example: Your version: 2.8 Server version: 2.9.1 Updating: 2.8 --> 2.9 --> 2.9.1
Example with major version change: Your version: 2.9 Server version: 3.1.1 Updating: 2.9 --> 3.0 --> 3.1 --> 3.1.1 Notice that patch 2.9.1 is not installed.
Or course the server wouldn't always be able to hold all the versions of the mod, and if someone had an old version of something and it couldn't be patched up to the latest version, the latest full release would have to be installed and patched up to latest.
What do you guys think? Missing anything? 8)
This is half technobabble.......I'm gonna throw my thoughts out there into the technobabble and see what happens.Some of that stuff will appear in the next MDC, such as sortable tables and tabbed organization (I'm drastically changing the layout of the Glest Guide itself to have drop down menus instead of some growing ToC on the first page). I'm thinking one problem with this, though, is that modders don't always use very consistent version numbers (if its even a number). For optimal usage, ALL modders will have to learn to adopt standardized version numbering, commonly accepted as A.B.C, where A is the major version, B is the minor version, and C is the revision (optional, and less commonly used). While letters such as a, b, and rc are commonly accepted in version numbering, they make too much of a hassle and should instead simply increase the revision (ie: instead of 2.12a, use 2.12.1). It's a very bad idea to name versions a full word, such as Gimli, since there's no way at all to know if version ASKFLJ is newer or older than version AFJASKF... This ensures its easy to make sure you have the latest version, and numbers are ALWAYS easier to understand than letters.
GUI: There would be an Addons button, that would have tabs for Mods, Maps, Tilesets, Scenarios, and Updates. Scenarios would have listed prerequisites, that would automatically be downloaded and installed with the scenario. The Addons button would blink(or do something) to notify an update or multiple updates are availble. All tabs would display a list that could be sorted alphabetically, by most downloads, and other things like size. Updates tab would be empty unless there is an update available(could have the update tab blink as well), and if there is an update available it would tell you your current version, and the latest version. The Mods, Maps, and Tilesets tabs would have preview images that could be clicked on to see larger. Maps could have a recommended tileset. For the sake or browsing I think each addon should have a little arrow at the far left that would open and close them, like in the older file managers with the plus and minus. Closed would be maybe 1/2 of an inch tall, enough room for the map's name and other basic info. Open would be as tall as needed to display a description, preview images, and other info. A Download&Install button could be in the open mode as well? A 'refresh lists' button could be added to get the latest info. And possibly a setup tab with setup options for things like automatically downloading updates and stuff like that.
Structure: A server would hold the content. GAE would check the server on startup, and update a config(cfg) file which would hold all the list data. When a download is requested, GAE would get the link from the config file and download the content and any prerequisites.
Updating System: GAE would check your version and check the server's latest. GAE would then update step-by-step, through each version on the server to the latest.
Example: Your version: 2.8 Server version: 2.9.1 Updating: 2.8 --> 2.9 --> 2.9.1
Example with major version change: Your version: 2.9 Server version: 3.1.1 Updating: 2.9 --> 3.0 --> 3.1 --> 3.1.1 Notice that patch 2.9.1 is not installed.
Or course the server wouldn't always be able to hold all the versions of the mod, and if someone had an old version of something and it couldn't be patched up to the latest version, the latest full release would have to be installed and patched up to latest.
What do you guys think? Missing anything? 8)
Code: [Select]<mod>
<type>tech|tileset|scenario|map|other</type>
<name>Mod Name (as it appears)</name>
...
</mod>
The type is used to categorize the mod. I will probably have to add a "pack" value to the type. Currently, pack's like the MegaPack are sorted as a tech, since the tech is the primary feature of it.Code: [Select]<mod>
<type>tech|tileset|scenario|map|other</type>
<name>Mod Name (as it appears)</name>
...
</mod>
Can a mod contain more than one type of content? How is this specified?
I am writing a simple test script (to learn the mod structure better, rather than to suggest this is production code) to find all dependencies for a mod.I'm not familiar with that tileset, but there's no such thing as texture inheriting (yet, and hopefully won't be. I don't want modders to get dependant on that). Most likely your download is shot or it was somehow broken otherwise... Oh, and the best question: is the image there?
Just to check I'm not going mad:
It says that vbros_pack_1 is missing some of the textures for pine_woods tileset; and indeed I cannot use that tileset in GAE because it also complains some of the textures are missing.
In Glest/MG, is there a surface2e.bmp etc that is it is 'inheriting' that isn't in GAE, or have I installed it wrong, or is it broken?
... And i don't like dependency checking as it's difficult to get the addons mounted in the right order with physfs, in case there are overwrites between addons.
It really doesn't seem to be there
can anyone else verify that the pine woods tileset in vbros 1 is broken?
can anyone else verify that the pine woods tileset in vbros 1 is broken?I would recommend against using anything from the vbros for testing. Their "quality assurance" is non-existent, and releases are frequently broken in some way or another (like inconsistent spelling between the file names and XML references).
It has to be a zip file to download though. You can't download an entire folder. It needs to be an archive. 7z is better than zip, and hopefully will be supported in the future.We've already got 7z in recent revisions, but it only works for some types -- something about solid vs. non-solid, and various other things I don't understand.
...
Ignoring tilesets/autumn/models/grass.tga 768.0 KB
Ignoring techs/megapack_v4/factions/persian/units/elephant_cage/sounds/tech_building_fall5.wav 148.6 KB
Ignoring techs/megapack_v4/factions/elves/units/tree_of_life/construction_models/tree1.g3d 1.1 KB
Ignoring techs/megapack_v4/factions/elves/units/wood_hall/images/wood_hall.bmp 12.1 KB
Ignoring techs/megapack_v4/factions/indian/units/thunderbird/sounds/eagle_fly2.wav 426.0 KB
=== Ignored: 675 121.3 MB ===
=== Included: 4681 587.2 MB ===
=== Models: 922 252.7 MB ===
=== Textures: 1040 183.1 MB ===
=== Particles: 120 83.5 KB ===
=== Sounds: 2360 149.9 MB ===
=== Units: 172 824.1 KB ===
=== Upgrades: 53 34.8 KB ===
=== Resources: 7 1.2 KB ===
=== Fractions: magic, norsemen, dark_magic, indian, persian, elves, tech, woodsmen ===
=== Maps: overgrown_city, tt_goldrush_v2, forest_for_four, waterworld, tropical_arena, domination_isles, one_on_one, conflict ===
=== Tile Sets: evergreen, desert2, winter, autumn, jungle, hell ===
=== Tech Trees: megapack_v4 ===
=== Scenarios: m4_kill_the_magic, m4_my_island, m4_beehive_castle, m4_amazones, m4_norsemen_attack, m4_persian_conflict, m4_indian_attack, m4_waterworld_light, m4_waterworld, m4_magic_can_do_it, m4_amazones_light ===
=== Mod check failed ===
(model BROKEN techs/megapack_v4/factions/persian/units/swordman/models/guard_attacking.g3d (referenced by: unit ../swordman.xml)) ('Error reading G3D file', IOError(2, 'No such file or directory')), ('File does not exist',)
(model BROKEN techs/megapack_v4/factions/tech/units/swordman/models/guard_attacking.g3d (referenced by: unit ../swordman.xml)) ('Error reading G3D file', IOError(2, 'No such file or directory')), ('File does not exist',)
(particle BROKEN techs/megapack_v4/factions/persian/units/worker/particle_splash.xml (referenced by: unit worker.xml)) ('File does not exist',)
(scenario BROKEN scenarios/m4_amazones/m4_amazones.xml) ('References missing map', u'in_the_forest'), ('References missing tileset', u'forest')
(scenario BROKEN scenarios/m4_amazones_light/m4_amazones_light.xml) ('References missing map', u'in_the_forest'), ('References missing tileset', u'forest')
(sound BROKEN techs/megapack_v4/factions/norsemen/units/mistletree/sounds/tower_attack1.wav (referenced by: unit ../mistletree.xml)) ('File does not exist',)
(sound BROKEN techs/megapack_v4/factions/norsemen/units/mistletree/sounds/tower_attack2.wav (referenced by: unit ../mistletree.xml)) ('File does not exist',)
...
Heres a question: If I have two TC mods which both modify the main_menu.g3d how does gae/physfs decide which one shows up at runtime?The one first in the search path. The order of this isn't well defined, so it could be different on different platforms.
GAE should NOT be bundled with the Megapack, for one simple reason: It's too darn big! I'm always a few versions behind on MG, and if there's no binary only version uploaded, I don't even download until there is.Well, I guess it is kinda big, but not that much...and I guess you download each version of MG then, cause with each version there is always a binary only version... :| ;)
Regarding 1. ( updating GAE ) is the most tricky thing in my opinion, especially regarding different platforms. I am curious how this will work! I for myself have no really practicable idea at the moment.
To my mind GAE should not be bundled with the existing old Glest game data, but rather pre-installed with a Glest+MegaPackV4 mod in the new mod structure.
Furthermore, I think (2) needs dependency checking from the get-go. And it doesn't hurt
In my dependency prototype I ignored the overriden 'core' files, because we need some agreement on how to handle them. I think they should go into a set folder within the mod archive, and that it would be much more robust if there was an XML file explicitly saying which artwork overrides whatever rather than an implicit naming system.
And on the client-side, I imagine a stand-alone wx-widgets app that does the downloading. From the beginning the downloader code however should be in the shared lib, with the full intention of eventually bringing it into the game UI.
In the game UI, there should be a top-level menu option for selecting between (1) mods, which naturally swap out all the UI artwork. The most-recent choice is remembered. But in the
<manifest type="core-data" name="malevolent_rising" version="1.6">
<!-- This pack contains custom tilesets and maps, turn default stuff off -->
<default-data>
<techs value="false" />
<tilesets value="false" />
<maps value="false" />
</default-data>
</manifest>
<manifest type="content" name="necribus_faction" version="1.6">
<dependencies>
<dependency name="malevolent_rising" version="1.6">
</dependencies>
</manifest>
The mods are more like classic glest extensions as opposed to total-conversions, and the distinction is something that will need GAE code changes with the TC picker work. I say that TCs are version 2!
Of these, the second seems to depend on the first, so right now we need a website. So who wants to help on the website and how do we go about hosting?