common data is a good or a bad idea?

[titi]

The current Megaglest version introduced a new idea called common data.
At the moment common sounds are all packed plain in one directory and referenced from each unit.xmls to avoid duplicates.

This has the advantage that this saves disc space because duplicate sound files are eliminated and the megapack is smaller by this, ( yes, maybe a lot smaller )

I thought a lot about this the last days and I am more and more coming to the conclusion that this maybe is a bad idea:
Imagine you want to create a new techtree and you want one megapack faction in your techtree. This is what was for example done for the imperial faction. Up to now it was easy to move one faction from one techtree to another ( assuming that the techtrees have somehow similar xmls/ressources ).
This is much more complicated now or you always have to move the full common data folder too. (Regarding only one unit it gets even more worse......)
The megapack is our reference at the moment which also teaches others how to do things and this will make it a bit more complicated for them to start.
Beside of this the common soundfiles are all plain in one directory now. A big mix of units sounds and its hard to see which sound file is used in which unit.

All in all I say:
"Size doesn't always matter" 

What do you think about this?

[will]

The proper way to do this is the same way package dependencies are managed in, say, linux: each mod can have dependencies.

I made a script for managing it, but it hasn't be adopted.

The concept is sound though:

The name is made of elements separated by periods.
The name of a mod containing a tech-tree is the first name, e.g. Military
The version of the mod then follows e.g. Military.1.2
If this is an extension and does not contain a tech-tree, then
the name of the extension follows e.g. Military.1.2.CyberStorm
These extensions also need version numbers e.g. Military.1.2.CyberStorm.0.3
Extensions can themselves be extended...

the sourcecode

This allows mods of mods, and mods that add factions to an existing mod, and such.  It is straightforward for the game to understand the dependencies of a mod, and to create the appropriate physfs mounts in the appropriate orders.
.
And the naming convention is human readable too.

It is easily forgotten that copied artwork not only takes up disk space unnecessarily, it also takes up RAM unnecessarily during gameplay.

[titi]

? I think we talk about different things here and beside of this we have no physfs in MG.

( "linking" factions is a completly different problem! )

[will]

*cough* merge?

you don't want to be adding things at this late stage to MG that conflict with GAE, right?

We right now have a download centre in MG where Imperial has a copy of MagiTech and we have Japanese that can only fight themselves.  If you tweak magic in megapack then imperial won't magically get the new version.

Project Red is interesting because it is possible to download the latest version always direct from github.  It could be a way forward, explicitly using git and submodules to manage the whole game assets for all modders.

The big problem to be solved here is how to distribute mods and such.  A discussion about the internal organisation of megapack impacts nobody because at the moment nobody can make a freestanding faction, they must make a techtree.

[titi]

I think you still don't get it. This has NOTHING to do with code! This is a data problem!
And the problem I talk about is NOT referencing factions!

[will]

code is the appropriate way to solve data problems.

worrying that its not obvious to unspecified new modders - in the few months window between now and Glest 4 - that they have to copy some common data folder is ignoring the elephant in the room - that the new MG mod-download code doesn't allow mods to add factions to existing techtrees.  These unspecified modders wouldn't be copying and duplicating the whole techtree literally byte-for-byte if they could add their factions to an existing techtree like magitech.

Ever wanted to play the vbros factions against, say, tech?  I did.

[titi]

ok I give up, this discussion is completly offtopic now. Why don't you start a topic where you talk about the faction problem and I can talk here about my intial problem?

Maybe I should split the topic directly before your firts post and you add some more text to your first post?
I am really willing to discuss all this, but not here and not inside this topic, because it deals with something different.

[Ishmaru]

I have a similar system done before the common data folder was created, for the same reasons the common data folder was created, to reduce size. I may not understand much bout programing, but wouldn't having shared data decrease the amount of ram needed to play, thus making game run better?

However, you are entirely right about making it harder for beginner modders to make mods based on factions in the megapack, not because of the extra work but because of the confusion caused by having data in other directories.  Not to mention wasting space having to copy sounds and textures that aren't even used in the faction
you are trying to copy anyways.

I can see two compromises for having commondata:

1) have a version each faction with their own dependencies as a separate download, for mod use only. (still have megapack with common data with megaglest)

2) At least there should be a dependencies.txt or common data.txt (whatever name you want) that is in the directory of each faction that list what files the faction uses from the common data folder.

[sojourner]

Titi,
Where does it save space?

If it is about packaging (since you mentioned disk), then it might be easy to copy the files on install/make.

If the aim was to save disk space post install, a simple script/program can be provided to 'export' a techtree/faction into a place of choice where modders can begin work.

Another script/program can be used to package an optimized techtree by using a simple 'diff' style command internally to find and consolidate common data in one place automatically [if required, it can maintain information about which were the duplicate files and what is the new location where they are now etc. This information can be used to export a techtree/faction too].

However it is not as simple as copy and nothing ever will be, but I am sure small in general is good for both download and install and excellent while running for sure...

[Omega]

Common data: oppose, because you can already do this mod-wise. Example: in military, the sounds are all stored in the faction folder, not the unit folders. You just have to put the proper path in the XML. This even works with resources in an entirely different techtree such as how Project Green uses Magitech's models to save space. Though, for the most part, it would be a good thing to encourage modders to use their own resources instead of sharing them.

At any rate, to sum that up, I oppose because I don't see the point: it can already be done with a tried and true method that we've been using forever.

[John.d.h]

Common data: oppose, because you can already do this mod-wise. Example: in military, the sounds are all stored in the faction folder, not the unit folders. You just have to put the proper path in the XML.

How is this different from the common data thing?  (Maybe I'm just missing something. )

Project Green uses Magitech's models to save space.

That's actually a PhysFS thing.

Though, for the most part, it would be a good thing to encourage modders to use their own resources instead of sharing them.

I agree.  I feel like the "common data" approach is really dealing with the wrong problem.  The problem isn't that duplicate sounds are taking up extra space.  The problem is that there are so many duplicate sounds in the first place!  Get a few friends and a microphone.  It's amazing how much work we get done in models, textures, and animations, but leave sounds completely neglected because we (myself included) lack the expertise.

[Ishmaru]

Why does it have to be a "official" location y cant megapack have its common data folder if they want it and mods that dont - dont have to use it (it shouldn't be auto made either) .  I prefer my systems for my mods. I don't want to have to alter it to fit a standard...

To be honest i miss days before the application data folder was introduced. When everything may not have been efficient but was simple, intuitive, and easy. It may not seem so bad as of now but im afraid that glest is going to eventually loose its ease of modding, which is glest biggest feature, soon...

[Omega]

Project Green uses Magitech's models to save space.

That's actually a PhysFS thing.

I thought it might have been, but Four Path Magitech has used links like this since long before PhysFS was implemented, so not sure about that.

Common data: oppose, because you can already do this mod-wise. Example: in military, the sounds are all stored in the faction folder, not the unit folders. You just have to put the proper path in the XML.

How is this different from the common data thing?  (Maybe I'm just missing something. )

I may be the one missing something, perhaps... Though it seems to me that common data would be outside of the faction, where as my approach is meant only for that faction.