Show Posts

This section allows you to view all posts made by this member. Note that you can only see posts made in areas you currently have access to.


Messages - will

Pages: 1 ... 27 28 29 30 [31] 32
751
General discussion / Rendering G3D using OpenGL
« on: 9 December 2010, 07:38:58 »
If I interpret the code correctly, G3D models are rendered in the following way:

Each model is number of meshes.

Each mesh has a number of frames.

Each frame describes the coordinates of each vertex.  Each frame must contain the same number of vertices - if a wagon has a wheel that turns, then the wheel might have 10 frames, and each describes the wheel at a different point of rotation.  The position of the vertices for any given rotation are computed by the model making tools and not by glest.

When loaded in the glest engine, it is stored in a vertex array.

Each screen update, the appropriate frame is selected for each mesh, and the appropriate vertex array is drawn.

My immediate thought is this:

In modern OpenGL (which I am unfamiliar with; I did OpenGL in the old days) the vertex buffer would be a VBO and you'd have only one VBO for each mesh, and each frame of animation would be a transform described in a 'uniform'.

In classic OpenGL, as each frame of animation is immutable you'd place it in a Display List.  You would have only one Display List for the mesh, and describe each frame of animation as a matrix or, better, a series of transforms (rotate/translate/scale) to make the matrix.

In both the VBO/uniform and Display List/matrix approach, you can have tweening of frames very easily.

Now the classic Display List / Model Matrix transform is 'deprecated', but my understanding is it still goes like a rocket.

Am I retreading old ground that others have thought about and discarded?  Or does it make sense to move from vertex buffers to Display Lists, and to compute matrix transforms for the 'unrolled' frames in the G3D models?

(Background: http://www.gamedev.net/community/forums/topic.asp?topic_id=558817 Display Lists are 10x faster than the vertex arrays glest seems to currently be using...?  My expectation is that Display Lists are likely a bigger gain for typical users than those numbers given in that gamedev thread suggest.)

752
Feature requests / Re: integrated mod installer
« on: 9 December 2010, 06:32:36 »
So we have a mod packer script that basically seems to work, and does a tidy job.  Its a python script, but its serviceable.

The mods are more like classic glest extensions as opposed to total-conversions, and the distinction is something that will need GAE code changes with the TC picker work.  I say that TCs are version 2!  The immediate goal is to repack all the classic glest extensions and get them hosted on a website so we can test the packing and downloading part.

There are two directions to now work in:

  • A website, so the glest_mod_pack.py tool can upload the mods
  • A mod installing tool

Of these, the second seems to depend on the first, so right now we need a website.  So who wants to help on the website and how do we go about hosting?

753
Tools / Re: glest_mod_pack.py
« on: 1 December 2010, 14:03:19 »
thx for your patience!

the problem with the paths were a unix/windows thing with path separators, and I think I've fixed that now.

the problem with the missing magitech is because the naming convention of mods means that your undead mod should be named "magitech.1.undead.1" or whatever, so the tool can see it extends magitech (and thus shouldn't package a tech-tree xml)

How big was your mod when you packaged it yourself without the tool?  And how big is it with the tool?  (Is the optimising and stripping of 'dev' stuff saving much at all?)

Some mods I test with have very very large savings, others not so much...

754
Feature requests / Re: integrated mod installer
« on: 30 November 2010, 12:59:53 »
At the moment my mod packer puts some xml in as a zipfile comment.

I think I'll change it so that it writes a /glest-mod-manifest.xml to the mod file describing the dependencies.

If the modder provides a /glest-mod-manifest.xml then it will be sensibly merged with the dependencies that the mod-packer finds.

I like that on many platforms that use version numbers, 'major' version number changes introduce incompatibilities but 'minor' versions do not, so therefore the tool can believe that something depending on mymod.1.2 can work with mymod.1.3 but not mymod.2.1

The tool is coming along nicely, I'd ask everyone to kindly test it and give feedback.

755
Tools / Re: glest_mod_pack.py
« on: 30 November 2010, 08:55:47 »
The newest version of the tool can optimise mods so they are smaller zip downloads and take less space on user disks.

It can take several MB off mods that contain lots of duplication

756
Tools / Re: glest_mod_pack.py
« on: 28 November 2010, 10:36:46 »
Excellent info!

I like the idea of putting all the 'ignored' files into a ._dev.zip.xy too rather than ignoring them; possibly putting *everything* into it, rather than trying to police it and remove *~ and thumbs.db etc

I've fixed some bugs, including:

  • parsing <music> from fractions xml
  • coping with non-case-sensitive paths of unix (all packed files have lower-cased names)
  • finding all the particles in <particles> tags

757
Tools / Re: glest_mod_pack.py
« on: 27 November 2010, 21:42:22 »
I have updated the tool again.

It should now hopefully find all your particles.  Are there any particles it still doesn't find, and how are they specified in the xml?

The new version of the tool should cope with the tech tree xml being missing, as long as the name follows the convention that an extension filename specifies what it extends; in your case perhaps magitech.1.yourmod.1 ?

The tool does look for <music> in the fractions, so what xml file specifies the music that it isn't including?

The tool still doesn't package overrides of the core files for the total conversions.  Are these files all in a common directory, or some other rule like that I can follow?

For the files you want included that are not strictly part of the mod, would be ok to insist that they are in some subfolder e.g. /info ?  I could look for files in such a folder and include them all, always?


758
Tools / Re: glest_mod_pack.py
« on: 27 November 2010, 12:02:04 »
thank you wciow!

this is exactly the kind of detailed feedback I can use to fix the tool!

1) what are the rules for /music?  Are these files specified in any xml?

2) the problem with mods that extend others - as yours does 'magitech' is really a limitation of the tool, and I really need to fix this to make the tool marginally useful :)  My approach will be to package magictech as a mod, package megapackv4 as a mod that extends magitech, and then you can package a mod that extends either of those and the tool will check the dependencies correctly... that'll already be in the next version next week...

2) can you zip up the mod you were trying to package (not the zip that the tool makes) so I can recreate these problems, rather than asking you to test every fix I try?

Thanks very much again for running it!

759
Feature requests / Re: integrated mod installer
« on: 27 November 2010, 10:20:07 »

760
Tools / Re: glest_mod_pack.py
« on: 27 November 2010, 10:19:28 »
Thanks for trying it out; sorry it didn't work first time; it was intended to be run from the "command line"

I've updated it, so it now asks you to enter the path to your mod if you don't use the command line, so now it should work for you!

761
Tools / glest_mod_pack.py
« on: 26 November 2010, 13:15:15 »
https://github.com/williame/GlestTools

You need python - and only python - to run it.  If you want to compress best, you can also install the 'xz' command-line compressor and ensure that is in your path (or, on windows, in the same directory as you run the script from I think)

It is Glest / MG / GAE agnostic, and useful for packaging mods of all those formats.

It enforces a version numbering system in the naming convention of mods.

To use it, simply put your mod in a nice directory, ensure the name of the directory follows the convention (if it doesn't, you get told the naming convention!), and then use the tool instead of a normal zip program.

The idea is that there will soon be a website for hosting these mods, or at least meta-data for them, and that there will be a friendly installer for players to use to get new mods and do updates.

But in the meantime, right now, hopefully this packager is useful in making classic mod zipfiles!

762
Feature requests / Re: integrated mod installer
« on: 25 November 2010, 23:28:30 »
update: a log from megaglest with vbros 1 along side: https://gist.github.com/716064

Is this stuff really broken?

E.g. does egypt's air_pyramid really reference technician/sounds/technician_ack1.wav, is it really missing, and what does G/MG/GAE do when sounds are missing?

763
Maps, tilesets and scenarios / Re: Tileset: Fernland
« on: 25 November 2010, 19:17:35 »
Are these files needed?

tilesets/fernland/models/deadtree.tga 768.0 KB
tilesets/fernland/models/texture_big_stone_.tga 192.0 KB
tilesets/fernland/models/texture_bush.tga 256.0 KB
tilesets/fernland/models/totem1a.blend1 140.7 KB
tilesets/fernland/models/totem3.blend 147.3 KB

764
Feature requests / Re: integrated mod installer
« on: 25 November 2010, 18:51:57 »
I started this thread, I volunteered to over-engineer a mod installer, so its natural I post most, but I've been trying to get others to lead the way as I'm the newcomer.

From the first post we asked "how we manage mods that add fractions vs mods that use the engine for alternative games..." and its clear that addons/ doesn't do this.  We can of course change addons and bend physfs to our will as it is not widely relied upon yet.

The bit that does seem agreed upon is that these mods will be deployed each as a single archive, using stored ZIP with XY solid compression for download.  And we will have xdelta+XY from the beginning too, since that's not where the bugs will creep in.

I see two types of mod:

  • (1) a free-standing game such as Glest+MegaPackV4, or Military of MRise.
  • (2) a mod that adds additional scenarios, fractions and/tilesets to (1).  Examples are numerous for the Glest tech tree.  I suggested the naming convention that lists the (1)-mod that it is extending.

To my mind GAE should not be bundled with the existing old Glest game data, but rather pre-installed with a Glest+MegaPackV4 mod in the new mod structure.

Furthermore, I think (2) needs dependency checking from the get-go.  And it doesn't hurt (1) either, since all the mods I've looked at - with the exception of MRise, which is nice and clean - are full of MB of debris and even broken links.

If a (2)-type mod has a scenario with the 'pirates' fraction in it, it'd better list the appropriate mod it is relying upon in the manifest so the downloader can ensure all dependencies are met.  I don't see this as being a stretch goal, I see this as hygiene and not too challenging.

In my dependency prototype I ignored the overriden 'core' files, because we need some agreement on how to handle them.  I think they should go into a set folder within the mod archive, and that it would be much more robust if there was an XML file explicitly saying which artwork overrides whatever rather than an implicit naming system.

Regards integration, I think that mod makers should be releasing early and often, even if their audience is their co-conspiritors and other modders.  I think the system should be full of small incremental releases to 'alpha' mods, and that a mod should have a 'stable?' flag when its uploaded so we know whether to only show it to players who have elected to see the bleeding edge.

And on the client-side, I imagine a stand-alone wx-widgets app that does the downloading.  From the beginning the downloader code however should be in the shared lib, with the full intention of eventually bringing it into the game UI.

In the game UI, there should be a top-level menu option for selecting between (1) mods, which naturally swap out all the UI artwork.  The most-recent choice is remembered.  But in the new game screen, if the 'advanced' check-box is checked, you can pick and mix between all installed fractions, maps, tilesets etc.

Its natural people are apprehensive of such naked ambition, but is any got any fundemental against the direction its going in, or is it just seeming to go too fast?

765
Feature requests / Re: integrated mod installer
« on: 25 November 2010, 12:03:14 »
I have written a very simple test script to see how successfully I can naively detect dependencies in a mod.

Here's the python code; it can just be run in python.

https://gist.github.com/715270

For a taste, here's what it thinks about megapack_v4:

Code: [Select]
...
Ignoring tilesets/autumn/models/grass.tga 768.0 KB
Ignoring techs/megapack_v4/factions/persian/units/elephant_cage/sounds/tech_building_fall5.wav 148.6 KB
Ignoring techs/megapack_v4/factions/elves/units/tree_of_life/construction_models/tree1.g3d 1.1 KB
Ignoring techs/megapack_v4/factions/elves/units/wood_hall/images/wood_hall.bmp 12.1 KB
Ignoring techs/megapack_v4/factions/indian/units/thunderbird/sounds/eagle_fly2.wav 426.0 KB
=== Ignored: 675 121.3 MB ===
=== Included: 4681 587.2 MB ===
=== Models: 922 252.7 MB ===
=== Textures: 1040 183.1 MB ===
=== Particles: 120 83.5 KB ===
=== Sounds: 2360 149.9 MB ===
=== Units: 172 824.1 KB ===
=== Upgrades: 53 34.8 KB ===
=== Resources: 7 1.2 KB ===
=== Fractions: magic, norsemen, dark_magic, indian, persian, elves, tech, woodsmen ===
=== Maps: overgrown_city, tt_goldrush_v2, forest_for_four, waterworld, tropical_arena, domination_isles, one_on_one, conflict ===
=== Tile Sets: evergreen, desert2, winter, autumn, jungle, hell ===
=== Tech Trees: megapack_v4 ===
=== Scenarios: m4_kill_the_magic, m4_my_island, m4_beehive_castle, m4_amazones, m4_norsemen_attack, m4_persian_conflict, m4_indian_attack, m4_waterworld_light, m4_waterworld, m4_magic_can_do_it, m4_amazones_light ===
=== Mod check failed ===
(model BROKEN techs/megapack_v4/factions/persian/units/swordman/models/guard_attacking.g3d (referenced by: unit ../swordman.xml)) ('Error reading G3D file', IOError(2, 'No such file or directory')), ('File does not exist',)
(model BROKEN techs/megapack_v4/factions/tech/units/swordman/models/guard_attacking.g3d (referenced by: unit ../swordman.xml)) ('Error reading G3D file', IOError(2, 'No such file or directory')), ('File does not exist',)
(particle BROKEN techs/megapack_v4/factions/persian/units/worker/particle_splash.xml (referenced by: unit worker.xml)) ('File does not exist',)
(scenario BROKEN scenarios/m4_amazones/m4_amazones.xml) ('References missing map', u'in_the_forest'), ('References missing tileset', u'forest')
(scenario BROKEN scenarios/m4_amazones_light/m4_amazones_light.xml) ('References missing map', u'in_the_forest'), ('References missing tileset', u'forest')
(sound BROKEN techs/megapack_v4/factions/norsemen/units/mistletree/sounds/tower_attack1.wav (referenced by: unit ../mistletree.xml)) ('File does not exist',)
(sound BROKEN techs/megapack_v4/factions/norsemen/units/mistletree/sounds/tower_attack2.wav (referenced by: unit ../mistletree.xml)) ('File does not exist',)
...

I really appreciate modders running it on their own mods and others they are familiar with, and then helping me understand:

  • what is ignored but shouldn't be - and how I would detect it
  • what is not resolved but should be, and what the path in the xml was to the file that's not really missing
  • any surprises
  • any encouragement or feature requests
  • any bugs or crashes, particularly if you get a stack trace from running the script

Its very easy to run; all you need is python, and know where the mod is (the mod should be in a folder, rather as you would unpack an archive)

The thing is, I haven't found a mod that this script doesn't think is broken!  I've squashed most of my mistakes, and iteratively found most, but now I really need help from expert modders to tidy up the last few things I've got wrong....

(I don't think this script will be used in any real GAE solution, its just a quick way to prototype the logic that might be used, and see if its useful)

766
Feature requests / Re: integrated mod installer
« on: 25 November 2010, 06:43:38 »
Yes, I was being very careful to suggest we 'silo' mods - that the mods installed would not be able to implicitly rely upon other content.

Are people comfortable with the zip-folder-with-version-number approach to containing mods?

767
Feature requests / Re: integrated mod installer
« on: 25 November 2010, 00:39:49 »
It really doesn't seem to be there

can anyone else verify that the pine woods tileset in vbros 1 is broken?

(In other news, g2xml doesn't support the old version 3 g3d files; I worked around it but thought it worth mentioning)

768
Feature requests / Re: integrated mod installer
« on: 24 November 2010, 22:25:07 »
I am writing a simple test script (to learn the mod structure better, rather than to suggest this is production code) to find all dependencies for a mod.

Just to check I'm not going mad:

It says that vbros_pack_1 is missing some of the textures for pine_woods tileset; and indeed I cannot use that tileset in GAE because it also complains some of the textures are missing.

In Glest/MG, is there a surface2e.bmp etc that is it is 'inheriting' that isn't in GAE, or have I installed it wrong, or is it broken?

769
Feature requests / Re: integrated mod installer
« on: 24 November 2010, 14:56:38 »
yes, I asked because that confused me.

I'm thinking we need dependency management, and then make physfs mount in the correct order.

I've worked with writing really big systems, and one simple versioning approach that I've seen work very well is to use major.minor numbers in the folder names.  To illustrate:

mods\MRise.1.2.zip\ <-- a zip file, therefore something downloaded and versioned by our new mod installer
mods\MRise.1.3\ <-- not a zip, therefore this is something this modder is making now

The modder runs the packager tool we write.  This tool finds the MRise.1.3 folder, and says "would you like to release version 1.3 of this mod?"

It then walks the files in it, looking for dependencies.  Any dependencies in MRise.1.2 will get copied into the 1.3 zip that is uploaded, other dependencies in other non-MRise folders are allowed, but are recorded in a config file in the new 1.3 zip, rather than copied.  So the packager records what dependencies - other mods - a mod has, and also what it publishes in the sense of maps, scenarios, tilesets, etc.  A 1.3 zip is made, uploaded, the MRise.1.3\ folder is renamed MRise.1.4\ and the zip then placed there.  So the modder moves on a version and carries on modding.

You can push this further, e.g. mods\MRise.1.3.ChazyFraction.0.6.zip would be a mod of a mod, which requires MRise.1.3 (or perhaps later) and so on.  Only modders and the curious see these files; game users just see nice names.

But fundamentally, the idea that things in addons\ are mounted in any order and can replace anything isn't holding.  We'd make the mods\ folder not work this way, but rather follow a strict hierarchy.

Obviously there are lots of finer details and tweaks; but what do people think about this general approach to encapsulating mods into fixed-name folders?

770
Feature requests / Re: integrated mod installer
« on: 24 November 2010, 10:11:21 »
Code: [Select]
<mod>
   <type>tech|tileset|scenario|map|other</type>
   <name>Mod Name (as it appears)</name>
   ...
</mod>

Can a mod contain more than one type of content?  How is this specified?

771
Feature requests / Re: integrated mod installer
« on: 23 November 2010, 06:35:15 »
XML is what modders have to use for the units, so why not for the mod description too?  CSV has the disadvantage of being flat and difficult to extend.

Now I'd like JSON to be usable everywhere we use XML today, but that doesn't seem to be necessary, just elegant.

XDelta on packed zips - as long as the zip is packed by the same zip program with the same settings, and that's something we control - is very nearly as good as on stored zips.  I know that if we xz the stored zip it is smaller, and that the xdelta between two stored zips is smaller, but its all not a lot.  If we work with compressed zips, compressed with a decent zipper like 7zip (kzip is sadly not open source, and its slow), then we can store compressed zips on the file system (which feels nice).  I mean, if the xz and xdelta advantage over deflate is worth it, then why not use bsdiff instead of xdelta and so on?

Ok, I bow my head in shame, suggesting xdelta on compressed zips.  It works surprisingly well.

If we want to go for good compression, we'd use bsdiff and stored .zip and .xy.

If we want things to be packaged faster we go for xdelta and delfated .zip, we just have to be careful to control the zip tool and options.  But then the zips are compressed on disk, which is nice.

Both these approaches bring up using PNG (and, honestly, JPG) instead of BMP/targa, and OGG instead of WAV.  If people did this, and we added support for g3dz, there'd be no difference between a deflated and a stored zip.

What is clear is that either a mod contains each and every artwork it needs - yuck - or a dependency checker is built into the packager tool so it detects all dependencies.

772
Feature requests / Re: integrated mod installer
« on: 22 November 2010, 20:47:52 »
given that the addons/ folder hasn't really taken off yet, its meaning could be changed.

It could be that there is a folder containing 'total conversions' - including the default one, which perhaps ought to be megapack - and that only one of these is allowed at any one one time - its pickable from the main menu, and the artwork for the menu screens etc is based upon it.  It could be that the name 'addon' is changed to perhaps something slightly more appropriate?

And then within each of these folders, there the mods of any total conversion are stored.

How does this grab people?

773
Feature requests / Re: integrated mod installer
« on: 22 November 2010, 18:45:14 »
Adding tar support to physfs is of course possible; however, it is performance presenting problems; tar is a solid format.

I'm not going to say what I think alternatives might be yet; I'm interested in what you folks think.


774
Feature requests / Re: integrated mod installer
« on: 19 November 2010, 16:21:05 »
I'll go add tar to physfs; a good way to learn my way around how it works too

775
Feature requests / Re: integrated mod installer
« on: 19 November 2010, 07:09:07 »
Quote
That's what basically the prototype does. I don't see any benefit in your approach and i actually don't really understand what you want to achieve.

ok.  Tar+xdelta for now then.  We can always make it more complicated later.  I want to code things and help code things, not be an upstart.

So there seems two directions I could code in:

1) code and submit to physfs tar support

2) the study of supporting multiple total-conversions and also mods that mod a total conversion

thoughts on what needs doing?

Pages: 1 ... 27 28 29 30 [31] 32