Future file format support

[will]

There have been discussions about future file formats for models and textures.

I thought it good to summarise the conclusions so we don't have to re-learn our conclusions when Glest merges and its time for wider support.

Textures should be lossy compressed and mipmapped with DDS.

Animated models should be stored as MD5.

Inanimate models should be stored as OBJf.

This would reduce filesize by an order of magnitude, and be faster to load.

The drawing performance of MD5 has not been compared to G3D but speculation is its not deal breaker.  DDS and OBJf will both be faster to draw in all cases.

[Omega]

Support, though I am not familiar with this OBJF, and not sure if it'd be worth the hassle of exporting to OBJ then running through this stripe as well (assuming I am understanding it correctly). It might be simpler, even if not more efficient, to just use MD5 for all models. Unless, of course, you want to explain the advantage of OBJF (the webpage didn't do a good enough job ).

[John.d.h]

not sure if it'd be worth the hassle of exporting to OBJ then running through this stripe as well (assuming I am understanding it correctly).

Scripts -- just set it and forget it.   Unit models are almost always animated, but this could be very useful for tilesets.

[ChupaReaper]

OBJ alone is an awesome format and would be nice for static models unless md5mesh without loading md5anims was to be supported, then again a one frame md5anim would do the trick and would barely take up space. Not sure on this OBJF format, maybe support for that and OBJ and if OBJF is smaller then it can be used for large complex models like the menu map.

[will]

I slipped OBJf into the discussion for the programmers to keep in mind - I foresee conversion being done by packing scripts and not users.

The background on OBJf for those interested in the nitty gritty details:

We all know the GL_TRIANGLE_STRIP is the fastest primative.  Nvidia even provide a lib so programs can stripify on model load, but it seems cleverer to just have mod packing scripts do it for us.

If we stripify all models - we could even do so for MD5 and call it MD5f or something - it not only draws quicker, it also takes half the space (both in GPU, disk and when zipped).

[will]

[This post has been split from the G3D model conversion topic on the Megaglest board, so as to avoid scattering the discussion.
-- John.d.h]

I would anticipate the total megapack size when 7zipped has actually increased though?

(7zipping a BMP or TGA is typically better compressing than the GZip (deflate) used in PNG (even with the best PNG packer called pngout).)

For fun I compressed megapack with glest_mod_pack.py:

417.1 MB compressed to 156.0 MB (37.4%) -> /home/will/Games/megaglest-3.3.7.2.zip.xz  

Out of curiosity I tried to get a rough idea of how much space DDS would save.

My simple script running in techs/megapack found 106MB of TGA/BMP
before: 106456838, after: 18969576

Anyway, they all compress down to 18MB (that's how much RAM would be needed in the GPU to run it too)

The original TGA and BMP 7zipped to 21MB
And those DDS zipped to 6MB

(click to show/hide)

Code: [Select]

#!/bin/bash

before=0
after=0

for f in `find -name *.bmp`
do
bef=$(stat -c%s "$f")
        before=$((before+bef))
convert $f $f.tga
nvcompress -alpha $f.tga $f.dds
aft=$(stat -c%s "$f.dds")
after=$((after+aft))
        echo $f $bef "->" $aft
rm $f.tga
done

for f in `find -name *.tga`
do
bef=$(stat -c%s "$f")
        before=$((before+bef))
nvcompress -alpha $f $f.dds
aft=$(stat -c%s "$f.dds")
after=$((after+aft))
        echo $f $bef "->" $aft
done
echo "before: $before, after: $after"


[titi]

why no DDS:

from wikipedia:
The DirectDraw Surface file format, from Microsoft, is a standard for storing data compressed with the lossy S3 Texture Compression (S3TC) algorithm ....

Whats wrong with this format:
1. it comes from microsoft ( always evil   )  
2. S3TC is a patent protected way! I think you will never see an opensource driver which will support this ( like the Ati one ).
For same reasons I think S3TC is also not supported by the open source intel drivers.

All in all its simply not a format what you want to use for the texture files. Its a special texture format what is used to use texture compression on graphic cards. MG already can do texture compression and uses it! It asks the graphics card and chooses the format dynamically. I think on nvidias proprietary drivers S3TC is already used ( maybe something better/newer if the card tells us about it )

We want to make free software so no, no support!

[will]

Microsoft is evil?  The children of last century want their meme back!

[james876]

windows do not execute this command under the model of conversion and compression, thank you!
D:\trunk\data\glest_game>glest_game.exe --convert-models=techs/megapack_new/=png
v3.4.1-dev-VC++: 1500-Mar 13 2011 9:52:00, SVN: [$Rev: 1692 $], [STREFLOP]
About to convert model(s) [techs/megapack_new/]
About to convert using texture format [png]
megapack_new directory without any files. Why?

[MJR]

Any word when these new features will be in a new full version release lets say version 3.41. Just wondering how long to wait for a new stable version.

[titi]

Currently targeted release date is end of march, maybe one or two weeks more

[MJR]

Thanks for the information. I hope it will be before the merger. Since most of what I like in MegaGlest wont be in the merger. See I don't do internet games with MegaGlest I only do the AI and what I've read about the merger the AI in MegaGlest wont be any more only Glest Advance Engine AI witch I think is inferior to MegaGlest current AI as far as a challenge goes. So I will not be installing any merger version of the program. So this next release of MegaGlest will most likely be my last upgrade unless you come out with another stable release of MegaGlest before the merger is finished.

[silnarm]

Microsoft is evil?  The children of last century want their meme back!

..Since most of what I like in MegaGlest wont be in the merger...

You seem to have been misinformed.

[John.d.h]

If an evil tree produces good fruit, then let's eat the fruit.  It's not like we're paying them for it.

My only concerns with DDS would be related to implementation and compatibility, not with its origin at the hands of evil-doers.

[titi]

Whatever, I think DDS should not be used because of reason 2!

[will]

DDS should not be used because of reason 2

that is a very valid point and I concede.

There are strong technical advantages, particularly concerning minimizing artifacts (especially in mipmaps) with pregenerating DDS.  If you do support compression-on-upload using the GPU's drivers it ought to be turned off in any 'highest quality' setting that is ever supported.  (IIRC the Doom3 engine shipped with both TGAs and DDS and only used the TGAs if you asked for maximum quality in the settings.)

[Yggdrasil]

Whatever, I think DDS should not be used because of reason 2!

Yeah, and reason 2 is quite wrong. Yes, S3TC is patent protected but only in countries with software patents.

There's the library libtxc_dxtn to get the full support into mesa. Most open source drivers can use it and will expose the opengl extension. This library is only needed for on-the-fly compression in the driver. Mesa is out of box capable to upload pre-compressed texture files like dds to the video memory. It's just copying memory which is not patent protected.

There's still a problem: Mesa normally doesn't expose the opengl extension EXT_texture_compression_s3tc, only ARB_texture_compression. So we can't be sure the graphics card is able to use the S3TC texture. Probably mesa would throw an error in glCompressedTexImage2D and glGet with GL_COMPRESSED_TEXTURE_FORMATS might let us check beforehand.

I tested it on my laptop with stock open source ati driver and it works.

[will]

Yeah the truth is somewhere in the middle.

If you pre-compute the DDS then, if your graphics hardware supports it, you can just upload it and the patent is not a problem (since the manufacturer of the hardware has licensed it).  This is the best of all worlds - its very good compression both on-download, on-disk and in-game.  Its also much better quality than on-the-fly compression.  It even speeds up game loading.

And all graphics hardware supports it.  I can't think of a single PC integrated chip even that doesn't.

*However*, there might be one chip out there that I don't know of.  Or someone porting Glest to iPad or Android or something.  We'd be potentially locking somebody out even if its no-one we can put our finger on right now.

If mod packing tools *always* produce full-file 'source' zips as well as 'optimised' zips, then maybe that's a way to have both?  But we'd need to be strict about people always making the full original textures (and .blend *ahem*) available imo.

[silnarm]

I think perhaps we just look for a .dds of the same name as any given texture (sans extension) and use if present, else load the g3d specified one as a Pixmap (or always load the 'original' into a Pixmap first, depending on quality settings, as Will suggested).

So there is always a tga/png and the modder can include their own dds as well, the dds would be used if appropriate, or if not present we automagic it through GL, as MG is doing already.

2. S3TC is a patent protected way! I think you will never see an opensource driver which will support this ( like the Ati one ).

Not to nit pick or anything, but the patent in question was filed in 1997, so in this case never == 6 years Too long to make a difference to us now, but its not never.

[hailstone]

It might be possible to do semi-realtime. Convert to disk on first run then use the dds after (not sure if that's what was meant by "automagic it through GL"). Probably better to do it with a packager though.

S3TC Support For Mesa Brought Up Again

[titi]

I don't really get why you want this format so much?

What we do in MG is load png(or whatever)  transform it to dds ( or whatever the card supports ) and then use it as a texture. I see no real advantage at the moment to change this. DDS is maybe used on consoles which have very little Ram for example and connot simply make this conversion on the fly, we can ...

All in all this discussion is not very useful at the moment as we will not have time to work on this kind of problems for a long time ....
I think many many other things comes first and this is very low priority.

[Omega]

Reasons to want it is increased performance and reduced filesize (assuming that only the DDS is used. If both the TGA/PNG and a DDS are used, the filesize is actually larger, but performance still increases). Extra performance is really useful. Just try zooming the camera out reallllly far and watch the frame-rate shudder.

I think you're again being a bit biased Titi. YOU don't have to use it, but it can still be implemented so other modders can use it. Though, I must agree, it is fairly low priority.