Preset mipmaps?

[ChupaReaper]

I've noticed with my mod having loads of textures at around 512x512 it loads ok but sometimes it can overload the game and should I finish all five factions, there's probably not going to be enough memory by the time faction 3 is complete, this usually causes an out of memory problem to do with 2d mipmaps.
Using a mass resizing tool of some kind (I've got a few) I can quickly resize all my textures to smaller sizes resolving the problem but loosing the quality of the textures.
So instead, textures could have smaller sizes preset for mipmapping maybe? I'm no expert on all this but maybe a directories amongst a model's textures called 2x, 4x, 8x, etc or something like this could store smaller versions of each texture. Thing is if they all have to be loaded into memory it doesn't really make a difference if they're predefined does it?
There are loads of PC games that have huge amounts of textures, much more than my mod and they manage so there must be some way to deal with this memory issue?

[hailstone]

Are mipmaps even being used? Doing a search for gluBuild2DMipmaps and glTexImage2D in the source shows that the level is set to 0 which is base only and therefore mipmapping isn't really being used (Edit: nevermind, I read the code wrong). Even if it is being used it takes an additional 1/3 of the texture size (the larger the level the less impact on size). So it's something you will need to work out with a texture allowance. I imagine other games dynamically load textures but for an RTS it really needs to be loaded from the start (maybe I'm wrong though).

There's a discussion on TA:Spring engine textures.
Other links:
http://www.dansdata.com/gz014.htm
http://stackoverflow.com/questions/3028574/finding-opengl-texture-size-in-bytes-after-it-was-loaded

[ChupaReaper]

I imagine with an RTS they will all need to be loaded at once I guess, with other games mainly FPS, the textures are drawn when a player is near them or in a level/zone/whatever that they are used in where as in an RTS all textures used by every faction and by the tileset must be loaded and ready to use as the player could view them at any minute. You could have it so the textures are loaded when the type of unit is first introduced into the game but once all the units are introduced we have the same problem as before anyway.

Would pre-creating mipmaps help at all or do they still need to be loaded into the game, I'm guessing GAE reads all the textures into memory and mipmaps them adding the mipmapped versions of the textures into the memory too. By pre-creating mipmaps it would mean GAE reads all the textures and then reads all their mipmaps into memory and all it does is skip the step where mipmaps are created and would thus not really solve the texture overloading issue? If this is the case, precreating mipmaps wont really help I guess.

Another suggestion would be to have a hi-res and low-res texture game setting. Low-Res loads low-res versions of textures if provided and mipmaps them. Whislt in Hi-Res mode, the original textures are loaded and no mipmaps are created, would decent graphic cards around 1gb dedicated memory handle this, if they would then this would be a nice solution.

Or maybe more control over how many mip maps are made, I'm gonna guess again and say that mip maps the original texture halved in width and height again and again until it's at something like 2x2? Instead maybe reduce the oringal texture to 64x64 and down so that large textures like 1024x1024 go straight to 64x64 then get halved and halved instead of going from 1024x1024 to 512x512 to 256x256, 128x128 then 64x64 and down. This would then save memory maybe? This is all just a total guess as I'm no programmer !

[will]

Graphics cards create mipmaps very fast whilst doing texture upload, so its not necessary to pre-compute them yourself.

Generally mipmaps trade more graphics RAM for faster drawing.  In your scenario you are wanting to use less RAM.

What you might want is that the engine knows its graphics RAM constraints and, during upload, downscales large textures so that they fit in that budget.

This keeping-to-a-budget logic is not so easy to code and get right, nor so easy to retrofit to Glest engines.  It'd be a big feature request.

Have I captured right what you're asking for?

One way to ensure you're not wasting lots of RAM on normal GPUs is to avoid non-power-of-2 texture sizes, apparently.  (They are starting to slip into mods, and frankly they shouldn't.)

[wciow]

I don't think we need to worry that much. I have 1024MB of Graphics RAM, which should comfortably fit all the texture requirements of Glest without the need to swap any data from normal RAM. The only thing I am unsure of is how much space is needed for the terrain textures?

[ChupaReaper]

Anything to increase how much GAE can take is what I'm looking for really, I can barely fit two of my factions into one game with all the bump and specular maps, now that they've been reduced to 256x256 and 128x128 down from the same size as the diffuse map they no longer overload GAE but with 5 factions total there might be a problem.

All my textures are the powers of 2 I hate messy texture sizes (some are like 128x64 and other different widths and heights are they ok?)

[Yggdrasil]

Texture compression is the way to go. That way the texture files are stored compressed on the graphics card. I have a hack to support .dds files. I need to refactor texture handling a bit before i commit this. Keep in mind texture compression is lossy and so is .dds.

I think MG already uses texture compression through the graphics driver which compresses the texture and transfers it on the graphics card. Open source graphics driver on linux don't support this out of box because the algorithm is patented (at least the most common s3tc). .dds is precompressed and just needs to be copied. We should probably try to support both ways as most gamers have proprietary drivers anyways.

[Omega]

Texture compression is the way to go. That way the texture files are stored compressed on the graphics card. I have a hack to support .dds files. I need to refactor texture handling a bit before i commit this. Keep in mind texture compression is lossy and so is .dds.

I think MG already uses texture compression through the graphics driver which compresses the texture and transfers it on the graphics card. Open source graphics driver on linux don't support this out of box because the algorithm is patented (at least the most common s3tc). .dds is precompressed and just needs to be copied. We should probably try to support both ways as most gamers have proprietary drivers anyways.

PNG is compressed too. Would it work?

[ChupaReaper]

Would I have to convert all my textures to .dds or could they be compressed when loaded, .dds is annoying to use that's all but .png I love to bits lol.

[John.d.h]

AFAIK, PNG is decompressed when it's used (so its only benefit is disk space, which is moot if you zip it up), while DDS can somehow stay compressed during gameplay.

[Omega]

Would I have to convert all my textures to .dds or could they be compressed when loaded, .dds is annoying to use that's all but .png I love to bits lol.

Looks like it. By the way, should you do that (and yeah, that's a lot of work with the number of models in your tech tree), be sure to save a non-DDS version in a loss-less format such as PNG, as lossly formats are not suitable for editing, but just for the final product.

[ChupaReaper]

Could a feature be implemented that overrides a g3d's target texture then? So if a dds texture is present use that over the one defined within the g3d (so if one uses Body.png, if there's a Body.dds, use that instead)?
Else not only will I have to convert all my textures but I'll have to re-export all my models too.
Edit: Or maybe a tool that can change g3d textures, maybe implemented into the g3d viewer which could be renamed as the g3d editor?

[will]

most GPUs compress textures internally - dds (and e.g. powervr on iphone) are just precomputed versions of this.

They are lossy compression, which is why I don't understand people's aversion to jpeg compression for textures (although double degradation is not cool).

Png is lossless compression and suites illustration more than photo-like images so often gain is not large for typical textures.

So you unpack a png, give it to the gpu, and (in normal games) it internally lossily compresses it...

So moving to dds is I think a good thing except it is tricky on opengl and it doesn't allow megatexture ideas

[John.d.h]

I don't understand people's aversion to jpeg compression for textures.

They get more and more degraded every time you save them.  It's an issue of long-term (re)usability.

[Omega]

I don't understand people's aversion to jpeg compression for textures.

They get more and more degraded every time you save them.  It's an issue of long-term (re)usability.

Plus, JPG cannot support alpha, and to be honest, texture should have alpha somewhere. It's a better idea to use one texture when possible, and since every unit should have team color somewhere, you need alpha. The reason to use a separate texture would be if you need transparency too. For example, Military's flamethrower has 2 textures, one for the unit itself, which uses teamcolor, and one for its flamethrower, which uses transparency for the handles.

However, I again note, that if you use a lossly format, you still should keep a lossless version as well (and perform all edits to this). You don't have to (and shouldn't) distribute these files with the mod, as that would increase filesize drastically, but you should back them up as well, and it never hurts to have them available online in case someone else wishes to do a deprivation work, for the reason John pointed out.

[Yggdrasil]

Else not only will I have to convert all my textures but I'll have to re-export all my models too.

Don't do that. I have a shell script for testing which converts all textures and modifies the models. This might be something for a modpacker which spits out an optimized addon as a zip file.

Maybe we should even skip the file extension in .g3d file and just search for something suitable during load.

most GPUs compress textures internally - dds (and e.g. powervr on iphone) are just precomputed versions of this.

Not exactly right. The GPU only has hardware for decompression. Compression is done by the graphics driver (that's the problem in mesa).

So moving to dds is I think a good thing except it is tricky on opengl and it doesn't allow megatexture ideas

Yeah, it's tricky in opengl. One needs to flip the t-axis. The class i used in my hack is from spring (and they have it from a nvidia SDK) which flips it in compressed format. I looked at many image loading libraries and all of them decompress it for flipping which is not what i wanted. It would still be nice to just use one library for texture handling and not libpng, libjpeg, ... and what not.

[ChupaReaper]

Else not only will I have to convert all my textures but I'll have to re-export all my models too.

Don't do that. I have a shell script for testing which converts all textures and modifies the models. This might be something for a modpacker which spits out an optimized addon as a zip file.

Having the g3d skip texture extensions will save me a lot of time and this shell script will be very useful, could it be set to update addons also, so instead of packings all the files into one 7zip or whatever every time one tiny change is made, it instead overwrites files that are newer?

[will]

Another pointer to my mod-maker, where conversion and so on could be done: http://github.com/williame/GlestTools/blob/master/glest_mod_pack.py

most GPUs compress textures internally - dds (and e.g. powervr on iphone) are just precomputed versions of this.

Not exactly right. The GPU only has hardware for decompression. Compression is done by the graphics driver (that's the problem in mesa).

That's just bad grammar on my part, and I didn't mean the sentence to be parsed quite so literally  If we're being literal, I'm not convinced there is compression happening in the hardware, I think the format is designed to be used compressed as-is.  That's certainly the case in powervr world, where I have some experience.

https://forum.megaglest.org/index.php?topic=6615.0 is a related thread.

SOIL looks like a very good small library for doing all our image loading?