Optimising G3D tool

[will]

I put it on github: g3dopt.py

It combines all compatible meshes.  It is an excellent compliment to G3dHack; if you run this python script first, you can usually get a few more K out of G3dHack optimisation.

It is optimising the model for drawing by reducing the number of glDrawElements and buffer switching needed.  This is apparently a big deal for things like webGL.

It was very tempting to interleave the vertices and normals and bump the version number up to 5.. but I didn't.

[will]

To be clear, optimising is this flow:

G3dScan it
g3dopt.py it
G3dScan it again

I'm presuming G3dScan can spot a mesh that has frames it doesn't need, right?

Thinking about it, it's obvious that the loader could also do the same trick to optimise drawing.  MG and GAE don't need people to run this script; they can join compatible meshes as they load them; its just array concatenation after all.

We really ought to be ensuring that frames are ordered opaque then transparent too, but I haven't bothered with that.

[will]

I still use G3Ds for hobby projects, mostly targeting webGL.

I had a landscape with 1000s of G3D trees on.  And it went quite slowly.

I run this g3dopt.py script on them - https://raw.github.com/williame/barebones.js/gh-pages/g3dopt.py - and the scene drew really fast afterwards!

I was stunned that the tool could have such an impact!

The extent to which various factors played a part: that it was webGL+Angle, that it was on low-powered laptops, that the models of trees Mr War gave me were exceedingly unoptimal - all these factors can be ignored!

It'll never hurt to run all your G3Ds through this tool and then Silnarm's excellent G3DHack optimiser!  And someone with a low-end graphics card might thank you for the thought!

Hope this helps,

Will.

[John.d.h]

I tip my hat on behalf of we outdated laptop users everywhere.

[Psychedelic_hands]

Why has not the megapack been run through this a billion times plz?

[John.d.h]

Why has not the megapack been run through this a billion times plz?

I'm sure the developers would be grateful if you did.

[titi]

@Will: I still don't get why things should be faster after optimizing.  Are there duplicate meshes in the models ? I mean jsust dropping some unused frames of an animation should not really affect the performance.
but keep in mind that especially for plants/trees you often have two meshes which are nearly in the same place! These are meant like this, because those have two different normals. So for example a branch has a upper side and a down side. Both meshes are needed! If you optimize them out it might be faster, yes, but the lighting of the model will not be the same in the game and things would look wrong.

But maybe there is something else you optimize. Please tell us ( if possible very exact ) what . Maybe the original glest models ( made with 3dsmax ) had unneccesary stuff in them?

[will]

hi titi,

this tool is not doing any removal of anything.  All it does it join together meshes that share the same textures and have the same number of frames.

So when drawing the Mr War's trees, which he made by lots of copy-n-paste-n-move of a mesh in blender, is now done with one draw call instead of 5 (or whatever).

The export scripts in blender could do the same optimisation, or the optimisation could be done automatically on load by MG.

[softcoder]

The python script looks very simple (good work) I'll see if i can integrate it;s logic into mg's model loading.

[softcoder]

Could you explain the following python code. I have most of it ported but the index part at the bottom confuses me:

Code: [Select]

def auto_join_frames(self):
        print "auto-joining compatible meshes..."
        meshes = {}
        for mesh in self.meshes:
            key = (mesh.texture,mesh.frame_count,mesh.twoSided|mesh.customColour)
            if key in meshes:
                meshes[key].append(mesh)
            else:
                meshes[key] = [mesh]
        for joinable in meshes.values():
            if len(joinable) < 2: continue
            base = joinable[0]
            print "\tjoining to",base
            for mesh in joinable[1:]:
                if base.index_count+mesh.index_count > 0xffff:
                    base = mesh
                    print "\tjoining to",base
                    continue
                print "\t\t",mesh
                for a,b in zip(base.frames,mesh.frames):
                    a.vertices.extend(b.vertices)
                    a.normals.extend(b.normals)
                if base.texture:
                    base.textures.extend(mesh.textures)
                base.indices.extend(index+base.vertex_count for index in mesh.indices)
                base.vertex_count += mesh.vertex_count
                base.index_count += mesh.index_count
                self.meshes.remove(mesh)


*Update: Think i've got it finally. Code is in svn head. Render performance seems better to me.

[tomreyn]

Turns out this is not as simple as we were thinking, and sadly the FPS gains are gone when the necessary corrections are made.

[softcoder]

I think the gains should still be there, we may join a few less meshes but we still do join a fair amount. Please make sure your result is accurate, I think i still see the increase on my end.

Thanks

[tomreyn]

My results (I always tested twice with a full exit and restart of MegaGlest, always using the same computer and INI files) are merely based on the "benchmark" scenario, which, I think, uses the "desert" tileset, which is the least demanding tileset. The camera angle you have there does also not show a lot of tileset objects. So if gains are primarily in regards to tilesets or don't affect Egypt and  Indian factions, this would explain why there is no measurable difference to the values the benchmark scenario reported before.

[will]

Sorry there was a problem, glad you found and fixed it.

Some of the promised performance can be returned though!  Just because some flags need to be preserved for selection etc doesn't mean you can't join the meshes; it means you have to track that outside the meshes, is all.

[titi]

OK I think we found a ( possible) problem with this when doing the same in MG while loading the models.

A Model contains several objects. each object has its own header containing infos like teamcolor, transparency, colors .....
If you merge things with different header infos the result will not be good ... you loose information and break teh models.
We saw a partly transparent blacksmith in japanese and so on . So be really careful with this .

MGs 3.0.8 beta does whats possible here without the danger of breaking things. But the general idea improved things a lot for some models.

[will]

Yeah those were the problems you found and fixed that I was referring to.

I was pointing out that you can still join meshes with different selection flags if you track selection separate from meshes after its been loaded.  This can be as simple as putting all the unselectable vertices at the end of the buffer and having a selectableVertexCount instead of a boolean.

[softcoder]

Doing that i don't think will do much since it still means a seperate render anyways for vertices with each flag. Looking at the code each flag requires different openGL settings for the render so you'd have to render one pass for each different type of property which is the same as original.

[will]

I was referring particularly to the 'not selectable' flag, which is not a render attribute, only a picking attribute.  I am guessing its the really big reason that the benchmark scenario didn't speed up.

If you have a 'selectableVertexCount' variable, rather than a boolean, in a mesh, then the unselectable vertices that are joined can be at the end.