G3Dv4 format spec

[martiño]

================================
0. INTRO
================================

G3D is a binary format used for Glest for 3D models and animations. Currently a plugin for 3dsmax exists. Glest is using version 4 of this format.

Many people asked us to make a plugin for Blender but we don't have the time for doing it, however this is the spec for G3Dv4 in the case anyone wants to take a look at it.

Glest uses a rather simple 3D format, in many ways similar to MD2 (the Quake 2 format).

The Glest code for loading 3D models is in model_header.h and model.h/cpp files in the "Glest Shared Library" graphics module.

EDIT: This spec refers to the latest version of the format, since it has changed in Glest 1.1.0 release candidates.

================================
1. DATA TYPES
================================

G3D files use the following data types:

uint8: 8 bit unsigned integer
EDIT: uint16: 16 bit unsigned integer
uint32: 32 bit unsigned integer
float32: 32 bit floating point

================================
2. OVERALL STRUCTURE
================================

- File header
- Model header
- Mesh header
- Texture names
- Mesh data

================================
2. FILE HEADER
================================

EDIT: All structures are assumed to be packed.

Code: [Select]

struct FileHeader{
uint8 id[3];
uint8 version;
};
This header is shared among all the versions of G3D, it identifies this file as a G3D model and provides information of the version.

id: must be "G3D"
version: must be 4, in binary (not '4')

================================
3. MODEL HEADER
================================

Code: [Select]

struct ModelHeader{
uint16 meshCount;
uint8 type;
};
meshCount: number of meshes in this model
type: must be 0

================================
4. MESH HEADER
================================

There is a mesh header for each mesh, there must be "meshCount" headers in a file but they are not consecutive, texture names and mesh data are stored in between.

Code: [Select]

struct MeshHeader{
uint8 name[64];
uint32 frameCount;
uint32 vertexCount;
uint32 indexCount;
float32 diffuseColor[3];
float32 specularColor[3];
float32 specularPower;
float32 opacity;
uint32 properties;
uint32 textures;
};
name: name of the mesh
frameCount: number of keyframes in this mesh
vertexCount: number of vertices in each frame
indexCount: number of indices in this mesh (the number of triangles is indexCount/3)
diffuseColor: RGB diffuse color
specularColor: RGB specular color (currently unused)
specularPower: specular power (currently unused)
properties: property flags

Code: [Select]

enum MeshPropertyFlag{
mpfCustomColor= 1,
mpfTwoSided= 2
};
mpfTwoSided: meshes in this mesh are rendered by both sides, if this flag is not present only "counter clockwise" faces are rendered
mpfCustomColor: alpha in this model is replaced by a custom color, usually the player color

textures: texture flags, only 0x1 is currently used, indicating that there is a diffuse texture in this mesh.

================================
4. TEXTURE NAMES
================================

A list of uint8[64] texture name values. One for each texture in the mesh. If there are no textures in the mesh no texture names are present. In practice since Glest only uses 1 texture for each mesh the number of texture names should be 0 or 1.

================================
5. MESH DATA
================================

After each mesh header and texture names the mesh data is placed.

vertices: frameCount * vertexCount * 3, float32 values representing the x, y, z vertex coords for all frames
normals: frameCount * vertexCount * 3, float32 values representing the x, y, z normal coords for all frames
texture coords: vertexCount * 2, float32 values representing the s, t tex coords for all frames (only present if the mesh has 1 texture at least)
indices: indexCount, uint32 values representing the indices

================================
6. G3D EXPORTER
================================
Working G3D exporter for Blender 2.6x as a python script: Download
Documentation: G3D support

_{Edit by Omega: Fixed error as noted by Yggdrasil here}

[seltsamuel]

gnarf thats me above session timeoutet  

uh by the way which glest version starts using v4 ?
need to know it for testing purposes

[martiño]

what if the head is animated too example he grins ?
legs 10 keyframes walk
head 5 keyframes grin ?
body 1 keyframe

The current 3dsmax exporter is not able of exporting meshes with a different number os keyframes, however the game should support it.

Lets say we have an animation that lasts for 1 second, then you would have a keyframe every 100ms for legs, 1 keyframe avery 200ms for head and 1 keyframe for the body, but everything shoud be ok.

if the textureflag is not set (0) ther is no Texturename if it is !0 there is a Texture name ?

This is true in Glest, because we only use diffuse textures, but in "Glest: Duelfield" we used more than 1 texture per mesh and then the number of names was the number "ones" in the texture bitfield.

uh by the way which glest version starts using v4 ?
need to know it for testing purposes

1.1.0-rc1, the thing is that all the models are still in G3Dv3, so until we start exporting all the units from the expansion a can not ensure that G3Dv4 support is free of bugs (we made some tests though). I fact I've detected a bug regarding the mesh properties while I was making the spec, that will be corrected in 1.1.0-rc6

[seltsamuel]

ok, then i will test it with this version when the converter is ready.
Is it possible to put your v4 exporter for 3dstudio online ? so that i can compare exported models ? and maybe 1 exported v4 model ?

Another thing for the Animation should there not somewhere be a place for
the speed of animation inside the mesh ? so that the right speed can be set with the model (maybe a basic speed) ?

Then for animation what data u store? absolute vertx for each frame or relative data from last frame ?

by the way i told that im going to make my own game (Hack and Slay RPG like Diablo2) and i think i felt the decision to take the www.ogre3d.org Engine because its stunning    maybe u want to look in there u only need to start the executable demos they have.

it has many advantages to use such an engine
1. u have not to deal with technical issues display related
2. u have model viewer and ex/importers for all great 3d designers

they seem not to have the restrictions because of the lightsources there.
they use for windows vc++ and as i know u use it too but have many ports for some systems.
u could for example only use their display engine and leave the rest as it is they did a good abstraction layer so u have not to fiddle with internals.

I tell u this because i want to know your meaning about the ogre engine or if u know one thats better.
I will decide to use it because the engine is in my terms technical absolute up to date, its cross platform and there is an derivat which uses #.net for use as massive multiplayer game (called www.realmforge.com). the engine supports many 3d formats natively and i must not worry about modelviewers designers im/exporters ... ...
whats your opinion ?


Thanx in advance
Seltsamuel

[Speedator]

Very nice martin-o!
I really hope we have a blender-exporter in next time!

seltsamuel: Yes, it is a nice engine(I am already working  for a project using it, too). But  it would be a little bit oversized for glest, especially because it already has an advanced  engine. In the other way I think RPGs take more advantages with ogre(or crystal space), so your decision is quite good.

Do you have any more informations to your project(don't post here because it is offtopic, but on a website or a uploaded dev-doc)?

[martiño]

is it possible to put your v4 exporter for 3dstudio online ? so that i can compare exported models ? and maybe 1 exported v4 model ?

You can donload the G3D exporter and viewer, they are in the package called "Glest Tools".

Apart from that there is a sample model in both G3Dv4 and MAX here:

http://www.glest.org/files/ (swordman_sample.zip)

[seltsamuel]

cool thx,

im getting further with blender im near to export an unanimated textured object the animation i think will get a bit tricky  


Seltsamuel

[lancelet]

Just a quick question: you say that meshCount is

Code: [Select]

uint16 meshCount;Is this correct?  If so, I would propose adding uint16 to the list of datatypes used by the spec to avoid confusion:

Code: [Select]

uint8 - 8 bit unsigned integer
uint16 - 16 bit unsigned integer
uint32 - 32 bit unsigned integer
float32 - 32 bit floating point
I have also noticed that the multi-byte types appear to be stored in little-endian format.  Can somebody please confirm this?

Thanks,

Jonathan Merritt.

[lancelet]

Naaarrrggghh!!!  

Jonathan Merritt.

[seltsamuel]

uh, u are right i not looked at the models already created i only try to obeye
martin-o´s filespec.

are u sure the file is not exactly as specified by martin-o ?
i cannot understand that the compiler adds things not declared to files ?

i will expect this issue


Seltsamuel

[lancelet]

Well, I'm still hoping for some confirmation (try the code!).

The reason the compiler would do this is speed.  It can optimize access of structures on multi-byte boundaries.  I can't find a good online explanation of it at the moment, but do a Google search for "structure packing" in C and C++ to find out about it.  It's also a common topic in network programming.

Jonathan Merritt.

[seltsamuel]

had not the time at the moment to hexedit the files but
even when its the case that there are wrong additional bytes, in my opinion
martin-o has to fix this bugs in his 3ds exporter because its not acceptable to
have compiler dependend things inside a filespec. the file must look as specified or cross plattform stability is not granted.

think martin-o must change in his exporter that he not writes the structure
but only the values of the structure to get rid of the overhead produced by the compiler.


Seltsamuel

[lancelet]

I have a reader for G3D working in Ruby now.  I think there's only one pad byte; at the end of ModelHeader (as I indicated earlier).

So, it should be simple enough just to indentify that there's a single packing byte at the end of ModelHeader.  It would save changing the rest of the spec.

Alternatively, ModelHeader.type could become uint16, in which case it would probably gobble up the packing byte.

Jonathan Merritt.

[seltsamuel]

when this is the only occurance of the packingbyte its a solution to use uint16.
the other thing to take the byte as it is and only ignore it is no real solution as i like it, because its compiler dependent. maybe some day later or another parameter for the compiler he does it no more and it vanishes.
in my opinion a filespec must be rock solid independent from compiler, os and processor (maybe write in the specs that data is stored in little endian as normal for x86 so other platform developer see and know it).


Seltsamuel

[MatzeB]

Naaarrrggghh!!!   Also this won't solve endianess issues for you unless you manually convert each field in the struct again. So solution 2 should be alot better:

2. The alternative would be reading the elements one by one. Similar to this code:

Code: [Select]

static inline uint16_t __swap16(uint16_t val)
{
    return  (((uint16_t)(val) & (uint16_t)0x00ffU) << 8) |
            (((uint16_t)(val) & (uint16_t)0xff00U) >> 8);
}

static inline uint32_t __swap32(uint32_t val)
{
    return  (((uint32_t)(val) & (uint32_t)0xff000000U) >> 24) |
            (((uint32_t)(val) & (uint32_t)0x00ff0000U) >> 8) |
            (((uint32_t)(val) & (uint32_t)0x0000ff00U) << 8) |
            (((uint32_t)(val) & (uint32_t)0x000000ffU) << 24);
}

struct SomeHeader
{
         uint16_t field;
         int32_t another_field;
         int8_t one_more_field;

         void save(FILE* f) {
              // I like defining this inline as I can compare fields and load/save implementation more easily then...
              saveUInt16(f, field);
              saveInt32(f, another_field);
              saveInt8(f, one_more_field);
         }

         void load(FILE* f) {
              field = loadUint16(f);
              another_field = loadInt32(f);
              one_more_field = loadInt8(f);
         }
};

uint16_t readUInt16(FILE* f)
{
         uint16_t result;
         if(fread(&result, sizeof(uint16_t), 1, f) != 1)
                throw std::runtime_error("file too short");
         // fixup endianess for big endian boxes
#ifdef BIGENDIAN
         result = __swap16(result);
#endif
         return result;
}

void saveUInt16(uint16_t value, FILE* f)
{
#ifdef BIGENDIAN
        value = __swap16(result);
#endif
        if(fwrite(&value, sizeof(uint16_t), 1, f) != 1)
               throw std::runtime_error("couldn't write word");
}

// the other write/load functions work analog int8's don't need to be swapped of course :)


There are also people around who like abstracting this stuff more by using little_endian_to_host() and host_to_little_endian() functions for conversions. Because there could be hosts that are even different to little endian and big endian. But that's stuff I don't care about as such a machine doesn't exist yet AFAIK

[seltsamuel]

timeout thats me above

[martiño]

The spec needs some revision! I spent ages trying to write a G3D file reader in Python only to discover that "struct packing" of C compilers leads to unexpected byte alignment of components of the file.

You are right. I'm aware of packing issues, all other headers are "packing-safe", this is a new struct I've added in v4 and I forgot about this stuff, however I think that the problem is solved if we change this struct to be:

Code: [Select]

struct ModelHeader{
   uint16 meshCount;
   uint8 type;
   uint8 padding;
};

I think this is a better way of solving this problem, instead of using packing directives or loading elements one by one. And in the future this padding might be used for something.

I've also edited the spec to add the uint16 type.

[MatzeB]

The spec needs some revision! I spent ages trying to write a G3D file reader in Python only to discover that "struct packing" of C compilers leads to unexpected byte alignment of components of the file.

You are right. I'm aware of packing issues, all other headers are "packing-safe", this is a new struct I've added in v4 and I forgot about this stuff, however I think that the problem is solved if we change this struct to be:

Code: [Select]

struct ModelHeader{
   uint16 meshCount;
   uint8 type;
   uint8 padding;
};

Huh? This is not about padding but about alignment! Lemme explain the problem: C++ Compilers are free to layout the data in a struct as they want and as what they think is fastest. You are relying on compiler internals by reading/writing out structs directly, you never know what optimisations a compiler might do. On x86 memory access is fastest when the address of the data is a multiple of 4. So the compiler will realgin your struct to something like this:

Code: [Select]

struct Modelheader {
   uint16 mechCount;
   uint8 __padding0[2]; // added so that the address of type is at a multiple of 4
   uint8 type;
   uint8 __padding1[3]; // added so that the address of padding is at a multiple of 4
   uint8 padding;
   // some compilers add 3 bytes more packing now to make the headers size a multiple of 4 too...
};
This is surely not what you want!

I think this is a better way of solving this problem, instead of using packing directives or loading elements one by one. And in the future this padding might be used for something.

You should still read the struct element by element, it is needed for Mac users anyway because they don't have a little endian machine which needs conversion of every field bigger than uint8 anyway!!!

[martiño]

struct Modelheader {
   uint16 mechCount;
   uint8 padding0[2]; // added so that the address of type is at a multiple of 4
   uint8 type;
   uint8 padding1[3]; // added so that the address of padding is at a multiple of 4
   uint8 padding;
   // some compilers add 3 bytes more packing now to make the headers size a multiple of 4 too...
};

You are right in the fact that this is compiler dependent stuff, and that is not a good idea to rely on structs for reading from files.

But as far as MSVC is concerned, and I'm 99% sure thet GCC does the same the alignment doesn't work like you said.

From MSVC help:

each structure member after the first is stored on either the size of the member type or n-byte boundaries (where n is 1, 2, 4, 8, or 16), whichever is smaller.

This means that if you store types fom bigger to smaller and the size of the struct is a multiple of the size of the first member no padding is going to be added.

[martiño]

You should still read the struct element by element, it is needed for Mac users anyway because they don't have a little endian machine which needs conversion of every field bigger than uint8 anyway!!!

Allright. I will deal with endianess problems at some point and this will surely include loading struct member one by one, although another option would be swapping data in disc.

[MatzeB]

each structure member after the first is stored on either the size of the member type or n-byte boundaries (where n is 1, 2, 4, 8, or 16), whichever is smaller.

This means that if you store types fom bigger to smaller and the size of the struct is a multiple of the size of the first member no padding is going to be added.

Seems you're right. gcc on x86 seems to do the same. In fact g++ version 3.2 and newer is implementing some vendore neutral ABI standards that msvc seems to follow too, so for gcc and windows this should be save as long as we're on x86 and not on exots...
Still I don't have a good feeling with this solution...

[MatzeB]

You should still read the struct element by element, it is needed for Mac users anyway because they don't have a little endian machine which needs conversion of every field bigger than uint8 anyway!!!

Allright. I will deal with endianess problems at some point and this will surely include loading struct member one by one, although another option would be swapping data in disc.

What do you mean by swapping data on disc? Creating a separate data package for mac users? That will most likely result in maintenance problems... (you know these "oops forgot to convert that one file...")

[martiño]

Code: [Select]

Seems you're right. gcc on x86 seems to do the same. Still I don't have a good feeling with this solution... in gcc it's not explicitely documented how the fields are stored internally.
Ok, think the packing option might be the best solution by now, although it would break the current format.

By the way, there is a way of specifying alignment that works on both MSCV and GCC:

Code: [Select]

#pragma pack(push, 1)

struct something
{
...
}

#pragma pack(pop)

[MatzeB]

Code: [Select]

Seems you're right. gcc on x86 seems to do the same. Still I don't have a good feeling with this solution... in gcc it's not explicitely documented how the fields are stored internally.
Ok, think the packing option might be the best solution by now, although it would break the current format.

I just did some research. There's a vendor neutral c++ ABI standard that gcc 3.2 and msvc7 are following. So we should be on the safe side here.

By the way, there is a way of specifying alignment that works on both MSCV and GCC:

Code: [Select]

#pragma pack(push, 1)

struct something
{
...
}

#pragma pack(pop)

Yes seems they added #pragma pack to latest gcc versions too (__attribute__((packed)) solution was nicer to read IMO...).
My opinion for now is leave it as is and read the struct field by field later. No need to fiddle around with #pragmas now when we're going to change it anyway.

[seltsamuel]

ok, so i will fill in a dummy byte in the converter.


but by the way how is it opted on 64bit machines ?



Seltsamuel