Object Persistence

[daniel.santos]

I'm in the middle of reading this article hailstone linked on IRC last night: http://www.gamasutra.com/view/feature/3683/a_templated_c_attribute_library_.php.  I'm also in the middle of un-breaking a lot of stuff, so I don't want to spend too much time on this article or its concepts right now.  However, I have a feeling you're onto something very important and I think this deserves a very close look.  Such a mechanism can potentially replace (or mostly replace) the way we:

• loading & save games
• loading & saving other objects, like models, textures, maps, etc. in a platform portable way (currently models & maps wont load correctly on PPC or ARM)
• transmit data across the network
• expose objects to Lua (instead of luabind)
• print objects for debugging (ala the new Shared::Util::ObjectPrinter)
• Facilitate debug windows that will allow you to see & change fields of game objects

If nothing else, such a proposed mechanism can be the primary interface to this other functionality, even if we keep much of the existing code (minus the per-object serialization/deserialization functions).  I used a similar library when working on Apache Tomcat, they have these XML persistence classes in the Apache Commons project that's somewhat similar, only it uses introspection instead of pointers.  Thanks for linking this hailstone!

One last thing I wanted to mention (slightly off topic) is that I was playing this game called Widelands recently.  Their UI is very strange -- you right click on windows to make them go away and you left click on game objects to get a pop-up menu.  But one of the options is "debug" and this opens a window that has a debug display of the object similar to my ObjectPrinter's output.  This could be very helpful in GAE and when we do the CEGUI, I think I would very much like such an option in the game.  Of course, it should only be enabled with a pre-processor flag because the information could be used for cheating.

EDIT 2009-12-10: revised list of affected mechanisms

[daniel.santos]

OK, after reading the whole article, here are my thoughts:

I don't like the idea of having each instance of an object bind it's values to a string.  I think a smarter approach is to have a single object (instance of a template) created for each serializable object type that stores the address to getter & setter functions.  This forces a function call of what would otherwise be in-lined functions, but that's OK for this purpose.  Otherwise, we end up with hundreds of copies of the same strings in memory without a good reason.  Example:

Code: [Select]

/** Base class of Attributes, so we can store them generically in a map? */
template<typename ObjectType>
class Attribute {
private:
    string name; /**< attribute name, unique to the AttributeManager instance.  */
    int flags;   /**< read-only, temporary, transient, etc. */
};

/** Sub-class (interface) which also specifies the type of the attribute */
template<typename ObjectType, typename T>class AttributeTypeImpl : public Attribute<ObjectType> {
public:
    virtual T get(ObjectType &o) = 0;
    virtual void set(ObjectType &o, T v) = 0;
};

/**  Implementation, that uses a getter & setter (for when your value has special needs) */
template<typename ObjectType, typename T, typename GetterType = T, typename SetterType = T>
class AttributeFuncCallImpl : public AttributeTypeImpl<ObjectType, T> {
private:
    boost::function<GetterType ()> getter;       /**< The getter function */
    boost::function<void (SetterType v)> setter; /**< The setter function */

public:
    T get(ObjectType &o)         {return o.getter();}
    void set(ObjectType &o, T v) {o.setter(v);}
};

/**  Implementation, that uses offset of the data member */
template<typename ObjectType, typename T>
class AttributePointerImpl : public AttributeTypeImpl<ObjectType, T> {
private:
    intptr_t offset;

public:
    T get(ObjectType &o)         {return getRef(o);}
    void set(ObjectType &o, T v) {getRef(o) = v;}

private:
    T &getRef(ObjectType &o)     {return *reinterpret_cast<T*>(reinterpret_cast<intptr_t>(&o) + offset));}
};

template<typename T> class AttributeManager {
public:
    typedef map<string, Attribute> Attributes;
}

class UnitAttributeManager : public AttributeManager<Unit> {
// contains specific code for serialization & deserialization of Unit objects.
}

I do want the ability to bind arbitrary values to objects in Lua, example: I may want to add a "pickleCount" variable to some Unit objects, but we can do this with a separate class hierarchy.  So essentially, I'm thinking of having the functions to access objects and their values from Lua work one way, and then allowing some type of get/set-Attribute Lua call to bind arbitrary values to game objects for the purpose of scripting.

I'm going to get back to my trunk refactoring now, but please let me know what you guys think. Given the number of mechanisms now in place or proposed for object state exposure & manipulation, I think a solution similar to this proposal is very appropriate.

[silnarm]

One last thing I wanted to mention (slightly off topic) is that I was playing this game called Widelands recently.  Their UI is very strange -- you right click on windows to make them go away and you left click on game objects to get a pop-up menu.  But one of the options is "debug" and this opens a window that has a debug display of the object similar to my ObjectPrinter's output.  This could be very helpful in GAE and when we do the CEGUI, I think I would very much like such an option in the game.  Of course, it should only be enabled with a pre-processor flag because the information could be used for cheating.

Yay!
_GAE_DEBUG_EDITION_ was intended for just this, as was the addition of DebugRenderer. All I've used the DebugRenderer for so far is 'colouring' cells and drawing big arrows all over the map, but I always wanted to have the ability to select a unit (or units!) and turn on a 'debug billboard', with detailed information on that unit and what it is doing (unit stats, command stats, & ultimately... AI state).  Just waiting on that damn GUI issue to be sorted

Ok, I peeked at the article when Hailstone mentioned it, and thought a 'template object serialisation' framework was a good idea, I decided I should actually read it and just did so.... always dangerous, I think I spend as much time reading about the patterns they are using as I do the actual article... and then I often start getting crazy ideas... anyway... enough of that.

I don't like the idea of having each instance of an object bind it's values to a string.  I think a smarter approach is to have a single object (instance of a template) created for each serializable object type that stores the address to getter & setter functions.  This forces a function call of what would otherwise be in-lined functions, but that's OK for this purpose.  Otherwise, we end up with hundreds of copies of the same strings in memory without a good reason.

string pooling should take care of that, does gcc offer something of this ilk? Not that I'm using that to argue against what your angling for, I think the per-instance object is crap too

Code: [Select]

/** Base class of Attributes, so we can store them generically in a map? */
... SNIP ...
[/quote]

It's late, trying to follow all those template parameters around is making my head hurt... I'll have to have a look at this tomorrow, as I said, I prefer the one serialiser per type, and a system like this would be quite useful I think... so it's definitely worthy of spending some time on, and 'rolling your own' is more fun too!

I do want the ability to bind arbitrary values to objects in Lua, example: I may want to add a "pickleCount" variable to some Unit objects, but we can do this with a separate class hierarchy.

If pickleCount isn't of interest to the engine, this could be kept only on the Lua side. To be honest, I'm still agonising over the Lua interface, and as the previous sentence indicates, and even currently thinking of having Units 'shadowed' in the Lua state... but I'm just throwing wild and off-topic ideas around now   Whatever we do decide with the lua interface, it could no doubt be made a bit easier with this.

... Given the number of mechanisms now in place or proposed for object state exposure & manipulation, I think a solution similar to this proposal is very appropriate.

Agreed, this can simplify a bit of existing code, and will make some proposed code easier & quicker to implement.

[daniel.santos]

OK, I edited my original approach and added some more items to the list of mechanisms this can (should?) be applied to.  I've worked out the concept for this mechanism a little more and I'll write more on it later.  In essence, I'm saying that we can create an object that will specify rather a function call should be used to get or set the values or if we can just manipulate it directly in memory.  This should reduce the overhead of function calls, as well as the generation of getter & setter functions in the object code, that the compiler will normally not generate for in-lined functions, while also facilitating access to values that require getter & setter calls (e.g., values that shouldn't be accessed asynchronously, etc.).  I like a lot of how Boost.Serialization works, but I just can't buy into everything they do.  So I'm thinking we can have a small macro:

#define GAE_ATTRIBUTE(field, getter, setter, flags) // some definition

The only real reason for the macro is to be able to stringize the field name, like we're doing in STRINGY_ENUM. So then a template<class ObjectType> AttributeManager (or we could call it ObjectSerializer, or whatever) that will serialize & deserialize objects of type Unit could be defined something like this:

Code: [Select]

class UnitSerializer : public AttributeManager<Game::Unit> {
public:
    UnitSerializer();
};

UnitSerializer::UnitSerializer() {
    GAE_ATTRIBUTE(id, NULL, NULL, 0); // don't use a getter or setter and no special flags
    GAE_ATTRIBUTE(hp, NULL, setHp, 0); // directly access from memory for reads, but call setter for writes
    GAE_ATTRIBUTE(ep, getEp, setHp, 0); // use both getter & setter
    GAE_ATTRIBUTE(rotation, NULL, NULL, Attribute::NO_NETWORK); // don't send this across the network
    // etc...
}

And actually, I'm wondering if this can be re-architected in such a fashion that we can simply add a single function like this to each class to further simplify it.  Example:

Code: [Select]

class Unit {
// blah blah blah
public:
    shared_ptr<AttributeManager<Unit> > getAttributeManager() {
        shared_ptr<AttributeManager<Unit> > ret = shared_ptr<AttributeManager<Unit> >(new AttributeManager<Unit>);
        ret->GAE_ATTRIBUTE(id, NULL, NULL, 0);
        ret->GAE_ATTRIBUTE(hp, NULL, setHp, 0);
        // etc...
    }
};

Anyway, the point is to keep it as simple, easy to implement, succinct, type safe and error-proof as possible.

[Loronal]

yah my maps are always getting errors when i load save games. They dont open they say the map is invalid and that the game is impossible.

[-Archmage-]

I'm not having any problems loading games........

[daniel.santos]

yah my maps are always getting errors when i load save games. They dont open they say the map is invalid and that the game is impossible.

I thought we fixed the problem with the path to the map being incorrect in saved games. (I don't remember which version that was, maybe 0.2.12a?)  Anyway, if you were using a PPC (some Macs) then none of the game would work because quite a hefty portion of the code is using non-architecture-neutral code with regard to endianness (see http://en.wikipedia.org/wiki/Endianness).  For development, the major benefits of this not mentioned in the bullet list at the top of the thread will be less code (both source code and generated code, so binaries will be slightly smaller) and it will be less bug-prone because we wont have to maintain separate code for various serialization mechanisms (XML, binary, Lua bindings, etc.)