WIP: New TMP-based Network Messages (for programmers)

[daniel.santos]

So first off, hello to everybody, sorry for having been away for so long.  Life can get pretty busy sometimes.

EDIT 1: TMP stands for "Template Meta-programming." It's a newer technique for writing C++ code that generates C++ code via the standard compiler.
EDIT 2: This post is intended for Glest programmers (sorry, no modding info here)

I started this out as an email to the list, but I figured that, with the size of it, this is probably a better place for it.

I'm still playing with a new mechanism here, and it's pretty cool.  However, it's a fair distance from being ready. None the less, I want to show it to you guys!  This is a first run actually implementing these techniques and it'll get cleaner as I learn this stuff better and implement it into pretty macros.  So don't let the ugliness scare you off too quickly!  Since this is a big posting, I'll separate it into sections.

[big]Summary[/big]
In a nutshell, we add some code to util.h (or some such) then a little at the top of network_mesage.h and, finally, we change the member definition of the message classes as follows:
 

Code: [Select]

class MyMessage : public Message {
private:
    DEFINE_MSG_MEMBERS(
        NetworkString<16>,  someString,
        int,                someInt,
        float,              someFloat
    )

public:
    // constructor, destructor, etc.
    virtual void read(NetworkDataBuffer &buf);
    virtual void write(NetworkDataBuffer &buf) const;
};
Then, in NetworkMessage.cpp, we define the read and write functions as follows:
 

Code: [Select]

void MyMessage::read(NetworkDataBuffer &buf) {
    metafunc_it::read(buf, this);
}
 
void  MyMessage::write(NetworkDataBuffer &buf) const {
    metafunc_it::write(buf, this);
}
All of the labor of sorting out types and such is performed at compile time.  No actual function call is made here when compiled with optimizations.  Instead, all of the code to do the read & write is completely unrolled into the function bodies, and there's no looping.  The only possible exception is if the compiler decides that something in NetworkDataBuffer or it's base class should be out-lined.

[big]Gory Details[/big]
For the gory details, this is what we add to util.h (or some such).  This is the code for iterating through a "static list" (i.e., a compile-time list) and performing a generic read/write operation.  This can likely be cleaned up with boost:bind or Boost.Fusion, but I haven't gotten that far yet.
 

Code: [Select]

#include <boost/mpl/list.hpp>
#include <boost/mpl/deref.hpp>
#include <boost/mpl/next.hpp>
#include <boost/mpl/begin_end.hpp>

// This is a recursive template.  Notice that it calls it's self in each function.
template<typename S, typename C, typename I = typename begin<S>::type> struct list_it {
    static void read(NetworkDataBuffer &buf, C &msg) {
        boost::mpl::deref<I>::type::apply(buf, msg);
        list_it<S, C, typename boost::mpl::next<I>::type >::read(buf, msg);
    }
    static void write(NetworkDataBuffer &buf, const C &msg) {
        boost::mpl::deref<I>::type::apply(buf, msg);
        list_it<S, C, typename boost::mpl::next<I>::type >::write(buf, msg);
    }
};
 
// This template specialization is what's called a "recursion terminator", as it prevents
// the above template from becoming infinitely recursive.
template<typename S, typename C> struct list_it<S, C, typename end<S>::type > {
    static void read(C &o) {}
    static void write(C &o) {}
};
Next, this is defined at the top of network_message.h
 

Code: [Select]

// This is the actual metafunction collection passed later on to list_it.  Here, I'm storing
// the pointer to the data member at compile time (when the template is instantiated), a
// feature introduced in C++98
template<typename R, typename C, R C::* M>
struct net_member_funcs {
    static void read(NetworkDataBuffer &buf, C &msg)        {buf.read(msg.*M);}
    static void write(NetworkDataBuffer &buf, const C &msg) {buf.write(msg.*M);}
};
Finally, this is what the Message-derived classes look like after pre-processing.  I'm using recursive vardic pre-processor macros, a new trick I've recently learned.
 

Code: [Select]

class MyMessage : public Message {
private:
    NetworkString<16>   someString;
    int                 someInt;
    float               someFloat;
    typedef boost::mpl::list<
         net_member_funcs<NetworkString<16>, MyMessage, &MyMessage::someString>,
         net_member_funcs<int, MyMessage, &MyMessage::someInt>,
         net_member_funcs<float, MyMessage, &MyMessage::someFloat>
     > metafuncs;
    typedef list_it<metafuncs, A> metafunc_it;

    virtual void read(NetworkDataBuffer &buf);
    virtual void write(NetworkDataBuffer &buf) const;
};
This constructs, at compile time, a list (that is actually just a template that recurses 3 times, plus a terminator) which stores each of my data members.  Passing the types of the data members shouldn't be necessary, because I can have the template figure that out automatically, I just haven't gotten the syntax right yet.
 
So then, when I actually have an instance of MyMessage and I'm ready to read or write, I just have to call metafunc_it::read(buf, this); or metafunc_it::write(buf, this); and it all happens by magic.
 
[big]More Than Just Network Data[/big]
So I'll get back with you guys on this when it's much more solid.  I also want to include features to read & write XML (not needed so much for network messages) and also to dump the contents of each message in the pretty formatted fashion I have now, which looks something like this:
 

Code: [Select]

MyMessage = {
    Message = {
        type = 1
        simRxTime = 5793459823
    }
    someString = "hello world"
    someInt = 42
    someFloat = 3.14159265
}
I'm doing "intrusive" so the classes have to be modified.  Boost.Serialization lets you do all of this unintrusively, which means you don't have to change the classes.  However, a.) it's a huge bloated fat monster weighing in at 400kb!!!, b.) it uses RTTI and c.) it is much more CPU expensive.
 
So all that would have to be added is a function (like below) and the functionality added to the member_funcs struct.  In practice, however, it will eventually be separate metafunctions cleverly glued together at compile time using techniques I have yet to learn, as having multiple metafunctions in a single struct is considered a "Blob" in TMP, which is an anti-pattern as it slows down compile time unnecessarily.  So the implementation of these is just the addition of a few functions to the class:

Code: [Select]

class MyClass {
...
    virtual void dump(ObjectPrinter &op) const;
    virtual void read(const XmlNode &node);
    virtual void write(XmlNode &node) const;
};

// in cpp file:
void MyClass::dump(ObjectPrinter &op) const {
    BaseClass::dump(op); // force call to base class, which should do the same thing this one does.
    metafunc_it::dump(op, this);
}

void MyClass::read(const XmlNode &node) {
    metafunc_it::read(node, this);
}

void MyClass::write(XmlNode &node) const {
    metafunc_it::write(node, this);
}
Further, all of these function definitions could be encapsulated in pre-processor macros if it made sense to do so.  This is inconvenient at times, especially when debugging or attempting to decypher code you haven't seen before.

All of this is actually the basis of the data engine of a new (and quite ambitious) project I plan on launching next year to better facilitate open source gaming in general.

EDIT: fixed formatting and added section I forgot.

[Omega]

Say, could you also add a section explaining what this change would mean to regular players/modders? My debilitated apperception cannot process your interminably operose dissertation (yes, that was a thesaurus).

[daniel.santos]

It has very little impact on modders.  Sorry for not being clear about that! It's basically a mechanism of cleaning up networking code and reducing the overall source code size (not the generated code, i.e., the size of the executable file(s)).  However, in the future, it might be a mechanism for linking the internal data of the game to a wide assortment of external things, like GUIs, Lua, XML (save game data), etc. and that likely would reduce both the source code size and the generated code size (at least for the feature set).

[silnarm]

Looks fairly nice, though I wont be looking anything up to figure out all that boost::mpl stuff any time soon

Have you considered how to handle collections yet?

I was working on a 'generic' serialisation / value-visiting framework a while back, abandoned in the end because it got really ugly (it was all macro based), but something that might be nice from it is using Reader/Writer objects to do all the 'work', so the classes modified (made serialisable) don't need read/write function pairs for multiple things (ie, network-buffer, xml-reader, table in lua-state, etc).

it went something like,

Code: [Select]

class Serialiser {
public:
    virtual void visitInt(const char *name, int value) = 0;
    virtual void visitFixed(const char *name, fixed value) = 0;
    virtual void visitVec2i(const char *name, Vec2i value) = 0;

    // for collections and sub-objects
    virtual void pushName(const char *) = 0;
    virtual void popName() = 0;
};


Not sure if it going to even be possible to do such a thing with what you've got, but would be nice to have, once everything has been modified to use the framework, 'serialising' to a Lua state is as simple as implementing a class LuaStateSerialiser : public Serialiser { ... }

[daniel.santos]

hmm, thanks.  That is definitely missing from NetworkBuffer.  It should derive from such a class that is 100% abstract (what Java would call an "interface").  However, I think it should rely on operator overloading, e.g.:

Code: [Select]

class Serialiser {
public:
    virtual void visit(const char *name, int value) = 0;
    virtual void visit(const char *name, fixed value) = 0;
    virtual void visit(const char *name, const Vec2i &value) = 0;
    template<Class T> virtual void visit(const char *name, vector<T> &value) = 0;

    // for collections and sub-objects
    virtual void pushName(const char *) = 0;
    virtual void popName() = 0;
};
I'm not even sure if you can have a pure virtual templatized function.  In fact, it sounds odd from the get go.  Also, we should use the American spelling "Serialization" because we here in the states are elitist a**holes who use "Z" instead of "S", because we want to think we're better in some way.   [insert stupid Bush quote here, e.g., "they hate us for our freedom!"].

OK, seriouzly now.  I'll be playing with this and working on it more.  In some cases, the name is needed and in others, it isn't.  I'm examining ways to do this non-intrusively or semi-non-intruzively as well.