Unit Testing

[hailstone]

I've been studying unit testing at university. Would it be good for GAE? I talk especially about the new network code since we have a design to base the tests around.

[daniel.santos]

hehe, yea, I suppose so.  I haven't made good use of unit testing in my career to be honest and I know that it can greatly reduce time spent on code because it tells you when you break it.  So sure, we'll just add two more jam targets, one to build the tests and another to run them.  I suppose we can put them under source/test.  I'm not familiar with unit test frameworks on C/C++ (I've only used them from Java) so if you have any suggestions, please chime in.  Come to think of it, we can stage the entire lobby network dialog in a set of unit tests and include any number of mechanisms to try to break it.  It will be interesting to see how some of this code functions outside of the environment that I usually see it run in too.   Some things may be too difficult to unit test without a lot more coding however, like world updates & such.  Units live in Factions which live in a World which has a Map and is all owned by a Game and that's a pretty heavy object structure.  But networking code & such can surely use it.

[bork]

A good idea really. Just don't be too eager to add unit tests for existing code, you might get disappointed a bit.
As for testing frameworks you should give Boost.Test a try definitely. Probably the best choice when you are not constrained by exotic platforms.

[daniel.santos]

Thanks for the suggestion bork.  I've yet to learn Boost and I understand that a lot of Boost is being integrated into C++0x, so it's something I've been meaning to learn more about anyway and I've thought about introducing it into GAE as a dependency, but we have enough broken at the moment that I think that part can wait.  But using it for a test framework, I don't mind at all.

[hailstone]

I suppose we can put them under source/test.

If I recall correctly, I read that it's good to keep the tests in the same directory as the class it is testing.

I'm not familiar with unit test frameworks on C/C++ (I've only used them from Java) so if you have any suggestions, please chime in.

I've only used JUnit but http://geosoft.no/development/unittesting.html recommends http://cxxtest.tigris.org/ . It uses python though. CPPUnit is meant to be a port of JUnit.

I've read that a testing framework is fairly easy to make so that might be an option also.

[bork]

I suppose we can put them under source/test.

If I recall correctly, I read that it's good to keep the tests in the same directory as the class it is testing.

Well, so, why it is good to keep them in the same directory? As far as I know, this might make some sense for Java, because of additional effort you might need to access package internals from testing package. However, having tests and tested classes in the same directory does not add anything for C++, except for more clutter maybe.

I'm not familiar with unit test frameworks on C/C++ (I've only used them from Java) so if you have any suggestions, please chime in.

I've only used JUnit but http://geosoft.no/development/unittesting.html recommends http://cxxtest.tigris.org/ . It uses python though. CPPUnit is meant to be a port of JUnit.

CxxTest looks rather attractive at first glance, but it's not very well suitable for large enough projects. It forces you to keep tests in headers which leads to recompiling whole test stuff when something changed. Use of setup/teardown methods is also not very good idea, because you'll have to dynamically allocate almost everything you need in your fixtures.

Just compare

Code: [Select]

class MyCxxTestFixture
{
public:
    void Setup()
    {
        this->someString = new std::string("Some String");
    }

    void Teardown()
    {
        delete this->someString;
    }

private:
    std::string *someString;
};

with

Code: [Select]

class MyFixture // you'll get this with Boost.Test, CppUnitLite and many others
{
public:
    MyFixture()
        : someString("Some String")
    {
    }

private:
    std::string someString;
};

You'll get this same issue with CppUnit, which rather verbose in any case.

I've read that a testing framework is fairly easy to make so that might be an option also.

Yeah, it's well known that Kent Beck advised to develop your own unit test framework, but I don't think he meant C++  I think one can write pretty usable framework for .NET in about in hour, but it's not so easy for C++.

[daniel.santos]

Well the area that it could immediately benefit the most is the networking code rewrite and I'm starting to think that it's a wonderful way forward.  For one, it wont require me to start up two instances of the game, go through the sequence of hosting a game on one side and joining it on the other -- they could execute in the same process in the test code.  For another, it seems like a good way to enforce loose coupling -- problems with the networking layer are discovered there and problems with the UI will be easier to troubleshoot (harder to unit test UI, but not impossible).  Finally, it will cause design problems which lead to tight coupling to scream at you, that is, when you run into tight coupling, it will become very obvious when writing the test for that code.

So thanks for all of the input on this topic bork.  I definitely prefer to keep tests separated from the original code.  As for the "in the same directory" concept, I agree that it's quite helpful with JUnit because of Java's package/directory hierarchy scheme for sources.  Even as such, when I've used JUnit in the past, I've always put tests under ./src/test.  [OT] I even had the SCM team (Source Control Management) tell me to take the "test" directory out because they thought I just created it for some test.  I had to explain unit tests to them.  I really better not get started complaining about my previous job, it sucked! [/OT] Back on topic, the other problem is Matze's jam-based build system just compiles all c/cpp files that it finds in the source tree you throw at it.  I'm sure it could be modified to perform otherwise, but that would also be breaking its paradigm (and I don't understand it that deeply).  As it is, I'll need to confront the build system to create a target to build the tests.

So I'm going to give CPPUnit a spin and see how it works out.  If it's simple enough, then that'll be my vote.

[daniel.santos]

I've implemented a basic test program using CPPUnit and modified the Jamfile to create it, but I'm not at all happy with how it works and will need to be fixed.  As it is, you have to run "jam testgae" and not "jam" or else you'll end up running two instances of gcc for the same source file at the same time (when using the -jx flag). But it's sufficient to get started at least.  I also discovered that I can't create an executable name that's the same as the subdirectory with the current build setup.  I don't like this and I'll probably modify it because it makes me sad. :'(  So the sources are under ./source/test, but the executable it generates is called testgae.  I'm not worried about using the same executable name as a standard executable or shell built in (/usr/bin/test) because if you've added the current directory to your PATH (the ".") then you get what you deserve! 

Either way, it's already proving beneficial so I'm glad this suggestion came up!

[daniel.santos]

So just to follow up on this thread, as of rev 306, the way you build the unit tests is with:

Code: [Select]

BUILD_UNIT_TESTS=yes jamAltnerately, you can edit the Jamconfig and add the following anywhere:

Code: [Select]

BUILD_UNIT_TESTS = yes ;
The test executable is named testgae and requires CPPUnit to be installed on your machine.

[hailstone]

I've made a template with CPPUnit (see trunk/test). I was wanting a tests folder in each directory and then have a project that compiled all the tests (only for includes in shared_lib). Shared_lib and game need to have individual test projects because game relies on shared_lib. It would be easy to add "tests" to the Glest dirs in the Jamfile.

I'm not sure how to cleanly organise it in VS. Maybe there is a way to use the compiled object files of the other projects without needing to add the source to the test project? Also need a place to store the files that you use for testing such as a ini file when testing the properties class.

EDIT: I found out that obj files that were compiled from other projects can be added to Additional Dependencies. The test files are the only files needing to show for the project then. The other project will be a dependency for the test project.

[silnarm]

I've made a template with CPPUnit (see trunk/test). I was wanting a tests folder in each directory and then have a project that compiled all the tests (only for includes in shared_lib). Shared_lib and game need to have individual test projects because game relies on shared_lib. It would be easy to add "tests" to the Glest dirs in the Jamfile.

I'm not sure how to cleanly organise it in VS. Maybe there is a way to use the compiled object files of the other projects without needing to add the source to the test project? Also need a place to store the files that you use for testing such as a ini file when testing the properties class.

EDIT: I found out that obj files that were compiled from other projects can be added to Additional Dependencies. The test files are the only files needing to show for the project then. The other project will be a dependency for the test project.

Cool, I'll have a proper look when I get a chance... if it all works out I'll add a 'template' test project to the VC solution file, or feel free to beat me to it .

[hailstone]

I found another testing framework.  http://code.google.com/p/googletest/

[daniel.santos]

I'm satisfied with cppunit personally.  If you look in the network branch, you'll see that I've had some unit tests for the networking code, although it's a bit trivial at this point.  However, it's when we get to writing unit test code that we definitely come into conflict with any over-use of the Singleton pattern.  I had to modify my stuff just to get these tests to work!

I should probably seriously consider finishing the networking re-write using TDD (test-driven development).  One nice thing about it is that when you get something to work and you later break it by accident, you know right away!