SegF on 3.9.0, Ubuntu 13.10 i686 headless (fakir), Map::getSurfaceCell

[tomreyn]

Two days ago, the headless server on fakir, the EC2 instance in Brazil, crashed with:

Code: [Select]

[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/i386-linux-gnu/tls/i686/nosegneg/libthread_db.so.1".
Core was generated by `./megaglest --ini-path=./ --data-path=./ --headless-server-mode=exit'.
Program terminated with signal 6, Aborted.
#0  0xb77de424 in __kernel_vsyscall ()
#0  0xb77de424 in __kernel_vsyscall ()
#1  0xb6de9aff in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#2  0xb6ded083 in __GI_abort () at abort.c:90
#3  0xb701b605 in __gnu_cxx::__verbose_terminate_handler() () from /usr/lib/i386-linux-gnu/libstdc++.so.6
#4  0xb7019273 in ?? () from /usr/lib/i386-linux-gnu/libstdc++.so.6
#5  0xb70192af in std::terminate() () from /usr/lib/i386-linux-gnu/libstdc++.so.6
#6  0xb701950e in __cxa_throw () from /usr/lib/i386-linux-gnu/libstdc++.so.6
#7  0x08234804 in Glest::Game::Map::getSurfaceCell (this=0xa87fe2c, sx=0, sy=0) at /home/softcoder/Code/megaglest/trunk/source/glest_game/world/map.h:286
#8  0x0874e4b8 in Glest::Game::Map::end (this=0xa87fe2c) at /home/softcoder/Code/megaglest/trunk/source/glest_game/world/map.cpp:324
#9  0x08778b2a in Glest::Game::World::end (this=0xa87fdf8) at /home/softcoder/Code/megaglest/trunk/source/glest_game/world/world.cpp:197
#10 0x082e1105 in Glest::Game::Game::~Game (this=0xa87fdc8, __in_chrg=<optimized out>) at /home/softcoder/Code/megaglest/trunk/source/glest_game/game/game.cpp:431
#11 0x08401f4e in Glest::Game::Program::~Program (this=0xa84b278, __in_chrg=<optimized out>) at /home/softcoder/Code/megaglest/trunk/source/glest_game/main/program.cpp:283
#12 0x083abd07 in Glest::Game::ExceptionHandler::handleRuntimeError (msg=0x0, getStackTraceString=true) at /home/softcoder/Code/megaglest/trunk/source/glest_game/main/main.cpp:722
#13 0x083ac361 in Glest::Game::handleSIGSEGV (sig=11) at /home/softcoder/Code/megaglest/trunk/source/glest_game/main/main.cpp:5567
#14 <signal handler called>
#15 0xb6f6b698 in main_arena () from /lib/i386-linux-gnu/tls/i686/nosegneg/libc.so.6
#16 0x085e20e7 in Glest::Game::ConnectionSlot::validateConnection (this=0xa8c7850) at /home/softcoder/Code/megaglest/trunk/source/glest_game/network/connection_slot.cpp:1535
#17 0x085e859e in Glest::Game::ConnectionSlot::update (this=0xa8c7850, checkForNewClients=false, lockedSlotIndex=-1) at /home/softcoder/Code/megaglest/trunk/source/glest_game/network/connection_slot.cpp:1502
#18 0x085f1d2d in Glest::Game::ConnectionSlot::updateSlot (this=0xa8c7850, event=0xb2cfc1e0) at /home/softcoder/Code/megaglest/trunk/source/glest_game/network/connection_slot.cpp:576
#19 0x085f21d1 in slotUpdateTask (event=<optimized out>, this=0xa87f380) at /home/softcoder/Code/megaglest/trunk/source/glest_game/network/connection_slot.cpp:175
#20 Glest::Game::ConnectionSlotThread::execute (this=0xa87f380) at /home/softcoder/Code/megaglest/trunk/source/glest_game/network/connection_slot.cpp:309
#21 0x0888b080 in Shared::Platform::Thread::beginExecution (data=0xa87f380) at /home/softcoder/Code/megaglest/trunk/source/shared_lib/sources/platform/sdl/thread.cpp:256
#22 0xb774b355 in ?? () from /usr/lib/i386-linux-gnu/libSDL-1.2.so.0
#23 0xb778fd6b in ?? () from /usr/lib/i386-linux-gnu/libSDL-1.2.so.0
#24 0xb7726d78 in start_thread (arg=0xb2cfcb40) at pthread_create.c:311
#25 0xb6ead01e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:131
This is the GDB backtrace on the core file it generated then. We also have this trace MegaGlest wrote itself, but this lacks line numbers:

Code: [Select]

terminate called after throwing an instance of 'Shared::Platform::megaglest_runtime_error'
  what():  surfaceCells == NULL
Stack Trace:
./megaglest:Shared::Platform::megaglest_runtime_error::megaglest_runtime_error(std::string const&, bool)address [0x8881bee] line: 0
./megaglest:Glest::Game::Map::getSurfaceCell(int, int) constaddress [0x82347e4] line: 0
./megaglest:Glest::Game::Map::end()address [0x874e4b8] line: 0
./megaglest:Glest::Game::World::end()address [0x8778b2a] line: 0
./megaglest:Glest::Game::Game::~Game()address [0x82e1105] line: 0
./megaglest:Glest::Game::Program::~Program()address [0x8401f4e] line: 0
./megaglest:Glest::Game::ExceptionHandler::handleRuntimeError(char const*, bool)address [0x83abd07] line: 0
./megaglest:Glest::Game::handleSIGSEGV(int)address [0x83ac361] line: 0
[0xb77de400]address [0xb77de400]
/lib/i386-linux-gnu/tls/i686/nosegneg/libc.so.6:()address [0xb6f6b698] line: 0
./megaglest:Glest::Game::ConnectionSlot::update(bool, int)address [0x85e859e] line: 0
./megaglest:Glest::Game::ConnectionSlot::updateSlot(Glest::Game::ConnectionSlotEvent*)address [0x85f1d2d] line: 0
./megaglest:Glest::Game::ConnectionSlotThread::execute()address [0x85f21d1] line: 0
./megaglest:Shared::Platform::Thread::beginExecution(void*)address [0x888b080] line: 0
/usr/lib/i386-linux-gnu/libSDL-1.2.so.0:()address [0xb774b355] line: 0
 The lack of line numbers is because I did not have add2line (binutils) installed - I just added this package to the headless installation instructions on the wiki.

We also have a DMP file available which I did not analyze, yet (please let me know if it's needed).

Please also note that I forgot to run the headless with the vps command line option, which caused it to write 7.5 GB of this (repeatedly) to standard error (or maybe standard output, not sure):

Code: [Select]

server command []
hang up in file descriptor

[tomreyn]

Looks like fakir ran into this one again.

Here's the end of its standard output:

Code: [Select]

*TESTING*: START Waiting for lagging client playerIndex = 1 [KINGOFGLEST] clientLagCount = 12.000000 [0.000000]
*TESTING*: START Waiting for lagging client playerIndex = 0 [nig] clientLagCount = 12.000000 [0.000000]
*TESTING*: FINISHED Waiting for lagging client playerIndex = 0 [nig]
*TESTING*: START Waiting for lagging client playerIndex = 0 [nig] clientLagCount = 14.000000 [0.000000]
Switching masterserver admin to slot#0...
In [/home/softcoder/Code/megaglest/trunk/source/glest_game/main/main.cpp::handleSIGSEGV Line: 5563] Error detected: signal 11:
Error saved to logfile [/opt/megaglest//.megaglest/error.log]
*ERROR* In [/home/softcoder/Code/megaglest/trunk/source/glest_game/main/main.cpp::handleRuntimeError Line: 628] [In [/home/softcoder/Code/megaglest/trunk/source/glest_game/main/main.cpp::handleSIGSEGV Line: 5563] Error detected: signal 11:
] gameInitialized = 1, program = 0x9c8f290
*ERROR* In [/home/softcoder/Code/megaglest/trunk/source/glest_game/main/main.cpp::handleRuntimeError Line: 647] [In [/home/softcoder/Code/megaglest/trunk/source/glest_game/main/main.cpp::handleSIGSEGV Line: 5563] Error detected: signal 11:

Stack Trace:
./megaglest:Glest::Game::ExceptionHandler::handleRuntimeError(char const*, bool)address [0x83abb5d] line: 637
./megaglest:Glest::Game::handleSIGSEGV(int)address [0x83ac361] line: 5568
[0xb77a3400]address [0xb77a3400] line: 0
[0xa73beb8]address [0xa73beb8] line: 0
./megaglest:Glest::Game::ConnectionSlot::updateSlot(Glest::Game::ConnectionSlotEvent*)address [0x85f1d2d] line: 578
./megaglest:Glest::Game::ConnectionSlotThread::execute()address [0x85f21d1] line: 309
./megaglest:Shared::Platform::Thread::beginExecution(void*)address [0x888b080] line: 256
/usr/lib/i386-linux-gnu/libSDL-1.2.so.0:()address [0xb7710355] line: 0
]
Message:
In [/home/softcoder/Code/megaglest/trunk/source/glest_game/main/main.cpp::handleSIGSEGV Line: 5563] Error detected: signal 11:

Stack Trace:
./megaglest:Glest::Game::ExceptionHandler::handleRuntimeError(char const*, bool)address [0x83abb5d] line: 637
./megaglest:Glest::Game::handleSIGSEGV(int)address [0x83ac361] line: 5568
[0xb77a3400]address [0xb77a3400] line: 0
[0xa73beb8]address [0xa73beb8] line: 0
./megaglest:Glest::Game::ConnectionSlot::updateSlot(Glest::Game::ConnectionSlotEvent*)address [0x85f1d2d] line: 578
./megaglest:Glest::Game::ConnectionSlotThread::execute()address [0x85f21d1] line: 309
./megaglest:Shared::Platform::Thread::beginExecution(void*)address [0x888b080] line: 256
/usr/lib/i386-linux-gnu/libSDL-1.2.so.0:()address [0xb7710355] line: 0

*TESTING*: FINISHED Waiting for lagging client playerIndex = 0 [nig]
In [/home/softcoder/Code/megaglest/trunk/source/glest_game/main/main.cpp::handleSIGSEGV Line: 5563] Error detected: signal 11:

=====================================

** Already in error handler aborting, msg [In [/home/softcoder/Code/megaglest/trunk/source/glest_game/main/main.cpp::handleSIGSEGV Line: 5563] Error detected: signal 11:
]
Aborted (core dumped)
ERROR: Server has quit unexpectedly.
And here's the GDB output:

Code: [Select]

[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/i386-linux-gnu/tls/i686/nosegneg/libthread_db.so.1".
Core was generated by `./megaglest --ini-path=./ --data-path=./ --headless-server-mode=vps,exit'.
Program terminated with signal 6, Aborted.
#0  0xb77a3424 in __kernel_vsyscall ()
#0  0xb77a3424 in __kernel_vsyscall ()
#1  0xb6daeaff in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#2  0xb6db2083 in __GI_abort () at abort.c:90
#3  0x083ab8b4 in Glest::Game::ExceptionHandler::handleRuntimeError (msg=0xb2c395bc "In [/home/softcoder/Code/megaglest/trunk/source/glest_game/main/main.cpp::handleSIGSEGV Line: 5563] Error detected: signal 11:\n", getStackTraceString=true) at /home/softcoder/Code/megaglest/trunk/source/glest_game/main/main.cpp:620
#4  0x083ac361 in Glest::Game::handleSIGSEGV (sig=11) at /home/softcoder/Code/megaglest/trunk/source/glest_game/main/main.cpp:5567
#5  <signal handler called>
#6  0x2f726564 in ?? ()
#7  0x085e20e7 in Glest::Game::ConnectionSlot::validateConnection (this=0x9cfc780) at /home/softcoder/Code/megaglest/trunk/source/glest_game/network/connection_slot.cpp:1535
#8  0x085e859e in Glest::Game::ConnectionSlot::update (this=0x9cfc780, checkForNewClients=false, lockedSlotIndex=-1) at /home/softcoder/Code/megaglest/trunk/source/glest_game/network/connection_slot.cpp:1502
#9  0x085f1d2d in Glest::Game::ConnectionSlot::updateSlot (this=0x9cfc780, event=0xb2cfc1e0) at /home/softcoder/Code/megaglest/trunk/source/glest_game/network/connection_slot.cpp:576
#10 0x085f21d1 in slotUpdateTask (event=<optimized out>, this=0x9d0be08) at /home/softcoder/Code/megaglest/trunk/source/glest_game/network/connection_slot.cpp:175
#11 Glest::Game::ConnectionSlotThread::execute (this=0x9d0be08) at /home/softcoder/Code/megaglest/trunk/source/glest_game/network/connection_slot.cpp:309
#12 0x0888b080 in Shared::Platform::Thread::beginExecution (data=0x9d0be08) at /home/softcoder/Code/megaglest/trunk/source/shared_lib/sources/platform/sdl/thread.cpp:256
#13 0xb7710355 in ?? () from /usr/lib/i386-linux-gnu/libSDL-1.2.so.0
#14 0xb7754d6b in ?? () from /usr/lib/i386-linux-gnu/libSDL-1.2.so.0
#15 0xb76ebd78 in start_thread (arg=0xb2cfcb40) at pthread_create.c:311
#16 0xb6e7201e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:131
A DMP file was created, too: 7a6007b3-9caa-5217-51c92832-645b489d.dmp