I think it has something to do with renderObjects. Its also used when you render shadows.
comment out renderObjects in game.cpp line 2672 ( renderer.renderObjects(avgRenderFps)
and line 2635 ( renderer.renderShadowsToTexture(avgRenderFps))
beside the effect that you will not see anymore trees and shadows you will see that the fps stay stable.
UPDATE:renderer.cpp line ~3637 VisibleQuadContainerCache &qCache = getQuadCache();
change this to VisibleQuadContainerCache &qCache = getQuadCache(true,true);
Then you instantly get the slow frame rates you normally get after camera movement. So I think this QuadCache does not work the way it was meant .. )