When I played 3.5.2 today i had an idea about how to tackle one of the major issues the game has when you have many units:
When you have many units, you often position them in a group, so that they fill adjacent fields, forming a rectangle. Then you select this group and order them to walk / attack somewhere at the same time. This means all these stacked up units try to walk to the same destination at the same time, and every single of them tries to find a good path by itself.
This is tough on the pathfinder, and while units which are in the row close to the destination will easily find a route there, the rows in the back actually have to calculate complex paths, and usually turn to the opposite direction at first. This makes surprise / hit and run attacks more difficult, since your units will take a while until they arrive, since they keep blocking each other at the beginning.
Thre's two ways to solve this I can think of:
1. Walk in formation
2. Walk 'row by row'
Ad 1:
One way to solve this would be moving in formation. This would mean that either the most central or only the outermost units should do pathfinding. Obviously this only works when all units have the same movement speed and do not run into obstacles along the way. So while this would be nice on the first thought, on second thought it may have a lot of problematic side effects.
Ad 2:
The second option is to make those units which are not as close as others to the destination wait one frame before they start to search for a path and before they start to walk.
I.e. the 'front row' starts path finding and walking first, then the second row, then the third etc. 'Rows' would be calculated as follows:
(a) Determine the unit (or any one of the units, if multiple) which has the shortest path to the destination. This unit is in row zero.
(b) For every other unit, calculate the distance between this unit and the unit determined in (a). The result is their row number.
Now when this group is moved to a given destination, each row starts path finding and walking with a delay which equals its row number. I.e. row zero starts immediately, row 1 starts one frame later, row two starts another frame later etc.
I think this would give us much more smooth movements, and put less load on the path finder
Softcoder already said on IRC that the problem and solution are far more complex than this. So I hope my and Coldfusionstorm's suggestions will lead to a fruitful discussion.