well the first step for custom AI is disableAi(factionIndex).
The second step is converting Units to LuaOO - objects
The third step is implementing the reactions (what it should do) of a unit.
And the last step is to implement event-handler and action-triggering.
ok, that was obviously not your question....
I'd say it this way: for really changing the AI without performance-problems you need to build the changes in C++, Lua gets very slow with multiple loops.
MG got a couple of better support-methods for this during the last month, but it is still not, perfect. So far it seems to me, GAE can even handle less AI control.