What made magic so hard? Maybe is because their malee ugets get team killed because of their heavy use of splash attacks? Just a thought...
Difficulty is a relative term. I would say it's not too difficult to play magic, just radically different from other factions. While most factions emphasize on production from buildings, magic focuses on producing one type of unit, the initiate, then morphing it up. The mobile summoner will produce the rest of the units, and is a bit more like a building, but at the same time, radically different. As mentioned earlier, there's magic lacks a proper defencive building. The golem is the main defencive unit, but has considerably lower health than most defencive units and its short range isn't made up from the fact it can move, as movement is limited and every fault the pathfinder makes critically reduces the golem's usefulness. As well, the impact type attacks fair well against buildings and poor against offensive units, which is quite the opposite of the golem's uses. The tower of souls is air-only, but since magic has so many ranged units and its air unit, the dragon, can attack air-to-air, the tower of souls is pretty much useless and I don't bother with them myself.
Of course, magic is also a touch more limited in the energy resource. Energy requires the construction of immobile energy sources, which can quickly flood a camp and if not built in a good location, can be detrimental to pathfinding. Dark Magic solved this problem by having energy sources produce energy (then a size 0 unit), eliminating the need to build more than one energy source. Magic's behemoth is rather tricky to build, as you must construct a very-late game wicker behemoth with an initiate at a point where you are likely not paying much attention to initiates, then morph that into the behemoth. And the behemoth, being a size 2 melee unit, does poorly in large battles, where pathfinding prevents it from even reaching the foe. Tech's battle machine can get around this with its ranged attack, though the AI has difficulties with both of these units.
The dragon is magic's redeeming quality, and does help empower magic a bit more. The AI doesn't use summoners very well in two ways. Firstly, while they can be used as an offensive unit, there's little reason to do so, as in most occasions, a summoner is best suited to remain at a safe distance from enemies, continuing to produce more units. Secondly, when a summoner is attacked, if they die before completing the queue, the resources are lost. Of course, this applies to all buildings, though summoners are far more likely to die if they become attacked than buildings, with their hefty HP. This could be improved by canceling the queue when health falls below a certain amount.
The fact that the two main units, which make up the bulk of your offensive force, the battlemage and archmage, have to be morphed from other units (the initiate and battlemage respectively) makes it far more difficult to handle magic. The total time to produce the units is slightly slower than other factions, even more so for human players since they must manually reselect the unit and tell them to morph again, for the archmage. And of course, the archmage's splash is particularly deadly to its own units. While it may not be the most practical, I think balance could be improved by having the splash only harm enemy units. Daemons, for example, are nearly useless once you have an archmage in the fray, because they'll meet the foe at melee range around the same time the archmage starts casting from afar, which will generally kill the daemons quickly, causing lost resources and allowing the enemies to advance upon the vulnerable mages faster.
Magic's ghost armour is pretty much a mirror image of the guard, and can be used in the same way. The drake rider is similar to the horseman, but can attack air and has a medium range attack. However, the drake rider is rather difficult to obtain, as it's morphed from the summoner, preventing that summoner from being used to, well, summon stuff. That also makes it take some time to obtain.
Of course, with all that being said, Magic does boast a strong early production, since their main building also functions as the producer of their principal unit, the battlemage, and the archmage does deal devastating splash damage - provided your own units stay out of the way.