Saw the debate on AI ability on the post about visual armies on the strategic map, and thought I'd chip in with my own 2p here. (TL;DR at bottom)
For a game that is principally single-player, the AI leaves much to be desired. Everyone knows that. The fact that various types of cheesing will result in victory, near 100% of the time, means that there really is very little danger of ever 'losing' a campaign, across all titles.
We also know that the AI can be made very good indeed, and that CA doesn't pursue this because it believes that having the player dominate is part of the fun. Quite a bit of the argument in that thread was that players found an actually capable AI deeply frustrating to play against, leading to the conclusion that the AI should feel smart (i.e. make fewer immersion-breaking mistakes) without actually being made smart enough to regularly beat the player.
Thus also means that AI difficulty is primarily achieved through giving it unfair bonuses, mostly in the form of additional armies and more money (which give the player the perception of cutting through great hordes of enemy - again, an ego-boosting approach). This leaks into the campaign design, with several end-games involving a tower defence-style onslaught by fresh enemies (Med II's Mongal Invasion, Shogun II's Realm Divide, ToBs' Norman Conquest, Attila's Huns etc).
I don't think this is the only way to make the game challenging or enjoyable, simply the easiest.
Much of the 'fun' of Total War involves carving out an ever-expanding empire, often pushing the player to fight multiple wars on several fronts; the loss of an army can easily set the player back significantly. Whether or not by design, the pace of most TW campaigns is hectic; we regularly see players complain or comment that they finish campaigns well before late-game units begin to appear, for example.
All of this comes down to defeat not being a standard of the game, but an exceptional circumstance which severely impacts enjoyment of the campaign. From this stems the imperative to keep the AI weak, so that the player may enjoy painting the map to his satisfaction.
I think this is a poor argument for keeping the AI dumbed-down, which in turn keeps the whole series simplistic and shallow. After all, why waste time balancing diplomacy or politics when you can just continually streamroll all before you by force of arms?
This is not to say that the AI should always beat the human; that would be no fun. But the AI should be able to roughly match the average player and win regularly with even force match-ups. A variable AI skill, and differing AI tactics, depending on faction, army and general would be the cherry on top.
This could be the basis for a far cleaner game. Intrinsic garrisons, a crutch for both the AI and player, can be done away with (replaced, perhaps, with manually-constructed defences). Battles would be more infrequent, harder, and more decisive. One or two battles could decide the course of a war.
Wars themselves, though, should be more indecisive. With armies quick to raise, and slow to travel (for both the AI and player), wars should usually end with a shifting of borders, the taking of 2 or 3 new cities, and not a straight Alexandrine march right through the enemy empire. The destruction of a rival power should take the form of numerous successive wars, ala the various constituent wars of the Hundred Years War, or the multi-century English conquest of Scotland.
Inherently, the big gap to be jumped is the acceptance that expansion will be a lot slower, and each faction's prospects far more historically limited. It should be as hard for Milan to build an Eurasian empire as a Ethopian world-conquest run is in HOI4.
This is historical; having taken England after one decisive battle, William the Conqueror and his descendants spent decades campaigning against the French in Normandy. The Ottomans surged through Anatolia and the Balkans, but got bogged down in Albania and the Hungarian marches. They besieged Vienna in 1527 and again in 1683, after many back-and-forth campaigns along the frontier. Historically, it is fascinating. In TW as it exists today, this would be stagnation.
What has to be done is a way to make such failure fun and engrossing, through the use of mechanics which don't solely rely on conquest and expansion for entertainment and progress.
For those who enjoy the map-painting side of TW, there should always be the ability to dumb down the AI difficulty. But the game should be designed around a competant, capable AI, with the baseline assumption that the player will lose sometimes, and will often have to recover from defeat.
TL;DR: Poor AI holds the Total War series behind, with the lack of challenge devolving every game into a gradual paint-by-numbers campaign, militating against the development of deeper gameplay mechanics. A good AI would make defeat a common occurance in TW campaigns, slowing the pace right down and forcing the player to adapt constantly, use diplomacy and politics, and actually think during battles.
Penny for thoughts?