Why are terrorists like the Taliban and al Qaeda or even insurgents on their home soil harder to defeat? 

There are many considerations.   Politically, policy-makers want approval over any military operation that has consequences at home and intelligence information found by the military has to be calibrated.

In the first study of its kind to combine military intelligence, attrition and civilian population behavior in a unified model of counterinsurgency dynamics, Moshe Kress and Roberto Szechtman of the Naval Postgraduate School stress the role of obtaining intelligence about the insurgency.

Absent intelligence, they write, not only can the insurgents escape unharmed and continue their violent attacks; but resultant poor government targeting causes innocent civilian deaths, which increases popular support for the insurgents and thus generates more recruits to the insurgency. 

Attacks on Taliban strongholds by U.S. drones show that deaths among civilians may end up hindering American lead efforts, according to Kress, and further says ill-targeted actions taken by Israel and Colombia also show that unintended deaths among civilians have led to increased support for insurgents.

In their paper, the authors model the dynamic relations among intelligence, collateral casualties in the population, attrition, recruitment to the insurgency, and reinforcement to the government force. 

Even under best-case assumptions regarding the government actions, they say that the government cannot totally eradicate an insurgency by force. The best it can do is containing it at a certain fixed level. The containment or stalemate points may be either fragile or stable. If the violence level is low, the containment point is fragile, in which case the insurgents can "break away" and eventually win. If the government commits large forces and applies a heavy hand (for example, the "surge" of United States forces in Iraq) then the stalemate point is stable. 

The model and analysis, they write, represent a best case situation from the government perspective under the parameters put forward where:

(a) government force is steadily reinforced by new units;
(b) it has unlimited endurance (it surrenders to the insurgents only when it is totally annihilated);
(c) the only recruitment to the insurgency is due to collateral casualties in the general population that generate resentment to the government, and therefore more recruits to the insurgency.

"If a government does keep its intelligence gathering capabilities high," says Szechtman, "it can keep a hold on the insurgency, and after a while, when the insurgents realize they can't win, a political compromise may be reached."

Article: "Why Defeating Insurgencies is Hard: The Effect of Intelligence in Counterinsurgency Operations – A Best-Case Scenario", Moshe Kress and Roberto Szechtman, Operations Research