Yeah, it's not easy once somebody decides you're a good target and start wiping you regularly. Big Mamas, Gunmen, and Knifers are the best defense against people sending small hordes. You could also invest a crap-ton of resources into Hobos (they don't defend) and send a screaming horde of drunkards at them every so often just to eat some of their troops too.