Most developers have some sort of a "gut feeling" why software development costs so much and even with best effort and intention, tends to consume more time than planned. This is nothing new, it's been pretty much a constant observation for 50-ish years and it won't get better any time soon. Here's the main reason why:
It's getting more, not less, complex
New software is built on top of existing software much the same as buildings are built on foundations, except here, we're not building on just one set of foundations but on a sprawling, multi-storied foundation of which most part are built using completely different, older technology, have strange, undocumented interfaces and behaviours, and for almost all of them, an average developer has practically no chance of reaching out to the original developer who built it, even if he, by some miracle of education or experience, actually notices that those layers exist.
You're reading this by using a web browser. The Mozilla Firefox browser, which is currently (start of 2018) not the most popular browser in the world, consists of 35.8 million lines of code. This is to be taken literally: people just as yourself, developers, have, over the course of 20-ish years written 35.8 million lines of program code, just so you could read this. Google Chrome apparently has much fewer lines of code, 18.1 million lines, but that seems too low, maybe they're not counting some of them, and it's a younger project, only 10 years old. And in both cases, the projects were built from existing projects and components.
Then, the browser is running on top of an operating system. Windows XP, which was released in 2011 and is now horribly obsolete, contained about 45 million lines of code. It's estimated that Windows 10 contains between 50 million and 60 million lines of code. Linux is about the same, with 23 millions of lines just in the kernel, and easily as many in the various graphical interfaces and common utilities.
And that's not even taking into consideration the code which runs in your keyboard, mouse, your screen, your network card, and inside your CPU.
Now here are two things you should bear in mind:
- Each of those lines of code could contain one or more bugs
- When a developer is writing new code, he's implicitly (and sometimes explicitly) hoping that the new code will seamlessly and correctly integrate with all that code which already exists, and that the existing code is bug-free.
When developing something which seems mundane and simple, like a web application, developers are relying that neither their new code has bugs nor all the existing code they are building their application on top of has bugs, that the documentation is perfectly correct and perfectly understandable, and that there is practically perfect communication between developers which are working on the project together as well as between them as a group and the clients which will be using the end-result.
The above image shows the possible interactions ("lines of communication") in groups of people with a varying number of members. Building software means that developers interact with bits and pieces of software written by other developers which they will never see or talk to, but still interact with them through those lines of code. Authors of commonly used libraries, modules, utilities, operating systems and everything else, might even be dead now, if they've started in 1970-ies, but still the whole software ecosystem interacts with the code they've created.
And navigating these things, especially the bits with communication, require experience and some awesome skill to not make the new project a complete failure.
Taking into account all of these points, now consider this one: still, after all that, with competent and motivated developers, managers and clients, with extraordinary effort and night shifts, more than half of all IT projects fail.
Software is eating the world, but complexity is eating software. That's why.
(for hardware, just multiply everything by x10).