Now this is a really old one with and one with a pretty clear lineage.
If your not familiar with the expression it means that something is very dead or there is no disputing it's dead-ness.
Often I would here this is my house, when a fly was swatted particularly well and the guts had made a very gruesome mess.
I guess we can start with what a door nail is?
They are the decorative and structural nails used to strengthen doors in very old construction and carpentry practices. You can see them in the photos above and below. They actually add considerable strength to the construction and hold together overlapping boards. They functioned like rivets do in modern days. They were hammered flat on both sides, or the inner side was bend down and then flattened into the wood.
So why is it dead?
The nail is said to be dead since it has been hammed so extensively it was totally unusable in future building practices, which was common back then since nothing (including nails) were mass produced and everything was reused.
Thus a door nail, was very dead and extensively beaten, much like the bug that was hit with much more force than was truly required.