I note that if you have a right to a house, you have the authority to enslave construction workers, as well as all those that supply the necessary materials houses are built with.

Since you, nor anyone else has such a right to enslave others, it is impossible for there to be a right to housing.