Rights aren't about deserving. You're starting from a flawed premise. And people are entitled to liberty within the sphere defined by rights. Who is entitled to trespass? That is the question people dodge, because the obvious answer of "nobody" tends to get in the way of their control freak inclinations.