People like to say libertarians are naïve, but they just take the concept of consent further than most. Being free to say 'no' when someone else tried to impose themselves somehow, and to respect the 'no' from others, is the simplest way to describe this.