Libertarianism (or anarchism, I use the words interchangeably) is an attempt to answer one question and one question only: When is the use of force justified? In other words, what should the law be? Libertarianism is a legal philosophy, only concerned with what the law should be and proportional punishment. Libertarianism is NOT a moral philosophy. Libertarianism doesn't say that all non-aggressive acts are moral or that any consensual act is moral; only that it is not justified to use force to combat non-violent behaviors one may find immoral and disagreeable. Many libertarians are able to distinguish legality from morality. While all legal laws should be moral, i.e. there should be no law which conflicts with morality, not everything which is immoral should be considered against the law.
Adam Smith in his work, A Theory of Moral Sentiments, talked about three type of justices. One is what he called commutative justice, which is don't mess with other people's stuff. Smith said that it is justified to use force against people who violate commutative justice. There are other virtues, or justices as well. According to Adam Smith, there is also distributive justice (not the welfare state), which is making a becoming use of one's own and giving everyone his due and treating them the way they deserve to be treated. There is also estimative justice, which is giving praise and esteem to someone (or something) who is deserving of it. According to Adam Smith, it is only justified to use force against those who violate commutative justice, whereas the other virtues, such as benevolence, cannot be extorted by force.
While it is only justified to use force against those who initiate the use of force against other people and/or their property, it is foolish to claim that the other virtues do not exist. Since libertarianism is a legal philosophy, it is only concerned with commutative justice when it comes to the legal realm, but to argue that the only justices do not exist just because they cannot be extorted by force is not consistent with reason or ethics.
There are many libertarians who claim to acknowledge the difference between morality and legality and that compassion cannot be achieved through force. There are also some libertarians who fail to distinguish between morality and legality, and therefore since only violations against people and their property should be illegal, any act which therefore is permissible must be moral as well. Such libertarians are known as vulgar libertarians.
One of the arguments many libertarians make is that the welfare state besides being immoral, is also not needed. Civil society, social norms, human empathy, ostracism are all nonviolent ways of inducing people to act in morally approvable ways without having to use the iron fist of the state. Since most people understand that there are moral obligations beyond what the law requires, there are ways to satisfy such virtues without having to use force to achieve it. Doing so is not only compatible with proportional punishment theory, but is also more humane, compassionate and effective than government enforced "charity." Libertarians who act is if any act that does not violate the NAP ought to be shielded from criticism undermine such arguments in support of civil society replacing the welfare state.
There are many acts which though legal, are worthy of condemnation. If a person is walking down the street and sees a person drowning and is able to save him without injuring himself (or putting himself in danger), one ought to do so. A person who fails to save a person from drowning, knowing that he is both able to do so and that he won't hurt himself by doing so, ought to be criticized for his lack of compassion. It is immoral to refuse to save someone else's life when there is little cost to you doing so. That being said, it should be perfectly legal to refuse to save such a person. There is only a legal obligation to save drowning people you push into the river. Those who are drowning not because of your actions there is no legal obligation to save, but there is possibly a moral one. One who refuses to help those in need ought to be criticized for it. They ought not to be fined, executed or imprisoned for it. Vulgar libertarians fail to understand the difference.
Is it moral to commit adultery? I would say no. One aught to not cheat and hurt those one claims to love, especially deliberately. But a relationship is not a slave contract. One owns one's body and any consensual sex should be legal. That does not mean that there is nothing wrong with committing adultery however. I would argue that doing so, while being perfectly legal, is immoral and there is nothing wrong in criticizing those who engage in acts one finds immoral.
I would argue that insulting people and being a bully to those who are undeserving of it is immoral. It may be immoral to commit suicide if one has young children one is responsible for or if doing so one knows will cause heartbreak to those one loves and cares about. This does not mean one should be forced to live against their will. Morality and legality may overlap in many cases but they are not synonyms.
In short, vulgar libertarians confuse morality with legality. They think any act consistent with the NAP is also moral. Any act that doesn't cause physical injury to another person or their property ought not only to be legal, but ought not to be condemned and criticized. As long as no force was used, such actions are moral. Insulting people, calling people names, failing to keep promises one makes, lying, committing adultery, failing to live up to one's social and familial expectations, being a peeping tom, refusing to help any person in need, betraying one's friends are all perfectly moral according to such vulgar libertarians since none of those acts violate the NAP. If that sounds like moral nihilism and a batshit crazy philosophy that makes libertarians sound like the sociopaths they are often accused of being, it's because it is.
