"My right to swing my fist ends where your nose begins." There is an end to your liberty, and that's when your actions affect the safety and liberty of others.

To give you an analogy — most people are thoughtful and mindful of everyone's safety and follow traffic rules, but for those that are not, they will be ticketed, jailed, and fined.