Which is exactly how I view it.
You say that it's possible because you have faith that people can engage in any behavior so long as all parties involved are happy about it. Unfortunately, in the long term, everyone reverts to the mean, and societies that depart from 'ideal' moralities implode. This is why empires invariably fail; they figure that they have conquered the known world and are so in control that they can do anything. So they do. And they depart from 'ideal' morality so far that the society just fails, bit by bit.
This is where the analogy of physics bogs down because we can test physical principles in the blink of an eye, while testing moral principles takes generations.
Note that theft isn't automatically excluded from 'ideal' morality. It may be that theft is a proper and healthy aspect of that morality. I don't believe that it is, anymore than I believe that gravity suddenly stops at a certain distance, but it's possible. The important bit is that there is an 'ideal' and that we should be applying ourselves to figuring it out.
You'll notice that I brought up prisons as an example of slavery.