I don't think we have to put infidelity into any sort of box or label it as agression or fraud. In a free society, people can freely enter into marriage/cohabitation contracts and can establish basic rules or even write actual contracts. In an arrangement where my mate and I "go steady", we have agreed to be sexually exclusive. If I cheat, that's breach of contract.
I can see how many people would feel it natural to have such contracts and enforce them, while for others that may be a thing that is not as big of a deal as for others and they would prefer to throw a few insults and walk away.
When it comes to misrepresentation of facts to extract monetary gain, that's fraud.
Private arbitration can easily handle both issues.
I addressed how infidelity sometimes results in monetary gain for one party through a misrepresentation of facts above.
I do agree with you, though, that not every couple will want to be bound by the same rules (or perhaps any at all).
Sorry, I separated the two issues for clarity, but again, breach of contract is breach of contract. I do consider someone misrepresenting facts to get married and extract monetary gain to be fraud. But, I do believe that the practical will be more important than the philosophical in an actual free society.
Free individuals will be able to engage in all sorts of contracts, some of which other free individuals may find abhorrent. Also, a free society will be a very diffuse society, so I can easily imagine some groups tolerating certain kinds of fraud.
Agree 100%