Is there anything really wrong with a casual relationship?

Staying with your long-term partner isn't quite the same as faithfulness to them. By staying with them and not looking for anything else, you prove your devotion. It's a little bit of "I love you so much that I don't need or want anyone else" or "I love you so much that I will give up everything else", whichever seems to make the most sense.