You need to understand, we men LIKE sex, unlike women, and tbf we are easily seduced. A married woman doesn't have sex with her husband, but we remain constantly on the lookout for young attractive women, it's in our biology, it's beyond our rational mind. If a man goes out to have a distraction, it is not something we would do to spite our partners, but just to alleviate an inner drive that is present all the time, every day as we go out on the street. Eventuay we cede to this constant pressure, while women remain blissfuy unaware that there exists any sex drive whatsoever.
Think of how much it bothered you that your husband would try to have sex with you every now and then, and how an escapade could free you from the need to satisfy his drive.
Stop giving so much weight to something as trivial as sex.