The husband was not invited to the sexy party with his wife