Not at all! I've seen swingers at this resort where I worked where the husband just like to watch other men or women fuck his wife and visa versa and I've seen full swap couples with or without watching/participation/mixed participation.
Maybe I use the term loosely, perhaps there are all different kinds of swinging?? My definition would be openly and honestly having sex with others while in a committed relationship.