Sperm production requires a slightly cooler temperature than body temperature. This is why the testicles are not inside the body like the ovaries are, but instead are positioned in the scrotum, which allows them to “hang out.”
In fact, the scrotum is lined with a very specialized layer of muscle that contracts to bring the testicles closer to the body when it's cold and then relaxes to allow the testicles to drop away from the body when it's warm. If the testicles sustain temperatures that are too hot for too long, sperm production is impaired. In an air and cloth environment, although it may be hot, sweat and evaporation along with the movement of the scrotum allows the testicles to cool off to some degree.
In a liquid environment such as a hot tub, there is no way for the scrotum to cool the testicle and it heats up and in some men there is very measurable sperm impairment. Just one session in a hot tub can impact the sperm count and movement for up to three months in some men. These effects are reversible with time. Some men are not very effected by a hot tub exposure, but if a man already has low sperm counts or motility, it’s certainly wise to avoid the hot tub. Boxer shorts may allow the testicles to breathe more than briefs but it has never been proven that wearing boxers instead of briefs makes a difference as far as birth rates.
A high fever can effect the sperm counts significantly. In some cases, the sperm count can drop to zero. This is usually a temporary effect. Because sperm production takes a couple of months, it can take over six months to fully rebound and return to normal counts.