The usual answer is that the temperature of dangling testes needs be lower than the body temperature to optimize sperm production. But why? During the gestation the testes descend from the chest across the whole body cavity to get outside of the body and this passage weakens the wall structure resulting in a higher propensity to hernias (Neil Shubin likes this example as showing our fish origins). On the other hand, it is unclear why wouldn't evolution be able to develop more heat-tolerant sperm. Indeed, the sperm sensitivity is extreme, clearly affecting our reproductive chances:...if the testes heat up by as little as 2°C, sperm formation is adversely affected. Sperm count will drop, the number of normal sperm will fall and the number of abnormal sperm will increase. Semen quality is naturally lower in summer compared with winter. Although semen volume does not change significantly, the total sperm counts per ejaculation fell from 320e6 in winter to 250e6 during July and August.
Given the physiological/reproductive penalties, there should be a good reason why it is done this way. Observe that desdendant tests and scrotum are not the universal features even for the mammals (though the majority of mammals do have their testes dangling outside). E.g., elephants wisely keep theirs sizeable testes hidden inside the body. It has been argued that the scrotum is an older primitive adaptation, and more recent testicular evolution among mammals “has proceeded from scrotal to ascrotal but never the reverse,” while the “loss of descensus is relatively rare”.
The biologists have been scratching their heads for many decades searching for a plausible rationale accountig for our pendulant ornamentations:...The traditional interpretation is that the scrotum evolved to provide a cool environment that would ensure for optimal spermatogenesis. It was also argued that the epididymis functions as a cold-storage site for sperm, and therefore it was the epididymis that was the prime mover for the evolution of the scrotum. We now know that both the storage and formation of viable sperm for most mammals requires that the testicles be maintained below body temperature, but why this is the case has never been adequately explained. Some theories of testicular descent include the idea that reduced scrotal temperatures function to minimize gamete mutation rates. Not only are scrotal temperatures typically below body temperature, but because of reduced blood flow to the testicles another hypothesis is that these conditions combine to create a hostile environment that functions to train or test sperm as a means of selecting for a high quality ejaculate. It has even been suggested that the scrotum evolved as a signaling device or form of sexual ornamentation to promote male social/sexual competition. As an extension of this hypothesis, it is interesting to note that scrotal testicles could be related to what is known as the “handicap hypothesis” (like peacock's tail). While this might seem to be a plausible account of scrotal testicles because the handicap they entail carries enormous potential costs, most traits that fit the handicap hypothesis become more exaggerated and more costly over time as a consequence of female choice and competition among males for limited reproductive opportunities. With the possible exception of colored scrota among a few species of primates, there is little evidence that this has been the case.
When the best idea is that we have testes outside of the body precisely to make our lives miserable, you know you are losing it. One very recent idea I've heard (which I like much more) is that the testes are located outside because we rely on a very archaic way of sperm activation, which no one else is using:...The integrity of the gonads is of paramount importance when it comes to reproduction. Because of vulnerability to damage, insult, and temperature variation, unprotected ovaries located outside the body cavity would be an enormous handicap/serious reproductive disadvantage. The same reasoning applies with equal force to the testicles. But unlike ovaries, descended testicles located outside the body cavity in the scrotum are common among many mammals in spite of all the obvious risks and disadvantages. We suggest that descended scrotal testicles in humans and many other mammals evolved to provide a situation specific means of activating sperm. As a result of consistent temperature differences between the male and female reproductive tracts (i.e., scrotal temperatures are typically maintained at 2-3oC below body temperature), we propose that the rise to body temperature that accompanies insemination into the vagina serves as one of several triggers for the activation of sperm. The rise to body temperature occasioned by depositing sperm into the vagina functions as a primitive but highly appropriate, situation specific trigger that augments sperm activation and functions to further increase the likelihood that a sufficient number of sperm will be able to pass through the cervix and reach the oviducts to fertilize an egg. Descended scrotal testicles evolved to both capitalize on this copulation/insemination contingent temperature enhancement and function to prevent premature activation of sperm by keeping testicular temperatures below the critical value set by body temperature. http://www.epjournal.net/filestore/ep07517526.pdf
It is an interesting hypothesis, as it explains testicond animals, like elephants: during the ovulation, their females rise basal body temperature, so the differential is still present; it is only the differential that matters. Furthermore, it explains odd habits of certain bipeds:...Humans show a peculiar cross-cultural nocturnal copulation bias. Having sex at night could promote any number of adaptive outcomes. Nocturnal copulation may have functioned to minimize vulnerability to predation and/or detection and interference from rival males, accommodate clandestine copulations, promote incest avoidance, and even enhance the likelihood of sperm retention by postponing the resumption of an upright posture by the female during the post ejaculatory period. Patterns of nocturnal copulation may also be a circadian adaptation to descended scrotal testicles. When daytime temperatures approach or exceed body temperature which is common in equatorial regions, testicular adjustments would fail to function favorably. However, in the evening and at night when ambient temperatures fall back below body temperature, more effective scrotal temperature adjustments are reinstated and the likelihood of optimal sperm viability may increase.
Isn't it exciting how one thing leads to another and then still another? Our mammalian lineage are the only animals keeping their testes outside of the body because our synapsid ancestor had a primitive and peculiar way of activating sperm during the ovulation. It was simpler to redesign the body than finding a proper biochemical solution to this seemingly trivial problem. Millions of years of nocturnal romance later, some mammals found a way of fooling this mechanism, which is also rather unimaginative - it is amazing that it took that long. The testicles promptly got back inside the body. Now the dangling testicles are on the way out. But for now we have them where we have them, and it is good.
Why are the mammalian testicles located outside of the body?
PS: another take on the evolutionary history of scrotum http://www.ias.ac.in/jbiosci/mar2010/27.pdf
...We conclude that: (1) testicondy represents the plesiomorphic character state for Mammalia, because of the lack of testicular descent in all Monotremata and almost all Afrotheria; (2) the scrotum evolved twice during the evolutionary history of mammals within Marsupialia and Boreoeutheria; (3) the distribution of mammals with a fast mode of locomotion (gallop) is aligned with the distribution of the scrotal life forms on the phylogenetic tree of mammals; at the same time, repeated loss of the scrotum in many taxa within Laurasiatheria occurs in groups that do not gallop. ("The evolutionary history of testicular externalization and the origin of the scrotum")