The obvious answer is that HFCS is cheaper than refined sucrose, but this does not quite answer the question. Making of HFCS is a costly operation that involves chromatographic purification
It is not cheap. The reason it is cheaper is that the US government intentionally keeps domestic price 3X higher than the world market price. American food companies operating in competitive markets cannot pay this obscenely high price so they move their production lines outside of the country (which works only for products with relatively long shelf life, e.g., candy) or they switch to HFCS.
This is nothing new. US sugar industry has been enjoying trade protection since 1789 (that was one of the first acts of the Congress). The initial goal was to gain the loyalty of the sugarcane farmers in the Louisiana Territory. Here is how the modern iteration of this far-sighted policy works:
...The U.S. Department of Agriculture (USDA) provides loans to sugarcane and sugar beet producers and processors that guarantee a minimum price regardless of the true market conditions. At the end of the loan term (generally 9 months), sugar producers and processors either (i) turn over to the government the sugar they produced as payment for the loan, or (ii) sell sugar on the market if the going price is higher than the USDA loan amount. USDA allocates a share of the anticipated U.S. sugar market to sugar producers annually. This allotment determines the amount of sugar an individual company is allowed to sell that year. If companies produce more sugar than their allotments permit, they are forbidden to sell it. Instead, they must store the excess sugar at their own expense until they have permission to sell in the future. The 2008 Farm Bill requires that these allotments be at least 85% of domestic sugar demand. Marketing loans and marketing allotment programs are contingent on the use of feedstock produced in the U.S.
...Tariff-Rate Quotas: U.S. sugar imports are strictly controlled by TRQs. The volume of these quotas is established annually by USDA, and the U.S. Trade Representative allocates the TRQs among countries. TRQs are the amount of sugar that can enter the country from abroad at a low or zero duty. The amount set aside for import under TRQs must meet US obligations to the World Trade Organization (WTO) – currently a minimum of 1,117,195 tons of raw sugar and 22,000 tons of refined sugar. The 2008 Farm Bill also allows USDA to increase sugar TRQs on April 1 of each year if a shortage is expected. Today’s TRQ allocation among 40 countries is based on US sugar trade as it existed from 1975 to 1981. Sugar production and markets have changed substantially during the past 30 years, yet the U.S. sugar quota system does not reflect that evolution.
Every so many years there are appeals to end this racket. Here is NYT (2003):
... Sugar growers in this country, long protected from global competition, have had a great run at the expense of just about everyone else - refineries, candy manufacturers, other food companies, individual consumers and farmers in the developing world. The nation's sugar program has become far too costly to America's global economic and strategic interests. Preserving the spread between domestic and world sugar prices costs consumers an estimated $2 billion a year, and nets the growers tens of millions annually. The sugar exporters who are able to sell to the United States also benefit from those astronomical prices. http://www.nytimes.com/2003/11/29/opinion/29SAT1.html
Nothing changed, of course. There is no other commodity in America that is more regulated by the government than sugar. Every time someone suggests "to do something about this situation," the degree of intervention further increased.
...National sugar policy has been affected by statute since 1789 when the First Congress of the United States imposed a tariff upon foreign sugar. The purpose of this and subsequent tariffs was to provide revenue for the government. From 1789 to 1930, a total of 30 Acts dealing with sugar were passed, not including several modifications. In 1842, however, the purpose of the tariff was expanded to include a policy of protection by subjecting refined sugar to a higher tariff than raw sugar, thereby attempting the promotion of a domestic refining industry and the expansion of domestic raw sugar production. It was not until 1934 that a federal sugar program was enacted. Higher government intervention was motivated by the belief that sugar production and marketing was regulated in both producing and importing countries more than any other commodity. The absence of true competition between domestically produced sugar and sugar produced with cheap labor or under subsidy in other countries led to further controls started with the first Sugar Act. http://edis.ifas.ufl.edu/sc019
...Politically, the oft-stated case for this type of protectionism is that it preserves jobs. In fact, despite “saving” a few low-wage jobs in the uncompetitive sugar farm industry, these tariffs bring the US a net economic cost. Since there would be an estimated 3,600 jobs at risk due to international competition if the industry was liberalized, it costs around $600,000 to preserve a single at-risk job through the tariffs. Considering that the mean wage of the industry is $37,000, the implication is that the opportunity cost to our economy of saving one job in the sugar industry is $563,000. This is about 3.5 times the average CEO salary... Not only are the tariffs not helping those they are intended to support but they are also simultaneously destroying jobs in downstream industries, which are forced to bear the tariff’s costs. These unfortunate industries include America’s far more efficient sugar refining and confectionery industries. While these industries pass a lot of the cost onto consumers, the tariff-inflated price of their input lowers employment and output. As a result, US sugar protectionism results in both significant economic costs and also punishes more efficient industries... In addition to making absolutely no economic sense, American sugar protectionism has inadvertently exacerbated many prevalent current problems. Compounded by the government’s subsidies for corn, a substitution effect occurs in which firms in the American food industry switch from using sugar to HFCS. With so many detriments and only minuscule benefits, it’s a wonder that sugar protectionism has survived for so long. http://dartmouthbusinessjournal.com/2011/05/u-s-sugar-protectionism/
The domestic growers justify their patronage
scaring the public with the visages of mass unemployment, acute shortages, unregulated markets, and other horrors that are usually invoked in such cases. Of course, the real reason is that this system is good for the government (it brings revenue), even better for the growers (that reap huge profits), and terrific for politicians (whose election campaigns are funded through these profits and whose main calling in life is to guard the hand that feeds them). Once this unholy alliance had emerged, nothing in the subsequent 224 years proved capable of ending the mutualistic relationship. It has no other purpose than its own existence, like other cancers.
Sugar is the longest running experiment in planned economy on American soil. That's the reason we have HFCS rather than sugar in our food and drinks.