Quantifying the Unquantifiable

Why Regulating Lead (and Other Public Health Threats) Is So Hard

Dans l’air du temps (d’actualité)
Quantifying the Unquantifiable : Why Regulating Lead (and Other Public Health Threats) Is So Hard - Ariel Méndez


Despite the predictable nature of many threats to public health, safety, and security, governments around the world struggle to find a successful systematic response to these dangers. In fact, lawmakers often respond in a knee-jerk, emotional fashion that all but ensures that the most effective means of protecting the public from harm are ignored. Frequently, policy responses address only a primary danger, leaving us still vulnerable to an even greater secondary danger. Although cost-benefit analysis is not perfect, it carries some clear advantages to other responses such as the increasingly popular precautionary principle. This paper uses the example of lead poisoning to examine the question of why regulators struggle to accept cost-benefit analysis, and opt instead for inferior alternatives based on irrational, and sometimes unfounded, public fears.


On the front page of The New York Times on August 15, 2007, readers learned that Mattel, the world’s largest toy company, was announcing the biggest toy recall in its history.1 Out of the nineteen million toys recalled, fewer than half a million of them were due to concerns about lead paint, yet it was the potential for lead poisoning that grabbed people’s attention. Dr. Michael Shannon, a pediatrician and toxicologist at Children’s Hospital Boston and Harvard Medical School, called it a “public health disaster” and said “the biggest story is the clear failure of federal agencies to protect us.”2

In the months that followed, public fear about lead poisoning rose, state lawmakers decided to tighten standards governing lead levels in toys, and public officials amended their statements regarding lead regulations. In one instance, the San Francisco Chronicle issued a correction after the spokeswoman for the federal Consumer Product Safety Commission (CPSC) announced that the government was pursuing a total ban on lead in all children’s products. It turned out that the CPSC was only taking steps to ban lead in metal children’s jewelry.3 Of course lead is not the only danger that threatens public health, safety, and security. People learn about new threats every day, but what is most alarming is that governments have not figured out a good way to systematically respond to these threats. In fact, some evidence suggests that legislators might be doing just the opposite: responding in a knee-jerk emotional fashion that makes it easy for public policy to ignore the most effective means of protecting the public from harm.

All dangers that threaten public health, safety, and security have at least three components:
• the magnitude of danger (how prevalent and costly the consequences are);
• the probability of occurrence (how likely something is to happen); and
• the cost of reducing the likelihood and/or magnitude of that danger.

The magnitude of danger, i.e. the estimated cost of the consequences, is measured from a baseline where the danger is not present. From this perspective, the cost of the consequences in a safe scenario is equal to the benefit gained if the danger is prevented in another scenario. Sometimes the danger isn’t from doing something, like buying lead-contaminated toys, but rather from not doing something, like deciding not to vaccinate a child, or deciding not to buckle your seatbelt while driving.

Despite failing to deliver a systematic method of evaluating the risks, benefits, and costs associated with various policy options, the so-called precautionary principle seems to be gaining popularity over the last twenty years. Alternative responses, based primarily on cost-benefit analysis, are not perfect but carry some clear advantages. They nevertheless have failed to catch on. This paper uses the example of lead poisoning to examine the question of why regulators struggle to accept cost-benefit analysis, and opt instead for inferior alternatives based on irrational, and sometimes unfounded, public fears. Sometimes those irrational fears turn out to be legitimate: It is possible to be right without the support of solid evidence. Although cost-benefit analysis relies on being able to quantify what sometimes proves to be unquantifiable, it seeks to articulate public fears and clarify the risks and benefits associated with each policy option. It is a way of asking, “What is the paramount concern, and what is the most efficient and effective way to address that concern?”

Problems with the Precautionary Principle

The New York Times Magazine called the precautionary principle one of the ideas of the year in its 2001 “The Year in Ideas: A to Z.”4 The following year, Jiménez Beltrán, then the executive director of the European Environment Agency, noted that the precautionary principle is explicitly cited in the 1992 Maastricht Treaty on European Union and declared that it “is increasingly relevant” as a means to provide guidance to policymakers in “situations of scientific uncertainty.”5

But what exactly is the precautionary principle? The Science &Environmental Health Network convened the 1998 Wingspread Conference on the Precautionary Principle and produced the so-called Wingspread Declaration: "When an activity raises threats of harm to human health or the environment, precautionary measures should be taken even if some cause and effect relationships are not fully established scientifically."6

The basic premise behind the precautionary principle is that it is “better to be safe than sorry.” If there is some doubt as to whether a proposed action will harm human health or the environment, the precautionary principle’s advice is simple: Do not proceed.

For anyone thinking of ways to actually implement this principle, it becomes clear that it suffers from incoherence, and as stated, actually prohibits the actions that it prescribes. One problem is that the world is full of scientific uncertainty and thus the precautionary principle commands us to take precautionary measures in just about every situation. Although this reduces the usefulness of the precautionary principle, uncertainty itself should not be viewed negatively. Henry Pollack observes that the relationship between science and uncertainty is a beneficial one. “Far from being an impediment that stalls science,”Pollack writes, “uncertainty is a stimulus that propels science forward. Science thrives on uncertainty… Indeed, one might argue that it is certainty, rather than uncertainty, that impedes science.”7

Cass Sunstein argues that a related problem is that the world is permeated with certain risks on all sides. Since nothing is ever completely risk-free, Sunstein explores the notion of “substitute risks.” Substitute risks are secondary risks that are incurred while attempting to avoid primary risks. Sometimes, the secondary risks are even greater than the primary risks. This is the “out of the pan and into the fire” problem.

To illustrate this scenario, Sunstein recalls a train crash that occurred in Hatfield, UK in 2000. After the crash, which injured dozens and killed several passengers, fully a third of British rail travelers started using the highway instead of taking the train. The problem was that driving in the UK is more than ten times as dangerous as riding the train. In the first thirty days following the train crash, it was estimated that five additional deaths from automobile accidents could be attributed to the additional drivers on the road. That figure, Sunstein notes, was “nearly equal the total number of deaths from train accidents in the previous thirty years.”8

Since risks are pervasive in the real world, Sunstein believes that the problem of secondary risks constitutes the biggest defect of the precautionary principle. If the stated goal of trying to avoid or minimize risks to human health or the environment is taken at face value, what guidance is there to choose between the primary and secondary risks that permeate all options? When it comes to distinguishing substitute risks, Sunstein writes that “…the [precautionary]principle offers no guidance…it forbids all courses of action, including regulation.It bans the very steps that it requires.”9 What is needed, Sunstein argues, is a careful comparative assessment of the risks and benefits – and the cost – that each option entails.

Business-As-Usual: The Precautionary Principle and Lead Poisoning

Cass Sunstein has written extensively on the question of how regulators should respond to the kinds of typical challenges that the problem of lead poisoning exemplifies. He argues that the kind of strong, emotional reaction and confusion that often follow an event like the Mattel toy recall get in the way of taking measures that could more effectively protect the public from the very dangers of which it is so afraid. Sunstein advocates cost-benefit analysis as the best way to achieve this.

But there is a catch: “In democratic nations,” Sunstein writes, “the law responds to people’s fears. As a result, the law can be led in unfortunate and even dangerous directions.”10 People’s fears do not always respond in a way consistent with the measured approach of cost-benefit analysis; the aftermath of the Mattel toy recall was no exception.

In the months following the Mattel toy recall, Washington state legislator swere unwilling to wait for federal lawmakers to address the problem and pushed through some of the toughest standards in the nation restricting the amount of lead and other toxins in children’s toys. As a result, the bill restricted the amount of allowable lead to 90 parts per million, far below the federal standard of 600parts per million.11

The measure, called the Children’s Safe Products Act, was overwhelmingly approved by the state’s House (92–2) and Senate (40–9) before it was signed into law by Governor Christine Gregoire.12 The Toy Industry Association, along with industry heavyweights such as Hasbro and Mattel itself, strongly opposed the measure, saying that the restrictions were unfair and would “cause complete chaos.”13 In a letter to the editor of the Seattle Post Intelligencer, Rick Locker of the Juvenile Products Manufacturers Association invoked the problem of substitute risks. Locker wrote that the bill was “a hastily written law” that would unintentionally ban car seats, a device responsible for saving thousands of lives.14When Governor Gregoire signed the bill, she specifically noted that car seats were likely to meet the standards of the bill, and took steps to ensure that the bill would not reduce the safety of car seats.15

Was the immediate decision to tighten restrictions on lead in toys the best response to the public’s fear of lead poisoning? Since there is no known safe blood lead level, the action by the Washington state legislature is certainly consistent with the precautionary principle. The U.S. Department of Health andHuman Services considers lead poisoning to be a “wholly preventable disease.”16Removing lead from children’s toys is an obvious step towards prevention, but regulators need to know what costs are involved if they are to pursue prevention in the most efficient way possible. Perhaps most importantly, in order to avoid being lulled into a false sense of security by attending to a minor risk while ignoring a major one, regulators also need to evaluate whether substitute risk sexist.

As it turns out, lead in toys constitutes a relatively small share of lead exposure to children. The largest source of lead exposure is in lead-based paint in older homes and lead-contaminated dust from old buildings.17 Even if the government succeeded in reducing or even eliminating lead in children’s toys,the most significant sources of lead contamination would still persist. If overall pediatric lead poisoning is the highest concern, then an effort to identify the largest sources should be made in order to determine the most cost-effective way of reducing overall exposure among children. Cost-benefit analysis is a way to accept the public’s fear about lead poisoning and then determine the least expensive way of preventing lead poisoning.

Cost-Benefit Analysis

In Cost-benefit Analysis, Richard Layard and Stephen Glaister describe cost-benefit analysis as the fairly simple and even intuitive process of summing the costs and benefits of each option under consideration and then simply choosing the option with the greatest net benefits. When considering option A, they write that “the rule becomes: do A if its benefits exceed its costs, and not otherwise.”18But since this rule overlooks the crucial process of measuring and estimating that is needed in order to assign values to the costs and benefits of any option, Layard and Glaister concede that this rule is deceptively simple.

Cost-benefit analysis does not address questions of distribution, timing,the value of future consumption, the ability to pay, and a host of other issues, but most opposition is centered on the necessary hurdle of measurement and estimation that is required in order to assign numerical values to costs and benefits in order for cost-benefit analysis to work. After all, intangibles cannot be summed; only numbers can. Some opponents object out of principle, believing that it is impossible to assign a cash value to something as sacred as human life.19

Others, including proponents of cost-benefit analysis, point out that the problem of measurement is not a moral one, but a technical one. How can one measure pain and suffering, benefits for animals, aesthetic and recreational gains,or overall well-being and health?20 Sunstein, himself an ardent supporter of cost-benefit analysis, admits that “An adequate cost-benefit analysis would certainly take those benefits into account.”21 Considering this deficiency, can any cost-benefit analysis really be “adequate”?

For some, the answer is “no” and the whole idea of cost-benefit analysis becomes meaningless. For example in 2001, the question of acceptable arsenic levels in tap water became the focus of public attention. “Americans may disagree about a lot of things,” said Rep. David Bonior (D-MI), “but drinking arsenic...isn’t one of them. When we turn on the kitchen sink, we ought to be able to drink what comes out without worrying about being poisoned or poisoning our family.”22Since nobody wants to experience lead poisoning, and there are no known benefits to having even trace amounts of lead in one’s body, Bonior’s statement about arsenic could just as easily have been about lead.

By this measure, the question of how much it will cost to eliminate sources of lead or arsenic, or whatever the particular danger may be, is never raised.Instead, cost is ignored and the command is to eliminate the risk at whatever cost. But in the real world, where scarcity of resources is a reality, the next question must necessarily be what the cost is – including the cost of any substitute risks and dangers – of eliminating or reducing the primary risk. Only after examining that question can the decision to eliminate or reduce the risk be properly made.

Cost-Benefit Analysis and Lead Poisoning

In order to appropriately address the problem of lead poisoning using cost-benefit analysis, the essential question to ask is, “What is the cost of lead poisoning, and how much are we willing to spend to eliminate or reduce that cost?” In 1972, Herbert Needleman laid the foundation for the now well-accepted inverse relationship between blood lead levels and cognitive ability.23 Nearly thirty years later in 2000, the EPA used that relationship in an in-depth analysis to compare the lost earning power of lead-poisoned children to the cost of removing lead paint from homes across the country. 24 The goal of the study was to determine whether the cost of removing the largest source of lead in children’s environments could be justified in terms of preventing the lost economic productivity that resulted from lead poisoning.

Depending on how stringent a standard is applied, the EPA estimated that it would cost a total of $51.7 billion to $100.6 billion to remove lead from all the targeted homes where children will be born over the next 50 years.25 The EPA then estimated that, based on an average lifetime earning potential of $366,021(1992 $), the average cost for each child in terms of foregone lifetime earning capacity was about $8,346 per lost IQ point.26 Using a variety of models, the EPA concluded that net national benefits of preventing lost IQ points through lead removal and abatement ranged from $173.5 billion to $-5.2 billion.27

In 2002, two years after the EPA study, Philip Landrigan led a team of pediatric researchers from the Mount Sinai School of Medicine, the New School for Social Research, and the Harvard School of Public Health to replicate theEPA’s assessment of how much lead poisoning in children was costing the country.28 One of the goals of the study was specifically to provide a value of the costs of lead poisoning in order “to counter one-sided and often ill founded claims about the high costs of controlling pollution.”29 In other words, the researchers saw that those in opposition to lead regulation had come up with figures about the costs of that regulation, and Landrigan et al. wanted to counter those figures with numbers of their own.

Landrigan et al. used the same model to assess lead poisoning’s impact on lost lifetime earning capacity to find that the total cost of lead poisoning inU.S. children amounted to a staggering $43.4 billion a year, equal to about 2.2 percent of annual U.S. healthcare costs.30 This study used a reasonable lifetime earning potential of $881,027 for males and $519,631 for females (1997 $), both figures higher than that employed by the EPA study. Nevertheless, Landrigan and his team claimed to have “consistently erred in the direction of conservatism.”

Since the study omitted such costs as pain and suffering, special education for children with lead poisoning, and costs for which no quantitative data were readily available, the researchers believed that the actual cost of lead poisoning in children was almost certainly greater than the figures they published.

This attempt to affix a dollar value to a problem like lead poisoning received heavy criticism after the study was published. One critic agreed that there exists a “well-known effect of lead exposure on neurodevelopment in children,” but nevertheless charged that “quantification of cost estimates may be overstating the scientific certainty of the disease-environment relationships,” and as a result, this particular study might cause researchers in the entire field “to lose credibility within the larger community of scientific and medical research.”31

Reliance on scientific certainty is a laudable goal, but it should not be used to discredit an otherwise useful analysis simply because it relies on estimates to quantify a known effect.  To require absolute scientific certainty before accepting any estimates is not only unhelpful, it is probably impossible.

Additional criticism of the model used in the EPA and Landrigan studies argues that lost lifetime earning potential should not be considered. For instance,Randall Lutter, an economist who is currently a deputy commissioner for the U.S. Food and Drug Administration, employs a willingness to pay calculation to find that each IQ point is actually worth between $1,100 and $1,900, about six times less than the figure cited by the EPA, and nearly twelve times less than Landrigan’s figure.32 In order for cost-benefit analysis to work, a numerical estimate is needed, but which one is better?

Lutter believes his figures are superior because they are “based on observed behavior” of the parents, who are the ones most likely to pay for the lead abatement. Arguing that “parents may value their children’s welfare less than their own,”33 Lutter concludes that it would be unfair, or “inequitable,” as he puts it, to ask parents to pay for costly lead abatement because the benefits go almost entirely to their children. He says that children are not only incapable of offering “credible commitments to repay such investments,”34 but they are also “likely to live longer and have much higher incomes than their parents.”35

Making Complicated Estimates

Making these kinds of estimates is where some might say things become interesting. Frank Ackerman and Lisa Heinzerling oppose the monetization of human life and health from a moral perspective: such values are inherently“priceless.”36 They further argue that even if one were to accept the moral validity of such an exercise, valuing lost IQ points (or any illness, for that matter) only in terms of lost economic productivity sorely undervalues the actual cost. They argue that there is an additional social cost that is missed by this analysis: “nothing is included for the change in the child’s quality of life and capabilities outside of work, nor for the impact on the parents.”37 For these reasons, they say that the EPA’s $8,346 figure is certainly too low.

Assigning values to these kinds of benefits is complicated even further by the fact that people value the same item differently depending entirely on context. One example of this can be found in Richard Thaler and others’ work on the “endowment effect.”38 The endowment effect describes the disparity between what people are willing to pay in order to gain something and what price they are willing to accept in order to give up that same item. Thaler discovered that the acceptable price to give up an item was usually higher than the price people were willing to pay in order to acquire the same item. With respect to valuing IQ points, this means that parents might be willing to pay different amounts to prevent the loss of IQ points than they would to recover IQ points already lost (if such recovery were possible). There is also the possibility that parents would value the loss of the IQ point from 100 to 99 differently than the loss of the IQ point from 90 to 89.

Another complication is that people see the same risk differently depending on how the risk is framed. Redelmeier et al. showed that medical patients respond differently to the same information depending on how it is presented. For example, more patients were willing to undergo a procedure when they were told that out of 400 patients, 350 of them were alive after five years than if they were told that fifty of them were dead after five years.39 All this throws considerable doubt on what the dollar value of an IQ point actually is.

These are only a few of the challenges one must overcome to assign values needed to perform cost-benefit analysis. But there is one other stumbling block in performing an adequate cost-benefit analysis for lead poisoning. A 2001 study published in The New England Journal of Medicine found that chelation therapy, the process of removing lead or other heavy metals from one’s blood stream, is not effective as a means of regaining lost cognitive ability from lead poisoning.40 This means that recognizing the major sources of lead exposure and taking preventative measures are the most important ways to protect healthy children, because once the neuropsychological damage of lead poisoning occurs, it is not reversible.

Most literature on lead poisoning agrees that old household paint and dust are the largest sources of lead exposure to children. Unfortunately, identifying specific sources of lead exposure as responsible for a certain percentage of lead poisoning is extremely difficult and to date has not been accomplished. This means that even if the largest sources of lead exposure such as lead paint and dust from children’s homes are eliminated, it may be impossible to anticipate precisely how much lower lead poisoning rates would be.41


It may very well be that Washington state legislators are correct, and that tighter restrictions on lead in toys are the most cost-effective way of reducing children’s exposure to lead. The answer cannot be known for certain unless all sources of lead poisoning are examined and a cost-benefit analysis is performed to evaluate all the options on the table. But the process of assigning values to the sometimes intangible things in order to perform cost-benefit analysis is inherently quite difficult and not entirely objective.

Cost-benefit analysis quantifies the associated benefits and costs of a proposed action, but the results will only be as good as the estimated values that go into the analysis. Simply because making those estimates is difficult and sometimes uncomfortable does not mean that the entire process should be abandoned. For example, with respect to the Washington State decision to tighten lead standards in children’s products, people ought to have some idea of the benefits of those new standards (How much are lead poisoning rates in children expected to decrease?) and also what new costs they can expect to bear (By how much will the performance and cost of toys be affected?).

Cass Sunstein argues that the government should never force people to pay for something they do not want.42 But does it make sense to favor giving parents the choice between paying more for a lead-free toy and paying less for a lead-laden toy that may cause irreparable cognitive harm to their child, simply because doing so increases parental choice? Increasing personal choice may be efficient from a market perspective, but is not always morally acceptable. Opponents and advocates alike raise valid points both against and for cost-benefit analysis. The trouble is that since both perspectives carry some validity, there is not any easy way to reconcile them. Opponents are correct to point out things such as the logical leap it takes to say that the value of a “statistical life” is $6.1million simply because it can be demonstrated that people will accept $610 in order to incur a 1/10,000 chance of dying while performing their job. That does not mean that those same people would be willing to accept $6.1 million in order to incur a 1/1 chance of dying, but nevertheless that is the figure that the EPAhas used to determine the statistical value of one life when performing cost-benefit analyses.43

On the other hand, the implication of saying that human life and health care priceless is that all dangers that threaten human life and health should be removed no matter what the cost. This is an admirable but clearly impossible goal which could unintentionally jeopardize additional lives. For instance, people often oppose nuclear energy because of the potential threat to human lives and the environment of nuclear waste or a nuclear accident. But these risks ought to be weighed against other risks such as the potential consequences of climate change and the estimated 26,000 annual deaths attributed to the emissions of coal power plants.44 Opponents are right; it is absurd to put a dollar value on something like a human life. But since policymakers must make these difficult choices anyway, cost-benefit analysis is a valuable tool that helps establish regulatory priorities.

Sunstein would like to see the advent of a “cost-benefit state” that would enable and support democracy by allowing people to make better-informed decisions about regulating public dangers.45 The executive office in Sunstein’s cost-benefit state would “publicize risk-related information,”46 but the problem is this kind of information is routinely buried for political or profit motives. We have already taken a look at how difficult it is to affix numerical values to costs and benefits, and these challenges are all compounded when conflicting interests try to push the result in one direction or the other. It is easy to say that “the government should provide us with bias-free assessments of costs and benefits,” but such neutrality is hard to achieve.

This paper has looked at the heightened awareness of lead poisoning that the August 2007 Mattel toy recall sparked, but the issues it raises are endemic to a wide array of regulatory challenges. New issues are continually emerging. For instance, a recent front page Washington Post story exposed the role of certain plastics in causing cancer and other serious disorders.47 While the specifics differ, the issues are the same. The question is how governments will respond to them.


Ariel T. Méndez is a recent graduate of the Paul H. Nitze School of Advanced International Studies (SAIS) and, for the time being, a stay-at-home dad. He worked at the Woodrow Wilson International Center for Scholars for the Environmental Change and Security Program in Washington, DC from 2001 to 2004. He then worked with community-based nonprofit organizations at the Center for Volunteer and Nonprofit Leadership of Marin in San Rafael, CA before attending SAIS. Mr. Méndez holds an M.A. in International Relations from The Johns Hopkins University School of Advanced International Studies and received his B.A. in Politics with Highest Honors from the University of California, Santa Cruz.