Ethics & Public Policy Center

The Congressional Budget Office Needs to Be Reformed

Published in National Review Online on June 2, 2017


Reactions to the Congressional Budget Office’s May 24 score of the American Health Care Act have naturally tended to fall into familiar partisan grooves.

Some Republicans have attacked the CBO in an effort to defend the bill, not only questioning the agency’s methods and pointing to its past failures to project the health economy but even (as in the case of Office of Management and Budget director Mick Mulvaney in an interview with the Washington Examiner) accusing it of abject political bias. Some Democrats, meanwhile, have defended the CBO in order to attack the bill, asserting that the agency’s scoring is unimpeachable and holding up its specific numerical predictions about the AHCA as simple facts.

I’m far from immune to the gravitational pull of this partisanship, and I’m sure what follows will not be free of it. When first reading the CBO’s score, my mind went right to some of its most bizarre and weakest points. There are many to choose from, and I agree with much of what has been written about them by many observers on the right. (This overview by the Galen Institute’s Doug Badger nicely covers some key concerns.) But it also has to be said that such criticisms of the CBO’s methods, and even criticisms of its intentions, don’t actually amount to defenses of the AHCA, just as the Democrats’ attacks against the bill are not defenses of the substance of the CBO’s score. The two sides of this argument are in many cases talking about two different things, and it is worth distinguishing them.

Drawing such distinctions would also help to clarify that some key concerns about some of the work of the Congressional Budget Office are not best understood as criticisms of it, and certainly not of its staff — which in my experience has always been defined by integrity and professionalism in the face of some enormously difficult pressures. The CBO is not politically biased. But its work is beset by challenges that run much deeper than that: They are structural, and require us to think about both the CBO and the larger congressional budget process in which it plays a part in terms of institutional reform.

The AHCA has its strengths and weaknesses. I’ve written about my sense of both, as many others have of theirs. But here my purpose is neither to defend the bill nor to criticize it. Instead, difficult as it may be, let’s try to think about what the CBO’s score of the AHCA tells us about the CBO — its capacities and its limits, and how its work might be best understood and also strengthened. No reform of the agency’s work could come in time to affect this iteration of the health-care debate, so in this debate we will need to make the most of the scoring we have, to take it seriously, and to try to learn from it. But this bill, for a variety of reasons, brings into the light some key challenges to the work of Congress’s scorekeeper that ought to be thought about with the future in mind.

Gaming the Scorekeeper
Part of the trouble the CBO often faces, though in this case a fairly small part, has to do with explicit congressional manipulation of the scorekeeper. Simply put, CBO has to score legislation under assumptions that can be manipulated by the Congress.

We can see an example of this in the statement, on page 16 of the CBO score of the AHCA, that “on the basis of consultation with the budget committees, costs and savings are measured relative to CBO’s March 2016 baseline projections, with adjustments for legislation that was enacted after that baseline was produced.” The “baseline” in this case refers to the starting assumptions the CBO works with about where levels of spending, insurance coverage, and other factors would stand over the coming ten years under current law — that is, if nothing were to change. Proposed changes to the law are then measured against this starting point.

The CBO makes its baseline assumptions for key programs available online. And if you look through them, you’ll find that the March 2016 baseline is not the most recent CBO baseline; a more recent one was produced in January of this year. It differs in some respects that would be quite significant for this particular score. For instance, in its March 2016 baseline, CBO projected that 18 million people would be buying coverage in the Obamacare exchanges in 2026 (if the law didn’t change), while in its January 2017 baseline the agency said 13 million would do so that year. That’s a big difference. Similar differences are evident in projections of Medicaid-expansion enrollment and in some of the agency’s spending projections.

So why would CBO use an older baseline as the foundation for its projections about this new bill? Basically because it was told to by the congressional leadership. That’s what “consultation with the budget committees” means. I think that using the more recent baseline would have yielded (modestly) better projections of insurance coverage but also somewhat higher projections of cost, so the decision to use the earlier baseline may tell us something about the priorities of the relevant policymakers — or of those whose votes were most in doubt.

A more familiar form of congressional manipulation of the scorekeeper — “gaming” the CBO’s assumptions and methods to achieve a certain score — was largely absent from this particular bill, to the detriment of its coverage score. The AHCA has been (so far) crafted with such haste and carelessness that there has been little opportunity to think through ways of juicing its score. But that sort of manipulation, in which elements of legislation are designed or tweaked with the peculiarities of the CBO model in mind specifically to achieve a certain score, is still an important part of the story of the agency’s verdict on the AHCA because of just how much of this kind of gaming happened in the crafting of Obamacare.

As it made its way through Congress in 2009 and 2010, the Affordable Care Act basically took shape through a long series of back-and-forth adjustments between the relevant committees and the Congressional Budget Office, intended to tweak the coverage and cost scores of the bill to accord with the idiosyncrasies of CBO’s modeling. Some forms of this gaming were cynical and reckless — the foremost example must be the CLASS ACT, a long-term-care program known from the start to be completely unsustainable, which was included in the bill purely to manipulate its cost score and then abandoned before it actually took effect. But almost every facet of the legislative design of Obamacare was a product of this kind of back and forth, so that the law in some respects was built to achieve a certain result in CBO’s model even more than in the world outside it. (On this subject, I’d highly recommend political scientist Robert Saldin’s recent book When Bad Policy Makes Good Politics: Running the Numbers on Health Reform.)

This wasn’t all cynical, of course. The CBO’s assumptions about the effects of various policy measures is the product of a kind of liberal consensus in health economics, so that the views of the designers of Obamacare were already fairly well in line with the assumptions baked into the model about the relative effects of mandates, taxes, regulations, and price controls on the one hand and of competition, consumer choice, state experimentation, and price signals on the other. Both Obamacare’s designers and the CBO’s model took the first set of policy tools to be far more powerful than the second. Maybe that’s not quite a political bias, and maybe it’s not even wrong, but it is a reflection of some implicit assumptions among health economists that have been too rarely questioned over time.

But the design of Obamacare did much more than reflect this broad view. It involved intense gaming of the particular features and quirks of the CBO’s health-economy model. And because its legislative design was so tightly calibrated to the parameters of that model, any changes to that design are almost unavoidably going to be scored as changes for the worse by CBO.

The classic example of this is the individual mandate. It is by this point basically a quirk of the CBO model that it judges the mandate to be exceedingly effective — far more effective than the evidence of the past few years would suggest. The AHCA would eliminate the mandate right away but retain most of the other key features of Obamacare for a two-year transition period, but CBO says that 14 million people would immediately lose or give up their health insurance, and that this would mostly be a function of their no longer being penalized for not buying it. Anything is possible, but that sure isn’t likely.

Late last year, a team that included Jonathan Gruber, the MIT health economist whose earlier work had a great deal to do with CBO’s assumptions about the mandate (among many of its other assumptions), looked into whether it still made sense to think this way about the effect of the individual mandate. As they put it in the New England Journal of Medicine:

When we assessed the mandate’s detailed provisions, which include income-based penalties for lacking coverage and various specific exemptions from those penalties, we did not find that overall coverage rates responded to these aspects of the law. Does that mean the mandate had no effect? Not necessarily. If its primary result was to make all Americans more likely to obtain coverage — whether or not they were subject to the penalty and irrespective of how much it would cost them — our analysis would not capture that effect.

Not exactly strong support for the CBO’s insistence on the power of mandates. But for now at least the agency persists in its view, and because the design of Obamacare answered precisely to its assumptions, its persistence in that view has an enormous effect on how it scores the AHCA’s different approach.

The same is true, if in less stark ways, for other key provisions of Obamacare that the AHCA repeals or replaces. They were fine-tuned to the CBO’s modeling, so changing them sometimes results in strange and very adverse projections. That doesn’t mean that changing them would not in fact result in very adverse real-world effects. The CBO’s model isn’t nonsense. Far from it. It just means that the particular eccentricities and weaknesses of the model — types of problems any health-economy model might have, in different forms — are exaggerated and amplified as a result of the gaming involved in crafting the law now being altered. In a sense, the CBO is scoring the AHCA by measuring how similar it is to Obamacare as much as by projecting its effects on the health system. So it penalizes the new law where it differs from Obamacare’s key provisions, and (as senators are now finding) the only way to get a better score out of the model in such cases sometimes is to re-create some of Obamacare’s provisions — even those that are failing in the real world.

Some of the more peculiar features of the score seem to be functions of this phenomenon. For example, the CBO’s overall coverage score of the AHCA is essentially identical to its coverage score of a bill Republicans sent President Obama in 2015 that would have repealed all of Obamacare’s insurance subsidies without replacing them with anything. (Or rather, it is identical to the effect on coverage that the CBO said that bill would have had if it also repealed Obamacare’s insurance regulations, as explained on page 3 of this CBO report.) In effect, spending almost half a trillion dollars on insurance coverage and market stabilization over ten years is scored as having almost no net effect on coverage levels.

Similarly, the version of the AHCA passed by the House and scored by the most recent CBO report spends about $46 billion more on Medicaid over the next ten years than an earlier version of the bill scored by the CBO in March. The agency accounts for that additional spending, but scores it as having no effect on Medicaid enrollment. These would seem to be functions of peculiar quirks of the model, brought to light by changes to a legislative architecture that was designed to capitalize on them.

This results in the exaggeration of some of the distinct eccentricities of CBO’s model. That it has such eccentricities is entirely understandable, and indeed unavoidable. Any complex health-economy model would. But when legislation is built around them, as Obamacare was in some key respects, CBO finds itself confronted with some very strange problems.

Layers of Assumptions
But the CBO’s report on the AHCA suggests some even more fundamental structural problems than these, which would pose a challenge not only to a bill intended to change a law designed to game the agency’s model but to any bill that doesn’t follow something like the general structure of a Great Society program. Some of these, too, are brought out in unusual relief by the nature of the AHCA — for good and bad.

One is the tendency of the CBO to model competition as having minimal effect on costs while modeling price controls to be efficient and effective. This problem is a matter of degree. It certainly may be the case that the CBO is justified in assuming that blunt policy instruments can be more effective in some instances. And there is no denying the complexity involved in projecting the workings of a vast consumer market rather than a centralized government program. Competition is obviously much more difficult to model than mandates and price controls, but the agency’s experience with Medicare Advantage and the Medicare prescription-drug benefit suggests that it tends to significantly understate the effects of competition — which obviously has consequences for its scoring of reforms intended to increase the market orientation of the health-care system.

CBO is aware of this problem, though it has tended to resist accounting for the effect of competition on its scoring even in retrospect. When the Medicare prescription-drug program (which involves competition among private insurers) turned out to cost 50 percent less in 2013 than the CBO had projected when the program was enacted in 2003, the agency undertook an impressive self-critical exercise to consider what had happened. The resulting report considered many of the factors that shaped the evolution of the program but ended up punting on the question of competition, arguing that “because other factors have affected costs per beneficiary, determining whether the competitive design of the program has been more or less effective than CBO originally anticipated is not feasible.”

Making such a determination would have been no simple matter, and the CBO has acknowledged that it lacks the tools to do it properly. In a 2011 hearing, for instance, CBO’s director at the time, Douglas Elmendorf, was asked why the agency did not account for the effects of competition in assessing Medicare-reform proposals, and he responded: “We are not applying any additional effects of competition on this growth rate over time in our analysis of the proposal. And again, we don’t have the tools, the analysis we would need to do a quantitative evaluation of the importance of those factors.” He later described the challenge of modeling competition as “a gap in our toolkit, and a gap we are trying to fill.” The agency has made some progress toward filling that gap, particularly in its assessments of premium-support reforms in Medicare, yet the gap remains significant.

But the greater problem revealed by the score of the AHCA is the difficulty of accounting for policy variation in the states. This is not a problem with the CBO’s model in particular. It would be a problem with any attempt to model the health economy when states have the ability to control and change the regulatory environment. And it becomes a particular problem when states are newly granted, or regranted, the ability to do that all at once.

The AHCA, in its most recent form, would give states that ability with regard to some key facets of health-insurance regulation. It would allow states to obtain waivers that would let them define essential health benefits for themselves, would let them loosen the age bands on premiums, and would let them constrain the reach of community rating in a few narrow circumstances, to create an incentive for younger and healthier people to buy coverage. The trouble for CBO is that it can’t know what states would choose to use these waivers and how they would use them.

In scoring the bill, CBO acknowledges that “the array of market regulations that states could implement makes estimating the outcomes especially uncertain.” That hardly begins to describe the uncertainty involved, of course. CBO continues:

Many factors would influence states’ decisions, as discussed below, and a projection of a specific state’s actions would be highly uncertain. As a result, CBO and JCT’s [Joint Committee on Taxation] estimates reflect an assessment of the probabilities of different outcomes (without any explicit predictions about which states make which choices) and are, by the agencies’ judgment, in the middle of the distribution of potential outcomes. Moreover, CBO and JCT’s assessments in this analysis should not be viewed as representing a single definitive interpretation of how H.R. 1628 should or would be implemented

These are massive caveats. But the report then proceeds to make a series of crucially important yet unexplained assumptions. For instance, it assumes that states that are home to half the U.S. population would not use any waivers, states home to another third would use them modestly, and those home to another sixth would use them more aggressively. Then it works through different implications of each approach, and in the end it adds all this up to arrive at some seemingly precise estimates of coverage levels and federal spending.

In places, the difficulty CBO analysts faced in projecting state decisions and their effects becomes especially evident. The report suggests, for instance, that CBO is not able to estimate insurance-premium costs in the states that would use waivers most aggressively (states in which about 50 million Americans live, according to the agency’s assumption), and yet the report does make coverage estimates for those states and also federal cost estimates, both of which would surely depend on having premium figures in the model.

The report also says that “a few million people” in such states would buy insurance coverage that the states would approve but that CBO won’t count as insurance because it would not sufficiently cover major medical risks. CBO counts those people among the uninsured, even if the states that would be newly re-empowered to regulate health insurance would consider them insured.

These sorts of challenges suggest that CBO confronts a very basic problem when it comes to legislation that affords states or localities meaningful leeway in program design and regulation. If that kind of policy devolution and decentralization is going to be key to the modernization of American government (as I think it needs to be), then CBO will constantly find itself asked to do things it is not equipped to do.

This already happens now, of course, and the agency’s report on the AHCA makes it pretty clear that CBO’s analysts understand this. It is evident in places in the report that its authors can see that some of what they’re doing — and particularly the pretense of precision they are expected to advance — is absurd. But they have no choice but to proceed toward comically exact-sounding conclusions despite immense uncertainty and layers upon layers of political guesswork that could never be called economic modeling.

Toward the end of the report, CBO offers a hint of how to think properly about what it’s offering policymakers. After laying out the numerous, varied sources of uncertainty bedeviling its work, the report concludes in this extraordinary way:

Despite the uncertainty, the direction of certain effects of the legislation is clear. For example, the amount of federal revenues collected and the amount of spending on Medicaid would almost surely both be lower than under current law. And the number of uninsured people under the legislation would almost surely be greater than under current law.

That is a very limited claim. It’s more than fair, and surely true. Spending would be lower, revenues would be lower, and coverage levels would be lower. But that is a far cry from the kinds of seemingly precise claims thrown around elsewhere in the report. And it is those kinds of claims that unavoidably shape how the legislation is understood.

Strengthening the CBO
That the Congressional Budget Office is structurally ill-suited to assess legislation that relies on competition and federalism to achieve its objectives is not an argument against such legislation — but neither is it an argument in its favor. If anything, it points to the need to modernize the CBO’s capacities, and to the need to lower our unrealistic expectations of the agency.

Maybe one way to advance both goals is to increase the transparency of the CBO’s work, so that it functions less as a mysterious oracle that serves up hard numerical pronouncements and more as a facilitator of the effort to consider the potential implications of legislation. But transparency is not as simple as it sounds.

Many observers over the years have suggested, for instance, that the CBO should offer range estimates, rather than precise point projections, when scoring the cost or other consequences of proposed legislation. That would give legislators some sense of the agency’s level of confidence and of the plausible range of outcomes. But in 2014, CBO director Elmendorf addressed that idea in a blog post in which he nicely articulated the challenges involved in making the CBO more open. For one thing, he noted, the CBO provides point projections because it is required to by Congress. But even apart from a formal requirement, it’s hard to see how a range estimate could actually be used in the legislative process. And perhaps most important, the nature of CBO’s work means, he wrote, “that we often lack a strong analytical basis for constructing such ranges. One obstacle is that most of the models and estimating techniques we use are not formal probability models, so they do not readily yield measures of uncertainty.”

CBO’s work involves much more of an art than the kind of almost mechanical science that outsiders might imagine when we hear the term “model.” Its health-economy model, for instance, as well described in this 2007 report, is an intricate mix of complicated spreadsheets that requires careful management and a fair bit of judgment. Opening up that work for real-time inspection could well unfairly undermine the CBO’s standing in the eyes of legislators and the public, who would frankly have trouble knowing what to do with the confusing mess that such transparency would reveal.

In his 2014 note about range estimates, Elmendorf gently expressed this sort of concern about what his employers would make of such information. “Surely it is useful for legislators to be aware of the uncertainty of budgetary and economic estimates,” he wrote. “But what else might they do with such quantification?” He was not wrong to worry. But that worry does not resolve the problem that causes people to want more transparency.

A broader transformation of the CBO’s work might better enable such transparency while also strengthening the agency’s capacities and responsiveness. Moving CBO (and JCT) toward more open-source modeling, for instance, could hold out some promise. Rather than produce stark hard-number projections behind a heavy veil, the CBO and JCT could act more as developers and stewards of an open public model, available online to anyone with sufficient technical prowess to use it. The two agencies, with outside help, would create and maintain the model, and would decide on a set of official economic assumptions and policy expectations to be used in modeling for formal budget-process purposes. Anyone could “score” any policy proposal using those official assumptions and could also use the model with other assumptions to project a proposal’s consequences under alternative circumstances.

Users could also propose improvements. And outside developers could build applications that made use of the underlying model to make it easier for non-experts to apply it to a variety of relatively straightforward circumstances. Such an open model would be a public resource, rather than a private secret. It would contribute to greater transparency — putting assumptions on the table to be debated and making the enormous uncertainty of projections better understood. And it would help CBO be a spur to policy innovation, rather than a bottleneck.

There are, of course, enormous technical challenges to overcome before anything like that could happen. (I discussed some of those here a few years ago.) Work already being done in this direction by the Tax Policy Center, the American Enterprise Institute, and some academic modelers could help some, but only some. And there are also many more practical challenges. After I wrote about a version of this idea in 2014, Elmendorf kindly got in touch with me to say the CBO was working to improve transparency but was severely constrained by congressional requirements and that a transition to such a mode of work would be enormously challenging.

That is no doubt true. But whether in this way or in others, some fundamental modernizing reforms do seem to be called for — not because CBO is politically biased or doing bad work but because it needs to adjust to how Congress has changed, how the country has changed, and how technology has changed.

Institutional Reform
Ultimately, such modernization would need to occur as part of a larger reform of the budget process in Congress, which is terribly broken now.

The Congressional Budget Office was created as part of the 1974 budget reforms, which also created the budget committees in both houses of Congress and the modern budget process. That entire edifice now seems on the verge of collapse. The process, which was intended to (and did for years) make Congress more efficient about budgeting, now exacerbates Congress’s worst vices — and particularly its paralyzing polarization.It has put at the center of the legislative branch’s work a process that invites the parties to fight over abstract visions (in their budget resolutions) rather than concrete policies, and so makes bipartisan agreement even less likely and makes Congress’s work even less focused, specific, and substantive. At this point, the very existence of the two budget committees strikes me as a detriment to the functioning of Congress. It is time for fundamental budget-process reform.

And on this front, too, the AHCA can teach us something important. The bill is being pursued through the reconciliation process, which is supposed to be a way of using the annual budget resolution to advance essential fiscal measures. But whether or not it technically fits as a taxing and spending measure within the parameters of the Senate’s “Byrd rule,” the bill being moved is surely not connected in any meaningful way to the budget process. The Congress has already passed a continuing resolution for the remainder of this fiscal year and yet is still, as a formal matter, pursuing the health-care bill as though it were an element of the 2017 budget process when everyone understands that it is no such thing.

Meanwhile, the work of crafting the bill in both houses has been pursued outside the structure of the committee system, through informal negotiation among different factions of the Republican conference. Senate Republicans have even felt the need to create an ad hoc group of senators — what any observer might call “a committee” — to develop its version of the bill outside the established committee system, which apparently cannot be made to work. We are all naturally focused on the travails of the presidency these days, but Congress looks to be very badly broken.

Maybe this ad hoc approach to health-care legislation suggests the shape of the next functional legislative process in Congress. But it surely also suggests a loss of faith in the existing forms and rules of the legislative process Congress now has. It should invite reflection and reform, in other words. And that reform should naturally make room for a modernized CBO to play a role more in keeping with the needs of the contemporary Congress.

This would be a great time for institutional reform in the legislative branch. It is badly needed. It could speak to the challenges of fragmentation, institutional decay, and cultural change in our politics, and to the need for both greater responsiveness and more-robust institutional structure. It could make for a bipartisan project that would not require the involvement of an executive branch beset by its own mind-boggling problems just now. The reasons are plenty. What is lacking is the will.

All of this suggests that the Congressional Budget Office is by no means at the center of Congress’s problems. It is, if anything, the best functioning of the elements of the 1974 budget process at this point. But along with the rest of that process, it needs to change — and the ways in which that is so have perhaps never been clearer than in the messy and complicated health-care debate we are now witnessing.

— Yuval Levin is the editor of National Affairs, a fellow at the Ethics and Public Policy Center, and a contributing editor of National Review.

Comments are closed.



RELATED PUBLICATIONS