A History of Thought in Empirical Methods

A history of how economists learned to believe, distrust, and redesign empirical evidence.

Empirical economics has one question, asked over and over for a century: what counts as an identified claim? Every generation has answered it differently, and every answer has been a verdict on the generation before. Theory supplies the identifying restrictions, said the Cowles Commission, and the correlations of the 1920s became clerical work. The restrictions were incredible, said the 1970s, and the great macroeconometric models became elaborate curve-fitting. Find variation that is as good as random, said the 1990s, and two decades of selection-corrected regressions became unreadable. Audit the estimator, not just the design, said the 2010s, and the workhorse difference-in-differences regressions of the credibility revolution turned out to have been averaging causal effects with negative weights. The field’s methodological history is not a staircase of accumulating technique. It is a sequence of admissibility standards, each installed by indicting the published results of the last.

This essay narrates that sequence. Its method is deliberate: every dispute enters through a number. Not a philosophy of causation but a disputed magnitude, the thing two camps computed differently from the same world: whether New Jersey’s minimum wage raised fast-food employment 13 percent or cut it 4.6 percent; whether a job-training program worth less than $900 a year could be made to look like a $15,578 loss; whether each execution deters twenty-nine murders or causes twelve; whether the return to a year of schooling is 6 percent or 10; whether the effect of divorce laws on female suicide is minus three per million women or minus five. A methods fight with no number attached is not a methods fight. It is a seminar about taste.

Prologue: Measurement Without Probability (1870s-1943)

The prologue is the one era in this history without an admissibility standard, and that absence is the era’s content. For roughly sixty years economists computed correlations, fitted curves, and read slopes off scatterplots under a presumption so universal it was never stated: a fitted least-squares relationship between two economic series is, by the fact of having been fitted, an estimate of the economic relationship the analyst had in mind. A downward-sloping price-quantity line was “the demand curve.” A high multiple-correlation coefficient was a confirmed theory. There was no gatekeeping concept, no identification, no probability model, no suspicion that the number might be an artifact of the machinery rather than a measurement of the world. What makes the era worth narrating is that its own participants dismantled this presumption piece by piece, each naming a problem the next century would spend itself on, and none possessing the apparatus to solve it.

The confident opening is Wesley Mitchell’s Business Cycles: The Problem and Its Setting. Faced with an embarrassment of rival cycle theories, Mitchell’s answer was a methodological commitment: define the phenomenon operationally from the record and measure it before theorizing. He decomposed series into trend, seasonal, cyclical, and irregular components, cross-checked them against narrative business annals, and produced a dated chronology: thirteen American cycles between 1878 and 1923, expansions averaging about twenty-four months, contractions eighteen to twenty-one (). This was itself a revision of a disputed number. Crisis-to-crisis theorists, counting only major commercial panics, had put the typical cycle at roughly twice Mitchell’s length; his four-phase definition, which counted recessions as well as crises, halved the received duration. Mitchell intended measurement to discipline theory. Posterity, unfairly, would remember him as the archetype of measurement divorced from it.

In the same year, an agricultural economist who believed in statistical demand curves quietly detonated the whole program. E. J. Working asked what, exactly, the statistical demand curves of Henry Moore and Henry Schultz had estimated. His answer required nothing but the Marshallian cross and a thought experiment about relative variability: the scatter of equilibrium price-quantity points traces the demand curve only if supply is the schedule that shifts while demand stays put; it traces supply in the mirror case; and when both shift, the fitted line is a hybrid resembling neither (). For weather-driven farm crops the implicit condition roughly held, and the crop elasticities of the 1910s and 1920s were defensible. For industrial commodities it did not, and a fitted price-quantity line could slope upward, tracing supply while its author announced a law of demand. Working did not overturn one number; he reclassified an entire literature’s numbers into the credible and the artifactual, according to a condition none of its authors had stated. I will flag now what the rest of this history confirms: Working (1927) is the only paper from this era that survives into the modern canon on its own terms, cited by the credibility revolutionaries seventy years later as the first statement that a price-quantity scatter, by itself, tells you nothing about either curve’s slope.

Ragnar Frisch generalized the anxiety from two curves to all of regression. His 1934 confluence analysis modeled every observed variable as a systematic component plus a disturbance, so that no variable is privileged as dependent, and built the “bunch map” to detect whether a set of variables contains one stable relation, several, or none (). His geometric parable is still the cleanest statement of the problem: a set of points lying in a line does not contain enough information to determine a plane. Ordinary least squares returns a coefficient anyway, confidently, even when the data geometry contains no information to identify it. Frisch coined the word multicollinearity for this condition. The bunch map itself was abandoned within a decade; the diagnosis has never been retired. That pattern, the diagnosis outliving the cure, is the prologue’s signature, and it recurs in every era that follows.

The climax is the collision between the era’s most ambitious construction and its most devastating reader. Jan Tinbergen’s League of Nations model represented the United States economy of 1919 to 1932 as roughly forty-eight equations, estimated one at a time by multiple correlation, then substituted into one another until the system collapsed to a single “final equation” whose characteristic roots classified the cycle (). The roots had modulus below one: the interwar cycle was damped, kept alive by recurrent shocks rather than by endogenous over-determination. And the fitted investment equations drove the interest-elasticity of investment toward zero, corroborated by the 1938 Oxford interviews with businessmen, demoting the interest-rate theories of the cycle at a stroke. These were real numbers with real stakes, and Keynes disputed neither of them. He disputed whether the entire apparatus could mean anything.

Keynes’s review was formally of the first League of Nations volume, the exposition of the method and its application to investment, not of the American model just described; his objections were aimed at the apparatus, and they travel. The review enumerated the preconditions a causal regression must satisfy before its fit licenses any inference: the list of causes must be complete, every factor measurable, the factors mutually independent, the functional form correct, the lags not chosen by fidgeting until something fits, and, deepest of all, the coefficients stable through time on non-homogeneous material (). Interwar macro data satisfied none of these, so the method was “statistical alchemy,” not yet ripe to become a branch of science. The steelman of Tinbergen’s reply deserves its due: most of Keynes’s objections are either absorbed by the method’s flexibility or are problems of all empirical economics, not special defects of correlation analysis, and a theory that implies a large interest effect is fairly refuted by a measured small one (). That last sentence is a genuine theory of testing, and Keynes never answered it. But Keynes won the logical audit, and his own rejoinder found the incoherence at the method’s center: the size of the residual was simultaneously Tinbergen’s detector of omitted factors and his measure of fit, a criterion that cuts both ways. Read today, Keynes’s preconditions are a table of contents for everything that follows: completeness is omitted-variable bias, independence is endogeneity, the fidgeted lag is specification search, and the demand for time-stable coefficients is, almost verbatim, the Lucas critique.

The debate resolved in neither combatant’s favor, which is why it closes the prologue rather than opening a school. Tinbergen won the war of practice: his models are the direct ancestors of everything from Klein-Goldberger to estimated DSGE, and he shared the first Nobel in economics largely for them. Keynes won the argument as staged: pre-probabilistic frequentism could not answer him. What both lacked was the same thing, and every prologue paper points at the identical hole. Working knew the scatter traced demand only under a condition on relative shift variances, but had no stochastic framework in which to state the condition as an assumption. Frisch modeled disturbances but treated identification as data geometry, not probability. Tinbergen estimated a simultaneous system with no theory of what simultaneity did to his coefficients. Keynes demanded completeness and stability with no machinery to test either. The loose thread the prologue hands forward is a missing foundation: there is no probability model of the data, and therefore no way to say what any fitted number is an estimate of. The man who supplied it had been Frisch’s own student, trained on the confluence analysis itself.

Era I: Identification by Theory (1944-1970)

The first era to write down an admissibility standard answered the question like this: an estimate means something when economic theory supplies the identifying restrictions. State it as a claim a reader could contest, because it was contested, immediately and from two directions. The world hands the economist a joint distribution of simultaneously determined variables; theory tells you which structural equation you are looking at inside the tangle; without theory-supplied exclusions and exogeneity assignments, a measured regularity is not knowledge but noise wearing a coefficient. The claim’s soft spot was visible from the start, and it is the era’s loose thread: the restrictions that did the identifying were assumed, never tested. The whole structure stood on credentials that no one audited for twenty-five years.

Trygve Haavelmo’s 1944 monograph is the charter. The prevailing practice, he observed, was incoherent: economists deployed standard errors and multiple-correlation coefficients while treating an explicit probability model as a violation of the very nature of economic data. His resolution was total. No statistical tool has any meaning without reference to a stochastic scheme; the standard objection, that economic time series are not independent draws, dissolves once the whole sample of n observations is treated as a single draw from an n-dimensional joint law; and any hypothetical system of economic relations can be transferred into a statement about that joint law, becoming “a statistical hypothesis in the Neyman-Pearson sense” (). A model, from 1944 onward, is a joint probability distribution over observables plus identifying restrictions. That definition is now so standard its authorship is invisible, which is the highest compliment methodology can earn. But Haavelmo also planted, almost in passing, the seed of everything that would later go wrong. The economist, he wrote, is a passive observer, watching a sampling procedure that Nature carries out. He raised autonomy, the property that a relation worth estimating stays invariant under interventions elsewhere in the system, and then the program he founded proceeded to assume autonomy rather than establish it. One era read the passive-observer passage as the reason a probability model was needed. A later era read the same sentence as the reason observation can never suffice and experiments must be found or built. Both readings are in the text.

Tjalling Koopmans converted the charter into a gate, and pointed it at a rival research program. His 1947 review of Burns and Mitchell’s Measuring Business Cycles names the identification problem as the criterion of admission: because many structural relations hold at once, the mere observation of regularities does not permit us to identify behavior equations among them, and without specifying the form of each structural equation, measurement of it “is not possible, and should therefore not be attempted” (). An unidentified measure is “an instrument of unknown reliability.” Burns and Mitchell were the Kepler stage of economics; but economic data are stochastic and economists cannot experiment, so the Kepler and Newton stages must be pursued simultaneously, and measurement without theory is a road that stops short. Notice what is being fought over. The disputed object is the entire NBER apparatus, reference cycles, phase durations, the finding that structural change over 1854 to 1938 was almost drowned by random variability between cycles. No point estimate was overturned. The fight was over whether the apparatus produced knowledge at all, which is exactly what makes it the founding dispute of this history: the combatants disagreed about the rule, not the result.

Rutledge Vining’s reply deserves a better reputation than a footnote to Koopmans’s stock phrase. His counterattack had three prongs, all of which drew blood (). First, Koopmans mistook a work of discovery for a work of estimation; at the hypothesis-seeking stage, the demand for identification and testing is simply inapplicable. Second, the aggregate economy is a “supraorganism” with behavior not deducible from its constituent particles, so a structural model built from postulated individual optimization is neither the only nor a demonstrated route to population-level fluctuations. Third, and sharpest, the toll gate was rigged: Koopmans’s own probability assumptions were untestable from the same data, yet they were being asked to condition the validity of every result, and requiring a Walrasian structure that had itself never been tested is an arbitrary restriction on research. The closing satire of Koopmans as a “Commissar of Research” was earned. Koopmans won the rhetorical battle completely; “measurement without theory” became a permanent epithet, and identification became the gatekeeping criterion for a generation. But the long verdict belongs substantially to Vining. The structural program Koopmans championed collapsed within thirty years, felled by critics whose atheoretical instincts were closer to Burns-Mitchell than to Cowles, while the NBER dating apparatus still officially calls American recessions. The cruel twist is that Koopmans’s identification theory outlived his structural program and became the backbone of modern causal inference, deployed by the very design-based economists his program would have dismissed.

Lawrence Klein built the standard into a working machine, and produced the era’s one genuine flagship number. His Cowles monograph modeled the interwar United States as a system of simultaneous stochastic equations, identified by a priori exclusion and exogeneity restrictions imposed before estimation, fitted with the Commission’s maximum-likelihood machinery precisely to handle simultaneity bias (). The number came earlier, from the 1945 prototype. The Washington consensus, built on a crude propensity-to-consume function, forecast a postwar return to depression: eight to ten million unemployed, six million by mid-1946 in the confident versions. Klein’s structural system forecast strong inflationary pressure and almost no rise in unemployment. The realized outcome was the boom; a better unemployment answer would have been about two million, and Haavelmo wrote to Frisch in February 1946 that Klein’s forecast was one of few, perhaps the only one, that hit the mark. Honesty requires the caveat the record itself carries: no contemporaneous Klein publication documents the forecast; the evidence is the 1945 Cowles calculations, later recollection, and the Haavelmo-Frisch correspondence. But the episode is the theory-first program’s best case, and it is a strong one: identification by theory, in its first serious outing, beat the reduced-form consensus by a factor of three to five on the most important macroeconomic question of the moment.

Read as a unit, the era is one logical argument rather than four papers, and its internal architecture explains both its power and its blind spot. Haavelmo defines the object: a model is a joint distribution plus identifying restrictions. Koopmans converts the definition into a gate: unidentified measurement should not be attempted. Klein compiles the gate into a machine and runs it. Klein was explicit about the lineage; his stated advance over Tinbergen was to relate the probability structure to the economic structure, a direct rendering of the Haavelmo program, offered as the answer to Keynes’s curve-fitting charge, theory first, probability model built around it (). The standard was stated, weaponized, and applied, in that order, within six years. What the sequence never contained was a step in which the restrictions themselves faced data. Vining had aimed his three unanswered questions at precisely this joint: why must behavior be studied through structural equations at all, are the atheoretical alternatives really unstable, and where is the evidence of structural-equation stability (). The internal critic named the failure mode of the flagship application before the flagship had finished being celebrated, and the program’s answer was silence, because on the era’s own terms the question was inadmissible: the restrictions were theory, and theory was the credential.

It is worth being precise about what Klein’s triumph did and did not certify, because the distinction is where the era ends. His win was a conditional forecast: given the war’s end, what happens to employment? It was not a counterfactual policy evaluation: what happens if the government adopts a different rule? The identifying restrictions had been imposed by theory and never confronted with data; Vining had already asked, and never received an answer to, the question where is the evidence of structural-equation stability. Within the era this looked like pedantry. The models forecast well, the restrictions were “economic theory,” and the enterprise scaled worldwide. The loose thread is exactly the unanswered question: restrictions assumed are restrictions untested, and a generation of policy conclusions was about to be run through parameters that no one had shown to be parameters. The demonstration that they were not would not merely amend the era’s results. It would reclassify them, from structural knowledge to artifacts of a particular policy regime, and the reclassification would arrive with a proof.

Era II: The Con (1970-1986)

The second era never installed a standard of its own, and I refuse to pretend otherwise; its contribution was the discovery that the existing standard certified nothing, delivered five times with increasing scope and no replacement. If the era’s standard must be stated, state it negatively: an estimate is admissible only if it survives the assumptions the analyst did not report. Invariance to the policy rule, credibility of the identifying restrictions, robustness across the specification search, recovery of a known experimental answer: each demand exposes a different hidden conditionality, and each was formulated as an indictment rather than a method. The era is a demolition, and the reason it reads as one continuous argument is that every paper widens the same charge. The reported number is conditional on choices the reader cannot see and the author cannot defend, so the number is not evidence.

Lucas cut first, at the structure itself. His 1976 essay presents itself as a single syllogism aimed at a narrow target, the case for permanent inflation built on econometric simulations of the long-run Phillips curve (). The estimated parameters of a macroeconometric model are agents’ optimal decision rules. Decision rules depend on beliefs about the policy process. Change the policy rule and the parameters shift systematically. Therefore a fixed-coefficient policy simulation is invalid no matter how well the model forecasts, because forecasting success and policy invariance are different properties and demonstrating the first says nothing about the second. The flagship number is the slope of the long-run Phillips curve. The large models, FRB-MIT-Penn and its cousins, traced a stable, exploitable trade-off: permanent inflation buys permanent output. Lucas argued the true long-run slope is zero, and that the estimated positive slope was a sample artifact with no bearing on the consequences of pursuing an inflationary policy. His second number is less famous and more quantitative: the Hall-Jorgenson evaluation of the investment tax credit, which treated the credit as permanent, understates the stimulus of a temporary credit by a factor he computed at roughly 4.5 to 7 depending on the credit’s stochastic regime. Getting expectations wrong is not a refinement; it misstates the policy effect several-fold. Notice the shape of the indictment, because it echoes something older. Keynes’s sixth precondition, that coefficients fitted to one period carry no warrant for another, had been dismissed as a philosopher’s scruple. Lucas turned it into a theorem with a mechanism, and the mechanism was the era’s own rational-expectations theory. The models were not wrong because macro data were dirty. They were wrong because their subjects were listening.

Sims went after identification, and his position is best understood as a third corner rather than an ally of either side. The exclusion and exogeneity restrictions that identified the large models were, he wrote, incredible: normalizations dressed as economic theory, imposed in numbers wildly exceeding what any theory could defend (). But where Lucas prescribed deeper theory, deep parameters invariant to policy, Sims diagnosed that prescription as replacing one incredible object with a more ambitious one, and prescribed less theory instead: the vector autoregression, treating all variables symmetrically, claiming only to forecast and to trace impulse responses. On the era’s flagship money-output question the VAR delivered a real number the theorists could not. Money Granger-causes output, decisively (a chi-square of 64.63 on 24 restrictions for the United States), but money innovations explain only about 15 percent of the one-quarter-ahead forecast variance of real GNP, peaking near 35 percent at three quarters and settling to 28 to 30 percent at longer horizons. The neo-monetarist claim that money is nearly all of the cycle was cut down to a quarter or a third. That number survived as a stylized fact. The methodological promise did not, and the failure is the era in miniature. Sims’s impulse responses required orthogonalization, and his Cholesky ordering of money, output, unemployment, wages, prices, and import prices is a recursive contemporaneous causal restriction, relocated from the equations to the covariance matrix rather than abolished. I place Sims in the demolition era rather than the reconstruction because his positive program failed on its own terms: the escape from identifying assumptions never happened, and the structural-VAR literature his paper spawned exists to defend explicit restrictions, exactly the activity his rhetoric disowned. Nobody escapes identification. The only choice is between restrictions stated and restrictions smuggled. Score the Lucas-Sims exchange across horizons and the split is instructive: Sims won the near term, the VAR became the working tool of macro time series almost immediately, while Lucas won the long term, his demand for policy-invariant deep parameters becoming, through rational expectations and the calibrated models that followed, the theoretical constitution of macroeconomics. Neither victory restored the thing demolished. The large models kept forecasting, respectably, for years; what they lost, permanently, was the claim to answer counterfactual policy questions, which had been the entire point of building them.

Hendry belongs to the demolition too, though he tried hardest to rebuild, and one paragraph does him justice at the price of his program. His 1980 essay stages a con to expose one: he fits UK inflation to a nonsense regressor, shows the fit beats the naive monetarist money-growth model on conventional criteria, and then reveals the regressor is cumulative rainfall (). Conventional fit statistics certify sense and nonsense alike; that demonstration became the classroom emblem of spurious regression. His cure was the golden rule “test, test, and test”: a hierarchy of admissibility filters (data admissibility, exogeneity, parameter constancy, encompassing of rival models) that matured into the LSE methodology. The diagnosis was universally accepted. The cure was bypassed. Applied microeconomics answered the era’s crisis by changing the source of variation, not by testing observational time series harder, and Hendry’s program survives as a regional tradition in time-series macro rather than the field’s standard. He is the era’s dissenting optimist: right about the disease, outvoted on the treatment.

Leamer’s 1983 essay is the era’s most quoted and most misread paper, and it earns the full treatment because its formal core is stronger than its slogan. The slogan, taking the con out of econometrics, indicts the gap between how econometrics was taught (one true model, estimated once) and how it was practiced (dozens of specifications run, one reported). The formal core is sharper (). Model the omitted-variable bias as a draw from a prior with covariance matrix M, the misspecification matrix. M is a pure prior concept: the data contain no information about it, so as sampling uncertainty S shrinks with n, total uncertainty S plus M stays bounded away from zero. More data cannot save you. The reported t-statistic is the inference from a specification search, not the inference the reader thinks it is, and no amount of sample size converts the first into the second. The demonstration is the flagship, and it was chosen for maximum discomfort: capital punishment. Isaac Ehrlich’s 1975 estimate that each execution deters seven to eight murders had been cited before the Supreme Court. Leamer’s own naive regression on 1950 state murder rates produced an even larger deterrent, 13.2 murders per execution with a standard error of 7.2, a seemingly healthy rate of return. Then he slid the specification across five ideological priors, each a defensible view of which controls are doubtful, and reported the extreme bounds: from deterring 28.66 murders per execution under the Eye-for-an-Eye prior to causing 12.37 under the Bleeding Heart. The interval straddles zero with room to spare in both directions. Any inference from these data about the deterrent effect of capital punishment, he concluded, is too fragile to be believed. The deterrence number was not revised. It was dissolved, and the dissolution is the point: the era’s characteristic product is not a better estimate but a voided one.

The reply came from sympathizers, which is what makes it damning. McAleer, Pagan, and Volker accepted Leamer’s diagnosis and then proved his cure was an old tool relabeled: the extreme bounds on a focus coefficient are a monotone function of the F-statistic for jointly deleting the doubtful variables (). Extreme-bounds analysis conveys nothing a standard specification test did not already provide, while inheriting exactly the arbitrariness it was meant to abolish, since the partition of regressors into free and doubtful is itself an unreported prior. On the flagship number their point was second-order but real: the fragility verdict against Ehrlich was a restatement of ordinary specification uncertainty, so the deterrence coefficient was left genuinely undetermined rather than proven fragile. Score the exchange honestly. Leamer won the diagnosis, universally; the profession conceded that specification search invalidates classical inference. McAleer-Pagan-Volker won the mechanics; mechanical sensitivity analysis never became central practice. And the actual cure came from neither: what eventually took the con out was a change in the source of variation, and the paper that announced it borrowed Leamer’s title to say so. An era that devours its own remedies within two years is not building a standard. It is clearing ground.

What Lucas, Sims, and Leamer argued in principle, LaLonde measured, and the measurement is why 1986 is the hinge of this whole history. The National Supported Work demonstration had randomly assigned disadvantaged applicants to a subsidized-employment program, so the experimental answer was known: earnings gains of about $851 per year for AFDC women and $886 for men, in 1982 dollars, with standard errors in the hundreds (). LaLonde threw away the experimental control group and asked what an econometrician armed with the profession’s standard toolkit would have concluded. He rebuilt comparison groups from the PSID and CPS, the survey data the evaluation literature actually used, and ran the estimators the literature actually ran: cross-sectional comparisons, regression adjustment, difference-in-differences on earnings growth, and the Heckman two-step selection correction. The results scatter across a range that should still embarrass: for men, the unadjusted PSID comparison gives minus $15,578, the CPS comparison minus $8,870, and across groups and estimators the estimates run from roughly minus $16,000 to plus $3,000 against a truth under $900. The women’s estimates err in the opposite direction, positive and too large. This is not a story about crude estimators failing; the sophisticated ones failed too. The Heckman correction produced estimates spread over about $1,546 of imprecise values with participation coefficients that were almost never significant, meaning the selection model was essentially unidentified without arbitrary exclusion restrictions.

The deepest finding is the one applied economists still underrate. The literature’s defense against bias was the internal specification test: check that the estimator fits pre-program earnings, check the selection equation’s diagnostics, and trust the estimators that pass. LaLonde showed the tests do not work. A closely matched PSID subgroup that passed the pre-training earnings test still delivered a cross-sectional estimate about $2,100 above the experimental truth, one training effect roughly 3.5 times the other. Estimators that fail the tests can be ruled out; estimators that pass them still miss. So the era’s last escape hatch, the idea that a careful analyst could validate an observational model from the inside, closed with a demonstration rather than an argument. LaLonde himself was modest about all of it, framing the paper as a Leamer-style sensitivity exercise and conceding that few econometricians expect misspecified models to replicate experiments. The profession read it as a verdict anyway, and the misreading was productive: the paper is cited as having helped push the Job Training Partnership Act evaluation toward random assignment, and it became the founding empirical case for everything the next era did.

Tally the era’s ledger, because its shape explains what follows. The long-run Phillips trade-off: voided by theorem. The identifying restrictions of the large models: declared incredible, with the declaration’s own alternative shown to smuggle restrictions back in. The published deterrence coefficients: dissolved into an interval from minus 29 to plus 12. The training-program evaluations: shown capable of missing a sub-$900 truth by five figures in either direction, with the profession’s specification tests unable to flag the failures. And every proposed replacement, deep parameters, atheoretical VARs, extreme bounds, encompassing tests, either demanded the unestimable or collapsed into the thing it replaced. The loose thread is a vacancy, stated precisely by the era’s last paper: if internal tests cannot identify which observational estimators are unreliable, the only credible standard is validation against a known experimental answer. That standard is useless as stated; almost no one has an experiment. But invert it and it becomes a research program: if nothing else recovers the experimental benchmark, go find variation that behaves like the experiment. What could an honest estimate look like? The next era’s answer was not a better model. It was a better accident.

Era III: Identification by Design (1986-2009)

The third era’s standard fits in one sentence, which was the point: an estimate means something when the variation driving it is as good as random and the design can be explained in one paragraph. Both clauses are contestable and both were contested, from inside the movement, within a decade of its founding. “As good as random” demoted the entire apparatus of parametric selection correction, the Heckman two-step, the maintained distributional assumptions, in favor of a physical or institutional accident, a draft lottery, a birthday cutoff, a bordering state, whose exogeneity you argue from how the world assigned it rather than from what you assume about an error term. The one-paragraph clause made transparency an identification requirement: a design is admissible only if a skeptical reader can reconstruct why the variation is clean without following you into a likelihood. The era’s own history supplies the three rebuttals. Exogenous variation that explains almost nothing manufactures answers out of noise, so strength had to join exogeneity. A clean design identifies an effect for somebody, and naming the somebody turned out to be a fight about what economics is for. And a design explicable in one paragraph looks trustworthy in a way its arithmetic may not deserve, which is the loose thread this era hands to the next.

The era’s tools were older than the era, and the chronology of their adoption is itself evidence about how standards change. Donald Rubin had defined the causal effect as a difference of potential outcomes in 1974, before any estimator, recasting randomization as what licenses assumption-free inference rather than what creates an effect (). Rosenbaum and Rubin had made selection-on-observables operational in 1983 through the propensity score, the scalar that balances a high-dimensional covariate vector, exact in theory and hostage to an untestable ignorability assumption in practice (). And regression discontinuity had been invented in 1960, inside educational psychology, by Thistlethwaite and Campbell, who then watched economics ignore it for four decades (). The tools sat dormant because the old standard had no use for them. Only after LaLonde convinced the field that its estimators could not recover a known truth did the profession go looking for designs, and found that psychologists and statisticians had already built several. The era is partly an invention and partly an exhumation, and it was honest enough, eventually, to acknowledge the priority.

The positive template came fast. Angrist used the Vietnam draft lottery, literal randomization by birthdate, as an instrument for military service, and found that service cut white veterans’ earnings by about 15 percent, resolving a literature that had scattered from positive to strongly negative by showing the naive positive correlation was selection (). Card used the Mariel boatlift, a sudden 7 percent expansion of Miami’s labor force in 1980, and found essentially no effect on the wages or unemployment of less-skilled natives; the only visible movement, a rise in Cuban unemployment and a relative wage dip, was mechanically the arrival of the lower-paid Mariels themselves (). Angrist and Krueger scaled the template to Census size: quarter of birth, interacting school-entry cutoffs with compulsory-attendance law, instrumented years of schooling for hundreds of thousands of men and returned a schooling coefficient of roughly 7.5 percent per year, close to OLS, which they read as evidence that ability bias was negligible (). Each paper’s causal claim rests on a paragraph a non-economist can check: lottery numbers were drawn from a drum; the boatlift was Castro’s decision, not Miami’s; nobody chooses their birthday to game the dropout law. That checkability was the revolution.

The Mariel number deserves its dispute, because the dispute exposes the design’s structural weakness a quarter-century before the field formalized it. Borjas reanalyzed the same episode in 2017 and reported wage drops of 10 to 30 percent for non-Hispanic male high-school dropouts, the group most directly in competition with the arrivals. Peri and Yasenov, and Clemens and Hunt, replied that the result lives in a tiny, reweighted, high-variance subsample and vanishes under standard sample definitions (). The weight of reanalysis favors Card’s near-zero, and it remains the field’s working number, but the deeper lesson is about the design: with one treated city, the answer is hostage to which comparison cities and which subgroup you slice, and conventional standard errors measure sampling noise, not counterfactual uncertainty. The single-unit natural experiment was powerful and fragile in the same joint, and the fragility eventually got its own method, of which more in the next era.

The era’s signature fight is the minimum wage, and it should be narrated at full length because everything the movement claimed, and everything its critics feared, is in it. On April 1, 1992, New Jersey raised its minimum wage from $4.25 to $5.05 while Pennsylvania held still. The competitive textbook model predicts an unambiguous employment decline; the pre-1994 consensus put the elasticity such that an 18 to 19 percent rise should have cut fast-food employment by perhaps half a worker to a worker per store. Card and Krueger surveyed 410 fast-food restaurants by phone before and after, and measured the opposite (). Pennsylvania stores fell from 23.33 to 21.17 full-time equivalents; New Jersey stores rose from 20.44 to 21.03; the difference-in-differences is plus 2.76 FTE per store, about 13 percent, with a t-statistic of 2.03. The design carried its own falsification test: New Jersey stores already paying $5.00 or more, unaffected by the law, fell by 2.16 FTE, nearly identical to Pennsylvania’s decline, so the “treated” stores outperformed both control groups. Two subtractions, checkable by a journalist, overturning a generation of aggregate time-series regressions: the paper is the one-paragraph standard incarnate.

Neumark and Wascher reran the identical design with different data and flipped the sign. Using payroll records for about 230 restaurants, collected with the help of the industry-funded Employment Policies Institute, they estimated a 4.6 percent employment decline in New Jersey, significant at 5 percent, an elasticity near minus 0.24 (). Their measurement case was genuinely strong: the employment changes in Card and Krueger’s phone survey had roughly three times the standard deviation of payroll records, with implausibly large store-level swings, the signature of survey error, and error of that size can manufacture a spurious null. Card and Krueger’s reply escalated to the data source that settled the norm if not the case: the BLS ES-202 unemployment-insurance payroll universe, comprehensive and government-administered, independent of both contested samples. It showed a near-zero, if anything slightly positive, effect. The exchange then turned on provenance: once both sides held “payroll” data, the question became which non-random sample to believe, and one of them had been assembled by an intermediary with a stake in the answer. Score it in two registers. On the New Jersey point estimate, contested but near zero; the post-2010 literature’s median own-wage employment elasticity sits around minus 0.04, far closer to Card-Krueger’s null than to minus 0.24. On the war of ideas, Card and Krueger won completely: minimum-wage research permanently moved to border-discontinuity and difference-in-differences designs, the monopsony model returned to empirical respectability, and the 2021 Nobel citation runs straight through this paper. The episode also established, by negative example, a norm nobody had thought to write down: labor-market results should rest on comprehensive administrative universes, not on convenience samples of either kind.

While the minimum-wage fight ran, the movement audited its other founding triumph and demolished it, which is to the movement’s credit and is also the era’s most instructive reversal. Bound, Jaeger, and Baker took the quarter-of-birth design apart (). The instrument was plausibly exogenous; the problem was that it was weak. The first-stage F-statistic on the excluded instruments fell from 13.5 with three quarter-of-birth dummies to 1.6 in the 28-instrument specification, with partial R-squared around one to four hundredths of a percent. Their demonstration is the most elegant demolition in the methods literature: replace quarter of birth with computer-generated random noise, and the second stage returns coefficients of about 0.06 with standard errors essentially identical to the published ones. The output gave no indication that the instruments were randomly generated. The apparent IV-OLS agreement that Angrist and Krueger had read as reassurance was mechanical: under a weak first stage, two-stage least squares is biased toward OLS, and 329,509 observations do not help because the inconsistency ratio, instrument-error correlation over first-stage R-squared, contains no n. Worse, a tiny exclusion violation bites hard when the first stage is tiny: family income differs by season of birth enough, given intergenerational transmission, to account for essentially all of the quarter-of-birth wage association. Staiger and Stock then formalized the pathology with local-to-zero asymptotics, showed that even a stringent first-stage pretest leaves substantial bias, introduced the F-greater-than-ten heuristic almost in passing, and delivered the corrected number: their preferred LIML estimates put the return to schooling at roughly 8 to 10 percent with honest, wider intervals, implying OLS was biased downward by about twice what Angrist and Krueger had concluded (). Hold onto the irony. The design-based movement’s flagship IV paper had the sign of its headline lesson backward, ability bias was not negligible, and the error was caught not by a rival school but by the movement’s own auditing instinct. Strength joined exogeneity as a permanent admissibility bar, and the first-stage F became a statistic no referee lets you omit.

The LaLonde benchmark got the same treatment: rehabilitation, then re-demolition. Dehejia and Wahba claimed propensity-score methods could close the gap (). On their male subsample the experimental truth was $1,794; the naive survey comparisons gave minus $15,205 and minus $8,498; their propensity-score estimates landed at $1,608, $1,691, $1,713, and $1,582, errors of a few hundred dollars, and the mechanism was transparent, discard every comparison unit whose estimated score falls below the treated minimum (1,333 of 2,490 PSID units; 12,611 of 15,992 CPS units) and estimate only on the overlapping subset. For a decade this looked like the observational counterattack: selection-on-observables works when you check common support. Smith and Todd then showed the closure was an artifact of a silently chosen sample (). The Dehejia-Wahba inclusion rule dropped about 40 percent of LaLonde’s observations, differentially excluding high pre-program earners and attenuating the pre-program earnings dip that makes training evaluation hard, which mechanically eased the selection problem. The experimental benchmark itself is sample-dependent, $886 on LaLonde’s full sample, $1,794 on the Dehejia-Wahba subsample, $2,748 on the early-random-assignment sample, and on the other samples the same matching estimators produce biases of several thousand dollars, in some cells beyond $16,000. Simple regression adjustment did about as well as matching on the favorable subsample, so the exercise did not even establish matching’s superiority; only difference-in-differences matching, which nets out time-invariant bias, robustly shrank errors to a few hundred dollars. The technical verdict goes to Smith and Todd. The war of ideas had already gone to LaLonde: the benchmark exercise, twice litigated, permanently privileged randomization, and Imbens would later note it played a role in pushing Congress to mandate experimental evaluations of federal programs ().

The era’s theoretical charter arrived in 1994 and turned out to be a confession as much as a license. Imbens and Angrist asked what IV identifies when treatment effects are heterogeneous, and answered: under exogeneity, exclusion, and one new assumption, monotonicity, IV identifies the local average treatment effect, the average for compliers, the subpopulation whose treatment status the instrument actually moves (). The caveat had been present at the creation; Angrist’s draft-lottery paper already flagged, in its own caveats section, that with heterogeneous effects the lottery recovers the effect for draftees and draft-motivated volunteers, not for true volunteers (), and antecedents of the local-effect idea have been claimed for earlier statisticians as well, so priority here is layered rather than clean. What the 1994 paper did was convert a footnote-grade worry into a theorem and a vocabulary, compliers, always-takers, never-takers, defiers, that relabeled the era’s numbers without renumbering them. The draft-lottery minus 15 percent became the compliers’ penalty. The quarter-of-birth 7.5 percent became the return for men whom compulsory-attendance law actually held in school. Same coefficients, newly narrowed reference population, and a new admissibility rule hiding in plain sight: an IV estimate is admissible only if you can name the subpopulation it describes.

Whether that rule is honesty or surrender is the LATE dispute, the era’s deepest fight, and it should be run at full length through the application both sides claimed: returns to schooling. Heckman and Urzua’s indictment starts from Marschak’s maxim, state the economic question first, then choose the estimator that answers it with minimal assumptions (). Defining the estimand as whatever the probability limit of the IV estimator happens to be inverts that order and smuggles in an interpretation nobody posed. Their calibrated GED example makes the stakes arithmetic rather than philosophical: a policy easing GED certification produces a single IV estimate of minus 0.032, which nets a gain of plus 0.199 to dropouts who switch up into certification against a loss of minus 0.336 to would-be graduates who switch down; 49.2 percent of switchers gain while the reported number is slightly negative. One number, two large offsetting effects, and a policy debate that would be conducted entirely in ignorance of the split. Nor is IV robust in the ways its reputation claims: each instrument defines its own compliers, so “the” return to schooling is not identified, only this instrument’s compliers’ return; and components of a vector instrument used one at a time can generate negative weights, so IV’s sign can be opposite the true effect. Keane pressed the same charge on the era’s founding numbers: a draft-lottery Wald estimate can read minus 20 percent when the true average draftee effect is minus 12, and the celebrated twins-based schooling return of 0.12 is incredible on its face because no plausible life-cycle model reconciles returns that high with the low schooling people actually choose, structural estimates running 0.05 to 0.08 ().

Imbens’s defense concedes the mechanics and contests the values (). Yes, LATE is local, instrument-specific, and not the parameter anyone set out to want; it is also point-identified under assumptions a reader can evaluate, which the population average is not. His clinching example is the bounds comparison: sharp Manski bounds on the population effect of minus one-quarter to plus one-quarter are identical across two data-generating processes whose complier effects are minus one-eighth in one and plus one-eighth in the other, so reporting only what is robustly identified about the population discards exactly the decision-relevant information the LATE recovers. A credible estimate of a narrow thing beats a non-credible estimate of the right thing; naming the subpopulation is not defeat but disclosure. Deaton, prosecuting alongside Heckman in the same 2010 symposium, aimed at the movement’s whole hierarchy of evidence: even a literally random instrument becomes “endogenous” in effect once agents select on idiosyncratic gains, and an internally valid number for a population we did not choose and may not care about is nearly worthless for using causes rather than hunting them (). Score it in the two registers this history keeps. On what IV mechanically identifies under heterogeneity, Heckman and Urzua are simply right, and the negative-weights point is now standard material. On the war of ideas, Imbens and Angrist won a rout: LATE organized the graduate curriculum, complier reasoning became the lingua franca of applied micro, and the 2021 Nobel canonized the position. The external-validity gap between them was not resolved. It was inherited, and the next era’s machine-learning turn, which treats heterogeneity itself as the estimand, is best read as both camps’ unfinished business.

Regression discontinuity’s exhumation completes the era’s toolkit, and the priority is worth stating plainly: economics did not invent its cleanest design; it imported it, forty years late, from educational psychology. Thistlethwaite and Campbell built RD in 1960 to reanalyze the National Merit recognition study and used it against their own earlier matched-group analysis, which had reported effects of public recognition on attitudes toward intellectualism, advanced-degree plans, and scholarship winning. The discontinuity analysis confirmed only the scholarship effects (at p below .025 and .01) and reduced the celebrated attitude effect to zero; of five reported outcomes, the matched-group design had been a false positive on three and understated a fourth (). Matching on background covariates manufactures pseudo-effects through differential regression to the mean; a jump at an assignment cutoff, when everything else trends smoothly through it, cannot. Economics adopted the design only after Hahn, Todd, and van der Klaauw relocated its credibility in 2001, from the deterministic assignment rule, which is not sufficient, to the continuity of all omitted factors at the cutoff, which is the load-bearing assumption and generates the density and covariate-smoothness tests of modern practice (). They also named the estimand (the treatment effect at the cutoff), proved fuzzy RD is a local Wald ratio inheriting the Imbens-Angrist conditions, and showed local linear regression cuts boundary bias by an order of magnitude. The forty-year lag is the era’s chronology lesson in one line: tools do not spread when they are invented; they spread when the reigning standard fails.

The movement’s most ambitious wing stopped waiting for accidents and started manufacturing them. Miguel and Kremer’s Kenyan deworming trial is the flagship: mass school-based treatment raised primary-school participation by about 7.5 percentage points, cut absenteeism by at least a quarter, and generated large spillovers to untreated children within three kilometers, at a cost near $3.50 per additional child-year of schooling (). Banerjee and Duflo built the manifesto on it: no theory could have predicted that deworming buys schooling at nearly twenty times the cost-effectiveness of hiring a teacher, so the parameter had to be measured, and the researcher-implementer partnership can manufacture variation observational data never contain, varying one input at a time, one price at a time (). Deaton’s cross-examination found the soft spot in the flagship itself: the deworming “randomization” was in fact alphabetical assignment of 75 schools to three groups, and with only three possible allocations the reported standard errors cannot say whether the differences arose by chance (). The point was a genuine hit, the core participation result nonetheless largely survived reanalysis, and the movement’s response to the whole critique was absorption rather than retreat: replication across sites, mechanism testing, pre-registration, then the 2019 Nobel. The transportability of the number, as opposed to its internal validity, stayed contested, and the next era made that contest explicit.

One more audit belongs to this era, though its consequences detonated in the next, because it cracked the workhorse everyone trusted most. Bertrand, Duflo, and Mullainathan asked a placebo question: invent state laws that never happened, assign them to random states and years in CPS data on women’s wages, and count how often standard difference-in-differences finds an “effect” significant at 5 percent. The answer was up to 45 percent of the time, and 44 percent even allowing arbitrary state-year error structure (). The mechanism is prosaic: long autocorrelated outcome series, a treatment dummy that switches on once and stays on, and conventional standard errors that ignore the serial correlation, in panels averaging 16.5 periods. The fixes were nearly free, block bootstrap 6.5 percent, clustering at the state level 6.3 percent, collapsing to pre and post 5.3 percent, which makes the two decades of practice they indicted harder to excuse, not easier. Prior practice has a steelman, serial correlation was textbook material and nobody had quantified its interaction with this design, and the paper’s own fixes degrade with few clusters, 8 percent at ten states, 11.5 percent at six, opening a small-cluster literature that ran another decade. But the headline stands uncontested: a large share of the borderline-significant difference-in-differences results of the 1990s, an era in which many published t-statistics hovered near 2, were artifacts of understated standard errors. Clustering became mandatory. “BDM” became a verb.

The era closes at its own victory party, the 2010 symposium in which Angrist and Pischke declared that better research design had taken the con out of econometrics, with Deaton and Heckman prosecuting and Imbens defending. Read closely, all three factions agreed the designs identify something cleanly; they disagreed about whether the something, a compliers-only LATE, a context-bound trial effect, is what policy needs. Internal validity was won. Its price was the question. And beneath the argument sat an unexamined confidence that is the true loose thread: the standard said a design explicable in one paragraph is trustworthy, and difference-in-differences is the most explicable design of all, two groups, two periods, subtract twice. BDM had shown its standard errors were wrong and been absorbed as a repair. Nobody had yet asked whether the point estimate itself, in the staggered-adoption panels the method actually ran on, was computing what the one-paragraph story promised. The workhorse estimators of the credibility revolution went essentially unaudited for twenty years, protected by their own transparency. The audit, when it came, was conducted with the revolution’s own tools, and it found the arithmetic.

Era IV: The Reckoning Inside the Toolkit (2009-present)

The fourth era’s standard raises the bar on its parent: design is necessary but no longer sufficient; the estimator applied to the design must survive an audit of its weights, its inference, and its sensitivity to the assumption that licenses it. This is a contestable claim, and the era’s opening years staged the contest before the audit even began. The credibility revolution had institutionalized itself in a textbook and a manifesto whose confidence now reads as a dare (; ): Angrist and Pischke’s Mostly Harmless Econometrics held that once the identification strategy is credible, common estimators are transparent conduits, quipping in its preface that if the estimates you get are not the estimates you want, the fault lies in the econometrician and not the econometrics (). The estimator was presumed innocent. This era convicted it. The same two-way fixed-effects regressions, event studies, and shift-share instruments the revolution had blessed as designs turned out to be estimators with hidden internal machinery, and under the heterogeneity actual data contain, the machinery sometimes computed averages nobody asked for, with weights that were not even positive.

The first contest, though, came from outside the design tradition, and it deserves to be narrated as a counter-reformation rather than a rearguard action, because the structural camp stated this era’s standard before the design camp turned it on itself. The structural tradition had exhibits, not just complaints. Rust’s 1987 bus-engine model estimated a fully specified dynamic optimization problem for a maintenance manager named Harold Zurcher and recovered an engine replacement cost on the order of $9,000 to $11,000, with the forward-looking model (discount factor fixed at 0.9999) fitting the replacement hazard far better than the myopic alternative; the honest fine print, absorbed later, is that the discount factor itself is not identified from choices alone, so the near-one value is a normalization, not a finding (). Berry, Levinsohn, and Pakes estimated equilibrium demand for automobiles and produced numbers no design can produce: where instrumented logit forced markups into a nearly uniform band, $4,630 on a BMW to $4,805 on a Chevy Cavalier, so that GM absurdly earned the highest margins in the industry, the full random-coefficients model delivered a dispersed, sensible schedule, a 1990 average markup of $3,753, 23.9 percent of price, running from about $801 on a Mazda 323 to $10,975 on a BMW 735, and cut the count of counterfactually inelastic car models from 1,494 of 2,217 to 22 (). When Angrist and Pischke published their victory manifesto in 2010, the reply landed in the same journal issue. Keane argued that all identification is theory-laden, so the real distinction is not few assumptions versus many but assumptions explicit versus implicit, and produced the flagship gap: reduced-form micro designs pin the intertemporal labor-supply elasticity near zero, while structural estimation with human-capital accumulation lifts it to roughly 3.8, dissolving the micro-macro conflict rather than reporting it (). Nevo and Whinston took the fight to the example Angrist and Pischke had chosen themselves, merger evaluation, and demolished the claim that averaging “similar” past mergers dominates simulation: similar mergers are often undefinable across industries, clean controls are contaminated, mergers are endogenous to exactly the unobserved shocks that invalidate the design, and the treatment-effect approach cannot produce the welfare numbers antitrust policy exists to weigh (). Their proposed bar, credible variation and an explicit model licensing extrapolation, is precisely the two-part standard this era ended up enforcing, stated by the losing side of the curriculum war. The design camp won the momentum; the structural camp wrote the era’s constitution.

Then the design tradition opened its most trusted tool and found the problem it had spent two decades not looking for. Priority in the two-way fixed-effects reckoning belongs to Goodman-Bacon, whose decomposition circulated from 2018 and showed exactly what the workhorse regression computes when treatment rolls out at different times: a weighted average of every possible two-by-two difference-in-differences, including comparisons in which an already-treated group serves as the control (). Those “forbidden comparisons” subtract the earlier group’s evolving treatment effect, so dynamics enter with the wrong sign. His flagship reanalysis made the stakes concrete: Stevenson and Wolfers’s estimate of unilateral divorce laws on female suicide reads about minus 3 per million women in the naive single-coefficient regression, but a timing-respecting event study gives roughly minus 5 per million; the workhorse had understated a life-and-death effect by masking its dynamics. De Chaisemartin and D’Haultfoeuille proved the general theorem: under common trends the TWFE coefficient is a weighted sum of cell-level effects whose weights sum to one but can be negative, so the coefficient can be negative while every underlying effect is positive (). Their reanalyses supplied the sign flips this history requires. In Gentzkow, Shapiro, and Sinkinson’s newspapers-and-turnout data, the published first-difference estimate is plus 0.0026, the alternate fixed-effects specification gives minus 0.0011, opposite sign and significantly different (t of 2.86), and the heterogeneity-robust estimator gives plus 0.0043, two-thirds larger than the published number; 40 percent of the fixed-effects weights are negative. The textbook union wage premium of roughly 8 to 10 percent fell to about 4 percent and possibly nothing under the robust estimator. Two decades of staggered difference-in-differences had been reporting numbers whose relation to the underlying effects was, in the presence of heterogeneity, unsigned.

The constructive wave rebuilt the estimator on convex weights, and the rebuild carried its own corollary that killed a second convention. Callaway and Sant’Anna disaggregated the estimand into group-time average effects, cleanly defined and aggregable however the researcher chooses (). Sun and Abraham proved that lead coefficients in the standard event study are contaminated across horizons, so a flat pre-period plot, the profession’s universal parallel-trends check, is not a valid test; in their Dobkin et al. hospitalization application the corrected estimates barely moved, yet the minus-two lead coefficient fell outside the convex hull of the very effects it supposedly averaged, proving the contamination operates even where the bottom line survives (). Borusyak, Jaravel, and Spiess derived the efficient imputation estimator from first principles, rival robust estimators showing simulation standard deviations 1.3 to 3.6 times larger, and applied it to a number with macroeconomic consequences: the Broda-Parker 2008 tax-rebate consumption response, whose implied one-quarter marginal propensity to consume drops from 15.9-to-23.4 percent to 7.8-to-11.4 percent, roughly halved, implying fiscal stimulus is weaker than the calibrated models that cite it assume (). The defenders of the workhorse deserve their steelman: TWFE is transparent, its pathology requires both staggered timing and time-varying effects together, and de Chaisemartin and D’Haultfoeuille themselves reported that fewer than 10 percent of surveyed AER applications used staggered adoption, framing the problem as a minority corner. The literature reversed that caveat within three years, and the reversal is the era’s loose thread showing early: in the administrative-data world of thousands of units treated at unit-specific dates, staggered adoption with heterogeneous dynamics is not the corner case. It is the default.

The shift-share instrument got the same audit with a stranger verdict: two incompatible acquittals, both correct. The Bartik instrument, industry shares times national shocks, had been used for decades on the strength of a vague intuition that national shocks are exogenous to any one place. Goldsmith-Pinkham, Sorkin, and Swift proved the estimator is numerically identical to using the local industry shares as instruments, so the exogeneity condition lives on the shares, differential exposure must be as good as random, and the credibility of any Bartik paper turns on which industries actually drive the estimate, made visible by Rotemberg weights (). Applied to the canonical labor-supply elasticity, the audit fails the flagship: Bartik two-stage least squares roughly doubles the OLS inverse elasticity, 0.63 to 1.22 with controls and 0.71 to 1.65 without, but the overidentification test on the share instruments rejects catastrophically (a J-statistic of 1,014.05), alternative many-instrument estimators scatter from 0.74 to 2.85, and over 40 percent of the estimator’s weight sits on five industries with oil and gas extraction the largest single share. The doubling, as usually run, is not credibly identified. Borusyak, Hull, and Jaravel proved the same estimator is also numerically equal to a shock-level regression, so identification can instead come from many quasi-random shocks, with the decisive corollary that the true unit of inference is the shock, not the region (). Their reanalysis of Autor, Dorn, and Hanson’s China shock left the headline coefficient of minus 0.596 intact but re-derived its standard error at the industry level, 0.114 with a first-stage F of 39.7 and six of seven balance tests passing, converting a possibly mis-clustered estimate into a defended one. Neither side won the identification question, because both equivalences are theorems; the profession adopted the pair as a menu a researcher must choose from and defend. The lasting rule is this era’s standard in miniature: exogeneity of the raw ingredients is not enough; you must certify which decomposition licenses your number and cluster accordingly.

Even the era’s constructive successes took the form of audits. Synthetic control, introduced by Abadie, Diamond, and Hainmueller on California’s Proposition 99 tobacco program, replaced the hand-picked comparison group with a transparent convex combination of donor states and replaced model-based standard errors with permutation inference; it revised the estimated cigarette-sales reduction from the prior least-squares benchmark of about 14 packs per capita per year to roughly 24 on average and 26 by 2000, nearly doubling the effect, because the old control group had included states that later passed their own tobacco laws (). Rare among this era’s stories, the method was promoted rather than deflated, and it disciplined the single-treated-unit design whose fragility the Mariel dispute had exposed. But the self-correction came anyway: Abadie’s 2021 survey walked back the original suggestion that bias vanishes as the pre-treatment window grows, to the honest version, only if pre-treatment fit is good, a method that fails silently when applied mechanically (). And Rambachan and Roth dismantled the last eyeball convention standing: since pre-trends tests are underpowered and conditioning on passing them biases the surviving estimates, they replaced “the pre-trends look flat” with partial identification under an explicit bound on how much post-treatment violations can exceed pre-treatment ones, reported as a breakdown value (). Their applications show why the convention had to die. The French restaurant VAT cut’s effect on profits survives handsomely, a robust confidence set of 0.07 to 0.31 with breakdown only when confounds could be twice the largest pre-treatment violation. The teacher-bargaining-law effect on women’s employment flips sign under even minimal smoothness relaxation, because the point estimate sits above a linear extrapolation of its own negative pre-trend. Two published conclusions, one robust and one inverted, distinguishable only because someone replaced a glance with a bound.

The audit then reached past any single estimator to the published record as a population, and found the selection. Brodeur and coauthors hand-collected 50,078 hypothesis tests from 641 articles in the AER, JPE, and QJE between 2005 and 2011 and plotted the distribution of test statistics: a camel shape, with a valley around z of 1.5 and a bump between 2 and 4 (). Under the maintained assumption that journal selection is weakly increasing in significance, selection alone produces monotone distortion; it cannot carve a valley. The missing mass just below significance, reappearing just above it, implies 10 to 20 percent of tests with p-values between 0.05 and 0.0001 are misallocated, inflated there by the authors’ own choices among specifications, and the valley sits exactly where the marginal return to inflation is highest. Ioannidis had supplied the underlying arithmetic a decade earlier: with realistic prior odds, power, bias, and competing teams, the positive predictive value of a lone significant finding falls below one-half for most designs, around 85 percent for a well-powered trial, 20 percent for a typical exploratory epidemiological claim, a tenth of a percent for a genome-scale search (). The skeptics have real points on both magnitudes, the camel can be partly manufactured by aggregating tests of different power and by rounding, and empirical false-discovery estimates in top medical journals run nearer 14 percent than 50, so “most published findings are false” is a proof of concept, not a measured rate. But the doctrine held, pre-registration became the institutional response, and the follow-up finding folds this dispute back into the others: p-hacking concentrates in IV and difference-in-differences, the era’s most audited tools. Some methods are more hackable than others, which means the census of estimates and the audit of estimators were the same investigation approaching from opposite ends.

Deaton and Cartwright closed the era by auditing the tool everyone had exempted. Randomization, they argued, balances covariates only in expectation, not in your trial; it buys unbiasedness, not precision, and a biased but precise estimator is often closer to the truth (). Their quantitative exhibit: with skewed treatment effects constructed to have a true average of zero, the standard t-test rejects the true null 11.2 percent of the time at fifty subjects per arm and still 6.06 percent at five hundred, and the Bahadur-Savage result guarantees no sample size escapes it. Their transportability exhibit returned to the deworming flagship: a Cochrane review spanning one Indian trial of over a million participants plus 44 trials covering 67,672 found substantial evidence of no benefit for nutrition, cognition, or school performance, directly contradicting the celebrated Kenyan attendance effect, while conditional-cash-transfer estimates of the “same” parameter span minus 8 to plus 38 percentage points across 29 studies, so there may be no single transportable number to discover. The RCT movement absorbed this critique as it had absorbed Deaton’s first, and collected the Nobel a year later. But the meta-point is the era’s thesis in one sentence: the randomized trial was received as the method that took the con out of econometrics, and Deaton and Cartwright showed the con had simply migrated, from the estimation to the unexamined leap of using the estimate somewhere else. Every audit in this era found the same thing. The defect was never where the certifying design was looking.

The era ends with its standard enforced and its subject escaping. The audits certified the estimators just as the data outgrew them: administrative panels with millions of units, treatments adopted at unit-specific dates, effects heterogeneous across thousands of cohorts, exactly the conditions under which “the” treatment effect stops being a number and becomes a distribution. Callaway and Sant’Anna’s disaggregated group-time matrix and Keane’s insistence that policy needs margins and distributions rather than a single local average are the same demand from opposite camps. And the census of published estimates left a question no estimator audit can answer: if the tools are now certified, what certifies the analyst’s path through the specification space? The loose thread hands forward a double problem, heterogeneity as the estimand and the researcher as the residual con, and the next era’s answer to both was to hire machines.

Era V: Machines and Administrative Data (2015-present)

The fifth era’s standard is still forming, and stating it honestly means stating it as a split: prediction and causation are two separate tasks with two separate admissibility bars, and a credible empirical claim keeps them apart on purpose. A prediction is admissible when it performs on a held-out sample behind a firewall; no assumption about the data-generating process is required. A causal parameter is admissible only when it carries a valid sampling distribution, which regularized machine-learning output categorically does not carry on its own. Machines are assigned the nuisance; research designs keep the identification. Every clause of that settlement is contestable, and the deepest contest, whether the era is a better toolkit for the credibility revolution’s question or a return to measurement-first economics with better hardware, is one I will state and refuse to resolve, because the era itself has not.

The founding provocation was imported. Breiman’s “two cultures” essay divided statistics into a data-modeling culture, which posits a stochastic model and certifies it by fit, and an algorithmic culture, which treats nature as a black box and certifies by held-out prediction (). His load-bearing exhibit is small and brutal: on a hepatitis survival dataset of 155 patients, logistic regression predicts with 17.4 percent error, a random forest with 12.3 percent, and the logistic model restricted to its “most important” selected covariates, the one the data-modeling machinery certified, errs at 22.9 percent, worse than the null rule of declaring everyone a survivor, which errs at 20.6 percent. A model that passes goodness-of-fit can predict worse than a constant. The steelman of the holdout position comes from Breiman’s own argument: the Rashomon effect, many models with equal accuracy, implies simple interpretable models usually sit inside the accurate set, so the manifesto half-refutes its own demand that black boxes be accepted. Economics took the currency, held-out accuracy, and refused the epistemics, and that selective import is the era’s whole shape.

Mullainathan and Spiess domesticated the import with a compartment label: machine learning belongs in the part of the toolbox marked y-hat, not the familiar beta-hat compartment (). Their teaching benchmark carries both lessons at once. Predicting log house values, a random forest beats OLS out of sample, an R-squared of 45.5 versus 41.7 percent, while scoring 85.1 percent in sample, which is the argument for the firewall in a single row of a table. And re-running LASSO on ten random partitions of the same data reshuffles which variables are selected while predictive accuracy holds steady, which is the argument against ever reading a machine’s selected variables as “the important ones.” Prediction is cheap to certify. Structure is not, and the machine’s coefficients are not structure.

Whether the y-hat compartment is a wall or a staging area was the era’s live dispute, and both sides had a real case. The wall position rests on impossibility results: post-selection inference fails without near-orthogonality conditions the data visibly violate, so a machine’s output can never be read structurally, and the compartments must stay sealed. The bridge position replies that the LASSO instability is exactly the problem orthogonalization was built to survive: you do not need the machine’s selected variables to be stable, you need the target parameter to be insensitive to which nuisance representation the machine happened to fit, and Mullainathan and Spiess themselves had listed prediction-inside-estimation as a legitimate category (). The bridge won, but on the wall’s terms: the crossing is licensed only by machinery that makes the causal parameter provably indifferent to the machine.

The bridge back to beta-hat is the era’s central construction, and its signature number should be quoted whole. Chernozhukov and coauthors’ double/debiased machine learning rests on two devices: Neyman orthogonality, a moment condition first-order insensitive to errors in the nuisance functions, and cross-fitting, estimating the nuisances on one fold and evaluating on another (). Together they let any learner that estimates nuisances at a modest rate deliver root-n normal inference on the target parameter. The demonstration is the effect of 401(k) eligibility on net financial assets. Raw, unadjusted: $19,559, confounded by the fact that higher earners both save more and are likelier to be offered a plan. After DML controls flexibly for confounders: roughly $6,300 to $8,100, less than half, and, this is the exhibit, stable across radically different machines, LASSO $6,331, regression tree $7,581, random forest $7,966, boosting $7,826, neural net $7,805. Five algorithms that share nothing agree on the answer, which is the evidence that the identification lives in the design, not in the learner. Wager and Athey built the heterogeneity lane: causal forests made valid by “honesty,” never using the same observation’s outcome both to place splits and to estimate effects, holding mean-squared error at 0.02 as dimensions grow from 2 to 30 while nearest-neighbor matching degrades an order of magnitude and its confidence-interval coverage collapses toward one-half (). Athey and Imbens legislated the settlement into the curriculum: state the goal first, predictive or causal, and choose the admissibility bar deliberately rather than by disciplinary reflex (). Honesty and cross-fitting are one idea under two names, a firewall between where the machine learned and where the claim is evaluated, and the fine print is real: the guarantees are asymptotic, rate-conditioned, and overlap-conditioned, so “any learner works” is marketing for “any learner that hits the rate with adequate overlap.” The truce holds where its conditions hold.

Running beside the causal-machine program is something older wearing new hardware, and it is the reason the era’s identity is unsettled. Chetty and coauthors took population tax records on more than 40 million children and turned intergenerational mobility from a scalar into a geography: a national rank-rank slope of 0.341, a 10-percentile rise in parent rank mapping to a 3.41-percentile rise for the child, concealing a nearly threefold spread across cities, the probability that a bottom-quintile child reaches the top quintile running from 4.4 percent in Charlotte through 10.8 in Salt Lake City to 12.9 in San Jose (). The methodological posture is the striking part: the paper published the correlations and explicitly refused to certify them as causal, deferring identification to later designs, which duly arrived in the movers research. Measure precisely first; identify separately, later, if you can. Satellites made the same move for places where official statistics are missing or corrupt: night-lights luminosity entered as a GDP proxy on the strength of a reduced-form elasticity of 0.28 to 0.32 and a structural elasticity of 1.0 to 1.7, and later subnational work showed how the relationship can weaken badly at the grid-cell level where sensors saturate and bloom, vindicating the original survey’s own warnings faster than the profession absorbed them (). Text became measurement under an explicit validation bar: a text-derived index is admissible only after audit against human-coded ground truth, the Economic Policy Uncertainty index validated against a 12,000-article human reading on one side, and Google Flu Trends, the 45-term nowcaster whose accuracy deteriorated until simple extrapolation from official data beat it, as the cautionary exhibit on the other (). The common discipline across both wings of the era is the same sentence: a machine’s fit on its own training data certifies nothing; certify against something the machine cannot see.

So the era’s open question, stated plainly and left open. On one reading, nothing fundamental has changed: DML, causal forests, and honest splitting exist to serve identification, the machines are hired hands, the design is still the employer, and the credibility revolution’s question, where is the as-good-as-random variation, remains the only question. On the other reading, the era’s most consequential products are not identified effects at all but measured objects, a mobility atlas, a luminosity panel, an uncertainty index, and the field is walking back into Mitchell’s 1927 program, define the phenomenon operationally from the record and measure it before theorizing, with 40 million tax records where the business annals used to be. If that is what is happening, then Koopmans’s 1947 complaint is not a historical curiosity but a pending docket item: an observed regularity can reflect several structural equations at once, and unidentified measurement is an instrument of unknown reliability. The night-lights elasticity that holds at the national level and fails at the grid cell is exactly such an instrument. The era has built a division of labor that presupposes an answer to the Cowles-versus-NBER question without stating one. That is the loose thread now hanging: whether the machines are better tools for the old standard, or the old debate reopening with better sensors, Koopmans’s complaint still pending.

Epilogue: The Spiral

One move, made five times, is the whole of this history. An era installs a standard of admissibility. Work accumulates under it. A successor standard arrives, and its first act is retrospective: it does not merely open a new frontier, it indicts the existing stock. The Cowles Commission’s identification requirement reclassified sixty years of fitted curves as measurement of unknown reliability. The Lucas critique and Sims’s “incredible restrictions” reclassified the identified macromodels, the proudest construction of the probability approach, as policy-dependent curve-fitting; Leamer dissolved the deterrence literature into an interval spanning both signs; LaLonde showed the evaluation literature could miss a sub-$900 truth by sixteen thousand dollars. The credibility revolution then built its designs on the rubble, and the reckoning that followed reclassified the revolution’s own workhorse output: staggered difference-in-differences estimates that averaged effects with negative weights, Bartik elasticities that failed their own overidentification tests, pre-trend plots that tested nothing, a published distribution of t-statistics with a carved-out valley just below significance. Each generation’s certificates were revoked by the next generation’s auditors, and the auditors were usually the same people, or their students.

This is not how the field’s theory behaves, and the asymmetry is worth stating exactly. Arrow-Debreu did not indict Marshall; it generalized him. A theorem, once proved, stays proved, and theoretical economics accumulates roughly the way its practitioners believe it does, by extension and refinement. Empirical economics does not. Its knowledge is conditional on an admissibility standard that is itself a historical variable, and when the standard moves, published results do not gracefully depreciate; they get re-audited, and some get condemned. The honest description of the field’s empirical stock at any moment is not “what we have learned” but “what has survived the audits conducted so far,” a phrase that carries its own warning about the audits not yet conducted. In 2008, a staggered difference-in-differences estimate in a top journal was knowledge. By 2021 it was a candidate for Bacon decomposition, and nobody could say in advance which side of the ledger it would land on.

It would be a misreading, though, to hear this as skepticism, and the reason is in the numbers this essay has followed. The audits are not corrosive relativism; they are the mechanism by which the field’s answers actually improve. The quarter-of-birth episode is the pattern at its best: a celebrated design, a demolition by random instruments, and then a corrected number, the return to schooling moving from an artifactual 6 percent to a defended 8 to 10, with instrument strength installed permanently as a precondition. The minimum-wage war ended with a better data norm and a professional prior near zero that two more decades of work has largely sustained. The TWFE reckoning did not conclude that divorce laws have no effect on female suicide; it concluded the effect was minus five per million, not minus three, and the correction ran through arithmetic anyone can verify. Even LaLonde, the most destructive paper in this history, is constructive in exactly this sense: it replaced a false confidence with a benchmark, and the benchmark disciplined everything after it. The pattern is demolition, then a rebuilt number, then a new admissibility bar that makes the old error harder to repeat. What never happens is the thing the staircase picture promises: a standard that stays put.

So the honest geometry is a spiral. The field circles the same questions, what does a price-quantity scatter identify, when does measurement without a model license inference, whose treatment effect does this design recover, and each pass is at a higher altitude: more data, sharper theorems about what the estimators compute, better-instrumented humility about the analyst. Working’s 1927 question about the demand curve is Goldsmith-Pinkham, Sorkin, and Swift’s question about the Bartik instrument, which variation is doing the identifying, asked ninety years apart with the same logical structure and vastly better machinery. Koopmans versus Vining is the pending question about the administrative-data atlases. Keynes’s sixth precondition is the Lucas critique is the transportability problem in development trials. The questions repeat because they are the field’s real subject; the altitude rises because the audits, however humiliating, leave tools behind.

Every substantive field of economics contains local chapters of this story, its own celebrated numbers installed under one standard and re-tried under the next, and readers will recognize their own literature’s version without my drawing the map. The general lesson is the one this essay opened with. The history of empirical economics is not the history of what economists found. It is the history of what they were willing to count as finding, and that second history explains the first. A field that periodically revokes its own certificates is doing something uncomfortable and, I think, unusually honest: it has decided that the standard of proof is itself an empirical question, revisable on evidence, and it has accepted the price, which is that no generation’s results are safe from the next generation’s definition of an identified claim. The spiral is not a failure to build the staircase. It is what intellectual honesty looks like when the subject is a moving world measured by interested parties with imperfect tools: the same questions, again, from higher up.