How scientific is our mental health policy?

At some point we must confront what we really understand about the science of mental health.

I found an opportunity for a refresher in the “science” of social science in the form of Jim Manzi’s 2012 book Uncontrolled: The Surprising Payoff of Trial-and-Error for Business, Politics and Society.

Manzi has made a pile of money doing statistical research for corporate America. His book talks about the scientific method and how it is used for business, economic and policy decisions. Manzi’s a right-winger whose policy ideas I mostly don’t agree with, but I do agree with his take on the predictive value of most social science research.

Manzi finds most of the research available to social scientists today inherently limited. People get as far as pattern-finding, which is not far enough to count as science. Within systems of “high causal density,” (the term Manzi uses to describe complex systems) the best we can do is apply a nonexperimental paradigm and make declarations about what theories a pattern that we find might support.

This is not enough to be predictive. Too many factors lurk within the data. As Manzi puts it: “Nonexperimental social science currently is not capable of making useful, reliable, and nonobvious predictions for the effects of proposed policy interventions.” Social scientists create models, but Manzi reminds us repeatedly that “the model is never the system.”

The best model we have for clinical work is the Randomized Field Trial, the technique used for evaluating whether medications should reach the market. Manzi notes that most of these studies are conducted over too short a term to be predictive of their overall benefit to people with chronic conditions. The appropriate measurement period for medication used to treat chronic conditions should be the lifetimes of all patients and all controls, and should measure all significant health indicators. In other words, we should aim to measure “holistic” wellness.

Even when an experiment finds a causal connection, we can’t always apply the results to new situations. To get a result that’s reasonably predictive in business settings, Manzi tests for whether causal connections change from neighborhood to neighborhood, from culture to culture, and over time. He runs hundreds of tests. He uses experts to identify new areas for additional testing, then he runs hundreds more tests.

Manzi’s bottom line: For reasonably useful predictive testing in a corporate environment it takes three key components: (1) senior political sponsorship; (2) an independent testing function led by an articulate, politically savvy and analytically inclined leader; and (3) a repeatable process that makes experimentation a part of how the organization makes decisions.

None of these three components have ever been present with respect to mental health and related public policy.