Where's the evidence?

Note – this is another policy-related posting which is probably not of much interest to most people!

In conversation the other day, the concept of the “social construction of science” came up, which reminded me, in a slightly tangential way, of a recent article I read about “evidence-based policy”. Evidence-based policy is something that I naturally incline towards – it focuses on asking and answering the question of ‘what works’ in order to develop, implement and resource social policy decisions. So, for example, in my line of work, the question would be ‘what works in terms of getting people from highly disadvantaged backgrounds into sustainable employment?’. Answers to this question at the moment would probably be based on a fair amount of information which is difficult to compare across programs, a healthy amount of values-based rhetoric and a nice dose of personal bias, rather than systematic, comparable, valid and reliable research – something I think needs to change!

However!! Evidence-based policy is one of these things that sounds good in theory but is actually difficult to achieve in practice. Or at least, I think it needs to be recognised that in practice, one person’s evidence base may be another person’s poison (or something like that). The argument is better articulated by the article I read, which summarises,

“There are three main kinds of challenge to the rational mission of ‘evidence-based’ policy. One arises from the inherently political and value-based nature of policy debate and decision-making…Secondly, information is perceived and used in different ways, by actors looking through different ‘lenses’…The third challenge to a rationalist concept of evidence-based policy is that the complex modern arrangements of networks, partnerships and collaborative governance are difficult to harness to the traditional forms of knowledge management, policy development and program evaluation in the public sector…”

Or to put it another way,

“Policy decisions emerge from politics, judgement and debate, rather than being deduced from empirical analysis. Policy debate and analysis involves an interplay between facts, norms and desired actions, in which ‘evidence’ is diverse and contestable."

From Head, B. W. (2008), ‘Three Lenses of Evidence-Based Policy’, Australian Journal of Public Administration Volume 67 Issue 1

So I guess the point is that even if we want to use rational, empirical evidence as the basis for decision-making, it’s difficult to escape the fact that even the question of ‘which evidence should we consider?’ is ridden with value judgements and inherent assumptions. In my original example, the question ‘what works in terms of getting people from highly disadvantaged backgrounds into sustainable employment?’ is, of course, based on an assumption that this is a good thing to do (not everyone would agree). Answering the question then involves another set of assumptions about what employment is, what ‘disadvantaged’ means, timeframes for evaluating ‘what works’ (is it as soon as someone gets a job or is it 2 years down the line?) and so on, and so forth. To some extent, this doesn’t necessarily matter as long as it’s recognised, but where it does matter is in terms of building up a sense of ‘truth’. If the question ‘what works in terms of getting people from highly disadvantaged backgrounds into sustainable employment?’ is seen as not worth answering by those in a position of power and with resources to expend on answering it, then the very question may never become part of any “policy truth”.

This brings me back to the concept of the “social construction of science” and I guess my thought is that any claim to have an absolute truth is invalid because it fails to recognise that as a society, we make choices about which facts we put resources into developing an empirical evidence base about, and which we don’t. And decisions about resources are influenced by politics, personal relationships, corruption, bias, personal values, imperfect information and who knows what else! Scientific truth may be based on rationality and empirical evidence, but the truths we focus on are entirely subjective.

As a slightly related aside, I came across a headline today that Smoking ‘costs NHS billions’. Shocking huh? Those pesky smokers coughing and spluttering away all our hard-earned taxpayer resources. That’s until you read further into the article, where you come across the fact that smokers pay around three times as much in tax as it costs to treat smoking-related diseases on the NHS. It’s a small but important example of how ‘facts’ can mean different things depending on their context. Interesting that the headline wasn’t Smokers single-handedly prop up NHS funding.

Anyway, my points may not be very well-developed as I’m writing this on tea break at work! But hopefully you know what I mean…