Is cherry-picking disability data at all fruitful?

Photo credit: Ambernectar 13

Disability is very high on the policy agenda. The rising aggregate cost of disability support has led successive governments to tighten up the system of sick leave, replacing “sick notes” with “fit notes”; to introduce a more stringent – and controversial – system of out-sourced capacity assessments; and to replace Disability Living Allowance benefit for working-age people with more limited Personal Independence Payments.

A continuous flow of news stories has highlighted concerns about the quality of services provided to disabled people and the withdrawal of services by local authorities in response to cuts in funding from central government.

Faced with intense media scrutiny, politicians are under almost irresistible pressure to cherry-pick – or even misrepresent – evidence to support their cause, and it would be naïve to expect otherwise.

In this pressure-cooker environment, just how important is it that politicians don’t simply “cherry-pick” data to fit their agenda? Professor Steve Pudney from the Institute for Social and Economic Research explains why it is essential and how he and colleagues are working to provide robust evidence to inform both debate and policy in this area.

Disability policy will always be contentious, and it is crucially important that policy is based on clear, robust evidence on the pattern and scale of care needs in the population and the capacity of individual disabled people to meet those care needs, either with public support or from their own resources. Research on the relationship between disability and access to support is hugely important but difficult to do.

Much of the existing evidence is based on the analysis of data from large-scale nationally representative surveys, in which randomly-sampled people are asked questions about any disabilities they might have arising from long-term health problems, and also their household circumstances, incomes and other resources.

In the UK, the Economic and Social Research Council (ESRC) and Government Departments have wisely invested in survey resources, so that policy-makers have access to data from several different surveys that give this kind of information. But measurement – particularly measurement of disability – is difficult, because of the many forms that disability can take, and the impossibility of going into great detail in the space of a short survey interview.

Measuring disability

Survey designers have responded to this challenge in different ways, so the available surveys include different numbers of questions about disability, and those questions are often phrased in quite different ways. For example, when asking about general mobility, the Family Resources Survey asks only a general question about difficulty with “moving about”. The English Longitudinal Survey of Ageing asks separately whether the interviewee is able to walk 100 yards, to climb a flight of stairs, or to climb several flights. And the British Household Panel Survey asks variously about difficulties with walking for more than 10 minutes, climbing stairs and walking down the road.

Everyone knows that the way you ask a question may have a big effect on the answer you get. For example, a question about the ability to climb stairs may not work very well in revealing the mobility problems of someone who lives in a bungalow, while a less detailed question on the general difficulty of moving about could work well.

People who do research to provide evidence for disability policy have to choose which data source to use. This choice is often dictated by considerations unrelated to the measurement of disability – and sometimes researchers choose a particular dataset simply because it is familiar or convenient to use. And it is very common for policy-makers to build the case for a particular policy by putting together evidence from different sources quite uncritically, without asking whether the different measurement approaches are in conflict.

Mis-using data

There has been widespread criticism in recent weeks of politicians for misusing data or “cherry picking” items of evidence that fit their prejudices without paying attention to the robustness of that evidence or its consistency with other evidence. The existence of multiple data sources embodying different measurement approaches gives scope for cherry-picking behaviour by policy-makers, unless we can show that there are no serious conflicts between the evidence that different measurement approaches give us.

What we have tried to do in recent research is to carry out the same analysis using the three different surveys listed above, to see whether all three sets of independent evidence show the same picture of the relationship between the severity of disability and the receipt of disability benefit. To do this, we focus on a particular population group – the over-65s – where disability is relatively common.

We also look at a particular disability benefit – Attendance Allowance (AA) – which is a universal benefit that can be claimed by anyone over 65 with a sufficiently serious disability, irrespective of their income. For each survey, we use the same statistical approach to distil the disability indicators into an overall measure of disability for each individual, and then relate that measure to the pattern of receipt of AA revealed by the survey.

We were reassured to find that all three surveys give much the same picture of the relationship between need arising from disability and the chances of receiving support through the AA system. That consensus picture (Figure 1) suggests that policy is rather well-targeted, despite the non-means-tested nature of the AA system. Contrary to the anecdotal evidence put forward by some policy-makers in the past, we find no compelling evidence that AA is claimed by many people without any significant disability.

In fact, if there is a targeting problem, it appears to be the reverse of this: many of the people who report severe disability in these surveys do not appear to receive AA at all, despite the fact that AA is intended to provide help for every older person who has care needs resulting from disability.

Figure 1: Proportion of people in receipt of AA by predicted severity of disability


We also find that receipt of AA is much more common among poorer pensioners. This is partly because disability is more prevalent among people with low incomes. But it is also partly due to the greater importance of the benefit payment to people on low incomes – who are consequently more likely to make a benefit claim. Figure 2 shows our predictions of the way that income influences receipt of AA.

For hypothetical 85 year-old severely disabled widows, the chance of getting AA falls from around 40% for the poorest, to 25-30% for higher-income widows. Figure 2 also shows the predictions for 65year-old married men with little or no disability – the chances of being an AA recipient are essentially zero, irrespective of income.

Figure 2: The influence of income on the predicted probabilities of AA receipt by survey for two benchmark cases

Figure 2: The influence of income on the predicted probabilities of AA receipt by survey for two benchmark cases

Faced with intense media scrutiny, politicians are under almost irresistible pressure to cherry-pick – or even misrepresent – evidence to support their cause, and it would be naïve to expect otherwise.

Independent bodies like the UK Statistics Authority play a valuable role in counteracting this tendency, but the first line of defence must be good statistical resources and independent researchers who are willing to question the robustness of their evidence. It is nice when this questioning suggests that the evidence is indeed robust.

Steve Pudney is Professor of Economics at the Institute for Social and Economic Research at the University of Essex. 

Further information

The research team also includes Professor Ruth Hancock and Marcello Morciano of the University of East Anglia and Francesca Zantomio of the University of Venice.

The project is funded by the Nuffield Foundation, Economic and Social Research Council and European Research Council.

Leave a Reply