Wednesday, 18 June 2014

[Guest View] Quality of Life Measures: QoLs, QALYs and Quantifying the Quality of Argument.

Michael Loughlin

About ten years ago I spoke at a debate organised by the British Dental Association at the Royal College of Physicians. The motion was ‘This house believes that Quality of Life measures are the way forward for local commissioning of dental services’. Speaking in favour of the motion was a key designer and defender of QoL measures in Dentistry, Prof Tim Newton. The motion was overwhelmingly rejected by the audience (made up mostly of dental practitioners) and I hoped that Prof Newton – concerned, perhaps, by so comprehensively losing the support of his target audience to someone asking ‘simple, naïve questions’ about knowledge and value – would get back to me with answers to some of the fundamental questions I had raised, but so far he hasn’t.

A couple of years before that I had raised serious concerns about the epistemic claims and political role of QoL measures in a book called Ethics, Management and Mythology. I looked at Alan Williams’ Quality-Adjusted Life Year and its role in health rationing, but the questions I raised were not specific to the QALY approach and concerned QoL measures in general. My focus was on the problems with the project of measuring value, to feed such measures into cost-benefit analyses to inform policy making in organisations and governments. Those criticisms reflected earlier published articles about the use of QoL measures in health policy, dating back to the mid-90s, in particular two articles in 1995 in Health Care Analysis called ‘Bioethics and the Mythology of Liberalism’ and ‘Rationing, Barbarity and the Economist’s Perspective’.

My concerns were not, in fact, restricted to QoL measures and reflected the general role of academics in the policy process. We ‘applied’ academics (now in the days of ‘impact’ more than ever) might like to convince ourselves that we are doing something both academically worthy and socially useful: providing insights and information that will improve policies and so improve people’s lives. But are we actually providing a different sort of service to those whom, we hope, will be our corporate and political paymasters, either directly or indirectly (most often, by securing grants –frequently, but not always, funded from the public purse – on the basis of having been commissioned to act in an advisory role to a large private or public organisation)? Are we providing what Michael Powers described as ‘rituals of verification’ in his important book The Audit Society in the late 90s? On this account, in a secular age, academics provide the sort of service to powerful organisations that agents of the church might have supplied in other times. Instead of reassuring the populace that the Creator sanctions the decisions of state, appropriate reassurance is supplied by creating the appearance of some manner of objective or scientific sanction for policies. In economic terms, the product we are being paid for is legitimacy. However honourable their intentions, academics in this role:

“serve to foster the illusion that decisions of state are the result of impartial, objective reasoning ‘informed’ by ‘experts’. They bestow academic credibility on policies and (most importantly) on the whole policy-forming procedure, making it subsequently harder for others – particularly those lacking academic standing – to criticise the policies. They convey a bogus sense of authority, the work of the favoured theorists functioning as another stick in the already impressive armoury of the powerful.” (EMM, p182)

Measuring devices are not the only way to achieve this goal, but they are one important way, particularly if they promise to quantify properties which would appear unquantifiable to most of us, such as ‘value’, ‘societal progress’ and ‘well-being’.

In the years since the publication of the book, I have received no answer to my naïve questions about QoL measures. What does it mean to measure value? How does the fact that many values appear incommensurable affect the meaningfulness of finding common measures? As one management theorist (Andrew Wall in Ethics and the Health Services Manager) put it: “How can anyone choose between more staff for the special care baby unit and additional domiciliary care for the elderly mentally confused; they have nothing in common but are equally deserving.” The whole enterprise of measuring them to find out which provides more ‘value’ for money seems intellectually – if not also morally – offensive.

So how does the information obtained in these measuring exercises constitute appropriate evidence to answer the sort of evaluative questions they are meant to answer – extremely serious questions about which services we should support, and which should be axed in the face of economic scarcity? Typically such exercises involve selecting particular groups of people and questioning them about the relative values they attach to certain outcomes, getting them to say how strongly they agree or disagree with particular claims about how they would value life with a certain condition, or in a particular situation, or asking them to put a numerical rating on the value of life with that condition or in that situation. Then the different answers obtained are subjected to some mathematical process to find an ‘aggregate’ value for the specific condition or situation being evaluated. This information about ‘the value’ of life with that condition or in that situation is then presented to policy makers as an outcome of the investigation. What assumptions does the QoL methodology make about knowledge and value and are those assumptions both sustainable and internally consistent?

While those questions have not been answered (and I’ll return to them in a moment) QoL measures have continued to be used by organisations such as the National Institute for Clinical Excellence to rationalise ‘priority-setting’ decisions – ie, to resolve questions about how to make the most rational use of the limited health budget, which treatments to fund and consequently which groups of patients are going to have to suffer and/ or die. The current political environment provides great opportunities for the designers of new QoL measures to expand into all manner of areas.

Governments across the world have a quite explicit agenda to save the current economic structures by imposing progressively greater ‘austerity’ on the bulk of the population. While people with houses worth more than £3 million may not face economic sanctions, groups including the disabled, health service users, students in higher education, workers in a wide range of public sector jobs, users of public transport, drivers, just about anyone paying into a pension scheme (the reader may fill in the rest of the list at her leisure….) will all find their real incomes reduced, both by direct restrictions on the incomes they earn and benefits they receive, and also by the need to pay for products and services that were once provided free at the point of use. By traditional economic measures, few governments will be able to claim that they are doing very well: as a Marxist analysis might predict, the system will fail in its own terms. While governments are good at changing the ways they measure such things as inflation and unemployment, as they make people poorer in real terms it is perhaps unsurprising that the current regime feels inspired not by any line from Marx, but one taken from an old song associated with the grotesque Liverpool comedian Ken Dodd: when counting success, don’t count money, count happiness. New measures are needed to show that we are, despite being poorer, not as unhappy as one might think[i] – there is a market for new ways of measuring what it means for people to do well, and despite the need for cuts in front line staff in vital service industries, there are jobs for academics who can provide such measures.

This is, of course, a rather pessimistic analysis of the role of QoL measures. To challenge it, a defender of such measures might want to say one of two things. (There may be other ways to challenge it that I haven’t thought of, but these would both be good ways.) S/he might say QoL measures are justified because:

1. They are a true measure of benefit/ ‘the good’/ what is good in life: a genuine realisation of what Bentham envisaged in the hedonic calculus

2. By employing such measures – by urging governments to use them, get academics to draw them up, do the conceptual and empirical work required by this process – we are likely to bring about progressive outcomes, such that the benefits of the exercise will demonstrably outweigh its costs.

The first answer invites the epistemic questions raised above. The second might at first seem to avoid them, as it does not involve any direct claim to provide correct answers to the question of how to measure value. But if QoL measures do not in fact provide valid evidence to answer questions about which policies are the best ones, then it is not clear to me how we can know that by employing such measures we are likely to contribute to social progress. If QoL measures do not genuinely measure anything then it is not clear what their benefits are, so neither is it clear how we can know that their benefits outweigh their costs.

Put another way, if you want to avoid the questions about how precisely the QoL measures actually qualify as measures at all and instead seek to justify them by saying something along the lines of “well, they’re the best we’ve got and they do help us provide better provision” then you simply beg the question, by justifying the QoL evaluations not by recourse to reasoned defence but by recourse to further unsupported evaluation.

In what follows, I’d like to reiterate the key questions and concerns I levelled at the paper by Williams (on ‘Economics, QALYs and Medical Ethics’, first published in 1995 in Health Care Analysis) in the book twelve years ago. My hope is to get an answer this time.

First of all, far from being progressive, QoL measures seem to be conservative in that their remit is set by the current agendas of policy makers. As Williams says, if you don’t want a particular answer, then be careful what questions you ask. So a recent survey commissioned by the council in Stoke asked people which local services they would least object to losing, not whether or not any local services should be cut – and certainly not whether or not national expenditure on armaments, or the pay/jobs of senior bankers, game show hosts, DJs and other economically unproductive persons should be cut and the savings invested in local services, paying people who actually work for a living. Such questions are beyond the remit of those commissioning the survey – should intellectuals be happy to lend credibility to the process of finding a right answer within such arbitrary limits?

So my question was: by finding methods to determine rational solutions within fundamentally irrational structures, do we in fact disguise, ignore or invite acceptance of the arbitrary nature of the structure that delimits the scope of our choices? Shouldn’t we sometimes, as intellectuals, be saying: given this range of choices there is no right answer, so we need instead to invite people to take collective action to challenge the systems that make it impossible to find an acceptable answer? To say we need to work with ‘the world as it is’ is to rule out, a priori, the possibility that ‘we need a new world’:

“There is very little in this world that is absolutely necessary: most of the things we call ‘necessary’ are only necessary given certain conditions. It is both astonishing and revealing that many theorists writing about health seem to be of the opinion that it is not their business to examine critically the conditions which necessitate the problems they discuss. Whether they realise it or not, such theorists assume the role of apologists for and servants to the status quo. Their ‘practical’ work involves helping the whole machine tick over effectively, or, failing that, explaining that the reason why it does not tick over right now is because no-one has yet thought of the theory which will fix it: there is simply no question that the machine may have a structural fault, that it is beyond fixing – however many broken bodies it churns out.” (EMM p161)

Secondly, there are serious logical and methodological questions about the relationship between gathering empirical information about what certain people actually do value, and generating answers to questions about what people should value, or what ‘really is valuable’. In the 1995 paper by Williams cited above, he makes it clear that the quality of life with a particular condition is determined subjectively: I cannot be wrong about the value I place on life with that condition. So in asking his subjects to determine the value of one year of life with a particular condition (asking them to rate that life somewhere between 1 – the value of one year of ‘normal healthy life’ – and 0 – being dead) he was eliciting the empirical data to provide the answer to the question “what is the value of one year of life with that condition?” Williams may have been unconcerned that this approach seems to assume subjectivism about value, dismissing without argument the views of Plato, Aristotle, Kant and all thinkers throughout history who argued that I might sometimes be wrong about what is valuable. He also seems unconcerned about the fact that if you ask different groups then you can predict different answers will typically be given, and there is ‘no right answer’ to the question as to which members of which groups you should invite to be your subjects. So, if you ask people who suffer with a condition to rate the value of life with that condition, you will get different answers than if you ask people who do not suffer with the condition, or people whose work involves treating those who suffer with it. On the question of ‘whose values to count’ Williams says:

“..the QALY concept is extremely accommodating in this respect. In principle it can accept anybody’s views about what is important in health-related quality of life, and anybody’s views about the trade-off between length and quality of life.” (Williams 1995)

Nor, once we have collected the data, is there any determinate answer to the question as to how one ‘aggregates’ it to find ‘the value’ of life with that condition. On the move ‘from individual to group values’ he says: ‘there is nothing in the QALY approach which requires aggregation to be accomplished in any particular way.’ He then describes distinct ways of arriving at the ‘collective’ view, following the soliciting of individual views, making it clear that a variety of different and incompatible ‘collective’ views could purport to be ‘based on’ the individual views expressed.

This was my reaction to that position twelve years ago:

“The theoretical adequacy of the QALY in determining the ‘best’ allocation of resources must surely be questioned, once we realise that it is compatible with many distinct, and mutually incompatible, methods for arriving at practical decisions. The objectivity of this approach is illusory, since there is no determinate answer to the question of how we organise the information acquired empirically; nor can there be any answer to the question of whether the views solicited in any empirical survey represent the ‘right’ answers, for the whole approach implicitly accepts that there are no right answers.

This same flexibility makes it useful to policy makers, since given the right method of aggregation a wide variety of policies could be given ‘justification’ with reference to the QALY. The device is also very ‘accommodating’ to those who want to make ‘priority setting’ seem more rational, since in addition to its bogus ‘objectivity’ it successfully disguises the incommensurablity of many of the values involved in decisions about the allocation of health resources. By insisting that people rank options, we assume that a ranking is intelligible: effectively, we make it a matter of necessity that all values must be commensurable. Thus, the problem of incommensurablity is systematically ignored, and it is assumed that it must always be in principle possible to make a rational choice between two options. Williams’ ‘empirical’ methods make decisions which can only be arbitrary appear non-arbitrary.” (EMM pp173-4)

Thus far nothing I have learned about either this particular QoL measure, or any other, has led me to revise this view. We have in this approach all the outer appearances of a scientific process – the gathering of information and its being subjected to mathematical methods of aggregation – but without any of the substance of genuine science. Suppose I asked an engineer to determine how much weight a bridge could support, and I received the response: “There is no determinate answer to the question of how much weight this or any bridge can support, but so long as you go through a sufficiently complex set of mathematical procedures, a range of answers can be acceptable.” Would anyone feel safe driving over a bridge built by that engineer?

QoL measures in dentistry can produce answers to several decimal places regarding the ‘impact’ of a particular condition and its treatment on matters as diverse as a person’s romantic life and confidence at work, but simply stating a number does not prove one has meaningfully or accurately measured anything. I can meaningfully say: “My sister is a lot happier in her new job than she was six months ago in her old one.” But if you insist on my stating a number to quantify precisely how much happier she is then you do not get me to give an answer that you can feed into some broader assessment. (Of how much happier people are generally than they were six months ago, or how much happier people who change their jobs are generally, or indeed how much well-being is associated with policies explicitly designed to promote greater occupational mobility.) Rather, you get me to add something nonsensical to a claim whose meaning was already as clear as it could be. Unfortunately, getting politicians to talk about well-being does not mean getting them to talk sense about well-being.[ii]

Having thought I had concluded, however, I cannot help but wonder if there might have been, rhetorically speaking, a more effective way to make my point. Maybe I should have foregone the arguments above and, rather, established a set of mathematical procedures which enabled me to ascribe a number to a movement’s relative success in answering questions regarding its rational foundations: a clear defence of one’s claims or thesis, involving judicious and clear appeals to grounds, warrant, backing etc. would equal 1. And a completely unsupported assertion, cast out with no thought as to its truth-value, would be 0. What do we award the proponents of QoLs and QALYs?

As a final note, if you want to pursue this little thought experiment it is good to remember:

1.1 Decimal points give the impression of precision, an impression of which whole numbers can only dream.

1.137 Use a decimal point and be liberal in your use of decimal places.

1110110 Rather felicitously, restricting whole numbers to 1 and 0 also suggests associations with binary coding, which again implies both rigour and a requirement for expertise.

1.2 Bearing this in mind will help you avoid the charge that all you’ve provided is something with all the rigour and intellectual content of one of those Buzzfeed “which Game of Thrones/Star Trek/Postman Pat character are you?” ‘quizzes’.


We all just loved the 2011 Royal Wedding, for instance. Of course the happiness had barely had chance to subside when we also had the Olympics in 2012, shortly followed by a Brit winning Wimbledon the following year. Based on knowledge of such happiness-inducing events, how can we fail to see that 2014 is a bumper year for happiness, with both the World Cup in Brazil and the Commonwealth Games. Surely the stress I experienced in 2012 at the thought of my daughter going into £50,000 worth of debt (to get a degree provided for free when members of the cabinet were students) must have been offset by the sheer joy of watching vans with Coca-Cola emblazoned on their sides riding past my place of work behind a flame carried by a celebrity’s daughter. Something which apparently generated so much happiness in 2012 it just had to be done Commonwealth-style in 2014.

While QoL measures have been the subject of this brief discussion, the concerns about the involvement of academics with the policy process go beyond those employed in constructing this sort of measure. For a fuller discussion please see the original arguments, in chapter 6 of Ethics, Management and Mythology.

No comments :

Post a Comment