Critical Evaluation

Some evaluators think that evaluation is a specific kind of research activity with its own characteristic methods, while others are think that the standard research methods can be pressed into service. The government has issued a splendid guide to evaluative techniques -- the Magenta Book -- which focuses on the techniques but hints at ways in which evaluation might take on a specific implication. For one thing, the term clearly implies that we attempt to estimate the value of something, quite deliberately. In other words, values are firmly on the agenda instead of being treated with some reservation as in conventional research methods. The question then becomes one of asking how values might specifically be researched.

There seemed to be two basic answers included in the Magenta Book. One turns on what might be seen as an economist's response to the issue of values. This urges us to consider the economic value of policy, and, above all, to consider the full range of costs of a particular policy. One cost which is rather specialist is  'opportunity cost'. Briefly, given that there is a limited amount of money, spending it in pursuit of one policy means that we lose the opportunity to develop another one. This poses a dilemma for policy makers. Even if increased sports participation has led to reduced crime rate, for example, can we be sure that the money would not have been even more effective in reducing crime rates if it had been spent on housing or education?

Specialist approaches to sport and leisure have identified even more interesting costs and benefits that are not at all obvious immediately. The benefits of spending on sport development, for example, might take various indirect forms such as making local citizens feel good about themselves at seeing money being spent in their area.  There are quite a few benefits that exceed the market price, so to speak, such as the educational or entertainment value of heritage sites (Powe and Willis 1996). There are several other examples of such 'merit goods', detailed especially in the work on the economics of the provision of the arts in of Evans (1999) or Gilhespy (see his files on this site here and here).

In essence, an economist will attempt to pin down, specify or operationalize these costs and benefits in various ways. In doing so, the same problems arise as face all social science research -- trying to find an accurate, valid and measurable indicator of costs or benefits. Sometimes, this problem can be played back to managers in charge of policy developments. The classic way to do this is to ask for suitable objectives for the policy for targets or performance indicators. Passing the problem back to managers does not necessarily lead to precise technical specifications, however. For example, many people will know already that it is really quite difficult to specify 'smart' objectives for policy, and very tempting to specify well-intentioned but not very precise ones instead.

The other aspect of value is equally problematic. We could ask about the value of policy by specifying our  'core values' in advance, although these face the same problems of specificity and validity faced by objectives. Another obvious option is to turn to participants or customers themselves and ask them for their subjective estimates of the value of the initiative. This leads the Magenta Book authors into support for qualitative research methods with the usual techniques of interviews, focus groups, observations and so on. Of course such subjective data is notoriously difficult to manage, especially if we have some overall evaluation in mind to be placed into a report. What if the dreaded  'one third rule' applies to sports development policy as much as it does to education policy -- whatever is done, one-third of the customers will approve, one third will disapprove, and the remaining third will not have noticed anything unusual. Such results do not seem to suggest clear directions for policy.

Of course, we might wish to use some externally agreed criteria of value, perhaps based in some moral philosophy. We might wish to pursue  'the greatest happiness for the greatest number', for example, borrowing from English Utilitarianism. We might decide that the main issue is to ensure that customers are treated as moral agents in their own right instead of as means -- which I suppose might be a version of the idea that policy  'empowers' people. We might try to apply some Christian notion of the good life to guide policy. No doubt you will already be able to see some problems arising, since all of these positions are controversial, and none of them can be guaranteed to deliver universal assent. Not only do philosophers themselves disagree about the merits of these value positions, but there are strong arguments to suggest that the public are pretty sceptical about all and any value positions these days. This scepticism has been well discussed by  'postmodern' theorists in particular, which is worth remembering when critics try to suggest that postmodern theory is a mere academic game and irrelevance.

To summarise so far, we can be cheerfully pragmatic about the options on offer. We might choose to use some  'critical rationalist' approach towards evaluation, demanding that policy makers specify their intentions and expectations, and then using basic social science techniques to research them, including those from economics. We might choose to use qualitative social research methods to investigate important aspects of perceived value among the customers or participants themselves. Everything falls nicely into place according to purpose.

However, there is one other kind of research that might be considered as well -- academic research. Here, the issue is not to provide evaluations which will be of direct use necessarily to policy makers, but to investigate issues that are academically significant. Interactions between people of different ethnic origins or genders might be of interest to those investigating social stratification itself, for example. If it so happens that sport or leisure organizations are chosen for case-studies, there may be some additional implications for sports or leisure managers, but the research itself will not be centrally focused on those implications.

Such research is particularly useful in investigating things that have not occurred even to the most reflective managers. 'Unintended consequences' would be one classic example. By definition, these cannot be incorporated into any rational policy or plan, but they are often really important. My favourite example turns on education rather than sports and leisure. Becker et al investigated student life at a prominent American university, and rapidly discovered that coping with the assessment scheme was the major concern of students. The assessment scheme had been designed with good educational intentions like providing feedback on performance, but it had become so dominant that students were forced to cope with it using a variety of dubious and sometimes illegal techniques  (such as plagiarism). It took a group of researchers coming in from outside to discover this, to describe it, chart its extent, and try to explain it.

I don't have any equally spectacular examples in sports policy, although one short study by Dunning and Waddington (2003) comes close (see my 'reading guide' here). The study starts from a classically academic observation that there might be some parallel between the pleasures of sporting activity and the pleasures of illegal drug use. We really could not expect managers and policy makers to even think of such a parallel, because they have their own specialisms to worry about. But for Dunning and Waddington, the parallels are worth exploring, partly because it enables us to research the important academic issue of pleasure. Of course, there are some policy implications too, of a rather startling kind. There is no strong division between good people who play sports, and bad people who take drugs, and therefore not much support for the policy that you can persuade young people to substitute sport for drugs. Of course, these policy implications are left rather speculative.

We are therefore left with not two main approaches but three. One quantitative approach attempts to specify costs and benefits. The second more qualitative approach addresses the issue of value. The third academic approach attempts to examine the context for policy and important aspects such as unintended consequences.

References
Evans, G. (1999) 'The economics of the national performing arts - exploiting consumer surplus and willingness-to-pay: a case of cultural policy failure?', Leisure Studies,18(2): 97-118,
Dunning, E and Waddington, I  (2003)  'Sport as a Drug and Drugs in Sport', in International Review for the Sociology of Sport, vol 38 No. 3: 351 - 368.
Powe, N. and Willis, K.  (1996)  'Benefits received by visitors to heritage sites: a case study of Warkworth Castle', in Leisure Studies 15: 259 - 275.

back to menu page