Critical
Evaluation
Some evaluators think that evaluation is a specific kind of research
activity with its own characteristic methods, while others are think
that the standard research methods can be pressed into service. The
government has issued a splendid guide to evaluative techniques -- the
Magenta Book -- which focuses on the techniques but hints at ways
in
which evaluation might take on a specific implication. For one thing,
the term clearly implies that we attempt to estimate the value of
something, quite deliberately. In other words, values are firmly on the
agenda instead of being treated with some reservation as in
conventional research methods. The question then becomes one of asking
how values might specifically be researched.
There seemed to be two basic answers included in the Magenta Book. One
turns on what might be seen as an economist's response to the issue of
values. This urges us to consider the economic value of policy, and,
above all, to consider the full range of costs of a particular policy.
One cost which is rather specialist is 'opportunity cost'.
Briefly, given that there is a limited amount of money, spending it in
pursuit of one policy means that we lose the opportunity to develop
another one. This poses a dilemma for policy makers. Even if increased
sports participation has led to reduced crime rate, for example, can we
be sure that the money would not have been even more effective in
reducing crime rates if it had been spent on housing or education?
Specialist approaches to sport and leisure have identified even more
interesting costs and benefits that are not at all obvious immediately.
The benefits of spending on sport development, for example, might take
various indirect forms such as making local citizens feel good about
themselves at seeing money being spent in their area. There are
quite a few benefits that exceed the market price, so to speak, such as
the educational or entertainment value of heritage sites (Powe and
Willis 1996). There are several other examples of such 'merit goods',
detailed especially in the work on the economics of the provision of
the
arts in of Evans (1999) or Gilhespy (see his files on this site here and here).
In essence, an economist will attempt to pin down, specify or
operationalize these costs and benefits in various ways. In doing so,
the same problems arise as face all social science research -- trying
to find an accurate, valid and measurable indicator of costs or
benefits. Sometimes, this problem can be played back to managers in
charge of policy developments. The classic way to do this is to ask for
suitable objectives for the policy for targets or performance
indicators. Passing the problem back to managers does not necessarily
lead to precise technical specifications, however. For example, many
people will know already that it is really quite difficult to specify
'smart' objectives for policy, and very tempting to specify
well-intentioned but not very precise ones instead.
The other aspect of value is equally problematic. We could ask about
the value of policy by specifying our 'core values' in advance,
although these face the same problems of specificity and validity faced
by objectives. Another obvious option is to turn to participants or
customers themselves and ask them for their subjective estimates of the
value of the initiative. This leads the Magenta Book authors into
support for qualitative research methods with the usual techniques of
interviews, focus groups, observations and so on. Of course such
subjective data is notoriously difficult to manage, especially if we
have some overall evaluation in mind to be placed into a report. What
if the dreaded 'one third rule' applies to sports development
policy as much as it does to education policy -- whatever is done,
one-third of the customers will approve, one third will disapprove, and
the remaining third will not have noticed anything unusual. Such
results do not seem to suggest clear directions for policy.
Of course, we might wish to use some externally agreed criteria of
value, perhaps based in some moral philosophy. We might wish to
pursue 'the greatest happiness for the greatest number', for
example, borrowing from English Utilitarianism. We might decide that
the main issue is to ensure that customers are treated as moral agents
in their own right instead of as means -- which I suppose might be a
version of the idea that policy should 'empower' people. We might
try to apply some Christian notion of the good life to guide policy. No
doubt you will already be able to see some problems arising, since all
of these positions are controversial, and none of them can be
guaranteed to deliver universal assent. Not only do philosophers
themselves disagree about the merits of these value positions, but
there are strong arguments to suggest that the public are pretty
sceptical about all and any value positions these days. This scepticism
has been well discussed by 'postmodern' theorists in particular
(especially Lyotard),
which is worth remembering when critics try to suggest that postmodern
theory is a mere academic game and irrelevance.
To summarise so far, we can be cheerfully pragmatic about the options
on offer. We might choose to use some 'critical
rationalist' approach towards evaluation, demanding that policy
makers specify their intentions and expectations, and then using basic
social science techniques to research them, including those from
economics. We might choose to use qualitative social research methods
to investigate important aspects of perceived value among the customers
or participants themselves. Everything falls nicely into place
according to purpose.
However, there is one other kind of research that might be considered
as well -- academic research. Here, the issue is not to provide
evaluations which will be of direct use necessarily to policy makers,
but to investigate issues that are academically significant.
Interactions between people of different ethnic origins or genders
might be of interest to those investigating social stratification
itself, for example. If it so happens that sport or leisure
organizations are chosen for case-studies, there may be some additional
implications for sports or leisure managers, but the research itself
will not be centrally focused on those implications.
Such research is particularly useful in investigating things that have
not occurred even to the most reflective managers. 'Unintended
consequences' would be one classic example. By definition, these cannot
be incorporated into any rational policy or plan, but they are often
really important. My favourite example turns on education rather than
sports and leisure. Becker et al investigated student life at a
prominent American university, and rapidly discovered that coping with
the assessment scheme was the major concern of students. The assessment
scheme had been designed with good educational intentions like
providing feedback on performance, but it had become so dominant that
students were forced to cope with it using a variety of dubious and
sometimes illegal techniques (such as plagiarism). It took a
group of researchers coming in from outside to discover this, to
describe it, chart its extent, and try to explain it.
I don't have any equally spectacular examples in sports policy,
although one short study by Dunning and Waddington (2003) comes close
(see my 'reading guide' here).
The study starts from a classically academic observation that there
might be some parallel between the pleasures of sporting activity and
the pleasures of illegal drug use. We really could not expect managers
and policy makers to even think of such a parallel, because they have
their own specialisms to worry about. But for Dunning and Waddington,
the parallels are worth exploring, partly because it enables us to
research the important academic issue of pleasure. Of course, there are
some policy implications too, of a rather startling kind. There is no
strong division between good people who play sports, and bad people who
take drugs, and therefore not much support for the policy that you can
persuade young people to substitute sport for drugs. Of course, these
policy implications are left rather speculative.
We are therefore left with not two main approaches but three. One
quantitative approach attempts to specify costs and benefits. The
second more qualitative approach addresses the issue of value. The
third academic approach attempts to examine the context for policy and
important aspects such as unintended consequences.
References Added
Evans, G. (1999)
'The
economics of the national performing arts - exploiting consumer surplus
and willingness-to-pay: a case of cultural policy failure?', Leisure
Studies,18(2): 97-118,
Dunning,
E and Waddington, I (2003) 'Sport as a Drug and Drugs in
Sport', in International
Review for the Sociology of Sport, vol 38 No. 3: 351 - 368.
Powe,
N. and Willis, K. (1996) 'Benefits received by visitors to
heritage sites: a case study of Warkworth Castle', in Leisure Studies 15: 259 -
275.
back to menu page
|