|
Ian Gilhespy
This paper presents some of the findings of a research project which examined the evaluation of the achievement of the objectives of cultural organisations. Specifically, the paper presents an analysis of the appropriateness and sensitivity of a range of performance indicators that were designed to measure the achievement of some of the social objectives of cultural organisations. The usefulness and relevance of the indicators is explored in relation to a number of interest groups (stakeholders). The conclusion is that certain indicators may be useful both in monitoring the achievement of objectives at an organisational level and providing evidence for advocacy purposes when reporting to supporting agencies but that they are less useful if attempting to compare amongst organisations or for use as evidence of some of the more intangible values underpinning cultural activities. Introduction Cultural organisations may entertain a series of social objectives alongside commercial objectives as well as other non-commercial objectives such as artistic ones. In Gilhespy (1999 p.41) the contention was developed that the range of objectives available to cultural organisations could be rendered down into a fairly small number and expressed in the form of a matrix. It was further contended that these objectives may be regarded as a series of strategic options available to the management of such organisations. The specific concern of this article is to evaluate methods whereby the achievement of some of the social objectives - namely, Access maximisation and attendance maximisation - may be measured and, perhaps, proven. This means looking at performance indicators although other forms of analysis such as 'willingness to pay' methods from economics (Hansen,1997 p.1) also have relevance. Performance indicators are used in commercial, public and voluntary organisations both to monitor the achievement of organisational objectives and to provide information for advocacy purposes. In this latter function performance indicators may be used as evidence for the efficacy of public or private support. The need for evidence that public money is being spent appropriately and effectively has increased in Britain and elsewhere in the West in the course of the last twenty years and continues into the third millenium. The reasons for this are briefly outlined in the next section. Context of Study In Britain and elsewhere in the West, the adoption of performance indicators by organisations either in the public sector or which gain public support has been fostered by governments and their agencies. In Britain, according to Her Majesty's Treasury (1983,p.3) the objectives of any activity in public life should be identified and defined and that methods should be found whereby the achievement of these objectives could be measured. The process is known as performance measurement. According to Lane (1995, 190) the portion of the resources mobilised by national or national governments expanded dramatically throughout the 1960s and 1970s but has been followed by a period of resurgence of economic liberalism in the 1980s and early 1990s. This resurgence was characterised in the West by a strong commitment to markets and competitive individualism (Hutton 1995, i) and led to a 'more market oriented public leisure service'(Nichols, 1999 p.20) one which gave a prominence to financial rather than social objectives. There is evidence, however, that the political climate in Britain and the United States is changing to one that is more supportive of the social benefits of leisure and cultural programmes. In Britain, the passage of the second National Lottery Act (1998) may lead to a greater emphasis on the community-based activities and to projects that are concerned with social inclusion (Cacchioli,1999). And Chris Smith (Creative Industries Task Force, 1998, p2), the Minister for Culture, Media and Sport has identified access as one of the four key policy themes for his department, the other three being education, excellence and economic value and has recognised the role of the arts in breaking down barriers in society. Nevertheless, as Nichols points out when discussing the social benefits of sport 'while the political climate is more supportive...there is also more emphasis on proving that they work - and it is not easy to do'(1999, p,20). The aim of this article is
to evaluate methods of proof. It reports on the appropriateness and sensitivity
of performance indicators designed to measure the achievement of certain
social objectives. The range of objectives is presented in the form of
a matrix (Table 1). The matrix is a parsimonious one but is one intended
to represent the range of policy objectives available to the
managers of cultural organisations.
The Research Base
Terms of Reference
i) over time( i.e. past performance);I have used two items of terminology from Carter (1991, 13), 'dials' and 'openers'. The term 'dial' refers to an indicator that can in some way be read like a barometer. An 'opener ' is like an alarm, a warning flag or as the discussion concerns the arts, a prompter perhaps. Indicators of the 'opener' type are appropriate to organisations with a role in supporting or monitoring a number of facilities. This could be a local or regional authority, a regional arts board or a national arts body. The analogy of the opener (Carter 1991) was devised to describe the function of performance indicators when being used to compare amongst organisations. A particularly anomalous figure would prompt further investigation. The use or ownership of dials and openers depends on the stakeholder group doing the analysis. Some further terms of reference also need to be established: i) efficiency concerns the best possible relationships between inputs and outputs (Gratton and Taylor 1992) and may be defined as output divided by resources consumed or that efficiency is a measure of unit cost - the cost of inputs divided by the output;Analysis The performance indicators have been compiled into two clusters as follows: i) attendance and target group indicators;The clusters are by no means distinct from one another but do provide a means of mapping the territory. The territory is mapped in more detail as each of the above sections breaks down into three sub-sections: relevance of indicators, interested parties (or stakeholders) for each indicator and the application of the indicators. An attempt was made in this research to develop categories of information that would function at an organisational level as part of a management information system. Attendance and Target
Group Indicators
Six ratios come into this category. Four are concerned with the attendance by a targetted group in relation to overall attendance. The groups are O.A.P.s, (in Britain the term for the over 65s), children, students and the unemployed. A fifth ratio is the number of concession visits (i.e. reduced price) as a whole against annual attendance and a sixth was concerned with subscription systems. Attendance and Target
Group Indicators - Relevance
*T.A. = Total Attendance over One year O.A.P. = Old Age Pensioner ______________________________________ I shall take the first five as a group. Each of these refers to the ability of the organisation to attract certain groups with reduced prices. The rationale for this sort of pricing structure has been referred to as compensation(Coalter, 1990) with the assumption that each group is disenfranchised in terms of finance relative to other sectors of the population. The arts are assumed to have merit good status (Cwi, 1982). These indicators have a number of virtues. They provide a powerful means of measuring access maximisation given that the access to be maximised is for those social groups identified. The indicators are appropriate, clear and easy to compile. Using the criterion of sensitivity, however, some caution is required. Not all those visitors who are of pensionable age or unemployed may take advantage of the concessionary pricing structure. Further to this not all visitors to arts organisations are paying for tickets for particular performances. Gallery spaces tend not to involve a charge or any sort of record of visit. Attendance and Target Group Indicators - Stakeholders In Table 3 the stakeholder groups
which may take an interest in the use of the indicators are ticked. User
groups and artists were not consulted in this research.
T.A. = Total Attendance over One year R.A.B.= Regional Arts Board L.A./R.A.=Local Authority/Regional Authority _____________________________________________ Table 4 illustrates the applications of these indicators as dials and/or openers. Attendance and Target
Group Indicators - Applications
T.A. = Total Attendance over One
year
A number of the stakeholder groups should find this set of indicators useful although for slightly differing reasons. At the facility level the measures may be employed in a bespoke manner according to the specific objectives of any one organisation. The ratios may be used to measure the achievement of objectives either over time (i.e.comparing performance from year to year or month to month) or against pre-set targets. Targets could be set for periods of time, type of artistic activity or particular events and could be changed according to shifting social and artistic priorities of the organisation. The achievement of targets might also be offset against attendance maximisation or revenue maximisation. It is assumed that organisations will seek to maximise attendance and revenue unless they have particular reasons (i.e. social and/or artistic priorities) for not doing so. User groups may find this set of ratios useful. One of the themes of change in the management of public facilities throughout the 1980s and early 1990s has been that public services become more user centred (Stewart and Walsh, 1992) or adopt a 'community recreation'(Haywood, 1995) approach. However, the focus of this research project was the managers of facilities. User groups were not consulted. The public agencies featured in Table 3 may wish to employ these indicators as openers if comparing performance with other organisations. However, the usefulness of these indicators in this capacity is compromised by a range of mediating factors: i) as noted earlier there is no policy in Britain of ensuring that opportunities to enjoy or partake in the arts are equitable. Thus, it is not possible to adopt some sort of benchmark ratio for organisations to achieve;Thus, when looking to make comparisons amongst facilities one can never quite compare like with like. It is, nevertheless, one of the roles of the regional arts board to set its own objectives and attempt to ensure that these objectives are met by the facilities that the board supports. This is the potential role of the 'opener' type of indicator. The indicators detailed in Table 2 would furnish the regional arts board with an insight into the geographical distribution of opportunities taken up by various social groups to partake in the arts. The sixth indicator in the category of attendance/target groups is subscription visits to total attendance. Subscription system are a means of encouraging repeat visits to a centre via reduced prices for a series of events. This measure is concerned with efficiency. It may be used as a dial either over time or against targets. There is no scope for inter-facility comparison as this sort of system is organisation specific. There are other deeper problems with these measures. The practice of such a concessionary pricing structure relates back to the social objective of access. With such a pricing structure the meaning of access becomes one of equity. The attempt is made to spread the benefits of the consumption of the arts across all sections of the population based largely on the assumption that the barriers to consumption are price related. It may, however, be the case that demand for the arts is price inelastic (Peacock, 1994) in which case this form of concession may be misconceived. It is also possible that the targeting of the particular groups in the above tables may be misconceived. The practice of targeting particular social groups is not based on any research that has highlighted those groups as being those that publicly supported cultural organisations fail to reach. What evidence there is of the consumption of the arts seems to indicate that there is a distinctive and enduring class composition to arts audiences (Tyrell, 1987: Evans, 1995). A distinction may be drawn between the objective of access for all and access for new audiences. The former is one that rests on the principle of equity (and in practice is attempted through concessionary prices) in which leisure generally and, more specifically, the consumption of the arts is regarded as some sort of welfare right (Clarke 1994). This meaning of access is well covered by the indicators discussed above. The latter is one that rests on the principle of social efficiency where efficiency is defined as 'the socially optimal outcome' (Towse 1993, 3). Here, the use of subsidy is appropriate if the people experiencing the arts would not have otherwise had the experiences. This meaning of access is less well covered by the above indicators. Ideally, the measurement system should be able to distinguish between those users who have a large amount of 'cultural capital'(Bourdieu 1985) and those users who have little. A monitoring system using the above indicators is not going to achieve this. Instead, there would have to be reliance on figures about new users or figures about the class composition of the audience based on the premise that the distribution of cultural capital is socially uneven. Evans (1999) has examined the social configuration of arts audiences in Britain at a national level from governmental(the General Household Survey) and other sources and concluded that the class composition of arts audiences is enduring. A further option would be to measure the proportion of expenditure committed to outreach work and education in relation to overall expenditure. These measures were not tested in this research. Attendance and Local
and Regional Population Indicators
Attendance and Local and Regional
Population Indicators - Relevance
*T.A. = Total Attendance over One
year
The second measure in this group is a ratio, total attendance in relation to local population as defined by the boundaries of local government. The data for this measure produced a range from .02 to 2.44. This means that one of the centres in the study gained attendance that amounted to 2 per cent of its local population. For another there were 2.44 visits per head of local authority population. This ratio is a very crude one in measuring effectiveness of service to the local population. Attendance may be comprised of visitors to the centre from outside the catchment area of the local authority. As previously stated only attendance by ticket purchase is likely to be recorded. But crucially, the ratio treats any one centre in isolation from the other cultural opportunities available in their respective areas. For these reasons this indicator is an insensitive one. The third and fourth indicators in this section can be discussed as a pair. The data for these indicators came mainly in the form of estimates from managers although some of the figures were derived from market research. The premise for these ratios is that an organisation in receipt of local or regional authority support ought to be serving those populations. The data range for local users as a proportion of total attendance was 65% with a low figure of 30% and a high of 95%. The mean average was 60% with a standard deviation of 22%. This spread of data is not significant in itself as the variation may be explicable either for reasons of situation or for reasons of policy. By reasons of policy I am referring to the interview data in which some managers expressed their agnosticism as to the geographical or social origins of their audience(s) in spite of their funding base. The data range for regional users - as defined by the boundaries of regional government - as a proportion of total attendance was 30% with a low of 70% and a high of 100%. Four of the organisations returned a figure of 100%. The mean average was 91% with a standard deviation of 10%. There is a second interpretation to this set of results. This relates to the economic value of arts activity. Cultural institutions often play a role in tourism and income generation for an area. An economic impact argument can be constructed wherein arts visitors are to be welcomed on the basis of their spending and the resultant local employment (Myerscough, 1988). In this case visitors to a facility from outside of the funding catchment are an economic good. Attendance and Local and Regional Population Indicators - Stakeholders & Applications For this section I shall discuss
stakeholders - as illustrated in Table 6 - and the applications of the
indicators - Table 7 - together.
T.A. = Total Attendance over One
year
T.A. = Total Attendance over One
year
The remaining pair of indicators in this section - local users as a proportion of total attendance and regional users as a proportion of total attendance - are powerful indicators for local and regional government concerned with value for money. They would function as dials and could be operational over time or against targets. To conclude the analysis section, some of the indicators tested are sensitive and appropriate instruments in the measurement of access and attendance maximisation. They are best employed in a bespoke or 'custom-made' manner at a facility level in relation to pre-set targets or over specified time periods (i.e. as dials). They are less sensitive and appropriate if to be used in a comparative manner amongst organisations or for benchmarking purposes (i.e. as openers). Limitations and Future Directions The adoption of performance measurement systems assumes that organisations are rational, goal-seeking and future-oriented. In essence, there is an assumption that organisations behave in a 'strategic' manner (Wilson et al, 1992). However, a number of organisations in the sample had not developed objectives or were unclear as to their policy priorities. Some of the interview data emphasised the need for organic development or community-led approaches which may not lend themselves to a rationalistic model of organisational behaviour. The corollary to this is that if an organisation is unclear about its objectives then the use of public money to support it may not be justifiable. This is particularly the case if the allocation of public resources is to be tied to the achievement of policy objectives. In Britain, the provision of public services by local authorities is conducted within the framework of 'Best Value'. A feature of the emerging Best Value framework is the need for a clear and demonstrable relationship between policy objectives and their achievement. An indication of attendance is no indication of the experience gained during that attendance. This raises a significant problem for the indicators developed and tested in this study. The indicators treat the users of the arts organisations as visitors or, perhaps, as consumers. They are silent on the nature and quality of the experience of the user. The consumption of culture is not a uniform experience. Nor is the consumption of culture a passive experience. Users may be both producers and consumers of the cultural product. The division of role between provider and consumer in a cultural organisation may not be clear cut. The role of the user may be a creative one, involved in a form of negotiation or dialogue with the artist or cultural product. The user may also be involved in a process of dialogue and negotiation with the management team of the arts organisation. Some of the interview data emphasised what may be termed cultural democracy (Bramham 1995). This concerns the development and enhancement of the users abilities to become culturally competent and constitutes the democratisation of cultural power. This leads to the building of confidence and skills in participants and relates the social aims of the arts closely to the aims of education policy and aspects of social policy. This raises a familiar problem for the social sciences in policy evaluation, what Bonet et al refer to as 'the accurate determination of the net result'(1997 p 94). They are referring to the problems encountered in the isolation of outcomes to a particular arts policy or programme relative to other social policies or programmes or, for that matter, activities in the commercial or voluntary sectors. The more long-term or abstruse the outcome the more difficult it may be to identify cause and effect. This may, particularly, be the case if the impacts are broadened to include environmental renewal, community development and the strengthening of cultural life (Matarasso1997 quoted in Kelly 2000). And for some critics the wait may not be worth it anyway. Kelly (2000) has noted the scepticism of leading arts managers in Britain such as John Tusa and Richard Eyre toward performance indicators. For the sceptics the use of performance indicators amounts to a demotion of the arts or a distraction from the artistic activity itself. An emphasis on the social impact of the arts may constitute a form of social engineering. The latter view has been neatly summarised by Bonet et al: '.. evaluation indicators.....implicitly incorporate a certain set of subjective value judgements which unconciously project the cultural prejudices of the evaluator in favour of what they believe to be socially, politically and artistically correct' (p.92).
Of course, an analysis of the positive social externalities of the arts should be contrasted with a study of the negative social externalities. The arts may contribute to a sense of exclusion rather than inclusion, for instance. The german word 'schwellemangst' - roughly translating as 'fear of the threshold' - captures the sense that some people experience when faced with the opportunity to consume the arts. More sinisterly, Pick(1980) has written on the plasticity of the arts and the ability of governments to mobilise cultural forms as forms of propoganda to serve a series of political purposes. His words are salutory but as I have noted before (Gilhespy 1999) the social purposes of the arts belong alongside a series of policy objectives including the artistic goals of innovation and excellence. The indicators discussed in this paper were designed to measure the social objectives of access maximisation and attendance maximisation. They achieve a level of usefulness and relevance. They may provide sensitive and useful information if the access to be maximised is defined in relation to certain social groups. In this study the achievement of access objectives has been provided by the information derived from the uptake of concessionary ticket prices. However, it may be that concessionary systems have more to do with a sense of welfarism in public provision than any analysis of who may or may not be using the centres. The focus is on equity. Whilst they have a degree of purchase for the objectives of access and attendance maximisation they are largely inarticulate on the longer term social impact of arts attendance. However, this may not be the problem
it may appear to be. Prescribing the outcomes of attendance of the arts
may be misplaced and inappropriate. Perhaps, we should not attempt to prejudge
or determine what the outcomes of attending the arts ought to be. If so,
the important matter is to ensure that people have the opportunity to experience
the arts and let them make what they will of their experience. Accepting
such an argument means that access maximisation is the only social objective
that matters. Access in this sense is about attracting and educating
new users to empower people to be able to make choices as to whether the
arts are for them or not.
References
|