Notes on:
Quinlan,K., Male, S., Baillie, C., Stamboulis A,
Fill J, and Jaffer Z. (2013) 'Methodological
challenges in researching threshold concepts: a
comparative analysis of three projects'. Higher
Education 66: 585--601
Dave Harris
The work by Land and Meyer has been very
influential, but it does not seem to have produced
a research methodology or a discourse about
methodology. These projects, undertaken by
the authors, refer to engineering. They all
used interviews with students and academics, but
there were other differences which affected
outcomes, including 'differences in project goals,
researchers' backgrounds, curricular context,
participants' experiences, negotiated or
independent knowledge and degree of
comprehensiveness sought' (585).
[Their definition of a threshold refers to the
metaphor of the portal and transformation].
The notion does engage academics in conversations
about teaching within their disciplines.
Lots of work has been done on economics, computer
science, and biology [refs on 586], designed to
focus teachers time, and help them discuss methods
of teaching, especially with difficult concepts.
A variety of methods have been used, including
interviews, surveys, observations and analyses of
grades and course feedback. There are no
quantitative approaches as yet, and discussions of
methods generally are not usually detailed.
The original notion had eight or potential key
factors, but two are particularly
'nonnegotiable'[according to a personal
communication from Meyer]: epistemological and
ontological transformation, enabling students to
see the world differently. However, these
features are not always well debated, and most
authors assume a need for different selections to
be present, although they are rarely saying why
they selected particular features: different
selections could well lead to different methods.
There is also the question of who defines
threshold concepts, especially for constructed
lists, while positivists simply 'may assume there
is a fundamental truth' about academic
disciplines. They similarly vary in terms of
the importance to be placed on social
context. Phenomenographic researchers focus
on perceptions of students and teachers, starting
with experience not theory, and showing different
ways of experiencing features of the threshold
concept [there is a reference to Bowden and Marton
1998 The university of learning: Beyond
quality and competence in higher education.
London: Kogan Page]. Researchers can
influence the research process themselves, by
choosing which questions to ask them how to follow
up, bringing in their own interests and background
knowledge: some will thematize and code, for
example. Some used grounded theory, concept
mapping and case studies as well as
phenomenography. More explicit
methodological discussion is needed [apparently
also addressed by Kinchin 2011 'Visualising
knowledge structures in biology: Discipline,
curriculum and student understanding'. Journal
of Biological Education 45 (4): 183-189].
Three case studies were chosen, all in engineering
and materials, all focusing on the first year of
study. Methodological differences were
suspected of making a difference to the outcome,
especially choosing the criteria for the focus of
research, and who defines what might be a
threshold. The three cases were discussed
after writing different drafts of case study
methods and discussing them with
participants. The three teams also engaged
in two exercises to identify key
differences. One involved concept mapping
with a comparison of the concept maps. The
second one involved combining data from the
different projects that dealt with a single topic
to show what the different teams made of these
data [there is another article on the results of
the second discussion, online]. Overall, the
project started in Australia and then got some
comparative work at Birmingham and Oxford.
Each project had different methods. The
Australian one used phenomenography, based on the
experience of Baillie. The approach
investigated whether the negotiated knowledge that
emerged did indeed change views of the
world. The point was to not define a
threshold concept but to study the experience of
teachers and students as they meet troublesome
concepts or what teachers see as transformative
ones. Trouble can arise from 'the cultural
context, the teaching, the teacher, the students
background or learning approach' or from
developments in the discipline (589). The
team met some resistance from the engineers asking
for a clear definition of a threshold concept,
however. Both main researchers were trained
in engineering and education, enabling some
specialist discussion of the engineering
concepts. The focus was on transformation
and troublesomeness. There was an initial
diverging phase, where tutors identified threshold
concepts in their own specialist areas of
teaching, then an integrating phase where they
came together to discuss what they might have in
common despite their diverse perspectives.
Particularly valuable were discussions between
students from different disciplines and years;
between students and academics; between academics
from different specialities inside engineering;
between academics in different engineering and
science disciplines; between academics from
different universities. Interviews or focus
groups were held, and workshops for students, and
for both students and staff. There were
audio recordings from interviews, focus groups and
workshops, notes taken by the researchers, and
handouts to be completed by participants.
Data were analysed to identify troublesome and
transformative concepts. Whole transcripts
or notes were analysed rather than segments.
The result was 'an inventory of potential
thresholds' (590) [reported in another paper
-these blokes know how to salami slice!].
Parts of the inventory were re-presented to
participants for further negotiation. [Then
there seems to have been some subsequent recoding
to identify 'overarching threshold concepts of
which previous items were examples, or underlying
threshold concepts on which other thresholds were
built'- these were based on the results of
participant negotiation and also conceptual
comparisons]. Teachers 'pooled' examples
they had had identified. Those that remained
were experienced as threshold by at least one
student, and most of the others had been
experienced as such by many students.
The second case study at Oxford interviewed 14
academics who tutor first year students.
Each tutor had to suggest one major threshold, and
their responses were then further probed thing for
'transformation, integration and troublesomeness'
and how they would know if students had got the
idea. The discussion was about thresholds
rather than threshold concepts as such, since
there had been a problem identifying a concept in
the Australian study. The curriculum is
delivered by a lecture series, with accompanying
problem sets for students. They also have a
tutorial usually in pairs. The idea is to
develop a long-term relation with small cohorts of
students who will tutor them over several
years. Eight students were also interviewed,
and they were asked to identify thresholds which
were then probed in the same way. More
attention was given to learning strategies and
perceptions of the tutorial process.
Students are asked to reflect back on their
experiences. They were also given a prompt
list based on topics tutors had identified.
The interviews were analyzed by reading the
transcripts and then identifying segments of text
which could then be coded. This identified
topics for further discussion, and included
thresholds, transformations, blocks and
barriers. Sub codes were then
developed. All the texts could be used to
find examples of these codes enabling the
comparison of transcripts [lots of references to
grounded theory here]. Concepts identified
as difficult in the interviews were then clustered
into a one page diagram. The team further
experimented with concept mapping [Novak, J (1990)
'Concept mapping: a useful tool for science
education'. Journal of Research in
Science Teaching, 27 (10): 937 – 49] to
illustrate vertical connections between
ideas. A workshop was held to review and
interpret findings, with some outsiders present,
and discussions lead to further data to 'verified,
clarify and elaborate the findings'.
The third case study at the University of
Birmingham was based on action research
approaches, with teachers reflecting on their own
students' learning. There was also an expert
on engineering threshold concepts. Student
and lecturer interviews were undertaken, together
with some focus groups. Both first and
second year students were interviewed, with the
intention of identifying troublesome threshold
concepts - students were asked if they struggled
with particular topics, how they understood them
and their application, how they decided, and what
they did to cope. And lecturers are asked
about how they viewed the module, which topics
they thought students would struggle with, what
their own experience was like, and how they
assessed whether learning outcomes had been
met. The result was an initial list of
troublesome concepts, although these might not
have been 'truly threshold concepts' (592).
Again concept maps were constructed where there
was 'considerable agreement that a concept was
troublesome', and also drawing on the syllabus and
learning outcomes, and textbooks.
Transcripts were reviewed to identify list of
topics and explanations for difficulty, and any
other information. This list was then
'refined' involving project advisers and education
[and engineering] experts 'familiar with threshold
concepts'. It seems that lecturers were not
very helpful because they did not 'reflect deeply
on the learning experience of students', so focus
shifted to perceptions of thresholds by students
and teachers, 'compared with textbook notions of
curriculum content'. The connecting
words associated with topics [provided by
interviewees] were added to the concept maps:
these connections 'describe the relationships
between potential threshold concepts'. This
helped probe presenting difficulties: phase
diagrams were seen as troublesome by many
students, but there are a number of concepts that
are required before they can be understood, and
the concept map helped identify them. The
words they actually used to explain their
difficulties helped understand the perception of
the [expert] relationships and subordinate
concepts - they mentioned the need to visualize
processes, and to use adequate maths, for example.
The goals of the research exercises were
different. The Australian one was intended
to inform curricular reform and build
consensus. The Oxford one was designed to
understand the role of tutorials. At
Birmingham, the aim was to improve the particular
first year modules. Researchers also
differed in terms of their backgrounds and
'epistemological orientations'. Australian
researchers had backgrounds in engineering and
education, and were keen on phenomenography.
At Oxford, the main investigator was an education
researcher interested in constructivism and
qualitative research. At Birmingham, the
principal investigator was a materials science
person interested in practitioner research with no
particular expertise in educational research as
such.
The context of the exercise was different as well,
for example whether it covered large courses or
single modules. Larger courses involve
problems of integrating underlying ideas.
The Oxford tutors working across different parts
of the programme were much more interested in
'more abstract thresholds', and 'problem solving
processes', or connecting maths and the physical
world, which are elements underlying all
engineering. The Australian researchers
found most interest in the integrating phase,
discussing generic engineering and its threshold
concept 'system identification' which has
different embodiments. The Birmingham study
by focusing on one particular unit did permit more
attention to difficulties and relations between
concepts in one specific module.
All three studies used interviews and focus groups
primarily, but interviewees differed: some were
more specialized and isolated and less interested
in integration, and their students were also less
likely to identify common principles. It was
difficult for senior tutors and lecturers to have
remembered their own experiences as
students. Sometimes student meanings did not
relate well to 'the underlying concept'. The
views of participants looking back were often more
useful. The Australian study moved on to
discuss their findings with people all over the
world in different contexts.
Another difference turned on negotiated or
independent knowledge: some interviews were
'conducted iteratively', in a process deliberately
attempting to build consensus, and this produced
negotiated knowledge. Others were
interviewed independently, so that each tutor at
Oxford gave their own opinion, and the researchers
decided whether common themes emerged unprompted:
even here, there may be a particularly strong
common ethos at Oxford. Both approaches can
be justified, but researchers need to be clear
about their research design instead of just
referring to identifying threshold concepts.
Approaches also varied to the extent to which they
'sought comprehensiveness and depth' in technical
understanding (595). At Oxford, more depth
was pursued by asking tutors to focus on only one
or two perceived thresholds, with the intention of
trying to grasp the effectiveness of
tutorials. Curriculum design was not
questioned. As argued above, more abstract
and general issues tended to emerge. At
Birmingham and in Australia, there was a more
exhaustive list, but more interest in the
technical nature of concepts themselves, and the
detail of what produced them as troublesome - as
in difficulties in grasping visual representations
with the phase diagrams. We still need to
know why matters such as graphical representations
might lead to difficulties. Substantial
inventories and then technical concept maps were
used to pursue these issues.
There are factors affecting data.
Researchers need to compare and contrast there
approaches and develop more rigorous
protocol's. There are still problems about
the best methods to use and analyze the data,
especially as interest in the threshold concept
widens to include different schools of
thought. A useful way forward might be to
begin with defining criteria for thresholds, then
thinking of different sorts of research methods to
explore different dimensions: if troublesomeness
is the most important, it makes sense to look at
student experience [rocket science!] [Apparently,
there is some useful American based research on
misconceptions in science, 597. This shows
troublesomeness or difficulty associated with some
concepts, but not their transformative or
integrative qualities].
If we were interested in integration, we might
begin looking for key principles that lie beneath
lots of examples and applications. Here, we
might offer students a variety of examples and ask
them to organize them conceptually, perhaps using
card sorts [Kelly grids!]: Again we would not know
about transformations [we would if we used Kelly
grids to look at laddering]. Each of the
other characteristics might have its own special
methodology. This bloke Flanagan who has
compiled a bibliography of threshold concept
papers has indexed particular words, with the
order of frequency of mention being 'threshold
concept', 'troublesome', 'transform', 'liminality'
and 'integrate'[looks interesting - Flanagan,
M. Smith, J (2006) 'Threshold
concepts: Troublesome topographies for the Google
generation', a paper given at the annual
conference of SRHE, Brighton]. We can see
that there are at least eight different and
distinct research questions.
Theoretical traditions of researchers will also
vary. In social sciences, we find multiple
theories 'to describe largely the same
thing'(597), yet researchers tend to live in
silos. Looking at threshold concepts would
permit wider discussion and enrich research.
If transformation is the essential feature, as
Meyer suggests, this might indicate some future
directions. Transformative learning has
already been discussed in adult learning theories
[I thought of Perry, who comes up in a minute as
an example of a 'student development
theory'. The reference for transformative
theory is Mezirow and Taylor (2009), although blow
me down, it's not mentioned in the
references!]. Student development theory
with stages of epistemological development are
also relevant, and context free, although this can
also be a problem, and it needs to be 'translated
into particular disciplines' (598).
Overall, simply seeking to identify threshold
concepts 'was ultimately unproductive. It is
a form of disciplinary essentialism and
positivism'. Differences including the
focusing on particular dimensions, and attending
to methodological issues are more
important. Better to say, for example, that
we are interested in tutors' perceptions of
integrative concepts that are transformative for
students. The methodological issues
identified are well known, but they need to be
contextualised in this sub field.
back to ed studs page
|
|