Evaluating Taught Doctoral Modules
I got the chance to draft the student evaluation questionnaire for the taught modules in the OxICFM CDT. I thought it would be interesting to discuss how I thought about the process.
There’s a lot of literature out there critiquing student evaluations of teaching (SETs). Some of this focuses on how SETs are used invalidly within Universities (e.g. promotion, sanction), and some focuses on the way that teaching is just very hard to measure.
My broad reading of the literature is that students are experts in their own experience of learning, but inexpert at directly judging teaching. Inviting students to judge teaching is normally a formally-invalid piece of survey design (however reliable its results might be), and introduces scope for students’ unconscious bias (e.g. on gender and race, but also on expectations about teaching) to enter the process.
Note: I am very open to suggestions about how to improve the instrument, if you have any, or to discussing if you’re thinking of adapting these for your own setting; I find these conversations stimulating.
The Questions
The module content was interesting. [Agree/Neutral/Disagree]
The module content was relevant. [Agree/Neutral/Disagree]
The module developed my professional skills. [Agree/Neutral/Disagree]
The teaching sessions were well-organised. [Agree/Neutral/Disagree]
The overall teaching delivery was good [Agree/Neutral/Disagree]
The teaching activities were useful for my learning [Agree/Neutral/Disagree]
The assessments helped me to develop as a scientist [Agree/Neutral/Disagree]
Feedback in this module enhanced my learning [Agree/Neutral/Disagree]
This module challenged me. [Agree/Neutral/Disagree]
Because of this module, I feel more ready to articulate and solve challenging problems in chemical synthesis. [Agree/Neutral/Disagree]
Because of this module, I feel more ready to articulate and solve challenging problems in chemical manufacturing. [Agree/Neutral/Disagree]
What should we stop doing in this module? (please note that unprofessional comments will be censored) [Free text]
What should we start doing in this module? (please note that unprofessional comments will be censored) [Free text]
What should we keep doing in this module? (please note that unprofessional comments will be censored) [Free text]
Broad Design Principles
What does it say about a University when it asks about something? I believe that SETs are one of the clearest symbols of teaching values in a University. If students are invited to comment on how good the lecturing was, there is a clear implication that this is what the University values. The CDT is trying to develop excellent, independent scientists capable of articulating and solving challenging problems in chemical manufacturing; I tried to limit the scope of questions to align with this goal. The survey tries to ask students about their reflections on their own development because this is what we care about.
In general, I tried to ask about learning rather than teaching. I broke this rule to ask about the organisation of sessions (which students should have a useful and valuable view on) and the ‘overall teaching delivery’ (which seems vague enough to elicit a holistic student view).
A Little Likert
I feel that the classic 5-point Likert scale pretends to a precision it just can’t claim. Picking between ‘agree’ and ‘strongly agree’ uses up students’ executive function without adding interpretable detail. I drafted blunt three-point Likerts (Agree/Neutral/Disagree) because I didn’t quite have the nerve to propose a simple Agree/Disagree. I had to consider what the role of the SET was when deciding this. I think all it can do is suggest areas to investigate, rather than constitute some final judgement of the module. Losing the granularity seems reasonable in this enhancement-focused conception of the survey’s purpose.
I also tried to ask a small number of questions. One of the things which strikes me about things like the NSS is that it’s really long. I got this one down to eleven Likert questions.
Focused Free Text
The exact instruction for the free text comments gave me the greatest pause. The questions are constructed as ‘what should we do?’ to try and focus student suggestions into concrete items without leading them unduly. My hope is that ‘we’ is interpreted as including both instructors and learners. Arguably, though, the question invites students to comment upon teaching activities directly. This opens me up to the established criticisms of the SET literature, but at the same time it respects the student’s judgement about their experience.
I included a professionalism note with all the free text questions, hoping to preclude some of the nastiness which can be so demoralising when reading student evaluations.
Reflections on Purpose
This whole exercise is easier to do in a CDT context than an undergraduate one, because the only objective of the exercise is student success. No-one is pretending that the evaluation should relate to things like promotion, and there is therefore no need to reach for things like student opinions on individual staff. I think this allows scope to properly explore the power of SETs as a prompt for reflection on learning, which seems like an under-exploited aspect of their power in my experience of undergraduate SETs.