arrow_back Back to Research

Formative Assessment and Questioning

Dylan Wiliam (interviewed on my podcast here) likes to think of formative assessment as "responsive teaching", and it is a fundamental part of every lesson I teach. When done well it allows me to get an accurate snap-shot of my class' understanding of a concept at a given time and adapt my teaching accordingly. Now, there are critics of formative assessment, such as David Didau, who argue that because of the distinction between Learning and Performance (see Memory section), formative assessment can only capture performance, which is an unreliable measure of learning. Whilst I see some validity in that claim, I would argue that the main purpose I use formative assessment is to identify and understand misconceptions, and for that there is no more effective tool. Needless to say, I am slightly biased when it comes to advocating the use of multiple choice questions for formative assessment, having created Diagnostic Questions, but John Mason's wonderful paper in this section provides an incredible guide to the different types of questions we teachers can ask. This section will be my attempt to survey the literature for the most effective formative assessment practices around.

Research Paper Title: Unskilled and Unaware of It: How Difficulties in Recognizing One's Own Incompetence Lead to Inflated Self-Assessments
Author(s): Justin Kruger and David Dunning
My Takeaway:
Before we get into the concept of formative assessment, we first need to establish if there is a need for it. Why can't we simply say to students "do you understand this topic?", or "what topics do you need help on?" and base our teaching decisions on that? Well, to put it simply, because students (like most novices) are not particularly good judges of their own abilities. This lovely paper describes this phenomenon as the Dunning Kruger Effect. In a series of tests of humour, grammar and logic, the researchers found that people tend to hold overly favorable views of their abilities in many social and intellectual domains. The authors suggest that this overestimation occurs, in part, because people who are unskilled in these domains suffer a dual burden: not only do these people reach erroneous conclusions and make unfortunate choices, but their incompetence robs them of the meta-cognitive ability to realise it. What are the implications for teaching? Well, if we assume that most students are novices in terms of their understanding of mathematics, then it is likely that they overestimate their abilities in mathematics. Hence, relying on their own judgement of their abilities is likely to be flawed, and thus we need another, more reliable measure. In the long term, this could be the results from a range of sumamtive assessments, but in the here and now of the busy classroom, our best effort is likely to come from a well designed question. This also calls into question the practices of traffic lights, thumbs-up, sticking postit notes on level descriptors, and other types of common self assessment strategies I have seen (and used!). How can we help students become better judges of their own abilities? Well, paradoxically the authors found that improving the skills of participants, and thus increasing their meta-cognitive competence, helped them recognise the limitations of their abilities. Once again, helping our students acquire more knowledge seems to be the key.
My favourite quote:
In sum, we present this article as an exploration into why people tend to hold overly optimistic and miscalibrated views about themselves. We propose that those with limited knowledge in a domain suffer a dual burden: Not only do they reach mistaken conclusions and make regrettable errors, but their incompetence robs them of the ability to realize it. Although we feel we have done a competent job in making a strong case for this analysis, studying it empirically, and drawing out relevant implications, our thesis leaves us with one haunting worry that we cannot vanquish. That worry is that this article may contain faulty logic, methodological errors, or poor communication. Let us assure our readers that to the extent this article is imperfect, it is not a sin we have committed knowingly.

Research Paper Title: Inside the Black Box. Raising Standards Through Classroom Assessment
Author(s): Paul Black and Dylan Wiliam
My Takeaway:
This is the bible of formative assessment, full of practical steps that are easy to implement and can make a huge difference. The key finding is that formative assessment raises standards of learning, but there is still plenty of room for improvement. Practical strategies that classroom teachers can use to improve students' learning include: feedback to any pupil should be about the particular qualities of his or her work, with advice on what he or she can do to improve, and should avoid comparisons with other pupils; for formative assessment to be productive, pupils should be trained in self-assessment so that they can understand the main purposes of their learning and thereby grasp what they need to do to achieve; and opportunities for pupils to express their understanding should be designed into any piece of teaching, for this will initiate the interaction whereby formative assessment aids learning. Two points in particular stood out for me:
1) Students can (and should) be trained in self-assessment. But this is difficult to do. We have seen from the paper above that novices find it difficult to judge their own abilities. We can try to help by providing things like "pupil friendly level descriptors", but these can be flawed, as Daisy Cristodoulou so eloquently explained when I interviewed her. Perhaps the best thing we can do to help students develop good self-assessment skills is to provide them with the necessary knowledge to understand the given domain, and provide the kind of constructive, useful feedback that we will discuss in the Feedback section.
2) Formative Assessment is (or at least should be) responsive teaching. It provides an amazing opportunity to find out what your students do, or do not understand, and react accordingly. As a result of reading this paper, together with the papers that follow, I ask three diagnostic questions at the start of every lesson, and adapt the lesson depending on the students' responses.
My favourite quote:
The main plank of our argument is that standards are raised only by changes which are put into direct effect by teachers and pupils in classrooms. There is a body of firm evidence that formative assessment is an essential feature of classroom work and that development of it can raise standards. We know of no other way of raising standards for which such a strong prima facie case can be made on the basis of evidence of such large learning gains.

Research Paper Title: Five Key Strategies for Effective Formative Assessment
Author(s): Dylan Wiliam for the National Council of Teachers of Mathematics
My Takeaway:
A wonderful, practical guide that discusses five key classroom practices that make the processes of formative assessment as effective as possible. Each is backed up by research evidence and illustrated with examples, and I would strongly advise reading the full paper as I will not do it justice here. The five strategies are:
1) Clarifying, sharing and understanding goals for learning and criteria for success with learners. A key strategy I took form this was sharing examples of other students' work is a great way to clearly convey the kind of work you are expecting in a way students can relate to more than abstract, confusing learning objectives.
2) Engineering effective classroom discussion, questions, activities, and tasks that elicit evidence of students' learning. This is all about planning effective questions. These might not necessarily be the kind of open-ended, probing questions that teachers are often advised to ask, but instead carefully planned, multiple-choice diagnostic questions. These have the advantage of being quick to ask, quick to collect information, and incorrect answers reveal the specific misconceptions students have. Needless to say, I love a Diagnostic Question
3) Providing feedback that moves learning forward. This is directly relevant to the Marking and Feedback section. The key point is that feedback should be more work for the student than the teacher, and the student should have time to read, respond to and act upon that feedback.
4) Activating students as owners of their own learning. This is fascinating. Dylan suggests that traffic-lights are a good way of encouraging this, which is a strategy that I have routinely dismissed as I have found students tend to just show green so I will leave them alone! However, Dylan adds the advice to ask students who demonstrate green cards to explain the concept to those with red cards. This gives students a stronger incentive to be honest.
5) Activating students as learning resources for each other. Two interesting findings here. Firstly, feedback students give to each other is likely to be more direct and hard-hitting than a teacher would give. Even more interestingly, the person providing the feedback benefits just as much as the recipient because she or he is forced to internalise the learning intentions and success criteria in the context of someone else's work, which is less emotionally charged than doing it in the context of one's own work. I would add an important caveat to this - that only works if the feedback is correct. Providing students with answers is an obvious way around this, but we must ensure students arrive at these correct answers in the right way.
My favourite quote:
The available research evidence suggests that considerable enhancements in student achievement are possible when  teachers use assessment, minute-by-minute and day-by-day, to adjust their instruction to meet their students' learning needs. However, it is also clear that making such changes is much more than just adding a few routines to one's normal practice. It involves a change of focus from what the teacher  is putting into the process and to what the learner is getting out of it, and the radical nature of the changes means that the support of colleagues is essential. Nevertheless, our experiences to date suggest that the investment of effort in these changes is amply rewarded. Students are more engaged in class, achieve higher standards, and teachers find their work more professionally fulfilling. As one teacher said, "I'm not babysitting any more."

Research Paper Title: Mathematics Inside the Black Box: Assessment for Learning in the Mathematics Classroom
Author(s): Jeremy Hodgen and Dylan Wiliam
My Takeaway:
This is a wonderful, mathematically specific paper that follows on nicely from the generic research that kick-started this section. Indeed, this paper is so good it makes a second appearance in the Marking and Feedback section. The authors discuss five principles of learning, which sound so obvious, but whose effects can be powerful. I will not do the principles justice here, so please read it in full:
1) Start from where the learner is - this is obvious, but difficult. The majority of maths teaching (in my experience, anyway) involves teaching something that the students have already encountered before, or which relies on pre-existing knowledge. Attempting to discover where students are at before you attempt to teach them new knowledge - and in particular which misconceptions they hold - is one of the most important parts of teaching
2) Students must be active in the learning process - we have seen throughout these papers that students remember what they think about, and sometimes learning needs to be difficult. If students are passive bystanders in lesson - and by that I mean if they are not thinking about the content of the lesson - they are less likely to learn
3) Students need to talk about their ideas - this is a tricky one, as students often find it difficult to articulate their thinking. The authors provide some really useful prompts and frameworks that can be used straight away, such as Can you put Amy’s idea into your own words?, What can we add to Saheera’s answer? Well, if you’re confused you need to ask Jack a question.
Which parts of Suzie’s answer would you agree with? Can someone improve on Simon’s answer?
4) Students must understand the learning intention. This requires an understanding of two key things: what would
count as a good quality work (success criteria), and where they stand in relation to that target. Knowing these things allows students to steer their own learning in the right direction, and is known as meta-cognition. Effective peer and self assessment are key to this, and practical tips for making both effective are provided.
5) Feedback should tell pupils how to improve - this will be covered in depth in the Marking and Feedback section
Practical ideas on how to achieve all of these are provided throughout this wonderful paper.
My favourite quote:
The changes in practice recommended here are not easily made. They require changes in the ways teachers work with students, which may seem risky, and which will certainly be challenging. The work we have done with teachers suggests that the teachers who are most successful are those who change their practice slowly, by focusing on only one or two aspects at a time. As they become skilled with these new ideas, and incorporate them into their natural practice, they can then turn their attention to new ideas. Teachers who try to change many things about their practice at the same time are unlikely to be successful.

Research Paper Title: Effective Questioning and Responding in the Mathematics Classroom
Author(s): John Mason
My Takeaway:
This is, quite simply, the best thing I have ever read on questioning in mathematics. There is so much good advice in this paper by John Mason that the best thing I could probably do is copy and paste the entire paper and have that as my Takeaway. However, I will resist and instead focus on a couple of the main things that have changed my teaching. The first is Mason's point that the distinction between open and closed questions is rather a pointless one. My work on Diagnostic Questions has shown me the value in a well-designed closed question in term of quickly finding out information about a class' understanding of a concept, as well as provoking fruitful discussion about misconceptions. As Mason so beautifully explains: "Questions are just words with a question mark: the notion of openness and closedness is more to do with how the question is interpreted than with the question itself". Then there is the section on using question effectively. A really important point that I had not considered was to avoid using questions for controlling purposes, and instead make use of direction instruction or statements. This ensures that questioning in the classroom is used only to promote mathematical thinking, and not behaviour management. I also find that asking students questions such as "why exactly do you feel it is appropriate to do that?" always leaves open the possibility of an unwelcome, and public, discussion taking place, whereas a simple "stop that" might allow the focus of the lesson to return to mathematics. Next comes "funneling" - something I am regularly guilty of - whereby you give your students so many clues in the desperate hope they will begin to figure out what is in your mind. Mason's advice in that instance is to simply tell them, and then enquire as to why they had not thought of it themselves. This is likely to be much quicker and more fruitful than kidding yourself that students have actually understood something because you essentially directed them straight to the answer. Finally, Mason emphasises the importance of being genuinely interested in not only in what learners are thinking, but in how they are thinking, in what connections they are making and not making. Techniques such as giving students more time to think, and asking for methods instead of answers can make a huge difference to the atmosphere in the classroom. I could go on, but I better stop there. This is a truly wonderful paper.
My favourite quote:
Although a very common activity, question asking is at best problematic and at worst an intrusion into other people’s thinking. By catching yourself expecting a particular response you can avoid being caught in a funnelling sequence of ‘guess what is in my mind’. By being explicit at first, then increasingly indirect in your prompts, you can assist learners to internalise useful questions which they can use for themselves to help them engage in effective and productive mathematical thinking. Above all, the types of questions you ask will quickly inform your learners of what you expect of them, and covertly, of your enacted philosophy of teaching. The key to effective questioning lies in rarely using norming and controlling questions, in using focusing questions sparingly and reflectively, and using genuine enquiry-questions as much as possible. This means being genuinely interested in the answers you receive as insight into learners’ thinking, and it means choosing the form and format of questions in order to assist learners to internalise them for their own use (using meta-questions reflectively). The kinds of questions you ask learners indicates the scope and breadth of your concern for and interest in them, as well as the scope, aims, and purposes of mathematics and the types of questions that mathematics addresses.

Research Paper Title: Is there value in just one?
Author(s): Caroline Wylie and Dylan Wiliam
My Takeaway:
This paper deals with answering a key question a teacher regularly needs to ask themselves: "when are my students ready for me to move on?", and was very influential when I was developing Diagnostic Questions. It makes the point that waiting for test data to confirm this is often unsuitable due to the time lag and the abundance of information, and relying on student judgement of their own readiness is flawed. With teachers needing to make this decision "on the fly" mid-lesson, the authors suggest that one well-written question, can do the job. The question should have three characteristics:
1) Designed for easy collection of information (I use one finger for A, two for B, etc);
2) Incorrect responses assist the teacher diagnose what students do not understand, and, ideally, provide ideas about what to do about this (this is all comes down to the choice of good distractors);
3) Correct responses support a reasonable inference that students understand the concept being assessed (students should not be able to get the question correct whilst still holding key misconceptions).
Whilst the information gleaned from asking this one question is not going to be perfect, there is no quicker, more accurate way of a teacher doing it mid-lesson. I ask three of these questions each lesson. If the responses of my students reveal misconceptions, then I respond accordingly - even if this takes up more of the lesson than I had intended. For my take on the use of such questions, see my Pedagogy videos.
My favourite quote:
The issue of “readiness to move on” is a familiar one to teachers, and the decision not to move on will initiate routines that are also familiar: to engage in whole class remediation, to pull aside a small group or individual students for additional assistance, to construct alternative learning opportunities that will assist students in their learning, and so forth, so that the teacher can later reassess the situation and proceed in the learning sequence.

Research Paper Title: Using Diagnostic Classroom Assessment: One Question at a Time
Author(s): Joseph F. Ciofalo and Caroline Wylie
My Takeaway:
This paper also looks at the use of diagnostic questioning in the classroom. There is a nice focus on the characteristics of good questions with some nice examples provided that illustrate these characteristics clearly. My favourite part is the description of how these questions might be used in the classroom, providing a suggestion with how teachers might deal with different response scenarios from students. I have recorded a series of videos about how I use these questions in the classroom each day here.
My favourite quote:
The goal is to help teachers better utilize questioning and discussion to improve student learning, a key strategy within formative assessment. By starting with the instructional decisions that teachers make, the intention is to create both a resource and a habit of mind that is of obvious value to teachers. Lesson planning is a natural part of every teacher’s daily activities. Embedding a focus on questioning strategies—through carefully constructed diagnostic items—supports teachers in improving instruction by providing them with a natural extension to their practice, rather than requiring a major shift in mind-set.

Research Paper Title: The memorial consequences of multiple-choice testing
Author(s): Elizabeth J Marsh, Henry L Roediger III, Robert A Bjork, Elizabeth L Bjork
My Takeaway:
We have seen in the first section on Memory the existence of the Testing Effect (or Retrieval Effect), whereby testing can actually enhance learning. This paper attempts to show that whilst multiple choice questions are often used to measure understanding (via formative assessment), they can also have a similarly positive impact on learning as other forms of assessment via the Testing Effect. Interestingly, these benefits are not limited to simple definition or fact-recall multiple choice questions, but extend to those that promote higher-order thinking. The authors also address something a common criticism of multiple choice questions - the fear that the choice of wrong answers (the distractors) can cause students to learn false facts. The authors find that such persistence appears due to faulty reasoning rather than to an increase in the familiarity of the distractors. There is no doubt, however, that good distractors have the potential to cause students to learn faulty information. The authors put it like this: "multiple-choice lures may become integrated into subjects’ more general knowledge and lead to erroneous reasoning about concepts." However before we make all multiple choice questions illegal, three things need saying.
1) The benefits of using multiple choice questions in terms of eliciting information about students' understanding in an efficient way, and positive effects on their learning via the Testing Effect far outweigh the costs.
2) The issue of learning false facts can be overcome via immediate feedback. This could be a discussion in a lesson when using a multiple choice question for formative assessment purposes, automated marking when using a system such as Diagnostic Questions, or through task-specific written feedback.
3) I am firmly of the beleif that you cannot fully understand a topic or concept unless you are also aware of the misconceptions that accompany it. These need to be confronted head-on (as discussed in the Joan Lucariello and David Naff paper in the Explicit Instruction section). I guess it boils down to when you introduce these misconceptions, and here I think there is an argument for waiting until students have grasped and had some success with the basics. They need to know what is right before they confront what is wrong.
My favourite quote:
More generally, the prevailing societal emphasis on testing as assessment is unfortunate, because it obscures the critical pedagogical aspects of testing. Tests, optimally constructed, can enhance later performance, provide feedback to the learner on what has and has not been learned, and potentiate the efficiency of subsequent study opportunities.

Research Paper Title: Feedback enhances the positive effects and reduces the negative effects of multiple-choice testing
Author(s): Andrew Bulter and Henry L Roediger III
My Takeaway:
This study develops the second point made in the paper above - namely that feedback is crucial both to boost the positive impact of multiple choice questions (increased retention via the Testing Effect) and reduce negative effects (acquiring misinformation via the lures or distractors). Subjects studied passages and then received a multiple choice test with immediate feedback, delayed feedback, or no feedback. In comparison with the no-feedback condition, both immediate and delayed feedback increased the proportion of correct responses and reduced the proportion of intrusions (i.e., lure responses from the initial multiple-choice test) on a delayed cued recall test. The advice for teachers is very simple - if using multiple choice tests, ensure students are given feedback. It is interesting to note that both immediate feedback and delayed feedback had the same beneficial effect. The authors point out that their version of delayed feedback was not synonymous with what may happen in a classroom, with students taking a test and potentially not receiving feedback for a week or two - delayed feedback in this study simply involved not being told the answer immediately after the question. However, this opens up an interesting possibility. We will see in the Marking and Feedback section that Bjork recommends delaying feedback as one of his desirable difficulties. This way feedback stops being a crutch for students, and also taps into Dylan Wiliam's key point that feedback must make students think. Perhaps the important thing here is not necessarily the time delay between the answer and the feedback, but what happens in between. If students answer a question incorrectly, do not know they are incorrect, and then subsequently answer another 100 questions in that matter, then that misconception is likely to be difficult to remove - after all, practice makes permanent. However, if they answer that question and no more like it before receiving feedback, any negative effects may be reduced to the same extent as they would be after immediate feedback, with the added bonus that students need to think more. This is pure speculation, but it makes sense combining the findings of several papers.
Educators should provide feedback when using multiple-choice tests
My favourite quote:
A pragmatic solution to the possible negative effects of multiple-choice tests is to ensure that students always receive feedback after testing. Feedback enhances the positive effects of taking a test and helps students correct their errors, thereby reducing the acquisition of misinformation. The latter outcome is especially important when the same questions and alternatives from a first test are reused on a later test, because the production of misinformation often increases the chance that it will be produced again on a later test

Research Paper Title: Multiple-Choice Tests Exonerated, at Least of Some Charges: Fostering Test-Induced Learning and Avoiding Test-Induced Forgetting
Author(s): Jeri L. Little, Elizabeth Ligon Bjork, Robert A. Bjork, and Genna Angello
My Takeaway:
A common criticism of multiple choice questions is that because students know the correct answer is one of the listed options, they can get it correct by recognition as opposed to the retrieval process from long term memory that we know can aid learning. This paper addresses that criticism by testing whether multiple-choice tests could trigger productive retrieval processes—provided the alternatives (distractors) were made plausible enough to enable test takers to retrieve both why the correct alternatives were correct and why the incorrect alternatives were incorrect. In two experiments, they found not only that properly constructed multiple-choice tests can indeed trigger productive retrieval processes, but also that they had one potentially important advantage over cued-recall tests. Both testing formats fostered retention of previously tested information, but multiple-choice tests also facilitated recall of information pertaining to incorrect alternatives, whereas cued-recall tests did not. In other words, so long as the multiple choice question is a good one (see papers earlier in this section for the rules of a good multiple choice question), then when searching for the answer students thoughts turn to reasons why the distractors are incorrect, and hence they exercise the very retrieval processes they have been accused of bypassing. This is huge, because it suggests that whilst multiple choice questions have all the advantages of speed of information gathering discussed earlier, they have an extra advantage of encouraging the students to think harder than if the question was presented without any options to choose from.
My favourite quote:
The present findings vindicate multiple-choice tests, at least of charges regarding their use as practice tests. In fact, our findings suggest that when multiple-choice tests are used as practice tests, they can provide a win-win situation: Specifically, they can foster test-induced learning not only of previously tested information, but also of information pertaining to the initially incorrect alternatives. This latter advantage is especially important because, typically, few if any practice-test items are repeated verbatim on the subsequent real test. From that standpoint, the advantage of initial multiple-choice testing over initial cued-recall testing is a truly significant one.