The Grading Podcast

Bosley: Solely my opinion, if there’s research to back it up, I don’t know of it yet, but that goal of the students is why they look at the percentage, the score, and then they stopped. Because again, the feedback didn’t help, you know, didn’t really add or subtract points. It was, here’s my goal, my goal was to get, you know, as high as a percentages I can get, all that tells me about that goal is the percentage at the top of the page. Doesn’t matter what the rest of it says. Now that I’ve gone to alternative grading and after I get my students trained on this, their goal is to get mastery of the learning targets. Now the feedback does matter. Now the feedback is helping them reach that goal if they didn’t.

Welcome to The Grading Podcast where we’ll take a critical lens to the methods of assessing students learning. From traditional grading to alternative methods of grading. We’ll look at how grades impact our classrooms and our students success.

I’m Robert Baleslea, a high school math teacher, instructional coach, intervention specialist, and instructional designer in the Los Angeles Unified School District and with Cal State LA.

Sharona: And I’m Sharona Krinsky, a math instructor at Cal State Los Angeles, faculty coach, and instructional designer. Whether you work in higher ed or K 12, whatever your discipline is, whether you are a teacher, a coach, or an administrator, this podcast is for you.

Each week, you will get the practical, detailed information you need to be able to actually implement effective grading practices in your class and at your institution.

Hi, everyone. Welcome back to the pod. Glad to be in your ears. I am Sharona, one of your co hosts for the Grading Podcast. And with me is our other co host, Boz. How are you doing today, Boz?

Bosley: I’m doing really good today. I’m looking forward to this episode. This is kind of one of our last of the Beginner episodes, even though most of those topics we’ll come back around to plenty of times, but yeah, I’m excited to kind of wrap this phase of the podcast up.

So in this one, we are going to be talking about feedback. In the episode on the four pillars this came up and it’s come up a couple of more times. So today we’re really gonna dive deep into what exactly we mean, not just with feedback, but also some of the other things that we have to do with our students to really make this feedback powerful and make our feedback loops and our relearning really work.

Sharona: Well, and I think what you said there is really important. This idea that feedback loops, I mean feedback loops are central to this grading system to any type of alternative grading. Because at the core, that’s how human beings learn. And there’s a video that I use in my classes, it’s a skateboarder learning how to do a specific trick.

And it’s actually labeled how humans learn or something like that. There’s a clean version, which we’ll link in the show notes. There’s also an explicit version that sometimes I show. But it really kind of drove home to me that this pillar, this feedback loop pillar, even though we’re sort of touching on it in the end of the intro, getting started part, it’s really the heart and the soul of this.

And so, one of the things that I have learned as I’ve been doing this is that there’s some really important things about not just feedback, but effective feedback. So what is it that allows feedback to be effective and also how the heck do you get students to actually use it?

Bosley: Yeah, and I think that last point you made is a really big one because feedback is not new. I mean, this isn’t something that’s exclusive to alternative grading. No offense to our higher ed listeners, but I do think the K 12 world does a little bit more with this and a little bit better and a little bit more trained, but that it’s not just us giving feedback. It’s how do we make our students really utilize it because. I know I’ve spent hours and hours and hours putting feedback on assessments and things and my students look at the number and that’s it.

Sharona: Well, and there’s actually research to that effect. They did a study, and we’ll have to link it in the show notes where they gave three different types of markings on assessments. They gave just a grade, either a letter grade or a numeric grade, they gave a grade and feedback, and they gave just feedback. And then they had students essentially respond to that feedback, I guess, or talk about it, or, I have to look up the actual study, but they found that if you gave a grade by itself, or especially if you gave a grade with the feedback, students didn’t even look at the feedback.

It just completely shortcut the whole thing. So we’ll definitely have to link that study in the show notes.

Bosley: Alright, so before we kind of get into how do we make it useful. What does good feedback look like?

Sharona: So, if you were to Google effective feedback, you’re going to get a lot of things about both in the classroom and outside the classroom. But in my mind, there’s maybe, I think I wrote down six specific elements, and I really want to talk about three of them, and then I know you want to talk about some of them. But the three that are really important to me are that they reference a goal, so there’s something that the student cares about, they are specific, and they’re timely. And I want to go in a little bit to each of those things, if that works for you. So, by goal referenced, what we mean is the student has to think that there is a reason for them to look at the feedback. So typically, in an alternative grading system, a lot of your feedback is specific to your learning targets, or to the assessment that is checking the learning target.

So for example, in my linear algebra class, every one of my assessments is directly aligned and labeled for which learning target it is. So the feedback is primarily about that learning target’s content. And I’m trying to make sure that the student understands how it aligns. Whereas in my history of math class, where I not only have learning targets but I have projects, my feedback is specifically talking to that project and telling the student, this is what you need to do or need to look at. But it’s because the students know they need to complete the project. So they have that goal.

And then I said it was specific. This is so critical because when we have these conversations, sometimes instructors are like, well am I supposed to just tell them what to do? And no, you’re not supposed to just give them the whole thing, but you need to point them very specifically to either where they need to improve or what they did well.

So, for example, I no longer give the feedback "good job" or "good answer" because that doesn’t tell the student anything. Instead, I will say that the answer is clear, complete, and correct. So they have all of the elements of what it is about that thing that is actually good.

And if it’s feedback on improvement, I will point to maybe the first mistake they made. I will say, hey, you know. You made an arithmetic error in this area or you didn’t include something I’m looking for and I might ask them a question or there’s a variety of things, but I’m going to point them to a place where they’re going to go look at their own work. So it’s very detailed.

And then critical to this is timely. One of the things that unfortunately students have been trained is that they can’t count on when feedback is going to return. So they can’t factor that into their learning process. For me, my goal is if a student takes an assessment, now my assessments are very micro, they’re very, very specific, there’s a lot of them, but they’re very micro, my goal is to get it back by the next class period because I only teach twice a week. So for me, those are three very important components to effective feedback.

Bosley: Yeah. And I want to kind of talk about each of those three a little bit as well. Actually one of them more than just a little bit, but those first two goal referenced and specific.

I think this is one of the big differences between the feedback I give now and the feedback I would give when I was still doing traditional grading. Like I said, I’ve always given feedback. You know, that was a big part of my education as a pre service teacher. I literally have books on feedback and the importance of feedback and all this.

So feedback’s not new, but because the goal of the assessment was different, when you’re doing traditional grading, the goal, for especially for the students, is what? It’s to get a good grade. Has nothing to do with learning. The goal for the students is to get as high as a percentage or point score as they can. Because that’s the goal, that’s what they’re focused on. With alternative grading, the goal isn’t getting a high percentage, because there is no percentage, the goal is showing mastery on whatever the learning targets are on that assessment. So that changes, a little bit, the focus of the feedback. And in my opinion, and this is solely my opinion, if there’s research to back it up, I don’t know of it yet, but that goal of the students is why they look at the percentage, the score, and then they stopped. Because, again, the feedback didn’t help, didn’t really add or subtract points. It was, here’s my goal, my goal was to get as high as a percentage as I can get. All that tells me about that goal is the percentage at the top of the page. Doesn’t matter what the rest of it says.

Now that I’ve gone to alternative grading, and after I get my students trained on this, their goal is to get mastery of the learning targets. Now the feedback does matter. Now the feedback is helping them reach that goal if they didn’t on that specific assessment. So that’s where, like I said, I think, and again, this is solely my opinion, I do not have research to back that statement up, but that’s one of the really big differences and when we hear, later in the show we’re going to talk a little bit more about why this isn’t as easy as it seems to do, but then the other one, the specifics. You talked a little bit about how you give feedback. So I want to give some, some specifics on what we mean with specific feedback.

Sharona: Before you go to that though, can I please jump in? I want to go back to goals for a second. O ne of the things that I think we’re going to have an episode on in the future is goal theory and how it relates to learning is actually a very, very hot topic of research right now. And so there’s actually research currently being done about how goal theory and learning, affect their learning. And so, I don’t think we want to go into it in this episode, but what we’re learning as we do this work in general is the nuance about goals, both the ones that we give as instructors and the goals that the students have, highly, highly impacts their performance in the classroom. So I think as long as we sort of put a pin in right now, goals and making these reference to goals is crucial, but there’s more work to be done on goals in general and how it relates to alternative grading. So we’ll definitely do an episode.

Bosley: Exactly. I kind of break my feedback into three different categories. And it depends which kind of mistake the student made as to which type of feedback they get. So one of them, my first one, is I will ask guided questions. I won’t actually tell them something. I will instead ask guiding questions to help lead them to their mistake or how to correct their mistake.

I really use this one mostly when students are making an error that is related to the learning target itself, that is a true error in what I’m assessing. So that’s kind of my first one. And again, I’ve heard the same complaints that you have about, oh, are we just supposed to, you know, spoon feed and point out their mistakes?

No, you don’t have to spoon feed it and tell them exactly how to correct it. But with good guiding questions, and there’s again, quite a few books and different papers out there about how effective questions and guiding questions can be in the learning process rather than just telling some, you know, telling a student something.

So, my other one is usually when a student makes some sort of basic math skill that’s not directly related to the learning target I’m assessing. They make a silly sign mistake or they do two times three and get five. So just one of these kind of really silly arithmetic or sign or maybe basic algebra mistakes. And in those cases, I’ll just tell them, I’m like, hey, you’ve made a sign mistake somewhere. Go find it. Like, so I’ll tell them specifically what the mistake, the type of mistake, but I won’t tell them where it was, or I might say, yeah, you made two sign mistakes somewhere in here. Go find them, go double check your arithmetic.

Sharona: Exactly. So, and that distinction between mistakes that are not content and mistakes that are directly related is really critical. And I don’t, I was thinking about it as you were talking because you and I grade a lot of the same classes or you sometimes help me because I teach this linear algebra class and one of the things that keeps coming up is that my students don’t give me a full equation when there’s an equation. And equations are something they begin to learn like I think in the fourth grade? Something like that. Fourth, fifth grade, somewhere in there. And it is not an arithmetic mistake. So it’s not that they did two plus two and got five and it was like the second they see it they will be able to fix it.

It’s that they don’t, they have some sort of fundamental misunderstanding of the role of an equals sign. And I saw this in calculus when people would write the equals sign on things that are not actually equal. So that one’s a little harder because I’m not teaching them what an equation is, in theory, and yet, them not knowing what an equation is, is coming up.

So where does that kind of thing fall in your specific feedback? It’s because it’s not directly my learning outcome, per se, but it’s also not a silly mistake.

Bosley: So that really depends on what the learning target is that I am assessing. And it will sometimes depend on how far we are into the semester.

And if I can tell this is something they forgot that time, or if this is something that might be a more fundamental misunderstanding. If it’s something that I think they just forgot that time because I’ve seen them do it in other places plenty of times. Then I might do that, my second category that, hey, you know, you, you, you have an expression that’s supposed to be an equation somewhere.

So if I, and again, that’s if I think my student just made that Homer Simpson d’oh mistake and it’s not more of a fundamental gap or misunderstanding. If it is more of that fundamental, then I might ask the questions, but if it’s that fundamental, I don’t know if that’s actually going to end up helping them guide them.

So in that case, I will often do my third one, my third broad style of feedback where I will tell them specifically where to go in a notes or an example to look at or tell them somewhere to go. So they can go in and look at it themselves. And then encourage them to ask questions in class or come to office hours to talk about this.

But, your exact example you just gave with making something that should have been an equation an expression, because they’ve left the equals sign out, I would tell them, hey, go look up at the definition of expressions and equations. What’s the difference? What were you supposed to do here? And then encourage them to, if you’re still not sure, come see me in office hours so we can talk about this.

Sharona: Well, and I think that’s a very crucial point because certainly by the time they’ve gotten to me, they’ve been utilizing these sort of more foundational skills for a long time. And the fact that they still don’t really have it ingrained means that me telling them, teaching them again is not going to work.

Something else has to happen. And for me, it’s that reflective piece. They’ve never been asked to answer that question for themselves by comparing their work to something else. So I think that reflective question or that reflective instruction is absolutely one of the most useful types of feedback we can give, because it has them take an action that they’re then going to either grapple with themselves or potentially come back to us again and we can engage more deeply with it.

So I love that type of not just "go look at your mistake and think about it" but "you have a mistake, go look at a correct one and compare."

Bosley: Yeah. So those are, like I said, the three general types of specific feedback, but most of my feedback will fall into. Either asking those guiding questions, pointing out the type of mistake but not where, or giving them some place to go specifically to look to see if that helps them correct.

Sharona: So, one of the challenges that I had in history of math is that all of a sudden I was assessing their writing. Now, I happen to have a master’s degree in marketing in addition to a master’s in math. So my writing is very, very, very good. But I don’t teach writing. So figuring out how to give feedback in a discipline area that was not mine and yet was something I know they’re supposed to be able to do was challenging.

And what I rely upon in that case is, I’m not comfortable pretending to be an English professor, for example. I’m not a writing teacher. But what I am very good at is explaining that I’m not following what they’re trying to say. So the type of feedback I give on something that is not in my core area of expertise sort of reflects back on me as the grader.

And I say, "I’m not following this. This isn’t making sense to me." And then I would ask the guiding questions of what were you attempting to say? What were you attempting to convey? So, therefore, there’s not sort of an out there almost a moral judgment of "you did good" or "you did poorly". It’s more that regardless of how well or poorly you did, I’m not getting it.

And since I’m grading it, your goal as a student is to do something to help me get it. So, that’s a thing that I feel pretty confident in, because I can just say, look, it’s not clear and it’s not working for me. And I think you need to try something else. And come talk to me if you don’t know what to do, because this might be something we have to work on together.

And we’ve mentioned this a couple of times that that leads me to a fourth thing that I want to start, but then I know you want to talk about, which is what to do with the feedback, the action orientation. So, you mentioned you told them to go look at something or things like that. I’m also thinking about telling them what their next step is.

Do they need to revise it and resubmit it? Do they need to prepare for the next time they’re going to try to tell me? Do they have to do something that’s missing? So not only am I going to say, you know, this is what is correct, this is what I’m looking for. But I’m going to say and what’s your next step? So there’s an action to be taken.

Bosley: Yeah, and with those examples that I was giving, like when I when I’m asked to kind of break these down, I have a really hard time showing how to be specific without also doing an action oriented feedback. Like I know they are two separate things, but giving a good example of one almost always includes the other as well.

Sharona: Well, except sometimes some of our feedback, like my feedback about correct answers. I’m still giving very specific feedback. It’s just not, that particular feedback is not action oriented because there’s nothing further to do in that moment. But I still have in the back of my head, because I said this is clear, correct, and complete, I’m hoping they’ll take away from that and therefore keep doing it. So there still is a sort of hidden action in there.

Bosley: Yeah. And I’ll sometimes do the same thing with. Especially that Gen Ed stats class that we both teach when we’re, you know, say, they’re asking about the being able to generalize the results of a study. And regardless of what their answer is, it should have a couple of key points.

And if they do get mastery on that, I will point out why they have, you hit all three of these to, like you said, hopefully encourage them and make them, understand that the next time they’re asked to do this, those are still the three things they need to hit and missing one of those could cause them to not get mastery next time.

Sharona: Now we’ve taught or tried to teach other instructors. We’ve tried to mentor others, instructors and how to do that.

Other than giving examples, sometimes I’m not sure what to do. What’s been your experience though?

Bosley: But I think those examples are probably the best way. In fact it, not too long ago, I was facilitating a EGI, which Equitable Grading and Instruction that’s what my local high school district calls this type of grading, but I am a certified instructor, so I was facilitating a group, and one of my colleagues, who I have the utmost respect for turned in one of their assignments that was really good but just didn’t quite meet the rubric requirements for proficiency. So I gave her probably two lines of feedback, like, Hey, you did these two things, and it’s done really well, but you left off this third point.

And I cannot give you the proficiency score without this last one. So go back and just add this missing piece and resubmit. And then she wrote me back saying, Oh my God, thank you. Now I understand what actionable feedback is. Like I’ve always heard this, I’ve never really quite understood it until I saw your feedback where you were given me an action to do.

You told me specifically why I didn’t get the proficiency score on the rubric, it’s because there was a specific part of that scale that was missing, and what I needed to do to raise my score from not yet proficient to proficient. And she was just like that, it all of a sudden makes sense.

Seeing that one good example made this actionable feedback a lot clearer to her. So I really do, I think the examples are probably the best way to show someone how to do this.

Sharona: I think that if I taught in a teacher prep program, which I hope to do, I don’t currently, but if I taught in a teacher prep program, I would actually probably have a learning outcome on how to give feedback.

Bosley: Oh, absolutely. Absolutely.

Sharona: Which I love because one of the types of alternative problems that we like to talk about when we’re doing our trainings is giving incorrect work as the problem and telling the student to fix it.

Bosley: Yeah, the error analysis.

Sharona: The error analysis problems. So that’s another way where you can not only give a student a more authentic and less cheatable assessment, but at the same time you can start teaching them about giving feedback, which, when they’re trying to struggle to give feedback, I have found makes them more responsive to feedback that I’m giving because they’re learning how to read it and they’re thinking about how to write it. So those go hand in hand, and that’s really fun.

Bosley: And then one of the other things that’s really key to giving effective feedback is also the consistency of it.

You know, not just the consistency of giving feedback, but the consistency of what that feedback looks like coming from you, the instructor. That consistency is really important because that consistency is what starts to train the students on how to use it.

Sharona: And that brings up quite a big thing there is because our students, certainly in college, but even in high school are coming to us having been trained that feedback’s not useful. Because there’s nothing to do with it. Like, when you take a unit test in a math class in the sixth grade, and it’s in a traditional grading system, and you get feedback, there’s nothing to do with that feedback. If you even get feedback, which, unfortunately, you don’t always. So…

Bosley: Well, I think you’re getting into… You know what? So we’re leaving, we’ve looked at what good feedback or effective feedback looks like. But it’s not just giving effective feedback that we have to do, especially when you are using this kind of grading system. It’s more to it than just that. And that first thing is what you were just saying.

We have to deprogram our students that what they’ve learned in traditional grading systems, that they’ve probably been in a lot more than a non traditional, where that feedback because there’s no reassessment or because that regardless of what they end up doing later on, they’re still getting punished for that mistake on that test, that feedback isn’t going very far again. Like we said earlier, they see the number and because the goal is to get a high grade, not the goal, isn’t learning. The number is what they care about. We have to. Really deprogram our students for them to stop looking at points, stop looking for scores and start really looking at that feedback and making the feedback loop work. Because if we give the feedback and the students don’t really take it in and don’t do anything with it, the feedback loop is broken and learning’s done.

Sharona: And what’s really interesting about the process of teaching them to use it. It is, at least in my world, the process of teaching them to find it is non trivial.

Bosley: Especially if you use an LMS system. If your feedback is on, you know, paper, it’s a little bit easier, but yeah, if you’re using an LMS system and most, I would venture to say most, if not most, at least a good percentage of instructors, both in the K 12 and in the higher ed, are doing now.

Sharona: So I actually take time after my first assessment. And unfortunately, I’ve had to learn it too because I use my LMS system one way but there’s probably three four or five ways my students can access it and each of those ways might have literally different technical elements of how to find it.

Bosley: And even then, the LMS system though that we look at does look you know different than what the students see. So you and I both use Canvas at Cal State. I also use Schoology in LAUSD. That’s our LMS system. I believe you’ve used other LMS systems, is that right?

Sharona: Not extensively. More it’s been my partner teachers. I used Moodle for a brief moment before I went into Canvas, but that was already six or seven years ago. So mostly I’ve been on, I’m pretty heavily a Canvas user.

Bosley: So, with both Schoology and Canvas, and from what I understand of Moodle and Blackboard from other people that I’ve worked with, there is actually multiple ways that you can give feedback. I’m thinking specifically in Canvas, I can think of three different ways that I could, three different places I could give feedback on a single assessment.

And that’s part of the consistency.

Sharona: Well, and it’s worse than that in Canvas. Because the type of assessment changes the places you can put feedback. So on an assignment in Canvas, you can actually do annotations, but you can’t do that on a quiz. But on a quiz, you can put feedback directly in the relevant question, and you can’t do that on an assignment. So, and then there’s like two or three other places you can put it. It’s a little bit chaotic.

Bosley: Yeah. So that’s the first thing. Before you can even start training your students on how to use your specific type of feedback and how to make that part of your feedback loop is you’ve got to train your students on where to find the feedback that you’re going to be giving.

And like you said, that is something that a couple of years ago we found out. It actually looked very different on our end than it did the student’s end. So that’s something as an instructor you might also need to do is really go and find out what you think your feedback looks like might look very different.

Sharona: So once you teach them where to find it, now you’re going to try to teach them how to use it.

Bosley: But just a quick point. That’s part of that consistency we were talking about earlier. If I’m, on one assessment if I’m doing it in the comment part of each individual question, and then the next time I’m doing it in the feedback part of Canvas for the whole assessment, that’s two very different places and the students would find it in different places.

And then on the next one, if I’m doing it in the comments of the rubric itself that’s a third option. I can’t, it doesn’t matter which one I pick. I have got to stick with that one every single time. Otherwise, it’s going to be a nightmare for my students.

Sharona: And what’s coming up for me with that statement is the recognition that we thought in our classes, and maybe historically this was more true, but we thought we were primarily teaching content.

Bosley: Yeah, let’s look at a typical student. So let’s take a typical student at the higher ed, a Cal State LA student. On average, how many classes would that student have in a semester?

Sharona: Minimum of four.

Bosley: Yeah. It’s even, let’s make it even broader. Let’s say probably three to five, maybe six classes.

Sharona: Yes.

Bosley: And at Cal State, all of us, like that is our LMS system. But there is absolutely no requirement that I use that LMS system. I could end up making my own kind of like spreadsheet or so even though Canvas is the LMS system for every class, might not have every teacher using it.

And then like we were just saying, there’s three or four different ways of putting feedback in an assignment for Canvas. I might be doing it in the feedback or the comment section of each question, you might be doing it in the feedback for the whole assessment and instructor C over there might be doing it in the rubric.

That’s three different things our students have to learn. Now let’s actually go down to the my world in the K 12 world where they have 6 to 8 classes. And again, even though it’s, you know at my district, I know not all districts do this, but my district, we are not just adopted an LMS system of Schoology, we’re mandated to use it.

Like we have no choice. I, by contract, actually have to have so many grades in that LMS system a week, but just like Canvas, there’s more than one way to put feedback. So now here are students with not just three to six, but actually six to eight different classes and each teacher, even though we’re using that same LMS system, there’s no requirement that we’re all using it the exact same way.

You know, even at a school site, I have never even attempted to go as far as train my fellow colleagues that we all have to put the feedback in place A instead of place B.

Sharona: Yeah, it’s never going to happen. And it’s not necessarily worth it. Because different assignments require different structures of feedback. So that works fine.

Bosley: But that’s all of those things combined is why we as instructors, as part of introducing our students to our classes, our routines, our procedures, and our grading system, we need to specifically teach them and show them where to find the feedback that they are going to get from you, what your specific feedback looks like.

And that’s, I think that’s another weird thing. That I’ve found, like I said at the beginning of this episode, feedback is not new for any of us. So this assumption that it seems like a lot of teachers have that I just give feedback that the students automatically know what to do with it is, is definitely a false assumption. And it starts with even as simple as where to find it.

Sharona: And I want to elevate the idea of the importance of teaching the students how to use it. This is not, for me anymore, just a practical, you’ve got to do it or it’s not going to work. At the end of the day, certainly in higher ed, I have a goal that is bigger than my class. I have a goal to really, really teach my students how to learn. You know, for me, teaching a student to be a lifelong learner is the absolute goal.

The awareness that I’ve come to, that my failure to teach students how to learn from feedback is actually hindering that goal of turning them into a lifelong learner. Because once you get out of a school environment, you’re never going to learn through grades again.

Bosley: Yeah. And that’s funny, you said that. I would challenge anyone, at least in this country, to show me a mission or a vision statement at any school K 12 that doesn’t say lifelong learner somewhere in one of those two statements.

So it’s supposed to be a goal, not just for higher ed, it’s supposed to be the ultimate goal for everyone in K 12 and into higher ed.

Sharona: And I understand that, but I guess what I’m saying is I didn’t understand how I was actually hurting that goal so badly. And now I’ve elevated that to the level of any other specific content goal, at least in my head, if not specifically in my grading system. So, I just think that’s super important.

Which leads me to the idea that we’ve now said how important it is. We’ve said that we need to teach them where it is and how to use it. But there’s another step to how, for me, which is that there’s different types of interactions in the classroom that could be centered around feedback.

Bosley: Okay.

Sharona: So for me, there’s at least three that I can think of. There’s an interaction between my feedback, the instructor’s feedback, and the student. That can happen privately, it can happen in their head, or it can happen in a conversation between me and the student.

But they could also give themselves feedback. That’s a possibility. Because I could, for example, hand out an answer key and ask them to compare, maybe on a practice problem or maybe on an actual assessment.

And then there’s also the possibility of peer feedback. So what have you used of the three different types of feedback, of the instructor’s feedback, self feedback, and peer feedback? What do you see, how do those play a role in all of this?

Bosley: So I, obviously the first one is one of the ones I rely the most on. I do, in my instruction, whether it’s in my role in K 12 or my role in higher ed, I have always leaned heavily on group work and group projects and group participation.

I do think that’s a crucial role in the education process, but when it comes to peer feedback, I’ve always been a little bit iffier on. One of our, one of the people that we had on the podcast a while back, Joe Zeccola, I think is an absolute expert on utilizing peer review and peer feedback.

And the fact that’s is one of his learning targets ,in his English classes, is both giving and using. So that is something I would like to do more of but I don’t think I do that extremely well, because I’m not sure how to train my students on giving that meaningful feedback to their peers. Yeah, so that’s part of my growth that I still need to do.

Sharona: I would say that I too, I have not used peer review as much, and there’s two things that come up for me. One is, I think where we do use it, both you and I do, is we do a lot of in class group work when we’re just learning. So before they’re doing some sort of like a mastery assessment, if we do, say, the practice problems for our statistics class, a lot of times we’ll do it in class and we’ll have them sit in a group and discuss it.

And in a way, there’s an element of peer stuff in there because they will show each other their answers before they actually submit it. So they’ll grapple with it, and I think we can put some intentional structure around that about how to talk to each other in that environment. So I think there’s an element of it there, but I agree it could be more specific.

Bosley: Like you were saying, we both do utilize this quite a bit. One of our normal activities that, that we do is, beyond just being an alternative grading, that Gen Ed Quant Reasoning with Statistics course, we do have a flipped classroom model to it. So anyone that’s not familiar with flipped learning again our first guest was actually the author of the flipped learning or the flipped classroom, Robert Talbert.

But the idea of it is instead of the primary first instruction coming from the instructor during class, and then all the work in practice is done outside, you shift that. So a lot of the initial learning is done before class, and then a lot of the practice is done in class. So when we’re doing that practice you’re right, I’m usually doing that in groups, and we’re having peer feedback with each other, or the students are. I just don’t think I’m good at capturing how effective that is, or really training my students. And that’s what I was saying when, or what I meant when I was saying that’s an area of my personal growth that I still need.

Sharona: I would agree that I don’t know that I feel that I do this as well as I could. And one of the things that’s holding me back, and I’m just going to be honest and vulnerable here, is I have done some reading about power dynamics and identities in the classroom. And if you’re not careful a lot of the challenging power dynamics that have to do with people who have various marginalized identities can really come out in a pretty ugly way in the classroom. So creating a safe space for peers to communicate with each other and to avoid some of those marginalizations of the extroverts versus the introverts or the men and the women or the…

Bosley: Majority and people of color.

Sharona: Gender conforming and gender non conforming, all of these things. And of course we teach at an institution that is majority minority. We have over 80 percent of our students. coming from Hispanic cultures.

Sharona: So, I would definitely recommend that anyone who wants to use peer feedback, peer review, to do some careful research and training on it before making it an integral part of this classroom. It’s just a fraught area, and it can be some of the most effective feedback. that people get and give. So I encourage it. It’s not, we probably should have someone on here specifically to do conversations about peer review and how to train students, how to do it. I think that’d be a fascinating episode.

Bosley: Yeah. Well, I know we have , down the pipeline we have a few more high school and K 12 instructors that are coming on. I think that would be interesting if any of them utilize that and finding out how they set that up and how they make that work.

Sharona: So I want to go back though, for a moment to the self review. So one of the things that I absolutely love is correction and reflection assignments. So what I mean by this is I ask the student, if they’re not doing a revision, if it’s maybe they’re earning a chance at a reattempt, for example, this is when I would use this, I will say, take your previous assessment, utilize the answer key, correct it, but then reflect on what you did wrong in the moment.

What did you recognize in the moment of completing the assessment that you have learned since then, that will now enable you to do better? So I tell my students, what I’m not wanting to hear is "I didn’t study enough" or some sort of vague thing that’s not very actionable.

Usually for my reflections assignments, the answer is typically one of three things. "I didn’t read the question closely enough", "I didn’t understand a fundamental piece that I now realize I didn’t understand", or "I just didn’t answer the question that was being asked". So it’s like usually one of those three things that they come to realize in the moment. Another possible one is "I ran out of time" that, because again, these are things that they need to recognize what is happening while they’re doing it.

Bosley: And I think that is a great thing to point out. You know that having students, regardless of settings or age, I think you can do this with very young students as well, but having them think about their thinking process, their working process that led them to those incorrect incorrect answers ot mistakes in papers is a huge part of the learning cycle. That self reflection. But like you said, it needs to be more than I didn’t get enough sleep last night, or I didn’t study, I’ll study more. Really? Getting into that metacognitive look at, Oh yeah, I made this mistake because XYZ, or I wasn’t clear on the difference between independent and dependent variable.

Sharona: And that level of reflection is where the learning happens, right? That is the loop right there. When they have that aha moment of, "oh, I consistently don’t read the entire question", that’s an aha moment, and they have dramatically and visibly learned something from that. And that’s when the feedback loop really kicks into action. And that’s a very exciting moment for me.

Bosley: This is a great point to be made that feedback doesn’t just have to be instructor to student. It can be student to student or even student to self.

Sharona: And probably the most powerful one is the student to self if they’re able to do it.

There’s one thing we skipped over a little bit that I’d like to go back to because I’m concerned a little bit that, you know, you and I each have about 100 students, sometimes we have a little bit more, and people might be listening to all of this and going, holy cow, I can’t possibly do this, at the level of regularity that I’m planning to assess, for a hundred plus students. So I do want to talk for a moment or two about ways to do this effectively and yet also efficiently.

So, in particular, we’ve talked a lot about personalized, individualized feedback. There’s a couple of ways to make it more efficient, and the first one that comes to mind is utilizing your proficiency scales, and their descriptions ,very effectively. So for us, I’m very minimalist on my proficiency scales. They have literally an emoji and a meaning behind the emoji. So the majority of my feedback is individualized. But I know that Joe’s proficiency scales, from the episode when we met, with him have entire descriptions. Always includes an argument for the thesis or things like that. So the first place the students get trained to go to is the proficiency scale level and they can then immediately begin that self assessment. "Okay, the instructor said I didn’t meet the components of a sufficient thesis statement. Let me see if I can figure out where I didn’t get it", right?

Bosley: Yeah, and that’s a you know, that’s kind of a great example of where do you want to put some more effort in to save you time at a different place? So you and I, and I do this with my K 12 world as well, tend to use more general rubrics and proficiency scales that we can reuse for that same learning target regardless of the assessment.

So that saves us time on writing the rubrics and the proficiency scales. Whereas the example you were just giving with Joe, he writes specific ones. Even if it’s a different assignment but the same learning target, he gives a different rubric with different, slightly different, definitions on his proficiency scales.

So he spends a lot more time doing, designing and putting those together. But then it saves him a crap ton of time when it comes to the actual feedback. Whereas you and I, we save some time on writing those proficiency scales and those rubrics, but then we end up spending more time on giving the feedback to the students.

So which one works better? Well, it really depends on you and your style as an instructor. But that’s a good example of, you know, spending in one place to save in another.

Sharona: And then another thing that we do, you’re absolutely right and I think that’s the key, is you’re going to have to spend a certain amount of time doing this. Which one feels better? Which one’s going to be more efficient for your particular situation? So for us, we get a lot of repetitive mistakes. We get the students, different students making the same exact types of mistakes. So we can automate that in a couple of different ways. Canvas has the ability to save your favorite comments. So we literally can click on one click and pull up the entire comment. You can do a cut and paste from like a general library of comments if you have them on like a Word document or a Google document.

I know that one of my colleagues, Dr. Silvia Heubach, she gives the students a key where she’s abbreviated everything. So if there’s an arithmetic mistake, it’s an A with a circle. If there’s an algebra mistake, it’s something else. And so when she actually does her annotations, she just uses the abbreviations and she annotates in the spot where it’s happening. So she doesn’t have to write, "you have an arithmetic error somewhere in the second step of your matrix", which is the way we do it. She would literally just, on that second matrix, put the symbol or the annotation, so that’s another way to automate.

Bosley: Yeah, and there’s another good example of Dr Heubach, using that shorthand and that, and that method that she does, saves her time there, but that means she had to spend more time training the students on how to use that feedback.

Otherwise, if you don’t, a student seeing an A with a circle above a certain part of the of their answer doesn’t mean anything to them. So again, where do you want to spend time to save time later. But yeah, there is a lot of ways, if you’re going to be doing the kind of individual ones like you and I do, there are a lot of tools that can still automate it, even though it’s individualized, because of, like you said, oftentimes, at least in our classes and in math classes, I’m assuming it’s probably similar in other disciplines, but there’s a lot of types of mistakes that are the same.

So I mentioned one earlier about an arithmetic error or a sign error, like "you have a sign error here go find it". That’s one of those comments that I have saved. And I mean, I can literally click one button on my keyboard and post that in the LMS system and their feedback. So even though it’s individualized, I’m still, I’m literally hitting one key on my keyboard and it’s putting that comment because I use it so much.

Sharona: And there’s a couple of independent tools out there that people love that do this kind of thing as well. So, I know, just off the top of my head, there’s Gradescope that people are using extensively. And there’s also just fundamental computer tools like TextExpander. And we’ll link all of this in the show notes, but these are things that can really shorten your time and allow you to give that personalized attention when it’s really needed.

Bosley: Yeah. Or even just as simple as pulling up a Google doc or a Word doc and typing it out the first time. And then going and cutting and paste from that document to your LMS system. We talked about the importance of utilizing tools in a couple of episodes, when we were talking about doing the grading architecture and a few other places, here’s another one where knowing your tools, knowing how to use your tools and using them effectively is going to make this a lot more feasible, a lot more doable, and a lot less cumbersome on you, the instructor.

Sharona: So I guess in closing for me, my closing thought is, this is definitely the biggest actual time component of alternative grading. Because it’s the heart. The heart is giving feedback to students and allowing them to act on it. The rest of it is super important, and it’s very structural. It needs to be in place to allow it to happen.

But once you’ve designed, like my courses now, I’ve been running them for several years. I don’t have to do a lot of design setup at the beginning of every semester. So what am I doing during the semester? I’m teaching and I’m providing feedback. I’m providing it in writing. Sometimes I’m providing it in person. Sometimes in the classroom. Sometimes in office hours, often over Zoom. But this is the heart of it. And what’s really cool about it for me is I’m talking math all the time. All the time. These feedback conversations, both in writing and in person, are about the content that I care so much about that I actually spend my days teaching this. This is what I want to do. This is why I’m an educator. So for me, this is the reward.

Bosley: Yeah. And going back to our very first episode when we were just kind of introducing ourselves and the goal of this podcast, we both told part of our origin stories. And that first attempt at mastery that we did together, that was not good.

Why did I never go back? It was because even with that bad attempt, the conversation at the end of the semester with my students was not how many more points do I need to get another grade? It wasn’t, can I get extra credit to get from an 89% to a 91%?

It was about the math. It was about the learning and with this feedback, it’s the same thing. I mean, that it’s just incredibly rewarding to get to talk about math and not playing, you know, silly point games or something like that.

Sharona: So what I’d like to do is I’d like to invite everyone listening to the podcast to submit to us your methods of giving feedback. If you’re doing alternative grading, what is working for you? And what questions do you have about feedback? So to submit those, you’re going to go to our website, http://www.thegradingpodcast.com. Click on Contact Us, and there is a place where you can actually attach a file. We would love to have these as voice memos.

So if you record a voice memo on your phone and attach it, so that we can potentially play it on the air. So, again, if you have something that we didn’t mention that you think is a really great way of giving feedback, we’d like to hear it. And we’d also like to start getting your questions. And once you have your questions, we will be, a new feature is we’re going to be picking a question of the month and asking people to submit not only our answers, but their answers. So again, that website’s thegradingpodcast.com, click on contact us, and you can attach either a text question, or your suggestions, or a voice memo, and we would love for you to become a part of the show. Any last closing thoughts, Boz, before we wrap up for this week?

Bosley: No, I just want to thank our listeners for listening. Thank you for for joining us and we’ll see you guys next time as we continue this journey talking about grades.

Sharona: Please share your thoughts and comments about this episode by commenting on this episode’s page on our website http://www.thegradingpod.com. Or, you can share with us publicly on Twitter, Facebook, or Instagram. If you would like to suggest a future topic for the show, or would like to be considered as a potential guest for the show, please use the Contact Us form on our website. The Grading Podcast is created and produced by Robert Bosley and Sharona Krinsky.

The full transcript of this episode is available on our website.