The Grading Podcast

80 – Alt grading vs trad grading

===

Sharona: We saw some very interesting things that people have who’ve been in this alternative grading journey with us, we have conflated some of the things and the reasons we do it, so they think we’re doing it because of the grading when we’re really doing it because we’re also trained in equity. We’re also trained in student autonomy. We’re also trained in motivation theory. We don’t always go into all those details. When they flipped it back to traditional so many of those behaviors started to come out.

Boz: Welcome to the Grading Podcast, where we’ll take a critical lens to the methods of assessing students learning, from traditional grading to alternative methods of grading. We’ll look at how grades impact our classrooms and our students success. I’m Robert Bosley, a high school math teacher, instructional coach, intervention specialist, and instructional designer in the Los Angeles Unified School District and with Cal State LA.

Sharona: And I’m Sharona Krinsky, a math instructor at Cal State Los Angeles, faculty coach, and instructional designer. Whether you work in higher ed or K 12, whatever your discipline is, whether you are a teacher, a coach, or an administrator, this podcast is for you. Each week, you will get the practical, detailed information you need to be able to actually implement effective grading practices in your class and at your institution.

Boz: Hello, and welcome back to the podcast. I’m Robert Bosley, one of your two co hosts and with me as always, Sharona Krinsky. How are you doing today, Sharona?

Sharona: I’m doing well. I know that we’re recording this in early January and there’s been a lot going on in LA, so I personally am doing well, but I wanted to maybe touch on what we’ve been dealing with here.

Boz: Yeah. For anyone of that’s been listening to us knows that you and I both located out in California and as of the time of this recording, about a week and a half ago, we had just some devastating fires break out in California. Luckily, you and I are both safe. Both of our families are safe. I did have several of my extended family members, a lot of my sister in laws and a niece, that were in the evacuation warnings, which means they were, you know, they were packed in and ready to go if they had to be evacuated. Luckily, like I said, they were, one of them especially was like right on the edge, I think like two blocks over they were evacuated, but yeah, we’re, we’re all safe. But we definitely want to thank, you know, all the incredible firefighters and that have been battling this and that are still battling. I, I think, especially if you’re not in California, you might not realize just how much it’s still going on. These three major fires that all broke out at the same time, the biggest one, the Palisade fire, as of about 12 hours ago, is still only 31 percent contained. So I mean, this is still very active and very dangerous fires, and I don’t know how many people still realize that that’s not in California.

Sharona: Well, and the other thing is people who don’t live in, because we specifically live in the Los Angeles area, and people who don’t live here have no real sense of the size of Los Angeles. Like more, I think I read something, I’m not 100 percent sure on my data, but more acreage is burned at this point than like the entire island of Manhattan in New York. So it’s hard to describe that it’s both extremely present and yet quite far away. Like I am 20 miles at least probably from the closest fire lines.

Boz: Yeah.

Sharona: And yet I’m extremely close by Los Angeles terms.

Boz: Yeah. Like I said, as of about 12 hours ago, those fires, those three fires combined had burned over 40, 000 acres. And like I said, there, two of them are still very much active.

Sharona: The island of Manhattan is 22 square miles of land. 14, 600 square acres is the island of Manhattan. And how much did you say it was just burned?

Boz: Over 40, 000.

Sharona: Right. So it’s a very large area, and yet, the vast majority of what we would call the Los Angeles area is not directly impacted. So, it’s this really weird dichotomy of this really significant, very severe natural disaster. And I think many of us know people impacted. I have colleagues who were in the evacuation zones, whose houses were fine, but were hit by looters, or people who were, trees came down on their houses. But again, not in the fire area because we also had a significant wind event.

Boz: Oh, yeah that that wind the night that the fires broke out that wind was crazy. And this is someone that grew up in Oklahoma. Like, if the winds blowing 20 miles an hour in Oklahoma the wind ain’t blowing. I mean, I seriously I was almost in a wreck because I was going to get my daughter from her martial arts class and one of these 70, 80 mile an hour gust hit my van and literally I was a half lane over. Like I, if the person next to me wasn’t paying really good attention, I probably would have hit him. Cause.

Sharona: Well, and I think you said he was blown as well though.

Boz: Yeah. I think that might’ve been part of it, but we, yeah, it, that wind was just, it’s, it’s nuts.

Sharona: So I wanted to shout out an appreciation. Every one of my colleagues from this community and other communities who’ve reached out this week for whatever reason, almost all of them have started with, I hope you’re okay. So I just really appreciate that. Yes, I am fine. And thank you for asking because many of our colleagues are not fine.

I’ve also been asked how to support people. And right now, the best ways if you’re out of the area to support people is donating to and a couple ones that I personally am supporting is the World Central Kitchen. They are here feeding most of our first responders, not only the firefighters, but all the National Guard and police that are here to help like prevent looting and do access control. Also the animal shelters in the area are absolutely overwhelmed. The Pasadena animal shelter is overwhelmed. And then if you know anybody personally, waiting to see what they need. Those are probably the top three ways that I’ve seen people helping and that I’ve been helping.

Boz: I would agree that the World’s Kitchen is another, like you said, they have been working endlessly to feed, or help feed the world army of people, because it really is. It’s not just the firefighters, it’s also law enforcement. It’s also, we’ve called up the national guard to help protect the areas with making sure that there aren’t people taking advantage of of this time, which is sad to say we have to do, but.

Sharona: So I’m sure we’ll talk about it. I mean, I’m seeing estimates that rebuilding is going to take several years, so it’ll probably come up again on the pod, but just wanted to kind of let everybody know we’re fine. And we’re grateful to be fine and shout out to all of our people who are not fine. And we’re here for you if we can be.

Sharona: So the easiest way to to go is the grading conference.com. It still works as a URL. It actually redirects to the Center for Grading Reforms website. But on the grading conference.com, you can find this year’s information and abstracts, submission is open, the conference is in June, and abstracts are due February 28th. So you do have some time, but they are open. We have really, I don’t know, it kind of feels like the conference is growing up a little bit, you know, like real professional.

Boz: Yeah, no, I think it grew up a lot last year. And as a result of that, we at the organizing committee, we had to grow up and get more sophisticated because the outpour of abstract submissions last year was. I mean, it was literally an exponential growth from the year before. I mean, it, it shocked all of us on the committee. It. It really, I don’t think I can explain how much additional work and how much surprise we were, we were like, holy crap, we have so many, and it really was. It was great. And at the same time, it was really kind of frustrating and sad cause we had to deny what we’ve never really denied any abstract before. Last year. Cause we just didn’t have.

Sharona: I wouldn’t say we didn’t deny any, but they would really have to be off topic, like not about grading reform.

Boz: Or no, there were, if someone had submitted multiple ones, that’s usually pick one or two, but that was. I don’t know if I count that as denying when you’re doing multiple ones.

Sharona: Right. We just picked the one that we think is going to be the best. So yeah. So this year with abstracts, we tried to get a lot more clarity on the types of presentation formats, what types of abstracts we’re looking for in those presentations what’s available as far as the zoom skills and how long those workshops and presentations are..

So if you go to the website and you click on abstract submission, you’ll see a whole thing about how to submit and what we’re looking for. And of course you can always reach out to us at the organizing team at info at the grading conference. com. If you have any questions about any of the things that we’ve done.

We’re also going to be looking for paper reviewers and volunteers to help host some of the sessions. So make sure that you’re on our email list. Again, you can email info at thegradingconference. com and we’d love to see you there. We’ve gotten quite a few questions about the cost of the conference and that is because registration is not up yet.

Rather it’s not open. There is now a registration page up. And just to let people know the entire conference, which is three days is $50. Or pay what you can if $50 is too much and student registrations are $25 and institutional registrations are for groups of ideally 10 or more and you reach out to us and we’ll explain that to you.

Boz: Yeah, and I’m sure we’ll definitely talk more details about that as the conference gets closer, so.

Sharona: So what are we talking about today? So we talked about the fires, we talked about the grading conference, but I think we have a topic for this today. And we don’t have a guest, which is unusual these days. So what are we talking about today?

Boz: Well, actually, I wanted to talk about you and your class. So we have been here

Sharona: People don’t want to hear about me.

Boz: Well, for this last semester, you got a new role this last semester, which we’ve talked about quite a bit, but one of the other things in that is you were forced to try something in the class that you and I have been teaching together for years, and it’s the class where we introduced alternative grading at Cal state LA.

So you were asked to pilot, to do an experiment on, using traditional grading in this class. Which I know you were just thrilled with. But that semester is over and I kind of wanted to ask you about some of the results. Well, I know we can’t get into too much detail, but just some overall comparisons of these two classes and what happened.

Sharona: Yeah, so this was an interesting emotional rollercoaster for me because. I want to give a little bit of background about why we had to do this experiment. In our institution, and by institution I mean the entire California State University system, we are grappling with our graduation rates and therefore our pass rates in our quantitative reasoning general education requirement.

Boz: So wait, but back up, what do you mean, what is remediation? What did, what impact was that on the actual students at Cal state systems?

Sharona: So math remediation is when students are determined in some way that they are coming to the university deficient in the basic mathematical skills they need to succeed in collegiate level math.

And if they scored below a certain level, they were required to take math classes that did not give them college level credit.

Boz: And there was up to three of those, correct?

Sharona: At Cal State LA, there were up to three. And so a bunch of research was done and it was found that this process was extremely inequitable. Actually schools that had higher levels of underrepresented minority populations used to have harsher requirements for testing out of these. So a school that had say a high black population, their cut score, the score you had to score on the placement test to get out of remediation, was higher than at a school that was predominantly white on the same test.

So a predominantly white school might have a cut score of 50, a predominantly black or Hispanic school would have a cut score of 60. Okay. So it was actually harder at a minority serving institution to test out of these remediation classes. And the problem was, number one, students were failing these classes at high rates and never even getting to collegiate level math. And there were time frames that if you didn’t get out within like a year and a half, you were not allowed to continue in college.

Like, this was a super harsh penalty. If you couldn’t do these lower level math classes that you had to pay for, but that did not give you collegiate level credit, you were kicked out of college.

And fed into Cal State LA. We are a feeder school into Cal State LA. I know how many of my students would take this math placement test and score into the bottom or the second lowest. And the amount of stress that they would have and trying to get through these courses.

And some of these courses were the most outrage…. Like I had one of my students come back to me and say, Hey, I’ve got one more chance to pass this class. Can you please tutor me? Cause you were the best math teacher I’ve had. I’d learn more from you. I was tutoring them on Roman numerals, right?

That was one of their units. Like being able to identify and add Roman numerals.

Sharona: So the first problem is the placement was harsh, the classes were hard. The second problem is they didn’t actually work mathematically. Students who got through them were not prepared for their first collegiate level math class.

So these classes that were "supposed" to prepare students didn’t work. So it was just.

Boz: And that’s really the original question and statistical experiment that College Bridge was doing and how we met through the slam program was looking exactly at this. It was taking students that would normally have gone through that remediation process, taking them as a senior in high school, giving them not the remedial, but the actual college level course with some additional supports, and trying to show that, yes, they can still do it without these remedial courses, that these remedial courses were not helping, they were a huge barrier. And we did. I mean, from year one, I matched or exceeded the overall Cal State pass rates.

Sharona: And so what happened is SLAM proved this, lots of other programs proved these courses didn’t work.

Because not everybody needs precalculus. Before it was all algebra and precalculus. Now there’s more options. There’s statistics. There’s quantitative reasoning. There’s discrete math. There’s things like that.

Now the classes in one sense have been wildly successful. So compared to when we had remediation, where if a student started in that second or third lowest class of remediation, the throughput pass rates were under 10%. Right? So a student who started in that lowest level, less than 10 percent made it all the way through collegiate math. It was slightly better with the higher courses, but still very, very low. 10 percent made it through 20 percent made it through.

So now we have 50 or 60 percent making it through, of the students who based on their scores would probably have placed in remediation.

So in the course that that we’re talking about today, there’s two flavors. There’s a supported course for students who, based on a multiple measure, are not college ready in whatever definition that means. And again, would have placed in remediation. And then there’s a group of students who would not have placed in remediation.

Okay. So we have a supported course and a non supported course. And the challenge is even since all the work we did, at one point towards the beginning, we had pass rates up to 70 percent in the supported course. Since the pandemic, those have fallen off dramatically. So people have been scratching their heads and saying, why is this happening? How can we improve it? And so the people who decided that the grading system was the problem had every good intention in the world. I want to give them that they had every good intention in the world. The argument was made that the alternative grading system itself had become a barrier to course success.

Boz: Based on, and why did they think that?

Sharona: They felt that that was true because they felt that students did not understand the grading system and they believe that grades are the motivator for student success.

Boz: And why did students not understand? Well, they can’t go to their Canvas grade book and see a percentage. They can’t see that 72. So if they don’t see the 72, how are they supposed to understand that they have a C? So, that was the crux of their argument, correct?

Sharona: It was. It was that everybody, quote unquote everybody, knows what a 72 percent means when you see it in a grade book. Now, you and I give a talk about grading as the misuse of mathematics and the measurement of student learning.

And we demonstrate mathematically that actually nobody understands what a 72 percent means in a electronic gradebook. And that most of the time until the very end of the semester, that 72 percent is a lie because Canvas’s default, and probably the default of a lot of other grading systems in a way that makes sense is that if an assignment has not yet been graded, it doesn’t show up in that weighted average. Because why would you put an assignment that hasn’t come up yet and hasn’t been graded in? Because it would deflate, so to speak, the percentage.

So anyways, there’s this assumption that, again, I argued vehemently against, that everybody understands what a 72 percent or an 82 percent in a gradebook means. And anyone who’s been listening to the pod for like more than a minute understands just how false all that is. But I have to admit, I was really kind of stressed last semester in the course of doing this because at various points, It looked like traditional grading was doing better.

So even all the research I’ve read, all the knowledge I have, all the stuff about grades are not motivational the way we think they are. All this stuff, it didn’t matter emotionally. I was like, Oh, crud. Like, what if traditional grading we have a better pass rate? Because at this point, for various reasons, the pass rate is the number one metric that we’re being measured on.

Boz: Which I can go on for a whole hour about why, although yes, an important metrics to look at in any kind of course evaluation. If that is the only thing you’re looking at, you are missing the point.

Sharona: I would agree. So I want to give a whole bunch of data because we’ve been talking vagueness and we’re already, 25 minutes into the episode. So let me get some details. Okay.

Boz: Yeah. So yeah, let’s, let’s get into the weeds. What happened by the time December rolls around and we are looking at both.

So there’s actually four classes that we’re going to talk about. So we have the supported classes and the two variations, the traditional and the alternative. And then we have the unsupported version of it. And again, we have traditional and alternative grading in those as well. So we have four different groups that we’re looking at.

Sharona: Right. So let me give you some overall data. First of all, the total number of students in all of these classes combined for the semester was 749. So this is a big group. This is a big data set. Broken into 32 sections of give or take 25. And of the 32 sections, exactly half were tasked with using alternative grading and half were used traditional grading. And that half was equally split between the supported and the non supported flavors. Okay.

Boz: So we did have maybe not the same amount of student sample size in each of the comparisons, but we did have a identical section counts. So, okay. Interesting.

Sharona: Now,

Boz: I didn’t know that I knew they were close. I did not know it was like a 16, 16 split.

Sharona: Right. And then on top of that, we had a total of 12 instructors. Eight of those instructors had both types of grading. So if they were teaching in the supported class, they had at least one alternative, at least one mastery. If they were teaching in the non supported, And then four instructors only had one type.

Two of those had traditional and two of those had alternative grading only.

Boz: And I was just full disclosure. I was one of the ones that only had alternative. I didn’t like

Sharona: Because people are not idiots and you would not have done the traditional.

Boz: Yeah. I would say, I would like to think that was by design. I don’t know if it was just because I am so limited on the timeframes of which ones I can teach.

Because again, my full time job, my real job is the high school job. So I’m limited to, basically the 4. 30 to what, however late they are.

Sharona: I’m pretty sure. They knew better, because you would not have agreed to do the traditional. So this is not a random, this was not a random assignment thing.

I mean, students, of course, didn’t know. They signed up for whatever. But this was definitely what you might call a matched block design in some ways. And then the other things that were the same is the courses were essentially almost exactly the same. We use the same learning outcomes. So it was still backwards designed based on learning outcomes.

It used the exact same assignments for the preparation, participation and practice. We still had the exact same clicker slides that we use across all of the teachers use the same clicker slides, exact same pacing.

Boz: Well, kind of the same pacing.

Sharona: We were teaching the new material on the same pace.

Boz: Yes.

Sharona: The material that was taught was on the same pace. There were differences of what happened in the classroom some weeks, especially, and I’ll go into that in a second. The courses were not faster or slower as far as the content delivery. So there were, and this course was designed from the ground up as an alternative graded course.

So as far as like, if you look at the four pillars of clearly defined learning outcomes, feedback loops, multiple attempts, those three pillars largely were still present in the traditionally graded class. So for example, there were quizzes, checkpoint quizzes that both classes did. And those quizzes were the same.

We used the same problems at the same time for the first two checkpoints for every single standard. And then the way that it changed is that the traditional class did not use marks indicate progress.

Boz: Yeah, but I do want to kind of point that point out that you just made because I do think it’s really important.

Sharona: And this is one of the very interesting things that happened during the semester is this word traditional grading? Those of us in the alternative grading community, we’ve sat down and we’ve said we have these four pillars and that if you have all four of those in some flavor, then you really are alternatively graded, but if you can’t get to all four, that’s okay.

Well, this "traditional" class had three of the four pillars. So it actually, in my view, really still kind of was an alternatively graded class, except this marks indicating progress thing of going back to points and percentages was like setting off a bomb on the class. So that was to me very interesting. Cause I was again in the middle of the semester, I was a little bit worried. I’m like, well, we have three of the four pillars?

Boz: Yeah. We, and we also had some retake abilities. We also were still collaboratively grading. We, I mean, even the traditionals, we were still getting together. So, anyone that’s taught in a course that is taught by multiple teachers, not just you, know that I might look at, and this is one of the experiments we do in some of our trainings, I might look at one Question and say, okay, it’s wrong, I’m going to take off three points. You might look at it and say, yeah, it’s wrong, I’m going to take off five points and someone else is going to look at it and say, Oh yeah, it’s wrong, but I’m only going to take one point off. That might not seem like a lot till you do that every single question and that adds up.

So even in the traditional graded classes, we were still doing this collaborative grading and coming to those agreements. I mean, these were, this is how we spend our Friday nights. As sad as that is.

Sharona: Yes. Okay. And then additionally in the quote unquote traditionally graded class every, we still had the grade based partially on those learning outcomes because there was 3 percent of the grade was on each of the sets of quizzes for each of our 10 major statistical learning outcomes.

So we were still really outcomes aligned even in the percentages. And every student got to take the quiz twice, that had that retake. So, so then it, this is where it came in. Some of the biggest differences came in. Okay. So the first thing is there were, I would say two huge differences in the courses that they called traditional grading, and only one of them was grading related.

So the first biggest difference is they put in a weighted category grading system. So 15 percent of the grade was based on our preparation assignments. 10 percent was on the participation assignments. 10 percent was on the practice assignments. 5 percent of the grade was on the presentation we do at the end of the semester.

So if you add that up, that’s 40 percent of the grade on non proctored, non checking for mastery except the presentation maybe assignments. And then 30 percent of the grade was on these quizzes, 3 percent each for the 10 standards. And then we had 30 percent of the grade on two traditional midterms. So 15 percent on an in class midterm, number one, number two, and then the final exam was the opportunity to redo those midterms and hopefully get them better.

Boz: Yeah. So I, in comparison, so you said, cause I want to make sure we kind of have this side by side comparison. You said the practice, the participation and the preparation that added up to 35%, correct?

Sharona: Yep.

Boz: So all three of those. In the, in the alternative graded class is all compiled into what we call the triple P, which is one of our learning targets.

So in comparison, it’s one out of 15 of our learning targets. So that is what?

Sharona: 6. 67 percent something like that.

Boz: Compared to 30. So there’s the first really big difference.

Sharona: And the argument that was made is, well, number one, students won’t succeed if they don’t do this preparation, participation, and practice stuff.

So essentially, we need to make it such a big part of the grade, because it’s the easy part to get, and therefore it’s going to bump their grades up. So put a pin in that, because that’s going to come back later.

Boz: There’s that control. There’s that,

Sharona: The behaviorism.

Boz: Yep.

Sharona: And then the other thing is I mentioned midterms. So one of the things that people have been uncomfortable about in our alternatively graded class, and in my mind it’s a reasonable lack of comfort, but not to be confused with actual grading practices, is our quizzes are asynchronous. They’re online, they’re asynchronous, and in an age of AI, we are starting to see increasing issues with that practice.

Boz: Yeah, and we’ve talked about these concerns we’ve got basically a mini series of AI episodes out there.

Sharona: So I have some sympathy for wanting to have a paper and pencil or at least a proctored environment exam. I have sympathy for that. The issue was they put it in because that’s what you do in a traditionally graded class.

And that’s just not the case anymore. I know plenty of traditionally graded classes that use online asynchronous quizzing. So on the part of the people working on the redesign, in my opinion, they confused traditional grading with traditional assessment practices.

Boz: Yeah.

Sharona: So, but other than that, like so much of this is similar, right? Retakes and the collaborative grading, the normed grading. And we even went so far as to try to calibrate how we graded the traditional exams and quizzes with the mastery. So we made decisions on how we did the points to attempt to keep the bar of passing, what’s good enough in a mastery class, is good enough in a traditional class. We tried to keep that similar.

Boz: Similar. Not necessarily, cause there were definitely some quizzes that, in fact, I even made the arguments that we shouldn’t be holding one necessarily to the same as the other because of the way that things broke down and I think I can think of three in particular, one where it made it easier to get mastery in the alternative, but the other two, it actually made it harder to get mastery in the alternative.

Sharona: So we did all of this and we’re watching the grades. And I got to admit there were moments in the semester where I was like, I don’t know what to do, but I want to get to the meat of it. And then like the results, cause we have the results, we have the results. And I want to be careful how I say this because some of this is data that I can’t release. Okay. So I’m going to talk about them relative to each other and not in absolute terms.

Boz: Yeah.

Sharona: Okay. So first of all, understand the pass rates between the supported and the non supported classes have persisted in being significantly different, which makes sense. Students in the supported class, even with support, are students who are coming with less skill in their bag already. We try to address that, we try to focus on what they have and not what they don’t have, but nonetheless, they are starting from a deeper hole. So, let’s start with the non supported class.

I was like, our pass rates, non supported are very high anyway. How much more room do we have to go? So without changing the grading system from this one that was designed, the mastery class beat the traditional class by about eight percentage points. So eight percentage points. Now I said without overrides because one of the things that became apparent during the semester, was that this 35 percent on the PPP was harming, dramatically harming a certain group of students.

Boz: Yeah, so I want to talk about that. Because that was one of the things that came up and that was one of the things that by the end, the same people that were saying, Oh, they didn’t understand non traditional grading was now saying, Oh, they didn’t understand how important the homework and stuff was.

Sharona: They didn’t understand the weighting.

Boz: Yes. Whichever. Like it.

Sharona: Yes.

Boz: But.

Sharona: With overrides.

Boz: But in fairness, because we designed this course, to be graded in an alternative fashion and not traditional grading. We never meant students to do all of these assignments. The idea of the triple P is give students a lot of different assignments, some that will help you prep for the lectures and the clicker slides, some based on the things we do in class, and some for practice. And then the student finds which ones of those help them the most. And then they do those. It was never, ever, ever designed for students to have to do all of these because there is a lot.

Sharona: Yeah, there’s too much.

Boz: And there’s a lot, because we wanted to give options to the students. That’s one of the foundational beliefs of a lot of alternative grading is people learn in different ways. No two people learn the same.

Sharona: Student autonomy and equity. Because some people have the time and space to prepare. Some people have the time and space to practice. Like, again, student autonomy and letting the system flex to adapt to the needs of the students without overburdening the instructors. That’s how it was designed.

And so now in all fairness to the traditional grading system, which I don’t want to be fair to, but I will, there was a huge number of students. I wouldn’t say huge, but there were definitely students for whom this 35 percent helped them pass. Because mathematically their quizzes and exams were not at a passing level but yet they were the students who would dot all the I’s and cross all the T’s and did do all this gargantuan work and had these very high percentages.

So the weighting worked in their favor. So, yes, it helped them pass. But I’m sitting here as the instructor going great, so I have students who cannot demonstrate they know this material on an exam, but I’m passing them through.

Boz: Well, it’s the group that we’ve talked about this, why we don’t have C’s in the alternative grading. Because usually the C’s are one of two groups. Students that have learned how to game the system and game the points and can play that game and can do all of those assignments and get all of those, those compliance points, so they just barely scrape through. Or it’s the students that actually do really well on the assessments and for whatever reason don’t do any of this stuff and it just kills the grade.

Sharona: So we saw that, by the way, in the grade distribution. So not only did the mastery class beat the traditional, depending on your measure, either by eight percentage points or five percentage points. But if you look at the individual grade categories, the A category in the mastery beat the A category in the traditional by 14 percentage points.

Boz: So, so it more than doubled the just overall pass rate difference.

Sharona: Difference. And the C’s in the mastery class, half of what the Cs were in the traditional. So we not only had worse pass rates, but now we have dramatically worse GPAs in the tradition in the non supported. Okay. This is just the non supported.

Let’s go to the supported. Okay. So again, without doing a grade override. Oh, and by the way, this was the group that was the driver. Everyone was basically happy with the pass rates in the non supported. If it had been just that, no one would have even bothered because it met every metric that was wanted.

Boz: Yeah.

Sharona: It was the supported class that people were horrified by. Now let me be clear, in both grading versions, neither of these pass rates were what I consider acceptable, okay? Part of that is that the demographic mix of our students have changed. We are still getting, at this point, we’re getting students for whom the COVID impact was in their algebra years, their first year algebra, their eighth, ninth grade years.

So I think we’re still seeing that filter through. So we definitely have more students coming in at lower initial skill levels. That being said, the mastery version beat the pass rate of the traditional, without the overrides, by 14 percentage points. Now we see a less of a difference on the grade distribution.

Boz: Yeah, it’s, it’s was higher, but we didn’t see the doubling that we saw in the unsupported class.

Sharona: Right.

Bosley: But we did, but the important part, we did see an even larger difference in the overall pass rate.

Sharona: Right. So a much larger difference in the overall pass rate. And even once we overrode it, Okay, so, so again, we had about 10 percent of the supported class, who, if we had not overridden this weighted grading system, they would have failed.

Boz: So what do you mean overridden? What did you do? Because I actually, if I remember right, you did two different things, did you not?

Sharona: Well, the first thing we did is we calculated the the grade both as designed. We also calculated it with just quizzes and exams, and then that we may have looked at it with just exams as well.

Boz: Yeah, so we took this traditional graded class that everyone understands and that, you know, is, sorry, that’s sarcasm. But yeah, we’re like, okay. We did all this compliance stuff because that’s what makes people do their stuff and it was hurting their grades. So let’s take that out and let’s see what they did with just the quizzes and the exams.

Sharona: And let’s be clear, we only did that for students that would help.

Boz: Yes.

Sharona: So that bothers me though, because if we were to look at the whole class just on exams, the pass rate might have been worse because there was a large contingent of students, especially students who ended up with a C, who ended up with a C because of the strength of their compliance grading.

Boz: I mean, we didn’t do that. We obviously would’ve never done that. ’cause that’s really that, that that wouldn’t be fair.

Sharona: Right? We would not do that to students, but it just reinforces that.

Boz: But yeah, so but.

Sharona: So we had to play games with the math.

Boz: Yeah. It comes down to this, if we played as with playing multiple games with taking out the compliance and just looking at quizzes and exams. And even just looking at just the quizzes or just the exams?

Sharona: Just the exams.

Boz: So two different major, so if we just looked at exams, how many exams were there?

Sharona: Two, plus the final as a potential retake.

Boz: So I’ve got two midterms and a final mm-hmm . Where the final isn’t a new grade, it’s replacing potentially the other two midterms.

So three, three data points is what my grade is. Based on even doing all that, what was the highest, like, what’s the difference between that highest pass rate of traditional versus the non traditional, which we didn’t play any games with. We didn’t like it’s what it’s always.

Sharona: So what we ended up with the final grades that were reported, the pass rate and the mastery was five percentage points higher than the non mastery. But as a, that doesn’t sound like a lot, but as a proportion of the total, it was actually a huge component of the total pass rate. So I have to be vague. I’m sorry. Cause I don’t have authority to release the actual rates, but it was. I would say on, based on a significance test, it was significantly higher against the best option of all of the games with the math.

Boz: Yeah. So we’re, we were looking at with.

Sharona: But here’s the thing. Playing those games, seven percentage points from one game to the next.

Boz: So.

Sharona: As originally versus the way it ended up, seven percentage points. And again, these pass rates are not good. So that’s a huge fraction of the total pass rate. Now, on top of all of this we have to think about the psychology. Because one of the big problems is that we had huge numbers of students who just stopped trying in the traditional, because mathematically, there was no path forward.

So at certain points in the middle of the semester, I was looking at the grades, and I’m like, we have this pass rate and there’s almost nowhere to go but down. Whereas in the mastery, there’s nowhere to go but up because in week 12, I’m like, yeah, X percentage of the class is passing. That’s a guarantee.

That pass rate only goes up from there, whereas the pass rate in the traditional almost always went down from that same exact point in the semester because people kept giving up. And also mathematically, especially at the higher end, if you have students who are up in like the higher 70s? It’s almost impossible to get that up from about week 12, and it’s very easy to bring it down.

Boz: Were there any other interesting results from this?

Sharona: Yes. So one of the most interesting results is, with one exception that was probably based on two students, every single instructor, every one of them that had both types, had better pass rates in their mastery version than their traditional the same instructor

Boz: And let’s be clear. You and I are extremely biased. Like if we were asked to teach both of these classes, it would not be a fair comparison I don’t know.

Sharona: I won’t even do it. So yeah.

Boz: There are instructors in this group that we’re teaching this that do not feel anywhere nearly as passionate as we do. And there were some instructors there that absolutely despise alternative grading. That are almost as biased as we are for traditional grading. So when we, and yes, some of those teachers and I love them to death, you have no idea how many times we’ve argued. But even their classes, when they had both and compared to them selves, the alternative grades were better.

Sharona: Now, and there’s another problem that we’ve been trying to address, and we just haven’t been able to crack it, is this idea of instructor roulette.

Okay, so we have a course that is about as controlled and similar as it is possible to make. I mean, I can’t swear because I don’t go in and like, I don’t spy on the instructors, but as far as I know, and part of it I know very well, all of these assessments are identical. All the designs are identical, we have ranges of pass rates that vary by instructor that are very high.

So those overall numbers where, oh, we did eight percentage points better. If you do just individuals instructor things, there are several instructors who their mastery rate passing is double their traditional. Double. Or 50 percent higher. And then some instructors, like the person I’m looking at on the screen right now, just happen to have really high pass rates in general.

But I mean, you didn’t have a traditional class, but you kicked it this semester. You really kicked it. Kudos to you. I think you had the highest individual section pass rates, except one person had one section.

Boz: I thought that was, I knew I was higher than the average, but.

Sharona: Yeah. No, there was one section that did better than you out of all of the sections. So, and I know for a fact you’re not grade inflating because I see your assessments and your feedback. So that’s there too. But yeah, I I’m stunned by the differences and how much the differences get hidden by these overall numbers.

Boz: Yeah.

Sharona: So, you know, it’s just a really interesting thing. And the other thing to remember is, like, as much as we tried to do a match block design and things, we saw some very interesting things that people have, who’ve been in this alternative grading journey with us.

Like, we have conflated some of the things. And the reasons we do it. So they think we’re doing it because of the grading when we’re really doing it because we’re also trained in equity. We’re also trained in student autonomy. We’re also trained in motivation theory. We don’t always go into all those details. When they flipped it back to traditional, so many of those behaviors started to come out. Like well, the reason we can’t improve the mastery is we give them too much student autonomy.

So we’re going to force them to not have autonomy. And it doesn’t work. And we know that, but, yeah. Sometimes I just feel like I’m overwhelming the people I’m talking to when I’m like, well, this research paper said this and this 20 years of research said that and so yeah.

So head to head the scores are in and it’s mastery grading or alternative grading for the knockout. Because let me tell you this semester, I haven’t heard a word about whether or not we’re doing traditional or trying to fix it.

Now, we’ll see if that stays true for the fall.

Boz: But I’m curious about the fall because one of the weird and I, I know you say there’s reasons for it. I still think it’s weird, but there is a huge size discrepancy of how many students take our gen ed before math classes in the fall versus the spring.

So, whereas we had 32 sessions, sections, just of our math 10 90 courses last fall. How many do you have coming up this spring?

Sharona: I believe I have six.

Boz: Yeah.

Sharona: But again, that is definitely advising because our institution has taken some research that says that students who don’t complete their math gen ed in their first year are much more likely not to complete college.

And because we know we have these pass rate problems. We can’t take the risk and we don’t know who’s at risk on the pass rate. We can’t let them take it in the spring because they might fail it and then they don’t get it done in their first year. So the psychology and the advising has determined that every single student, every single incoming freshmen needs to take their, their math, quantitative reasoning requirement in the fall.

And they have paired it, I believe with one of the other GEs that students then are told to take in the spring.

Boz: Yeah. Well, like I said, We can debate some of that logic, but the point is there is a huge, there’s a huge difference. It’s, it’s, you know, almost a one to five difference for every five classes there are in the fall.

There’s only one in the spring. So I am curious with the math leadership to see what happens. I’m hopeful that the conversation is dead. Like you are right. But I won’t believe it’s dead until we get to fall and it’s not come up.

Sharona: I don’t see how it’s not dead because yes, it was 14 percentage points. But again, this does not have a good pass rate overall. So that’s a huge proportional increase.

Boz: It’s a hundred years of research to back up what we were doing. I

Sharona: know, but here’s the thing, and I’m sure that many of our people listening, if it’s not at your institution, it doesn’t count because you have unicorns as students.

And we now have unicorn poop proof. Gosh, that was very.

Boz: You have unicorn poop.

Sharona: Yup. We have unicorn poop proof that even just the component of the grading system, just two things, but primarily one, just points and percentages, can be, even with all the other good stuff, the retakes, the alignment to the learning outcomes, the use of feedback, although I would argue that the points of percentages destroyed the feedback. So that’s a big part of it.

Boz: Yeah.

Sharona: But even just changing the points of percentages and to some degree insisting on traditional exams decimated this group of students and I feel terrible for them and I hope they come back.

And if they come back, they will come back into a mastery graded structure, alternatively graded structure. I’m having trouble with my language because we still call it mastery at the school because that’s what Canvas uses. But yeah, the results are in. Knockout blow for alternative grading. I’m going to try to write it up.

I have to get an IRB letter, but I’m going to try to publish this because this data is pretty cool at the end of the day. But I got to tell you, there were some times early on where I was nervous because student behaviors early on are different. Right? Especially because you cannot pass our class, like you cannot be shown to be passing until about week 11 because that’s the first time we’ve assessed enough that you’re passing officially.

So all the stuff before that is like, well, they have three and that’s, by this time, that’s likely to pass. But, you know, I have to kind of put all this language around it to say, well, it’s not guaranteed because you need seven and you know, ah, So I hope that people found this interesting. I hope people listened all the way to the end I don’t know. Any last thoughts about this experience?

Boz: You know this is something that you and I have talked about the the whole semester with just the the ridiculousness of it and that the fact that we’re doing this pilot on something that’s been done for a hundred years.

I really do. I hope this is the end of it. And I hope you are able to publish it. I do think this will be, it’s got a decent sample size and I think it would be some great stuff to actually get out there. Cause that’s one of the big complaints, even though it’s gotten better is that. We don’t have a lot of research to show. Yes, there’s lots of research to show traditional is bad. We don’t have a lot of research to show that alternative is, you know, any better.

Sharona: Well, and, and research that is not based on a single instructor in a single institutional setting. With a single group of students and that is the good thing here.

We’ve got 749 students in four different, essentially matched block design as best we could. They really tried to do a good experiment with matching the instructors.

Boz: Careful.

Sharona: They tried. I said.

Boz: This is still not an experiment.

Sharona: This is true, but they tried to do a good study.

Boz: Yes, because we did not. We did not randomly assign the students, which is, you know, that would be an interesting discussion. There is a huge impact on. Like that does have a bigger impact than, than I think non researchers really understand, but we did also, and we can look more at this, but to try to deal with that is we did have a lot of similar time schedules.

So, if there was an eight o’clock traditional, there was a early morning non traditional. So.

Sharona: And I think within the individual instructor, they flipped a coin as to which section was mastery and which section was alternative. So it’s, it’s the best we could do. Yeah, it was not possible to randomly assign, but it was the best because.

Boz: We can’t randomly assign students their schedules.

Sharona: But it’s about as random as you can get given the constraints. So, yeah, not a true, not a true experiment in that sense, and yet the best design they could come up with. And, and by the way, one of the reasons we didn’t yank a bunch of that triple P out, because the argument was, if we cut down the amount required, We might accidentally cut out the stuff that actually helped students succeed, and therefore it would no longer be a true comparison.

Boz: Like I said, again, it was designed not to have to have students do every bit of it. So we will be talking more about the grading conference as we get closer to, you know, some of those deadlines. And we’ve got some interesting interviews coming up.

Sharona: Please share your thoughts and comments about this episode by commenting on this episodes page on our website.

www. thegradingpod. com or you can share with us publicly on Twitter, Facebook, or Instagram. If you would like to suggest a future topic for the show or would like to be considered as a potential guest for the show, please use the contact us form on our website. The Grading Podcast is created and produced by Robert Bosley and Sharona Krinsky.

The full transcript of this episode is available on our website.

Boz: The views expressed here are those of the host and our guest. These views are not necessarily endorsed by the Cal State system or by the Los Angeles Unified School District.