The Grading Podcast

147 – More Complicated Story

===

Boz: The good news, grades went up abroad. And then the next header, the complex news

Sharona: The complicated news, opportunity gaps persisted

Boz: So not bad news, not bad news, but-

Sharona: No, but not what they were hoping.

Boz: Welcome to The Grading Podcast, where we’ll take a critical lens to the methods of assessing students’ learning, from traditional grading to alternative methods of grading. We’ll look at how grades impact our classrooms and our students’ success. I’m Robert Bosley, a high school math teacher, instructional coach, intervention specialist, and instructional designer in the Los Angeles Unified School District and with Cal State LA.

Sharona: And I’m Sharona Krinsky, a math instructor at Cal State Los Angeles, faculty coach, and instructional designer. Whether you work in higher ed or K-12, whatever your discipline is, whether you are a teacher, a coach, or an administrator, this podcast is for you. Each week, you will get the practical, detailed information you need to be able to actually implement effective grading practices in your class and at your institution.

Boz: Hello and welcome back to The Grading Podcast. I’m Robert Bosley, one of your two co-hosts, and with me, as always, Sharona Krinsky. How you doing today, Sharona?

Sharona: I am doing well, and I’m very glad to say that I’m doing well, because I’m been recovering from a minor medical procedure for the last week and a half that’s had me a little laid up. So wasn’t sure I was gonna be up to this, but I am. I’m very excited about that, so here I am ready to hit the hot seat. How are you doing? What are you up to?

Boz: First, I’m glad to hear that you are recovering, so I’m glad to have you here and able to record, and I hope your recovery continues to go well.

Sharona: Thank you. …

Boz: This is an interesting day. So at the time of recording of this, my daughter, although she graduated last year, is going to prom tonight. So- … really excited. She’s going with, one of her judo buddies that is more like a brother than, a friend. And interesting enough is actually going to prom at a school that is ran by a principal that used to be one of my principals and is still a good friend of mine. So it was funny when I called him up and said, “Hey, my daughter wants to go to your prom. What do we have to do?”

Sharona: Nice. Yeah. I think prom is very different than when we went to prom back in the day. The process used to be you just- Invited whoever you wanted, and the person bought two tickets, and that was it. And I think it’s different now. I don’t know.

Boz: Oh, yeah. I think it’s- No we had to do this whole guest pass thing and the paperwork, and yeah. Which I get, but.

Sharona: My kids very intentionally avoided their proms, which was very easy for Nathan because it was the pandemic, the beginning of the pandemic. Yeah. And Aaron, I managed to get him to say yes to a girl who asked him, I think, to homecoming or one of the … it was a formal dance, but it wasn’t prom. But then he was like, “Absolutely not for prom, Mom.” All right. Okay. All right. I failed the female species, but- … I did my best. Did my best, y’all. Oh, boy. I did wanna mention at the top of the show here, though, we mentioned this, I think, last week, but there are two upcoming faculty learning community things coming up that I just wanted to mention as a side note before we get into the episode. Is that okay?

Boz: Of course.

Sharona: We’re gonna put it in the show notes, but this week, I believe, the faculty learning community run by Drew Lewis and Melanie Lanahan is starting up again. It’s a really great introduction to redesigning your course to use alternative grading. No experience necessary, but it’s good if you have at least identified a course and have some thoughts on what you might wanna do. So go to our website at thecenterforgradingreform.org and check that out. And then our one specifically for math instructors that’s a little bit more intensive that runs in the fall is also open through the Math Association of America. So I just wanted to make sure people knew that there’s some great opportunities to join a community of practice and learn how to do this in the next bit of time.

Boz: And speaking of Melanie and if that name sounds familiar to our audience she is one of the grading conference organizers. But did you see what she posted in Slack the other day?

Sharona: I did, and I just … Sometimes in this job I just get this little moment of, “I can’t believe this.” So do you wanna talk about it, or do you want me to talk about what she posted?

Boz: Yeah, so Melanie went to a webinar through, I think it, it was called Top Hat. Where Dr. Susan Blum was one of the I don’t know if she was the presenter or keynote or one of the presenters, but yeah Dr. Susan Blum was presenting, and as she was doing this and giving out like some of her links and things after the webinar, actually posted a registration link to the grading conference. So I wanna give out just a– that is, again, so cool. Dr. Susan Blum has been such an inspiration to this community, and she has been such a big supporter of the grading conference. It really is an unreal feeling. Last year, she showed up and basically went to every single session. It was absolutely amazing, and to hear that she is promoting the conference on her LinkedIn and in other places, it is just… it’s so cool to hear that, and just a big thank you if she happens to be listening to this, just a huge thank you. That’s incredible.

Sharona: Yeah. And it’s a really– It was a interesting– I wish I’d been able to go to this. I, like I said, I’ve been recovering, but this webinar that she gave the title of it is “Alternative Assessment or the End of Learning.” And it’s basically that the theory of it i- is AI is changing learning, assessment should change, too. And I definitely am on board with this idea that AI is so disruptive that we really– alternative grading is no longer an option, it’s a necessity. So I’m sure we’ll do more of that in the future, but I just-

Boz: Oh, it’s not like we haven’t done some of that in the past as well.

Sharona: No, but I had a great conversation this week with someone from my university, and we were talking about this disruptive nature, and it’s just so stark. Anyway, I don’t wanna go into it right now, but I love the fact that Susan Blum is talking about this as well. Yeah all.

Boz: Right. So if you don’t wanna talk about that, what do you wanna talk about today?

Sharona: Thank you for asking. So we saw an incredible article on the Grading for Growth blog a few weeks ago, and I really wanted to dive into it because I think it is such a well-done study. The title of it is “Specifications Grading and Equity: A More Complicated Story Than We Hoped.” I absolutely love articles that give us an opportunity to look at the nuance and not really stay at this sort of 50,000-foot level that sometimes we get trapped at. So I wanna see if you’re willing to take a deep dive into this article. It’s by Brandon Yik, Lisa Markochuk Oh, I hope I said that

Boz: Sorry if we did mispronunciation, if we did.

Sharona: Yeah, and Marilyn Staines. And really excited to dive into this article.

Boz: Yeah and it’s interesting especially Brandon, he has actually presented– He, he’s come to the conference many years. He’s presented at least two or three years. I think some of the others in this might have as well, but I know Brandon has been very active in the Grading Conference. In fact, I think he is presenting again this year, and it might even be some of the stuff that this article is about, I think.

Sharona: Could be, I don’t know the conference schedule well enough, to be honest. Although, before we do get into this article, I do wanna call out this word equity, because we’re gonna talk about specific dimensions of equity in this article, but there’s so many different dimensions of equity. And our keynote for the first day of the conference, Sarah Silverman, is gonna be talking about a different one, which is what happens when different equity needs from different groups conflict with each other and with the instructor’s needs. So we’re not gonna talk about that today, but I so recommend that keynote.

Boz: Yeah, equity has really become this huge umbrella term. And you’re right, there are a lot of different, when we really dive into what do we mean by equity, there’s a lot of different groups, a lot of different viewpoints to come at this from. And sometimes those actually have opposing needs, which is really interesting. A- and again, what makes our jobs as educators so daggum difficult.

Sharona: Yeah. It really is hard because a lot of the work that I did in the pandemic to accommodate all these access needs, I am now finding is enabling behaviors on some of my students that are bad for them.

Boz: Yeah and we talked about that ’cause we’ve had Silverman on the podcast, and we– that was really a lot of what we talked about was, that what can be good for one group can actually be harmful for another group, and just the need of really being informed when you’re making any kind of decisions, whether it’s in the grading or your pedagogical approaches or just how you run your class. So it w- it was an interesting episode. I am also really looking forward to her keynote, so Be sure to, if you haven’t already registered for the conference, be sure to do that and we hope to see you all at that keynote

Sharona: And make sure to tell your institution about it. I think we’re up to a record 19 or 20 institutional registrations this year.

Boz: Oh, wow.

Sharona: So reach out to us at info@thegradingconference.com if you wanna get more information about the institutional registrations, so that’s pretty exciting. Why, so why don’t you introduce us to this article?

Boz: So yeah, like you said, this came out on the Grading for Growth blog, which is one of our favorite sources to steal from.

Sharona: Absolutely. …

Boz: Came out on April 20th, so it is not even quite two weeks old yet. And it really is, it’s the three people that you were talking about is a team from University of Virginia, where they had taken a large enrollment general chemistry lab course and redesigned it for a specs grading

Sharona: Which we’re seeing a lot of. We’re seeing specs being the default redesign on these science lab courses, which I think is very interesting. Makes sense, but-

Sharona: And I wanna point out here at the very beginning of this article, this is the blog post but they actually have a published article in the JACS, which I wish I had looked up ahead of time, Journal of something chemistry. Sorry. I’ll look it up while we’re, when, while we’re talking, but this is a published article in a peer review journal. It’s the Journal of the American Chemical Society.

Sorry. JACS.

Boz: And we’ll have links to, to this if it’s not behind a paywall on the show notes, right?

Sharona: Yep. Even if it is behind a paywall, we’ll give you a link. But yes, absolutely

Boz: Besides just overall results, they were also looking at four identified opportunity gap areas. Those four things specifically that the- they were looking at were gender, first generation status, underrepresented minorities, also referred to as URM, and transfer student status. So those, again, we talked at the top of the show, there’s so many different facets that you can look at. These are the four that they, were specifically interested in, and they looked at these results with those four groups in mind.

Sharona: Now, what I love about this, though, is they also took it a step further, and they have an intersectional lens called a systemic advantage index that combines all of these identities into a single measure. So the more of these identities an individual student has, the different their index score.

Boz: Which is really interesting. It makes sense ’cause, a student can be a, a female and a first gen or, can be a underrepresented minority male who’s also a transfer student. I’ve never seen this, I’ve never heard of this S-A-I, or Systematic Advantage Index, but it makes complete sense, and yeah, I love that they did that.

Sharona: I think I might be putting in a data request to our data center to say, “Hey, can you create this thing for us?” Because we have, we also have multiple equity variables identified. We have four actually, we have five. The one we have that you and I don’t usually use is transfer status, because the course that we work with is a first year freshman course, so we have so few transfer students in that course that it doesn’t make any sense. The one that we also have that they don’t list in this particular study is Pell eligibility, which is a proxy for economic disadvantage.

Boz: Yeah. And but we’re gonna be comparing some of our results to some of the results that they found here. We also at one point basically had a head-to-head where we looked at our course through a traditional grading lens and the alternative grading lens that it was designed for. We’ve done a whole episode on that. The results were pretty one-sided and pretty telling. But yeah, so we’re, we are gonna be bringing up some of that up as we go through some of the results that Brandon and his team found.

Sharona: So I wanted to take a minute to give the context of their course before as we start to dive into this article.

Boz: Okay.

Sharona: Does that make sense?

Boz: Oh, absolutely.

Sharona: So they are looking at their general chemistry lab courses. These are one-credit courses that are taken by most first and second year students at the University of Virginia, typically alongside the corresponding lecture course. The grades that the students get in the lab courses are separate from the grades they get in the lecture course, which is again is relatively typical. They enroll about 1,600 to 1,700 students in the fall in their Gen Chem Lab course for one, and then 8 to 900 in Gen Chem 2 in the spring.

Boz: So these are good sized courses. This is not a one-off class that they’re talking about here.

Sharona: And they don’t really specify the size of the lectures versus the labs. My experience with these types of R1 institutions is that the lectures could be anywhere from 100 to as much as 800 in a course, and the, but the labs are pretty small. So the lab courses are usually 25, 35, 40. I don’t know in this case how big there were, but there’s probably a lot of different lab instructors-

Boz: Yeah. …

Sharona: That we’re looking at. And they closely followed the framework laid out by Linda Nilson in her Specifications Grading book. S- so the individual assessments were pass/fail based on clearly defined expectations aligned with learning objectives, and they had a bundling system, so in order to get higher grades, you had to demonstrate mastery of more objectives. It’s more that they said that they reorganized these than they made them completely different-

Boz: Yeah. …

Sharona: Between when they had done by the traditional. And so it’s more structuring and evaluating than a wholesale redesign of the course content.

Sharona: Absolutely. And so even though I don’t know that they go into it, we’re just aware that this is a thing. And so they were trying to figure out, they wanted grades to improve. They had a whole bunch of things. They wanted to close opportunity gaps. So the question is, what happened?

Boz: Yeah. So we’ve set it up enough. We kinda did enough foreplay here. Let’s, what were some of the results?

Sharona: So I like their headers that they have in this article. The good news, grades went up across the board, and that’s great. We want, better grades are better than worse grades. That’s a good thing, if it’s not because we dropped our standards, right? Yeah. Because that’s the number one accusation we always get. “Okay, your grades went up. How much did you lower your standards?”

Boz: We’ve actually had several episodes, including- … those Harvard episodes where we talked about that. But we’re going to assume because, like they said in this their grading was really a restructure and bundling of their learning targets and how they assess those learning targets. It does not sound like they completely redesigned the course and changed what they were looking for. So we’re going to assume for time being that the rigor of this course was pretty similar to-

Sharona: Similar or harder.

Boz: Yeah.

Sharona: My experience is that it usually gets actually harder.

Boz: Yeah.

Sharona: So they saw grades increase for URM students. They saw grades increase for first-gen students. They saw grades, and these are, like, A’s. They saw them for transfer students, and they were all statistically significant

Boz: They saw a massive one in transfer students.

Sharona: Yes.

Boz: Which again, maybe because that’s not something we’ve ever looked at, but I found that really interesting that for whatever reason, the transfer students had a huge increase, percentage of students that received A’s.

Sharona: And I think it’s very interesting. They link to an article, but earning a higher grade so not just passing, but earning a higher grade in a foundational science course increases the likelihood that a student continues in a STEM major.

Boz: Yep.

Sharona: And they’ve linked to that article as well. So I thought that was interesting. So it’s not just passing, which in our course we, we look at pass rates a lot because our course is not for STEM majors. This is a course for STEM majors. Yeah. And so not just passing, but getting one of those higher grades is also important.

Sharona: Absolutely. Absolutely. So not only do we see significant improvements in pass rates, but what grades those passing students got skewed so much towards A’s and B’s. Our C’s were-

Boz: Al- almost non-existent. They- Yeah … they really were. And we’ve talked a lot about our hypotheses of why that is, the kind of characteristics that the C students in a traditional class kind of fell into, how one group of students that profile that would get C’s are the ones that lived on the partial credits, on the just doing a bunch of stuff and never really getting anything, but just kinda living on those participation points and all the compliance stuff, and then getting partial credits. Yeah. And those students in our class didn’t pass.

Sharona: No. They, it’s not that they didn’t pass. Yeah. They separated themselves-

Boz: Yeah. …

Sharona: Into the ones who were just gaming and the ones who were gaming, but also just needed a little extra time to actually get it.

Boz: Yeah.

Sharona: A-

Boz: And the other kind of profile was the opposite. It was the student that didn’t do all that compliance stuff, didn’t do every little thing but actually knew the material. But because of all the other stuff, for whatever reason, they weren’t either choosing not to doing it or weren’t able because of life. Yeah. And those students in our classes’ grades shot way up.

Sharona: So this next part of the article is where their data and our data diverged

Boz: Yeah. But- … I wanna go back just for a sec- No. Okay … ’cause I don’t wanna undersell this. Th- this raising grades, the tide that raises all ship kinda thing, that is not to be undersold. That is an incredibly important finding, and it’s a finding that we’ve seen, it’s a finding that they have statistical evidence here. It’s a finding that so many of our other colleagues in different fields have found. So I don’t wanna undersell that. That is a really important point.

Sharona: I agree. We talk, especially when we do a lot of the equity work I’ve been at, they talk about how something that raises all boats is not enough, but it’s a really good start. Yeah. We really do want to raise all boats. It’s great if we can raise the boats and close the gaps. That’s awesome. But I would hate to close the gaps but not raise the boats. Yeah, exactly. I, can we get both, please? So in their case, they definitely saw the ri- the, a, a lifting tide, it rose all the boats, and it didn’t close the gaps as much as they would’ve liked.

Boz: Yeah. So let’s get into the… ‘ In this article they really go into detail to the background, some of what they did. I think they undersell themselves a little bit with the good news. But yeah the next header for this and I love how they wrote this. So they had the good news, grades went up abroad, and then the next header, the complex news.

Sharona: The complicated news, opportunity gaps persisted.

Boz: So not bad news. Not bad news, but-

Sharona: No … But not what they were hoping.

Boz: Yeah.

Sharona: Because they found that the systematically advantaged and systematically disadvantaged students did not meaningfully close. And I love what they write here, because they talk about the fact that they looked at effect sizes and things like that, and if you look at their paper, they did a robust statistical analysis on these gaps to show that these gaps were statistically meaningful. And people have heard me complain, I’m not gonna swear about the type of complaining I do, but it really bothers me. When I first started doing equity work, I requested definitions. What was a gap? And the people that I was working with could not define it. They were like just look, the numbers are different.” And I’m like, “Different numbers does not a gap define.” Obviously if they’re the same, then there’s no gap, but just because they’re different, you need to look at the statistics.

Boz: Yeah. There, there’s a thing called statistical significance, and there’s a reason why we look at it. There is natural variability in data. It’s supposed to be. Yeah.

Sharona: Yeah, there is natural d- variability, and there’s also effect size, which is you could have variability in the data, but the effect size being small means it’s not relevant difference. So there’s lots of statistics you can apply to this, but I love the fact that they said that the effect sizes remained unchanged. And they then did some modeling of the probability of course success, and they said the disparities were stark under both grading systems, both the traditional and the specifications. The odds of a transfer student succeeding in the first semester course in traditional grading were about 1/11th of those as a first year admit student. Under specs grading, they increased, but were still substantially lower odds of success. So that’s disturbing, however, in my opinion, and I do think we should say this here, what this really speaks to is these gaps are complicated. It’s not just grading. It’s not just the instructor. It’s not just the course. Yeah. Because a lot of blame, at least in my environment, falls on the course, the individual instructor, the individual department. You need to fix this. It’s your course doing this. And I think it’s so much more complicated than that.

Boz: Yeah. And, we’ve said this before, and we’re gonna continue saying this, changing your grading, while we think extremely important, is not a silver bullet. It is not going to fix everything. In fact, if all you change is your grading, yes, you’re gonna have some effect, but really what we have found is I like to refer to it as the linchpin that made everything else work. You talk about, the fire blanket that it gets, the grading gets out of the way of everything else you’re trying to do.

Sharona: I’m definitely in love with my own new metaphor of the fire blanket smothering the fires of learning. It’s just laying on top of the whole course, and the course in my mind is an obstacle course, so you c- if the fire blanket’s on top of it, you can’t see it. It’s taking all the air out of the room. It’s smooshing down everything. You take it off. Now you can see the course. You can see where there’s fires. You can see where it’s at, and there’s opportunity now, but there’s a lot more work to do. But I do think it is the fire blanket. I think without it, it’s almost impossible to do much of the change that we need.

Boz: Yeah. And you’ve brought this up. You’ve, talked about the work that your late great mother did for so many years really decades, and how you tried a lot of it and a lot of other great, research-based pedagogy, and you just, you weren’t seeing the results you were wanting until you did the grading change.

Sharona: Yeah, and I’m having a little bit of a interesting experience this semester. My, my grades, my pass rates are not going to be good this semester. I was not able to retain the vast majority of my students in my class. We just had exam three, and I think 12 of my 29 students took it.

Boz: Yeah.

Sharona: But it’s because it’s complicated.

Boz: It’s complicated, and, to be honest, you didn’t really get to run the course the way you would have ran it if you had complete control.

Sharona: That may be.

Boz: You played a little bit in, in the games where you could, but …

Sharona: Yeah, but I lost too many of them too early for it to be that.

Boz: Yeah.

Sharona: So I, I just think that the reality is the whole experience is complicated right now. I will tell you, the students that I have retained are having the best experience of their life, regardless of what grade they end up getting. So I feel really good about that, but it’s complicated, and I’m okay with that. So then they have a third heading, though. I wanna get back to the article. The third heading in this section, “An important nuance: Lower grades also increased for some groups.” The proportion of students getting low grades, C, D, F or withdrawal, also increased slightly for some groups. They say that the increase was larger for URM and first-gen students than for their more advantaged peers. I think what they’re seeing, which we have seen, is that bimodal separation.

Boz: Yeah. But I’m curious ’cause when they say low grades, they did include the C’s and I get C is not a great grade, but it is a passing grade. I’m curious what that data would look like if they took the C out of that group.

Sharona: I’m … I question that as well. But they did point out, they say here, “It underscores why implementation decisions matter.” So I think they’re questioning some of the specific design decisions they made within the course.

Boz: Yeah.

Sharona: Because they say, “The pass/fail structure of specifications grading may interact with students’ resources, time, and prior preparation in ways that they don’t yet understand.” So that’s one of the things that we noticed in our course, and the reason that we don’t use those first two semesters of data much is we made dramatic design changes in our course over multiple semesters.

Boz: Yeah, I mean- So … we’ve continued to tweak pretty much every semester, but you’re right, we had and we’ve talked about it. We, our, the, our first attempts were not good.

Sharona: Interesting, to say the least.

Boz: But yeah, we made some massive structural changes, and even though we kept tweaking pretty much every single semester, we didn’t see the massive shifts that we did at that year three mark.

Sharona: I also… but I wanna go back, that we, we have actually higher Fs than we might have had under traditional grading. Not really, I wouldn’t say higher. I would say out of the DFW category- we don’t have very many Ds. Like, when they drop below a passing rate they tend to drop all the way down to the F.

Boz: Yeah.

Sharona: So there is that split, and I don’t like to track degrees of failure, right? So I don’t track a whole lot. I can, but Ds versus Fs once you drop below passing. But of course, our course is also ABC no credit.

Boz: Yeah.

Sharona: So once you hit that no credit, I guess what I’m looking at, the people in the no credit group tend to be way down at absolutely nothing, as opposed to I’ve got five or six or seven when I need nine.

Boz: I- if our course was an ABCDF instead of an ABC no credit and we looked at where the traditional D sat, you’re right. We don’t have students in that group. We just don’t. I mean- … most of our students that are failing at least in my courses, and I think you’ve seen the same when you were coordinating the course and looked at the data across all sections, is the students that are failing they’re at zero, one, two, maybe. I don’t even have that many threes.

Boz: Yeah.

Sharona: And a- and we’re looking at anything from zero to six standards mastered. Half of them had zero.

Boz: Yeah.

Sharona: One student… No, sorry, a sorry, 4% of the students had one, it’s just very stark. Whereas- In the traditionally graded class, which I don’t have a split out, but just in points-based it, first of all, it wasn’t 44% of the class that had it, 48 and a half percent of the class got that. And I can’t distinguish. I don’t know. I don’t have the d- I could go back and look. I have details somewhere, but they’re all just lumped into that, so I can’t even really tell you. But it’s interesting. But the supported class had a 44% no credit rate. The unsupported class, the class of students who were deemed college ready in mathematics, still lots of URM, lots of first gen, all that kind of stuff, but we only had a 14% no credit.

Boz: Yep.

Sharona: So we’re still trying to crack that, if a student comes and their multiple measures show that they’re not prepared for college mathematics, I’m glad to have a 56% pass rate, but a 56% pass rate versus an 88% pass rate, that’s brutal.

Boz: A- and like we said, this is not something that our grading system alone is gonna fix. That, that is looking at how we use that supported time in our other pedagogical decision making and choices.

Sharona: It’s not gonna fix it alone, but this change alone in our data went from, so our standards based one was a 56% pass rate. Our traditional one was a 43% pass rate.

Boz: Yeah. And-

Sharona: It does help …

Boz: it does help. And that, like we designed this course with standards-based grading in mind. So a lot of those same practices that we use, a lot of those same things based on the four pillars of alternative grading did find its way to the traditional graded course.

Sharona: Yeah.

Boz: And in fact, if we go back to, even though it’s a very different course now, but if we go back to the actual through pass rate before any of this we are mountains ahead of where we were there. Going back to this article first I love how open and honest, and that’s something you’ll find a lot in this community, but just open, honest, and vulnerable about what they found and what they didn’t find in their data.

Sharona: And again, this is the blog post. There’s so much more in the actual published article that we can’t even have time to get into. But they have done A great job looking at all of the different research questions and the methods and going into great amounts of data and analysis. This is a tremendous article that I would highly recommend people read.

Boz: But going back and comparing what we saw in our course, after several years of some major redesign and some tweaking, did we see the increase? Yes. Did we see the closing of most of the gaps we were looking at?

Sharona: We did. So we did a statistical analysis on three data points, the GPA, meaning the grade point average in the course- specifically, the DFW rate, and the DFU rate. Now there’s a, the difference between those two is the DFW rate is everyone, includes everyone who intentionally withdrew from the course. The DFU rate is, excludes people, students who went through the process of withdrawing, but still looks at people who have what we call an unauthorized withdrawal, meaning they disappeared from the course before a certain time. They end up getting a no credit, but we track those. So those are two different things. So basically the DFW includes students who came through in week eight, week nine and said, “Hey I don’t want this course anymore. I’m gonna drop it.” And then the DFU does not include those people. Those people get excluded from the data. So we looked at those three data points on four equity measures, which are gender, URM status, underrepresented minority status, first generation, and Pell eligible, which for us, Pell eligible is a proxy for economic disadvantage. We closed all of the equity gaps in the GPA with the exception of a very tiny GPA gap between male/female in our non-supported course. But what was interesting is the females did better.

Boz: Yeah.

Sharona: So we still had a gap, but it was flipped.

Boz: Yeah.

Sharona: So there’s that one. We also closed all of the equity gaps. I shouldn’t say cl- yeah pretty much closed. We looked at the close and the effect size, so the only way that we considered it significant is if it had a medium or higher effect size and showed a statistical gap. So if at a low effect size, we, it basically, even if it statistically was a gap, we just said it’s not worth looking at. And the only one that persisted in the DFW rate was in our supported class, so the non-college ready group in our underrepresented minority category. And one of the weird things there that I just wasn’t able to account for is we are almost all Underrepresented minority. We have a very small number of non-underrepresented minorities. So there might be a sort of a strange difference in those groups because of who chooses to come to Cal State LA and is not an underrepresented minority.

Boz: Yeah.

Sharona: Because we are so heavily, over 75% of the students who took this class were underrepresented minorities.

Boz: You, you said we closed the gaps. Now, I have, in, in my K-12 world, we’ve been looking at different equity gaps the entire 20-plus years that I’ve been an educator, and when someone says closes gaps, they can mean one of two things. So I want clarification from you of which one you mean.

Sharona: Okay.

Boz: So sometimes when they say we closed the gaps, it means we took a gap between two groups and we made that gap less. It’s still a gap. There’s still a difference. There’s still a , statistically significant difference, but what might have been a 10-point difference is now a, six-point difference. So we made statistical significant gains, but there is also still a statistically significant gap. So both things can exist. Or closing meaning you actually brought the two groups close enough together that they are no longer statistically significant and effect size difference. So which of those two did you mean?

Sharona: So when I meant close the gaps, I meant they are no longer statistically significant and/or if they are statistically significant, the effect size is low.

Boz: Yeah.

Sharona: So in the case of the DFW rate, the only one that had any statistical significance at all were URM, and only the supported class also had an effect size of medium or high. And again I misspoke. I said 75% of our students were URM. I’m sorry, 90% of our students in this class were URM. So there’s just something weird about those sizes of population that might be impacting the data, but all the rest of them were no longer statistically significant. And they were before we did this.

Boz: Now, going back to the message I said earlier, that grading alone does not fix everything. It is not a silver bullet and I really hope that Brandon or any of the members of his team hear this. We are talking about things that we did after three years of doing this and of making major changes before we even started collecting our data. I don’t know when they actually made their change, but looking at the gaps in these years and knowing the pandemic is in there, I am guessing this was, this data was collected pretty soon after they’ve started their redesign. I am really hopeful for Brandon and his team if, they continue to look at this and they continue to look at their courses and some of those pedagogical decision-making and how they actually do things in their class. And as they refine that, I am hopeful for them that they are gonna see some of these equity opportunity gaps, reduce if not completely go away like what we did.

Sharona: I agree. I agree. I do wanna clarify, we’re saying Brandon and his team because we know Brandon. Yeah. I’m not sure who’s first author on any of this because on the article, he is listed as first author on the article, but Marilyn Staines is listed as, listed first on the actual blog post. So Marilyn, Lisa, Brandon, we’re not sure what the relationship between y’all is. So we just happen to know Brandon, so that’s why we’re using

Boz: him. That’s a good point. I do wanna apologize to the the-

Sharona: and he is listed first on the official article, but he’s not listed first on the blog post

Boz: But yeah, no you’re right. That’s a good point. I keep bringing up Brandon’s name because, I-

Sharona: We’ve seen him present.

Boz: We know him … we’ve seen him present. I’ve actually talked to him in some of our some of our social gatherings during the conference, so I think I have sat in as, part of the organizing support that we give our presenters at least two different things that he’s run, so yeah. I’ve gotten to know him. Never actually physically met him, which I’d love to do, but, so yeah I do apologize. But yeah.

Sharona: I do wanna go to the last couple sections of this blog post though, because I really like what they say. There’s a section called What This Means for Instructors that I love some of their sections, because again their story was complicated. Yes, grades rose, equity gaps did not close or opportunity gaps did not close as much as they had hoped. What does this mean? And what they say. They said, “The research points to a few directions worth exploring.” So can I mention a couple of- No,

Boz: absolutely …

Sharona: the ones?

Boz: Yeah.

Boz: I really like pairing grading reform with other support structures. Like they said specs grading can create an environment where, revision and things are valued and it’s not just point mongering, it is about learning. But students need time. They, they need time. They need a- access to the meaningful feedback. And what they found and what they wrote is, and I’m gonna read this directly, “In particular, first gen students and transfer students may benefit from proactive outreach and structured support that helps them navigate the new system.” So this idea of, really understanding that those two groups, for whatever reason in particular, really do need some support that’s not just direct academic support, but actually in navigating the new system of the class and just the new system of being at college.

Sharona: And I definitely think that is something that’s become very stark for me with this course this semester, is the level of resistance the students have to sitting down and trying to figure out where to find the feedback and how to use it. It’s just so foreign that just making it available is not enough. You gotta teach them how to actually utilize it and what to do- Yeah … and why it matters.

Boz: Yeah. And there’s one other thing that they point out that I’m really glad they recognize, they point out. It’s something that we’re starting to see more and more in, in lots of research, is to look beyond the grades. That’s their actual bolded heading, “Look beyond grades.” And a- again, I’m gonna read directly. “Our study measured final course grades, which is an important but incomplete picture.” “Future research needs to examine students’ experience with specs grading”

Sharona: Specifications. “…

Boz: How it affects their sense of belonging, their confidence in their relationship to failure and revision, but also desegregated by different social identities. We cannot design for equity if we only measure grades.” Again, I think that’s incredibly powerful. We are starting to see a lot more research that’s going beyond the grades, that’s, really getting into the student experience and how those experiences might be different for different groups of students.

Sharona: And I think there’s a really important paragraph a little bit beneath that. They have a heading called “Where Do We Go From Here?” And it says, “We are at an interesting and important moment. Specifications grading has spread rapidly through chemistry and other STEM disciplines, driven by instructors who are genuinely motivated by better pedagogy and greater equity. The practice is here to stay, and that’s probably a good thing, but the research base is still catching up to the enthusiasm, and studies like ours are still rare. We need more of them.” And it goes on to say the types of studies we need. So I love this. I agree. At the center, we’re gonna try to, support this type of work as best we can and fund things like this as we can.

Boz: And continuing from that same paragraph, I, they point out something else. And it was interesting ’cause I was just having this conversation with the instructional leadership team, or ILT, at one of the schools I support in my, day job. “We need qualitative work that illuminates why these gaps persist even when overall grades are improving.” I was just having that conversation. Quantitative data and quantitative analysis is great at identifying where problems are, but to identify the why, you have to do qualitative studies. You have to collect qualitative. And again they point that out. We- we’re starting to see a little bit more of that. We’ve got a, I think a couple things sessions in our grading conference that it is talking about some of that qualitative research. So we still don’t abandon the quantitative, but if we wanna answer the why, we gotta get into qualitative research as well.

Sharona: Absolutely. I could not agree more.

Boz: All right, so we are already coming up on time. I’m glad you were able to be here and physically okay to sit with us for an hour, but is there anything else that you wanna point out or talk about before we start wrapping this one up?

Boz: I am too, and again, if you haven’t already registered

Sharona: I like to just give out the main center one, which is http://www.centerforgradingreform.org, but you can go also to thegradingconference.com, and that should redirect you to the conference page on the center’s website. So you can go to thegradingconference.com, or you can go to centerforgradingreform.org. Those will both take you to the same place, and you can register. Registrations are going really well. Get your institutions involved, and we can’t wait to see you at the conference.

Boz: All right. And until then you have been listening to The Grading Podcast with Boz and Sharona, and we’ll see you next week.

Sharona: Please share your thoughts and comments about this episode by commenting on this episode’s page on our website, http://www.thegradingpod.com. Or you can share with us publicly on Twitter, Facebook, or Instagram. If you would like to suggest a future topic for the show, or would like to be considered as a potential guest for the show, please use the Contact Us form on our website. The Grading Podcast is created and produced by Robert Bosley and Sharona Krinsky. The full transcript of this episode is available on our website.

Boz: The views expressed here are those of the host and our guest. These views are not necessarily endorsed by the Cal State System or by the Los Angeles Unified School district.