The Grading Podcast

86 – Pharmacy grading

===

Sharona: I have to say, though, that the final sentence of this article might be my favorite summation of all of this. So they say, While there is no single ideal way to approach grades, a good grading system should uphold high academic standards, accurately reflect student learning outcomes or competencies, motivate students to learn, reduce undue student stress, make students feel responsible for their learning, minimize conflict between faculty and students, and give students feedback that they will use

Boz: Welcome to the Grading Podcast, where we’ll take a critical lens to the methods of assessing students learning, from traditional grading to alternative methods of grading. We’ll look at how grades impact our classrooms and our students success. I’m Robert Bosley, a high school math teacher, instructional coach, intervention specialist, and instructional designer in the Los Angeles Unified School District and with Cal State LA.

Sharona: And I’m Sharona Krinsky, a math instructor at Cal State Los Angeles, faculty coach, and instructional designer. Whether you work in higher ed or K 12, whatever your discipline is, whether you are a teacher, a coach, or an administrator, this podcast is for you. Each week, you will get the practical, detailed information you need to be able to actually implement effective grading practices in your class and your institution

Boz: Hello, and welcome back to the podcast. I’m Robert Bosley, one of your two co hosts, and with me as always, Sharona Krinsky. How are you doing today, Sharona?

Sharona: I’m tired. I am very, very tired. Not only am I working all my jobs, but I’m moving in two days from the date of this recording. It’s only up 11 floors in my building, but it’s a full on move. Between that and working way too many jobs, I’m exhausted.

Boz: Yeah.

Sharona: So hopefully that doesn’t come across. This should energize me. How are you doing today?

Boz: I’m actually kind of tired too. I was the night before this recording, I was up most of the night being sick and throwing up. So, yeah, I have very little sleep last night, so feeling better today. It was just something I ate last night, but yeah, hopefully, hopefully I’ll be able to recuperate today and get ready for the, the week coming up. But.

Sharona: Well, I hope you feel better. So given that you were up all night, do you have any idea what we’re talking about today?

Boz: Well, you know, we’ve had a couple of interesting experiences and come across a couple of interesting articles. So I think that’s kind of the direction we’re going to go, but what experience did we have recently that kind of kicked this idea off?

Sharona: So, we have this talk that we also have, I think it’s the second or third episode of this entire podcast, that we give called Grading as the Misuse of Mathematics and the Measurement of Student Learning. We’ve given that talk several times. We gave it this week in a professional development session. I’m giving it several times traveling around the country in the next few months. But it never fails that we ask certain questions and we always get the same answers. It doesn’t matter what subject matters in the room. It doesn’t matter whether we’re talking to K 12 teachers or higher ed teachers. But the one that really is very interesting is we ask people to define the difference between different letter grades.

Boz: Well, let’s back up a little bit. I know we’ve said we’ve done this. It is actually one of our early episodes, but real quickly, you know, when we’re talking about this talk, this misuse of mathematics, like we really get into some of the nuts and bolts of why a lot of the math that we use in traditional grading is, you know, producing unreliable results and why we think, you know, traditional grading is a poor measurement of actual student learning.

Sharona: Well, and I remember that Robert Talbert did an episode on this as well, that grading is actually not numerical data, that it’s qualitative data. So what it, I believe the post is when it’s a number, not a number.

Boz: Exactly.

Sharona: So we kind of go into the math of that. So we did that this week.

Boz: Yeah, which we always enjoy doing. That is actually one of probably our, it’s A) it’s a great entry point to start talking about alternative grading because, and we’ve talked about this before, when you start looking at making changes without really understanding why you’re doing it. You can actually end up reproducing some of those same things that you’re trying to come up with a new way of grading to avoid. So we, we really do enjoy doing this particular talk. It is one of our most commonly requested ones and pretty much the entry point to anytime we start working with a new group. This is where we start by really jumping in and dissecting and looking at the issues mathematically with how traditional grades are calculated.

Sharona: Exactly. And we almost always start with, what’s the purpose of grades? So what do most people say the purpose of grades is?

Boz: To sum it up in kind of a generic answer, grades are supposed to communicate with students something about their learning or communicate to someone about the, you know, a level of students learning.

Sharona: And when we follow that question up, so usually, yes, people say that we also sometimes get the ranking and sorting piece, but most of the time K 12 educators, higher ed educators. They say it’s to measure student learning or to communicate the level of student learning. What’s the second question we ask?

Boz: Well, once we ask that, we always follow up that with, okay, then what’s the meaning of a specific grade? Like, you know, what’s the meaning of a B? How is a B different than an A or a C?

Sharona: Well, and I love asking about the B because the A tends to be, oh, they’re just extremely excellent. They’re amazing. C means to be, usually is barely adequate. So the B kind of gets you that middle level of, of what it is.

Boz: And, and I think it makes people think a little bit more about it because you’re right. You know, the, the F, C, and A are the, those are really easy to, to kind of define or, or I don’t know about easy to define, but.

Sharona: Simple to say that we think we know what they are.

Boz: Exactly.

Sharona: So what do people usually say about the B?

Boz: So what people say about the B, and again, kind of generalizing that, but that the students have learned, or shown mastery, on a certain percentage of the material. That’s what a lot of them kind of boil down to.

Sharona: So then it gets interesting. I agree with you. That’s what most of them say. They know more than half, most, maybe not all the materials, something like that. Then what’s the next question we ask, which is the really fun one.

Boz: And this is where it gets interesting is we then ask them, when you’re having a new student come into your class and you see they had a B in the previous one, do you think that’s what that B actually means?

Sharona: And they almost always say, It depends on the instructor.

Boz: So everyone thinks they know what an A, a B, a C means and that their grades really do message that, but then they don’t trust them when it’s coming up from someone previous.

Sharona: And I really wonder if not having really thought too hard about that, we as instructors generally think that, but the minute you start thinking about it, you might realize, even if you don’t admit it. That your grades don’t do that either. It’s just interesting.

Boz: The question is then if we agree the grades are supposed to communicate the amount of student learning, and we think we have an idea of what a B means, then why do we not trust them? Why do we not believe them? And I think it’s because we understand, even if it’s subconsciously, how much stuff goes into those grades that aren’t anything abouT learning.

Sharona: People will often say things like, Well, I know that my students won’t do the homework if I don’t grade it. But they’re also getting really high homework scores, but not doing well in tests, or a different student might do well in tests and not do well in homework. So I can’t tell, even in my own grades, without looking at the detail.

Boz: And then it’s, there’s funny, there’s another question. It’s a little bit off the topic we’re going to talk about today, but not really. That I always think is hilarious. And part of our demonstration as we go through to show some of the mathematical fallacies with traditional grading, we set up a mock test, a fictitious test, and ask teachers to figure out how they would divide the points up. And then we give them a fictitious student and we ask them, okay, if they did these things. What would the grade be? And we often get a huge variability. The school that we’re talking about that we just did this with a few days prior to this recording, actually, they were closer. They only had like a 20 point spread, which is for the size of the room was a smaller spread than what we usually get.

Sharona: But to clarify, this fake test has to be good for all disciplines. It has to be good for all grade levels. So it’s very vague, right? We just do, Hey, you’re giving a test. It has three groups of questions. We described group a, we described group B, we described group C, and then we tell them to give some points to the group, not down to the individual level question, because we can’t pick a discipline that everyone could work with.

Boz: Yeah, and and we’re obviously making some assumptions with doing this but it was really interesting because there another thing comes up when we do that and it came up in this one that we break those three groups out in this fictitious test basically by whether you want to call it Blooms or DOK, but your first set of this really basic recall, the second set is a little bit more difficult. And then you’ve got these kind of stretch questions, bonus questions that is a much higher level of thinking about the material. If we say that the fictitious student doesn’t get anything right on these, this hardest level.

Sharona: But they get everything right on the easiest level.

Boz: Yeah. And what was the comment that we get, which is not unique by any means whatsoever?

Sharona: Well, the general gist of both this situation and others is that combination of scores couldn’t possibly happen in my class. And we’ve heard this in so many different ways. A student who doesn’t do any homework could not possibly succeed on my exams. A student who does all the homework could not possibly fail my exams. A student who gets all of the basics right couldn’t possibly not get any of the complex right. A student who gets all the complex right couldn’t possibly not know the basic. Like we’ve literally had every combination in every discipline.

Boz: And I think we even get this even more so and, and I might be wrong about this there might be a lot of other subjects out there that see this, but I think we get this even more so in math because of how material builds on each other, Oh, a student can’t possibly be able to graph or solve exponential functions until they can do linear functions. We’ve talked about that on the podcast before about how.

Sharona: Well, it’s not that the math builds. It’s that the belief is that the math builds and therefore you can’t do the later stuff before the earlier stuff, but that’s not a belief that I dispute.

Boz: And I would agree that most of the instruction of mathematics is fairly linear to a point, but that doesn’t mean the learning always is, but yeah, that we get this kind of argument in so many different fashions, so many different ways that it is quite comical.

Sharona: So coming out of that conversation and the fact that I’m going to be giving this talk to wildly varying groups of people over the next four months. I thought it’d be interesting to go back to the literature and see what’s out there about traditional grading systems. And I wanted to specify it by traditional, we’re not talking about A, B, C, D, F, although that is traditional.

When we say traditional, we’re really diving into the points and percentages style of determining those final end of term marks. So we came across this article. Do you want me to introduce it? You want to introduce it?

Boz: No, go ahead.

Boz: Yeah, but it cracks me up that the point you were just making about how many things we do as an educator based on what we think is going to happen to the students after they leave us, and at the same time, everything that our students come to us deficient in, we blame the teachers below us. So college, college blames high school that we don’t teach anything. High school blames middle school goes all the way down to the, you know, parents not preparing. But it’s funny how we do everything, because of the next level and yet blame the level behind us. It’s almost a universal truth that I just find funny of how much of that we do.

Sharona: And my mom, people on the podcast may or may not know, my mom was a math educator as well. She spent her career at Cal State Dominguez Hills. She worked with K through eight pre service and in service teachers, but she was in the math department, so she was specifically in math education. And she used to, Explain the whole cycle of preschool and then elementary school blames preschool and then high school blames, and you go all the way up until you get to your pre service and in service teachers and they get blamed for what happens in preschool.

So like it’s a whole cycle and she used to say you have to break the cycle in many places. You have to shatter the cycle, ideally simultaneously. So that’s our goal. I would say that’s our goal and one, at least my personal goal with grading, is since grading permeates the entire cycle all the way up through pre service and in service. teachers and instructors. I want to break it. So that’s why to me going all the way to the professional school level was very interesting. Because ultimately when you get to higher ed, now you’re talking about, well, what they need to be ready for if they go into a master’s program or a professional certification. So doing pharmacy education. Also, pharmacy is one of those life or death skills, right? Pharmacy is one of those that the stakes are really high. You mess up someone’s meds, It’s bad.

Boz: It’s bad.

Sharona: So, we came across this article, we’ll link it in the show notes, and one of the things I really enjoyed was literally the third sentence in the introduction, "alternative forms of grading such as competency based assessment specifications grading and ungrading have recently become more popular as educators have begun to realize the downsides of traditional grading systems." So, I’m like, right there. I’m sold. They used our alternative grading language. I’m happy.

Sharona: So as we were reading through this, I got to admit we nerded out, totally nerded out on this article because one of the first things they introduce is something that neither you nor I were actually familiar with, which is something called a Kuder Richardson reliability score. And it is a statistical measure of the reliability of assessments. And I was like, Oh my goodness, statistics and grading. Yay.

Boz: I’m actually surprised you didn’t talk and start your nerd out by the rich history of grades that this actually goes into, cause I know anytime we’re working together and we’re presenting, especially on a intro type of PD, you always take the. The history part of grades. And you always enjoy getting into just some of the weird traditions that. You know, are considered traditional practices now, and the historic reason of why they exist.

Sharona: Yeah, well, I don’t want to become a broken record. But one part that that history did cause me to realize, and this is probably a future subject of research for me, It says, because considerable global variations exist in how grades are structured and assigned, this review focuses on traditional grading schemes within the U. S. I am not familiar at all with grading systems outside of primarily the U. S. and Canada, so I think that could be a very interesting thing to look at is what’s happening globally.

Boz: Yeah, and they actually bring that up in this article, a part of the Issues with traditional grading is the hodgepodge of different elements that go into that final grade.

Sharona: Exactly. But I do want to talk for a minute about this reliability score. So yes, nerd out on the history, totally good, all over it, but I do want to talk about this reliability scale because one of the things that we do find quite a bit is that people’s perceptions of the accuracy of their assessments tends to be wildly optimistic might be a way to say that people believe that their assessments are measuring what they think they are measuring.

Boz: So yeah, in statistics, when we’re taking measurements, there’s two things that we’re really concerned with. And those are, is the measure valid? And is the measure reliable? So, in this article, and what you were talking about, they’re looking at those two when it comes to assessment.

Sharona: Well, and specifically being able to distinguish, to differentiate between the two, because reliability is producing consistent results. Validity is producing accurate results.

Boz: Yes.

Sharona: And so like if my scale is off by 10 pounds, but it’s reliable. It’ll always be off by 10 pounds. I’m not going to comment low or high, but valid is how true the measurement is.

Boz: That’s this Kuder Richardson reliability score is a statistical measurement specifically done on assessments on tests to see how reliable that test is.

Sharona: Well, and then it comes with the standard deviation it’s always reported with the standard deviation because you not only want the reliability score, but you want to know how variable that particular assessment is over time. That’s how it gets its reliability. So what I found fascinating they gave this great graphic for those of you that are statistics nerds. where they compare a score of B to a score of C. And they compare what happens if you have a 90 percent KR score, Kuder Richardson reliability score, and a 10 percent standard deviation versus, say, if you have a 60 percent reliability score with a 10 percent standard deviation. So they’re keeping the variability within the measurement the same, but they’re changing the overall level. And it’s kind of shocking how determinative the reliability is, or rather how how hard it is to distinguish between a B and a C student if you don’t have an assessment with this really high reliability score.

Boz: Yeah. So to give a little bit of background on this particular test and how the numbers work in it, the KR 20 score or, which is what the Kuder Richardson and is usually referred to as the KR 20. Goes from about a 0.2 to a 0.95, like that’s kind of the range. It’s similar to, we have a lot of measurements and statistics that kind of go from that zero to one, or maybe negative one to a positive one score. And the closer it is to one, the more reliable of a test it is that it is measuring what it’s saying it’s measuring. But when you look at that with the standard deviation, and this is what this article was part of the research was pointing out is how incredibly high. That score needs to be along with an extremely low standard deviation to actually be able to distinguish between a B and a C student.

Sharona: And the way you get a higher score is the test questions that you put on your assessment or whatever you’re asking as part of the assessment, they have to be internally consistent in order to get that high score. So if you have a KR it’s considered an excellent score, and it’s reasonable, according to this article, it’s reasonable to assume that the student’s score would reflect actual learning if you have a 9 or above. But it turns out that even with a 9 or above, if your standard deviation is too big, you start to get a lot of overlap between the B and the C student.

Boz: Yeah, which means maybe if a student gets a B, maybe they actually have more learning than the C student, but then again, maybe not. So being able to distinguish between a B and a C, maybe sometimes this test, your test can do that.

Sharona: So then the article goes through a number of statements and a number of assumptions. So it says the most basic assumption is concerning grades is that they accurately measure learning, and then they just showed this display of the KR 20 to say, not so sure. The second thing that they point out now in this, again, this is a pharmacy education article, is if you start putting things other than assessment of knowledge and skills, then the overall grade gets even less reflective of learning. So, for example, participation, completing homework, they give a table here of the variations of what happens when you include these other things. And that’s part of what we do as well in our talk is we do the weighted averages.

Boz: Yeah. That’s one of the components that we’ve never looked at the reliability of a test, but we’ve looked at kind of the three things that we point out in our discussion is the reliability of teacher subjectively grading a test, but then we also the second component that we looked at is exactly the same thing that this article is doing. This research did, which is looking at the effect of those weighted categories. It is extremely common in most higher ed, at most high schools, and I would say probably most middle schools, that your grades are made up of not a pure average, but a weighted average.

So you have different categories, and those categories, those differences of if I’m giving my homework, 10 percent of the grade and you’re giving your homework 15 percent of the grade. That doesn’t seem like that much of a difference, but it actually can end up making a huge difference in the outcome of the students final grade. So that kind of is the second point of our discussion when we talk about what’s the misuse of mathematics. And that’s exactly what this research was doing it and basically did the exact same thing that we did to show how a student could get anywhere from an A to a C, depending on those weighted categories without changing any of the actual student learning.

Sharona: Now, the third thing that they talk about in this section of the article is something we’ve never actually talked about. We have never, I don’t think, Boz, that you and I have ever had this conversation, although maybe we’ve had it sort of subtly, the number of questions that differentiate a grade.

Boz: Yes. And now, so, and this is again, going back to a statistical concept. And I would disagree. I think we have talked about this a little bit. Especially with a good friend of ours. And I don’t know if I have permission to use his name, but I’m going to use it anyways, Owynn Lancaster. We did a lot of work with him through College Bridge and with some of our early work with alternative grading and why traditional grading, we shouldn’t be utilizing this, but there’s two ways a test can build reliability. One of them is through quantity. Asking more questions, which is what this article is also is pointing out the difference between asking a 10 question test and a hundred question test. You can build reliability by doing more questions, especially in a, you know, something like a true false, like a scantron type, yeah, Scantron type test. That is one way to build reliability is ask more questions.

Sharona: However, that brings up its own issue, right? So then in a hundred question test, you might have 10 questions that distinguish one grade level from another. But then what if those are all relating to a single topic or to multiple topics? Like, so now, yeah, you have an 80 or an 85 versus a 75. But you don’t know if the difference between those students is a single thing that they’re supposed to complete or multiple ones.

Boz: Yeah, am I missing one or two questions off of each topic on the test or am I missing all of the questions on one topic and the rest of them I’m really good? But yeah, that was the reliability point. And this is what Owynn would bring up is the other way to increase the reliability of the measurement instead of just asking more questions is to ask better questions. And that’s where I think you and I have had conversations. In fact, we’ve done a whole thing at the last CMC conference we went to about authentic assessment in the day of AI. So how do you ask better questions? Because those can also lead to more reliable measurements. So you can do it by quality or by quantity.

Sharona: Agreed. Agreed.

Boz: I don’t think this article talked about the quality of the questions as much. It definitely talks about the quantity aspect with, in a 10 question test, one question differentiates a letter grade, whereas a hundred. Question test. It’s actually 10 questions.

Sharona: So the fourth point in this section of the article made me laugh. Because they talk about the considerable variability that exists among instructors. But the last sentence of this particular paragraph says, If elementary and high school educators formally trained in assessment exhibit variability in grading, it is reasonable to assume that pharmacy instructors will do so as well. And when I read that, I couldn’t help thinking about how often you accuse me of being arrogant when it comes to higher education versus K 12. And yeah, I mean, I definitely have come across a lot of people in that in their own subject matters feel like they know what to do because they’re experts in their subject matter and I just also remember Eden Tanner talking about how that would have been hubristic of her, and she’s an unbelievably well accomplished biochemists.

Boz: And she’s one of our keynotes this year in the conference that I’m looking forward to. It’s funny usually the keynotes, like we try to make a big deal of trying to get the keynotes onto the show this year with the exception of one, all the keynotes we’ve actually already had on the podcast.

Sharona: I actually think all three of them have been on the, on, oh no, you’re right. Asao Inouye has not yet. Asao has not. We gotta get him on before the conference. But I thought that was very interesting that, yeah, if K 12 educators, you guys have a lot more formal assessment training, you might not have grading training because we found that nobody has courses in grading 101, but you guys do learn about formative assessment and summative assessment and all those things to a much greater degree than higher ed professors do.

Boz: Yeah. But I kind of wanted to move on to, cause a lot of this article that we’ve talked about so far is research that backs up everything that we’ve been saying, but this article goes a little bit further. It talks about not just the deficiencies of traditional grading systems, which is part of their title. It actually also talks about why it’s so difficult to make any changes. and recommendations for changes to make grades better.

Sharona: But before we go that far in the article, there is one other thing I want to point out.

Boz: Okay.

Sharona: That they say, a key issue regarding the validity of grades in the specific context of health professions education is whether grades align with clinical performance. So once you get up to a graduate level in a professional expertise area, so medical, law, engineering, health professions, whatever, you now start to have. a real, a really real world expectation of performance. So for them, especially making sure grades align with that, I think is, is important. So I just want to say that this is part of what’s motivating them in this article.

Boz: And I actually think we see that across quite a few variety of subjects in areas. I mean, I know what, what are the favorite shows that my daughter and I like to watch together is a show called The Rookie. But one of the, one of the kind of key points and, and where a lot of the humor comes into this is the difference between when someone leaves the police academy to their first year. The gap of knowledge. We see this in teaching. This is why we have student teaching. We see this in the medical field. This is why you have your clinicals. And this is why you have that those internships. So we see this in a huge variety, this difference, this deficit from where you’re learning ends your academic learning journey ends to when your actual practice and career starts. And it’s almost become so accepted that you’re going to have the career side has to figure out a way to deal with it, whether it’s like I said, student teaching, whether it’s, you know, your your first provisionary year where you’re on probation and you’re having to do all these guided or assisted calls, like if you’re in medicine. So it’s just become such a acceptable practice that the career side knows that they have to deal with it. Exactly. Yeah.

Sharona: So I just thought that was important to point out, again, because we keep saying that instructors look to the next thing and try to teach to it. And now we have this sort of culminating point where they’re like, well, wait a second, do grades in a pharmacy education setting actually match clinical performance? So now you were mentioning though, that the barriers to revision, I just wanted to make sure, there’s a lot more in here about the motivational aspects of grades that we’ve gone over before. So I think we can skip that part. Just go down to, you’re right, the barrier part was really interesting.

Boz: Yeah, because they talk about some of the, the barriers of doing any kind of grading practice reforms or changes. And that you know, grading has this, you know, huge cultural and logistical nature that it’s not just a simple practice that it really does have deep roots that makes this really difficult to make any kind of reform.

Boz: So what does that mean?

Sharona: So if you’re in pharmacy or medical or, or those things. You have to, once you graduate, you have to match with an institution that is willing to take you on the next level of your training. So this would be doctors going to residency and pharmacy students going to residency. So there’s a big complicated system in this country to try to take the students who graduate, because there’s a very limited number of slots of institutions that are willing to continue the next level of training. You have to have resources, you have to have funding, because these students are not quite ready to be just let out into the world on their own. So they have to make choices. And I remember, my sister’s a doctor and I remember her going through the match process and it’s a combination of where does the student want to live? What type of pharmacy or medicine do they want to be in a hospital? Do they want to be in a clinical setting? Do they want to be at a corporate? You know, pharmacy and there’s a whole complex process by which you try to place all of these graduates Into their next step because they can’t move on until they do this supervised thing.

Boz: Yeah, which is what we were just talking about this kind of but as an educator or as someone that, you know, might be, might have some control over how the education process works at my institute. Yeah. If I know this, if I know that going to, you know, making a reform of going to a pass fail, which is one of the recommendations that a lot of people that we’ve talked about a lot of the you know, grading reforms like Guskey and Feldman have talked about. If that’s going to hurt my students ability to get to this next stage, then you’re right. It is not only illogical for me to do it, I would say it’s harmful to my students because I’m putting them at a disadvantage. So the system is perpetuating its own self.

Sharona: However, there’s a counter to that, and they did this in some of it, some years that they looked at this, pass fail sometimes does match higher. Because a student who has an A, B, C, D, F, but has all C’s is going to be less attractive than someone with all A’s in a multi level grading system, but yet A’s and C’s would be treated the same in a pass fail. So, you couldn’t distinguish. So it’s a real complicated thing, but they’re finding that pharmacy educators are starting to look at this and saying, Hey, wait a second. These GPA averages, they’re not actually working for us, so that might be changing, and it’s something that we need to be looking at.

Boz: Yeah, it talks more about some of the culture aspects of, you know, some of these barriers. But I also, I, part of that cultural difference and part of those barriers. And I’ve said this a couple of times. I said this last time we were talking with Dr. Patrick Morris, I want to see if there’s a way to do some sort of research. But the people that are running the educators education system are the ones that at some level had success at it. And I, I really do believe that’s one of the biggest barriers to doing any kind of real grading or assessment practice reform.

And you’ve said this hat because we’ve succeeded at some level that is part of our identity and challenging that identity can be really difficult to do. So it talks in this article a lot about the culture, the culturistic aspects of why doing any kind of reform is difficult. It didn’t talk specifically about that one, but I do, I want to find out see if there’s a way that to research that question, because I think that’s a huge part of why grading reform is so difficult.

Sharona: Now, there’s another huge barrier that they talk about that, in my opinion, gets harder and harder the further, further up the educational scale you go, is the time, they say here, the time, expertise, and resources required to develop and implement practices that overcome this. So most faculty at these levels have not. Let me repeat, not been adequately trained in the complexities and nuances of how to measure learning, how to provide evaluative feedback, and how to motivate students to learn without using points or grades. So when they give the example, they have these things called they call them OSCEs. They are objective structured clinical examinations. Okay. So that’s one of the measures in pharmacy education. And they said, Assessing learning that way requires skill in case development. So you actually have to be able to write cases that adequately measure these objective structural clinical evaluations. That’s hard.

Boz: Yeah.

Sharona: That is hard. And it takes time. And even with more traditional forms of testing. You need true expert knowledge, but then you also need expert knowledge in actually constructing tests and constructing test items. So with the level of lack of funding we have in some of these areas and lack of, of expectation of training, it’s no wonder that we don’t. invest in this.

Boz: And you said that you think, you know, this is a issue, the more of an issue, the further you go up, which I would agree, but we’ve talked about this a lot of times, how many, how many of us K 12 educators actually had a class, you know, that was called grading 101, where we really talked about the art and science of grading and going just past that. How many of us have, you know, gone through any real test authoring courses. And cause I’m sorry. Yes. I, as part of my K 12 bachelor’s degree, we definitely had some psychology classes, the adolescent mind, the psychology of teenagers, you have these kinds of classes, you do have some usually some courses about testing, but not really test writing, not, not looking at it in. In fact, I think most of my understanding of, of test writing actually comes from my statistics background. And even I had to go in and relook up and refresh myself on this KR 20 measurement. So the amount of test authoring. training that we have.

Sharona: So in my job as a, as a coordinator, the bulk of my work is actually writing assessments. It’s a lot of the work I do. In addition to setting up the canvas courses and things like that, but trying to write more authentic assessments and then trying to work with my faculty to understand why these assessments are a more accurate measure if they are. And sometimes I mess it up. Sometimes we write a question, we’re like, wow, that just that didn’t do what we wanted.

So they finish out this article with a bunch of recommendations, and some of them are very interesting to me because of where this particular educational component sits at the very top, at the very end of a student’s educational journey. Because they actually say here, traditional tiered grading systems may fail to accurately discriminate authentic learning and in turn produce flawed rankings of students. So one of the things that we’ve said is two of the purposes of grades are communicate learning and rank and sort. And now this is a group that’s saying actually that’s not ranking and sorting very well either.

Boz: Yeah.

Sharona: And a lot of the times in the lower level classes were like, Hey, we just shouldn’t be ranking and sorting. Like that’s not our job at a first year math class. But now they’re saying, and not only that, it doesn’t actually work.

Boz: But I love this they have a whole table where it basically summarizes the issues with traditional gradings and some of the recommendations. And I’m looking over these recommendations and is this not majority of this not four pillar stuff?

Sharona: Oh, absolutely. Yeah. I mean, I like the one that says tiered grading can create an adversarial relationship between faculty and student. And their recommendation is use a competency based grading system with a set criteria for assessing competency and create a remediation or corrective pathway so that all students can achieve the intended learning outcomes.

Boz: Or the very bottom one. Grades are an evaluative form of feedback and are less effective at improving learning than descriptive feedback. The recommendation? Separate grades from feedback. Offer more descriptive feedback. Offer more sampling of knowledge opportunities, but avoid assessment fatigue. So, you know, we we’ve got, you joked that we’ve gotten even away from using words like, you know, did not meet standard or met standard or exceeded to just emojis like. My taking even the language out of it, even though

Sharona: Man, students will make meaning out of anything. I use a fricking cloud or a little man with a thinking face and they’re like, I got the thinking face. I’m like.

Boz: But it’s still, it’s looking at the number or the result versus looking at the feedback, which is another thing we’ve talked a lot about. We’ve got, you know, research out there that, that did this and tested this. What kind of. was the best feedback to give, you know, whether it’s the score, the score and feedback, or just the feedback. And this is saying the exact same thing.

Sharona: I have to say, though, that the final sentence of this article might be my favorite summation of all of this. So they say, while there is no single ideal way to approach grades, a good grading system should uphold high academic standards, accurately reflect student learning outcomes or competencies, and reflect student learning outcomes or competencies. Motivate students to learn, reduce undue student stress, make students feel responsible for their learning, minimize conflict between faculty and students, and give students feedback they will use. Holy moly. Wow. That’s pretty cool. I agree. I agree with every word of that sentence.

Boz: Oh yeah. We’ve talked about either how traditional grading doesn’t do this or how alternative grading helps support it. Like the relationship. I love that it’s talking about the relationship between the student and the educator. Because that is such a big, a big factor in student success. And I know you’ve said it a million times that grading is one of the most personal things that we do relationship wise with our students.

Sharona: Yeah. We’re at some level, we’re passing judgment on them as individuals, or at least that’s the way it’s perceived.

Boz: Well, yeah, especially when you’re using traditional points and percentages and points. I just said points twice. That’s okay. That’s how many times we do them. But it puts us at an adversary role with our students. We become these point mongers or gateway to the points that really does build this adversary relationship. Whereas the relationship I want to have with my students is they see me as, someone that’s going to help them get to the learning goals that they have, not someone that’s guarding all the points and distributing the points, whether I like you or not.

Sharona: Yeah. Well, that reminds me of that article of why football, was it why football is better than school or something like that? Yes. Of 28 reasons. I want to be the favorite coach. Of the winning team that people dump the Kool Aid on, you know, I mean, their standards are not lower. I mean, this happens at the very tippy top of sports. So why can’t we do that academically? I think that would be really cool.

Boz: All right. So we are coming up on time. So do you have any last minute thoughts you wanted to put in here?

Sharona: My last minute thoughts have to do with the conference. We just closed our abstract submissions. We have amazing abstracts that have been submitted. We’re going to be going through that process. Registration is open. We have the institutional registrations available if anyone wants to get their institution to come in. I just really feel like we have so much momentum. And I just want to see it keep going in this, in this world of grading reform.

Boz: And speaking of the grading conference and deadlines, the deadlines for volunteering has already passed, but I want to give a shout out and a thanks because, we had so many people that are offering to help and volunteer their time during this conference.

We might actually have to turn people down because we’re not going to have enough roles.

Sharona: That’s amazing.

Boz: That’s how giving and how responsive this community is. It is absolutely amazing. So thank you, you know, thank you to our listeners. Thank you to the community enlarge and thank you for all the, those that have reached out and put in that they are happy to volunteer their time during the conference.

Sharona: Yes. Ditto.

Boz: All right. And then we hope you guys have enjoyed this and we’ll see you next time.

Sharona: Please share your thoughts and comments about this episode by commenting on this episodes page on our website, www. the gradingpod. com. Or you can share with us publicly on Twitter, Facebook, or Instagram. If you would like to suggest a future topic for the show or would like to be considered as a potential guest for the show, please use the contact us form on our website. The grading podcast is created and produced. By Robert Bosley and Sharona Krinsky. The full transcript of this episode is available on our website.

Boz: The views expressed here are those of the host and our guest. These views are not necessarily endorsed by the Cal State System or by the Los Angeles Unified School District.