The Grading Podcast

Boz: But what did Doug, recommend?

Sharona: Well, I love this question because this is the question that’s kind of at the heart of so many alt-grading conversations.

Boz: Yep.

Sharona: So he says, “Before we get the to the tricky dilemma of how we might measure whether learners are reaching the goals we set, one key question demands consideration.”

And then he quotes people like John Warner, who we just had on the pod a few weeks ago, raises others. But he says, “Is it proficiency or is it growth? Are we measuring whether they can do something or know something, or how much more they can know or do it since before we started measuring?” And this is a big one.

Boz: Welcome to The Grading Podcast, where we’ll take a critical lens to the methods of assessing students’ learning, from traditional grading to alternative methods of grading. We’ll look at how grades impact our classrooms and our students’ success. I’m Robert Bosley, a high school math teacher, instructional coach, intervention specialist, and instructional designer in the Los Angeles Unified School District and with Cal State LA.

Sharona: And I’m Sharona Krinsky, a math instructor at Cal State Los Angeles, faculty coach, and instructional designer. Whether you work in higher ed or K-12, whatever your discipline is, whether you are a teacher, a coach, or an administrator, this podcast is for you. Each week, you will get the practical, detailed information you need to be able to actually implement effective grading practices in your class and at your institution.

Boz: Hello, and welcome back to The Grading Podcast. I’m Robert Bosley, one of your two co-hosts, and with me, as always, Sharona Krinsky. How you doing today, Sharona?

Sharona: Well, I think I texted you yesterday, “Is the semester over yet?”

Boz: You did.

Sharona: I did. And it’s week five, ~uh, ~out of a 15-week semester. So I’m doing well, but just this, I’m struggling a little bit ’cause I’m teaching five days a week, and it just feels like I’m always at the university, and I don’t know.

I, I’m having some weird struggles. I’m not gonna lie. Having some weird struggles this week.

Boz: Well, you and I have taught, together for, for over, what, a decade now?

Sharona: Yeah, a decade …

Boz: but we have taught together, at Cal State LA for, what, seven years now?

Sharona: Yeah.

Boz: We often talk about this pretty much every semester around somewhere between week 8 to 10, you hit that point.

Sharona: No, I’m later. I, it’s usually 12 to 13. Like, 12 is kind of my worst week because I’m like, “Oh my God, this semester’s-” Well- “… never gonna end.”

Boz: ~Yeah, but that- And then- … that’s when it’s the worst, but ~you usually start hitting it around somewhere between 8 to 10, and then it gets, it’s worse around 12- 12 and then 13

sometimes

Sharona: 13, and then all of a sudden I’m like, “It’s done.”

Boz: So, you are hitting it. You’re starting it a little bit early this semester.

Sharona: This is not good. Although, to be honest, I think it’s because I have so many other fun things going on that I would rather be paying attention to. I love my classes, but I’d rather have three days a week that I was teaching, and I know how privileged that makes me sound, and I’m so sorry, but I’m just whining. But yeah, that’s, that’s how I’m doing. How about you? How are you doing?

Boz: You know, we’re, we’re a little bit later in the semester, at LAUSD than just week five, so we’re, we’re starting to kind of get into the middle of it. You know, a couple weeks ago I, I did talk a little bit about this first big round of PDs that, ~um, ~the division I work for at LAUSD had done. Well, we’re starting to come up in a couple of weeks on the second one. So th- that’s gonna be, a lot of my time and effort going into working and, and trying to smooth those out, take the feedback that we got from those first rounds. ‘Cause we got a lot of great feedback, but we got definitely got some feedback of where we can improve, so we’re, you know, really trying to take that to heart. So the only thing I, I don’t like about that is it takes me away from my schools and from, you know, the teachers I’m working with. But know it’s ultimately for ~the, ~the betterment of the product that, the division that I work for turns out, so-

Sharona: Yep. You’re a little bit on the I’m prepping PowerPoints side of the

Boz: Yeah. In fact, that’s, you know, we’re recording this on Saturday, which is usually when we record when it’s just you and me, and yesterday, Friday, I was at the district office all day, and that’s all we were doing was practicing, rehearsing, and getting these, what, 12 different PDs that we run, getting them ready to go, and I think they start in a couple of weeks.

Sharona: Well, I feel for you. You’re in PowerPoint hell. But you’re doing good work. We talk a lot about some of the teachers you’re working with, and they are very lucky to have you, definitely.

Boz: You know, looking at the next couple of weeks, I just have to mention, I’ve looked at our interview calendar of what’s coming up. We have really got some great interviews coming up, and I just kind of wanted to put, a little bit of preview out for them just so people are looking, for them, ’cause

Sharona: I guess it- Well, and just in case you’re sick of hearing just the two of us. ‘Cause we’ve, I think we’ve done, like, three of the last four have been, and then plus this one, plus maybe one more week, so just yes, please, please tease, entice-

Boz: So we have got, , three that I wanna talk about. Two of them are returning guest. One of them, Becky Pepler, we, who I think we’ve had on two or three times already. But yeah, I’m really excited about that. She’s got a new book that’s coming out that will probably be the main point of our interview, but she’s just, she’s a ball of energy. She’s fun to have on. She’s incredible in the space that she’s in with alternative grading, and she really has kind of like specialized and focused on the reassessment aspect of it. So that’s, I, I’m, I’m really excited about having her back on. She’s just a fun guest to have.

Sharona: I also wanna say, though, that despite whatever order you’re gonna talk about these, and we make no guarantees about when these episodes are clicking out.

Boz: But Becky is not our only returning guest. We’ve got another returning guest scheduled. Who else do we have that’s coming back?

Sharona: So I’m really grateful Mark Aronson is coming back. We interviewed him about a year ago, and his school, one of the things I’m curious to ask him about is that was the school that had that honors designation on the transcript.

Boz: Yeah.

Sharona: And he said back then, “Hey, let me come on in a year, and, you know, let’s talk and see how this went.” And he reached back out and said like, “It’s been a year. Wanna hear?” We’re like, “Yes, please.” So I’m really excited to have him back, and I’m looking forward to that conversation. But then we have one more.

Boz: And we have, and this one’s not a returning one, and this is one I am really excited about, because I have been wanting to get someone on from this university ever since I heard about this university. It’s Evergreen University, and the reason I’ve been so interested is they don’t have interim grades at the university at all. Like, I, it is such a, like, extreme alternative grading, and I have been wanting to get someone on from this college for so long. We had a group of them at the grading conference. Unfortunately, I wasn’t able to be in that session. I’ve actually gone back and listened to it. Since listening to it, I wanna have them on even more, so I am so excited that we have finally got someone from Evergreen coming on.

Boz: I’m really curious to see what that experience ends up being, how that goes, what comes of it, if anything. So yeah,, if you’re there or if you’re in the area, yeah, come by, find our booth. We’d love to talk to you. Well, I say we, I mean we the center, ’cause technically I won’t be there.

Sharona: Yeah. And?

Boz: But, but my mini me will be.

Sharona: Yes.

Boz: So-

Sharona: Yes …

Boz: I can’t be there ’cause LAUSD will already be in session by then, but our Cal State systems won’t be. So I’m not going, but my daughter will be going.

Sharona: Yep. So we have hired your daughter as a booth attendant, and we’re getting her trained up, so the next generation is starting to happen. So there we go. Not that she’s necessarily gonna go in this direction, but how could she avoid it, right? So anyway, so that’s gonna be real- so there will be a Bosley at- Yes … the booth. Yes. Just not this Bosley.

Boz: All right … speaking of universities, that’s kind of where we’re gonna go with today, but we– you have talked about this for a very long time, and recently we’ve talked a lot about, the changing environment with universities, and kind of the danger that AI presents, but also more of the fact that it’s revealing a bigger, darker, issue. So we’ve been talking about that quite a bit recently, and we just came across an article that just came out a few days ago, came out on September 22nd. Do you wanna give a little information about this article? ‘Cause I think this is where we wanna talk about today.

Sharona: Yeah. So this article is actually part three of a three-part series, but this is the first article that I stumbled across.

Boz: Is the, the last of the,

Sharona: of this three-part The last of the three.

Boz: Yep.

Sharona: And it’s called The Ways Forward on Measuring Learning, and we’ve been talking since, like, day one that this podcast launched that the purpose of alternative grading, for a large degree, for many of us, the purpose is to measure learning. Now, there are people in our community who disagree with that. Some people wanna measure growth and other kinds of things, but at least step one in alt-grading is discussing what you’re measuring. And so th- this author’s, ~uh, ~his name is Doug Letterman, his, ~um, ~perspective is that not measuring learning is a problem for higher education.

Boz: Yeah, and that’s kind of one of the, ~um, ~underbelly things that, you know, you and I have talked about that AI is kind of, bringing to the light So this is, like you said, a, the third article in a three-article series. We might, you know, mention the other two a little bit today. We also might go back and, and do the other two in another episode.

Sharona: Okay. So the first article is called Failure to Measure Learning is Hurting Higher Education, and the subheader there is the subtitle, “The public doubts the value of what colleges are providing. That could change if institutions proved that students actually develop the capabilities and skills we promise them.” And the second article is called Why We Haven’t Measured College-Level Learning. So in, in the subtitle there is, “Past efforts have failed because of the tension over whether they’re for us, colleges, or them, our overseers.

Let’s try again so colleges can prove they’re giving students what we promise them.” So that’s, was the lead up to this article that we’re gonna talk about today.

Boz: A- and like I said, we might mention them a l- and talk a little bit about them. But I definitely, if we don’t wanna come back to those, ’cause that second one, I found that second one almost as interesting and as, important as this third one. I really did. Yeah. I like that second article a lot. And in fact, when we were looking at that a- and kind of looking at all three of these, we took an unusual little turn, and I, I wanna spend a little time before we get into the article talking about that, which is, you know, if universities are even trying to measure learning. So we kinda went and looked at some mission and vision statements of a couple universities and found some of them kind of interesting.

Sharona: Well, I guess one of the things that I was a little bit confused about when I started looking for these things is I started by looking at mission statements, right?

And I kind of thought that some of the mission statements of universities were to educate students. Like, I thought maybe that was the mission

Boz: you, like, you would think it might be the main-

Sharona: I- might be the mission? …

Boz: main? I think

Sharona: that could

Boz: be- But,

Sharona: okay … ~the main~

Boz: purpose.

Sharona: But to be fair Learning for learning’s sake is not always sufficient for an institution to exist.

Boz: Okay.

Sharona: ~So, so ~I think that many of these mission statements go beyond learning as being the mission as to why they’re trying to teach students, why they’re trying to have people learn. Like, what’s the purpose of that? So for example, the Cal State LA mission statement, it says, “Cal State LA transforms lives and fosters thriving communities across Greater Los Angeles. We cultivate, cultivate and amplify our students’ unique talents, diverse life experiences, and intellect through engaged teaching, learning, scholarship, research, and public service that support their overall success, wellbeing, and the greater good.”

Boz: It does have the word learning in it at least

Sharona: once. It does. And to be fair, like when I… You know, my, my theater company, our mission is we transform, we, we build community through the performing arts. That’s what our mission statement is at Encore. But I guess the difference is we actually mention the performing arts in our mission. If you just took that first sentence, “Cal State LA transforms lives and fosters thriving communities across Greater Los Angeles,” there’s no indication we’re talking about a university. It could be a healthcare system. It could be a bank that does lending. It could be , a nonprofit about homelessness.

So I went looking for some other ones. So then I was like, “Well, maybe that’s a Cal State LA specific failure.” So I said, “Okay, well, let’s look at, let’s look at the mission statement for, say, Cal State Long Beach.” Right? ‘Cause maybe, maybe Cal State LA is just, just really not there. So I looked at that one.

Boz: What did that one say?

Sharona: So it starts with, “As a leading global university…” So okay, so now we have some context. “We provide quality educational programs and services to our students from around the world. In doing so, we promote global understanding and cultural exch- exchange, and foster mutual respect and appreciation for different cultures, languages, and perspectives.”

Boz: There was a learned appreciation, but there wasn’t really a lot about learning itself in that one. Right. ~So I’m~

Sharona: like, “Okay, so maybe the Cal States are not the right place to look. How about UCLA?” ‘Cause I’m like, I’m working my way up. ~Um, ~so UCLA says, “UCLA’s primary purpose as a public research university is the creation, dissemination, preservation, and application of knowledge for the betterment of our global society.” So I’m like, okay, at least it’s more direct. And they emphasize that they’re a research university, and as a research university, the emphasis can be on creation and dissemination and preservation of knowledge, not necessarily the teaching of that knowledge.

But I’m like, all right. Okay. Still not quite what I thought we would have.

Boz: Yeah.

Sharona: And so then I went to University of Kansas, and I went to University of Kansas ’cause I presented at University of Kansas this year. Their mission is to educate leaders, build healthy communities, and make discoveries that change the world. Okay, to educate.

Boz: Yeah. That not only is it in there, ~it, ~it was one of the very first things.

Sharona: Right. ~But- ~But I’m like, okay, so ~th- ~these mission things, none of them are really hitting at it, right?

Boz: All right. So, well, when you were doing that, I, for fun, went and looked up where I got my bachelor’s, which is a smaller university in Oklahoma called the University of Central Oklahoma, which I, I s- I can’t rave enough about my experience there. But their, their mission was, “The University of Central Oklahoma empowers students to become ethical, creative, and engaged citizens and leaders through the commitment to transformational learning experiences.” Okay. L- little bit- Closer … little bit better, but they don’t just have a mission, they also have a vision, and they also have, their core values.

Sharona: Okay.

Boz: They have, ~um, ~six core values, but one of them is, “At the University of Central Oklahoma, we help students learn by provate- providing transformational learning experiences across six core areas. Transforma- transformational learning is a holistic process that places students at the center of their own active and reflective learning experience.”

So it may, okay, might not be just in the mission, but it’s part of, you know, these three things that often… The mission and vision almost always every, every kind of institute has those. But a lot also have this third one of the core values, so.

Sharona: And I went there too with Cal State LA. I was like, “Okay, so maybe I’m looking at the wrong thing.” And I found values, and then I found institutional level learning outcomes. And the, I’m not gonna read the values for Cal State LA, they’re fine, but again, they’re not… They’re the values of the institution, not the values for the student. Like, there’s a real mismatch between where these documents are directed. And I’m not a huge fan of students as customers model. , Like I don’t think that we should be putting a, such a commercial value for the customer thing on our students. But our students are the individuals whom we are supposed to be serving first and foremost, I think, but apparently that’s not necessarily true. So once we finally got to the learning outcomes level, then I was like, “Okay, now we’re finally talking about learning.” And so for example, I’ll read just one of you. Institutional learning goal number one at Cal State LA is knowledge, mastery of content and processes of inquiry. Cal State LA graduates have a strong knowledge base in their academic major and can use powerful processes of inquiry in a range of disciplines. They engage contemporary and enduring questions with an understanding of the complexities of human cultures and the physical and natural world, and are ready to put their knowledge into action to address contemporary issues. So now I’m like, okay, so we have now stated three levels down in university documents that this is what our students are supposed to learn. I can get behind that. And there’s, yeah, there’s four institutional learning outcomes and then there’s, there’s different program learning outcomes, there’s four, , general education learning outcomes, and then there’s some graduate learning outcomes for the graduate programs, so.

Boz: All right. So, and this was just kind of a fun detour ’cause that as- Well- Well, yeah …

Sharona: sort of a detour, but really kind of gets to the heart of the problem.

Boz: Yeah.

Sharona: Right?

Boz: And, and that one of those issues is what is the purpose of a university?

Sharona: Right. Well, and also as going back to the article is what Doug Lederman is suggesting is that if we wanna prove to the public that there’s value in a higher education degree, we have to document and measure something, and he’s arguing it should be learning-

Boz: Which I think you and I would agree completely with.

Sharona: Right. That that’s the value of what a student is getting from a higher ed degree is what they have learned. And now part of that learning could be how much they grew, but it’s really the learning. And so the first question he asks in this article is, “So what do we wanna measure?” And this resonated for me ’cause of some conversations we had this week with some of the groups that we work with when we’re down at a course level and we’re struggling to define these learning outcomes, right? And these are courses that have been around for a few years or a decade or several decades.

Boz: Or a century.

Sharona: Or a century or longer. And so yet we’re struggling to define, because in order to measure these things, we have to know what we’re measuring. We’re struggling to define it, and we’re kind of asking ourselves, especially in the AI time, what do our students really need to know and be able to do?

Boz: Yeah. And I mean, we’ve, we’ve had this conversation, I think it’s come up in math quite a bit, ~um, ~for the last, you know… Heck, it w- it was coming up when I was a, a student at the University of Central Oklahoma. So this has been at least 30 years, probably longer, but it has become a much more widespread and complex conversation now, with AI absolutely being the catalyst and the gas on the fire kind of thing. But yeah, this conversation’s gotten really, really big, really complex very quickly.

Sharona: And what I appreciated about Doug’s article is in this step one, he da- does a, I think, a bigger dive than what we did in this little detour- Yeah … as you called it, and he dialed down into some commonalities. So all of these institutions, they have different missions, they serve different groups, they have different funders, but they really dial in to capacities such as critical or analytical thinking, communication, quantitative reasoning, ethical or civic capacity, breadth or depth of knowledge, and working with others or across differences. So, like, that’s a, that’s a pretty core set of things that might give us a hook to try to figure out is that what students should be learning? That’s what we should be developing in these students is competencies or skills in these types of things.

Boz: Yeah, and if those are, then at the course level, how many of our current things that we teach or assess really are targeted or geared towards one of those kind of, pathways?

Sharona: And some of them I’m proud to say that we’ve incorporated because in our math classes we do have the Common Core standards for mathematical practice, but some of them are not in there. I don’t teach typically in a math class ethical or civic capacity, and yet there’s an entire book called Weapons of Math Destruction. Not mass- Yeah … but math.

Boz: Math, yeah.

Sharona: That talks about some of the very, very detrimental ways math is used. And now that I’m teaching this quantitative reasoning for everyday life course, where we’re looking at the mathematics of voting, it’s really becoming apparent that, you know, I’m working with my students on a daily basis to say, “Here’s three ways you could do math on this set of data. Which one should we use?” And that which one is absolutely not clear from the math, ’cause all three are legitimate math you can do, math processes, but one results in a racially gerrymandered district. One results in a partisan gerrymandered district. One results in completely, disempowering a minority group. So which one do you pick, right?

Boz: Yeah, and I think that’s an interesting point that I try to make with my statistics students at every level of, that I teach statistics. And I don’t think it’s something that enough people really understand, is how easy math can be used to manipulate an argument the way you want it to go, and it still be valid math, and an understanding of that. But, how do you gauge the ethical use of mathematics? And it really is a bigger question then ’cause it, mathematics is not just arithmetic. It’s not just, you know, eight times seven. But yeah, that’s a really deep question that I don’t know we really get that much into.

Sharona: We certainly don’t, I don’t think at the undergraduate level. ~Um, ~definitely this is a conversation happening at higher levels. So I mentioned, Moon Duchin, who I got to see present at MathFest, who is the leading mathematician around, democracy and voting rights, and has an entire lab called the Data and Democracy Lab. And so they do a ton of work in the mathematics of democracy. And so I think in those levels, these ethical conversations are happening, but I don’t think we should wait. I think 18-year-olds should be starting to have ethical conversations about the use of all of the disciplines that they’re studying.

Boz: Absolutely. And I, I would say younger than 18. Okay. I would say this should be, down into the high school years as well. And you, you will see that some subjects definitely lean, lend themselves more to those conversations.

Sharona: I’d like to rephrase what you said. Some, some lend themselves more to it Just because ethics is a little bit more central to their discipline.

Boz: Yeah.

Sharona: ~But, but I think every discipline has… ~It’s not a stretch to say ethics in science is critical.

Boz: Oh,

Sharona: ~if, if- ~Ethics in statistics is critical, you

Boz: know? If you, if you don’t think ethics in science or in mathematics and specifically statistics is useful, go back and look at the history of how science has been used to do unbelievably horrific things, how statistics has been used to, you know, perpetuate racist ideas, and yeah.

Sharona: But what I recall is when my kids were in middle school, and they had their, like, seventh grade US history, they started talking about ethical things and civic things in seventh grade. I don’t know that they were talking about the ethics of mathematics in algebra class, in pre-algebra. I don’t know that they were talking about the ethics of science in their intro biology where they were learning what a cell is, and I think that’s wrong. Like, I think- Yeah … you’re right. We need to, to start prioritizing these conversations across the board, and one of the arguments that used to be made was, “Well, there wasn’t any time. We had too much content to cover.” And that’s potentially an opportunity that AI is giving us, slash if we can get any of this technology that is developed in, that some of the things that we used to spend a lot of time teaching in class are just not worth teaching anymore ’cause they can be done by tools.

Boz: And that kind of comes to, I would say, the second part of this first step of what do we want to measure, that also gets into the what is it that we think our students should be learning.

Sharona: But before we go there, there’s a statement right above that that I wanted to just highlight-

Boz: Okay

Sharona: that Doug’s proposing. We’re seeing these commonalities that we’re saying, but there are some divergences between people within academia and higher education and people outside of it. So Even when we’re coming to say what should they learn and what do we wanna measure, there are multiple constituencies. So it, this is worth having a conversation, and we need to think about how to have it to really emphasize bringing together different groups. Because as he says, frameworks that are being developed within higher education often prioritize deeper knowledge of a discipline than is in many of the workforce based frameworks that are looking more at career skills. So I just wanted to say that this is not a done deal of what we are measuring.

Boz: Yeah.

Sharona: And that we need to have more conversations at every level. We should be having conversations about Calc One. We should be having conversations about an entire degree program. We should be having conversations about higher ed, like every level needs to be having these conversations.

Boz: Yeah. And like you said when we kinda started this off, w- getting into those conversations at the classroom level, at the course level, it really is interesting to see just how hard it is to have those conversations. And we’re not talking about with people that are new to the subject. We’re talking about people that, you know, have taught a, a class like calculus, which has been around for a while and have taught it for years. I remember you once, saying years ago you went to, was it a whole session or even a whole conference on what should be taught in Calc One, and it was, it was a argument. I mean, it was a real argument, wasn’t it?

Sharona: Yeah, it was at MathFest. I wouldn’t say there was an argument at MathFest because, there was some consensus, there were some differences, but what happened was I brought that back with me and I had a faculty member at my institution look at me and go, “What are you talking about? Everybody knows what’s in Calc One.” And I’m like, I was literally just at eight hours of talks where people were not arguing, but debating what should be in Calc One and seeing different models of different institutions. And yet I go to a place where math discipline folks who are not at these conferences, who are not involved in this conversation, just assume that whatever’s in their brain is what everyone agrees should be there.

Boz: And, and with your experience as a coordinator

Sharona: Ugh, my God

Boz: Okay, how true is that statement that everyone knows what’s in calc?

Sharona: We spent eight years meeting every Friday night with a team of basically the same people, and still we’re debating nuances and details of our stats class.

Boz: Okay, but no, what about calc though?

Sharona: I can’t even get them to agree on whether or not we should be able to use calculators in calc.

Boz: Okay, and that- I mean,

Sharona: I …

Boz: and it went way beyond that. I know you didn’t- Way beyond that … y- I know you didn’t coordinate, the first level calculus, for very long- Yeah not like the stats, but you did coordinate it. I did. You did, you did work as a cor- as a coordinator of that course. How m- how much fighting was there when it just came to what was gonna be on a test?

Sharona: It was unbelievable, and it was really personal too. Like, people would really come after me that I was failing personally to write tests that were appropriate to the subject matter, and yet I would have two people fighting me in exact opposition to each other- even though they were both fighting me. They weren’t fighting each other ’cause they didn’t know about each other, but one is saying, “You have to put this thing in,” and the other one is saying, “You have to take that same thing out.”

Boz: Yep.

Sharona: I was just like, “Oh my God.”

Boz: So the comment of, “Oh, it’s calculus. Everyone knows what’s in calculus” is so unbelievably- Yeah … comically incorrect.

Sharona: Okay, so step one was what do we measure? Often people think that the next question is how do we measure it, but Doug puts another step in the middle there.

Boz: Okay. But what did Doug, recommend?

Sharona: Well, I love this question because this is the question that’s kind of at the heart of so many alt-grading conversations.

Boz: Yep.

Sharona: So he says, “Before we get the, to the tricky dilemma of how we might measure whether learners are reaching the goals we set, one key question demands consideration.” And then he quotes people like John Warner, who we just had on the pod a few weeks ago, raises others. But he says, “Is it proficiency or is it growth? Are we measuring whether they can do something or know something, or how much more they can know or do it since before we started measuring?” And this is a big one Are we striving for competence? Are we striving for growth? And at what… I’m gonna add one more. At what point?

Boz: At what point? I was about to say the same thing. And the thing to add to this is the when part, because… And that’s the heart and soul, in my opinion, of the differences between traditional and alternative gradings, is we have a very different definition of the when. If we’re looking at proficiency and not growth, when is that proficiency measured?

Sharona: It’s a bigger conversation than is really at an, like you have to have that again, that conversation has to happen at every level. You have to have it at whatever artificial time barriers we have. So right now, if we’re structured into a term, you have to ask the question of when in the term.

Boz: Yep.

Sharona: But then you have a sequence of classes. So do they have to be proficient in all calc one skills by the time they’re done with calc one? Or is it okay if it’s by the time they’re done with calc three? Is it okay if it’s, they’re done by the end of their first two years of an engineering program? Is it okay as long as it’s before they graduate with a bachelor’s? What about a master’s? Like, there’s so many points that you have to say, “Is this the moment?” And one of the things I do love about some of the rubrics that I’m seeing outside of an institutional level is they’ve defined different benchmarks of proficiency. So there’s your starting point, a couple of milestones along the way, and then when you’re considered, truly proficient. Or in the case of when we looked at, you know, Engineers Canada and some of the stuff they’re doing with competency-based, they define different competency levels. They’re like, “This is a competency of a beginning level.” Yeah. “This is a competency of a developing learner. This is the competency we expect when they’ve graduated undergraduate. This is the competency level we expect at a fully certified engineer. This is the competency we expect if you’re gonna teach others,” right? So they don’t define just, it’s not a binary. It’s not you’re competent or you’re not, and it’s not even that it’s you’re approaching competency. It’s like there are different levels of competency. There’s one level of competency of I can use calculus.

Boz: Yeah.

Sharona: And there’s another level of I can teach calculus. Now, oftentimes we don’t actually learn the use part until we teach it.

Boz: Yeah. And, and even within the use, there’s different ways of using calculus and different– just like any other math subject, but especially calculus, that, you know, yeah, maybe I’m not proficient in this one ’cause I don’t need to be for how I use it. The way a marketing analysis or financial, analyst might use calculus is very different than the way an engineer might use it.

Sharona: Yeah. And this is another one of those things where we can do the math, right? We could, in theory, measure either one of these, proficiency or growth.

Boz: Yeah.

Sharona: Which one should we use is a human decision. It’s not a math decision. So one of the things he, that Doug says is that he leans personally towards a value-added learning measurement, which is growth instead of proficiency.

Boz: But he also, in that last paragraph, and I do think this is a important aspect and point, that creating a system that measured growth would be much more, complex and expensive

Sharona: And then also the issue he’s trying to address, which is restoring public confidence in higher education’s value rather than accountability or internal improvement, is whether or not the product, the students that come out, are actually prepared or not, which is proficiency.

Boz: Yeah.

Sharona: So even though he personally is like, “Well, I love the value-added model,” because of his other priors and other things that he’s interested in, he’s like, “To answer my dilemma that I’m trying to solve, the problem I’m trying to solve right now, which is restoring public trust in higher education and its value,” he’s like, that probably has to be a proficiency model.”

Boz: ~Yep. ~Yep. And that, that’s what he says at that last line, is at least to start.

Sharona: Okay. So then he gets to step three.

Boz: And this is where both of them– I mean, we love the step two question, ’cause that gets at so much of the underbelly and the foundations of what we believe in alternative grading, but the third one is our bread and butter.

Sharona: And I love the fact that he actually said he’s gonna spend less time on this, ~um, ~because he’s not educated in it at the end of the day. He doesn’t say that, but he’s like, he’s realized that the more important task is defining what we wanna measure, and how is it really hard, and he doesn’t really have any good answers. And I really wanna talk to him, ’cause I’m like, “Yeah, we have, we think, a few options for you.”

Boz: That third step, you said it, but you didn’t really call it out. The third step is how might we measure?

Sharona: Right.

Boz: I love the two questions. I really do. Like you said, the second point, the second reason why he was planning for this part to be shorter is- And I’m gonna read it directly. And two, the how is really hard.

Sharona: Yep. But if we’re not up to big things, then what’s the purpose?

Boz: Yeah, but that, I mean, that also gets to a, another reoccurring thing that, that we come across a lot, and I think is one of the biggest difference between K-12 and higher ed, is the understanding and speed of which we try to do this when we do decide we’re gonna try to change the way we measure things. We were working this last a few days ago, we were working with McMasters. We’ve got this ongoing FLC after we did that, and I mean, we’ve got deadlines and, or not deadlines, but goal dates for launching classes that were a year, year and a half, two years after the training. So that’s one of the big advantages, I think, that, you know, the, the leadership of McMasters is giving the grace and the understanding that this is hard. It takes time, and that’s not something we see uniformally when we deal with, with groups

Sharona: Well, and I was thinking about a whole bunch of the things he said about what makes it hard, and the things that we encounter, which is things like scale. You know, it’s hard to scale truly authentic assessment. But this is a problem that’s been around a million years. If you look at the history of specifically math education and things like when Harvard decided to add certain classes to their, to their requirements for entry, there weren’t enough teachers. Like- Yeah … the, the requirement came before you had the instructors. And so this value thing, it’s like we just don’t have enough money and the resources in the system to hire enough instructors to truly do authentic teaching and learning at scale. It’s just, so until we are willing to invest those kinds of resources, we’re going to grapple with what’s the best we can do in the context we have. And so a lot of people, including me sometimes, are like, “Well, you know, the timed proctored test is not ideal.” And I’m like, it’s not, but it does put some external pressure onto a student’s learning that they can’t just kind of put it off and put it off and put it off. At some point they and we have to know can they replicate this work? Can they do this skill out of their own internal resources?

Boz: Yeah.

Sharona: And that is one of the things we’re just gonna have to figure out how to live with.

Boz: So I wanna get back to kinda some of the things that he says on, and again, he admittedly wants to keep this part short because he’s doesn’t have the expertise, and like he said, it’s hard. But he’s got a short version, and there’s a couple bullet points, and I kinda wanted to talk a little bit about the first one, ’cause I’ve got mixed feelings about that.

Sharona: Okay.

Boz: Did you read that, ~section?~

Sharona: Which one?

Boz: The test.

Sharona: Yes.

Boz: Okay.

Sharona: ~And, ~and okay, so what are your mixed feelings? ‘Cause I also have thoughts.

Boz: He talks a little bit about the prospect of some kind of examination that could accurately measure whether students have the capabilities, skills, knowledge, and habits of mind is, is enticing The whether you can create that is a whole different question. But the very next sentence says, “It would satisfy those who favor a significant level of cross-institutional objectivity over the subjectivity judge- or subjective measurement that plagues our historical approach.”

Sharona: Mm-hmm.

Boz: I have mixed emotions or mixed feelings about that one because how do you measure that test? ‘Cause that’s, where a lot of people have this weird belief about testing, about traditional grading, about it being objective.

Sharona: Well, okay. So I’m gonna push that back on that a little bit.

Boz: Okay.

Sharona: In the form of licensing exams, so medical boards, law school boards, engineering certifications, most of those are considered to do a generally okay level, especially the ones related to human safety. They do manage to get those relatively okay.

Boz: Okay.

Sharona: But if you were gonna do it on creative writing, there is no objectivity in that discipline. Like, creative writing is creative. The, the test is do people wanna read it or not, right? So I think the question becomes Again, he’s sort of setting up a binary, I, I don’t think he’s intending to, but it’s like it’s this or that. A test or something else. And I think there is a role for a test in certain situations, in certain contexts.

Boz: Yeah.

Sharona: But the problem I have is that we don’t need this at every single level. And so-

Boz: You know, I, I, as a K12 educator, ’cause I don’t think you guys have to do this at the higher ed, which we could have a whole episode on. But I had to take three qualifying test, actually four, ’cause I had to take another one when I came to California. But in the state of Oklahoma, I had to take three qualifying tests to show that I had developed the competencies and the proficiencies through my attaining of my bachelor’s degree to show that, yes, that I could, A, actually know a little bit about teaching and, and could be left with the understanding that I can do it. B, that I knew what I legally could, can’t, and shouldn’t do as someone who is responsible for the most precious thing, which is someone else’s kids. Right. Which is not something I take, you know, lightly at all. Whether or not, we could just handle that kind of responsibility. I took three different tests. The Oklahoma General Educator’s Test, the Oklahoma Subject Area Test, and the Oklahoma Professional Educator’s Test.

Sharona: Yeah. You wanna, you wanna know what tests I took?

Boz: Yeah, I know.

Sharona: Algebra. Did I know enough algebra to get my master’s? Do I know enough analysis to get my master’s degree?

Boz: Yeah, but did… Those allowed you to teach, but did you have to take any kind of qualifying test to show-

Sharona: No

Boz: that you could teach?

Sharona: Nope. All I needed was the master’s degree. So I do think there is a role for tests. I also think that we have an opportunity and, like, we’re, we’re facing– So many people are concerned about the AI apocalypse, and I think there are legitimate concerns, right? The reports coming out of OpenAI that their AI agents are going rogue on government websites is, is very frightening, okay? Like, they’re trying to hack DoD. This is not a good thing.

Boz: Hey, the part that I found more scary was, where AI has been able to solve and prove math proofs and problems that we’ve been working on, and we couldn’t do it.

Sharona: Right. So I know there’s tons of AI ethical issues, and AI can possibly solve some things. So one thing could be, for example, in your teaching and learning test, you could be given a scenario and asked how would you handle this situation, and you could write a free form, long form answer that in the past would’ve taken people time to get a rubric, norm that rubric, score that rubric, score your thing. AI can do those things. AI is smart enough now that you can train it to get pretty darn good at some of the more sophisticated evaluation pieces, at least at certain levels. I don’t know that I would wanna do it all the way up, but I think it’s a possibility. So I do think there’s a role for testing at certain places.

Boz: But there was also the second bullet point that he talked about in the how, which is the portfolio student work.

Sharona: Yeah.

Boz: Which is another one that, yeah, I find really, really interesting and It’s very strange in my experience working with different educators over the years, is A, the understanding of what a portfolio is, but B, the worth of them. Like some teachers in some areas, like that is their bread and butter. That’s their whole thing. And others, it’s if they have it at all, it’s kinda like some side thing that, okay, the students can do this if they wanna move their grade up a third, you know, from a B to a B+ or B- to a B. But yeah, there’s a really wide range of use and understanding of what a portfolio is.

Sharona: Well, and I’ve argued for a long time, I call it what you grade, but what I really mean is what you assess is what students are gonna learn. When you look at them and ask them a question, whether it’s a question on a test, a verbal question, a portfolio thing, what you ask them to show you tells your students what they should be learning. ‘Cause they’re looking for, for guides, right? They have this mass of information coming at them, and they’re trying to figure out, “What do I really need to know? ‘Cause I can’t possibly absorb all of this. I can’t retain every word coming out of an instructor’s mouth.” So what you put into your final assessments or all of your assessments, and that’s what they’re gonna iterate on, ’cause when they get it wrong, they’re gonna think about it, especially if they’ve got a revision process. So I love the portfolios because I can really get a lot more things in there where I can say, “Hey, it’s important that you’re able to do an artifact of this type. It’s important that you’re able to give quantitative reasoning of this thing. It’s important that these things are there.” And I love that about those portfolios, is that they are a tool that is much more diverse than a single format of inquiring into student knowledge and learning.

Boz: Yeah. All right. Well, we’re coming up on time already. I don’t believe I actually thought we might have had time to talk about the other two at all. I don’t know if we did this one justice, but, um, I do want to kind of conclude with the same way that he concludes, ’cause I think there’s a really big step there or big idea there. So he, he actually, this is how he finishes this article: “I’m running out of steam here, as are you if you’ve made it this far probably. So I’ll conclude with this. If the grand vision that I’ve just laid out came to pass, the next few steps would involve using the new information we’ve gleaned to embrace more curricular and teaching approaches that research has shown to work best.” And the reason I kind of want to point this out is that’s one of the big arguments not just grading reform, but educational reform that I have seen, and I’ve talked about it a lot, but the, we’ve taught it this way for 100 years, why change it? The this is how I learned it, this is my best way, you know, the best way is the way I was taught, the way I will teach. And yet I understand why people have those, those ideas. But we have to get past that. We have to really dive into, there’s so much research about learning, about student learning, about motivation, that we really need to refocus not what we think is best, not what we’ve done in the past, but really become a profession of scientific inquiry and use of that inquiry.

Sharona: I couldn’t agree more.

Boz: All right. Is there any last-minute things that you wanted to bring up?

Sharona: I’m not going after that. Absolutely not. I’m good.

Boz: All right. Well, I wanna thank everyone for listening. This has been The Grading Podcast with Boz and Sharona, and we’ll see you next week.

Sharona: Please share your thoughts and comments about this episode by commenting on this episode’s page on our website, http://www.thegradingpod.com. Or you can share with us publicly on Twitter, Facebook, or Instagram. If you would like to suggest a future topic for the show or would like to be considered as a potential guest for the show, please use the Contact Us form on our website. The Grading Podcast is created and produced by Robert Bosley and Sharona Krinsky. The full transcript of this episode is available on our website.

Boz: The views expressed here are those of the host and our guest. These views are not necessarily endorsed by the Cal State system or by the Los Angeles Unified School District.