The Grading Podcast

===

Jordan Kostiuk: And then the idea will be at the end of the semester, students will compile some number of problems that they’ve received feedback on. Um, and what we’re gonna ask them to do is, like, describe, like, what was the feedback you got? How did you, how did you revise this? Um, I’m, I’m trying to model, like, kind of like, like i- if this were, um, like a publication, right? Like a response to the reviewer, right? So here’s … Like, give us your revision, but also respond to the, the comments that you got. Um, and then I think additionally, they’ll, they’ll solve a couple new problems

Boz: Welcome to The Grading Podcast, where we’ll take a critical lens to the methods of assessing students’ learning, from traditional grading to alternative methods of grading. We’ll look at how grades impact our classrooms and our students’ success. I’m Robert Bosley, a high school math teacher, instructional coach, intervention specialist, and instructional designer in the Los Angeles Unified School District and with Cal State LA.

Sharona: And I’m Sharona Krinsky, a math instructor at Cal State Los Angeles, faculty coach, and instructional designer. Whether you work in higher ed or K-12, whatever your discipline is, whether you are a teacher, a coach, or an administrator, this podcast is for you. Each week, you will get the practical, detailed information you need to be able to actually implement effective grading practices in your class and at your institution.

Boz: Hello, and welcome back to The Grading Podcast. I’m Robert Bosley, one of your two co-hosts, and with me as always, Sharona Krinsky. How you doing today, Sharona?

Sharona: Let’s see. You asked me how I’m doing. You didn’t ask me where I was. I’m confused.

Boz: Yeah, ’cause we did, last time we recorded, you were back in LA, and you are now, but you haven’t been in LA the whole time since the last recording, so

Sharona: And I think the last recording was, like, a week ago, but no I made a quick trip to Chicago, had an amazing opportunity to work with some folks at the Department of Veterinary Medicine at University of Illinois Urbana-Champaign. So I’m tired but happy. I have had an intense… This is my fourth day in a row doing something directly relating to grading for more than 10 minutes, so this has been a pretty amazing time the last several days. So I’m good, I’m tired, and my semester starts tomorrow. Yikes. I know. So I’m fully in the middle of a course redesign of a course I’ve never taught, so it’ll be fun. It’ll be fine.

Boz: Yeah. It’s funny though, ’cause, our semesters start tomorrow at Cal State LA, but it’s not just our semesters, ’cause my daughter is also starting there tomorrow. At

Sharona: Cal State LA, so-

Boz: At Cal State LA, and she is really excited. The funniest thing I think she’s excited about, though, is she’s taking a intro judo class, which is crazy.

Sharona: Which is actually something you took.

Boz: I did not take it. I helped teach it. Oh.

Sharona: You helped teach it …

Boz: but what’s, it’s funny because, she’s been doing judo most of her life and she is-

Sharona: So in other words, she’s gonna start teaching it within a class or two is what’s gonna happen.

Boz: But it posted on Canvas last night, and she was looking through the materials, and she’s really excited ’cause I guess it looks like, from what’s been posted, there’s, the goal is to learn these 15 different basic throws, and one of them she doesn’t know. So she’s really excited about learning this new throw that she actually has not ever done even though she’s been doing it for 13 years.

Sharona: And it’s definitely gonna be fun and interesting and exciting semester, and good news, and we’ll be sharing lots of good news on the pod over that period of time.

Boz: Yep. But we are not here alone today, Sharona. So, tell our audience who is with us today in the virtual studio?

Sharona: So I am super excited to introduce someone that I know and that I just realized I’m not 100% sure how to pronounce his last name, so I’m gonna give it my best. Jordan Kostiuk?

Jordan Kostiuk: Great job.

Sharona: Close? Okay.

Jordan Kostiuk: Yeah. No, that’s it.

Sharona: So Jordan I have known and been working with for, gosh, I don’t even know how many years now, six or seven or eight. Jordan is an associate teaching professor in the Department of Mathematics at Brown University. I probably should say Dr. Jordan Kostiuk, but anyway he’s a former algebraic geometer turned full-time teaching faculty and scholarship of teaching and learning practitioner. I know him through two of my worlds, so not only through alt-grading, but also because we work together on the team-based inquiry learning materials and in that community. So welcome, Jordan.

Jordan Kostiuk: Happy to be here.

Boz: All right. Now, there’s a couple things that we will probably be getting into, but before we get too far into this, something we always like to ask our new guests is just how did you get involved in this kind of crazy world of alternative grading?

Boz: Okay, so hold on. I wanna back up just for a second. So you started this with linear algebra in a three hundred person course?

Jordan Kostiuk: Yes.

Boz: Okay. We have had tons and tons of people on, and we always ask at Origin Story, I don’t know if I’ve ever met anyone Dr. Tanner maybe, but that actually started with a class that large. ‘Cause that is usually one of the big topics every year at the grading conference and a lot of, things that we get pushback on or questions of, one of them is always how do you do this at scale? How do you do this with these large classes? That, and I’ve just never heard of a linear algebra being that large. Yeah, I took, my calc one and calc two were I think four, 400 and 300 plus, but linear algebra that large? Wow.

Sharona: And I wanna, so I wanna say one thing and then ask you. I think we’re gonna want to dive deeply into how you manage this at this scale, ’cause that is right now the number one class question we’re getting. But I wanna make the comment that your linear algebra and the one that I rave about all the time are the same course. We have the-

Jordan Kostiuk: Yes …

Sharona: same materials. Literally we use the same workbook in the same pacing. We might have nuances like you might do vector spaces with polynomials, I might not, but other than that, we’re literally using the same course. So that’s- Yeah, totally … super exciting.

Boz: Yeah, but I’ve seen some of the stuff you’ve done, Sharona. Doing that at 300, ’cause how big are your classes when you teach that class?

Sharona: They were 25.

Boz: Yeah, exactly.

Sharona: Okay. So let’s walk this through. First of all, can you describe your system? What do students have to do to get an A versus a B versus a C?

Jordan Kostiuk: So this will be good. So I’ll step you through what I was doing ’cause so I’ve been on sabbatical for the last year and so as I’m coming back in-into the fall, there’s gonna be some I’d say mechanical changes, but I’d say philosophically it’s the same. So the TBIL curriculum has twenty-five sections in the activity book. Each section corresponds to a specific learning outcome and at the bottom of each section there’s an assessment that’s designed to assess progress towards that learning outcome. And so I I call them fluency assessments in my course. But really what I did was I designed an asynchronous online protocol for this to happen. So we might cover two to three outcomes in a given week and then so every week students will get a, a batch of fluency assessments. And so there’s gonna be two or three associated to new outcomes, and that’ll be their first attempt. And then on a rolling basis, they’ll do second or third attempts.

And right Boz, you were talking about the scale. There’s a lot of things that play in the scale. But one of those scaling decisions was, okay, I don’t really have the capacity to do unlimited reattempts so we’re gonna make this like really structured, and we’re gonna give you three attempts at predictable times. So that, that was a boundary I knew I needed to set early for this to be manageable.

Sharona: So I have a question then because I know exactly the assessments you’re talking about ’cause I use the same ones. Bosley helps me grade them, so this is why there’s an issue. So student does an assessment, you said asynchronously online. They submit it. How is it graded? Mechanically. Is it auto-graded? Do you have graders or TAs? What level of feedback are you giving on an individual fluency check?

Jordan Kostiuk: Yeah. And so this is great opportunity too. So the, so the main reason that I’ve been able to handle this, this scale is as much, I won’t take all the credit for the TBIL at Brown University. The main the main reason I’ve been able to scale this is I looked at the pedagogy and I said, “You know what? I think this is a reasonable thing to do with 30 to 50 students.” And so I challenged myself to be like, “How can I be effective course coordinator? How could I train graduate student and postdoc instructors to engage in this pedagogy with me?” So when I say 300 students I really mean six coordinated sections of 48 students. I’ll teach one myself but then the other instructors are typically a postdoc or a graduate student. And I do some pre-training, and there’s kinda like a semester-long mentoring there. And the system I have serves 300-ish students each semester. But, as far as the mechanics go, we’ve got one instructor and an undergrad TA for 48 students. And so tying back to, like, how does this grading work for those fluency assessments I’ve got one instructor grading those individually on Gradescope. So it doesn’t happen automatically. True to that mastery-based grading, right? It’s either it’s fluent or it’s not. Right? So there’s no partial credit. And I think the comments are pre- kind of predictable, right? So for me I know exactly what to expect. But for my novice instructors I’d say it’s a pretty typical curve where it might take, several minutes looking at the first few submissions to see oh, okay, what are what are students getting? But then eventually it gets pretty quick. But I’d say in total instructor’s probably grading maybe two to three hours a week, maybe a little more. Some of those outcomes are pretty Students make all kinds of mistakes and you might need a li- a little more attention there.

Sharona: Yeah, I think we’ve seen pretty consistently, at least I think I have and I think you have, Boz, too, that we can do a class of 30, possibly 40, but we usually have 25 to 30, on a single outcomes assessment in about an hour, especially if it’s the first checkpoint. So if we’re doing three checkpoints, it’s three hours for a single class of 30. I would suspect, Boz, you could probably get to 40 in close to the same amount of time, maybe a little bit more.

Boz: I, I- I don’t know what you

Sharona: think …

Boz: yeah, I’ve been doing it long enough and Jordan was saying, it definitely takes someone that’s new to it a little bit longer, not because the grading itself, but just some of those comments. Like there’s certain outcomes that I know there’s going to be some algebraic mistakes on expanding a binomial. It’s just one of those that I know, a good chunk of students are gonna make. So those comments are almost copy-paste, and in Canvas we do have those where we can set up our own comments that we can then just post in. So it’s not auto-graded, but it, there, there’s those very typical mistakes that the comments end up being very similar, so yeah they grade pretty fast once you’ve seen it enough to know, okay, yeah, this outcome, these are the mistakes I’m gonna see. I’m gonna see some, goofy algebra in some of these expansions, or I’m going to see, these kind of mistakes.

Sharona: So what I think is interesting, though, is that I’ve spoken to a bunch of people about large classes recently, and I had a conversation at a dinner at MathFest with a colleague who was actually a biologist who came to sit with us, Katie Mattaini, and at least in math, it doesn’t– I don’t see, as a math professor, how I could do really good multiple choice checks in some of these topics. I really want the s- I wanna see all the students’ intermediate work. But apparently in some of these other disciplines like chemistry, physics, biology, where they’re much more conceptual, they are able to write really good multiple choice questions to get at some of this conceptual understandings and things so they can automate using a multiple choice much more quickly. So everything I’ve heard recently about scale in mathematics, absolutely it’s at scale, but it’s with lots of people grading, like whether they’re undergraduate TAs, graduate TAs, post-docs, instructors. So it really becomes about building that team.

Jordan Kostiuk: Yeah.

Boz: Now I did have a question, ’cause this is a topic that has been coming up a lot on this podcast and just, in academic circles for the last couple years. You said you do this, some of this testing asynchronous. What impact, if any, has the, the recent explosion of AI had on your class or on your grading structure? Has it had any? Because linear algebra is… for a while, AI had a real hard time with, things above basic math

Jordan Kostiuk: Yeah, no that’s a great question. And it does so kinda to start with the answer so that, that was a big reason why I’m changing some things heading into the fall. I kinda started TBIL just before GenAI kind of exploded on our campus. And if I’m being honest I do think it’s at the point where, you feed in one of these fluency assessments into ChatGPT or Gemini I think you’re gonna get a high quality response that at the scale we’re grading I can’t honestly say we’d be able to distinguish. And right over those three years, it was really a kind of about balancing this like, okay do I feel, do we know that enough students are navigating this assessment system in good faith for, for us to feel good about it? And I think that’s really what it came down to. And so for the first few years that I did it, I was reasonably confident that like most students were, following the rules, approaching this in, in good faith. I, was satisfied that the incentive structures were in the right spot. Nor was I naive. Like they were– they’re gonna have… But I think when you teach a large course, like you can only wear so many hats for so long. And so for me, the investment in the– I wanna make sure I’m doing the active learning pedagogy really well. ‘Cause even if the students were using AI, like us as instructors I guess I’ll resist swearing on the podcast, but like we, we know when students are bull crapping us when you’re having a conversation with them. You can’t fake that.

And so for, for a few years that, that was like good enough, and then I think finally in the last kinda spring semester it kinda got to a point where we were noticing that like what I noticed in my first year of team-based inquiry learning was you could just have more high-quality spontaneous conversations with students about the course content and that’s what really sold me that the team-based inquiry learning was quote unquote working was like, yeah, I’m having richer conversations with my students about linear algebra, and this is how I know at a high level that this is working. And then we got to a point where it was like me and a few instructors were like, “Eh, I’m not so sure that we kinda have that fidelity anymore.” And so thinking through kinda how to, how do we reinforce this and just rebuild kind of some integrity that, that kind of got lost over time in, in support of having those conversations.

Boz: See, those conversations, first I completely agree with you. Having the conversations with my students is what sold me on alternative grading. You said it sold you on the TBIL. That’s what sold me on alternative grading, ’cause the first time I did it, it was a disaster. But by the end of the semester, I was having conversations with my students, not about points and not about grades, not about, can I get extra credit here, or how many more points do I need, but it was about the statistics. So I completely agree with you on the sold part. And we’ve been hearing this, and this is something that’s come up a lot when we start talking about AI, is just the power of those conversations and why, I don’t know why it seems, especially math educators want to resist this idea of using oral conversation or oral exams as part of their assessment toolbox, because you can learn so much more through a conversation than you can with any kind of test. You’ll never convince me otherwise. But yeah that conversation can be so powerful, whether you’re using it actually as assessment or whether you’re using it as kind of that assessment check of, okay, do I believe this work is actually student work or is this something else? Because this last year Sharona and I taught in this class that she coordinated with, it was a bunch of other professors, and that was one of the things that I loved about the class was the asynchronous testing, because it, it gave me instructional time back, and I lost that argument. It, it came back into the class this last year, mainly, not mainly, exclusively because of AI fears.

Sharona: Not just fears because I wanted to comment on one thing about that, and you said, Jordan, you’re trying to bring integrity back. My concern, ’cause I’m seeing some, some literature come out about this and people saying this, students are so uncomfortable with their own capacity to do this. They don’t believe in theirselves, their self-efficacy, and they’re very worried that everybody else is doing it. So between those two forces, they just don’t believe in themselves. So it’s not even a matter of integrity, I think, at this point. I think many students would like to do it on their own. They would like to learn to do it. They just don’t think they can. They don’t think that their grade will reflect their own ability if they’re competing against students who are using it, and they don’t know how to bridge that gap. So I have a lot of compassion for the pressures on students to use this, and I am therefore reformulating Bosley resents bringing the checkpoints back into class. I actually enjoy it because I have made a conscious decision to actively engage with my students during assessments. Literally, as they are taking it, I will be having some of those oral conversations with students, which I think is gonna just make a number of people’s heads explode. But so be it. Sumo.

Jordan Kostiuk: No, yeah. There’s a lot of complex factors that are competing with each other. And this feels like a a good time. In, in a certain sense I’ll backtrack, but it’ll help where eventually … This is a really long-winded answer ’cause I, I know Sharona wanted me to get to the point of how do students get A, B’s, and C’s.

Sharona: No, you’re good. You’re good.

Jordan Kostiuk: But, so th-this mastery-based grading was always just like one important but small piece in the total assessment structure. So there were these fluency assessments, but students also had like problem sets, which for me, like I put a lot of emphasis on the writing and the process. And I won’t get into like the details, but like the final grade breakdown was still like a percentage-based thing. So an A was like ninety percent overall, a B was eighty percent overall, and it was just like, okay each outcome corresponded to one percent, and then the problem sets were like twenty-five percent. And then I had a take-home final and some other things. And so there was like a small piece that was mastery-based grading. The other stuff, I’d say there was also like a little bit of alternative grading. But at its core, like it was still about, okay, how do I calculate some number that students are gonna recognize?

Heading into the fall and I worked– and I won’t take all the credit for this. I worked with one of my graduate student instructors, Anna Dietrich, and one of my undergrad TAs Eddie Nguyen. So we worked all spring about just okay how do we want to revise assessment and attend to, our various concerns about AI and integrity? And so I think the system that we have in place right now I can share a syllabus later in the link. It is just on my website. But I would say this looks more like what I would recognize as a bonafide mastery-based grading thing. So we’ve got three learning outcomes, like three big learning outcomes, and one of them are those core fluencies. And so they’ll get like an A, B, or C based on their core competencies. And then students are gonna submit a portfolio at the end of the term that will document kinda progress on problem-solving. So we’re still sorting out like the logistics of that. But right, so the fluency component is happening in person synchronously. And so we– and like we wanted, we needed that to have more fidelity with that core competency. But it still didn’t sit right with me. I feel like we can learn a lot about what students can do in a timed environment when they don’t have access to other resources. But solving new problems and communicating them well that for me is not a time-bound thing. And so it didn’t sit well with me to be like, “Okay, I don’t trust my students to submit anything that’s not in front of me there.” So it was really important for me to have this other component of the revised grading structure that was like, “No, you’re gonna have all semester to work on this, and I want you to reflect on feedback that you get, and I want this portfolio to represent what you can do in those conditions and then piecing that together a-a-as part of that final grade.”

Yeah.

Boz: So is the portfolio completely new, or is it just something that you’re revamping or revising with going into this fall?

Jordan Kostiuk: It doesn’t feel completely new, but I think if you looked at the syllabi, it would look completely new. So I had never done a portfolio version before, but the problems that they’re working on. So my, my course always had a problem set component, so just two or three problems. And I usually, like leaning on the t- team-based inquiry learning terminology, I’ll call these flexible extensions. And so the point of students working on these problems was to practice using the core competencies that they’re developing in class to expand new areas of knowledge, new applications new mathematics and so, the problem sets always existed, and it’s just they were graded by the undergraduate TAs and then we didn’t really do anything oh with them after that. But now what we’re gonna do is each week the undergrad TAs are only gonna grade one of those problems. We’re gonna have designated portfolio problems. And instead of the problems getting a numerical grade it’s just gonna be written feedback focused on their communication and their problem-solving.

And then the idea will be at the end of the semester, students will compile some number of problems that they’ve received feedback on. And what we’re gonna ask them to do is describe what was the feedback you got? How did you revise this? I’m trying to model like if this were like a publication, right? Like a response to the reviewer, right? So here’s… give us your revision, but also respond to the comments that you got. And then I think additionally they’ll solve a couple new problems.

Boz: And I love that idea. And Sharona, you have used portfolios in some of your courses. I know there’s been one or two other times that you’ve wanted to and it didn’t quite work out. But I have never really used them. And it’s something I want to. Like it I think the idea of, the portfolio where the students get some agency in what they choose to showcase their learning throughout the semester or throughout the year is a great idea. We’ve talked a lot recently, in the last year and a half on this podcast about some of the different motivation theories, one of them being the self-determination. I think that, I love that idea. It’s something that if I, ever had an opportunity, especially in my K-12 world, to really go back into a classroom, it’s something I would like to try. But yeah I love that idea and having it focused not on just the student agency of getting to choose to highlight what they wanna highlight, but that focus on the reflection, the metacognitive part of it, and how you, how that reflection and how this component led to better understanding something else is a- another piece that I think I would love to have in a portfolio.

Not just reflecting on how the feedback and stuff helped me here, but how that later helped me better understand or better infor- improve on something else.

Sharona: And I agree with everything you just said, Boz, and I also like what you said about the whole responding to reviewers sort of mental model.

Because the class that I’m working on right now, I have in my head that they’re gonna do three projects, which probably are actually gonna be portfolios. I’m calling them projects ’cause I didn’t know what they were, but they’re actually gonna be a portfolio that combines some in-class quizzing and exam that they did on that topic, along with some other things they’re gonna have to put in there.

But I’m gonna have them pick one of the three, and and they get to choose which of the three they want to do this on to do an oral presentation at the end of the course. And I kinda had in my head a very low stakes version of a thesis defense, right? Where they’ve put together this stuff, it’s got a lot of stuff that’s custom to them, and then they’ll be given some guidelines to do a five-minute presentation and then five-minute Q&A from their peers.

And I’m gonna give the peers some structure of the types of questions to ask, but it’s really supposed to be more like, “I’m interested in this.” If you can’t an- if they ask a question you can’t answer, that’s totally fine. You can say, “That is not an area that I talked about,” but then having the skills to say, something that might be related, and I’ll answer it this way.”

So just to kinda have some fun with the last couple of weeks and see what everybody’s been working on because this class is about voting systems, finance, and a little bit of statistics. So there’s some really interesting things I think we can do. So I’m excited to use that mental model of a thesis defense, even though they’re not really writing a thesis or anything like that.

So you’re gonna be doing that this semester. Are you still using the asynchronous fluency checks and just gonna live with those, or what are you doing about that?

Jordan Kostiuk: No. So the, those fluency assessments are gonna take place in person. So not in class. I gotta be very careful about the messaging ’cause I don’t want my students to think– my students are just gonna look at the syllabus and say, “Oh, okay, the course has exams now.

We’re going back to exams.” And so we’ll see. I think by the time they go through the protocol they’ll realize that it’s the same in spirit. But basically we’ve got– I’ve just bunched them bigger. So we’ve got three evening slots of two hours. And so this’ll just be each each assessment day will have a, an item for every outcome up until that point.

And then I think we have a shorter one that we’ll– We’ve– Brown has something called a reading period, so I’ll take advantage of that space. And then the final exam will just be, like, a big three-hour block and it’ll have every, every every outcome and students can take as many as they need to, or I’m hoping a bunch of them won’t come.

Boz: So it sounds like you do our, our– your final kinda or at least that part of your final like we do, where it’s just a final opportunity for a student to show proficiency. So if I’m a student and I’ve already shown proficiency in all these, or fluency is what you call it, which I love shown fluency in all these skills, I don’t actually have to take that final.

Is that right?

Jordan Kostiuk: Exactly.

Boz: Yeah. So that’s exact-

Jordan Kostiuk: I’d rather they not.

Boz: Yeah. That’s exactly how we do ours, and I understand some of it, but the amount of, again, especially math educators that we’ve talked to, and they’re like no they have to still, do the final, and they have to show everything on the final.”

And I don’t– Like I said, I understand some aspect of it, but at the same time I’m like, why? If I was already shown that I can do this, why do I need to show it again in this high-stakes timed environment when there’s other things I should be focusing on, like the stuff that I haven’t shown that I necessarily have shown fluency with yet?

Jordan Kostiuk: Yeah. E-e-e-exactly. I’m with you and folks struggle with that. I think for me– ‘Cause right, the, in, in the universe of possibilities as I was considering, okay, how do I change this, right? One option on the table was always to let’s just not touch the fluency assessments.

Just keep this asynchronous online ’cause it’s re- let’s be honest, right? If I have a high degree of trust in the system and that students are going through this in the way that I want them to, it’s really convenient. And it is how I’d like s- things to do. And so one option on the table was just, okay, what if I do a final exam, right?

That, that’s in person. And really what this is serving is like an integrity check. Which like I don’t love that framing. I don’t like the idea of, okay, I’m going to introduce this exam Because, at its core it was the reason I would introduce that exam is ’cause maybe I don’t kinda trust what’s happening over here, and that just didn’t sit right with me.

But it was also like, I’ve got my core competencies and I think different folks will approach their TBIL course very differently depending on the kind of institution that they’re at. But for me, the core competencies are just were not sufficient to form the entirety of the, the assessment structure.

Like I know there are instances of TBIL where it’s like these are the outcomes and of course they’ll, they’ll do projects, they’ll kinda cover other things, but at its core it was, okay, if a student can demonstrate mastery on these core outcomes, then that’s A level work.

Sharona: That would be my class.

Jordan Kostiuk: Yeah.

Sharona: I’m just being honest.

Jordan Kostiuk: No, yeah. It,

Sharona: exactly, right? But to be clear- and depending on

Jordan Kostiuk: your, your-

Sharona: I’m not Brown, and that does matter. The context does matter. I love Cal State LA. My students are not your students. Yeah. In any way, shape, or form.

Jordan Kostiuk: E- exactly.

Boz: Which- E-

Jordan Kostiuk: exactly …

Boz: which is why anytime we’re doing any kind of, PD or talks or training on how to redesign your course using alternative grading, one of the first things we say is you’ve got to really look at your course.

What’s the context of it? What’s the audience? Who’s taking this course? What’s the … I- is it in a sequence? So all those things matter, which is also part of the reason why you can get t- three alternative graders together like this, and if you ask us, we probably have between the three of us at least five or six different ways that we actually set up our structure of grading.

Yeah, probably a lot more than that, Sharona.

Sharona: I think I’ve g- I think I’ve got five myself,

Boz: but that’s why, because it really does matter. And like you were saying earlier, Jordan, it also allows you to personalize s- to the, your grading structure in a way that feels right to you and your pedagogical beliefs and, your just b- your morals and beliefs of what is good for students.

And that changes from course to course. My algebra II in a high school setting has different needs and different requirements than my freshman level stat class at Cal State LA. That’s just fact. So that’s one of the things I’ve always loved about alternative grading is that it does l- allow you to set up, this personalized grading structure that feels right for your students, your class, and you as an educator.

Sharona: And I wanna piggyback on that because I think one of the discomforts that a lot of us who went to asynchronous outside of class assessments is struggling with when we bring them back in is we’re feeling like we’re controlling behavior now in ways that we intentionally got away from, right? I have reframed that in my mind to look at it more from a student perspective, and particularly a student learning perspective.

Because again, I feel like the students are feeling pressure to use AI because of their lack of confidence and self-efficacy. If I put in a structure where AI is not an option because and I’ll explain what I saw at Illinois in a minute for many students, that actually relieves the pressure. It puts different pressure on, right?

Because now they’re really having to learn it, but are they more likely to authentically learn it if they know that their assessment environment is not gonna allow them to access AI? Okay. I hope so.

Boz: So I- I’ve got to push back on that, Sharona. I’m sorry, ’cause we had this conversat- I can’t remember if it was with Dr.

Linda Nilson or Dr. Susan Bloom, ’cause we had those interviews back to back. But we had that discussion of, the cat and mouse game of trying to chase h- ways to ensure, the misuse of technology wasn’t being done, and it is. It’s a cat and mouse game, and if that’s the only thing we’re doing, and I’m not saying it is with you, but if that’s the only thing we’re doing, we are always going to lose.

The technology is moving faster than we can actually prevent. And I’m sorry, you can think you’re in as much of a proctored room as, as you want, there are so many smart devices now. E- even the s- the AI glasses now- … are getting to where it’s hard to tell that they’re AI glasses,

Sharona: and I understand that we’re not gonna be able to prevent all of it, but I really do believe, I still believe in students, okay?

And what I mean by that is if you ta- if you introduce just enough grit that most of the students are gonna not do it, I think that’s important. Now, what I wanted to say was, because I was at University of Illinois, and they have a comput- computer b- computer-based testing facilities. So they’ve got four rooms.

They have a safe place for people to put their stuff, and they’ve got a couple of proctors walking around. So what happens is you get back the asynchronous nature, because then the computer system, which is Prairie Learn instructors can build their assessments in an extremely flexible system, and students just sign up for spots.

So they’re like on the hour, every hour, they let the next set of students in. Is it perfect? No. But does it get us our time in class back? Yes. Does it prevent the majority of AI at the moment? Yes. Does it take pressure off the students who think that all of their colleagues are going to use it? Yes. So it’s a middle ground.

It’s not perfect and I still think you have to build the relationships with your students, but I just think that too much, it’s too frictionless now. It’s too friction- the temptation’s too high. Our students don’t have those self-regulatory systems. So I’m not opposed to it. That’s just the reality. I think it’s a reasonable addition to our tools, and I still think we should be doing oral exams too.

Yeah. But see, I can do oral exams if I don’t have to give paper-based quizzes inside my class because of my concerns, so yes- I would love a full oral exam … I understand what they said. You can’t chase it with… You can’t… I guess what I would say is you can’t fight technology with more technology.

You’ve gotta put human beings in the loop to help fight some of this stuff. But there are human beings involved because they have proctors walking around, so anyway that’s my pitch. I’m excited to get to work. We’re gonna be doing a project with PrairieLearn to try to get some alt-grading support into their computerized testing processes.

So what are you looking forward to? So then Jordan, tell us what your, this next year looks like. Are you still teaching the linear algebra? Are you doing this in anything else? What’s going on?

Jordan Kostiuk: Yeah. This year looks like linear algebra. I think I’ve pretty much gone over that assessment structure.

And then for the first time, I’m also gonna be doing some like more SoTL work with that. I know Boz mentioned self-determination theory. So we’re gonna be doing data collection in my linear algebra course for this year. We’ve been doing data collection on the course for a long time.

But this year my focus is on how are students perceiving the assessment structure. Whereas I would say, my previous studies have been on like the active learning component. So we’ve got an instrument that’ll measure changes in sense of competence, relatedness, and autonomy.

So I’m curious just how students will navigate that system. I’m really curious about haven’t quite talked about this yet, but like the challenge of doing all grading in a coordinated section, ’cause I have to, give guidance to instructors. And I’ve been thinking a lot about how do I be as not dictatorial as possible, right?

So there’s that balance between I want course policy to be consistent but then, especially in the portfolio, right? I’m expecting that there’s going to be situations where Instructors have different opinions about the quality of the work that they’re assessing. And so thinking through how will I as course coordinator manage those differences,

Sharona: The answer is eight years of weekly meetings where you have to try to facilitate people like Bosley arguing with other people.

I– My recommendation is hire Bosley as your personal coach. ‘Cause that’s how I got through that exact situation.

Boz: Yeah, but it does, it comes down, and Sharona, you and I have actually done a guest post on the Grading Grading for Growth blog about this very thing. S- this idea of actually meeting and calibrating your scoring.

I- if you’re gonna have a group of, different instructors that, yes, will view, especially, work at this level that there’s not just maybe one right answer, but those opinions and calibrating those so you’re all on a similar page so that, yeah, the grading is equitable.

So my, if, if you and I, Jordan, are grading the, group of work that it isn’t unequitable where your students are getting the benefit of some of your beliefs, and I’ve got this really strict, staunch take and nobody is getting fluency

Sharona: But we did. We literally would take a single piece of student work, throw it up on a shared Zoom screen, and then we would all express our opinions about that specific piece of work, not just in vague generalities, but literally what do we each see?

And what was very interesting about that process is a lot of times you would find that someone else would pick up on something that you missed, like something that they didn’t like and that you agreed with once it was pointed out to you, but you missed it. So you s- you start to build this body of, as instructors, mistakes understanding, right?

That you all agree, okay, we all see this mistake, but we’ve decided we don’t really care about that mistake as a group. And you just have to get people to agree that once the group agrees, that’s what it’s gotta be. You gotta go with that. So even if your personal beliefs aren’t that thing, once you get to agreement as a group, so as a group, you might decide this mistake, w- we’re gonna note it, we’re gonna give feedback, but it doesn’t matter.

This mistake over here is super critical, and we as a team agreed that would change. And it takes us about I don’t remember what it was at the beginning. At the beginning it was hours. I think towards the end we could probably get through norming a single learning outcome in 30 to 40 minutes

Boz: Oh, it ju- usually less than that, ’cause we’d spend the first 10 minutes talking about, how the course or how the classes went that week and what was coming up and-

Sharona: No I was thinking-

everything else … I, no, the whole meeting was usually an hour and a half, but a single learning outcome we could do in 30 to 40 minutes of, if it’s the first time doing it that semester with that group of instructors. Because by then we’d had instructors who had been doing it for five or six years, and then we would have one or two that were brand new.

In the beginning though, oh man, there were some knockdown, drag-out fights, and I as the coordinator had to facilitate those, and that is not what I would’ve considered my strength early on, especially when Bosley was one of the ones arguing. ‘Cause that was not helpful when my own coach is arguing with the other people.

But anyway. But yeah. No it’s a thing. But see, it, it’s

Jordan Kostiuk: interesting ’cause I’m like, thinking through my stance and I don’t know if this is the right stance. We’ll see how I feel about it at the end of the semester. But I think when I started this coordinated approach, it was, like, I think I had a goal of okay, I want students to have the same experience regardless of sections, and I don’t know.

I don’t know if I feel that same way because I think I have a better understanding now of how important it is to give instructors autonomy when they’re figuring this stuff out. So like my orientation to these normings are I’m going to make sure that we have really high standards, right?

I feel like my role is to set the floor for what that fluency will have. And– But if an instructor decides that there’s something important to them, and they need like a higher floor than… and if they want to ride or die with that, as long as they’re prepared to have the necessary conversation with their student about “Hey, this is really important to me, and, you might have a friend who is in a different section that differently, but like I really need you to address this on your next go.”

Right? So it’s less– It’s not that I want the instructor to have my exact standard, but it’s more about how can I support that instructor in communicating that standard with their student? ‘Cause I think that’s what’s gonna be really important for– Right, if I’m thinking about the graduate students and postdocs if I want them to take any of this alt-grading with them permanently into their practice, then I think I need them to feel like they have a high integrity in that grade and that they can facilitate that conversation.

And I think there might be places where, you know, if they have a higher standard or they feel like it’s important and I’m unable to convince them to lower it I’m toying with the, the, the right call in that moment might be just to let them go and support them in that conversation So I have

Sharona: two thoughts, but so does Bosley apparently.

Do you wanna go first, Boz?

Boz: No, I think the key thing that when we were doing this in the statistics is even though Sh- Sharona as the coordinator would come in and would have some non-negotiables, like, but for the most part it was when we came into these, these calibration and these norming calls, it wasn’t her convincing everyone else of what the standard should be.

It was us as a group looking at student work and discussing and coming to a, collective agreement that we could all live with. Now, yeah, occasionally there were a few things that, were absolute, nos or things that Sharona as the coordinator was like, “No I’m not negotiating on this.”

But those were so very few, and it, so it really was, it, so we get as, as instructors, we get that autonomy through that argument and, being convinced and having those group discussions of, okay, as a collective, this is what… And sometimes those did change from semester to semester, like where that bar would move just a little bit because of the different instructors in the group.

Sharona: But there’s two other things also that I wanna mention, which is The one area I would push back is I absolutely agree that instructors’ personality and identity have to play a critical part, and they can have different bars. I’d like to try to avoid it, either avoid it being part of the final grade, right?

So just because you’re gonna have that doesn’t mean it has to be part of the final grade. But we had the weirdest four-level rubric scale, okay? Because normally when we teach PDs, we say that if you’re gonna have two levels of success, you should only do that if that difference matters for your grade wrap-up.

Okay? So if you’re gonna distinguish between good and great, then there should be a difference to the final grade. And then we violated the heck out of that for this course, and the reason is exactly what you’re saying. We had instructors who had such a high bar that if we didn’t give them a place to go on the proficiency scale that was above, then they wouldn’t give good enough.

They would not agree to whatever the group agreed to. So what we did is we had two levels. We had a check and a check plus. There was no distinction in the final grade between a check and a check plus. They were equivalent, and we didn’t norm on that. So we said, “We just decide good enough or not, and then you, the instructor, decide is your feedback good or is it great for this particular student.”

So we were giving them the power to communicate with the student and potentially get students to strive for that check plus. It doesn’t have to be in the final grade. It could also be in the final grade, right? So that’s another option. The key is you wanna try to get agreement on good enough, and then so like- you said, the floor. So what I would recommend is go ahead and get everybody together to agree on the floor together, and then deal with from the floor to the excellence. There’s a lot more leeway there- Totally … in my opinion. So that’s what I would do, is I would definitely give room to grow in the rubric.

And I might also point out to those instructors, I don’t know if your students come back to you to ask for letters of reference ever, but having that good to great in the proficiency scale, even if it doesn’t end up in the final letter grade, it’s gonna show up in the letters of reference, and that’s a much more powerful place to put it.

So that’s my unasked for advice to you as a fellow course coordinator.

Jordan Kostiuk: No, I appreciate it.

Boz: All right, so we are coming up on time, but I wanted to go back and make a really quick point about some of the AI discussion we were having earlier. And I really do think that the best, as instructors, our best defense is a, a three-prong approach, which is good pedagogic good pedagogy in the classroom, a grading system that takes, you know, the cheating incen-incentive away, which we obviously believe is some form of alternative grading, and the relationship building with the students.

So my question to both of you is, we’ve talked a lot about the TBIL, and both of you guys are, have been doing it for a very long time. So if I’m an educator and I’m hearing this and I’m unfamiliar or I’m not using it, where can I go to, find a-and be part of this TBI- TBIL community and start using some of this, you know, good peda-pedagogical approach?

Jordan Kostiuk: Yeah. So I’d say your best place to start is tbil.org. And so that’ll have probably most importantly… Oh, sorry kicked the desk. Most importantly will be the TBIL resource library. So there we’ll have curricular materials for pre-calculus, calc one and two linear algebra which we’ve talked about here.

And there’s also a nice page kind of publications from the TBIL community so that tbil.org.

Boz: All right. We are quickly coming up on time already. Sharona, is there any last-minute thing that you wanna touch on before we start signing off?

Sharona: Just anyone who’s still listening, please send us good vibes for the beginning of the semester.

We need all the vibes we can get.

Jordan Kostiuk: Agreed.

Boz: All right. I wanna thank you, Jordan, for joining us taking a little bit of time out of your day to come and talk with us. And for our listeners, you’ve been listening to “The Grading Podcast” with Boz and Sharona, and we’ll see you next week.

Sharona: Please share your thoughts and comments about this episode by commenting on this episode’s page on our website, http://www.thegradingpod.com.

Or you can share with us publicly on Twitter, Facebook, or Instagram. If you would like to suggest a future topic for the show or would like to be considered as a potential guest for the show, please use the Contact Us form on our website. The Grading Podcast is created and produced by Robert Bosley and Sharona Krinsky.

The full transcript of this episode is available on our website.

Boz: The views expressed here are those of the host and our guest. These views are not necessarily endorsed by the Cal State System or by the Los Angeles Unified School District.