The Grading Podcast

42 – Dr. Gaines

===

Dr. Gaines: But when I reflect on the class, like, my class, without any extra prompting from me, they got together, Wednesdays were the days they had quizzes, they got together after class at a time that worked for everybody to go to the dining hall and eat pancakes to celebrate making it through another quiz. Even outside of their accountability groups, I would walk around the building to the study areas, and I would see different groups of different students studying together, or students saying, oh, so and so is like, really good at this. They already met this learning outcome. You should talk to them.

Boz: Welcome to the grading podcast, where we’ll take a critical lens to the methods of assessing students learning from traditional grading to alternative methods of grading. We’ll look at how grades impact our classrooms and our students success. I’m Robert Bosley, a high school math teacher, instructional coach, intervention specialist, and instructional designer in the Los Angeles Unified School District and with Cal State LA.

Sharona: And I’m Sharona Krinsky, a math instructor at Cal State Los Angeles, faculty coach, and instructional designer. Whether you work in higher ed or K 12, whatever your discipline is, whether you are a teacher, a coach, or an administrator, this podcast is for you. Each week, you will get the practical, detailed information you need to be able to actually implement effective grading practices in your class. and at your institution.

Boz: Hello, and welcome back to the podcast. I’m Robert Bosley, one of your two co hosts, and with me as always, Sharona Krinsky. How are you doing today, Sharona?

Sharona: I am doing well. I know I say that every week, but I really enjoy recording the podcast. So no matter what’s going on in my life that day, by the time we sit down to record, I’m actually doing well. So I’m going to say that again today. And I’m going to kick it right back to you. How are you doing, Boz?

Boz: You know, we’re starting the first part of our serious testing stretch at my high school. So that’s always exhausting. I’m more involved in that than I’d like to be, but like you said, this part always is fun. So I’m doing better now that I’m not in my high school doing some of the silly mandated testing.

Sharona: Exactly and we are grateful again to have a third person with us in the virtual studio. I’d like to introduce Dr. Teresa Gaines. We’ll be calling you Gath. Dr. Gaines is an assistant professor at Haverford College in the chemistry department. They received their PhD in organic chemistry from Georgia State University. Their research interests include making libraries of novel molecules to probe at the relationship between their structure and activity to create potential drug candidates for target proteins. Gath is leading a research project on creating and assessing chemistry card games for classroom and individual use as active learning tools. Outside of research, Gath enjoys spending time with their dog, Hamilton, and making fermented foodstuffs like kimchi, miso, and cheese. And we could have a whole podcast on that side of the thing. But welcome Dr. Gaines may I call you Gath?

Dr. Gaines: Absolutely. Thanks for having me.

Sharona: Absolutely.

Boz: So unless there’s something you wanted to add to the intro, one of the things we always like to ask I guess the first time that they’re on is what’s your origin story? How did you get involved in alternative grading? Like what brought you into this?

Dr. Gaines: Yeah, my origin story really kicks up in my first tenure track position, but my thoughts on grading and doubts on grading as a system, and kind of classroom hierarchy as a system, started when I was in graduate school where I had a difficult relationship with a student, where we weren’t seeing eye to eye, despite both of us really trying to make the relationship work for the classroom where there were disputes about grades, how grading was done, why we’re taking off points for this sort of thing. And I was just a TA and I really didn’t get to make the decisions. But I was the person that students would come to. And a mentor at the time, when I spoke to them, they told me that the classroom is a dictatorship. It’s supposed to be a benevolent dictatorship, but it’s a dictatorship and you just have to say, and that should be the end. They didn’t see why I was engaging with this student and that never really sat right with me for ways that I really couldn’t express at the time. However, post grad school, after my post doc where I got to teach my own classes, I taught primarily organic chemistry, and then fast forward to my first tenure track position at a small state school in Mississippi. When the pandemic hit, I had a lot of opportunity to try new things. Some things that I had tried before we went remote. I implemented oral exams in organic chemistry, because I really wanted to have something that I could look at and have a conversation and say, okay, I really understand what you know and how to get you where we’re trying to go. That was great in person, a little bit harder to do on Zoom.

There was a lot of conversation on Twitter at the time about ungrading, at least in the chemistry space. And so that was the first time that I had heard any of those terms. So I saw specs grading, standards based grading, ungrading, and I tried to wrap my head around all of these. And ungrading was the only one that I was like, okay, I think I can visualize myself applying this to an organic chemistry class. So the first year that we were fully remote, I ungraded my organic chemistry class. And I don’t think any time that you implement some sort of alternative grading that it’s perfect at the first time or the second or third or fourth or fifth.

But something that kind of reminded me of that first conversation about the classroom being a benevolent dictatorship was that since I, in the version of ungrading that I did, students and I decided the final grades together, and they only received feedback from me throughout the semester. I encouraged them to revise and resubmit, and we would have a lengthy reflection process before determining our midterm grades and our final grades. And because I had relinquished a lot of the dictatorship powers that my previous mentor said I just have and should hold on to, I was able to have very authentic conversations about growth, about mindset, about progress that I wanted to have with students all along, but the structure of the classroom with me having all of the power was really inhibiting in a big way. And so, yeah, that’s kind of my origin story. After ungrading I’ve been interested in implementing different types of alternative assessments or alternative structures all together in the classroom.

Boz: So, that’s interesting. I don’t know, and correct me if I’m wrong, Sharona, I don’t know if we’ve ever talked to anyone that Ungrading is where they started. We’ve had lots, you know, lots of people that have gone to ungrading, but not.

Sharona: Courtney Sobers, I think, started with ungrading and I think Katie Mattaini may have started there too. And one thing that’s sort of coming to mind as I think about this is how many of the courses that you teach are for students who need chemistry for their major, and how many are sort of just service courses to something where it’s a side thing? Like, what’s the proportion in a chemistry department?

Dr. Gaines: Ooh, that’s a great question. So when I was in Mississippi I would say it was about 50/50. Maybe they weren’t a chemistry major, but if they were a biology major and they needed chemistry and organic chemistry those are the classes that I that I taught. But currently at my current institution I would say it’s upwards of 90 percent need this course, maybe 95 percent need this course for their major and very few students take it as a service course.

Sharona: And the reason I’m asking that is because what you’re describing about having conversations about growth and all this stuff sounds amazing. And given the current students that I’m working with, I am in a pure service course. It’s not even a pure service course, it’s actually a GE course. So these are students who are required to take a quantitative reasoning course, there’s only really three, maybe four on campus that meet this requirement that are not on a direct pathway to calculus. And so the students I get have essentially zero interest in being in my class.

Dr. Gaines: Oh wow.

Sharona: And, and that is making it hard, like that’s smacking me in the face right now. So I don’t even know how I would approach ungrading this, if my students kind of have no interest in the content at all.

Dr. Gaines: Yeah, that’s a difficult situation to me. I will say many of our students in chemistry, my current institution is a little bit of a special case, but at every other institution I’ve been at, students take chemistry because they need it to go to med school, and they also tend to not be interested in the content, but, like, they know that they have to check this box. And so, I’ve had experience in the past where there’s been like a frustrating level of disengagement or only engagement so much that to say, okay, what will my GPA be? And then like, it’s just off of a cliff after that.

Sharona: So we met you when we invited you to be a keynote at last year’s Grading conference.

Dr. Gaines: Yes.

Sharona: And at the time, you had just completed your first year at a new institution?

Dr. Gaines: Thats correct.

Sharona: And if I recall correctly, you chose not to do alternative grading that year due to the need. Well, I, I’ll let you describe why, but could you go from a, just a tiny little bit of that into. So what did you do this year? Cause it’s been almost a year.

Dr. Gaines: Yeah. So my first year at my current institution Haverford college, we’re located just outside of Philly, I decided not to do any sort of alternative grading in my course because the turnaround was just too fast. I was finishing up my last semester in Mississippi. I needed to move across country, over a summer. And then I was teaching all new courses that I had never taught before. For most of my teaching up until this point, I had taught organic chemistry primarily. But my first two years here, I’ve taught an advanced third year level laboratory, an integrated laboratory, and then first year chemistry. I’ll be back on the cycle for organic chemistry soon, but I was teaching classes I’d never taught before. And with the move and everything going on, I wanted to at least experience the class, how it’s been taught before coming in and like trying to pull all the levers. So this year I definitely went in and pulled a lot of the levers where for the first year course that I taught, I ended up implementing specs grading. One of the nice things about taking that first year to observe was that I was, I gave myself more time to prepare to implement specifications based grading. And in the system that I used, we had homeworks and quizzes. The quizzes were in person but students could retake them, and each question was linked to one or more learning outcomes. And then it was just ones and, and zeros. Did they meet it for this assessment or did they not? For homeworks, they could annotate, and resubmit, and then I would give more feedback and we would follow that process until we met all of them. And for quizzes, we would find a time for them to come in and sit them again, and we just tracked those outcomes throughout the whole semester. In the end, we settled on a certain scheme where if you met a certain number of these outcomes by the end of the semester 85 percent of the time they were assessed, you would get an A and then like scale down from there. The one piece of, I guess, a traditional class setting that I kept was the final exam. Mostly because, and even though test scores aren’t the greatest comparison, if I was changing everything else about the class, I wanted something to compare. So I gave the exact same final that I gave the traditional course the previous year, when I taught it. So that’s when I, that’s what I’ve been up to for the past year.

Sharona: So how’d they do the second year on that final as compared to the first year? The same? Better? Different?

Dr. Gaines: The first year in traditional, which I’ll go ahead and say, like, it’s my first time teaching the class. I definitely had a better handle on the content the second time around. But in the first year, the average on the final was somewhere in the mid sixties as far as raw percentage. The specs grading class, the average was an 83. So at least with that one way to compare significant improvement.

Boz: So I want to come back to that improvement in a second, but I had another question because last fall I was teaching in a new course myself that was a coordinated course. So I didn’t have the freedom to do the kind of grading I would prefer to grade. And it was a traditional grading. And I know my experience with doing that was really hard after, you know, doing alternative grading for like six years, going back to traditional was really difficult. So I’m curious, since you had been doing it for a couple of years, what was it like going back to that traditional grading?

Dr. Gaines: It’s hard. When I’m grading exam questions and I have to say, okay, the total question’s worth five points, but this student missed this one thing. So is that worth two points off? Is it worth two and a half points off? If I talk with them and they like, can explain to me what the issue was can I just give it one point off? Like, it’s a lot harder to go back to that traditional system. Something that I did implement was a revision system, which thankfully in my department, a lot of different faculty members have some sort of revision or earn back system. And so for the class when I taught it traditionally they could get earned back on their tests. And so they could get, they had to do like a reflective revision type exercise where they go through with the key. What are the things that they missed? Why did they miss them? What are the things that they need to do to make sure that they’re prepared the next time this topic is assessed. And so I did implement that, but I had a good bit of flexibility and precedence within the department to do that sort of thing. But going back to traditional grading is hard because all of the questions that I have about what is a point, what is a score, what is an A or 3. 0 mean, I don’t have answers for those any more than I did before then.

Boz: See, and that’s one of the things that we do in a lot of the trainings that Sharona and I do is just giving a random problem to a room full of educators with the key and like, okay, you’ve got 10 points. How much is this worth? And I mean, the spread. We’ve done this literally with thousands of people at this point.

And our spread is, you know, usually somewhere between seven to nine points on a 10 point scale.

Sharona: Yeah. We’ll get scores of one to eight, two to nine in a room full of instructors. Yeah.

Boz: Yeah. And that’s always been one of the things that we try to point out about traditional grading is it’s, you know, this average and the math is all supposed to be so objective, but no, it’s really subjective, when you get to the partial credit, unless you’re just doing ones and zeros, they got it or they didn’t and no partial credit. I mean, it is all incredibly subjective. And I don’t know about you, but I know when I was doing traditional grading, especially in high school, where I might have, you know, 120 students taking the same test or a similar test. By the time I finished, I would have to go back to where I started and kind of look and make sure, cause I wanted to make sure I was being equitable in what I was taking off, you know, the partial credit that I was taking, it would take me forever. And people always ask about the workload and the time it takes to grade. And I’m like, I might be grading more, but it takes me a lot less time to actually do the individual grading cause I’m not going, okay, this is seven points they made this mistake. Is this a two point deduction? Is this a one point? .

Sharona: Okay. I have so many questions that I’m starting to write them down. . . Okay. So first question is class size. So how big are the classes that we’re talking about?

Dr. Gaines: Yeah, so when I ungraded my class was about 25 students for the first semester. And then we did un grading again in the second semester and in that section I had 20. When I did specs grading this past year, I had a class of 16. So the first year chemistry course that I teach is a five day a week intensive version. There’s a, there are larger three day a week sections, but because they have different course numbers, I felt liberated to change the class as I pleased. But this coming year I’ve convinced the other two people who are teaching all of the other larger sections to also adopt specs grading. So long as, like, I, like, share the materials and templates. So this coming fall, we’ll have, across four sections, about 120 students.

Boz: Nice. That’s very cool that you’ve come into a place relatively new still, cause this was this your second year or, or yeah, this was your second year, right?

Dr. Gaines: Yep. This year is my second year.

Boz: So yeah. So convincing people by the third year that’s great that you’re bringing people along with you. I love that.

Dr. Gaines: Oh, thank you. They also kind of knew to expect it, because when I applied for this job, I still had my job back in Mississippi and so I really only wanted to move if I knew I would be able to do some interesting things research wise that I was limited by. And so my research talk was about my ungrading project, not my wet lab synthesis and they hired me. So they knew something like this would be coming eventually.

Sharona: So, now what about the upper division? You said you were doing an advanced upper division lab type thing. Where’d you go grading wise with that? Is that what, like, what is its traditional structure and what structure did you use?

Dr. Gaines: So the interesting thing about this class is the structure changes based on who’s teaching it. And every year, somebody else somebody new teaches it. So something that as a department, we’re trying to standardize a little bit more is coming to consensus about what are the core learning outcomes we want for this course, no matter who teaches it. So when I teach, essentially so to contextualize where this course fits in our curriculum all students do a research based thesis or capstone project, regardless of their major. So, in chemistry, usually this is like a, they work in a research lab their senior year if not, if they haven’t started earlier, and then they do a thesis based on that work. In order to prepare them for the process of doing independent research, this third year lab is like an introduction to a research experience. So we have a rotating cast of whoever teaches it because the students for that quarter or that semester will work on that professor’s research project and really kind of put into practice, like, okay, like, you’ve learned how to keep a notebook now, it’s on our own to do it, or we need to make a presentation, we need to write a paper. So all of those sorts of elements are in there somewhere, but depending on who teaches it, the emphasis on each part might be a little bit different. My approach for that class, I taught it last year and then I also taught it this year, was we’re going to have a system of I’m going to give you an assignment and then I’m going to give you feedback and I’m going to give you a chance to make those revisions. And if they’re done, it’s 100%. So not quite specs grading but definitely feedback focused because, again, if the purpose of this class is to get them the experience and to figure out what things are important in the research space, kind of an all or nothing percentage type of traditional system, to me, doesn’t make a lot of sense. And not giving them a chance to revise and resubmit also doesn’t make sense in that context, to me, but other people who’ve taught it may think differently. .

Sharona: And so did you do it traditionally in the first year and then change it or, or how much change happened in that course?

Dr. Gaines: Yeah. In my first year because I taught one half of the course and I was partnered with another person who taught the other half. And so we had conversations before the semester started about how we wanted to do grading. And because they had taught this before I asked them, okay, like, what are the major assignments for this course and we ended up like kind of going half and half where I said, you know, let’s give them like a revision system where we ask for the assignment first. We don’t give them a grade. We just give them all of the feedback, detailed feedback. And then once they submit it again if they met all of those things that we stated then we can just give them a 100 for, for that assignment. And they ended up co signing onto that. The only thing that was graded traditionally was the final project report, and for that we devised like a standard rubric, but we still invited students to give us a first pass where we only give them feedback. It was just there was a rubric added on top of that for extra clarity.

Sharona: Okay. Oh, go ahead. I was just saying, and this year?

Dr. Gaines: This year we did something similar. I taught it again, but I was partnered with a different person this time. This person is a visiting faculty member and so they’re new to the campus and they asked if I had taught this before and I said, yes. And they asked, can we just do what we did last time? And I was like, sure. That’s okay with me. Okay.

Boz: Yeah. So what I was going to say is I love how, even though you did do traditional grading that first year, last year, you were still implementing some of the, you know, four pillars that that is the foundation of alternative grading. And this came up on our last interview and it’s coming up again. You know, we did an episode a few back on a blog post by Dr. Robert Talbert about, or was it, was it Robert or was it David?

Sharona: It was, I think it was Robert,

Boz: About, you know, 20 small things you can do to implement immediately with it and still, you know, within traditional grading. So, it’s funny that we’ve done that and then the last two or three interviews, we’ve seen examples of people doing just that. So if you’re at a institutional context where you can’t go all the way to alternative grading, or if you’re in a circumstance like you were because it so much was new, you didn’t want to go all the way to some form of alternative. You can still implement, you know, parts of it that still is so beneficial to students and still better than just the very typical traditional grading.

Dr. Gaines: Absolutely.

Sharona: Okay. So I have another question, cause I just have so many questions coming up. I want to go back to what you said about the benevolent dictatorship. And this is something we’re going to hear about at the grading conference this year as well. Dr Laila McCloud is going to come and talk about identity and power and her experience in the classroom. How is this new power structure working for you now? You’ve sort of told us that you’ve mentally rejected the benevolent dictatorship. Do you feel you’ve been able to rebalance the power structure as much as you want? Do you want to do more? Like where does that playing out for you right now?

Dr. Gaines: Yeah. So this is a really awesome question that I’ve been thinking about, especially since I implemented specs grading in my first year chemistry class. Some of the learning outcomes were concept based or skill based. But we also had engagement focused learning outcomes that were measured not necessarily by, you know, you can’t, it’s hard to measure or say, like, yes, a student was reflecting metacognitively. But if they engaged in that exercise, they received the mark that it was met for that.

And some feedback that I got when I ungraded, but also when I specs graded with first years was, wow, meta cognition is very hard. And you’re asking me to think a lot about things that I normally don’t think about and it felt like a little bit of an extra burden. And so it’s kind of taking that in hand where last, not last year, the year before, so there’s a biannual conference in chemistry education. It’s every two years on the even numbered year. Two years ago and then again, this coming summer, there was a session called Disrupting Grading, which was kind of the catch all umbrella term for anybody doing anything with under like under the umbrella of alternative grading. And so I hung out in that session. It was an oral session to listen to all the talks for as long as I could. And an interesting theme that emerged was like, or a question that emerged, was how do we help on board students into these metacognitive practices. When I ungraded in organic, the biggest piece of feedback that I heard loud and clear was, Oh my God, there’s so much reflection. Please don’t ask us to do this much reflection. Even though at the same time, with the same hand, they said, I’ve learned the most by doing all of these reflections.

Sharona: Well, I asked how the power structure was working for you. And you mentioned that. And then I want to go back to the metacognitive and are they ready for it thing? Because that’s a whole hot button for me.

Dr. Gaines: So the power structure is interesting, because if we go in this direction, where it’s like a traditional hybrid or more traditional appearing, and even when I had first years, the power structure is assumed to be where I have all of the power and students have none of the power. And what I say goes, even though like, I have to work really hard to break up that conception. And so really where power structures come up is with, at least in specs grading for me over the past few years, is with academic integrity issues where, right, normally the professor says, you know, I observed this you haven’t convinced me that this isn’t what occurred therefore, this is your penalty and there’s not a lot of room for discussion in many cases. But due to specs grading, but also just like implementing certain alternative practices that allowed me to engage with students on a personal level, where they trusted me to have their best interests in mind. A lot of those conversations I’ve been very aware of the fact that this is a power structure that’s not in line with what I’ve been trying to communicate or implement. And so I’m thankful at my institution, we have a student run honor council who we can call to have a mediated meeting. Because with those power structures, if I had multiple students that I was confronting, even if we had meaningful conversations, there might be presumed bias if the outcomes are different. So having an external student run party that I can tap to and say, you know the reason why we’re reaching out to Honor Council isn’t to put you in trouble or mark anything down. We’re just having a mediated meeting. Nothing leaves this meeting. But I want us to come to a resolution in a place where I have an external party to hold me accountable, to being fair, to being equitable. So that’s been a very interesting way to kind of exercise that power structure outside of the co deciding grade structure that I had in ungrading.

Boz: Okay. So can I jump in Sharona or did you?

Sharona: Yes, I’m going to let you, cause I’m like so much to say, go ahead, Boz.

Boz: Well, no, I, cause what you were just saying, you know, is one of the topics that comes up a lot, the whole idea of cheating. Especially, now, you know, going back to when we were remote I don’t know. I think you were still in Mississippi at the time that pandemic hit. I don’t know how long you guys were out. We were out for a year and a half. Like we were completely remote for, for a year and a half in my high school and two years at Cal State. And there was lots and lots of talks about integrity and cheating and you got, we got to do all these digital proctoring softwares and all this stuff. And I’m sitting there going, you know you’ve got that much cheating because you’ve incentivized it when you do traditional grading, because it’s not about learning. It’s all about points. It’s all about getting as much points as you can. So you, you’ve really given incentive to cheating.

Whereas with alternative grading, if you take a lot of that incentive away by either doing revisions or doing reassessments or, you know, however you can fix those mistakes. And you make it more about learning than point grubbing. A lot of the cheating takes care of itself. One of the things I’ve personally done since coming back from pandemic, like I don’t do any testing in class. All of my testing is asynchronous. Part of that is because of the types of questions we ask. They’re more difficult, even though they’re math questions, they’re more difficult to cheat on without it being obvious that you’re cheating on. But, a lot of it is, if the student’s focus is on learning and getting proficiency or mastery of learning targets and not point grubbing, they don’t do it as much. So, I’ve actually gained a lot of instructional time back by not doing my assessments, most of my assessments actually last time and everyone’s Oh, but then they’re all cheating. I’m like, actually, no, they’re not. And I can, I can show and prove that. No, I’ve got actually less cheating than most people have, even when you do it.

Sharona: And one of the ways we know that is because of how bad their initial assessments are, like if they’re cheating, they’re cheating badly. But I, you know, I’m interested about this thing with the power structure and the academic integrity, because I definitely have not divested myself of as much power as I might eventually want to in my gen ed stats course. But my response to cheating is I look at a student, I’m like, you cheated. It’s clear. I have a log that shows that you cut and pasted a chat GPT answer in like, this is not even a conversation. So your penalty is you get no feedback and you wasted one of your attempts. I don’t report to academic misconduct. I’m like, you have a natural consequence. I’m not in the business of making chat GPT better. So I’m not giving you any feedback. You want to try again? No. Yes.

So like at the end of the day, the student doesn’t pass because as far as I’m concerned, there’s no evidence of learning unless they engage with me. So I have both cases right now. I have two students who very clearly Chat GPT use and one of them, I spoke, came on board and is authentically trying the assessments and may succeed, may not, but it’s authentically trying. The other one took 12 weeks to even get in touch with me, had a conversation this week and is now like I’ve locked that person out of all the assessments because I told them you got to come in in person. I can’t do this anymore. And they’re not. So okay. They’re not going to pass.

But I wanted to go back to what you said about the metacognition because I’m getting a lot of pushback in my institution that maybe first year students aren’t ready for this. However, Bosley, what did you guys just do? What did you guys just do at LA Unified?

Boz: Yeah, my district just passed its first new grading policy in almost 20 years that goes way towards alternative grading and specifically a form of standards based grading called EGI and even has some of that metacognitive parts to it, but.

Sharona: What bothers me about this whole first year student students aren’t ready is you have to start at some point. If you don’t start it in the first year, you’re going to have the same conversation in the second year. If you don’t start it in the second year, you’re going to have the same in the third year. And I do think there’s elements about student psychology and development that I think we need to be aware of, but I just kind of don’t buy the I’m not ready to think about my thinking. Like I just kind of don’t buy it now. Maybe there’s too much of it, right? Maybe there’s degrees, but you know, my students. The only time at the moment they really have to get metacognitive is if they, in our class, they have to prove evidence of learning twice. If by the third time they’re not done, with both times and they have to take it the fourth time in between the third and the fourth, I make them do one of those reflection and corrections assignment. So they have to pick one of their quizzes, compare it to the answer key and tell me what did they do wrong technically. But then also I tell them, what did you do wrong in the moment? And I tell them it can’t be that you didn’t study enough that’s not what I’m talking about. What happened when you were sitting down to take this assessment? Did you not read it? Did you misread it? Did you realize you didn’t know something you were supposed to know, and now you know it? Like, I give them prompts to basically, it’s gonna be one of these things. So, is that sort of a more moderate level? And, cause I don’t have any of those engagement learning options, which, maybe I need them.

Dr. Gaines: Oh man, it what you’re talking about reminds me, oh my gosh, can I find this person’s name very quickly? So the terms that I’m thinking of are like, there’s reflection in action and reflection on action. When it comes to like how we’re thinking metacognitively, Oh my gosh. The name is either David Schoen or Daniel Schoen.

Sharona: I’m seeing Donald, donald Schoen, reflective practice in early years education.

Dr. Gaines: Yeah, so I’ve been kind of reading some of their work and other people who’ve kind of like combined different things. There’s a great book called Iterate by John Sharp and Colleen Macklin. That is really talking about iteration as a process more broadly, but they relate it to iteration is how you build expertise in creative endeavors and technical endeavors. And so they go through a lot of like history and background and thoughts on this. And I, I really liked that system because when I asked students to reflect, I do get a lot of the reflection on action. So like, what didn’t you do before kind of this in the moment part, but the reflection like in action, like, Oh man, I realized that I just didn’t review my functional groups when I needed to on this assessment. So I, I really love that. I do want to follow up.

So even though I got these comments that the metacognition is way too much, I got similar comments where, cause metacognition is one of our engagement learning outcomes for the specs graded course. Another one is participating in class community. And so very intentionally at the beginning of the semester, I partnered up students with accountability buddies and we really set the stage for like, what is this? This isn’t somebody that you need to come to and like confess every week, but every week they had a chance to talk to this person about what are their goals for this class? Have they changed? Are they on track to meeting these goals? Have they made any steps towards meeting those goals? And they had class time to do that and talk and one piece of feedback that I got in a similar vein because my class is mostly first years, but sometimes I’ll have a couple of second years or maybe a fourth year who just needs this as a service class. And one of the feedback that I got, and I’m assuming it’s from a second year or fourth year, they said first years aren’t ready for this type of community engagement. But when I reflect on the class, my class, without any extra prompting from me, they got together, Wednesdays were the days they had quizzes. They got together after class at a time that worked for everybody to go to the dining hall and eat pancakes to celebrate making it through another quiz. Even outside of their accountability groups, I would walk around the building to the study areas and I would see different groups of different students studying together or students saying, Oh, So and so is like really good at this. They already like met this learning outcome. You should talk to them.

And so I feel the same way about metacognition where for community, I think it’s a lot easier for me to see where metacognition is largely invisible. But I looked at that comment, like students aren’t ready. And I’m like, yeah, maybe they’re, you know, not doing this to the highest degree, but it’s making a difference in their day to day experience of this class that is impactful. And if we can convince other subsequent classes that they’ll take to have a similar structure, then they might eventually get to that point. So, I, I think maybe if there were different ways of making the metacognition visible to students in a way that they see their growth, that might, like, take off some of the or alleviate some of the cognitive load. Because at least in the feedback, they feel like it’s a constant treadmill, but they don’t feel like they’re getting anywhere. And it’s hard for them to see that, like, no, like you, you are, you are going somewhere.

Boz: But that’s funny. And you mentioned that when you first said that. They say that, but then also at the same time, they say they’ve learned more in your class than almost any other class. And that just, I think it was Harvard or it was one of the Ivy leagues that did a study, you know, decades ago that showed how just how bad we are at judging how we learn. They took all these students and they’re like, you know, what’s the best way for you to learn, what kind of structures and they’re all like, you know, yeah, the traditional lecture and stuff. But then the ones that didn’t have that actually learned a lot more. So it just kind of reminded me of, yeah, they might not like it or might not see the benefit. Like you said, they think they’re on a treadmill, but they’ve actually ran a marathon and just didn’t know it.

Dr. Gaines: Right.

Sharona: It sounds like you have some research that’s in your discipline. Which I don’t know if you’re still doing it, but looking at these novel molecules. And then do you also have an education research project?

Dr. Gaines: Yes. So right now my lab, it’s a little lopsided. This year will be a little bit more even split between organic synthesis and then chemistry education. Where the big thing that we’re working right now is a review article of chemistry games and education literature. Because that’s where I really want to do a lot of my work in the education space. But there’s not a lot of literature that explains why this is such an interesting area to focus on. So we’re assembling that ourselves by doing a review and then also a meta analysis.

So my students in that space, I have two fourth years right now who are working on that project and another third year who’s working on that one. But because we’ll have a hundred students in specs grading next year, we’re going to run a study for our sections of specs grading. I have a student who will be a senior next year who’s super excited about working on that project. And then like a couple of other like small projects here and there, more so, like a scholarship of teaching and learning emphasis where it’s like, this is a really interesting classroom phenomena. I would love to study this, like in this context type of thing. So a couple of those going on as well, but, yeah. We have a few different projects happening in my research group right now, but it’s good.

Sharona: I kind of feel like a matchmaker right now because what I want to do is I want to take you and I want to get you in touch with Dr. Eden Tanner. I don’t know if you’ve listened to the podcast at all, but Dr. Tanner is at Ole Miss, I believe, and is also, number one, also in drug delivery big, huge lab in drug delivery. And, when she started teaching, she teaches Gen Chem, 170 students in one section, and she saw the equity gaps that the institutional data had. And she took that researcher’s perspective and said it would be the height of hubris to think that I’m going to be any different. So I need to do something differently.

And so she went to some form of alternative grading that I can’t entirely explain because it still used, I think it was basically standards based, but it used a lot of multiple choice quizzes and scantrons and things. Cause again, 170 students, right?

Dr. Gaines: Yeah. So, so many students.

Sharona: Proof is in the pudding. Out of 170 students she had 169 take the final, which was the ACS exam, and she had an 168 of them pass it from what I understand that’s unheard of.

Dr. Gaines: Yes. That is unheard of .

Sharona: So I don’t understand this, but I wanna take you and connect you with her. And then we just spoke with an episode that’s not even out yet with two biology, neurobiology?

Boz: One of them was biology, the other one was

Sharona: biology, education, wildlife.

Boz: There was like wildlife, fish and something biology.

Sharona: Oh yeah. Department of Wildlife, Fish and Conservation Biology. So two biology professors at UC Davis. Who are now going into this and there have the same questions you’d have like, wow, there’s not any real literature here. We need to do some. So they have a project going on. So I think I want to like match make all of you. I want to get all the previous guests

Dr. Gaines: Oh, that would be great.

Sharona: together in a big zoom and be like, okay, you need to work with you and you, and then we have an engineering education project that’s in process. And one of the things that’s really hard with all of these is, I don’t think anyone has hit on exactly the right protocol for measuring student changes in student movement, because we don’t want to use DFW rates. That’s just a terrible measure. So we, on my study, we were using identity and growth mindset protocols, and we ended up looking into some motivation achievement goal theory. But now Brie and Rob are using something else. Do you remember what they said? Oh, You can look up the transcript. But they were using a different protocol. That had, that it was like a different one of these metacognitive protocols. And so I think that there’s an area here where everyone who is trying this is trying to approach it from a different psychosocial perspective of human behavior and we haven’t quite, everybody who does this is like this is clearly working. This is clearly working. And yet we can’t measure it in a way that dBER, I guess is what it’s called, Discipline Based Education Research, would accept it as a qualified study. So, I’m kind of fascinated by this. So we want to get you in contact with Brie and Rob because of the protocols.

Dr. Gaines: That would be amazing. Something that because actually I just had a meeting with my student who wants to lead this project for, for next year. And talking about like, how do we want to collect data? And like, of course, scores and averages, if we have them take the same final again we can take all of those things and compare them. But then the question is, well, something that might be interesting is to go and catalog, like what are the student experiences in this sort of class? And how can we, and figure out like, how can we say if this is similar or significantly different than a student experience in a traditionally graded class that is similar. Which is a lot harder to do than just giving surveys making an interview protocol, validating that your questions are making sense to students so that they are answering the question that you’re trying to ask and not what they think you’re actually saying. So it just, it would definitely be more involved than one year. But, my student seems very interested in the process of developing an interview protocol even if it’s just like hammering out the kinks so that then we can go and collect that experiential data from interviews and then code it and summarize, okay, what are the ways that students are experiencing a specifications graded course?

I think another issue with alternative grading and discipline based education research is it’s not well defined. You can ask a million people how they do on grading and each person, it will be slightly different.

Sharona: Well, it’s worse than that. Each person does it differently in each class. Like you do it in two ways, I do it in three. So it’s actually very contextual to the placement of the class. One of our interview people, TJ Hitchman said, there’s an inverse relationship between the motivation of the students for the class and the amount of structure you need. So the less motivated they are, the more structure you need. But what we’re finding is that whole issue you had with your students saying too much metacognition. Well, that’s the problem with the interview protocol because we’re trying to talk to students about their experiences and they’re not used to thinking about their experiences. It’s the same problem.

Dr. Gaines: It’s, it’s so hard.

Sharona: Boz, did you find the protocol that they were talking about or the theory?

Boz: It wasn’t Self Determination Theory, was it?

Sharona: Oh, it was. It was. It was self determination theory. So there’s another one that we can throw at you that that’s the one that yeah, these two researchers from UC Davis, they felt that that actually came out of their interview protocol. As they started coding everything they were finding connections with self determination theory, which I now have to go down a rabbit hole and learn.

Dr. Gaines: Yeah, actually a lot of my other colleagues in the chem ed space, when I talked to them about specs grading, they’re like, yeah, you should look at that in the context of self determination theory. And I was like, Absolutely. I mean, we need a theoretical framework but we’ll look at a couple of others. One that would be an interesting study that I think would be difficult for me to do.

But if you are able to partner me with somebody else who’s doing this in their class there’s something called the ICAP framework, I C A P, and I forget what the letters stand for, but the way that this works is you have someone observe or you either record a conversation and there’s a tier for like, you know, is the conversation just like regurgitating? Is it someone linking new content together and making an inference? Is someone explaining something in detail to demonstrate their knowledge? And so you can put it somewhere on this framework. I think it would be very interesting to observe or record conversations in office hours or just ask hours, or even if the class has a large active learning component. In a specs graded course versus a traditional course to see what types of conversations are happening.

Sharona: Well I just looked up the main article and it stands for, it says, this is a framework that defines cognitive engagement activities on the basis of students overt behaviors and proposes that can be categorized as interactive, constructive, active, and passive. So those are the I C A P of the four. So that is fascinating. Yeah, I feel like what I’m hearing from the people we’ve been talking to recently, as well as my own research, is that we are grappling with finding the correct framework on the psychosocial side because the ed researchers I’m talking to are engineering education, chemistry education, math education, biology education, and almost all of them are coming out of their original disciplines. So they have undergraduate degrees or whatever in engineering, in chemistry, in biology, in math. So we’re coming at it from, discipline background that doesn’t match with human behavior cognition,

Dr. Gaines: right?

Sharona: Oh my goodness. This is just fascinating. This is so exciting.

Dr. Gaines: Although something that I would love to partner with somebody in, if anyone is listening since a lot of us are not, don’t have our PhDs in whatever discipline based education. I think coming together and like having like a, even if it’s just like a slide or an infographic of what is self determination theory. What is constructivism? All of these like really popular theoretical frameworks and just like, a digestible bite. So as you’re like going through, like, if we can get like 10 people to like, make one of those a year and just like put them somewhere that would be helpful.

Sharona: Well, I’m also thinking maybe, we’re looking at the next directions for the grading conference, Boz. I’m kind of thinking a mini conference on education research into alt grading and bringing together people who are actively doing ed projects to share the social frameworks and to make connections like just a one day mini conference specifically focusing on research around alt grading might be a fascinating place to go.

Boz: That really could be a very, very interesting place to go with it. I hate to do this, but we are actually coming up on our time already. I mean, I think we could go on for another hour easily. This is, this has been a lot of fun. I do also, remind me, cause I don’t remember, we usually you know, post all of our recordings of the grading conference every year. First to just people that were registered, but then eventually to anybody are some of our keynotes are different. And I don’t remember if you were one of our keynotes.

Sharona: I don’t think we posted our keynotes, probably.. We don’t typically and this year our keynotes are only going to be for participants. So do you remember though, I think that we either did not record or we did not post to anybody but participants for the keynotes?

Dr. Gaines: I think it was only posted to participants.

Sharona: Yeah.

Dr. Gaines: But my memory could be wrong, but I still have my slide deck,

Sharona: right? That’s one of the things you get for registering it. But any closing thoughts before we sign off for you and what’s next?

Dr. Gaines: Yeah, I am super excited for next year, specs grading on a large scale, ungrading and alternative grading on a larger scale has been. My most asked question. So I’m excited to try it and experience all of it to eventually share again. But I’m super excited. I’m glad to have partners and my colleagues who are willing to jump in and try something that they’ve never done before and just heard about from me. Yeah, I’m, I’m excited. I’m optimistic. I still feel good about this whole alternative grading thing. So

Sharona: I’m so glad to hear that cause of course we’re kind of crazy evangelists. Boz over to you.

Boz: So a couple of things. First, it’s starting to become kind of a podcast tradition now. Anyone that’s doing any kind of research, like what you’re doing, basically an open invitation, either when your research does come out or when you hit because you’re so at the early parts of it, if you just get to a benchmark and want to talk more about it, we’d love to have you back on. And I also just wanted to thank you again for last year’s keynote. It was one of the highly received keynotes, anyone that has an opportunity to listen to it and hasn’t, I highly encourage you to. It was a great keynote last year. And like I said, this is, I couldn’t believe when I looked up at our clock and I’m like, Oh wow, we’re almost, we’re almost at our hour and we could easily keep going. So thank you so much for coming on. This has really been a lot of fun.

Dr. Gaines: Oh, thank you so much for having me.

Sharona: And everyone register for the grading conference. It’s coming up really soon.

Please share your thoughts and comments about this episode by commenting on this episode’s page on our website, http://www.TheGradingPod.Com. Or you can share with us publicly on Twitter, Facebook, or Instagram. If you would like to suggest a future topic for the show, or would like to be considered as a potential guest for the show, please use the contact us form on our website. The Grading Podcast is created and produced by Robert Bosley and Sharona Krinsky. The full transcript of this episode is available on our website.

Boz: The views expressed here are those of the host and our guest. These views are not necessarily endorsed by the Cal State System or by the Los Angeles Unified School District.