The Grading Podcast

91 – Feedback with Joe

===

Joe Zeccola: If we are practicing skill and knowledge acquisition, the only way is to give them reattempts. And why are we averaging it? I, I, again, I, you can make a list that we would, we would the three of us, whether it’s actors, athletes, whatever, skill people who, it took them a while to figure it out and then became great. No one talks about the, took ’em a while.

Boz: Welcome to the the Grading Podcast, where we’ll take a critical lens to the methods of assessing students’, learning from traditional grading to alternative methods of grading. We’ll look at how grades impact our classrooms and our student success. I’m Robert Bosley, a high school math teacher, instructional coach, intervention specialist and instructional designer in the Los Angeles Unified School District and with Cal State la.

Sharona: And I’m Sharona Krinsky, a math instructor at Cal State Los Angeles, faculty coach and instructional designer. Whether you work in higher ed or K 12, whatever your discipline is, whether you are a teacher, a coach, or an administrator. This podcast is for you. Each week you will get the practical, detailed information you need to be able to actually implement effective grading practices in your class and at your institution.

Boz: Hello and welcome back to the podcast. I’m Robert Bosley, one of your two co-hosts, and with me as always, Sharonaa Krinsky. How are you doing today, Sharonaa?

Sharona: I am doing well. I actually got a little bit of a slower start this morning, which is fantastic because I had quite the week. It was a amazing week, but I was in three different cities within six days, so that was pretty fun. I do want to start, though, with a shout out. I had an amazing experience at the University of Kansas this week. They invited me in for an all day workshop on grading reform and there were probably maybe 50 people throughout the day. And what a incredibly welcoming group of people. The Center for Teaching Excellence, the CTE there, is fantastic. And I just had a fabulous time. I am, I’m so excited by the efforts they’re making there. How is how are you doing today?

Boz: Well, I’m a little bit tired this morning ’cause the night before we were recording this was my oldest daughter’s prom. So had a little bit of a late night playing chauffeur for her, but she had a absolutely time of her life, so it’s worth it. But yeah, little bit draggy this morning.

Sharona: Well, and our third occasional co-host is here, joe Zeccola. Welcome back to the pod. How was your week?

Joe Zeccola: Good, thank you for asking. Really excited to be here. We moved into a new house, so we’re getting more moved in. Living with my, my parents moved in with us so we could all be together and so that’s creating adventures for all five of us. My 6-year-old, my wife, myself and my parents. Good week though, because as Bosley knows, I’ve been losing some weight and building in strength training and I’m down to the lowest weight I’ve been at in three decades, so that’s nice.

Sharona: Oh, oh my gosh. That’s fantastic.

Joe Zeccola: Yeah, that feels good. Congratulations. Yeah. I am a, I’m a walking, talking advertisement for the weight loss drugs that people are using as long as they’re smart with them. ’cause they’re, they’re not a fun ride, but they work really well. And then you add in the strength training and you feel a lot better. So it’s it’s nice to feel better and start getting my energy back. And I’m actually looking at buying clothes in smaller sizes, which is fun. Oh wow. So I won’t be the much fatter version of Bosley anymore. We’ll be similar sizes.

Sharona: Well, and you said something before we started recording about this being the most fun part of your morning. And I have to say, yesterday I was just kind of down in the doldrums, I think I’m hitting Boz that that mid-spring semester slump that I hit right around week 11, week 12. And I was like, I just wanna crawl into a bed and sleep for a week. Because this past week when I was traveling, that was my spring break. So I have no break. And then this morning I’m like, oh, I can get up because I’m recording today. So I agree with you, most fun part of my morning.

Joe Zeccola: That’s awesome. I got a lot of sleep last night. It was my one night to sleep. Tomorrow I gotta get up at four ’cause we gotta take my dad to get hearing aids put in at like 6:00 AM So yeah, it was nice to get some sleep and to do this. And I’m in the doldrums as well. I’m way behind on my grading. The nice part is we’re gonna talk about feedback. I’m giving a lot right now, so that’s where my head is. But honestly, like this fires me up to, to enjoy what I’m doing. ’cause it just reminds me why I’m doing what I’m doing.

Boz: Yeah. And we always are thrilled when we have you on, you know, you and I go way back, so, and we don’t get enough time to just

Joe Zeccola: No.

Boz: Hang out and enjoy. We do not.

Sharona: And Joe and I go way back to my history of math class, which was at least a year ago, you know.

Joe Zeccola: Do I think it feels, it feels like two, I think we have two year a year.

Sharona: Think it was two years ago, right?

Joe Zeccola: Yeah. Yeah. And, and we immediately hit it off and, and we also have the theater connection that I’ve not got a chance to really work on, but Yeah.

Sharona: Yeah. Well, anyone in LA wants to come see my son in Anastasia. It’s in a couple weeks. You can go to my theater group’s website. We’re gonna invite you, Joe. You should come see him.

Joe Zeccola: We should. He’s gonna be, it’ll be fun.

Sharona: It’ll be fun. All right. Boz Joe teed it up. But what are we talking about today?

Boz: So, yeah, we asked you to come back on because, not long ago there was a article in the Grading for Growth blog that we’d love to gather material from. Anyone that’s not subscribed to that you really should be that.

It’s ran by Dr. Robert Talbert and Dr. David Clark, great friends of the podcast very much into the weeds of this kind of work. But they had an article, a guest post on improving the feedback given to students was the title. And I thought it was really interesting ’cause it’s based on some research that was done in a specific math class. But when I saw the conclusions, I kind of wanted to, bring you on Joe, to talk about this and to see if we think this is actually more than just issues that we might see in a math class, or are these things that you think that you see personally in your English classes? Or do we think these are more universal issues? And then also talk a little bit about how we’ve personally dealt with some of these issues.

Sharona: And what I think is interesting for this particular article, Boz, is most people think that most math problems have an answer, right? That that’s the standard thing. Like in math, you have it so easy, we just have to know if the students have the right answer or not. But this is based on proof writing and there’s actually better and worse ways, or more elegant ways or simpler ways. There’s like all these different ways to write a proof and have it be correct. So it’s kind of a nice moment where there’s not a single right answer in math. And so that the feedback becomes really important.

Boz: Yeah. You know, and I’ll admit, I used to think this too, but even more base level mathematics, there might be a single answer, but there’s lots of ways to getting there. And some are more efficient, but take more skills. Some are, you know, take a little bit longer to do, but can be done with less knowledge, you know, of mathematics. And I know that’s one of the things that has always bothered me is when you have, and I think this is probably more seen in mathematics, but when you have people that aren’t just looking for the right answer, but looking for the way they did it, like it’s, yes, you got the right answer, but you didn’t do it my way. So I’m still taking things off, but.

Sharona: Or on the flip side, one of the biggest criticisms we get about common core math, especially in the elementary level, is why aren’t you letting my kid do it the way that we all learned? Why aren’t you letting them add in two column addition? Why are you forcing them to do these weird other ways? So we kind of get it on both sides, right?

Boz: Yeah, but let, let’s jump into this article. The very first problem that it calls out is that students don’t use feedback. So, Joe, I want to ask you first, is that something that you think is just in mathematics?

Joe Zeccola: Well, I guess the way to preface that is no, it is not. Right? That’s very much a problem with our profession. I mean, I have, teachers and students telling me, so I won’t tip my hand as to what locations, that the feedback, if it’s given at all isn’t given in ways that can be used. But I mean, like the key line that I, my first note when I realized I was gonna be taking notes when I read this, my first note was when I read right away, their second disclaimer was feedback is not for revision in the study. Right. So right away we start with, okay, so it’s not actionable. So, you know, for lack of a better word, who cares.

Boz: Exactly.

Joe Zeccola: Right.

Boz: And, and you know, this is something that I, I’ve talked about several times ’cause I, I’ve always given what I would consider fairly extensive feedback. And yeah, for a long time it was a very frustrating process because I would spend hours doing this and students would spend seconds looking at it. Because what’s the only feedback that they cared about was the overall percentage or the overall score? Because my grade, when I was doing traditional grading, that’s the part that’s important to them because that’s the part that it’s not about, it’s not about growth, it’s not about learning, it’s about point gathering and point accumulation.

Sharona: If I could just join that though, when we say the feedback is not intended for revision, that’s different than not intended for learning because I think most of us would say that we give feedback so that students could learn. I think what you’re saying specifically, Joe, is the disclaimer was because we’re in the traditional points system, this particular research, when we say revision, we mean revision on that same assignment. Right? It’s not just getting better for the future in some sort of vague way.

Joe Zeccola: Right. And that’s, and that’s my issue, right? Because it’s getting better for the future in some sort of vague way. So you wonder why the feedback isn’t paid attention to, Bosley’s point, the reason kids weren’t paying attention to his feedback back then is he wasn’t building into his class a structure that said, now go redo it. Right? And, and what it made me think of, and I’d said this when we were just talking before, was again, when I did something wrong, I, these days because of where, you know, the journey with you all is taking me, I equate almost everything to when my dad first taught me how to play baseball. And I, find it amazing because my dad would not in any way call himself a teacher, nor would I call him a teacher. But his instincts sure were right. He started with what I did well. So I was never discouraged when I, when I learned. And more importantly, he showed me what I did wrong and then let me do it again till I got it right. Right? And is that not how we all learn?

Boz: Yeah. I mean, I had the exact same experience with my dad when he was coaching me on how to be an offensive lineman and how to play football. It’s the exact same thing. You do it. Okay, this is what, this is what you did wrong. This is what you can make better here, try this. And then you do it again and you do it again, and you keep doing it until you can do it well.

Joe Zeccola: And you feel really good about yourself when you do it well. And there’s nothing that solidifies learning better than seeing yourself take feedback and take a step forward.

Boz: Yep. But what was missing out of both of our experiences, I’m gonna guess, my dad never said, okay, that was a, that was probably a seven and a half outta 10 on that block.

Joe Zeccola: And walk away. We’re done. Sometime in the indiscriminate future you can get better.

Boz: Well, even if he was having me do it again, and just kept grading them and scoring them, am I gonna be listening to the feedback more? Am I gonna be listening to the score more?

Joe Zeccola: Right.

Boz: Because the score was what’s gonna make my performance or my grade better?

Joe Zeccola: And also though like just, I mean this is taking it to a different place that we don’t really want to go. But I mean, does the score align to the feedback? And what I mean by that is, I dunno what a seven outta 10 is. What I do know is if my score says not quite there yet. Right. And they know they want to be there. Yeah. The feedback and the score are saying the same thing. You’re almost there. Make this change.

Boz: Yep.

Joe Zeccola: Right. So again, actionable and aligned to a grade as opposed to, I got some rando points. And it’s over, you know? But again, I just wanna say before it sounds like we’re bashing the article on this amazing blog that has all this great thought for us. Their conclusions, I think are our conclusions.

Boz: Oh, yeah. No, and I’m not bashing anything she said. I’m completely agreeing. I’m, I just, I want to have this conversation, taking it past just what the actual research was on and do we think this is more universal than just this math class? And is her conclusions actually more valid because of that? Yeah.

Sharona: Well, and I wanna just clarify since we are going into the article, the disclaimer is not that this person didn’t think that feedback should be this, correct? It’s that the research was done in a context where the feedback was not given in a revision environment.

Joe Zeccola: Correct. Right. They were just describing the research terms. Right. They weren’t, they weren’t saying what they thought was best practices. They were just saying, this is where we got this data. Right.

Boz: But I, I would take it a step further. ’cause before I did alternative grading before I completely went away, I did have retakes, I did have, not revisions, but chances for students to make up those quizzes or retake them or replace them. And it was still the focus though in traditional grading is on the points, right. I really think that you can do a lot of great things and they’ll have some impact. But you are hog tying yourself if your grades are still based on points, percentages, and averages, because that’s always going to be the most important thing that the student looks at because that is what’s making their grade.

Joe Zeccola: Yes.

Sharona: Now that being said, they did point out a couple of other problems why students don’t use the feedback that I think are important. So if we say, okay, we know, and most people in this blog know, let’s say we do manage to get the feedback aligned and a revision, there’s still a couple of problems. Even in our alternatively graded classes like statistics, we sometimes have terrible trouble getting people to actually read and use the feedback, even though it is very carefully aligned and uses all these best practices. So one is time. If you don’t give time in class for the student to look at it, the chances of them looking at it and using it go way down. And that was what they found. That’s my opinion too.

Joe Zeccola: I could not agree with you more. Because of this adventure I talked about, moving with my parents, I’m more behind on my grading practices. So I haven’t given kids as much self-assessment work as I would normally do on the essays, yes. But I try to do it on everything. So I’m just, I’m making sure they’re getting the learning they need. So what I literally was thinking about this morning and independent of this podcast, ’cause, I’m in the middle of looking at some essays and what I was thinking was, I need to give them, right after spring break when we’re resetting for the home stretch, a day or two to look at where they are at everything, be able to self-assess, think of where the feedback is, think of where to go. And literally my thought was if I don’t provide the time in class and say it’s important by how I use instructional time, they’re not gonna look at it. And why should they?

Boz: And I, I think you just said two incredibly important things, Joe. First, and this is one of the recommendations that was put forth in the article. At the end of it, they did a summary and a call to action and their very first thing was give students time to review the feedback during class. But I think you said two really important things, Joe. It’s not just giving them time in class to look at it, it’s giving them time to do something with it. Whether it’s doing a self-reflection, whether it’s doing some sort of group activity, looking at feedback and doing peer reviews. Doesn’t matter what it is, but it’s giving them something to do with that feedback and giving them that time in class to do it. Second point that you said that, I cannot overstate the importance of that statement, if you don’t give it importance in class, why would students think it’s important? If it’s important to do, then you should be making it as an educator a priority in instructional time for it. ’cause that’s what sends the message to the students. So.

Joe Zeccola: Even if it becomes a homework assignment, devote time in class to setting it up to talking about it. Everything about the way you use it needs to suggest this matters. And just throw in and say, do this on your own. That doesn’t say that.

Boz: Exactly.

Joe Zeccola: Right? I wrote something just to say the reflection thing. When, when I read the piece, I wrote something just a quote where it says, students need to be told or given a chance to reflect on how the feedback might relate to a different task and as discussed in the previous section, that often means they need an explanation about why something is correct or incorrect. So the why, right? They also need direction on what actions they should complete as a result of the feedback. And I just was reminded of one of my first sort of aha moments, it was when Bosley and I were working together. The first one that sort of underscored that I was taught early on by a lifelong educator to really prioritize reflection. And I think I’ve said this on a different podcasts, my students will all roll their eyes collectively. No one makes them reflect more than me. But the aha moment was when I stumbled upon this great quote from John Dewey, arguably the founding father of public education, we do not learn from experience, we learn from reflecting on experience.

Boz: Yep.

Joe Zeccola: And again, without fostering that space and giving some guidance telling students how to use this stuff. I mean, again, we’re the ones who are teaching them. You gotta teach ’em how to learn too. Right? Obviously if you’re getting college seniors, one would hope they’re already doing that, but all of us know, ” they should know this already” is the enemy of education, so you make sure you foster it.

Boz: Yep. And Sharona, that was one of the, the big changes that you made in the statistics class last fall, I think it was where we actually beefed up the reflection assignment that we give in the stats class after one of the checkpoints.

Sharona: Actually, I think we’re doing this semester for the first time.

Boz: Was it the change this semester? Okay.

Sharona: The change this semester. So what we found was we were giving five attempts to demonstrate proficiency at certain learning outcomes twice. And almost no one took the fourth attempt. A lot of people got it within three, that’s fine. But of the ones that didn’t get it, many, many, many of them skipped the fourth and just did the fifth attempt. And so we were thinking, well, that fourth attempt doesn’t seem very useful, but if we cut it, then there’s only four. And how are they gonna do it? So we moved, we previously used to have, in order to take the fourth attempt, you needed to do a corrections and reflection assignment. That was just a couple of questions. Now, instead of it, we said, well, what if we move that assignment to right after the first attempt? Have everybody do it, but make it much more intentional.

Boz: Yep.

Sharona: You know much more, what exactly did you get wrong? How does it differ from the answer key? Why? Why did you get it wrong? What was going through? Do you understand what is the correct thing? So we really beefed it up, but we expanded it too, and we’re trying to do it within the class time. Now, I don’t know yet how well it’s working because training of instructors to do that is also a component.

Boz: Yeah. And even though you and I both know how important it is to give class time to do it, i, I am unconvinced that most of the classes that don’t have that supported hour long workshop are actually able to have the time right now to do it without understanding the importance of it. That’s gonna be something that’s skipped over. But.

Sharona: Yeah, I’m less concerned about the unsupported class because my experience with the unsupported class is that the students who are not taking advantage of these opportunities have different reasons than they do, the supported class is the ones where we get the students who are coming from a much less college ready background and need more intentional time and instruction on this stuff than the unsupported. So, I don’t know. I’m less worried about it, but I agree, it’s still for those students who do need it, it’s a problem. But.

Joe Zeccola: And that’s always the thing, right? Like the thing that makes me be more intentional about it is I, I get hung up on that should thing in terms of I don’t know who the kid had last time. Even if I do, I don’t know what they necessarily learn. So the more we do it, the better. I agree with you. You want to focus on the ones where you know they need support. ’cause you’ve got the data that says that whole group needs it, right? But you never know which kid in there, all of a sudden you do something deliberately that no one else asked them to do deliberately. Right. Stuff starts happening.

Sharona: Yeah. It’s just more of a pick my battles thing. Oh yeah. I’m not gonna get on the instructors in the unsupported class because.

Boz: Some hills are worth, some hills are worth charging Amazon, and

Sharona: I’ll I’ll on the, on the supported class. But I did think that not having time, not reading it carefully was mentioned. The other one that I thought was really interesting, and this goes more about how students use it, is it said many students reported that they would read the feedback and make quote mental notes unquote, about it as they prepared for exams.

Joe Zeccola: I love that. I love that. One of my favorite pieces of advice I ever got in college that I keep saying to my students over and over again, and it, it holds so true. And I’ll, I’ll tell you the joke that lines up with it in a minute, is if you don’t write it down, you’re not gonna remember it. So the mental note is gone into the ether within moments. Right? And it came from a teacher who used to say to me, one of my writing instructors for screen writing, used to say, when you have an idea, you better write it down. Right? And here’s a funny one. I have an idea for a book that my wife has wanted me to write. My wife is convinced that how much I’m into all the stuff that I do, and that’s pre, by the way, grading work with y’all. So it was more about the way I do cognitive worksheets with kids that I go through thought process.

wrote down a title for a book and I still can’t find it and it’s killing me. Right? And I wrote it down, it’s somewhere in my phone in pages of pages of notes that apparently I didn’t title well ’cause I can’t find it. Right? And so that’s the point, like, again, a mental note. Yeah, good luck. Like I wrote the thing and I can’t find it much less giving them something to actually do with it that hits multiple modalities and makes the task long enough and worthwhile enough that the transfer is likely gonna happen. More importantly, the imprint in a memory is gonna happen.

Boz: Yeah. Well, hey, I have never heard you say that so if you ever do like, I think that is an incredible idea for a book and working with you. I know you do it better than almost anyone I’ve ever seen. I I know we’ve had other English colleagues on this podcast. I, I know I’ve actually talked to Dr. Ashleigh Fox about some of that stuff and you guys have talked and hit it off and absolutely love some of the stuff you do along that. So I think that’s a great idea. You definitely should write that book.

Joe Zeccola: Yeah, I should pitch it with her because if I find someone whose jam it is and we do a little co-write, that might be, A) a professor’s gonna be a better person to want to have on it. But also I just think having someone else to sort of brainstorm with, because I just did something else. And I’ll say quick, we’ll move off it, but I just did a cognitive organizer this fall that I wasn’t that successful with and I wasn’t sure why. And then I did it with, that was with 11th graders who are AP students. Then I tried it with my 10th graders because the spring of the 10th grade year I do a sort of prep version for AP. They were better at it than my 11th graders. And now I’m going, what did I do differently in the process of teaching this? And I’m literally really scratching my head because it is not a talent gap with the students.

Boz: Yeah.

Joe Zeccola: It is not like my 11th graders are strong. I did something different. And they were all looking at me going, this was so easy. I love that you gave us this. And I’m like, oh my God, that was the point, but I don’t know what I did differently. So having someone to sort of think about that and maybe doing some research on it would be fun. But going back to the feedback aspect, again, just how intentional do we make it. And just to give you guys something to bump off on the way you gave me. The other thing that really hit me on the article, and this is huge, is, where is it? Where is it? I’m looking at the summary and call to action.

Yeah, so it’s the fourth bullet. Even when the work is good, tell the students what components of the work you evaluated and one of my big things that I’ve gotten better and better at, and this is an English teacher thing, so I think this will resonate more with English and social studies teachers, but it’s important. Is when you’re grading if you grade everything, it’s gonna take you an hour, right? So I pick things that are most important with me with each essay, and I pull things along little by little and I’ll give rubric feedback on other stuff. But I literally tell them, like, especially early on, because of the cognitive stuff we were talking about, I focus on argument. I focus on how well are you supporting your, your claims with evidence and reasoning, right? But I tell them, right, and also for the feedback purpose, tell them what you looked at so they know what they’re working to improve. And it’s such an important distinction. And that’s also, again, they also point to rubrics as well, which is also gold.

Boz: Well, and I think that’s a great leeway into the second problem that this article brought up. ’cause I think this is almost as big, if not as big as the first one. And it’s one I think is overlooked a lot more than the first one, which is students don’t understand the feedback. And I don’t know how many trainings I’ve done where, you know, we, we think, especially as you get into to high school and to higher ed, that students have been given feedback for years. Of course they know what to do with it. They might have been given some feedback at, at different levels for years. They haven’t been getting your feedback right. They hadn’t been getting the, the nuanced way that you do it. So actually teaching the students how to use your feedback is just as important and I think more overlooked than. Giving students, you know, time to actually look and deal with the feedback.

Sharona: Well, and there’s a surprising piece to this that I don’t know if it’s getting worse. It’s definitely a theme I’m hearing right now is I find that students don’t read the feedback carefully enough so they don’t understand it because they haven’t actually read it. So the very first thing they come to say mean, they say, I don’t understand. And I say, what don’t you understand? And they say the feedback, I’m like, great. Read me the feedback. And they’re like, why would I read you the feedback? You wrote it, right? It’s sort of the mental thing. I’m like, no, no, no, please go ahead and out loud. Read that feedback to me. And halfway through the sentence, they missed something there. It’s, they literally missed a word, missed a phrase, missed something. Yep. So if you ask them to read it out loud I. It’s, it’s one of the first stages. They don’t understand it. ’cause literally they’re not reading close enough.

Joe Zeccola: And just to circle that back with the whole idea that the feedback in this study wasn’t actionable. In your case it is. That creates that loop for that conversation. I’m looking at a book I was mentioning to y’all. It’s instructional feedback. It’s Smith, Lipnevich, and Guskey and Thomas Guskey. And it’s called Instructional Feedback. The Power of the Promise, the Practice. And when they talk about the principles of effective feedback, and it’s actually what, the is chapter one. What they talk about is a framework for thinking about feedback. The last sort of italicized instruction provide students the opportunity and a reason to respond to your feedback, right? So you’re creating the space, but also there’s a reason you have to do it. Right? And then in the second chapter where they say characteristics of effective feedback, it says again, what do I want students to do with this feedback? And if we don’t have something other than learn. Right, which again means what, when you get into the, the details, then what are they gonna do?

Boz: But going back to this stated problem, I’m gonna ask you this again, Joe, do you think that is a math only? I love how I, I wish we only record audio. There’s sometimes I wish we did video too. ’cause he’s almost falling out.

Sharona: Don’t bring that up. I’m thinking about it.

Boz: Oh God.

Joe Zeccola: We’re gonna be on it. We’re on YouTube about going to video it YouTube channel soon, right? Yeah.

Sharona: Yep. I’ve thought about it.

Joe Zeccola: Well, I’ve lost enough weight that I’m not quite as mortified at the thought. Still mortified, but not quite mortified. But yeah, it is funny ’cause I feel like Bosley and I share the same brain minus the okie slash goomba difference. But right. Yeah. I mean, it is not just math. It is all of us. And that’s the thing. We’re assuming students speak our language and we don’t think about how we make it a shared language. And if it’s not a shared language, then again, who are we talking to? It’s an echo chamber. The kids, not even the kids. ’cause we’re talking about college students students are not supposed to understand it unless it is written for them or spoken to them. Using materials and resources they are made to use as tools as opposed to you’re speaking without any of that. Right.

Boz: Well and that’s kind of why I wanted to bring it up ’cause I didn’t want anyone that either has read this blog or that, that listens to this podcast thinking, oh yeah, I can see why that’s a problem in math or any kind of high technical field. ’cause you’re using terminology that the students don’t know or you’re using high level math, academic language. No, it’s not that. It’s again your students. You need to teach your students how to read and understand and use your particular feedback. ’cause everyone does it differently.

Joe Zeccola: A hundred percent.

Boz: If you look at mine and Sharona’s feedback on the same test. We have very similar philosophies. We have very similar mindsets. You look at our feedbacks and they’re different. They might be pointing out the same issue, but we’re write ’em in different ways. So just.

Joe Zeccola: And that’s every teacher. Yeah. That’s all of, that’s the universe of teachers. Yes.

Boz: So just to assume that because students have had feedback for many years that you giving them feedback, they’re gonna automatically understand it, is absolutely ridiculous. And it surprises me how many teachers, when I do PDs and stuff and I bring this up, they kind of have an aha moment. Like, oh wow, you’re right.

Joe Zeccola: Yep. Well, I mean, just to go one step further, looking at the proofs, again, the reason I’m really aligned to this in a way I didn’t expect when I said to Sharona, yes, yes, yes, let’s do the gig. And then promptly forgot the time. Is I’m in the middle of grading essays. That scoring and giving feedback on essays and looking at those proofs and looking at essays, there are so many moving parts to performance tasks that kids turn in. There are so many different things you can focus on, and I think you’ve got three different buckets to talk about.

You can’t focus on everything. What are you focusing on? And then I would argue, and I think both of you would agree, please tell me, that many of our colleagues don’t know what they’re focusing on. Which means the grading, it depends on how the thing hits them when they read it. So for example, with an essay and Sharona, this would, for your history of math, things like that, this is exactly what you and I were problematizing, what are we trying to assess? Right? That sometimes it’s how good the writing is. Sometimes it’s the other thing. And for me, I’m very focused in on, I love it if the writing’s great. That’s irrelevant to me until I see that they can argue, right? Because it’s argumentation that is the big thing that takes them all the way through college. So I can’t be wowed by great writing that’s a poor essay. What I’ll say is, the writing’s amazing, but I’m zeroing in here. But how many teachers have been taught? ’cause it’s not necessarily they’re just doing it wrong, to know what they’re grading and what they’re focusing on and how does that get to the kid, the students.

Boz: Well that a great point. And Sharona, I’m sorry, I’ll let you jump in in a second. I know you’re wanting to, but that’s a great point. Any kind of alternative graders, our foundation is built on our learning targets and one of the pillars is our assessments are directly aligned to those. When I first started as a math instructor, my first test I’m going through students work and whatever, and I’m just taking off points on things that are wrong. Well, some of what, what was wrong might have been what I was trying to assess, and some of it might have been they made a third grade arithmetic error that for whatever reason. So, yeah, my feedback wasn’t focused because I wasn’t focused on what I was grading because my grades were based on, how many points and percentages they got in homework and quizzes and tests and all this other stuff. But that’s a great point that if we are not focused on what we’re grading, our feedback’s not gonna be, and how are students gonna actually focus then?

Sharona: So what I was gonna say, Boz, that it follows directly on what you were saying. It’s actually, I just wanna point out it’s not just negative feedback that needs this alignment. Positive feedback needs it as well. So you can’t say something was correct or good without saying what is correct or good about it. So a lot of times in my linear algebra class, especially if an answer is completely correct and I’m gonna give it a proficient mark, I will say answers are clear, complete, and correct, ’cause those are the three things I want.

I want clarity, I want it to be mathematically correct and I want it to be complete. And so I will say those three adjectives to describe the correct stuff. And I was looking in the article. And they said that there’s another study that they linked that we’ll put in the show notes that they studied four math faculty collect whose grading was collected over entire semester. And the two most common form of feedbacks was check marks with no other feedback and direct edits. So the check marks, first of all, you couldn’t always tell what line of the proof they were or of problem they were talking about. Was it the logic, was it the writing, was it the notation? What was correct about it? And then the direct edits, they would change some wording but not say why it was changed. Like why was this wording more important than this other wording?

And that ties back to Joe what you and I found when I freaked out after that first project got turned in in history of math. Because I’m like, oh my God, the writing is terrible and I don’t know how to teach writing. What am I gonna do? And then you asked me this question, you said, well, how’s the math in this? And I’m like it makes no sense. And that’s what you said. You said, well, they don’t understand the math. And I was like, whoa. So I was like mind blown that I could give the feedback based on the math, which I know how to grade and how to assess and how to help. And you said, and you were right, the writing was gonna fix itself. So I think which,

Joe Zeccola: yeah.

Sharona: Yeah. So both positive and negative is super important to give useful feedback on because how can they do more of the good stuff if you don’t tell ’em what’s good?

Joe Zeccola: And you know, it’s funny, I was saying this earlier, I mean, the positive feedback is so much more important in that when we do something well and get it right. We want to keep getting it right. The research around reading strategies across studies shows that when kids become aware that they utilize one or more strategies that good readers use, they want to acquire more of those strategies that they understand that they’re doing something well, makes them want to do more things. Well, for me, Bosley, one of the big things that Bosley taught me that was really helpful for quick feedback was there’s a couple of Google Chrome apps that allow you to use the clipboard more quickly. So I literally have cut and paste that I start using and usually what I’m scoring in an entire assignment that’s not a performance task, but just a quick reflection or something, I can do the whole assignment in less than a half hour for a class because I use cut and paste and end up falling into five to six different categories anyway, so I just cut and paste them. But what I do is I use language from the proficiency scale. For when they’re doing well as well, saying your thing does this, and I quote the scale that I want to drive them to anyway, same thing that Sharona is doing is saying, I’m using the language that I wanna reinforce. And the research certainly shows that that’s what people respond to. And then just, I wanna add one more thing and that, and Guskey’s got a new book with is it Fray and Fisher, or Fry and Fisher. Do we know,

Boz: I think it’s Fry and Fi Fisher.

Joe Zeccola: So Guskey Fry and Fisher Grading with Integrity. And it’s early on in the text, but the coolest thing, and it just should remind us all, and I, the reason I wanted to say it is just, it tells you how important feedback is in this way. The research shows, hang on, let me.

Sharona: Well, to be clear, it’s spelled Frey, whether or not it’s said Frey or Fre, but it’s F-R-E-Y-E-Y.

Joe Zeccola: Yes. I just don’t know how it’s pronounced. Right. The effect of grading on students’ performance and basically, so one of the earliest studies on how students learned is affected by grades in teachers comm was conducted in 58. And from this study, what they found out was that research results showed that students who received standard comments with their grade achieved significantly higher scores than those who received only a score and a grade. And you can infer from that a check mark, right? Those students who received individualized comments, that even better this led Paige to conclude that grades can have a beneficial effect on student learning, but only when they’re accompanied by specific or individualized comments from the teacher follow up studies conducted years later, confirm Paige’s results. And the thing that they highlight is that it doesn’t have to be lots and lots of comment. It could be a couple quick words that focus with the grade. But the point is the kids feel seen, they feel like it’s targeted. They feel like they’re getting something they can use. And this is research that now we’re talking about 70 years worth, a little short of 70 years worth of data that support it.

Boz: And once again, you have keyed up the third problem from this article perfectly. That third problem is students can’t apply the feedback. And I want to actually read the first two lines of this part of the article because you used almost identical wording and I don’t know if it was subconscious ’cause you read this. I don’t think it is. ’cause I think I’ve heard you say it before, but the article reads the third problem is that students can’t apply the feedback they get to new task. We tend to only provide students quote feedback. We also need feed forward.

Joe Zeccola: So just, so again, just the fact that we share the same brain, one of my notes for this is the feed forward stuff is gold is what I wrote when I read. I mean, there it is, right? Like, what are we doing? And again, I wish, I mean, I know so many caring educators who resist this stuff, but again, why are we teaching in a classroom different than we teach in other places? Like, I don’t get it. You know what I mean? What’s our goal?

Sharona: Yeah. And I have a historical answer that is

Joe Zeccola: Yes. Right? Yeah.

Sharona: The answer is we wanna rank and score students and suppress a bunch of ’em.

Joe Zeccola: There it is. Right? Yeah. So in other words, the system ain’t broke. It’s doing what it was designed to do.

Boz: That’s a great point. We are trying to force a tool to do something that it was designed to do just the opposite, right?

Joe Zeccola: Right. It was designed to put, to decide who the ditch diggers were to, to quote Caddyshack, right? Yeah. Yeah. Ugh.

Boz: But I kind of wanted to bring this idea of feedback versus feed forward, back to our dad analogies when it came to him coaching you baseball or your dad coaching you baseball and mine football. Our feedback wasn’t just, you know, oh, you didn’t do this right. It was, drop the hips further, explode more. It was telling us what we need to do, not just what we didn’t do or what needs to be improved. It was giving us feedback of how to apply that, what we need to do, drop when you’re in your three point, drop that foot back a little bit further or. Did you experience the same thing with your dad with baseball?

Joe Zeccola: Oh, a hundred percent. And the thing I was just thinking was, I’m looking at him right now through the window. He is looking at me like I’m nuts. ’cause. He’s just seeing me talking through the window. But again, I’m gonna tell him this later when we have dinner or something about how great he was at this. He also like, it’s amazing the things you remember because again, I didn’t reconnect to this until I started doing this work with both of you is I remember the flood of warmth that fell over me when I realized I was doing something right in baseball because I wanted to be a baseball player. Right. I remember that, translating to, and he’s never gonna listen to this. He’s a lawyer in Vegas. If he does, we both remember this moment where my friend Steve was a much better athlete. We had a moment where we were both pissed at each other and he was gonna try to throw a ball by me and I was gonna try to hit it. And I remember I was so angry, but I had enough confidence in me that, and this kid was a way better athlete than me. He threw the ball as hard as he could, hit a ground ball right back at him. We both looked at each other and he was a little pissed ’cause I hit it and we never talked about it again. We both talked. He goes, you’re right. You did hit that one right.

And it’s because I wanted to be good at this thing. And my dad started with just noticing what I did well and then tons of specifics. Do this, do this, do this, adjust this. You know, when you throw, make sure you step towards who you’re throwing. But it was that twofold thing of starting with these are the things you’re doing well and now here’s how you get better. This is what needs to be adjusted. And that combination made me feel really good about it, you know, and, and maybe the kid doesn’t like it or the student doesn’t like the particular task, but the knowing they’re doing something well makes it a lot easier to fix the things that aren’t going well. We’re human beings. Right? And it’s just like those two things are just fantastic. And again, very specific. I could not agree with you more.

Sharona: So I don’t have the dad sports thing because my parents were so anti sport, like I can’t even tell you for themselves, for us everything. Right? Just really anti sport. But I am hearing what you’re saying and thinking about something going on in my life right now. So we mentioned at the top of the podcast, I’m very involved in theater. And we, last night we were at rehearsal and we did a full run of the show. And then everyone sits down for what are called notes. So directors give notes and everyone has their phone out or their notebook or whatever. Everyone has a way to capture what is about to be told to them. And those notes are about what just happened for the purpose of fixing stuff for the next time, or keeping it. So there are positive notes, there are negative notes. They also talk about how you never give fellow actors notes unless requested by the actor. It’s only for the di the creative team to give notes. They’re the only ones empowered to give notes. And at the very beginning, you know, like the director says, Hey guys, I, one of my pet peeves is I don’t like to give the same note twice. You are expected to take the note, meaning the next time you do this, you need to apply it.

And so that structure, there’s no grades, there’s no any of this. This is, you are taking notes, I’m giving you notes, and it’s this instant feedback thing. And the actors will, can you clarify this or the director will say, don’t ask questions. Come up to me after. So there’s different structures, but what a great way of doing feedback.

Joe Zeccola: Or feed forward.

Sharona: Or feed forward.

Joe Zeccola: Right? Yeah. So my mentor in theater directed over a thousand plays, A man named Davey Marlon Jones like a second father to me. And I’ll never forget when I was I, I ended up being his longtime assistant director. I’ve directed about 30 plays myself. That’s where Sharona and I have a nice connection. But I remember being a sort of extra in a production of Trojan women and it wasn’t an extra, where there’s four or five of us that played guards that were on stage for, you know, the women of Troy who were being held prisoner. And I remember once being lost in a place and I was in this place and I just realized I was not where I was supposed to be, so I just dropped off the set and just walked out. And then his note to me was, great job jumping into the ocean and getting outta the way. And everyone laughed. But the whole point was even then, he was like, good instinct. You knew you were not where you were supposed to be, and you got the hell off stage, right? And all of his feedback was, you did this, well do it again. Right? That was the way to do it or do it this way next time. Right?

And that’s a great example. I do a lot of, by the way, Sharona, connecting my students to Constantine Stanovsky, the idea that there’s directors of root and directors of result, right? And I don’t care about result. I care about doing it the right way. And if you do it the right way, the result will come about. So again, my theater life really does connect to my teaching life. It’s all about process that will create product as opposed to product being all, because then you’re just, you’re working backwards from, as you guys were talking about the right answer and doing it your way as opposed to do it the right way and let it evolve.

Sharona: So I do want to look at the last problem pointed out in this article though, because I hadn’t thought about this exactly the way it’s posed. So the problem they state is we may not apply feedback equitably. And I guess I know this at some level, but I guess it hadn’t really hit me the exactly this way. They say that faculty often have to judge whether a student has sufficient understanding of the concepts when the evidence from the students is ambiguous. And we actually talk about this all the time in statistics. We actually argue and debate about whether or not we think a student in a given situation, especially a borderline, has shown sufficient understanding. So we do understand how hard this is, and it says in these situations, we give some students the benefit of the doubt. Assess other students as having a misunderstanding. So what do you think of the, either either of you, Boz, you’ve been in this more with us in the, in the norming calls, equitable feedback, equably applied feedback?

Boz: Well, everyone has some implicit biases, whether we’re aware of them or not, and we definitely see these when we do our norming calls, which, by the way, anyone that ever does any kind of common assessment, if you’re not getting together and calibrating on how you’re grading those, that data is pretty useless. So, I still can’t believe this is a process that we’ve actually had a huge group of math teachers willing to do this completely on their own. But yeah, we get together for hour, hour and a half, two hours and sit and argue about student grades weekly.

Sharona: We get together hour, hour and a half weekly during the semester, on a Friday night most of the time.

Joe Zeccola: Yeah. I remember designing my last year at Santee as part of my national board certification. I designed an EPT prep class for our students because I was scoring them and then I saw what it needed and I tried to help our kids game it out because it was, like you said, it was a scourge. It would not stop keeping them in. Yeah, I, I certainly think this is a problem. I think it’s, for me, I think it’s, it cuts both ways. And what I mean by that is the idea I. Because I would disagree with the language because they use it going by the problem Number four, title, we may not apply feedback equitably. And then the next sentence is, there’s evidence that faculty do not provide feedback and grades equally to our students. Well, I would argue that equal and equitable are not the same thing.

Boz: Absolutely.

Joe Zeccola: Right. So I definitely don’t apply fee to give feedback equally. I try to be equitable. And I think to Boz initial point, we have to think about how we, where our unconscious biases are, right? Where those are, where the implicit biases. That said, just to give you an example of something I did that was, I would argue quite equitable but not equal at all, is that the first essay I graded last night was from one of my most struggling students in my AP Lang class the first half of the year. And I was blown away by how coherent the argument was. It did everything I needed to do. The kid’s proficient and I marked them proficient in all the key areas and gave feedback in the areas they weren’t. But I was blown away. I didn’t expect it to be that good. The kid has certainly doubled down in terms of effort. You can see it.

So I did sort of track and then I got to one of my best students who wrote a higher scored paper. But because they did something that made no sense for their argument at the end, I dropped the score for one of their learning targets to make them redo it because I know the kid’s gonna redo it, he’s one of my best students. He’s also this kid who combines being a really good baseball player with a really good student. So he’s certainly my jam, but if anything, I’m giving him a harder score because what I’m saying is I wanna see you do this, and I guarantee you his parents be like, thank you, because that was a sloppy mistake. Fix it. But I think that would demoralize the other kid.

So I would argue not equal, very equitable. Right. This is a kid who knows he’s getting it, but I’m saying, no, no, this makes no sense. Go fix that. It’s gonna take you a half hour, 45 minutes. You know,

Sharona: I’m going to sort of challenge a lot of things that are sort of hidden in this statement. First of all, I don’t think you can great equitably, like in general, like it’s just, it’s not possible. These are human beings and we are human beings. Yep. Right. A grading is just fundamentally inequitable. And especially if we use some of the guskey stuff for, for rollup or it’s you personally as a human being, judging another human being.

Joe Zeccola: And embracing that judgment as opposed to rejecting it. Right. Exactly. In the name of some fake calculation score you made up. Right.

Sharona: Right. So there is no such thing as equitable. We do need to check our implicit biases at the door and be aware of those if we can, but if we’re just committed to each student and we have defined our learning outcomes, well. It’s just gonna kind of fall out and.

Boz: Well be reduced at least.

Sharona: Yeah, yeah. No, no. I mean, the grades are gonna fall to where they are. I’m not saying it’ll fall out equitably. I’m just, we need to strive for equitability, but we need to redefine what we mean by that. Because this whole, I mean, yeah, equitable. If a weaker, I guess one of the questions I have for you, Joe, is hopefully that student who was struggling in the beginning didn’t suddenly turn in a much stronger paper because the first thing I was gonna say is AI.

Joe Zeccola: I, I a, I didn’t say it, but I’ll say it. I, I checked the AI detector, I looked at the stuff. I’m interested in some of the phrasing, but they’re about to start timed rights in class, so I can align those. But what I did notice, because I’m behind in my grading, this essay was a month ago, although. It doesn’t freak me out because they’re, they’re at another revision right now, so I’m giving this back to them to use to fix their revision. So they’re, it’s going to time out perfectly. ’cause I also in the middle of a unit getting an old paper back and then they can’t do anything with it anyway. What’s the point? Right? So I want the action to be improve these aspects on the other one.

But the other thing that I noticed all spring long was there was more buy-in. Like this kid sort of connected to me more is tapping my fist more deliberately. Like there’s something more connected. Something in her said, I’m gonna try harder. So when I read it I was like, well, we’ll see. ’cause I wasn’t optimistic and then I was impressed because the stuff wasn’t there. And this seems more there. It wasn’t flawless at all. But yeah, that is what I was looking at. And my evidence is I saw the money getting put in the bank. I did not expect this result. ’cause oftentimes kids who struggle with this struggle for a little longer.

But what I was gonna say was, we do have our perception of where our kids are and we do try to do better than that. But just, I’ll give you one more example. The last essay I graded before I went to bed last night was a kid who is one of those, I call them stealth bombers, that they sit there quietly and then they just blow it up when they turn something in. It’s so strong and this kid will force their self to speak ’cause my class is discussion based, but not a lot. And I was really excited to read her essay.

It kind of stunk. It’s still like, and by stunk I mean it’s proficient but it’s barely proficient. This kid’s usually better. So again, my perception was based on what I know the kid can do. And then basically that’s what I said. I go, this is meh, but you’re doing what you need to do. So I think once you embrace the way we grade, we can reduce it a little bit. Because again, what we’re looking for is proficient and what the kid got, which might not be the happiest thing she got in the world, is she could hear me going, eh, you know, like sort of sad tromboning the feedback because I expected this thing to be great. It wasn’t great, but. I’ve trained myself enough with the kind of calibration we used to do at Santee and other places and just thinking about how I think about grading, it was proficient. So there’s no penalty. And I just think, I think we can reduce it. I, I agree with your, but I think your point is really the key, Sharona, that if we embrace the thing we’re supposed to embrace, which is use our judgment to get people where they want, like every coach does and every profession, like every teacher does everywhere else, there’s gonna be some bias that we can’t do anything about. We just have to really look at the implicit stuff. Am I holding something against certain types of kids? Does this kid’s personality trigger me in a way that makes me, you know, somehow hold them back and things like that.

Boz: Well, we’re coming close on time, but there is one point of the summary and the call to action of this article. And we’ve talked about a lot of these kind of throughout, but there’s one in particular that I wanna directly address before we end this episode and that is their second bullet point of their summary and call to action, give students an incentive to learn from the feedback through revisions or re-attempts. And I would argue that if you are using traditional grading, you are hog tying this, you are hindering this process. Because if my first mistake, my first attempt is still calculated in my final grade, is it if I’m still paying the price for those mistakes, even though I have learned and I have gotten better, you’re doing exactly the opposite of what you’re trying to do. That, and this is part of the reason why, if you’re doing a traditional class and you’re not doing any kind of revisions or re-attempts or something that allows the student to actually learn from the mistake, of course they’re not gonna use the feedback. You’ve gotta give them a reason to learn from that feedback.

Joe Zeccola: Oh boy. I mean, yeah, that’s the thing that to me was so great about reading this. I just feel like, obviously we’re picking stuff that we want to discuss, but it just shows you how the thinking is all pointing in the same direction. And what do we want them to do? Are we trying to teach them, are we trying to practice skill and knowledge acquisition, or are we trying to practice score keeping? Yep. Right. If we’re practicing skill and knowledge acquisition, the only way is to give them reat attempts. And why are we averaging it? You can make a list that we would, the three of us, whether it’s actors, athletes, whatever, skill people who, it took them a while to figure it out and then became great. No one talks about the Took ’em a while.

Boz: Exactly.

Joe Zeccola: Unless they’re talking about it to say, see, anyone can do this. But their value is never, ever impacted by the, took him a while.

Boz: Exactly. Now, how many people know how many times the attempt to make the light bulb versus the what’s the business quote?

Go ahead. It was,

Sharona: I think it was like 999 failures for one success or something on the, yeah, on the light bulb.

Boz: And nobody cares about the failures.

Sharona: Well, and the other one that always gets me is Michael Jordan, I don’t know if he’s still that, but at the time when we were growing up, he was the best basketball player of all time at the time.

And

Joe Zeccola: I’ll never agree to that. I’m a Knicks fan. Okay.

Sharona: Well, but the point was he, at the time, he was considered one of the greatest, and he missed 70% of the time when he was given that that three point shot, that final one, he had like a 30% success rate or something like that. And that was considered great. It was fabulous.

Joe Zeccola: Well I do that when I do, traditional grading workshop for my kids. The greatest hitter in the history of baseball by almost any metric would be Ted Williams. I think now, when that they’ve merged statistics, you would say Josh Gibson, but either of them, their score is at their peak, they batted 400, which means 40%. That means they failed 60% of the time. They’re a solid f in traditional grading, right? I mean, like, what, what do we, what do we, and that’s the greatest ever, ever.

Boz: Correct me if I’m wrong, pretty much anyone that’s over a 350 lifetime batting average is in the Hall of Fame.

Joe Zeccola: No. Anyone who’s over. Like probably 3, 3 10 or three 15. Oh, okay. So it seems this quotes right? Like yeah, because people like Joe DiMaggio, who’s one of the greatest hitters ever had a 3 25 lifetime average. So that’s a way higher bar than most Hall of Famers. And in today’s hall of fame class, you got people who 250, 260 ’cause it depends on what else they’re doing. So yeah, I mean it used to be, and, and it’s interesting because of technology and also the way we use advanced statistics, baseball isn’t as timeless as it used to be in terms of numbers equalizing. They have to look at other things. But yeah, it’s, it’s always been a game that is about just trying to minimize failure a bit because failure’s what you do, right? It it, yeah. The best players are failing 70% or more of the time.

Boz: All right. Well we, we are coming up on time, so Sharona, is there any last thing that you want to close on?

Sharona: Just, I wanna ask people to maybe write in if they feel like they’ve got a really good feedback activity. I’d love to showcase some, some, especially stuff that you do in class with students once you give feedback. I’d love to maybe bring those up on a future pod or link them in the show notes. I’d love to see what people out there are doing with feedback.

Boz: And love to possibly steal some of them to make our own practice better.

Sharona: Steal some of them, invite people on to be a guest interview. Something like that. How about you, Joe? Any last thoughts?

Joe Zeccola: Just how much this article drove me back to those two texts that Guskey co-wrote just reminds me there’s some good thinking, both grading with integrity and instructional feedback. There’s some good thinking around feedback out there. We also, I’ll do some looking to see if Guskey’s got any more pieces on it, but just again, it’s practical because it makes it clear that we only have so much time to do this. And that’s, why I think we all love Guskey so much. Is it, it’s all of his stuff is focused with an eye towards a teacher can only do so much at any minute, but how do we maximize the time that we spend doing this? And there’s some really good thinking around that, that again, I’m just reminded I haven’t dug into it enough. Yeah.

Boz: All right. Well, I wanna thank you for always, Joe, coming on. It’s always a thrill to have you on. I can’t believe that how quickly this hour’s gone by, so can’t wait to have you back and for everyone else, we’ll see you next time.

Sharona: Please share your thoughts and comments about this episode by commenting on this episode’s page on our website, http://www.thegradingpod.com. Or you can share with us publicly on Twitter, Facebook, or Instagram. If you would like to suggest a future topic for the show or would like to be considered as a potential guest for the show, please use the Contact us form on our website. The Grading podcast is created and produced by Robert Bosley and Sharona Krinsky. The full transcript of this episode is available on our website.

Boz: The views expressed here are those of the host and our guest. These views are not necessarily endorsed by the Cal State System or by the Los Angeles Unified School District.