60 – The Role of Depth of Knowledge (DOK) in Aligning Assessments to Learning Outcomes: An Interview with Erik Francis

Today on the podcast Sharona and Bosley sit down with Erik Francis. An international author, educator, presenter and professional development provider, Erik has over 30 years of experience in education. He has written a number of books and blogs on Depth of Knowledge, Inquirying Questions, and other topics. In this conversation, we take a deeper look at Depth of Knowledge and how it might be used to allow us to better align our assessments with our Learning Outcomes.

Links

Please note – any books linked here are likely Amazon Associates links. Clicking on them and purchasing through them helps support the show. Thanks for your support!

Resources

The Grading Conference – an annual, online conference exploring Alternative Grading in Higher Education & K-12.

Some great resources to educate yourself about Alternative Grading:

Recommended Books on Alternative Grading (Please note – any books linked here are likely Amazon Associates links. Purchasing through them helps support the show. Thanks for your support!):

The Grading Podcast publishes every week on Tuesday at 4 AM Pacific time, so be sure to subscribe and get notified of each new episode. You can follow us on Twitter, Facebook and Instagram – @thegradingpod. To leave us a comment, please go to our website: http://www.thegradingpod.com and leave a comment on this episode’s page.

If you would like to be considered to be a guest on this show, please reach out using the Contact Us form on our website, www.thegradingpod.com.

All content of this podcast and website are solely the opinions of the hosts and guests and do not necessarily represent the views of California State University Los Angeles or the Los Angeles Unified School District.

Music

Country Rock performed by Lite Saturation

Country Rock by Lite Saturation is licensed under a Attribution-NonCommercial-NoDerivatives 4.0 International License.

Transcript

Erik Francis: What is the intent and purpose of standards? Because what I like to say is that we’re teaching on an 18th century schedule in 19th century classrooms with a 20th century curriculum. And we still have a no child left behind mindset where we’re focusing on skills and stuff. But what exactly is the intent and purpose of a standard these days? And that’s really what I’m focusing on lately, especially through my work with DOK and questioning.

Boz: And throw on top of that the grading front with a hundred year old system.

Erik Francis: Oh yeah.

Boz: Welcome to the grading podcast, where we’ll take a critical lens to the methods of assessing students learning from traditional grading to alternative methods of grading. We’ll look at how grades impact our classrooms and our students success. I’m Robert Bosley, a high school math teacher, instructional coach, intervention specialist, and instructional designer in the Los Angeles Unified School District and with Cal State LA.

Sharona: And I’m Sharona Krinsky, a math instructor at Cal State Los Angeles. Faculty coach and instructional designer, whether you work in higher ed or K 12, whatever your discipline is, whether you are a teacher, a coach, or an administrator, this podcast is for you. Each week, you will get the practical, detailed information you need to be able to actually implement effective grading practices in your class and at your institution.

Boz: Hello and welcome back to the podcast. I’m Robert Bosley, one of your two co-host. And with me, as always, Sharona Krinsky. How are you doing today, Sharona?

Sharona: So I always love that you ask me how I’m doing so that I get to answer first how I’m doing, but I’m doing really well today. We got to do some absolutely fantastic professional development work with a school in Michigan about authentic assessment in the age of AI. And given that the entire school has gone to standards based grading, it just was such a nice expansion of what we normally get to do. We’re not having to persuade people that they should do standards based grading or alternative grading. We’re not having to start at the beginning of what’s wrong with traditional. So I had a good time. How about you?

Boz: You know, I really enjoyed that too and what I thought was kind of fun and refreshing is that’s the first time we’ve ever been with a group that at least, with zoom technology was as proficient, if not, almost all of them, more proficient than we were. So, it was a unique experience working with someone that was that good at working online.

Sharona: Exactly. I mean, because not only are they standards based grading, but it’s a virtual school. So they are really super used to and very fluent with zoom technology. So that was, that was a lot of fun. You know, having done that today, I’m very excited about the guests that we have today. We have a little bit of a different guest and this person who I’m about to introduce actually probably is a lot more on the assessment and /or teaching side of things then specifically grading reform. So I’d like to welcome Erik Francis, an international author, educator, presenter and professional development provider. He has over 30 years of experience, which I may have beat you if you only have 30. I’ll just make that note. And he’s the author of the incredible new book And Inquiring minds want to learn, posing good questions to promote student inquiry. Also has a couple of other books. Now that’s a good question. How to promote cognitive rigor through classroom questioning. And the book that really drew us to you, Erik, which is Deconstructing Depth of Knowledge, a Method and Model for Deeper Teaching and Learning. And he’s the owner of MavErikk Education, which provides professional development guidance and support on planning and providing teaching and learning experiences that are standards based, socially and emotionally supportive and student responsive. So welcome Erik to the pod.

Erik Francis: Thank you. Thank you guys for having me. And thank you guys for allowing me to be a part of this and speak on this forum.

Boz: So one of the things that we always like to have our new guests kind of talk to our audience about is their origin story. Just how did you get into grading reform and in your case, just general educational reform as a whole.

oming an educator back in the:

So a lot of what I was doing was working a lot with standards and understanding them as goals and expectations, and even requirements that we had to have them on the board. But I also started using them to justify and validate why I was teaching, how I was teaching, what I was teaching. So I want to be creative as a teacher and I remember my administrator always said to me, if it can address and assess the standard, you can do it. And I would do it. So I’ve always had this very, very strong proclivity and interest in standards based, not just only standards based grading, but standards based learning.

As it’s kind of evolved with going from no child left behind to going to common core. Now we’re calling it next generation learning standards. One of the things that’s fascinated me and one of the things I want to actually write about in a new book is what exactly is the intent and purpose of a standard. And if you look at the history of it, it’s actually quite fascinating, because if you read Diane Ravitch’s stuff, she’ll explain how standards, not only people realize this, is that standards were actually first created for access and equity, and that’s kind of surprising if people think about that because they don’t really associate that word with standards, but the equity and the access was that everybody who regardless of their location or the socioeconomic status had the right to a high quality education.

And they use those standards to really set more of a criterion for enrollment and acceptance into post secondary education or what we would call college back then. Then they became indexes of efficiency and ensuring alignment with the standardization of the system. And then what’s interesting is that during the fifties, they kind of went away and then it became politicized in the eighties. So that’s kind of what’s been my fascination with standards now. It’s like, what? Is the intent and purpose of standards and what have they become and how can we basically not only assess student learning, but address and assess learning loss using the standards so that every child can rise to reach and go beyond those goals and expectations set by the standard. So that’s kind of where I am now with it and that’s kind of my history with it. With teaching, it was always about ways to basically use my curriculum instruction to validate what I wanted to teach and how I wanted to teach it and why I wanted to teach it. But now it’s more so looking at this. Okay. So what really is the intent and purpose of a standard these days?

Boz: So you said something interesting and it jumped out to me because I was just talking about this exact point in that training that Sharona was talking about earlier, but , in what you were describing with the history of standards. You said they kind of went away in the 50s. So when were they here before? Like, how far back does this go?

base movement starting in the:at it actually started in the:has been in effect since the:

Well, one of those governors was from Arkansas, Bill Clinton, and when he came in, he said, we’re going to have national standards and the states all said, no, 10th amendment of the constitution you can’t force us to do that. So he said, fine, you all go create your own standards and assessments. And then they, the states all said, no 10th amendment constitution, not all power, not delegated to the federal government will go to the states. So that’s how it got tied to actually the law, the federal law of the Elementary and Secondary Education Act to say, fine, if you want title money, you every state needs to create standards and assessments. That show the kids are achieving these goals, and that’s when back in the day, as we remember with our experience, those standards assessments were like nice to know. Like, I remember that it was schools where I worked like, Yeah, you want to come in for testing day? That’s fine. But then when no child left behind, it became need to do, and then it became not so much about even assessment evaluation. It became accountability and enforcement, and that’s where the punitive part of it was.

And then, you know, Common Core, it was more so the bargain to say, well, do you want to be absolved of these requirements for No Child Left Behind, and hey, we’ll give you some money if you adopt these standards. So 46 states said, ooh, money, and didn’t realize they had to be joined in this testing consortium. And, so that’s where we’re kind of at right now, I think. It’s like, What is the intent and purpose of standards? Because what I like to say is that we’re teaching on an 18th century schedule in 19th century classrooms with a 20th century curriculum, and we still have a no child left behind mindset where we’re focusing on skills and stuff. But what exactly is the intent and purpose of a standard these days? And that’s really what I’m focusing on lately, especially through my work with DOK and questioning.

Boz: And throw on top of that the, the grading front with a hundred year old system. Oh, yeah.

Sharona: Well, there’s something I want to add there too, that you said that I want to put some nuance on. You that in Horace Mann’s time, it was about equity and access, which I agree with for, white male students.

Erik Francis: Absolutely.

Sharona: So, if you look at the history of the math education to which dovetails very much with the Horace Mann stuff it was because white men did not have equal access depending on location and access to quality educators. So now when we get to a nation at risk, we’ve thrown in gender and race and socioeconomic class and all of these other things, which is one of the reasons it becomes so politicized is because there are states for whom they’re all good and fine with equity and access for white men. We’re not so thrilled about it for women and minorities. So just wanted to add that layer.

Erik Francis: No, I think it’s great you did because you’re totally right. Because when you say that standards were originally designed for access and equity, people cocked their eyebrows at you. It’s kind of like the same thing. What did it mean being by all men are created equal? Who are the men we’re talking about? You know, not to get really, you know, controversial by it than that. But the thing is, this is that how has that changed and evolved? You know, when we talk about access and equity, like you said, white men doesn’t matter social economic status or location, didn’t matter if you’re in an urban center or you are working, you’re living in the territories, you still had access to that. But again, one gender, one race.

Now, if that’s still what standards are and the purpose of standards, is it living to that promise? Is that fulfilling to that promise? And what can we do to make sure that not only it’s accessible to where everyone has the opportunity to basically teach and learn that standard, but it’s also accessible where the kids can basically access that standard in their own unique way. That’s where the equity comes in. So how can I access that in my own unique way? How can I achieve that standard in my own unique way in terms of equity? Equality is everyone gets the same standard, but equity is, now how can I access that? And that’s sometimes I think the hard part where it comes to education because equity I believe, and this is my two cents, is tied to differentiation, individualization, and personalization. And and there’s so many different things when it comes to equity out there where they say this is equity. No, actually, that’s equality. I mean, there’s there’s actually even a poster if I can say that was misinterpreted where I don’t know if you ever seen the one where everybody gets a bike. And that basically, everyone gets a bike, that’s equality. Everyone gets a bike to fit them, that’s equity.

ights for the Disabled in the:

So when you look at that even for assessment, and say, okay, so the kids can, has this standard, and they may not be able to access it in the way that the assessment has, but why is it that if that student can do that using some other different resource, using some other different tool and they can achieve that standard, now that can’t be used as evidence? See, that, that’s the thing I always get confused about. It’s, it’s like, if I gave you three books of Green Eggs and Ham, The Outsiders, and Pride and Prejudice, and the standard said determine a theme of story from details in a text and explain how to detail support the theme, if the student can access that text and relate to that text because that’s how they understand it and they can do it. Does that mean that they can’t do it because they didn’t know how to do it with the other two texts? That’s the thing I always ask when it comes to assessment.

Sharona: I think we could spend an entire podcast on this.

Erik Francis: I know.

Sharona: Because I completely agree with everything you just said. But I think this is actually a decent entry point to one of the challenges we keep coming up against and butting up against that we wanted to bring you on for. Which is overlaid with these standards, right? So we have these standards. We talk a lot in this podcast about using standards to create learning outcomes for the course that are based on context and cultural relevancy and all those things. So we use the standards as a stepping point. But. Overlaid with these and coming from sort of the same father of what we would call the alternative grading movement is Bloom’s taxonomy, which is one thing. And then there’s this thing called depth of knowledge. And I say this thing because other than listening to Bosley occasionally say D O K, dah, dah, dah, dah, dah. I actually know little to nothing about depth of knowledge. And I guarantee that most of my higher education faculty don’t know, unless they have a crossover, they don’t know anything about it. We sort of know Bloom’s taxonomy. So can you give the Cliff Notes version of what DOK is, how it came about and plays in? And, what happens when it crashes headlong into some form of alternative grading that uses proficiency scales?

Erik Francis: So the interesting thing about DOK is that it is not what it was originally tended to be. And what I mean by that is that it’s evolved. And unfortunately there’s been a lot of misinterpretations and there’s also some miscommunication about what exactly depth of knowledge is. It’s not another way to do Bloom’s taxonomy. In fact, it is not even a taxonomy in terms of, we look at taxonomies that not only categorize, but also scale. And Norman Webb, who created Depth of Knowledge, he’s the creator of it. He actually created it as a criterion for alignment studies. Where you would take your standards and you would code them a 1, 2, 3, or 4 based upon its cognitive demand. And then you would take your assessment items and code them as a 1, 2, 3, and 4. And he would do holistic assessment alignment studies where if you had 6 items and 4 out of the 6 items addressed the standard at the DOK level demanded, then it’s fully aligned. If two or three out of the six did it, it’s acceptably aligned. If one or none of it, it’s, the standard is insufficiently aligned.

So that’s what originally what it was. What depth of knowledge has become, it’s a measure of cognitive rigor. And that’s what Karin Hess did with it. She superimposed Bloom’s taxonomy and the depth levels of depth of knowledge. So you see the cognitive action verb that introduces a standard and you look to see where it is in Bloom’s and that’s the level of thinking students must demonstrate. Depth of knowledge looks beyond the verb. So you have to ask yourself, if the standard says analyze, well what exactly do they have to analyze and how deeply do they have to analyze it? Or if it says understand, what exactly do they have to understand and how deeply do they have to understand it? That’s described in detail by all the words and phrases that follow the first verb of an objective of a standard or a learning intention or objective or target.

What I did with it is I made it into a method and model for teaching and learning using and drawing upon both Norman Webb and Karin Hess’s work. So, when you determine the DOK level of the standard, that’s the ceiling of assessment, that’s what Karin Hess called it, so what that means is that when I know the DOK level of the standard, that tells me that’s the deepest level the kids could and should be assessed on an on demand standardized assessment. So if the standard is a DOK 2, then that’s the deepest level the items on the assessment will assess the kids. They won’t assess beyond it unless it’s math. Because math, the way they get away with it, is that they have the standards of mathematical practices and those are all deeper DOK levels. And it only won’t be assessed at it, because that makes it, that makes it a test. It’s too difficult. Assessment is about the range, so when you determine the DOK level of the standard, you’re saying that this is the deepest level the kids could and should be assessed, and this is the range of DOK levels in which the kids could and should be assessed.

r Francis, Mr Francis, I know:

Can you explain how you attained it? A DOK 3 is either you’re justifying or verifying. That’s where you’re using evidence. In math, that’s proofing. In English Language Arts, that’s if this is the theme of the story, then how does the information in the text strengthen and support that theme. Or you’re considering and critiquing, which is now you’re asking hypothetical questions like, what if? Or you’re basically asking argumentative to say, here’s two sides of the argument. Which one is the accurate or appropriate and why?

At a DOK 4, now you’re exploring and extending. So that’s where you’re going to go deep within a subject area. That’s when you’re going to go across the curriculum. You’re going to go through multiple texts or connect multiple topics or go beyond the classroom. I’ll say this right now for the world of assessment, an on demand standardized assessment do not have any DOK 4 questions. Doesn’t matter if people say this is DOK 4, a lot of times they’re doing that to say, this is why my assessment’s good. Okay? You’re not going to be able to capture that in one segment of time. A DOK 4 can take hours. A DOK 4 can take days, weeks, months, a semester, a year. So, that’s from that level of assessment, and you can’t capture that on an on demand standardized assessment unless you’re doing a writing assessment where you synthesize multiple texts. That’s the closest you can get to a DOK 4.

Boz: That’s really interesting. One of the things beyond my work with grading I was department chair at my school when we adopted Common Core and when we started , the SBAC and the CAASPP testing and trying to make these transition and really going from Bloom’s to DOK, you know, this DOK thing was something new to us back then. One of the single biggest misconceptions that I still see, we’ve been doing common core now for almost a decade, but this idea that D O K means hard, it’s something, you know, if I’m solving this multi step four term polynomial, that’s hard, so that’s a D O K three. No, the D O K does not mean how difficult something is. Like, like you were saying, just solving an equation, just going through the process of solving it, even if it’s a hard equation, does it make it D O K three or four? It’s actually still D O K one.

Erik Francis: And this is the simplest way I say it, because you have to simplify it. If I can find it on Google or I can do it on my calculator, it’s most likely a D O K one. So if I said what is 542 times 398? You can’t just say that off the top of your head. You have to work that out. Or I can just use my calculator. I can tell you it’s 215, 716. That’s still a DOK 1 it’s hard. The complexity is how can you use multiplication to find the product of multi digit numbers? Now I got to go explain how I did that. So you might say stacking factors, partial products, add them, get that I’m going to say in my old age, what I’m starting to like his place value understanding where I take 398 split it up into its place value. Do 542 times 300 plus 542 times 90 plus 542 times eight drop my zeros to make a triple digit times a single digit.

And then bring them back to get my partial product. Sharona might say, I like the lattice method. Or someone else might say, I like the stick method out of Asia. Or someone might say, I like using arrays. That’s DOK 2. Okay, it’s not about, and that’s the thing, is that it’s not the more difficult it is. Because difficulty is subjective. What you may think is hard, I may think is easy. But complexity and demand, What exactly and how deeply do I have to comprehend and communicate that? That’s where DOK comes in.

And sometimes it’s hard for our on demand standardized assessments to capture that because it involves a lot of communication. When you see the items that say, you know Erik solved the problem this way. Explain how he did this. Okay, that could be a 3 because I’m explaining or justifying that. Or, Erik solved it this way and he got it wrong. Where did he go wrong? That’s a DOK three. Okay. If I am just plugging in formulas, if I’m just using algorithms, okay, that is a DOK one.

If I can explain how I did it, that’s a DOK two, a DOK three could be. Okay. I need to find out which three numbers or which three values add up to form a right triangle. Get me a right triangle. So someone might say 3, 4, 5. I got a right triangle. Why? Because 3 squared plus 4 squared plus 5 squared equals 16 plus 9 equals 25. I know it’s right. Okay, so what about 4, 5, 6? Well, no. Why? Because of this. Okay, so now I know the the legs are longer than the hypotenuse. So what kind of triangle do I have? See what I mean? I’m doing that there. So that’s the whole thing about it. It’s that it’s not the way you’re teaching because that’s another misnomer. How are the students comprehending and communicating their learning? Are they just answering what it is? That’s a level one. Are they explaining how they did it? That’s a level two. Can they justify their responses, results, or reasoning, or someone else’s? That’s a level three. Can they explore and extend it? So in math, STEM is a DOK 4.

One of the simplest ways I do this with teachers, I say, have your students go out in the real world, And bring it back to the class and teach the class how this math can be used in the real world. That’s a D O K 4, okay? English Language Arts, when you’re doing author studies, genre studies, when you’re comparing informational text and literary fiction about the same topic. I used to have my kids, they would we read about Abraham Lincoln in a textbook. And then I would say, okay, now I want you to, you have a choice. You can watch Spielberg’s Lincoln, you can read Gore Vidal’s Lincoln, you can read a children’s book about Lincoln, and you can read Abraham Lincoln Vampire Hunter. How does that address Abraham Lincoln? How does that present Abraham Lincoln? That’s a DOK 4.

In history, when you look at history in the past, present, and future, right now we’re living in a DOK 4 when it comes to this election, because we’re asking, has this ever happened before in the past? That’s a DOK 4. What’s really interesting is that the enrichment courses of art and music and P. E. and C. T. E. Those are the DOK 4’s is for the core content classes because they’re going to say to them, Hey, this is how you can use geometry to create perspective. This is how you can use sound in science and lyrics in literature to create music. This is how I can use angles to figure out how I can basically get that into the soccer goal without it hitting the goalie, whether it be out the goalie getting to it. So that’s where we really need to tap into that. But those enrichment courses are also looking at them to say, how can we turn students from students of the work to teachers of the work to disciplinarians of the work to having expertise? And that’s why I even say in all the content areas.

Sharona: Okay, so much here. Two places I want to go. One I want to go first, but I do want to get to the other one. Yeah, where I want to go first is, I think you’re articulating something that’s really bothering me about something I’m currently working on. And that is something you said before, which is the issue of using DOK levels as proficiency scales. So I’m going to read you an example of something that I’m literally, I could not put my finger on why this bothered me, but I’m going to read this to you. Okay. Okay. So what this says, this is a English language arts standard. So the standard says, analyze how particular lines of dialogue or incidents in a story or drama, propel the action, reveal aspects of a character or provoke a decision. So what level, what DOK level is that standard? Would you say? Okay. I can read it to you again.

Erik Francis: Yeah.

Sharona: Analyze how particular lines of dialogue or incidents in a story or drama propel the action, reveal aspects of a character, or provoke a decision.

Erik Francis: Okay. You got a lot of stuff going on there.

Sharona: Right. But the reason I’m going to ask you what level it is, is because then I’m going to read you their proficiency scale.

Erik Francis: Okay.

Sharona: So that’s where we’re going with this.

Erik Francis: All right.

Sharona: Best guess.

Erik Francis: Best guess. Okay. I can tell you why. I could tell you. It’s a DOK three.

Sharona: Okay? Okay.

Erik Francis: But the whole thing is not A DOK, the individually, excuse me. The whole thing is a DOK three. Why? Because I have to deconstruct that standard into its individual objectives, and you have three separate objectives in there. Actually, you got more because here’s the thing I’m gonna ask you. Remember, it says analyze right? So what comes after analyzing that in that standard?

Sharona: How particular lines of dialogue or incidents in a story or drama do dot, dot, dot, dot, dot.

Erik Francis: Okay. So now I’m going to ask you this. Am I assessing, are they analyzing a story or are they analyzing a drama? Cause I got two separate things going on there. Okay. So if I’m looking at story, I may be working on that. first quarter by doing short stories. I may be working on that second quarter by doing novel studies. Okay. Third quarter is where I may get to my plays, my drama, where in middle school I might be reading Ann Frank and in high school I may be reading Shakespeare or in elementary school the Monsters on Mulberry street. I think that’s the name of that story where you do that. Okay. So that standard, I’m gonna be teaching that different times in different contexts. Okay, now it says, propel the action. What does that look like, sound like, and feel like? And that’s where you use the language of DOK because if I’m analyzing how the particular lines of dialogue and analyze how the incidents in there, which are two other things propel the action.

Now I might be looking at examples that I have to explain. That’s a D. O. K. 2. Now the other one says propel the action and the other one. What is it? Wasn’t it.

Sharona: Reveal aspects of character or provoke a decision, but honestly where I need to go Let’s not even go that far because I think we already have enough right? I want to read to you the proficiency scale Okay, because I’m struggling with the proficiency scale and I think we’ve already identified One of the main issues is the the standard but this is a four level scale I’m gonna start from the bottom of not enough evidence So, not enough evidence. The assess criteria says student has not yet provided enough information to determine what the student can do in relation to the grade level standard. Okay. Now, the next level up, needs support, which is still not considered good enough for the grade, says student can recognize the effect of dialogue in a text. Okay. So then the next level up says partial understanding student can analyze the effect of dialogue in a text. So we went from recognize to analyze in the same proficiency scale. And then mastered is student can analyze the effect of dialogue in a text and make connections. So is this what you’ve sort of seen this, this butting heads of a proficiency scale And standards or like,

Erik Francis: yeah,

Sharona: what’s going on here?

Erik Francis: Well, the first thing is this, and this is where the Standards Based Grading people sometimes look at me, like, don’t ask that question, Erik, because if you say, Oh, we’re doing standard based grading, I go great. Which part of that standard are you grading? Okay. Because if I can basically analyze how the lines of dialogue in a story propel the action, that’s one objective. See. And I have to basically do that as part of that objective. Can they do that? Okay. Then I have to say, can they do that where it reveals the aspect of drama? So you’re actually, because this is why it means that the standards are so deep. You’re not, which part of the standard are you hitting? That’s what I always ask when it comes to assessment. Which part of the standard are you addressing and assessing? And the other thing is, how closely did that item address and assess the standard? That’s the hard part I have with it. So I would look at

Sharona: What about

Erik Francis: Go ahead.

Sharona: Even if we simplify, because I didn’t realize quite how off this was and how complicated. So let me, I’m trying to get at one thing that I’m trying to understand right now. Let’s say it wasn’t quite as complicated. It just said how particular lines of dialogue in a story propel the action. Let’s say it was just that.

Erik Francis: Okay. Can you, can you, can they do that? Then that’s proficiency.

Sharona: Well, but they’re saying recognize is support, analyze is partial and analyze and make connections is mastered. Is that okay for a proficiency scale?

Erik Francis: The thing I would look at it to say is are you getting away from the standard?

What does the standard say? They can analyze how the lines of dialogue in a story, let’s just even simplify it further, propel the action. That there is proficiency. Okay? Now, can they explain the lines of dialogue in a story? Can they explain how the action is propelled in the standard? That would be approaching.

See what I mean? Can they read the lines of dialogue? Can they understand or can they summarize how the action is propelled? Okay? So, you gotta build it up. And with no evidence, I used to use no evidence to say, You didn’t do it. You know, I mean, that that was my thing. And this is actually why I tell schools. I say this, I say, what if instead of giving that zero and a lot of this is the, you know, amelioration that Rick Wormeli talks about. And I know it’s hard.

I was, you know, I lived and died by my zeros too, because that’s the way we’ve been taught and told, you know, but what if they just said no evidence or gave an I and said to the kids, okay, okay. Here’s the thing. We’re going into the grading period here, where I’m going to give you a summative grade. I got a lot of no evidence. It’s better that you choose the number than I do. You see what I’m saying? So, it’s not that they’re, it’s not that they can’t do it, it’s that they provided me no evidence that they could do it. And what you’re doing is you’re keeping it open to them to say, Well, until I get some evidence, then basically you know, I can’t, you know, I can’t give you, I can’t say the proof now.

Some teachers say, Well, what if I get all this late working and did it? Okay, well, then you have to shape it to say that. Well, what can I give them to say that they can do it? You know, because that’s all you I used to do that with my students all the time where I used to sit like every at the end of the end of semester. I didn’t do anything. Everything in your class this semester. What can I do for extra credit? And I used to laugh at them and say nothing. I said, Why are you giving me extra work, because you didn’t do it? So they would say, blank you with a colorful word, and I’d say, wait, there’s a way. And they’d say, what? And I’d say, if you can show me that you can do this, and you can show me and explain to me how it’s done, and do it at the level demanded by the standard, I’ll give you, and this is where I actually, I’ll admit to you I made a mistake, because I should have given them full credit.

I’ll give you a 59. 5. For the quarter for the semester all that work for 59. 5. Why? Because it’s better than a zero. Okay. Now, what do I got it now? Here’s what about what would I do? They get the 59. 5 They got to get it to me before the final or they ain’t got to get to me before the exam Okay, now they’re going into the exam not with a zero They’re going with a 59. 5 that you know I got a round up because I can’t have decimals in my gradebook So now it’s a 60 but because it’s so fresh in their head Now they’re getting on the final 100, 95, 90, 85, 80. What they get for the semester usually rounds out to like a B or a C.

So I used to have teachers get mad at me like, you know, you let them get away with that. I go with what? An education? You know, and, and, and I used to talk to this cause I was a dean of students when it came to attendance and I would convince kids to go back to class. And the teacher would say, you know what, you missed so much work I don’t know how you’re going to pass this class. And you know, kids are not going to go, well, I’ll prove you wrong. They’re going to fly. Yeah. So what I used to do is I used to say that basically, you know, I made this comparison, think about this way. What do you do when you miss a bill?

Sharona: You pay a late fee.

Erik Francis: You pay a late fee. Well, you don’t automatically, because first you call up, and you say, Oh my gosh, I’m sorry, I forgot. Is there any way I can pay this bill without the late fee? And usually you sometimes have a very sensitive person on there who will say, Yeah, we’ll waive the late fee this time. And if they don’t waive it, what do you say? Where’s your supervisor? And then you get the supervisor who has some flexibility, or they say no. And then you have to pay the late fee, replace that with work and grades, you know, replace that to say, Well, where’s your principle?

My thing was always about it’s not about the grade. It’s about the learning and we make mistakes and things happen and we don’t know why they happen and we don’t know why the kids are responding the way they do. So what can we do where we can talk about alternative grading that we grade so everyone can succeed to the best of their ability or their capability? What can we do to say, I used to say to my kids all the time, I say, is that your answer? Yeah. Is that your final answer? Yeah. And back in the day, letter grades. Is that your A answer? I think it is. Well, guess what I think? Well, you don’t think it’s an A? I said, well, do you want an A? Yeah. Okay. Do this. Cause right now you’re at a C, but if you want an A, this is what you can do. It’s up to you. So you see, I’m putting that back on a student and they’re choosing their own grades.

So, so when it comes to that, when it comes to that, you know, I know I kind of meandered off and this is the great thing about conversations like this because it can go everywhere. But the standard should be, this is what it means to demonstrate proficiency. That’s it. Doesn’t matter what the DOK level is, if the kids are hitting that and they’re hitting that with consistency. You need to decide. Are you going to say that they’re proficient because they’re consistent? Or are you going to say that it’s mastery because they’re consistent? Because that’s the whole other thing. I, my thought about mastery is you’re a master, which doesn’t mean that you can do this perfectly. It means that you can do this extensively, okay? So I always like to say going beyond the standard is mastery. But if I’m hitting that and I’m hitting that mark and I’m hitting that consistency, that’s what you should get.

And when I talk to people about this and they say, well, the standard is only a DOK one. Right. If that’s what it means to be proficient, then you give them a proficiency score because they earned it.

Sharona: So much to think about.

Erik Francis: Well, hopefully that’s why we’re gonna continue these conversations afterwards. Exactly. Rome was not built in a day and neither will your understanding of DOK. Because you have a lot of things you’re dealing with here. You’re dealing with 60 years of Bloom’s Taxonomy, and there’s nothing wrong with Bloom’s Taxonomy, okay? And it’s really funny because I have a great relationship with Tom Guskey. And at one point, I think Tom Guskey was feeling, you know, we talked about this that I was saying, don’t use Bloom’s, use D O K. And if that’s what I was saying, that’s the misinformation. We shouldn’t use blooms, we should use D O K. No, we should use both, which is what Karin Hess did with it. Because now you know what the thinking is, but what does that look like, sound like, and feel like? What does it look like to analyze? What does it sound like to understand? Yeah. Because if I said to you, analyze whether the information is in the text, you’re going to say, or nope, that’s a level one, but they’re analyzing. You’re right. But why? Because it’s higher level thinking. You’re right. But why? Because that’s where it’s categorized in Bloom’s. You’re right. But why? I don’t know. I don’t know the guy. Okay. But now you’re saying, analyze how the information is presented. Now I have to explain. Analyze how the information supports the central idea and theme.

Now I have to justify. Analyze how two or more texts. Now I have to explore and extend. That’s what I mean by what’s the language you’re using.

Sharona: Well, and I’m going to put one additional fillip in there. The language you’re using is discipline specific. Because if I say evaluate this function at a point that is a D O K one. If you say evaluate the student’s work on this text, that’s like a D. O. K. 3 or 4 because you’re passing judgment, getting evidence. So the word evaluate is not even enough. The word analyze is not even enough.

Erik Francis: In fact, they even say evaluate expressions is considered to be a D. O. K. 2. But it said evaluate. Even when I say explain, okay? Just because it says explain, that doesn’t mean it’s automatically a D. O. K. 2. If I said explain that, that’s a D. O. K. 1. Explain how? That’s a DOK2. Explain why, or explain what if. Now I’m in a DOK3. Explain what, but you see how I’m looking beyond the verb? There’s a simple way to do this. And it does not throw the baby out with the bathwater. And it’s not transformative. It’s extensive. We unpack and unwrap the standards generally by what? Circling the verbs and underlining the nouns. And then we make a T chart. And that T chart, we list all the Bloom’s Verbs underneath skills, and all the key concepts, or the nouns, or noun phrases, that name the stuff.

Does anyone know why we do that?

Boz: You know, I’ve done, I have done extensive PDs of doing that, of doing that exact thing. Going through, circling the nouns and the verbs and then matching it to the word wheels, the Bloom’s wheel or the DOK wheel and go, Oh, it says this so it’s this part.

Sharona: And? Why?

Boz: Why? I can’t tell you because, because we were unpacking the standard. That’s why we were doing it.

Erik Francis: Do you know, what if I told you the why is this. That’s how Larry Ainsworth taught us with unwrapping standards in his first book. The book that he came out with with Doug Rees. Still a great way to do it, but that worked with No Child Left Behind standards, when all the kids had to do was demonstrate they could use the skills and they knew the stuff. So that’s why we circle verbs and underline nouns. One of the things that we were misinformed about these new standards that started with the common core, that’s continuing with these next generation science standards is that they’re not cognitive or content based, they’re contextual.

So when I teach about deconstructing stairs for depth of knowledge, I want them to take a red pen and circle the verbs. And then I want you to take a blue pen and I want you to underline the nouns. And I don’t even say verbs because I see a lot of teachers go like, Oh, that’s a verb. I got to circle it. Oh, that’s a verb. Well, the kid may not be doing that action, but it’s a verb. I got to circle it. Right? So like how the author responded. Oh, responded. That’s a verb. You know, no, it’s the author responding, not the student. Okay, there’s a reason why I use red and blue because that’s the color of an archery target. Okay? If you look at the circles of an archery target, it’s red, black and blue. The black is the print of the standard. The red is the verb. So you hit that target. The blue is the noun. So you hit that target. So I’m hitting the cognition. I’m hitting the content, but the bullseye is a yellow and that’s why I say you need to take a highlighter and highlight everything that comes after that verb and don’t stop until you get to a period or a semicolon that’s followed by another cognitive action verb because everything in that yellow, that’s the depth of knowledge demanded.

It’s not, I’m analyzing, great, you’re demonstrating higher levels of thinking according to Bloom’s, according to Solo, according to Marzano, according to Fink’s, fantastic. But what exactly are you doing? I’m analyzing whether the information is in the text. That’s a DOK 1. Why? Because all I gotta do is answer. Analyze how the information, this is all in the yellow. Is in presented in a text. Now I got to explain that analyze how the information that’s why I got to teach supports the central idea. Oh, I got to teach that and theme teach that of a text. That’s all in the yellow. What exactly and how deeply so when you guys work with people with standards. Ask them this, what exactly and how deeply do they have to do that?

Well, first thing I do, no, no, no, no, no, no. Look at that standard. What exactly and how deeply? We’re going to deconstruct it later to get there. But what exactly do they have to deeply? So, I like to compare the DOK levels to a race. And each DOK level is the finish line. So a DOK1 is a dash. Get to the finish line. A DOK 4 is a triathlon. Why? Because you’re using multiple skills. two and three are an obstacle course that different complexity. So I show a backyard blow up obstacle course and I show a AmErikan ninja course and I have them as blocks, but then I turn them on their side to show the same finish line. And I say, what is this image represent? And I actually asked the PE teachers, what is this saying to you? And he goes, Well, it’s not saying a triathlon is better than a dash. It’s just a different type of race. I go, and that’s the DOK levels. Do I have to get to a dash? Yeah, that’s a DOK 1. Are they running a triathlon? Which means that I may be doing STEM. I may be reading multiple texts. I may be talking about history’s impact on art and literature. That’s a DOK 4. The DOK 3 is Am I running an obstacle course in my backyard where I just got to get to the finish line, or am I running an AmErikan Ninja where I got to figure out, okay. I can’t get to that one part, but what part can I do and how do I swing to that part and use that to get there? So that’s what a way I look at with DOK.

Boz: And your point to all this, if I can kind of summarize it is if the student is supposed to be running a dash and they run the dash and finish the dash. That should be the level that we’re trying to get them to that. That should be what we’re expecting. It shouldn’t be, okay, we’re, we’re expecting you to do a dash, but now we’re going to try to do these proficiency scales to where we’re going to throw obstacles into the dash for no reason, because we need to get the higher level of DOK, you know that they can’t have a DOK 1 be my proficient. I’ve got to get that up to DOK three to get to efficiency, which is exactly what I see in a lot of designing of proficiency scales, especially when they’re using learning targets that are more closely resembling the actual common core standard and not using something that’s a little bit more broader, like the next generation or the standard mathematical practices.

Erik Francis: Well, see, that’s the whole thing is that what I like to say is that as educators, we are responsible to get the kids through the standard. So I’m saying through the standard.

Okay. Which means that our responsibility is to get the kids through that standard. Okay? They may not get there in the time that we have, but that’s why we have RTI, Intervention Support. That’s why we’re looking at, and people don’t look at, RTI is not so much, this kid is this, this kid is this, this kid is this. It’s about time. Excuse me, I’m giving you interventions during this time, I’m giving you targeted interventions beyond the classroom time. I may be giving you strategic interventions beyond the classroom time, the day beyond the quarter semester, but do we all believe that kid can get to reach the finish line to cross it? Okay, that’s good teaching. My responsibility is to get the kids to achieve that standard. I don’t have to like it, but this is what it means to demonstrate proficiency in this grade level, in this subject, with this content. My responsibility is to teach that and to get the kids through it. My responsiveness is get the kids to, through, and beyond it.

So now, and that’s why I turned into an RTI. So if I start with a standard and say, can you do that? They say, no. Well, can you do this? No. Can you do this? No. Can you do this? Yeah, good. Let’s start there. You’ll like this because you’re math teachers. I show teachers that they can achieve a high school master and it’s a DOK 3 in 5 minutes just by doing this process. The standard is, explain why the product of a non zero rational number and irrational number is irrational. So, I said, can anyone who’s not a math teacher do that right now? And they all go I’ll go, alright, let’s tear it. What is the product of a non zero rational number and irrational number? Okay, what word stands out for you? What word looks familiar for you? Product. Okay, so product. So what are we doing? Multiplication. Why? Because that’s what a product is. That’s where you get the answer. Okay, so we just teared that down to third grade. Okay. Okay. Now let me bring it back to high school. So what’s the product of a non zero rational number and irrational number? It says it right in the standard. It’s irrational. Why? I don’t know. Okay, well what’s a rational number? I don’t know. What’s an irrational number? I don’t know. Okay, now I’m in middle school, because that’s where we learn it. Now I’m going to elementary. What does rational mean? I don’t know. What does irrational mean? I don’t know. Okay. What is a nonzero number? Five, two, seven. Okay. Great examples, guys. D O K twos. But what’s a nonzero number? It’s a number that’s not zero. Fantastic. Let’s start there. Now give me a number. That’s not zero. Two. Okay, let’s make a class definition before we get conceptual because if I get conceptual with you, you’re going to go, I like to say it’s like the line in Jaws. You say Barracuda, people go, huh, what? You say shark, we got a problem on the 4th July. So right now, before we go Barracuda, we’re gonna go shark.

What can I do with two? You can multiply it. Okay, great. So that means I can multiply, add, subtract, divide it. Right. So I can do properties operations. Class rule number one. Can I count the two and get a physical amount? Yeah. Okay, good. Class rule number two. Can I measure two? Yeah. Okay. Rule number three. Can I find it on a number line and reach it? Yeah. Okay. Rule number four. So that’s the four rules. So now I want you all to tell me what your non zero number is and how does it meet the four rules? Okay, so now it’s, I’m gonna DOK 2 learn it. Now, here’s where I get through with DOK 3. What if I put a negative in front of your number? Does it still meet the class rules? Okay. So then they’d say, well, let’s think about it. I said, well, can we do the properties operations? Yes. Can we count to it and get a value? Yes. Can we measure it? Some people say no. And you know who says the most that to me? Science teachers. You can’t measure a negative number. I go, are you teaching temperature? Are you teaching sea level? Where’s my economics teacher? Your bank account? Okay, so we can do that. Can we find a number line? So what do we know about negative numbers? So they’re, they’re rational and rational means they make sense and irrational means they don’t make sense. What numbers don’t make sense? Fractions, Mr. Francis. Okay, now I got to hear it back down to third grade. What’s a fraction? One half with a fraction. Three fours. What’s a fraction? Two thirds. Okay, good. What does it mean? Part of a whole. What does that mean? How do I make part of a whole? It means you got to divide it. Did anyone ever teach you that a fraction is a vision problem? No. Yeah, you take the denominator, you divide into the numerator. For those of you guys who need to simplify, put the big number outside the house, put the little number inside the house. Okay, if I, what do you say, one half? Can I divide two into one evenly? No. What do you get? A decimal. What is that? Point five. Okay, good. Is there any numbers after it? Zeros. Are they numbers or placeholders? Placeholders. So that’s called a terminal decimal. And I make everyone do a karate chop, do a terminal decimal. And then, depending on your audience, you gotta teach them about, you can say, so if I divide three into two, what do I get? I sometimes, depending on my audience, I’ll say, have you ever seen the devil decimal or the devil fraction? Because if you divide three into two, I like to call it the Motley Crue fraction. Point six, six, six, shout at the devil. Okay, that’s called the repeating fraction. Repeating decimals, excuse me, repeating decimals.

Sharona: You are Bosley’s new favorite person on this planet.

.:

Sharona: It depends.

Erik Francis: All right. It’s the same thing with English language arts. The worst question we can ask the kids. What’s the central idea or theme of the story? Do you want what you taught me, what I think, or what this book says? You know, I had a school I was working with in North Carolina. First of all, I want to say thank you for letting me just talk and share the experience. Because this is the thing. Assessment is, I look at it to say, it’s not about the pitch you throw at a batter. It’s, can they explain how to hit that pitch? Can they justify why they hit that pitch that way or didn’t? Okay? So, the standard was determine the theme of the story from details in the text and explain how the details support the theme. So, they used. a publishing company that the initials come after the letter G before the letter N and then after the letter G again. So that’s a DOK3 to figure out who that publishing company was. So you’ll get it afterwards.

So the thing is this, is that the question was, what’s the central idea theme of this, of Jack and the Beanstalk? What’s the theme of Jack and the Beanstalk? And I had three options. Don’t listen to your parents. Don’t trust strangers. Mistakes can be corrected. The correct answer was C, but 60 percent of the kids picked B. They all picked the same answer. And the kids who picked A, they picked it because they understood listen to your parents. So, the school was all upset. They don’t know how to determine theme. I go, wait a minute, wait, wait, wait, wait. Let me go in and talk to them. And that’s what I used to do with kids. That’s why I love doing what I can do because I can talk to kids and kids are like, that’s the guy who talks to us about our assessments. I go, okay guys, remember that question you had? Yeah. Okay. The correct answer was C and 25 percent of the kids go, yes. Okay.

But wait a minute. A big portion of you picked B, don’t trust strangers. Why? And they were quiet and no one ever asked them that and finally, one of the student goes, well, if he didn’t buy the beans from the stranger in the first place, there wouldn’t be a mistake to be corrected. Yeah. Yeah. That’s what I thought. And you know what? If the giant didn’t trust Jack to go into the castle and he stole the golden goose and he basically did all that. So now what are they doing? They didn’t answer the item correctly, but did they achieve the standard? Cause they had to determine the theme of the story from details in the text and explain how the details support the theme.

Boz: Yeah, we, we just got through talking about assessment and how assessment can look like so many different things and where you collect that evidence doesn’t have to be that state test that multiple choice test that traditional assessment because what you just described right there. Yeah, they missed it on the multiple choice, but then they were able to explain it.

So

Erik Francis: Well,

Boz: what have they shown?

Erik Francis: This is also a Teacher Pay Teachers issue. Okay, because You can enter a standard on Teacher Pay Teachers, and I’m actually teaching people to do this, and this is how I’m basically trying to meet halfway. So there’s a science standard that says develop and use a model to explain the functions of a cell and the way the parts contribute to the function. There’s four different objectives in there. First, they’re gonna develop and use a model. That’s a DOK3, because they have to consider how to do it. They have to use a model, which is Really a DOK 1 because they have to recall how to do it. They have to explain the function of the cell. That is a DOK 1 because they’re just recalling information. But then they have to explain ways this parts of the cell contribute to the function. That’s a 2. So when I go on Teacher Pay teachers and I pull off that worksheet where they’re only identifying parts of a cell. Yeah, it addresses the standard, one element of it. It’s a cell and there’s the parts. So it’s insufficiently aligned.

Same with math. If I have a standard that says, multiply four digit numbers by one digit number and multiply to two digit numbers using the properties of operation or place value understanding and the objective says, describe the strategy, explain the reasoning. And I pull a worksheet off there that just has two digit by two digit numbers. It’s acceptably aligned because at least it’s addressing half the standard. But to make it more fully aligned, they have to describe the strategy and explain the reasoning. And to make it completely aligned, there better be problems on there where they have four digits times one digit, or four digits times two digits, or three digits times one digit, or whatever.

See, that’s the whole thing. And that’s where we as teachers need to come back and control the curriculum products we have because I can give you that worksheet and say, okay, now I want you to do those two digit numbers. Take one of those numbers and put it on the top digit. The multiple canned and make a three digit times a one digit number. Now what I want you to do is take two of those numbers from your first problem, put them on the second problem, and now I want you to do a three digit times a three digit. See? So that’s how I’m taking control as a teacher. And that actually is less work for us and more work for the kids. But what if I said that’s what the kids will remember when they go to the assessment? Oh, that’s when Mr. Francis made me do this. Oh, that’s how you do it.

Sharona: So I have another question I’ve been dying to ask, and I know we’re long Boz, we’re just gonna have to deal with it because I really think this is important. Because you said something that I want to clarify because of something that we’ve been teaching, which is, you said, if it’s only a DOK one standard, then you should only assess it a DOK one proficiency level. Does that mean that the entire assessment is restricted to that, or that we’re only checking the proficiency scales on the specific standards within the assessment at that? So what I mean by that, for example, is we’re trying to encourage faculty and instructors to do more authentic assessments. And many authentic assessments are at a higher general level because they are gathering evidence from the world and things like that. My argument has been, we’re going to get the authentic DOK 1 level evidence we need, and I can give a proficiency score on that, but inside a bigger, more authentic assessment. What do you think of that?

Erik Francis: Absolutely. I want to make sure this is clear. Okay. The DOK level of the standard informs you the deepest level the kids could and should be assessed on an on demand standardized assessment. Okay. When you’re working with N. W. E. A. When you’re working with Galileo, when you’re working with your state assessments to make a fair and equitable and balanced assessment that the items on there should assess up to that D. O. K. Level.

Now, math is an anomaly because math has the standards of mathematical practices. That’s And that’s where we have to understand the curriculum we get because we’ll get these units and it’ll talk about this is the standard addresses and then I’ll talk about the core ideas and enduring understandings and then boom, they list all seven of the mathematical standards and practices. Okay. And you’re just like, okay, but you’re using that to deepen the complexity. There is nothing wrong in your teaching and your assessment. with going beyond the standard. This is set up for on demand standardized assessments.

In fact, what I tell teachers is if you think it’s a deeper DOK level, fine. How are you going to teach it that way? Okay. And it’s not about the teacher, how they teach it. It’s about how the kids learn it. I mean, I can take two plus two and take it to a DOK four with just two plus two. We started it. What’s two plus two, four. You’re right, but why? Because when I do this, now I’m in a DOK2. Now here’s a DOK3. Could 2 plus 2 equal 5? No. You’re right, but why? Because 2 plus 2 equals 4. What do you mean? Well, when I have 2 and 2 and I add them together, I get 4. So what would I need to do to make it 5? I have to add one more. Great, you understand the concept of addition. Now what if I use addition, decimals, and rounding? Could 2 plus 2 equal 5? So someone actually did it.

And the first time I’ve had this, I learned this because someone tried to embarrass me when I said, what’s two plus two? They said five. And I said, what do you mean? And they go, you’re going to do it. And they told me how I did it. I’m in front of 600 people was in Bangor, Texas six years ago. Okay. And I said, wow, what mathematically allows me to do that? And she said to me, triangles. And I said, thank you for teaching me something new. Notice I’m taking that with me. Okay. Six years later, that’s what we’re teaching the kids to do. Not for the exit ticket out the door, not for that test. At the end of the week, and not even for that assessment, it’s 10 to 20, 30 years from now where they go, Hey, you remember when you tried to embarrass me and bang her, and now I can do and talk?

And I made this as part of my presentation. So for the first time ever, somebody said yes. And I said, you’re right, but why? And I said, they said, because if you have 2.2 plus 2.8, that would equal five. And I go, wait a minute. What’s 2.2 rounded? I said two and I said, what’s 2. 8 rounded? And they said three. I said, no, you didn’t prove it. Why? Cause what? But it equals five. I go, no. Could two plus two equal five. You made two plus three equal five, which is great using decimals and rounding. But what would you have to do? So if I did 2. 3 and 2. 4, that would equal 4. 7, which equals five when I round, see. So it’s not like that. Yeah, but that’s D O K three and math proofing.

I used to, when I taught math, I used to give the kids the answer key. I said, here’s the test. They said, all the answers are there. I go, right, tell me why they’re all correct. Or I give them a worksheet and I put down right and wrong answer. I go, which ones are correct and which ones are incorrect and why? They walk into class. They say, here’s the problem. Here’s the answer. Which ones of mine are correct and incorrect and why? That’s the simple thing. But if you notice, I’m doing less and they’re doing more. If I say to you, how does The Outsiders address the idea of family in Chapter 1, and the kids can tell me, well, the family is dairy and soda pop, but they think of Dally and Johnny and Steve and Two Bit like brothers, so what’s the message? Family’s not who you’re just related to, family’s who you feel close to. You’re right, but why? And they can give me evidence from the text, that’s a DOK 3.

When I’m doing scientific inquiry, you say, you have to get this outcome. How could you do it? Okay, it didn’t work. Why? That’s a DOK 3. Historical arguments should George Washington be credited as the first president of the United States, or should we acknowledge the other presidents appointed on the Artist Confederation? Yes or no. You’re right. But why? See, that’s the simplest thing. So just to make it clear on an on demand standardized assessment, it should not go beyond the D. O. K. Level of standard. You assess to it and through it unless it’s math, but an authentic assessment. You can go anywhere

Boz: I think that’s gonna kind of bring us to a great stopping point because I think that is an excellent point that what you were saying that the assessment should stop at the the DOK level of the standard itself is for those on demand state, you know, state type of tests. If we’re talking about, you know, other types of tests, then absolutely. In teaching past the DOK level of the standard, absolutely. Don’t get that message wrong, but I think that’s a great point to kind of, you know, In this on I, I do want to thank you for coming on and having this conversation.

I do know that you have a, a really new book that has just come out in the last six months. Inquiring Minds Want to Know.

Erik Francis: Right. Inquiring minds want to learn, right?

Boz: Inquiring minds want to learn, I’m sorry.

Erik Francis: That’s the National inquirer. Inquiring minds want to know is the National Inquirer. Inquiring minds want to learn is the Erik Francis book.

Boz: So I would love to hear more about that and have you back on again to talk about that book. Sharona, any kind of closing comments on this?

Sharona: I was just going to give Erik, do you have a 30 second elevator speech about the new book that you could give us?

Erik Francis: Yes. So it’s all about how can you basically turn your standards, your learning targets and objectives into good questions. And how can you use those good questions to engage students and promote student inquiry?

Sharona: Okay. Well, I know what book I’m buying as we speak.

Erik Francis: Well see, here’s the thing, because we’re at the world of assessment. One of the things I’m challenging people is this, and I’m actually going to a school district. You see, I said you were in Michigan. I’m going to Michigan next week. At some point, I want to challenge them to say how can you turn that standard into a question and give the kids sample items or sample text that they can use examples to support their responses to that question. So, like, for example, with math, I could say how, how do you fluently multiply multi digit numbers using a standard algorithm? Here’s 20 problems. I want you to pick five today and answer that question. And that’s the standard.

So, so, but let’s make that the sequel because I know we’ve talked a lot and I always say this DOK is the concept questioning and inquiry is the process that will get the kids to comprehend and communicate their learning at that DOK, level and test questions are not questions. They’re items. Sometimes they’re phrased as interrogative statements. And I actually talk about in the book. Bad questions, unfortunately, are questions we see on tests like the trick questions, which one..zzzz okay, what are you trying to do there? Or, multiple choice. No, that’s an item. Sometimes it’s phrased in interrogative state. Yes or no? That’s a cognition killer and a conversation killer. Because there’s four ways to answer it. Yes, no, I don’t know and I don’t care. You know? Leading and loading questions.

If I said to you, why is The Adventures of Huckleberry Finn inappropriate? Well, maybe it’s not. Maybe it’s based upon the era that’s looking at the book because that’s the whole thing by it. So and I talk about how you turn bad questions to good questions and factual questions are good questions because they get the kids to think they stimulate thinking lower levels according to blooms, but they’re still stimulating thinking. But that’s a sequel conversation for us. How about that, guys?

Boz: Yeah,

Sharona: absolutely.

Boz: Love to have you on again. I’m a couple of books behind already so, but this is definitely one that I’m going to be looking into as soon as I get caught up on some of my other books I’m behind with right now.

Erik Francis: I appreciate that thank you so much. And you know, please keep in touch with me and anyone who’s listening to feel free to reach out. I mean, Conversations are no cost. You know what I’m saying? And you know, that’s the thing I love talking about it. I try to engage with people like, let’s have a conversation about this. Let’s talk about this. And that’s I think the best way we learn. And that’s the best way we spread good messages out there, especially when it comes to instruction, learning and assessment, especially when it comes to DOK cause there’s so much misinformation out there.

Boz: And then we’ll be sure to link, not just these two books that we, that we were just mentioning and then talked a lot about, but also your third book, now that’s a good question. So we’ll have all three of those linked. Go, go check them out and we’ll see you next week.

Sharona: Please share your thoughts and comments about this episode by commenting on this episodes page on our website, www. thegradingpod. com. Or, you can share with us publicly on Twitter, Facebook, or Instagram. If you would like to suggest a future topic for the show, or would like to be considered as a potential guest for the show, please use the contact us form on our website. The Grading Podcast is created and produced by Robert Bosley and Sharona Krinsky. The full transcript of this episode is available on our website.

Boz: The views expressed here are those of the host and our guest. These views are not necessarily endorsed by the Cal State system or by the Los Angeles Unified School District.

Leave a Reply