– Hi everyone, I’m Niya Bond, Faculty Developer here at OneHE, and I’m thrilled to be joined today by Emily Gravett. Emily is gonna be talking to us about transparency in teaching and learning. Emily, if you would share a little bit about your background and your interest in this topic.

– Sure, thank you. Thanks for having me, this is wonderful to be with you all. So this is me on this slide, Emily Gravett. I’m an Associate Professor of Religion at James Madison University in the State of Virginia in the United States. I also, for many, many years, worked at teaching and learning centers, a variety of teaching and learning centers in the United States, including the one at JMU. And so I spent quite a bit of time not only teaching my own courses, but talking to other people and trying to help other people reflect on, and potentially even improve their teaching. Today, and this is just an inclusion measure to describe visuals, but today I’m wearing a red kind of top and some big earrings, and I’ve got my clear glasses on, and I’ve got brown hair and brown eyes, and there is a picture of corn behind me.

So I’m coming to you today to talk about transparency, and thanks Niya and Dasha for organizing this and for giving my course a plug. We’ll see what we can accomplish in 20 minutes. This is a topic that we could spend a whole hour or 90 minutes on, so I’ll just give you kind of the basics and then hope to spark your interest in thinking from there. So I came across the idea of transparency in learning and teaching maybe about 10 years ago through the research of Mary-Ann Winkelmes. She had done kind of a broad scale study of like seven schools and 1,800 students and 35 faculty, and faculty had been trained by her and her team to make two assignments more transparent. And we’ll get into what that means obviously a little bit later. And then each faculty member taught two courses, one with the more transparently designed assignments and the other without. And then she sort of measured what happened. She and her team measured what happened, the effects of this teaching intervention to have more transparent assignments. The effect was measured by the amount of transparency that students perceived, student self ratings on predictors of academic success. So academic confidence, sense of belonging, and improved mastery of skills that employers value, which we know leads to better GPAs and better retention rate. There was a direct assessment of student work and then also short-term retention rates were studied in this research project that she led. That was really cool. And it turned out, of course, that transparently designed assignments were better for everyone and led to sort of more academic success and more retention. But were especially good for students in kind of vulnerable or underrepresented populations. So good for all, and especially good for some. And so this is how I came across this research, and it seemed kind of like an easy intervention to do. And so I just started designing my assignments more transparently.

So I’ll convey here the three principles of transparency. If you’re thinking about making your assignments more transparent, and these can be little in-class assignments, these could be semester long assignments. You could even think about transparency at the level of like a program or a major or your curriculum. But for the purposes of this talk today, I’ll just kind of focus on major significant assignments in the context of a class, that might be worth like a percentage of the grade for students or that they might be working on outside of class, that kind of thing. So the first principle of transparency is purpose. The second is task or tasks, sometimes plural. And the third is criteria. So three principles of transparency, purpose, task, and criteria. So let me go into a little bit of what each of these are.

So for purpose, you might be asking yourself, when you’re thinking of an assignment and you’re focusing on purpose, and wanting to be a little bit more transparent about purpose with your students. You might ask yourself what skills and/or knowledge the assignment will help students to learn? and how you can convey that to them, how you can make that clear to them, how you can write that down on a piece of paper for them or say it verbally in class. So what skills or knowledge is this assignment helping the students to practice or to learn or to master? Are there clear learning objectives for the assignment? So some of you may be familiar with the idea of learning objectives. You might be familiar with something like Bloom’s Taxonomy or its revisions. Some of you might be familiar with like Fink and his Taxonomy of Significant Learning Experiences. These are things that if we had a much longer period of time together today, I would delve more deeply into. But for now, I can just kind of point you in those directions. But the idea is that there’s different levels of learning or different sort of types of learning, different sorts of outcomes or objectives or goals or aims that students might be trying to reach in your particular class, or in this case, in a particular assignment. And so are those clear to students? What are those? Do you have in mind your own aims for the assignment? Why are you having the students do it? Something I like to think about in terms of purpose is how maybe the assignments objectives align with any overall course goals or objectives that you might have.

So in the State of Virginia in the United States, we’re having to put together curriculum and syllabi where we’re very clear about what it is that this course will help students to accomplish or help students to achieve. So those show up in the form of objectives or goals. So those are overarching for a particular class or course or semester or quarter. But then individual assignments probably need to be supporting those overarching goals or objectives for the entire course. So how does this smaller assignment, this one piece of the class, fit into these larger aims that you’re trying to help students to reach or to meet? You might ask yourself when you’re thinking about purpose, why is the assignment important in this class, but also beyond in the so-called real world: What impact might it have on students? How might it help them in terms of career or jobs? How might it make them better or more effective citizens? What kinds of, you know, different life skills might it help them to acquire? So I teach religion, something that we talk about a lot in my courses is empathy. So how might an a particular assignment help students to practice empathy or to gain skills in empathy? I think of that as a skill that’s good, not just in school, but in the world beyond. So when you’re thinking about purpose, it’s sort of like, why? Why am I having my students do this assignment? What’s the point? And if you can’t identify that, I think students can smell that, or students can sniff that out, right? They then might complain about busy work, like why am I having to do this? It’s not clear at all how this fits into the class, why we need to do this kind of work in this class. So it’s nice for you to get clear on why you’re having students do what you’re having them do. And then if you can get clear on it, then you can convey it to them.

So I wanna give you an example. This is far from my areas of expertise, if I have any areas of expertise. This assignment comes from the TILT, the Transparency in Learning and Teaching website, which Mary-Ann Winkelmes and her team have developed and kept up over the years. So you can find this assignment and many other examples on the TILT website, and those links will be in the slides provided later. But this is an example from biology, because I figured probably not everyone on this webinar would be in religious studies like I am. And so I wanted to provide some kind of different disciplinary examples. So here’s an example of purpose from an actual assignment that was designed to be more transparent in a biology context. So this instructor on the assignment that they gave to students, writes, “The purpose of the exercise is twofold: first, to simulate the process developmental biologists go through in the laboratory when observing a novel mutant, and second, to help you think about downstream physiological effects of loss of function of various genes. This assignment will help you practice the following skills,” and then some skills are listed, observing and describing, et cetera. So this instructor is being very clear with students, “Here’s the purpose, it’s twofold. I’m having you do this exercise because. I have reasons, these are the reasons. I’m not keeping them secret, this isn’t a mystery novel, I’m gonna lay it all out there for you so you understand why we’re doing what we’re doing. And here are the skills that I’m hoping this assignment is going to help you to practice or to master.”

The second principle of transparency, as I mentioned, is task or tasks. So this is sort of the ‘how’ part. So if you’re thinking about being more clear, more explicit, more transparent with your students about the task of an assignment, you can ask yourself some questions. First of all, maybe, “Does the assignment make sense given its purpose?” That’s always a good check. How can you make the genre or type of assignment more obvious or more explicit? So when I say “research paper”, I might mean something very different than even a colleague in my same department means by “research paper”. So not sort of taking for granted some of those genres or some of those types of assignments, but really laying it out for students so that they understand what it is that you’re asking. If there’s an audience you want students to imagine or some sort of role you want them to inhabit, this is the time to lay that out as well. Is there an order or a series of steps that students should be following? So some of you may be familiar with like scaffolding or sequencing or sort of stepwise assignments where you’re building up to something, or you’re having students have a lot of support at the beginning and then you’re kind of taking it away as they get more and more adept, or you’re having multiple staggered deadlines with opportunities for feedback, these are the sorts of steps or the sort of order that I’m envisioning here for the task, and just again, kind of laying that all out for students so they know what they should be doing in what order. If there’s any pitfalls that you know that you’ve come across before having done the assignment in the past perhaps, or just heard rumor from colleagues that you know students are struggling in certain ways, you might lay those out for students as things to avoid. This is the principle of transparency that is basically trying to get you to think about what you’re asking students to do and how.

So again, the same example from this biology assignment. “This exercise is asking you to think about this series of mutants. All these mutants survive to adulthood, but display phenotypes that are based on underlying developmental defects for each mutant.” So here’s where the instructions really are very clear. “For each mutant, examine the images and watch the associate video,” here it’s linked. “Then, answer the following questions for each mutant.” And then there’s a list of questions, and I’ve just excerpted this so you don’t get the full list. But again, this is on the website so you can find this later if you wanna take a look more deeply. But you know, describing the phenotype of the mutant, how does it differ? And then there’s this series of questions. So here’s the task of this assignment, is taking each of the mutants, looking at the images, watching videos, then answering a series of questions.

The third and final principle of transparency that I wanna mention to you all today is the criteria principle. So the questions here are: By what standards will the assignment be graded, or evaluated, or assessed? For the purposes of our presentation today, we can kind of use these terms interchangeably, even though they’re not. What are the standards? How are you gonna be grading this? Students, I think, have a right to know that in advance. Or is there a checklist of expectations? Are there requirements? These can help guide students, so those can be conveyed to them early on. As I mentioned, I think these things should be conveyed in advance. So have these criteria been conveyed in advance? Preferably, when you’re giving them the assignment so that they can be working on the assignment in tandem with the criteria. I remember an assignment in graduate school where I had to do reading responses to some of the books that we were reading, and I was just kind of playing with them and having fun with them and doing the assignment, but being a little bit flexible about it and thought I was doing fine. And then at the end of the semester, I got like a, not a great grade on the reading responses, and I was really surprised and I asked the instructor, and it turned out I hadn’t been doing them right along the way, but I had no idea ’cause I wasn’t getting that feedback, but I also wasn’t aware of the criteria that I should have been fulfilling. And I would’ve maybe done those reading responses a little bit differently if I had known what the criteria was and if I had known I was not meeting the criteria. So this is related to the next question that you might consider. What opportunities will students have to practice before the deadline, and who will give feedback, maybe even besides you, the instructor? So is there opportunity for practice? Is there opportunity for feedback? Is there opportunity for students to know whether they’re on the right track or not, according to the criteria that you’ve laid out?

Research shows that it’s really beneficial for students to see examples of assignments, preferably multiple examples, so they don’t latch onto or lock in to just one as the only one correct way of doing something. And these examples can be particularly effective for students to review if they’re annotated. And I’ve even in class, in my own classes, had students actually annotate the examples themselves according to some kind of rubric or some kind of criteria that I’ve given them for that assignment. But what’s working well, what’s not? Are they meeting this expectation, are they not, right? Where’s the strength? Where is an area of growth? So this principle is focusing on how will this work be assessed, and making sure that’s obvious and known to all students. And this is where I think that we get the results from the intervention that Mary-Ann and her research team found. Because some students come into college, particularly students who have families of college-going students, you know, maybe having a little bit more familiarity with the rules of college or what’s sometimes called the hidden curriculum. And some students may be coming in and not knowing anything really, about how to do well in a college classroom. And so we wanna make sure that we’re not sort of inadvertently rewarding students who are coming in with knowledge and prep and skills that they didn’t really gain in our class, that they had kind of an advantage coming into our classroom. So here’s the example from biology. The instructor is saying, “As scientists, we should strive for specificity and accuracy. As such, I encourage you to avoid vague description or unclear hypotheses. Note that in the example given below,” so there’s actual example, “the student clearly describes the appearance of the worm’s movements and how they contrast with wild type,” and I left out some stuff here just for the sake of the slide, it’s already taxed you enough. “Furthermore, her hypothesis, though ultimately incorrect, provides a direct physiological mechanism by which the phenotype could arise. Finally, she identifies the actual underlying cause and cites the related peer-reviewed primary journal article. Note also the citation style used.” So this instructor is providing an example and then noting what the example does well, pointing out particular features that the assignment should have, like a citation style that they’ve selected.

So this is an example from a biology context of criteria. As I mentioned, the TILT website has lots of different examples from lots of different disciplines and lots of different course topics. I’ll go ahead and have Niya or Dasha put into the chatbox an assignment that I have created, a movie review assignment. And if I click on it here, it might open up for us. And I can just show you what this has looked like for me when I’ve put it all together. So I teach a religion and film class. I ask students to write a movie review because that’s a common genre of writing in film studies courses. So I lay out what the purpose of this assignment is, just like I’ve described it to you, it’s a common form of writing, common assignment, it’s good for you to practice. I link it to my overall course goals. So I pull from my overall course goals to convey how this specific assignment supports them. I have a task that I ask them to move through. So I have them imagine that they’re a movie critic for a fictitious local newspaper. I tell them that they need to choose a recent movie, I tell them they need to watch it twice, I tell them they need to take notes on it. We’ve been practicing note taking in the class so they know what this means. And then I tell them to write a review for that movie following the format that we’ve been discussing in class. I tell them what the end result is, there’s a deadline, and then I’ve got a rubric. So often rubrics are used for the purposes of the criteria principle, but it doesn’t have to be a rubric with like a table or a matrix. These can be sort of checklists or just pros descriptions. Some of you may not want to be sort of limiting the creativity or the types of assignments that you receive from students. And so there’s lots of different ways to convey the criteria to students. But rubrics are one that I’ve really appreciated and have found really helped students, and have reduced a lot of the grade complaints that I used to get.

So let’s go back to our slides, you all can look at this assignment later and ask me questions about it if you want. I want to point you to a resource that my colleagues and I at the University of Virginia, which is a university right down the road from me where I live, created. So we took Mary-Ann Winkelmes’ research into transparency and the TILT project and we created a rubric for measuring transparency on kind of major or significant or substantial assignments. When you get the slides at the end of our session today, you’ll be able to click on the link there in the title, “Measuring Transparency,” and that should take you to, I think, like A PDF that includes the actual article that we wrote and a description of the rubric and the rubric itself. It’s also available on the TILT website as well if you poke around under resources. But this is a rubric essentially if you want to do a little bit of self-reflection or a little bit of assessing your own assignments. So if you have an assignment and you’re wondering how transparent it is, or if you could make it more transparent, how might you do that? You can use this rubric to help move you in a more transparent direction. I don’t think there’s really probably such a thing as a fully transparent assignment or a fully, like opaque assignment. But probably this is a spectrum, and there’s ways of being more or less transparent depending on your discipline, depending on the level of your course, depending on your personal teaching style, depending on the time in the semester and how many other types of assignments and things like that that they have encountered before. So there’s definitely not one right way to do this by any means, but this rubric is maybe just a resource for instructors to be taking a look at their assignments through this new transparent lens. So I’m happy to be reached at my JMU email address, which I have put on the final slide here in case you wanna email me and ask more questions about transparency. This is a topic that I’ve talked about quite a bit and worked on for a little while. I’m so happy to kind of field email questions after the fact, but also to do a little Q&A for about the next 10 minutes or so here in this space.

– Thanks so much, Emily. We do have three questions that have been queued up in the Q&A section. I can just go one by one if that’s okay.

– That’s great.

– All right-

– I promise no answers. I just am very excited about there being questions.

– Okay, from Ray, “When providing annotated examples, are they graded, and are they from the same assignment?”

– Yeah, so I would… I mean, in my experience when I provide annotated examples, I do provide them from the same assignment if I can. I mean, if I’m doing an assignment for the first time or it’s the first time I’m teaching a class, then I probably haven’t done it before. But if it’s an assignment I’ve done before or it’s a class I’ve taught before, I do provide examples of that same type of assignment. And again, kind of usually I try to do multiple examples. I strip them of any identifying information, and/or I ask students permission if I can share their examples. Sometimes I even include examples from the same class that I’m teaching right then and there. And again, I just sort of ask students permission if it’s okay to share. And students, especially if you’re shining a light on them kind of positively, students are usually pretty happy, happy to have their work highlighted. Like I mentioned, sometimes I do the annotations as like a group activity or as a class discussion. So you know, putting up an example and then having students look at the rubric, and sort of what would they notice, how would they evaluate it? If they were giving it a grade, what would they do? What would they, you know, highlight and call attention to? So you can actually ask students to do the annotation as an activity in class if you’re wanting to go over the assignment. But otherwise I just would annotate them myself and share that like PDF with them.

– Great, thank you. It looks like the second and third question actually go together from Carla, “Three critiques of TILT, I have heard through the pilot program I am running at my institution, too much handholding, potentially easier to use AI, students are not the experts in pedagogy to give feedback. How do you address these concerns?”

– Yeah, so the handholding one I hear all the time. And so that’s why I kind of mentioned like, okay, if you wanna allow a little bit room for more open-endedness or more creativity or more flexibility with your students, there’s obviously lots of ways to kind of toggle each of these, particularly maybe in the criteria area if you wanna allow for more creativity, for instance. I am sympathetic to the handholding, and at the same time, I also feel like if we’re gonna be assessing student work, then we need to make sure that we’re teaching them how to do it. And so we need to be really clear with everyone and make sure all students are on the same page about what’s expected of them, because otherwise, we’re gonna be advantaging some students and disadvantaging others, because some students are gonna be coming in with more knowledge and more prep. So if it’s something that they’re gonna get graded on, then I think it is our obligation to make sure that we’ve taught them how to do it and that we’ve been very clear about what the expectations are. I know I need a lot of handholding when I’m learning how to do something for the first time. I need to watch videos, I need to get feedback, I need to practice a bunch, I need to fall and fail. And if that’s handholding, then I think part of teaching is sort of handholding, because we’re helping students move from a novice or a beginner position to having a little bit more comfort and facility with a concept or with a skill. The AI question is… That’s a bigger, I mean, I think that’s like a bigger question than I can really answer. I’m sort of the like, “Don’t feed our inevitable overlords camp about AI.” So I’ve been kind of trying to like avoid it, which is of course a losing battle. So you should probably not take any advice from me about AI. But yeah, I mean AI is gonna completely transform higher education, you know, and I don’t really know in what ways. But yeah, that’s something we’re gonna have to think about with all of our assignments, writing especially.

– Perfect, and did you talk about the students and feedback giving- .

– Yeah ! So I completely agree that students are not experts in pedagogy in the United States. Not all faculty members are experts in pedagogy either. So we don’t get trained necessarily to know how to teach. When you’re in K-12, when you’re teaching younger students in the United States, you do actually get trained, but at the professor level, that’s not always the case. Students are sort of expert in their own experiences of an assignment, but they don’t necessarily know what’s best for them. So for instance, in my classes, I do weekly quizzes. They don’t necessarily always like it or understand it until I explain that this is something that actually, you know, sort of frequent testing and retrieval can help them learn something better. I think I might need a little clarification if whoever asked that question would be willing to put it in the chat in terms of how that might play into transparency. Because transparency, with the principles that I’ve talked about, is really mostly up to the instructor to be deciding in terms of what the purpose is, or what the task is, or what the criteria is. But again, happy to try to answer that better if I got a little bit more information about how students’ lack of expertise may kind of affect the operation of this idea.

– Well, it was a question from Carla. So Carla, we’ve given you permission to talk if you want to say and clarify-

– Or type, it’s fine, or not. I can just fumble around-

– Yeah, on an invitation, everything is an invitation.

– It was more about allowing students to participate in the adult… Oh yeah, yeah, yeah. Yeah, I understand that. And I have sometimes, I mean, this isn’t exactly what you’re asking, but I have sometimes really involved students in this process and asked them like, “Why do you think we should do this assignment?” and then kind of build that in in terms of an explanation, or “Which parts were you finding difficult?” and then built that into kind of my commentary about the pitfalls or, you know, involving students in the annotation process. They aren’t gonna be experts on the things that I’m teaching them yet. And at the same time, they are the audience or the beneficiaries of that teaching. And so I think it’s really interesting to get their perspective. It’s sort of like how I feel about course evaluations. It’s like they have utility, but that utility is somewhat limited. But they can talk about some things, and then they can’t talk about other things as well. Oh yeah, and in terms of, you know, student self-reported data in the research on TILT, yeah, that’s a limit of self-reported data. People are often not very good about indicating us, you know, gains that they’ve experienced or knowledge that they have. And so Carla’s absolutely right, you know, when we are looking at TILT data and thinking about the effectiveness, we’re taking some of that from students self-reported experiences. Some of it’s from the assignments themselves that students produced, and so that’s a little bit different. But some of it is self-reported, and self-reported data has its own limitations. That’s absolutely right, Carla, thanks for raising that. I understand what you’re asking now a bit better, I think. I do see a question about would it be possible to explore the impact of AI on transparency in education? That was the sort of the AI, yeah. The more transparent your assignments are, the easier it may be to punch into ChatGPT and get a response. Interestingly, since we have just a moment, I can share an anecdote. My partner put some of my transparently-designed final exams into AI with two different student profiles. One was like a student who was very like eager and headed to grad school, and one was a little bit more of like a just-trying-to-get-by type of student, and then figured out what those students created by AI would produce for my final exams. And it was really interesting for me to read because some of the responses were very clearly hallucinated, and some of them I would’ve given full marks for. And so this was an interesting exercise for me, maybe also to do in the future in advance next time so that I can see what the questions are that maybe I could improve or the questions that would be easier to cheat. It also did give me a reference point to have the work of the AI already there in case I suspected any students of using AI when they weren’t supposed to. I could just easily compare it to the results that AI had produced. But yeah, the questions about AI are broad and tricky. Thank you everyone, this has been wonderful.