– Thank you all so much for being here today. It’s wonderful to see names that I recognize and new folks joining. So I’m excited to share and just dig into some microdesign strategies that we can use for our online assessments. So as Olivia mentioned, I’m a professor in academic technology at Bakersfield College. I know that there’s one other Renegade in the room today, so thank you for joining. And I am going to be going over some different microdesign strategies that we can use to make our online assessments less vulnerable to AI misuse. You can access the slides with this QR code, but I’ll also go ahead and drop it in the chat for everyone just so you have it for your reference.

So today I’m going to start off by focusing on AI and academic integrity. We’re going to explore some of the benefits and limitations of online assessments, and then we’ll wrap up with a discussion of the microdesign strategies and how we can use those strategies to help make sure that students are not offloading learning, but they’re engaging critically with our courses. So before we dig in, I want to ask everybody, what is one snack that you can’t resist at night? If you can go ahead and drop it in the chat. Your favorite snack. I know that some of us are coming up on nighttime, some of us are just after lunch. Some of us are almost ready for bed. Oh, excellent, excellent. I saw cookies. That’s my favorite. My daughter calls me a cookie monster. I love cookies. Pringles, wonderful. Popcorn, another one of my favorites. Chocolate. Who can resist? Okay, yes. Nerd Clusters. Excellent. Hershey’s Kisses, delicious. So imagine yourself sitting next to your favorite snack on the couch. How many of us could actually resist taking a handful of that snack just sneaking in a little bit of a little snack? Imagine we’re next to an open bag of potato chips. How many of us could resist taking a handful of potato chips? Yes. Excellent, April. Not many of us could resist, right? It gets even harder if we’ve been working all day, if we haven’t eaten, we’re really tired, we’re really hungry, we’re sitting on the couch next to that snack or a bag of potato chips. Many of us are going to eat a handful of potato chips, right?

So this is really very similar to how students experience AI. It’s really a lot like sitting next to an open bag of potato chips. Many people just cannot resist that. And we know that many of our students are really busy. They’re juggling childcare. We know that the majority of students, even at residential four-year institutions have responsibilities beyond just school. And so for students, it’s really difficult to resist using ChatGPT to complete that essay. It’s incredibly easy to copy and paste that prompt and create a high quality essay in a matter of minutes tops. And so as instructors, what we can think about doing as we’re redesigning our assessments is we can think about how we can effectively close that bag and put it into our pantry. So it’s harder for students to access, and it just makes it much less likely that students are going to turn to AI to offload learning. When we think about doing this, we can think about this quote by James Clear. He’s the author of “Atomic Habits”, and he says, “We do not rise to the level of our goals. We fall to the level of our systems.” And as we’re thinking about our courses, we can think about how we can design systems that help to promote academic integrity for our students. And that help effectively, again, move that potato chip bag, close it, and put it into a pantry. I love that Karen has brought up a really good point that potato chips and AI are designed to be addictive, right? They have that excellent crunch for potato chips, and AI is meant to keep users engaged on the platforms. And so these things are really designed to keep people using them, whether it’s potato chips or AI. And so we want to think about how we can build systems that help support our students in learning and not create guardrails so that students can’t offload learning to AI.

So when we think about online assessments, there’s a problem that appears. And for those of us that are used to teaching online, it’s a similar problem that we see with online courses. So when we copy and paste our face-to-face courses into an online environment, it often augments the limitations of the online environment without realizing the benefits. And the same is true with online assessments. So if we take the example of a discussion – in a face-to-face discussion, I can go into class, I can say, “Hey everybody, share your thoughts on the reading this week.” And students will respond, I’ll be able to jump in. I’ll be able to immediately guide the conversation in ways that encourage students to think a little bit deeper. I’m able to move past that vague prompt. But in an online discussion, I don’t have that immediacy where I’m able to steer the conversation. Instead, I’m going to have to probably experience a lot of AI-generated content. I’m also going to see a less engaging discussion. And so when we think about our online assessments, we want to think about how we can take the benefits and maximize those while minimizing the limitations. So let’s just take a moment to look at some of the benefits and limitations. As you can see here, we have several benefits. Multimodality, mobile friendliness, interactivity, and anytime, anywhere learning.

So let’s go ahead and just take a moment and drop in the chat something that we’ve seen in an online assessment where we’ve seen folks really engaging with these benefits. How have we seen students use these benefits in our online assessments? I’m seeing wonderful examples here. I’m seeing… I scrolled too far back. I went back to Hershey’s Kisses. Of benefit everywhere, but not with online discussions or online assessments. So I’m seeing peer reviews. April, I’m so glad you brought up the no computer, but has a phone. We’ll talk about that in a little bit. Engaging in discussions, polls, we also see quieter students feel more comfortable. There’s actually some really interesting research that shows that- so we can see invisibility as a limitation of online assessments. It can also be a benefit, right? It can help students feel a little bit more comfortable, a little bit more emboldened to participate. So April, thank you so much for bringing that up. Videos, multimodality, love that- with questions, interactivity. Again, students not having laptops or I’ve, you know, in my work I’ve spoken with students who have laptops, but they’re really old and slow, so it takes forever to turn on. So it’s a lot easier to just be able to turn on and, sorry, pull out the phone that they already have with them everywhere. TikToks, love that. Work on breaks. At work. Excellent. Yep. Yep, we all appreciate being able to use our phones. Being able to complete work whenever they need. These are wonderful examples that we’re seeing that really help to leverage the benefits of online assessments. And some of these benefits really center on that anytime, anywhere.

So I’m seeing a lot of focus on how online assessments really help reduce barriers for our students. When I first started teaching online, I asked students to share how they use, sorry, I asked students to share if they were working or if they had children, and I would make notes in the Canvas notes about, you know, whether they had job or outside school responsibilities. But I stopped writing that down because every one of my students was balancing outside responsibilities. Video responses. Awesome. These are beautiful examples of how we can leverage some of the benefits of online assessments. So again, going back to multimodality, we can think about how we can create audio or video or images and integrate that into our online assessments. We can have students create those, that content, students consume a lot of multimodal content, but they may not have a lot of experience creating multimodal content. So that can be really helpful. I love the focus on accessibility, especially in terms of mobile friendliness. We know that a lot of our students don’t have access to home internet. It’s actually gone up, so it’s about 20% of students in the US rely on their phones for access to home internet. And so it’s really important for us to consider how our assessments play on the phone if students are expected to complete them outside of coursework. We saw some excellent examples of interactive assessments. So polling, videos with embedded quizzes, all of those are harder to offload to AI tools. And then again, anytime, anywhere, learning, being able to support students who need to use that 10 minute break at work to complete an assessment. Or a student who needs to be able to nurse while they’re finishing their assessment at 3:00AM. There’s been some really great research about student parents out of the Wheelhouse at UC Davis. And there’s one piece in particular called “Like a Juggler”. And they talk about the pull that a lot of student parents feel where they feel like they have to choose between their coursework or helping their kids be successful in school. And so the more that we can help students have our courses fit into their lives, the more that we’re going to be able to reduce those barriers of access.

There’s also some limitations, right? There’s that delay that I talked about in the example of the online discussion. There’s a sense of invisibility, which can help democratize engagement, but at the same time, it can also embolden behavior that we may not necessarily want to encourage. We know that sometimes people don’t behave the nicest in online environments. They may not act as they would in a face-to-face. And so, sometimes that can be good, that can increase participation. Other times we need to remind everyone that we’re still humans and treating other humans like humans. But in terms of invisibility, students may feel more invisible. They may feel like, how is the instructor ever going to know that I completed this assignment with AI? And of course, the biggest one is that frictionless cheating. So the ease in which students can complete an essay is again, too big of a temptation to expect students to resist. And so that’s why we can think about microdesign strategies that can redesign our assessments to leverage those benefits, minimize the limitations, and ensure that our students are learning. Alright, so time to get into the microdesign strategies. So the first strategy is thinking about how we can create mobile-friendly assessments. So in the chat we saw many examples of why this helps us best serve our students. I’ve also dropped in the chat a couple of examples. On my website, I have some free resources, eight mobile design principles, and three of those focus specifically on strategies we can use to create mobile-friendly assessments. And I also, just for fun, shared the mobile design micro-fellowship with you all. I hosted a micro-fellowship just this semester, and folks shared different strategies that they used in their courses. It was really engaging. So if you want to see some examples from other faculty, feel free to check that out. One of the things that has really stood out to me recently, I’m wrapping up an NSF grant, and I found that there’s a statistically significant difference in how students are using their phones based on their demographics. So we found that student parents and part-time students are more likely to use their phones to complete assessments.

So when we create mobile-friendly assessments, not only can we better support our student parents and our part-time students, but we can also help to create more engaging assessments that students are less likely to offload to AI. So, one really easy way we can do this is if we just, in working with a faculty- he was a math instructor- and instead of having students type out an equation and submit a Word doc, he simply had students write their equation on a sheet of paper, take a picture with their phones, and submit it. That’s a really easy small change we can make, and it makes it much harder/much less likely that students are going to use AI to complete that assessment. Another strategy we can use is inspired by Fabiola Torres, an amazing instructor at Glendale Community College. And she has students share pictures of course concepts as they’re out and about in their everyday life. So if I was teaching science, I could have students take a picture of a plant adaptation that they see on campus. While you could generate those images pretty realistically with AI, the more specific you are about where the image is being done, the less likely it is that it can generate something specific. So, when I was testing this out, I found that it was able to create an image of, you know, a plant adaptation in a typically residential US suburb, but it wasn’t able to do a picture of a plant adaptation on the Bakersfield College campus. So, always good to test it out before you launch these things, but it’s a great way because students are already carrying their phones with them. It’s easier to turn to their phone than to go back to their computer and to generate it that way.

The other thing we can think about, so strategy two, is we can have students submit chats. This is an excellent strategy that we can use, that we can have students engage with AI as a tutor. So, we know that tutoring has long been established as a really powerful strategy for learning. And, using AI as a tutor, is a really great way to use AI to support student learning. So, research shows that when students engage with AI as a tutor, it can help increase learning. So, we can have students interact with Gemini or Perplexity, both have guided learning options, they can submit this transcript and then we can score students based on, or give students a grade. Or, if we’re doing ungrading, however we want to do it, but we can check students’ learning by looking at how students are submitting, how students are interacting, how deep is their interaction, are they just saying yes or no, or are they going to engage more deeply with AI as a tutor? The great thing about this is you can’t get AI to generate something that looks in this format. So, when I’ve tested this, I’ve been able to create, you know, transcripts like me-you-me-you typical staged transcripts, but it doesn’t show the speech bubbles. So, this would be something that you can ensure that students have completed on their own.

Another thing we can do, and we talked about this in terms of just multimodality, but we can have discussions become multimodal both in terms of what students produce, but then, also, in terms of what we produce. So, for multimodal discussions, we can ask students to create a short video where they’re sharing, you know, their thoughts on a topic. We can have students, again, the image, you know, when we’re having image-based discussions. One of my favorite discussions I do is I have students at the beginning of the semester share a picture of something that brings them joy. And this is a really powerful chance for me to get to see students’ kids, their pets, all the things that make students amazing, and make students driven to successfully complete the course. So that’s another great strategy that we can use. I’ve also been teaching courses on Introduction to AI and Teaching with AI. And in one of those courses, instructors are supposed to redesign an assessment to make it less vulnerable to AI. So, one instructor, instead of changing what students produced, she changed her text-based prompt to a video-based prompt. She was getting really tired of discussions that were filled with AI slop. And so she found that just having the video-based prompt, it was captioned, so it was accessible, changed the game. So after she did that, the discussions were much more authentic because students would’ve had to go through much more steps to be able to offload their learning. She was able to put that potato chip bag in the pantry. We can also think about how we can refine grading criteria. So, this is an excellent time for us to test out our assessments and see what is AI really good at doing, and what is AI not good at doing? What do we really want to make sure students are developing? What are their unique human skills that we want to make sure students are really developing, not just for our courses, but for their futures in the workforce? We know that AI and LLMs tend to struggle with accuracy, so that can be one way that we can refine our grading criteria to focus on unique human skills. And we also know that there’s, you know, voice and things that we want students to develop. So we can refine our grading criteria to really emphasize those human skills that takes testing and playing around with our specific assessments.

All right, last one. So the last one we can think is – how we can rethink reflections? So, as we rethink reflections, we can think about how we can use reflections very strategically in our courses. So, reflections are a powerful opportunity for learning, but we want to make sure that we’re using them when they’re actually measuring the student learning outcome or when they’re helpful for students metacognition. Another course I teach is Introduction to Microsoft Office, and I found that I thought that this course was going to be immune to AI. So, in my discussions I have students complete practices with different, you know, features in Microsoft Office and share their experience. And in one, they had to share their experience using the accessibility checker. So, they had to reflect on their experience. And most of the discussion posts were a few sentences, but there were a few that were paragraphs about their experience using the accessibility checker. And I thought, that’s interesting. So I tried it with ChatGPT and lo and behold I got something very similar. And so, in this case, I just made one small tweak and I asked students to submit a screenshot, and I realized the learning outcome for this assignment was not reflecting on their experience using the accessibility checker. It was, “Can you use an accessibility checker?” And so, I redesigned the assignment to have what students submit show that they use the accessibility checker, and then identify what areas it missed. And so, just having that extra requirement of the screenshot made a really big difference and helped to prevent some of that AI misuse. Okay, so I wanted to leave us plenty of time. I know that we started session by thinking about how we can crowdsource, sorry. How we’ve seen those benefits in action in our online assessments. And now, as we’re thinking about what we shared, I see the chat has been really active with some great ideas and great questions. So, we’ll have some time to answer some questions and also to crowdsource. So, if you want to take a moment and just drop in the chat some different strategies that you’ve used, some small changes that have made a big difference, or, maybe some strategies that have been shared today that you’re interested in adapting or using for your own course. Feel free to drop that into the chat. And Olivia, do you want to do the Q&A?

– Yes, thank you very much Alex. We’ve got two questions in the Q&A one in from Claire. A lot of these strategies rely on current AI capability restrictions. That is, they can’t make a photo on Bakersfield campus, they can’t do speech bubbles, this won’t be a constraint forever as they continue to advance. Will we continually be trying to outrun AI with these strategies?

– Very good question. So, I would say, there’s always going to be- the way that I think about AI is I like to think of it as a calculator. So, my daughter is in fifth grade and she’s not allowed to use a calculator for math. So she is able to, you know, multiply and divide huge numbers. As she gets further, she’s going to be using AI to integrate in specific ways. And so, when we think about the inherent, the tool, what these tools were built, even as they develop, there are some fundamental aspects about these tools that will make them perform in the same way. Now, if we start seeing LLMs that are built not using generative AI, then it will be a different discussion. But, the tools that we’re seeing, are built off of generative AI models. And so, there are going to be inherent shared weaknesses, especially as we’re seeing so much data being used to help to further these models. The most accurate and the most reliable model of ChatGPT, I think, was ChatGPT 2 before they started basically scraping the internet to just get as much data as they possibly could. And so, that’s why no matter what, we’re always going to see issues with hallucinations, and accuracy, because these tools are meant to generate. They’re meant to create new content, that’s what the G in GPT stands for. It’s not meant to be reliable and consistent. And so, even as they develop, they’re still developing on this trajectory, even though it’s exponential, even though we’re able to see them create more and more content, we do see the same issues over and over again. And that hyper-local, hyper-accurate is something that generative AI is not on the trajectory to address.

– Thank you. One from Georgia. How do you get students to share their chat history? Should we ask for every assignment or only certain types of assignments?

– Good question. So, it depends. So, there’s different tools that you can use. You can have students use Gemini or Perplexity, I’ve been testing out the free version and they’ve all been very, you know, sufficient for what students’ purposes would be. Students can simply just right-click and print as a PDF and submit it that way. They wouldn’t need to worry about accessibility. So, when students are generating content, they don’t need to meet the same accessibility standards that we as instructors do. So, they could do the print as PDF and share it that way. We can also, if we’re using a tool like PlayLab. PlayLab allows you to create your own custom bot, so you can upload your own information and you can see students chat, or you can see basically analytics that way depending on if a student is signed in, it may be anonymous or not. But really, the easiest way is to just right-click and print to PDF.

– Okay. Anonymous question here. Some students do not use AI while others love it. It is the future, but right now there are significant environmental concerns where students do refuse to use them. I’m not sure how to create some of these AI assignments with this in mind. Adding in, I should add that I teach to hundreds of students a term, and Karen has written, “not my future.”

– Yeah, yeah, and so I would say one microdesign strategy would be to integrate it as a tutor. The other microdesign strategies are things that will not ask students to use AI. I would recommend including in a getting to know you survey at the beginning of the semester, just where students are at with AI. Because we may be surprised when early on, I think maybe about six months after ChatGPT came out, I hosted with the library workshop for faculty and students on AI, and students were very concerned about privacy, ethics, and those are all really important things to consider. We can also, you know, AI is very interesting to a lot of people right now. So, even if we don’t have students use AI, I think it’s really important that we integrate it into our topics. You know, have students look into the research on data centers and, you know, I know that they’re proposing a data center in our area right now. And the impact of data centers on communities is tremendous. There’s some terrible examples of data centers in South America where people are not able to get clean drinking water and the data center consumes a huge amount of clean drinking water every day. Meanwhile people are having to drink water that is, you know, going to shorten their lifespan. So it’s very, I think it’s really important that we all educate ourselves on kind of the underbelly of these tools and so that we can have really informed conversations and we can help our students become advocates for a society that we want to see in the future.

– Okay, a question from Susan. With strategy four, can you give an example of how you would emphasize human skill? How you would emphasize–

– Emphasize human skill.

– Oh okay. Thank you for telling me what strategy four was. It was, oh, that was a test.

– Okay, so that’s a good one. So I would test it out and see specifically for your assignment. You know, what stands out? There’s been some research that shows that for instructors who are very familiar with their assessments, they can kind of tell AI-generated content. I’m sure many of us have seen it ourselves. And so, you can see like what is it lacking, what’s it missing that’s so important from the student, but, in the case that we’re not able to find something, one of the biggest ones is accuracy. So, I talked about generative AI, it’s built to generate, to create to, you know, come up with new answers. It’s not meant to be valid or reliable. So, when we’re designing assessments, if we focus on reliability and accuracy, that’s going to be helpful. One caveat with that, is tools like Perplexity. Perplexity is much better at being accurate with information on, you know, especially that’s available online. And so that could be something that you want to kind of test out a little bit. And, I know that we’re running out of time, I do want to just highlight the Swiss cheese approach. So, I don’t know if you all have seen the Swiss cheese approach from Rundle at all, but they recommend that basically multiple measures, multiple imperfect measures can help to support academic integrity. So, each of these measures alone, there’s no golden, you know, golden ticket that’s going to make every assessment not AI-vulnerable, but if we stack together multiple layers, then it can help to, you know, move that potato chip bag. Just like, you know, just closing the potato chip bag on the couch is not enough. We have to close it, we have to put the lid on, we have to like put it in the pantry, maybe we have to put it in the garage. You know, there’s multiple layers that we need to be able to stop. Eat a healthy snack, you know.

– Okay, last question. When you require screenshots or transcripts, how do you incorporate that into the grade, if at all, and what’s your reasoning behind that choice?

– So, for that example, in working with students on the various grants that I’ve been doing, I’ve been talking a lot with students and the biggest thing that students want is more opportunities for practice. So this can be a really great way to integrate more opportunities for practice where we can kind of monitor their learning and make sure that they’re getting where they need to go. So, you know, and if we’re teaching a STEM course, it could be a great way to integrate that practice and, you know, have a check. That could be another opportunity for a meaningful reflection in which students can share, you know, how they’re using that. How they’re doing that for learning.

– So I know time is going and people may have to leave, but we have two more questions, so if you’re okay, I think Alex we’ll handle the two questions and anyway people can go back to the recording. So, from Samantha, students’ interest in back start buy-in in the course topic seems to play a large role on whether students will offload their critical thinking to AI. What strategies would you suggest for instructors who teach gen ed courses or courses to non-majors where they see low student engagement and high AI use?

– Yes, so Swiss cheese approach is going to be helpful for that one. You know, I think that… one of-the one thing that we can do is in those surveys at the beginning of the semester, asking students, you know, where they’re at can go a long way. So, I’m thinking about when my kid was a toddler and she was having tantrums, I would say, what do you need? Like, and oftentimes she would come up with things that I had no idea, I didn’t even think of, like, you know, she needed to play with me for 10 minutes instead of whatever we were doing. So, that could be something like, what do you need to make this course interesting? Like, you know, how can we really connect this to your lives? I do think, you know, assignments like the joy, that joy assignment, that is huge because students start to see how this course connects with the things that drive them, the reason why they’re there. Being able to get a job where they can provide healthcare to their kids. Another great example are those short opportunities for students to connect things to their courses, sorry, to their lived lives. So, I was working with a math instructor and he has students go around, you know, he’s teaching like intro to like basically a gen ed math class at a community college. But he has students go around and take pictures of, you know, different angles they see and name the angles, and share that. And it’s a really easy way to integrate and get some, you know, students are able to show things that are interesting to them. So, different ideas.

– Okay, I’m going to do the- this will be the last question and, but, you know, it’ll be recorded and so people can, and we’ll make sure all the questions are answered. So, this is from Colleen? Yes, Irish name. It’s the Irish name for girl. I’m having a hard time with exams and homework that are not being recorded with videos. This has been an issue and I don’t have time to watch 70-plus videos each week. I’ve pulled exams down to two recorded per course, but they still use AI on the homework and then fail the exam.

– Yeah, yeah, yeah. Oh my goodness. And, you know asking students to do videos, it’s nice in theory but, man, you can get slammed with grading ’cause it takes a lot, a lot longer. So, some different strategies that you could do for that. The sky’s the limit with your creativity. So, if you are able to revise your assessments, you know, instead of having a midterm paper or, you know, an exam, if students can have a discussion, you know, maybe you have students sign up for Zoom-based discussions and the group is Tuesday mornings at nine, Thursday night at 10, or you know, you come up with different times and students sign up for the groups based on the times they’re available. Then they can have a Zoom-based discussion where they talk to each other and then you can grade that one. So, then effectively, instead of having 70, you can grade 10. If you have groups of seven, that can make it a lot easier and it gives students an opportunity to interact meaningfully with each other. Goes back to, you know, oral exams and providing opportunity to for that. And, Mary, 1.5 speed is always helpful. Yes.

– I’m just going to say, Alex, thank you so much. There’s very, very informative. These comments are coming in on the chat and, I would say, there’s a huge amount of content in the chat as well. So, a big, big, thank you from us at OneHE and everybody here.