– Okay. Hi everyone. I’m Niya Bond, the Faculty Developer here at OneHE. And I’m so excited to be joined today by Carl Whithaus. Carl is the professor and director of University Writing Program at the University of California, Davis and is here to talk to us today about the PAIRR Project. Now Carl, can you tell the community what that entails?

– Sure. So the idea is peer and AI review and reflection. And it’s really a project that’s about how do you integrate both peer review, and then AI feedback into the writing process for college students. And I can say a little bit more about the importance of reflection in that, and thinking about how you take feedback both from AI engines but also from peers. You sort of triangulate those, and then make decisions about how you, as a student, are going to change your writing to get what you want to achieve out of it. And so it’s not letting the AI engine drive what you’re doing, but it is thinking about current writing technologies and how do we give students agency, how do they have critical reflection and ethical use of AI in their writing processes.

– That’s amazing. And you know, research shows that peer interactions like peer review can deepen engagement and enhance motivation. I’m wondering if you could tell us a little bit about how those two things came together. So peer review and kind of AI collaboration combined.

– Yeah, so it’s really interesting. In November, 2022, like everybody else, we were, you know, aware of writing technologies, but ChatGPT 3.5 sort of crashed into higher education, K-12 education. And at Davis, part of our response and the group of faculty that I work with, we didn’t want to be just go back to blue books because I mean it’s like trying to write without word processing in the 1990’s. Like it’s here, you’re gonna have spelling-grammar check, how do you incorporate it? But how do you still have the construct of writing? How do you have knowledge about writing? How do you grow that so students are rhetorically aware making decisions. And so we’ve decided to focus on how do you use feedback from AI? How do you teach students to incorporate it in critical and ethical ways? We tested it out in 23, 24, originally with 3.5 and then with 4.0 by the time that had come out. But we wanted to do it in ways that weren’t just dumping students intellectual property back into OpenAI’s model, but had some privacy protections.

And so what we did in summer of 24 is we applied for a California Education Learning Lab grant and we partnered, not only were we thinking about UC Davis, but we partnered with three California state universities. So Sac State, Cal Poly, Solano, CSU Bakersfield, and then four community colleges, American River, College of Marin, Glendale, and LA Mission College. So all across the state and the eight campuses faculty at them got together and we submitted to the California Education Learning Lab and they awarded us funding. We were one of five large projects funded and we have two years. So in 25, 26 and 26, 27, we’re gonna be working with faculty at all of the eight campuses. It’ll end up being around 115, 120 faculty, probably 12,000 to 17,000 students. We’ll be using the PAIRR model, but we’re also going to be studying how folks are writing in classes where there is no PAIRR intervention. So they’re using AI or not using it depending on just what happens. And then we’re going to be reporting on that both the professional development that faculty are doing, but also how does the PAIRR model itself work? And that’s the research side of things.

And I think one of the things that might be really exciting for you all is we’re also developing it as all is open educational resources. So part of the California Education Learning Lab’s mission is create open educational resources. You’re not just creating, you know, private things that you then go out and market, but how do educators in California community colleges, CSUs, UCs, how do they get access to it? And then since it’s open educational resource, really educators around the country around the world will also have access to the PAIRR materials.

– That’s amazing. What a great resource and what an inspiring collaboration across so many different institutions and organizations.

– Yeah, it’s really fascinating. The collaboration process has been, I mean it’s the eight site leads, but it also is faculty members who are taking part in professional learning communities. So in fact this summer we’ve been meeting with faculty at each campus and folks have been thinking about, okay, here are the prompts, you know, from the prompt library that you can use. Here are some recommended readings that you can incorporate in classes. Here are responses to those. Here’s the overall flow of PAIRR as a model that you can incorporate into your class in terms of the software, in terms of the responses to readings. And then all of this is available both in, we have the PAIRR packet, which is basically a Google doc that’s accessible to folks. We have a Canvas Hub, we have a Canvas Commons that people can go to. And then you can pick and choose and use as much of it as you want or as little of it as you want.

Obviously we’re not studying everybody who’s using it. We’re only working with, you know, the eight different campuses in terms of the research side of things. But one of the things we felt very strongly about is that as we’re developing the materials and letting them get out to people, that we make them public right away. It’s almost the model of like open source software development. We’re sharing these through public webinars. We really want people to play and use it. And then we’re gonna have an iterative process of how do we refine and make it better? And also how do we share a prompt library that’s prompts that are tested by multiple people so that instructors can use those with whatever AI engine they’re using. And I can tell you a little bit more about where we are. We’re using the platform MyEssayFeedback now, but in many ways PAIRR is platform agnostic. So you could use it like we originally ran it with ChatGPT 3.5, and 4.0. We’ve used Claude in 24, 25, but it doesn’t matter what particular chat bot or software you’re using, you can use the PAIRR approach across different platforms.

– Yeah, and I like that you all provide kind of like a baseline process, but then educators are empowered to kind of use it in their context, and as you noted, pick and choose, you know, how they want to integrate it or when. Could you talk a little bit about the specifics of the PAIRR process just for those who aren’t familiar with it yet?

– Yeah, so a PAIRR starts out and there’s a number of recommended readings. We have sort of four different areas. We ask folks to do a reading that focuses on reintroducing AI. So how do you think about it? How do you use it in critical and ethical ways? So there’s some short readings about that. We also introduce readings about language bias and then also about linguistic justice and equity. So thinking about problems with large language models, the way that they tend to standardize, the way that they will correct language in ways that don’t necessarily allow students to hold onto their voice or to make choices about what they’re trying to do. So students can make choices to have a more standardized voice, but they can also want to hold on to things that are closer to them. And we want them to be aware of that in making deliberate choices. And then we have a fourth area of readings that focuses on ecological or environmental impacts of AI. And we very much want students to be conscious of the energy use, the water use that’s going on. It’s hard to get definitive answers about the consequences of this around AI, but we want individuals to make a choice and students could very well opt not to use it at all because they see environmental harms coming from it. But we want folks to be knowledgeable and aware of the issues that are going on around AI as they use the model.

So there are those readings, there are discussion-set questions, but then we get down to sort of the second step of the process where students are drafting their essays and then they go and do a process of peer review where they’re getting feedback from at least two different peers about their essay. And then the third step in the process is the feedback from AI. And there we have two different base prompts. One that’s a reader based, one that’s criterion based. Instructors can choose from those. We’re ultimately developing a library of prompts that give students guided feedback and students will see that feedback. And the question is, we then have a reflective activity where students compare the AI feedback and their peer feedback and they do that initially after getting the feedback from the AI and peers. They’re giving their response to it and they’re thinking, you know, here are the pluses and minuses that are given with both of those. And then they go and they’re finalizing their essay and we ask them to have a revision memo that reflects on what pieces of the feedback did they use.

So there’s two moments of reflection here in the PAIRR process. One is right after you’ve gotten the AI and the peer feedback, right? So you’ve gotten feedback from two different peers, you’ve gotten feedback from AI, what do you think is the AI feedback? And we’ve heard so far often like the AI feedback tends to be more directive, tends to be here let’s fix these particular things or change these things. Some of that’s tied to the way we do the prompts where there are particular suggestions that have asking the AI to provide students, but peers often tend to have more situational knowledge, tend to understand the assignment or other things that aren’t made explicit in the assignment or the rubric. But you know, from just being in class and peers will mention those things. And there’s also a human element where that feedback involves motivation. But students reflect on those and it’s important that they’re doing it soon after they’ve gotten feedback from these sources ’cause it’s like, oh, what’s working and not, we want them to sort of articulate what’s valuable, what do they think about it. But then also then they go back and they’re revising the essay and they articulate in this revision memo or header to their paper, what did you actually use? And in some cases, what do you actually use two or three weeks down the line when you’re revising a paper is different from, you know, what you felt immediately. So that’s an important process and then that gives them an opportunity to reflect and look back longer. So that’s sort of the PAIRR model in terms of the curriculum.

And the cool thing about that we found is if you’re doing this like in a first year composition class, you, as an instructor, might make PAIRR into, not PAIRR but AI and AI and writing, into a major theme of the course. And so you could use the four recommended readings or the required readings in the video or you could, we have like three or four other different sources. You could build it into a major theme for the class. At the same time, we’ve used PAIRR like in environmental science courses, we’ve used them in plant science courses at Davis where they’re large lecture courses, you’re doing one writing assignment, but like AI and writing is not gonna be a theme for the faculty member or the TA leading the discussion section. It’s just one little tiny thing in a course about like how coffee is grown, right? But it’s there, it can be integrated and it’s part of sort of a writing intensive or writing in the disciplines course where you can use a thin version of PAIRR and have students work on it while they’re working on advanced writing within their disciplines. And so that would be sort of my synopsis of here’s how PAIRR as a curriculum model works. And then there’s interesting things of pre and post surveys and focus groups and all that, which is just the research side of the project.

– Well I love that and I’m so glad you gave examples of different educational contexts because I am an English and writing educator myself. You know, in smaller courses where there is kind of more time to do that reflective work, but as you mentioned, maybe in a large lecture course this can still be integrated even in a smaller way. And I think that’s so important to consider across different disciplines. Now regardless of what educational context someone’s coming from, you have tons of resources, you’ve mentioned some just now that educators can come and look at and use. Can you tell us more about where those can be found and what kinds are available?

– Oh yeah, I’d be happy to do that. So I will drop it into the chat, but I’ll also go ahead and say it. So the base URL to find out information about the PAIRR project is Writing.UCDavis.edu and then it’s the slash and it’s PAIRR, although the trick here is it’s pair with two Rs P-A-I-R-R, so it’s peer AI review and reflection. So give us the double R’s, but if you go to that website, you’ll see about the project, the people involved. But the tab that I would really click on would be curriculum resources. And there you will see links to the PAIRR packet. You’ll see instructional materials. We will soon have links up to a Canvas Hub and a Canvas Commons, which will provide materials within those learning management systems for folks. But if you click on the PAIRR materials packet, which is the first link under the PAIRR instructional materials at that website. You will get access to a Google doc, which contains the AI literacy readings. We have some for students, we have ones for faculty members. Gives a overview of the reader response and the criterion based feedback prompts, which are the two main models that we have. We will be adding to it as we test things with faculty and develop a library of prompts that’ll be accessible here.

The other thing with the packet is it provides access to the common rubric to student reflection assignments. So it’s really a set of curricular resources that a faculty member can turn to and use. And as I said, the faculty at the, even the faculty at the eight different schools that we’re working with right now, people are drawing on different phases of this and we’ve got four elements that we’re requiring everybody to do. But if you are just coming across this as an open educational resource and you’re playing with it, it’s really how do you adapt this and make it useful for your class? And I think part of the thing that’s neat about this is it can be used, as I said, in writing intensive, you know, disciplinary courses. You could use a thin slice of it, but if you’re teaching first year composition or advanced composition or another writing course, even one where you wanna have a theme of writing and AI, there’s a lot of materials here that are tested by folks that you can then incorporate both sort of content as well as process of writing.

– Yeah, well I’m really excited to follow your journey and keep up with all of the releases that you’ll be having as I’m sure many in the community are. I know this is a long-term kind of research project, but will you be releasing information as you go? I know you’ll be releasing resources, but how will you be reporting out?

– Yeah, so we’ve been holding a number of public webinars. So we’ve held two this summer. We will continue to do that and those will be announced through our website and also through the California Education Learning Lab website. The Learning Lab is also putting together a collection of publicly available resources, not only from the PAIRR project but also from other, both AI and just general education projects that they funded throughout the state of California. And so I would check back to the website that I mentioned are also look up California Learning Lab and look at their publicly available sources. And that’s where announcements will come out.

– Amazing. Well, we always like to leave the last word to our experts, so no pressure. But if there was one final tidbit you could leave the community with about PAIRR, what would it be?

– Yeah, I think my final tidbit would be PAIRR is a model where we’re trying to think and work with educators on how do you use this technology that is reshaping writing in ways that help students become critical and ethical users of AI. And I know there is concern about any use of AI for writers, but I think PAIRR provides us a model that embraces the technology but does so critically. And I don’t think it’s possible for us to go back and write or educate, you know, work with students in ways that you don’t use AI at all. But I do think we wanna make sure that students aren’t simply doing cognitive offloading that ends up hurting their learning and depriving them of agency. I think we want them to know how these tools work and be able to make decisions. And there may be moments where you don’t wanna use AI, but if we as educators don’t engage with it, then we’re cutting our students off from learning. And I think PAIRR provides a way of learning critically and ethically about it. The other little…

So I’m adding two tidbits here. The other thing that I would say that’s cool about PAIRR is it really is open educational resource development. And it’s the type of thing that we’re not only working with faculty at eight colleges in California on, but really want to get feedback in terms of the public webinars, in terms of the Google site. We are around on Bluesky and X and in fact, and we really are taking feedback from people and trying to incorporate that into the next iterations. I’m really excited about the prompt library that we’re building. We have reader based one, so we have two, I suspect we’re gonna have 20 or so that are tested in multiple different ways by many, many faculty. And then we’ll be able to give other people a good base to work off and if you play with it, let us know about it. Let us know what works, what doesn’t work, and we will be refining them and continuing to share them going forward.

– That’s amazing. And it’s inspiring too to have that kind of global reach for conversation and collaboration because, you know, even with skepticism, I think so many are curious about these technologies and how and when and if and why they’re gonna fit with modern education. And so I’m just really grateful to have chatted with you today about this project and to see where it goes.

– Excellent, well thank you for taking the time to talk with us.