Three young adults sitting on yellow seating in a hallway, looking at a laptop.

ASU students debate with famous philosophers using generative AI

Stephanie King

July 25, 2024

With generative artificial intelligence (AI) tools becoming increasingly available to students and statistics showing a future workforce that’s automated by AI, David McElhoes knew he had to adapt his Introduction to Philosophy of Science curriculum.

“Experience with AI is going to be necessary – if it isn’t already – in the workforce,” said McElhoes, a instructor in Arizona State University’s (ASU) School for Historical, Philosophical and Religious Studies. “If you don’t have those skills, you’re going to be at a disadvantage.”

The course averages between 75 and 100 students each semester, with a wide range of majors, including philosophy, psychology, engineering, data science, physics, computer science and more. This broad range of study also carries a range of experience for students using generative AI. McElhoes’ goal is to bridge the gap, encouraging students to explore the use of generative AI to enhance learning.

So McElhoes developed a series of assignments encouraging students to use generative AI while advancing the course objectives. As part of ASU’s AI Innovation Challenge, McElhoes is experimenting with prompt engineering to ensure that students can simply copy and paste prompts into ChatGPT.

AI-enabled assignments create interactive debates

A series of assignments, first offered by McElhoes in spring 2024, puts students in the role of a major philosopher they are learning about in the course. The idea is for students to interact with generative AI tools, such as ChatGPT, to have a critical discussion or debate with a rival philosopher from around the same period.

For the first assignment, students take on the persona of Karl Popper, one of the most influential philosophers of science in the 20th century. The fictional assignment is set in the 1930s on the steps of the University of Austria in Vienna, where Popper engages in a heated debate with another famous philosopher, Rudolf Carnap.

“Debate is a very big part of philosophy,” McElhoes said. “The AI is up to speed on everything that was going on then so it’s an engaging exercise where there’s a nice back and forth [between the students and the AI].”

The assignment is provided to students as a prompt for them to enter into their generative AI tool. Here’s a look at the first few lines of the written prompt students will use to begin the assignment:

I would like to play a game. You pretend to be Rudolf Carnap. I will pretend to be your nemesis, Karl Popper, and you will address me as ‘Popper’. We are having a debate on the steps of the meeting hall where the Vienna circle is about to convene. Your tone, as Carnap, is always respectful, humble, and polite. Several of our undergraduate students are observing our exchange. As Popper, I aim to convince you and your students to weaken logical positivist doctrines so that they better match what is now known as logical empiricism, and to defend Popper’s views from your criticisms.

The initial prompt provided by McElhoes is detailed, totaling 679 words that describe the scene, the character and the assignment. The prompt continues:

The game begins with you providing a quick description of the scene, followed by you asking, “Good day to you, Popper! Tell me, what do you find problematic about logical positivism?” You will await my response to you. I will attempt to deliver a compelling criticism of the logical positivist view. You will then attempt to rebut my criticism as Carnap would. Each rebuttal you provide to my criticism ends the round, and you will then give the user the option to offer another criticism of logical positivism, or to defend Popper’s philosophy from a criticism offered by Carnap, or to end the game.

Students work in teams to take on the role of Popper, taking turns articulating what they have learned about logical positivism and then trying to challenge it from Popper’s perspective. The entire exchange takes place within the generative AI chatbot, in this case, ChatGPT.

For example, ASU student Ava Dominica Gilormini Lapham, playing the role of Karl Popper, responds with:

Well, Carnap, briefly, logical positivism is the radical version of empiricism that uses insights on language to solve problems with philosophy through the acceptance of verificationism, the analytic vs. synthetic distinction, linguistic frameworks, and a logic for induction. It is this final point, logic for induction, that poses a key problem I have identified with logical positivism.
The problem with logical positivism is that it relies so heavily on confirmation or positive confirmationism; however, confirmation cannot be justified. This is because induction is not justified, as derived from Hume’s Inductive Skepticism. If we justify generalizations of “All Fs are Gs'' through confirmation, it has to be deductively or inductively justified. Induction is not justified because of Hume, and we cannot justify these generalizations deductively either since there is always an imagined or possible scenario in which not “All Fs are Gs.” Given this, confirmation cannot be used to justify scientific generalizations of the form “All Fs are Gs.” Confirmation cannot count as evidence or scientific knowledge.

The GPT responds in the role of Rudolf Carnap:

Your criticism, Popper, indeed raises important considerations regarding the justification of scientific generalizations within the framework of logical positivism. It prompts us to reflect deeply on the epistemological foundations of our philosophical positions. However, allow me to offer a perspective from within the logical positivist framework. While acknowledging the challenges posed by Hume's skepticism towards induction, logical positivism maintains that empirical evidence still plays a crucial role in supporting scientific hypotheses.
While it's true that induction cannot provide deductive certainty, it does offer a pragmatic basis for inference, allowing us to make reasonable judgments based on observed regularities.

McElhoes walks through these arguments with students in class so they are prepared for the assignment: “They’re put in a position where they’re getting a dynamic response from generative AI that’s surprisingly good,” McElhoes said.

Each student criticizes, defends and responds to earn points for how many of the rival’s students they convinced to join Popper’s side during the debate. At the end of the game, the points are tallied, and the generative AI creates an image of the fictional scene at the conclusion of the debate.

McElhoes finds that the students are drawn to the gamification of the assignments, even replaying the game several times. “They’re motivated to do this on their own,” he said. “They’re getting AI feedback, which is pretty good, and as they keep playing, they get better at it.”

During the course, there are a total of three assignments that use generative AI. These assignments task students with practicing how to explain complex philosophical ideas clearly, target the objective well and articulate a cogent argument. They also match the exams McElhoes gives his students, in which they’re given an argument that misconstrues the material, and students must share two reasons why it’s a bad argument.

What McElhoes appreciates about this curriculum is that the students gain a new perspective and way of thinking. “Just seeing the improvement on the final exam is my favorite thing about the class,” McElhoes said.

Testing ChatGPT to maximize student learning

While McElhoes noticed significant improvement in students’ performance from the first assignment to the second, he also discovered that students didn’t immediately apply those skills on the exam. So, with access to ChatGPT Enterprise through the AI Innovation Challenge, McElhoes is tweaking the assignments so that the prompts are better. He updates the exam to ensure the assignments and the exam are more closely matched, making it more obvious that students’ skills are moving from the practice setting against the AI to the exam.

McElhoes also shared a few aspects for educators to be mindful of: first, recognize generative AI’s limitations. His first incarnations of the game – which included the creation of a custom GPT – were too complicated, and inaccurate scoring was an issue, as well. His advice? Keep it simple, and the AI will respond well.

Additionally, McElhoes mentioned that ChatGPT Enterprise is not great at assessing students’ work yet. “So it’s not going to replace graders,” he noted.

“It’s just as good as a textbook, as far as a pedagogical aid, as long as you use it right,” McElhoes said.

While McElhoes has allowed students to use their preferred generative AI tool iterations, he will require students to use ChatGPT 4.0. “ChatGPT is, at least from the perspective of this assignment, the superior AI as far as pre-training has gone,” McElhoes said. “It’s like someone at OpenAI has my course notes or has taken my course and has pre-trained the GPT with all the different perspectives that I talk about in my class. It’s perfect for my class.”