Since OpenAI’s ChatGPT became publicly available in November 2022, the field of higher education has been focusing on its impact and applications — faculty want to understand how this will shape their work and the student experience.
Largely missing from many conversations, however, is a discussion of how scientific approaches may be used to study ChatGPT and other generative AI tools in the context of higher ed. With the technology itself evolving rapidly, establishing a framework for examining its implications is critical; we need to know what questions to ask, and to continue asking, even as the answers continuously change.
At Columbia University’s Science of Learning Research Initiative (SOLER), our work is dedicated to examining the academic experience of our students and instructors through a scientific lens. Doing so involves leveraging research rooted in the Scholarship of Teaching and Learning (SoTL) — a systematic inquiry into student learning to improve teaching practices — and analyzing insights that have been drawn from academic and institutional data. The goal? To advance the teaching and learning experience.
Our team has begun engaging in research related to how students are using generative AI tools and we’ve learned that we need a systemic approach to researching the impact of these tools over time so we can better understand how to leverage them. Here are three methods our team has been using.
Observational Research
At SOLER we’ve been conducting observational research to get a better sense of the existing habits, understanding and attitudes our students and faculty have about generative AI tools. Much of the discourse about generative AI in higher ed has focused on issues of academic integrity. To inform these conversations, observational research — without intervention — is the necessary foundation. Our researchers aim to ascertain what students and faculty know about the technology, how often they use it and for what purposes, and how they view its usefulness or appropriateness in various academic contexts.
Some of our key observational methods include anonymous surveys and focus groups, which offer “safe spaces” where students can be forthcoming about their habits. We’ve found that collecting this information is crucial to properly support faculty, who have a great need to understand their students’ behaviors and attitudes. Our instructors have questions about retention and academic success — they want to understand how the use of these technologies relate to student outcomes. Our efforts to analyze data have helped us shine a light on these issues.
In the coming academic year, SOLER will partner with faculty in Columbia’s Graduate School of Architecture, Planning and Preservation and the Office of Academic Integrity to examine student attitudes about the use of ChatGPT. The investigation will serve as a starting point for a study that will ultimately test the tool’s impact on student learning in a real estate finance course, which brings us to our next research approach: true experiments.
True Experiments
True experiments are a critical research methodology because the sample groups must be assigned randomly between control or experimental groups, and all variables except the one being studied are controlled, in order to best determine causality. We’re designing true experiments that explore prescriptive questions about the ways the technology should be deployed as an instructional tool — this is a key element of advancing teaching and learning in higher ed. When it comes to investigating generative AI tools through an SoTL Research framework, essential questions combine elements that are discipline specific with more general considerations of the student experience.
We believe true experiments on ChatGPT should be designed to address two major areas:
- Experiments should be incorporated into assignments, especially in the context of writing papers and computer programming, and should examine questions about student motivation, assessment, revision processes and academic integrity.
- Experiments should examine how “AI tutors” provide personalized feedback and explore the impact on learning and attitude-related outcomes for students, and how these outcomes compare to those achieved with more traditional resources.
Hybrid Research
A third core approach is implementing hybrid research that examines how students opt to use the technology when given explicit access but limited instructions. This method combines elements of the above approaches and fills a conceptual gap by addressing the following question: when given access to the technology but limited guidance, how do students choose to use it?
Observational research entails simply encouraging students to use the technology in a given class and then asking students to report on their usage. A true experiment might involve establishing two conditions in one curricular context, such as two sections of the same course given the same assignment. In one condition, students receive limited instruction; in the other, students receive specific guidance on how the technology should be used in the context of the assignment. Using a combined technique with this structure in place, a researcher could examine whether the two groups exhibit different patterns of behavior, learning outcomes or attitudes.
Along these lines, SOLER is currently developing a project in collaboration with faculty at Columbia Business School that will explore how groups of students reach consensus about using AI image generators. Our goal is to understand how the patterns of usage shape the interpersonal dynamics of the group members.
As the field of higher education finds itself navigating this rapidly changing technological landscape, adapting is our only option. We must make systematic and rigorous efforts to understand and leverage new technologies — and we must seriously consider ethical and moral questions, especially ones that pertain to diversity and inclusion, like who benefits from these tools, and why?
These complex issues can be meaningfully addressed by taking a scientific approach, using robust research frameworks, and with institutional support for these efforts. If we examine how students and faculty are experiencing emerging technologies through a scientific lens, we can achieve more than just keeping up — we can map out a path to a brighter and more equitable future.
