UX Research: ChatGPT
Investigate students’ mental models of ChatGPT and how these influence usage and learning
Objective
Results show that incomplete mental models lead to distrust, struggles, and fears of overreliance
Result
UX Researcher in team of 5
My role
Scope
Mars - June 2024 (University project)
01/Context and Research Process
Background
Since its launch in 2022, ChatGPT has become a staple in how many students work, study, and create. Statistics show that roughly 60% of students use it for everything from finding information to developing ideas. Yet, at the time of this project, little was understood about students’ perceptions and mental models of AI tools such as ChatGPT. Understanding these concepts is essential to facilitate design decisions that support students’ ability to learn and use AI tools efficiently.
ChatGPT is essential in students´ workflows, yet students perceptions and mental models are poorly understood.
Research Planning and Preparation
The UX team and I began by asking, ‘What do we need to learn and why?’ We then narrowed the focus areas, defined participant eligibility, and selected the research methods that would provide the most relevant and actionable insights.
Research Goal
Understand students’ mental models of ChatGPT and how these influence information retrieval.
Participant Profile
University students with active experience using ChatGPT for academic purposes.
Methods
A combination of think-aloud sessions and semi-structured interviews was used, a pairing proven effective for uncovering users’ mental models.
Session Setup & Recruitment
The sessions were prepared with detailed protocols and carefully crafted interview questions, refined through iterative review, brainstorming, and pilot testing.
To find the right participants, we developed a targeted recruitment strategy, reaching out through social media and direct email to connect with those who matched our criteria.
Understanding the Data
The data analysis was approached with the goal of uncovering the real stories behind the data. Using the AI tool Revoldiv, we transcribed each session recording, capturing not just words but also movements and visual expressions. Using an open coding approach, I created a custom coding template that marked each code with its time stamp and participant origin for full traceability. Thereafter, the team and I distilled the codes into representative themes to uncover the students’ mental models.
What Did Students say?
Believable, But Not Enough
“I trust it when it matches what I know, otherwise I always double-check”
Trust in ChatGPT was rarely given freely. Students cross-checked its responses against their own prior knowledge, often guided by a gut sense of whether something “sounded right.” The lack of clear references was a consistent barrier to full trust, making independent fact-checking part of their normal workflow.
Afraid of overeliance
“There is a risk that you stop thinking a little for yourself”
Students worried that relying too much on ChatGPT could harm their learning. They feared losing critical thinking skills, becoming less personal in their work, and even being misled by incorrect information. Misinformation, plagiarism, and cheating were common concerns, alongside the belief that some topics simply require real human interaction to truly understand.
How It Works? Not So Sure
“I am not realy sure where the data comes from, I guess the Internet?”
Most students weren’t sure how ChatGPT actually works. Some called it a robot, others an algorithm, and many assumed it just “pulls from the internet” without knowing how. The lack of real-time data left them puzzled, and they often couldn’t explain why it had this limitation. This made students feel unsure and distrustful in some situations.
Don’t Know How to Talk to AI
“I think how you write influence the answer, but I dont know how”
Students understood that prompts shaped ChatGPT’s answers, but few had mastered the skill. Most borrowed prompting habits from Google or other AI tools, applying them with mixed results. Even when they tried to add context or keep questions concise, answers were often too long or off-target.
02/From Research to Solution
After we looked at how students interact with and talk about ChatGPT, it appears, as often is the case, that their mental models are partial. They understand that how you prompt is important but not how to do so and they struggle to understand how the tool works and where the information comes from. Often they assume that it works similar to tools they used in the past such as search engines.
What Do the Findings Say About Mental Models?
Design Opportunities
By incorporating elements that increase transparency of how the AI functions and where the information comes from, students may become more empowered and able to utilize ChatGPT in academic work. The UX team and I placed forth the following concrete design recommendations.
Help students understand how ChatGPT works by implementing tooltips or access to a side panel to display its inner workings.
Make it easier to understand how to prompt by including prompting tips, premade prompts, or making it visible how the AI “reasons”.
Give students a way of understanding where information comes from by providing a summary of source origin, actual references, or further reading quotes.
Research That Predicted the Future
his study was carried out in 2023, but since then, AI tools such as ChatGPT have advanced a lot. Through the time I have amused myself with keeping track of how the design has evolved to identify if the UX/UI has changed to better facilitate accurate mental models and improve the tool efficiency, keeping my research findings in mind.
It has been fascinating to see AI leaders like ChatGPT adopt design solutions that echo the findings and recommendations from our study.
Examples in Today’s AI Tools
Chat GPT
Built-in references turn answers into traceable, verifiable information for students.
LM Studio
Thought previews reveal how the AI reads your prompt and guides better prompting.
Mistral
Confronts the fear of overreliance by making students reflect, not just consume.
Final Reflection
This project reminded me that UX research isn’t only about usability, but about shaping how people think with technology. Seeing AI tools adopt features that echo our findings has been incredibly validating.
More Projects
Creating a user-centered solution to simplify household chores through extensive user research and participatory design methods.
Task Ease
From project concept to live product: conducting UX research, prototyping in Figma, and designing for hundreds of users.