What is a cognitive lab and how does it work? WIDA Researcher Mark Chapman explains

A lot of work goes into developing an assessment. At WIDA and the Center for Applied Linguistics (CAL), there are teams of language testing experts dedicated to managing all stages of the test development process; from preparing new test content for field testing to measuring the validity and reliability of test content. (Read our recent article: Everything you’ve ever wanted to know: ACCESS test development)

One important component of the test development process is a cognitive lab. Cognitive labs are a research method that WIDA and CAL use to collect detailed information about students’ test-taking experiences. Cognitive labs involve observing student behavior, hearing directly from students and educators and are particularly helpful when trying out a new kind of test question or a new way for students to engage with a test.

During cognitive labs, WIDA and CAL work with students at different grade levels and at varying levels of English language proficiency to understand the impact of all test changes on a diverse group of students, including those with Individualized Education Plans (IEPs).

Last fall, WIDA and CAL staff visited schools in Indiana, Virginia, Colorado and Tennessee where they spoke with students, teachers and some school leaders. The goal of these visits was to explore a potential change in how students respond to ACCESS for ELLs Online Speaking tasks — specifically how the students react to having two chances to respond to a task.

Mark Chapman, a researcher at WIDA, and Shauna Cannon, a language testing specialist at CAL, visited Pioneer Bilingual Elementary School (Escuela Bilingüe Pioneer) in the Boulder Valley School District in Colorado to conduct one round of these labs.

“Shauna and I had a lovely time at Pioneer,” Mark said. “We were warmly welcomed by Heather Riley and Sandy Sirepisios who helped facilitate the activities with the school’s second and third graders.”

Seven educators and 17 students participated in the ACCESS for ELLs Online Speaking Cognitive Labs.  

“There are different ways of conducting cognitive labs,” Mark said. “For this cognitive lab, one person administered two different Speaking tasks to a student while the other person observed how the student engaged with and responded to the tasks.”

Mark and Shauna looked for things like  

• whether the student was confused by having two chances to respond
• whether the student chose to say more in response to the second opportunity to speak; and if so, whether the student produced more original language compared with their first response

Each student’s teacher was also present and observed the student responding to the Speaking tasks. After administering the Speaking tasks, Shauna conducted a short interview with each student. Interviews were conducted in English and Spanish.

“We focused our interviews on why the students chose to respond once or twice to the task, their opinion about having two chances to respond and how they thought this compared with the one opportunity they get now,” Mark said.

Following the student interviews, Mark and Shauna interviewed teachers to get their feedback on the new Speaking task design.

“We really wanted to hear from educators about how they thought their students had performed when they had two chances to respond,” Mark said. “And we also wanted to know whether they thought this new design feature might help their students do better on the test.”

These cognitive labs were recorded. Now, WIDA and CAL researchers will review the data — including the observation notes and audio recordings — to learn more about the impact of introducing two chances to respond to Speaking tasks.

“I’m excited to learn from the data so that we can better understand whether this change will allow students to better demonstrate their language skills and have more of an opportunity to show what they can do,” Mark said.

From Mark, Shauna and everyone involved in the ACCESS for ELLs Online Speaking Cognitive Labs: “Thank you to the wonderful students and teachers who participated! Next time you see an opportunity to participate in a WIDA cognitive lab, think about signing up! We would love to work with you.”

Learn more about cognitive labs and the test development process on the Building a WIDA Assessment page.

Mark Chapman is the senior innovation researcher at WIDA. His work focuses on researching innovative assessment practices to enhance language tests for young multilingual learners. Mark also conducts research into the validation of Speaking and Writing assessments for young multilingual learners. He publishes his work in journals and edited volumes and regularly presents at domestic and international conferences.