What is pilot testing? A WIDA staffer finds out

By Katie Stenz

I’ve been communicating about WIDA assessments for several years now as a marketing specialist. You may think that means I know everything there is to know about test development, but I don’t – not by a long shot.

This spring, WIDA held pilot testing as part of the redesigning Kindergarten ACCESS project. When I found out about this work, my first thought was, “Yay, I get to work with educators around the consortium!” My second thought was, “I don’t really know anything about pilot testing.”

My curiosity set in (I knew that at some point in my career I’d finally make my journalism professors happy):

• What is pilot testing?
• Why is pilot testing important?
• What does administering a pilot test look like in practice?

So, what is pilot testing? The goal of pilot testing is to collect student performance data on redesigned or new test items and tasks. It is one of many essential educator reviews that test content must go through on its way to becoming a finalized assessment. Pilot testing typically happens after this redesigned or new test content has gone through educator-powered test development events like bias, sensitivity and content reviews and cognitive labs.

Why is pilot testing important? Pilot testing gives us much-needed feedback on test content and usability and determines what should be included in the field test, which is the next step in the test development process. The goal of a field test is to ensure that all those redesigned or new test items measure what they are intended to measure.

That all makes sense. But what does administering a pilot test look like in practice? My colleague Anne Lies and I visited Antonia Pantoja Charter School in Philadelphia, Pennsylvania, to observe Kindergarten ACCESS Pilot Test 2024 administrations. (Between WIDA and the Center for Applied Linguistics, at least two pilot test observers went to several sites around the consortium to ensure we got multiple perspectives on test administrations.) Here’s how it went:

Before school: Anne and I arrive at Antonia Pantoja Charter School, which is a dual-language, K-8 charter school, and are warmly greeted by everyone we encounter, including our hosts, Beth Silano, ELL coordinator, and Dennise Lopez-Colon, K-12 ESL teacher.

Beth and Dennise have already gone through the Kindergarten ACCESS Pilot Test 2024 training and prepared test materials accordingly, so they use this time to review the testing schedule for the day and make sure all pilot test materials (which I can confirm were kept in their yellow DRC box and in a locked room) are ready to go.

Morning: Dennise picks up the first student and brings them into the testing room. I ask for the student’s consent and then I turn on the tape recorder so that WIDA has examples of student responses for future training courses. Beth sits near the student and administers the pilot test, which is a paper-based, one-on-one, interactive assessment that covers all four language domains. Beth and the student are interacting with one of four separate versions of the pilot test given at participating sites throughout the WIDA consortium.

The pilot test is somewhat longer than Kindergarten ACCESS. This allows my colleagues at WIDA to try out the largest possible number of new items to find the ones that will work best for the redesigned Kindergarten ACCESS.

Beth makes sure to score the student’s responses to each task even though the student will not receive an actual proficiency level score for the pilot test. Pilot performance data helps WIDA staff understand whether the new test content is accurately assessing English language proficiency. Pilot test administrators are using a revised scoring rubric, so WIDA is collecting feedback on that experience, too.

Beth and Dennise administer the pilot test to two more students. I make note of how excited students are to get stickers when they finish – something to keep in mind as we think of ideas for field testing!

Afternoon: After lunch, Beth tests one more student, and we’re done testing for the day. Anne and I interview Beth and Dennise about their pilot test experience. They share insights into all sorts of things, like the age appropriateness of test items and the usefulness of the test graphics.

(Interviewing the participating educators is a critical part of the pilot testing process. WIDA needs to understand how test administrators experience the assessment, too. If WIDA staff aren’t in-person at a pilot testing site, test administrators share their feedback via a survey.)

Drawing on my curiosity about pilot testing in general, I (KS) ask Beth (BS) and Dennise (DLC) to weigh in on a few of my own questions. Here’s our back and forth:

KS: Why did you volunteer to participate in pilot testing?

BS: Because of the students we serve. Their needs, what’s appropriate for them and their instruction are always at the forefront of my mind. Since they are five years old, it’s my job to make sure they have a voice.

DLC: As an EL teacher who works with kindergarteners, I wanted to see student growth with the pilot test – it’s such a tiny bit of time between when students join us in fall to when they are taking Kindergarten ACCESS in spring.

KS: Why is it important for WIDA to host test development events like pilot testing?

DLC: It shows us you care. You are taking time to learn about how we give this test and what we think of it. It’s our responsibility to work together to help all kids – no matter what school, city or state.

BS: It gives validation to the test that you’ve taken the time to watch students take it, get our feedback and bring it back to your team to work on changes. Having you come here has given me a different perspective. I know now there are reasons why everything is done the way it’s done.

KS: Even though testing students isn’t everyone’s favorite activity, what are some of the positive outcomes that you’ve observed of testing students?

DLC: I see what my kids can do and what they’ve learned from me. It puts everything together. Then, I can continue to do what’s working or make changes. Beth and I use the data to make plans for students and their instruction and it helps us explain to classroom teachers what students can do.

BS: I like seeing their growth from year to year. During testing we look for little moments like today when we saw a student do so much reading. I don’t work with kindergarteners as much and it’s the test that helps me see what they can do.

Before calling it a great day, we tour the school and snap a few photos of all of us. Then, Anne and I sign out at the front desk. It’s been an incredible experience! We are so fortunate to be able to learn from these two phenomenal educators and their students.

After school: Anne and I debrief on what we observed during the day and catalog our pages of notes. Then, we send all that data to our colleagues at WIDA and CAL so that they can analyze it and begin to understand what worked and what didn’t work as we prepare for field testing.

Learn more about test development on the Building a WIDA Assessment page.