A third audience question explored the struggle between assessing students on established standards and assessing them on content they have not had the opportunity to learn. Gordon asserted that we need to use a kaleidoscope model when making high-stakes decisions where we examine the many “hues” of a test taker. He believes it is definitely possible for more to be done, but he acknowledged that it is a difficult task, especially in our world’s changing demographic landscape. Another question came from an educational consultant and former university professor in measurement, who asked the following: “Do you think validation should remain within the purview of the educational measurement profession, or do you envision concerns of stakeholders that fall within contest and pragmatic perspectives could actually work their way into interpretive and validity arguments? If so, can you give us an example of what that might look like?” Kane responded that many changes have already been made to incorporate varied perspectives into validity arguments in testing programs. To what extent did the scores, used for awarding scholarships, correlate with their future levels of student success? She cautions, however, that portfolio-based assessments tend to be less reliable that other forms of testing. Similarly, von Davier proposed a focus on collecting “predictive validity evidence” where students would be evaluated the year after being admitted to determine if the selection process is identifying students who would actually succeed. Such careful evaluations can provide formative feedback and facilitate long-term fairness in making decisions. Ercikan recommended also evaluating students who are not selected into the program and identifying potential patterns or trends that describe their assessed abilities. Each school is likely to want their favorite student to receive a scholarship, prompting the question, “What rules do we use to minimize bias and maximize efficacy in selecting 11th and 12th grade students for our leadership program?” Kane suggested a social moderation alternative where all involved schools agree on a common set of standards they use to evaluate students. A representative from a youth life skills and leadership program in India had concerns about the validity of their student-selection process based on portfolio-based assessments, with decisions made by individual schools. Audience members at the conference expressed some concerns worth noting here.

DIF detection, if inaccurate, can lead to the development of biased tests and threaten the consistency of score meanings across subgroups.

They caution that we must be very specific about our claims regarding validity and fairness with diverse populations.

Nwea map test validity and reliability