Validity as a tool of evaluation

Keller-Margulis, Milena A. Existing approaches to measuring writing performance are insufficient in terms of both technical adequacy as well as feasibility for use as a screening measure. This study examined the validity and diagnostic accuracy of several approaches to automated text evaluation as well as written expression curriculum-based measurement WE-CBM to determine whether an automated approach improves technical adequacy.

A sample of fourth grade students generated writing samples that were then scored using traditional and automated approaches and examined in relation to the statewide measure of writing performance. While you should try to take steps to improve the reliability and validity of your assessment, you should not become paralyzed in your ability to draw conclusions from your assessment results and continuously focus your efforts on redeveloping your assessment instruments rather than using the results to try and improve student learning.

Reliability is the extent to which a measurement tool gives consistent results. It does not have to be right, just consistent. Student learning throughout the program should be relatively stable and not depend on who conducts the assessment. Issues with reliability can occur in assessment when multiple people are rating student work, even with a common rubric, or when different assignments across courses or course sections are used to assess program learning outcomes.

Validity is the extent to which a measurement tool measures what it is supposed to. More specifically, it refers to the extent to which inferences made from an assessment tool are appropriate, meaningful, and useful American Psychological Association and the National Council on Measurement in Education. In order to be valid, a measurement must also and first be reliable.

Validity is often thought of as having different forms. Perhaps the most relevant to assessment is content validity, or the extent to which the content of the assessment instrument matches the SLOs. Content validity can be improved by:. On the other hand, one of the things that can improve validity is flexibility in assessment tasks and conditions. Such flexibility allows assessment to be set appropriate to the learning context and to be made relevant to particular groups of students.

Insisting on highly consistent assessment conditions to attain high reliability will result in little flexibility, and might therefore limit validity. The Overall Teacher Judgment balances these ideas with a balance between the reliability of a formal assessment tool, and the flexibility to use other evidence to make a judgment.

Used with permission. Search all of TKI. Search community. Assessment navigation Home. You are here: Home » Using evidence for learning » Working with data » Reliability and validity Reliability and validity.