Measures and Outcome Assessment

We work to carefully track outcomes in each of our students, whether in terms of psychological functioning and growth, academic functioning and growth, or other matters of priority (e.g., basic enjoyment of the tutoring). Our "core battery" includes the measures that we seek to track in all students. This page provides more details about key pieces of this battery, as well as certain of the measures themselves, in pdf form. 

Though many families do choose to release their outcomes data - for observational use, in deidentified form - such that we can track OPT's aggregate effects over time, this aggregate evaluation is not the main purpose that drives our measurement. Our first priority is to facilitate student growth to the best of our abilities. Our measures help us know when we're on the right track, student-by-student. 

Can OPT perform assessments, even if students do not ultimately participate in OPT's tutoring? 

In many cases, yes. Contact Jillian Strayhorn for more information about this. 

Rating Scales of Overall Psychological Functioning

  • Parent and Teacher Ratings: We periodically seek ratings on parent- and teacher-versions of the Psychological Functioning Scale for Children and Adolescents (Strayhorn & Strayhorn, paper under preparation) and the Psychological Skills Rating Scale (Strayhorn & Strayhorn, paper under preparation), as well as the Vanderbilt ADHD Rating Scales (Wolraich et al., 1998a, 1998b, 2003), the Columbia Impairment Scale (Bird et al., 1993), and the Strengths and Difficulties Scale (Goodman, 2001). These scales are all quite broad-spectrum; they can serve as outcome measures for a heterogeneous group of children and youth.
  • Child and Adolescent Self-Ratings: When possible we seek self-ratings from our students on somewhat modified versions of these same overall functioning rating scales. We also seek students' ratings on the first item of the Fordyce Emotions Questionnaire (Fordyce, 1988), which contains a simple 0 to 10 rating of life satisfaction.

Functioning Narrative

  • We also assess functioning through parent interviews, particularly before tutoring has begun. From the details parents give us we build a narrative, which parents also check for accuracy. This narrative is then scored by blind raters on a variety of scales, often the CGAS scale (Shaffer et al., 1983) and/or the HONOSCA Scale (Gowers et al., 1999). 

Academic and Intellectual Skill Measures

  • We assess reading recognition with the Slosson Oral Reading Test, 3rd Revision (Slosson & Nicholson, 2002) and the Test of Word Reading Efficiency, Second Edition (Torgeson et al., 2012). We assess reading comprehension with the Gray Oral Reading Test, 5th edition; the Test of Reading Comprehension, 4th edition; and/or the reading comprehension subscale of the Wide Range Achievement Test, Expanded Version (Robertson, 2001). In some cases we also use the mathematics subscale of the Wide Range Expanded, and sometimes we also use the nonverbal reasoning portion of the test. We also often use verbal sections of the KBIT (Kaufman & Kaufman, 2004). 

Performance Measures of Psychological Skill

  • We have ongoing efforts to further develop and implement performance measures that ask students to: generate options for hypothetical problems; rank order the quality of options for hypothetical problems; generate role-played social conversations; explain the ethical principles that they consider most important; recall examples of positive behavior that they have carried out; do reflective listening; and fantasy rehearse aloud good handlings of anxiety- or anger-provoking situations. These performance measures, among others, have the potential provide another sort of evidence of students' competence development.

Process Measures

  • We measure the number of sessions students hold with their tutors; the amount of time spent in the tutoring; the total number of words decoded and/or read in reading instruction, if applicable; the number of "sections," "chapters," or curricular "books" read; and the number of psychological skills exercises done and/or mastered. We also record the student's impressions of the tutoring, and of work and reading more generally. 

The psychological skills we hope to nurture in all of our students include: the effort-payoff connection, self-discipline, goal-setting and goal-monitoring. One way we work to teach these specific skills is by guiding students in setting their own process-related goals (for example, "reading 10,000 words" or "finishing the next programmed manual" or "mastering the Brainstorming Options exercise"). All tutor-student pairs set such individualized goals, and tutors meticulously track their students' progress towards those goals, updating the student each step of the way. Goal attainment is celebrated enthusiastically, often with a congratulatory certificate or letter sent to the student via snail mail. 

Parent Practices Scales

  • In some cases we will also track certain parent practices, including, for example, parent behaviors that promote a positive emotional climate and parent behaviors that enhance parent authority with kindness and fairness. These are self-report measures that parents may opt into or out of, as they prefer. 

References:

  • Bird HR, Shaffer D, Fisher P, et al. (1993). The Columbia Impairment Scale (CIS): pilot findings on a measure of global impairment for children and adolescents. Int J Methods Psychiatr Res, 3:167-176.
  • Fordyce MW (1988). A review of research on The Happiness Measures: A sixty second index of happiness and mental health. Social Indicators Research, 20, 63-89.
  • Goodman R (2001). Psychometric properties of the Strengths and Difficulties Questionnaire (SDQ). Journal of the American Academy of Child and Adolescent Psychiatry, 40, 1337-1345.
  • Gowers SG, Harrington RC, Whitton A, Lelliott P, Beevor A, Wing J, & Jezzard R (1999). Brief scale for measuring the outcomes of emotional and behavioural disorders in children. Health of the Nation Outcome Scales for children and Adolescents (HoNOSCA). Br J Psychiatry, 174:413-416.
  • Kaufman AS & Kaufman NL (2004) Kaufman Brief Intelligence Test, Second Edition (KBIT-2). AGS Publishing, Circle Pines, MN.

  • Robertson GJ (2001). Wide Range Achievement Test–Expanded Edition. Wilmington, DE: Wide Range.
  • Shaffer D, Gould MS, Brasic J, et al. (1983) A children's global assessment scale (CGAS). Archives of General Psychiatry, 40, 1228-1231.

  • Slosson R & Nicholson CL (2002). Slosson oral reading test: SORT-R3. East Aurora, NY: Slosson Educational Publications, Inc.

  • Torgesen J, Wagner R, & Rashotte C (2012). Test of Word Reading Efficiency, 2nd Edition (TOWRE-II). Austin, TX: Pro-Ed.

  • Wolraich ML, Feurer I, Hannah JN, Pinnock TY, & Baumgaertel A (1998a). Obtaining systematic teacher reports of disruptive behavior disorders utilizing DSM-IV. Journal of Abnormal Child Psychology, 26, 141– 152.

  • Wolraich ML, Hannah JN, Baumgaertel A., & Feurer ID (1998b). Examination of DSM-IV critieria for attention deficit/hyperactivity disorder in a county-wide sample. Journal of Developmental and Behavioral Pediatrics, 19, 162– 168.

  • Wolraich ML, Lambert W, Doffing MA, Bickman L, Simmons T, & Worley K (2003). Psychometric Properties of the Vanderbilt ADHD Diagnostic Parent Rating Scale in a Referred Population. J. Pediatr. Psychol, 28, 559-568.