Denver Public Schools ('22-23)

Program Standardization and Efficacy: 2022 - 2023 Highlights

Compared to the 2021-22 school year, this year's Denver program was markedly more standardized. Working with the Denver Grade Level Tutoring (GLT) team, Cignition curated a uniform program design, building on the successes of the preceding year. Although some schools maintained autonomy, student selection, group tutoring size standardization, and regularized scheduling were recurring elements.

The data demonstrates the effectiveness of those guidelines in program design. Overwhelmingly, sessions were scheduled during the school day. Only 4 of 263 groups (1.5%) were scheduled outside of regular school hours. Those four groups took place on campus before school started. Tutoring groups comprised three or four students (an average of 3.57 students per session), homogeneously grouped by math proficiency. Groups met consistently, meeting on the same days and times each week. Each group was assigned a lead tutor, who conducted all of the sessions unless they were unable to attend, in which case a substitute was employed. The lead tutors attended 90.9% of their assigned sessions. These changes led to higher levels of student engagement and improved student outcomes, as shown in Figure 2.

Data Informed Instruction

Last year, Cignition, in collaboration with the Denver GLT team, set 5 goals for improving the 2022-23 student outcomes. Four of those goals revolved around improving our use of student data to provide more targeted instruction. Those goals were:

1. Teachers will be able to select standards to be covered and the time frame for those sessions to align our efforts with their classroom more closely. 
2. Assessments will measure the learning gains in reference to the specific standards that were covered in the student’s group. 
3. Assessments will provide a standards analysis so teachers and tutors can help identify misconceptions and address individual student needs. 
4. An improved feedback tool to accurately communicate student interim progress to their instructors.

All five goals were reached by the start of the 2022-23 school year. Teachers had the ability to pinpoint specific standards for tutors to concentrate on for each individual group. This enabled tutors to tailor their focus according to the data from classroom assessments and insights from our pre-diagnostic evaluation. Weekly student progress reports now included outcomes at the standards level, allowing teachers to make adjustments to the instruction as needed.

These changes gave us a new tool to measure student outcomes: Standards Mastery Percentage (also referred to as Standards Progress.) This assessment involved students solving representative problems and explaining their thought processes. Furthermore, tutors directly observed students' methods to arrive at their answers, ensuring a comprehensive understanding of the student's approach. From these observations, students were measured on a three-point Likert scale:

• Complete Understanding (1.0 point)
• Partial Understanding (0.5 points)
• No Understanding (0.0 points)

The program's effectiveness was measured by comparing student engagement metrics against our Standards Progress. The hypothesis was that as students were more engaged, they would demonstrate progress by mastering a larger percentage of the standards they worked on. Our key engagement metrics were attendance (70% was our standard), overall engagement (80%, as measured daily by the tutors, was our standard), and contact hours (25 hours was our standard.) In Figure 3, the Standards Progress score of students who met none, one, two, or all three standards was averaged. As expected, more engaged students mastered a higher percentage of the standards they engaged.

¹ Absolute Gain score is calculated by subtracting the percentage correct on an identical diagnostic administered near the beginning of the series of sessions from the percentage correct on a diagnostic test administered near the end of the series of sessions. (Absolute Gain = Post-Diagnostic Score - Pre-Diagnostic Score)

² Standards Mastery score is calculated from the student progress measured by the tutor in each session. Students are given a formative assessment from the standard addressed in that session. In addition to assessing whether the answer is correct, the tutor also requires the student to explain how they arrived at the answer, taking into consideration the process demonstrated in obtaining the answer. Student progress is measured with a three-point Likert scale”

• Complete Understanding (100%)
• Partial Understanding (50%)
• No Understanding (0%)

The Standards Mastery score is the average of these scores. In essence, it is the percentage of standards attempted by the student that were actually mastered.

³ Cignition’s goal for each student is to see them demonstrate conceptual mastery of each concept attempted. Using formative assessments, tutors measure student progress and use that data to inform instruction. As described in note 2, student progress is measured and recorded each session by tutors. The measurement “demonstrated growth” indicates that the student moved from “No Understanding” to “Partial Understanding” or “Complete Understanding”.