7 Assessment Results

Chapter 7 of the Dynamic Learning Maps® (DLM®) Alternate Assessment System 2015–2016 Technical Manual—Science (Dynamic Learning Maps Consortium, 2017) describes assessment results for the 2015–2016 academic year, including student participation and performance summaries, and an overview of data files and score reports delivered to state education agencies. Technical Manual updates provide a description of data files, score reports, and results for each corresponding academic year.

This chapter presents Spring 2021 student participation data; the percentage of students achieving at each performance level; and subgroup performance by gender, race, ethnicity, and English learner status. This chapter also reports the distribution of students by the highest linkage level mastered during Spring 2021. Finally, this chapter describes updates made to score reports during the 2020–2021 operational year. For a complete description of score reports and interpretive guides, see Chapter 7 of the 2015–2016 Technical Manual—Science (Dynamic Learning Maps Consortium, 2017).


In this chapter we describe the results that were reported as part of the 2020–2021 assessment administration. However, due to the confounding factors of assessment administration changes and COVID-19, these results should be interpreted with caution and should not be directly compared to previous assessment administrations.


7.1 Impacts to Assessment Administration

The 2020–2021 school year was significantly impacted by COVID-19. Overall, participation in DLM assessments across all states was lower than what would typically be expected. This decrease was not uniform across demographic subgroups. White students made up a larger percentage of the student population in 2020–2021 than in prior years, whereas African American students, students of Hispanic ethnicity, and English learners made up a smaller percentage of the student population. There were also fewer students who were placed in the Foundational and Band 3 complexity bands, which are used to determine the starting linkage level in each subject (see Chapter 4 of this manual for a description of linkage level assignment). Further, data from the spring teacher survey indicated that students may have had less opportunity to learn, and that many students experienced difficulty with remote learning.

For a complete discussion of student performance and the potential impacts of assessment administration changes and COVID-19, see (Accessible Teaching, Learning, and Assessment Systems, 2021).

7.2 Student Participation

During spring 2021, assessments were administered to 30,895 students in 18 states Two states chose to extend their testing window through September 2021. For these states, results are included for students who completed their assessments by the close of the standard consortium testing window on July 2, 2021.. Counts of students tested in each state are displayed in Table 7.1. The assessments were administered by 14,271 educators in 9,035 schools and 3,796 school districts.

Table 7.1: Student Participation by State (N = 30,895)
State Students (n)
Alaska 137
Arkansas 2,289
Delaware 284
Illinois 2,785
Iowa 841
Kansas 882
Missouri 2,337
New Hampshire 239
New Jersey 2,765
New Mexico 75
New York 5,123
North Dakota 225
Oklahoma 1,973
Pennsylvania 4,718
Rhode Island 365
Utah 3,192
West Virginia 562
Wisconsin 2,103

Table 7.2 summarizes the number of students assessed in each grade and course. More than 9,000 students participated in each of the elementary (grades 3–5) and the middle school (grades 6–8) grade bands. In an effort to increase science instruction beyond the tested grades, several states promoted participation in the science assessment at all grade levels (i.e., did not restrict participation to the grade levels required for accountability purposes). In high school (grades 9–12) almost 10,700 students participated. The differences in high school grade-level participation can be traced to differing state-level policies about the grade(s) in which students are assessed.

Table 7.2: Student Participation by Grade or Course (N = 30,895)
Grade Students (n)
3 535
4 4,573
5 4,704
6 859
7 876
8 8,681
9 2,933
10 1,594
11 5,752
12 302
Biology 86

Table 7.3 summarizes the demographic characteristics of the students who participated in the spring 2021 administration. The majority of participants were male (67%) and white (68%). About 5% of students were monitored or eligible for English learner services.

Table 7.3: Demographic Characteristics of Participants (N = 30,895)
Subgroup n %
Gender
Male 20,682 66.9
Female 10,213 33.1
Race
White 20,923 67.7
African American 4,750 15.4
Two or more races 3,110 10.1
Asian 1,196 3.9
American Indian 696 2.3
Native Hawaiian or Pacific Islander 166 0.5
Alaska Native 54 0.2
Hispanic ethnicity
No 25,794 83.5
Yes 5,101 16.5
English learner participation
Not EL eligible or monitored 29,402 95.2
EL eligible or monitored 1,493 4.8

In addition to the spring administration, instructionally embedded science assessments are also made available for teachers to administer to students during the year. Results from the instructionally embedded science assessments do not contribute to final summative scoring but can be used to guide instructional decision-making. Table 7.4 summarizes the number of students participating in instructionally embedded testing by state. A total of 2,177 students, across 10 states, took at least one instructionally embedded science testlet during the 2020–2021 academic year.

Table 7.4: Students Completing Instructionally Embedded Science Testlets by State (N = 2,177)
State n
Arkansas 529
Delaware 5
Iowa 117
Kansas 204
Missouri 1,226
New Jersey 1
North Dakota 51
Oklahoma 41
Utah 1
Wisconsin 2

Table 7.5 summarizes the number of instructionally embedded testlets taken in science. Across 10 states, students took 15,881 total testlets during the instructionally embedded window.

Table 7.5: Number of Instructionally Embedded Science Testlets by Grade or Course (N = 15,881)
Grade n
3 891
4 900
5 2,844
6 1,101
7 1,136
8 3,413
9 1,035
10 1,265
11 1,988
12 1,306
Biology 2

7.3 Student Performance

Student performance on DLM assessments is interpreted using cut points, determined during standard setting, which categorize student results into four performance levels. For a full description of the standard-setting process, see Chapter 6 of the 2015–2016 Technical Manual—Science (Dynamic Learning Maps Consortium, 2017). Additional cut points were set for grade 3 and grade 7 in 2018–2019. For a description of the cut points for those grades, see Chapter 6 of the 2018–2019 Technical Manual Update—Science (Dynamic Learning Maps Consortium, 2019). A student’s performance level is determined based on the total number of linkage levels mastered across the assessed Essential Elements (EEs).

For the spring 2021 administration, student performance was reported using the same four performance levels approved by the DLM Consortium for prior years:

  • The student demonstrates Emerging understanding of and ability to apply content knowledge and skills represented by the EEs.
  • The student’s understanding of and ability to apply targeted content knowledge and skills represented by the EEs is Approaching the Target.
  • The student’s understanding of and ability to apply content knowledge and skills represented by the EEs is At Target. This performance level is considered to be meeting achievement expectations.
  • The student demonstrates Advanced understanding of and ability to apply targeted content knowledge and skills represented by the EEs.

7.3.1 Overall Performance

Table 7.6 reports the percentage of students achieving at each performance level from the spring 2021 administration for science. The spring 2021 results were fairly consistent with performance in prior years, with the majority of students achieving at either the Emerging or Approaching the Target performance levels. At the elementary level, the percentage of students who achieved at the At Target or Advanced levels ranged from approximately 12% to 15%; in middle school the range was 16% to 20%; and in high school and end-of-instruction biology, the percentages ranged from 6% to 26%.

Table 7.6: Percentage of Students by Grade and Performance Level
Grade Emerging (%) Approaching (%) Target (%) Advanced (%) Target+ Advanced (%)
3 (n = 535) 66.0 21.1 7.1 5.8 12.9
4 (n = 4,573) 64.2 21.3 10.5 4.0 14.5
5 (n = 4,704) 67.5 20.2 11.5 0.8 12.2
6 (n = 859) 63.0 20.8 12.9 3.3 16.2
7 (n = 876) 61.5 22.4 13.1 3.0 16.1
8 (n = 8,681) 56.2 24.2 17.6 2.0 19.6
9 (n = 2,933) 56.3 25.5 13.7 4.6 18.2
10 (n = 1,594) 62.0 26.8 9.2 2.0 11.2
11 (n = 5,752) 54.8 29.2 13.3 2.8 16.1
12 (n = 302) 74.5 19.2 6.0 0.3 6.3
Biology (n = 86) 58.1 16.3 19.8 5.8 25.6

7.3.2 Subgroup Performance

Data collection for DLM assessments includes demographic data on gender, race, ethnicity, and English learner status. Table 7.7 summarizes the disaggregated frequency distributions for science, collapsed across all assessed grade levels. Although state education agencies each have their own rules for minimum student counts needed to support public reporting of results, small counts are not suppressed here because results are aggregated across states and individual students cannot be identified.

Table 7.7: Performance Level Distributions by Demographic Subgroup (N = 30,895)
Emerging
Approaching
Target
Advanced
Subgroup n % n % n % n %
Gender
Male 12,277 59.4 4,905 23.7 2,891 14.0 609 2.9
Female 6,213 60.8 2,528 24.8 1,268 12.4 204 2.0
Race
White 12,354 59.0 5,118 24.5 2,889 13.8 562 2.7
African American 2,877 60.6 1,115 23.5 625 13.2 133 2.8
Two or more races 1,921 61.8 726 23.3 396 12.7 67 2.2
Asian 833 69.6 236 19.7 105 8.8 22 1.8
American Indian 367 52.7 190 27.3 116 16.7 23 3.3
Native Hawaiian or Pacific Islander 105 63.3 36 21.7 20 12.0 5 3.0
Alaska Native 33 61.1 12 22.2 8 14.8 1 1.9
Hispanic ethnicity
No 15,314 59.4 6,272 24.3 3,530 13.7 678 2.6
Yes 3,176 62.3 1,161 22.8 629 12.3 135 2.6
English learner participation
Not EL eligible or monitored 17,605 59.9 7,068 24.0 3,967 13.5 762 2.6
EL eligible or monitored 885 59.3 365 24.4 192 12.9 51 3.4

7.3.3 Linkage Level Mastery

As described earlier in the chapter, overall performance in each subject is calculated based on the number of linkage levels mastered across all EEs. Results indicate the highest linkage level the student mastered for each EE. The linkage levels are (in order): Initial, Precursor, and Target. A student can be a master of zero, one, two, or all three linkage levels, within the order constraints. For example, if a student masters the Precursor level, they also master the Initial linkage level. This section summarizes the distribution of students by highest linkage level mastered across all EEs. For each student, the highest linkage level mastered across all tested EEs was calculated. Then, for each grade, the number of students with each linkage level as their highest mastered linkage level across all EEs was summed and then divided by the total number of students who tested in the grade. This resulted in the proportion of students for whom each level was the highest level mastered.

Table 7.8 reports the percentage of students who mastered each linkage level as the highest linkage level across all EEs for each grade. For example, across all third-grade EEs, the Initial level was the highest level that students mastered 43% of the time. The percentage of students who mastered as high as the Target linkage level ranged from approximately 21% in grade 12 to 50% in grade 8.

Table 7.8: Students’ Highest Linkage Level Mastered Across Science EEs by Grade
Linkage Level
Grade No evidence (%) Initial (%) Precursor (%) Target (%)
3 (n = 535) 16.1 42.8 11.8 29.3
4 (n = 4,573) 5.6 40.0 13.8 40.6
5 (n = 4,704) 5.1 40.2 13.2 41.5
6 (n = 859) 8.4 33.3 18.0 40.3
7 (n = 876) 7.8 30.7 20.7 40.9
8 (n = 8,681) 4.9 27.3 17.5 50.4
9 (n = 2,933) 7.1 29.0 28.0 35.9
10 (n = 1,594) 6.1 32.4 33.1 28.3
11 (n = 5,752) 4.6 28.7 30.5 36.1
12 (n = 302) 21.5 38.1 19.5 20.9
Biology (n = 86) 5.8 34.9 22.1 37.2

7.4 Data Files

Data files were made available to DLM state education agencies following the spring 2021 administration. Similar to prior years, the General Research File (GRF) contained student results, including each student’s highest linkage level mastered for each EE and final performance level for the subject for all students who completed any testlets. In addition to the GRF, the DLM Consortium delivered several supplemental files. Consistent with prior years, the special circumstances file provided information about which students and EEs were affected by extenuating circumstances (e.g., chronic absences), as defined by each state. Three new special circumstance codes were available in 2020–2021: 1. Student could not test due to COVID-19; 2. Teacher administered the assessment remotely; and 3. Non-teacher administered. State education agencies also received a supplemental file to identify exited students. The exited students file included all students who exited at any point during the academic year. In the event of observed incidents during assessment delivery, state education agencies are provided with an incident file describing students impacted. For a description of incidents observed during the 2020–2021 administration, see Chapter 4 of this manual.

Consistent with prior delivery cycles, state partners were provided with a two-week review window following data file delivery to review the files and invalidate student records in the GRF. Decisions about whether to invalidate student records are informed by individual state policy. If changes were made to the GRF, state partners submitted final GRFs via Educator Portal. The final GRF was used to generate score reports.

In addition to the GRF and its supplemental files, states were provided with two additional de-identified data files: a teacher survey data file and a test administration observations data file. The teacher survey file provided state-specific teacher survey responses, with all identifying information about the student and educator removed. The test administration observations file provided test administration observation responses with any identifying information removed. For more information regarding teacher survey content and response rates, see Chapter 4 of this manual. For more information about test administration observation results, see Chapter 9 of this manual.

7.5 Score Reports

The DLM Consortium provides assessment results to all member states to report to parents/guardians, educators, and state and local education agencies. Individual Student Score Reports summarized student performance on the assessment by subject. Several aggregated reports were provided to state and local education agencies, including reports for the classroom, school, district, and state. No changes were made to the structure of aggregated reports during spring 2021. Changes to the Individual Student Score Reports are summarized below. For a complete description of score reports, including aggregated reports, see Chapter 7 of the 2014–2015 Technical Manual—Year-End Model (Dynamic Learning Maps Consortium, 2016).

7.5.1 Individual Student Score Reports

Individual Student Score Reports included a Performance Profile section, which describes student performance in the subject overall. In 2021, a Learning Profile section was added to the reports, which provides detailed reporting of student mastery of individual skills. The Learning Profile section was added due to changes in the year-end blueprint in 2019–2020. A cautionary statement was added to the 2020–2021 Performance Profile and Learning Profile, which indicated that the 2020–2021 academic year was significantly impacted by the COVID-19 pandemic, and mastery results may have reflected the unusual circumstances for instruction and assessment. For more information on validity considerations and scoring and reporting in flexible scenarios, see (Clark et al., 2021).

EEs were reordered in the table in the Learning Profile to match blueprint order, and the hyperlink for the DLM website’s font was increased and moved to the footer before the copyright statement.

A sample Performance Profile and a sample Learning Profile reflecting the 2020–2021 changes are provided in Figure 7.1 and Figure 7.2.

Figure 7.1: Example Page of the Performance Profile for 2020–2021.

Example Page of the Performance Profile for 2020–2021.

Figure 7.2: Example Page of the Learning Profile for 2020–2021.

Example Page of the Learning Profile for 2020–2021.

7.6 Quality Control Procedures for Data Files and Score Reports

Changes to the quality control procedures were made only to the extent of accommodating the revised score reports for spring 2021 (i.e., checking to be sure changes were correctly and consistently applied). For a complete description of quality control procedures, see Chapter 7 in the 2015–2016 Technical Manual—Science (Dynamic Learning Maps Consortium, 2017).

7.7 Conclusion

Following the spring 2021 administration, six data files were delivered to state partners: GRF, special circumstance code file, exited students file, incident file, teacher survey data file, and test administration observations file. An incident file was delivered describing the impact of one reported incident. Overall, between 6% and 26% of students achieved at the At Target or Advanced levels across all grades and subjects. However, these results should be interpreted with caution due to the confounding factors of assessment administration changes and COVID-19. Lastly, minor changes were made to score reports to aid in interpretation.