The National Assessment of Educational Progress (NAEP) program — which produces the “Nation’s Report Card” and other critical education data — should make changes to understand and reduce its costs and open the door to innovation, says a new report from the National Academies of Sciences, Engineering, and Medicine.
The NAEP program helps educators, policymakers, and the public understand educational outcomes by administering and analyzing assessments of what fourth, eighth, and 12th graders in the U.S. know and can do in reading, mathematics, science, writing, and other subjects. The program has incurred substantially increased costs over the past two decades and currently costs about $175.2 million per year. The program is administered by the National Center for Education Statistics (NCES), part of the Institute of Education Sciences within the U.S. Department of Education. The independent National Assessment Governing Board (NAGB) sets policy for NAEP.
“NAEP has not kept pace with the more streamlined approaches being pursued by other testing programs, and as a result is facing serious cost-containment issues,” said Karen Mitchell, retired senior director of the Medical College Admission Test at the Association of American Medical Colleges, and chair of the committee that wrote the report. “The changes we recommend in our report can bolster the future success of this valuable and respected program.”
A Pragmatic Future for NAEP: Containing Costs and Updating Technologies says about 28.6 percent of NAEP’s budget — the most expensive assessment component of the program — is spent on its unusual approach to administering assessments, in which outside contractor teams and computers are sent to local schools. The report endorses efforts by the program to use local school computers and local school staff as proctors in place of contractors, and says this change could save as much as 18.7 percent of NAEP’s budget by 2030 — more savings than previously estimated.
To improve the program’s technical quality, reduce burden on schools, and decrease costs, the assessment should also be administered in longer sessions that allow 90 minutes for student cognitive testing questions.
NAEP should consider integrating testing for subjects that it currently assesses separately — such as reading and writing — which would be more efficient, cost effective, and align more closely with student learning. The program should also prepare a detailed plan and budget for modernizing NAEP’s measurements of long-term trends, weighing those plans carefully against other program priorities.
The report says NCES should implement automated scoring, which could reduce costs by about 0.7 percent (roughly $1.25 million) of NAEP’s annual budget. NCES should investigate the potential of adaptive testing, which may not reduce costs, but could improve statistical estimates and the test-taking experience for lower-performing students. NAGB and NCES should also create a more structured process for question development, and complete an analysis of the value and cost of different types of assessment questions.
A greater proportion of NAEP’s budget for analysis and reporting should go toward innovations that will make its data more usable and better understood — including finding ways to make raw data available to researchers more quickly, and improving the sophistication of the online NAEP Data Explorer, the report says.
The committee that wrote the report found it difficult to obtain complete information to explain NAEP’s costs and how they are connected to its programs. NCES and NAGB should develop a clear, consistent, and complete description of current NAEP spending to ensure that major decisions can be supported by the budget.
Well over 28.7 percent of NAEP’s budget goes to management, planning, support, and oversight — a category that includes costs associated with federal employees; contracts specifically for support functions; and management and planning portions of all other contracts. The report recommends that NAGB and NCES commission an independent audit of program management, decision-making, and costs to identify ways to streamline these processes. The report also recommends that NAEP’s research activities be separated from management and planning, with an identifiable budget and program of activities.
As NCES develops its new platform for administering tests — the Next-Gen eNAEP platform — it needs to pay close attention to costs for technology support, the report says. NCES should regularly evaluate whether it can use open-source software or software built by other vendors in its new platform. NCES should also make sure there is adequate expertise to support and oversee the platform, as well as seek expert guidance from app developers and educational technologists about the platform’s projected costs.
The study — undertaken by the Panel on Opportunities for the National Assessment of Educational Progress in an Age of AI and Pervasive Computation: A Pragmatic Vision — was sponsored by the U.S. Department of Education.
Report
https://www.nap.edu/catalog/26427/a-pragmatic-future-for-naep-containing-costs-and-updating-technologies
Press release
https://www.nationalacademies.org/news/2022/03/nations-report-card-assessment-should-make-changes-to-contain-costs-allow-innovation-in-education-testing-and-research-says-new-report
Panel: Opportunities for the National Assessment of Educational Progress in an Age of AI and Pervasive Computation: A Pragmatic Vision
https://www.nationalacademies.org/our-work/opportunities-for-the-national-assessment-of-educational-progress-in-an-age-of-ai-and-pervasive-computation-a-pragmatic-vision-for-2030-and-beyond#sectionCommittee
Commissioner's Remarks-Statement of National Center for Education Statistics Commissioner Peggy G. Carr
https://nces.ed.gov/whatsnew/commissioner/remarks2022/3_24_2022.asp