The QRIS evidence base is limited; studies are often observational and lack strong causal designs, many studies were conducted in the early stages of QRIS implementation, and studies frequently focus on children ages 3 to 5, excluding those ages 0 to 3. Although the following review focuses on the limited causal evidence for QRIS, examples of important observational studies are discussed when relevant, particularly if they suggest critical directions for future research.
The research discussed here meets our standards of evidence for being methodologically strong and allowing for causal inference, unless otherwise noted. Each strong causal study reviewed has been assigned a letter, and a complete list of causal studies can be found at the end of this review, along with more details about our standards of evidence and review method. The findings from each strong causal study reviewed align with one of our eight policy goals from Table 1. The Evidence of Effectiveness table below displays the findings associated with QRIS (beneficial, null,v or detrimental) for the strong study (A) in the causal studies reference list. Findings for study B, which meets the criteria to be considered methodologically strong, are excluded from the table because the findings cannot be clearly classified as beneficial or detrimental to prenatal-to-3 wellbeing, but nevertheless warrant discussion. The assessment of the overall impact for each studied policy goal weighs the timing of publication and relative strength of each study, as well as the size and direction of all measured indicators.
Table 2: Evidence of Effectiveness for QRIS by Policy Goal
| Policy Goal | Indicator | Beneficial Impacts | Null Impacts | Detrimental Impacts | Overall Impact on Goal |
|---|
| Access to Needed Services | Reduced Enrollment in Low-Quality Child Care | A | | | Trending^ Positive |
| Nurturing and Responsive Child Care in Safe Settings | Child Care Quality Improvement | A | | | Trending Positive |
^Trending indicates that the evidence is from fewer than two strong causal studies or multiple studies that include only one location, author, or data set.
Access to Needed Services
Both of the strong causal studies included in this review examined the impact of QRIS on the types of providers parents selected to provide care for their children. A longitudinal study of licensed center-based providers rated in North Carolina’s QRIS between 2007 and 2009 found that providers with lower initial QRIS ratings had lower student enrollment at later time periods, and these effects were concentrated among programs in areas of high competition.A,vi Providers with lower initial ratings had approximately five and eight fewer students at three and five years post-initial rating (effect size at five years post-rating was 0.18) and that at five years post-rating, programs with lower initial ratings had approximately an 8 percentage point reduction in their capacity utilization (the percent of total student capacity being used). The authors hypothesized that the lag in parents’ response may have been due to parents’ lack of desire to transfer children currently enrolled in a care setting and there may have been a lag in the provision of this type of information to parents. Regardless, this evidence is consistent with the idea that parents respond to low QRIS ratings by moving their children out of the care of these providers.
A second longitudinal study on the effect of QRIS enactment on parent choice of child care providers found that the implementation of QRIS led to a shift from parental to nonparental care.B However, different types of families responded differently: economically disadvantagedvii families were more likely to use relative care as compared to parental care; economically advantaged families were more likely to use nonrelative, informal care as compared to parental care. Neither the full sample or either sub-sample of families was more likely to select formal child care providers (home- or center- based child care) relative to parental care. This study also found that, in the full sample of families, QRIS enactment led to approximately two more hours per week of nonparental child care use. QRIS enactment was also linked to lower likelihood of paying for care in the disadvantaged subsample; the opposite was true in the advantaged sample. Impacts of QRIS enactment differ across subgroups and child care types, but overall evidence suggests that QRIS may lead to increased use of nonparental care.
Although both of these studies included ECE programs serving children from birth through age 5, neither study disaggregated findings by age of children served to identify impacts specific to the care of infants and toddlers. No other strong causal studies have examined the impact of QRIS enactment or rating levels on the choices that parents make for the care of their infant and toddler children.
Parents’ Ability to Work
The longitudinal study of the impact of QRIS implementation also examined indicators related to maternal employment.B In this study, the author hypothesized that QRIS enactment would increase the demand for higher-quality care, formal care, and providers who participate in the state QRIS; these changes would subsequently result in higher family expenditures, driving “down-stream” changes in maternal employment, in response to increased expenditures. The findings demonstrate that QRIS enactment can lead to an increased likelihood of maternal work and increases in mothers’ amount of work (weekly hours, annual weeks of work) and earnings. However, these impacts were concentrated among the sub-sample of advantaged mothers only. These findings do suggest that there may be far-reaching consequences of QRIS; future research should explore this more thoroughly, especially among parents of infants and toddlers and to better understand if these impacts are beneficial for families.
Nurturing and Responsive Child Care in Safe Settings
Very limited causal evidence exists that examines the link between QRIS and quality improvement of child care providers over time. One study of North Carolina’s QRIS found that lower initial ratings of center-based providers led to improvements in ratings over time – in the first two years after the initial rating, there was a statistically significant gap in the probability of achieving at least a four-star rating between providers rated lower and higher at the initial time point.A However, by three years after the initial rating, this gap had closed, suggesting lower-rated providers had increased their ratings. Although not statistically significant, the trend for the probability of a achieving a three-star rating or higher was similar. Additionally, this study found that lower initial ratings led to increases in future measures of observed qualityviii by 0.25 and 0.21 points at four and five years post-initial rating, respectively, aligning with when programs were expected to be re-rated. The latter is equivalent to a “0.36 effect size with respect to the standard deviation observed at baseline,” meaning that “the estimated effect of receiving a three-star rating instead of a four-star rating is over 1.2 program-level standard deviations” (pp. 855-856). The authors also assessed the impact of lower quality ratings on changes in other measures of quality (e.g., staff qualifications and experience, ratios, or physical space requirements), but found null results.
Although causal evidence on the impact of QRIS on child care quality and quality improvement is limited, a large number of observational studies have been conducted at the state level on the validation of QRIS. Whether or not these systems are valid, defined as “how well the quality measurement and rating process are working to differentiate meaningful levels of ECE program quality in a QRIS” (p. 3),18,21 is critical to understanding the effectiveness of QRIS. Validity is typically assessed through validation studies and, if designed rigorously, state QRIS validation studies can help us better understand if QRIS ratings are linked to actual quality measures and what the critical components of these complex systems may be. Unfortunately, many validation studies were conducted early in the history of QRIS (e.g., concurrent with implementation or in newly implemented systems) and suffered from methodological and design issues, such as difficulty recruiting providers, nonexperimental study designs,ix and small samples.17,18,22 Relevant to this review, many validation studies also focused on the preschool population, rather than on infants and toddlers. Two recent syntheses of validation studies can help provide some insight into the association between QRIS and quality, but these results should not be interpreted as causal due to design concerns with individual state validation studies.
A synthesis of nine QRIS validation studies conducted between 2013 and 2017 found positive associations between QRIS ratings and at least one measure of observed quality, although most states found mixed results.18 Among the four state studies examining infant and toddler classrooms, results were also mixed, although three states found at least one positive association between QRIS ratings and a measure of observed quality. A synthesis of Race to the Top-Early Learning Challenge (RTT-ELC) state validation studies found that “higher-rated programs scored higher on independent assessments of program quality than lower-rated programs” (p. 17).17 These findings were similar to those of the earlier synthesis: both reports concluded that overall ECE program quality was not high; that QRIS distinguish between high- and low-quality programs, but not consistently between individual rating levels; and that differences between high- and low-rated programs were small.17,18 These syntheses included some of the same state validation studies, so consistency in results is to be expected.
One longitudinal study examined the impact of QRIS implementation on the supply of child care labor.B This study found that QRIS implementation increased new hires (by approximately 4 percent for the full sample), but also increased job separations (by 2 to 3 percent per quarter); these effects essentially negate each other. The study also found that QRIS enactment led to small increases in monthly earnings for child care workers (0.7 percent) and new hires (0.8 percent). The size of these labor force and earnings effects were relatively consistent in subanalyses by worker educational attainment and age, but tended to be somewhat larger for less-skilled and younger workers. Although it is unclear from these analyses if these results are beneficial, to the extent that wellbeing is improved within the child care workforce, children may benefit if this leads to better caregiver-child interactions. Future research should address these issues to better understand the impact of QRIS on the child care workforce.
- An impact is considered statistically significant if p<0.05.
- Competition was defined by the authors as the number of other ECE programs within 5 miles of a program at the initial observation year; high competition was defined as competition above the median level. For more information, see study A (Bassok, Dee, and Latham, 2019).
- The author defines economically disadvantaged families as those in which the mother has a high school degree or less; economically advantaged families are defined as families in which the mother has more than a high school degree. For more information, see study B (Herbst, 2018).
- Observed quality was measured by the Environment Rating Scale scores (e.g., ITERS, ECERS).
- Many state validation studies included limited covariates as controls in statistical models and very few states controlled for baseline equivalence of groups.
