Data Use in Education Networks
Principal Investigator(s): View help for Principal Investigator(s) Michelle Shumate, Northwestern University
Version: View help for Version V1
| Name | File Type | Size | Last Modified |
|---|---|---|---|
|
|
text/x-rsrc | 5.6 KB | 11/14/2025 12:04:PM |
|
|
application/vnd.openxmlformats-officedocument.spreadsheetml.sheet | 25.8 KB | 11/14/2025 12:04:PM |
|
|
application/vnd.openxmlformats-officedocument.spreadsheetml.sheet | 33.5 KB | 11/14/2025 12:04:PM |
|
|
application/vnd.openxmlformats-officedocument.spreadsheetml.sheet | 16.6 KB | 11/14/2025 12:04:PM |
Project Citation:
Shumate, Michelle. Data Use in Education Networks. Ann Arbor, MI: Inter-university Consortium for Political and Social Research [distributor], 2025-11-14. https://doi.org/10.3886/E240321V1
Project Description
Summary:
View help for Summary
Nonprofits and public organizations may benefit from using evaluation data to improve programs and managerial decisions. Guided by organizational learning and institutional theory, this study investigates how partners, coalitions, and technical assistance organizations influence these organizations’ professional use of evaluation data within collaborative networks. Data were collected from 449 organizations embedded in an ego-centric network of partnerships and purpose-oriented coalitions on education reform.
Funding Sources:
View help for Funding Sources
Army Research Office
Scope of Project
Subject Terms:
View help for Subject Terms
network;
education organizations;
data use
Geographic Coverage:
View help for Geographic Coverage
United States
Time Period(s):
View help for Time Period(s)
2018 – 2020
Data Type(s):
View help for Data Type(s)
survey data
Methodology
Response Rate:
View help for Response Rate
For the organizational surveys, the response rate was 51%.
Sampling:
View help for Sampling
This study uses interview and survey data from 26 cross-sector coalitions to improve educational outcomes. These initiatives were sampled through the following procedures: 1). We compiled a database of collective impact coalitions working on education by identifying member affiliations with national networks (e.g., StriveTogether, Collective Impact Forum, Ready by 21, and the Campaign for Grade-Level Reading). Collective impact coalitions follow a structured partnership model where partners are formed under a common agenda, share measurement data, engage in mutually reinforcing activities, and communicate regularly under the leadership of a backbone organization (Kania & Kramer, 2011). We successfully recruited 13 collective impact coalitions. 2). We used the following matching criteria to recruit another 13 coalitions that were less structured compared to the first half of the sample: geographic (e.g., population density, coverage area, and the number of school districts or municipalities), demographic (e.g., race, and poverty rate), and labor market (e.g., unemployment rate, and median income). All the sampled coalitions were in 13 states.
The matching process was conducted to minimize institutional differences across educational contexts, and to enable the research to compare the outcomes of networks of different design features while controlling for potential confounding factors. This matching technique helps strengthen the internal validity of the study by reducing the likelihood that observed differences in outcomes are attributable to underlying differences in the coalition environments. The use of matched pairs from the same state also ensured consistency in the policy and regulatory context affecting the coalitions, further enhancing the ability to isolate the impact of the coalition activities on educational improvements.Coalitions in our sample varied on several factors: the model of collaboration (collective impact or not), year of establishment (2006 to 2015), location, size (M = 36, S.D. = 10.5), composition (what types of organizations are in the network), and social network structure. Notably, nonprofits, education organizations (e.g., K-12 public schools, early childhood schools), and government agencies (e.g., the Department of Public Health, Department of Children and Families, Department of Labor, and Department of Social Services) made up the majority of each of these coalitions, with less representation from the business sector. There is no co-membership across coalitions.
The matching process was conducted to minimize institutional differences across educational contexts, and to enable the research to compare the outcomes of networks of different design features while controlling for potential confounding factors. This matching technique helps strengthen the internal validity of the study by reducing the likelihood that observed differences in outcomes are attributable to underlying differences in the coalition environments. The use of matched pairs from the same state also ensured consistency in the policy and regulatory context affecting the coalitions, further enhancing the ability to isolate the impact of the coalition activities on educational improvements.Coalitions in our sample varied on several factors: the model of collaboration (collective impact or not), year of establishment (2006 to 2015), location, size (M = 36, S.D. = 10.5), composition (what types of organizations are in the network), and social network structure. Notably, nonprofits, education organizations (e.g., K-12 public schools, early childhood schools), and government agencies (e.g., the Department of Public Health, Department of Children and Families, Department of Labor, and Department of Social Services) made up the majority of each of these coalitions, with less representation from the business sector. There is no co-membership across coalitions.
Collection Mode(s):
View help for Collection Mode(s)
mixed mode;
telephone interview;
web-based survey
Scales:
View help for Scales
Professional use of outcome data in a coalition is the dependent variable of this study and is measured at the organizational level. We adopted the 6-item scale about professional use of evaluation data by Lee and Clerkin (2017). It draws on multiple theoretical frameworks to capture the implementation of performance measurement and inform better operational and management practices. We asked the respondents to describe the extent to which their organization engaged in data use activities on a 5-point frequency scale, with 1 marked as “not at all” and 5 marked as “to a very great extent.” Example statements include: My organization uses data to improve management decisions, improve strategic planning, improve services for clients, appraise employee performance, support budget decisions, and for outreach and public relations. A confirmatory factor analysis showed that all the items can be loaded to a single construct, demonstrating a high reliability (M = 3.98, S.D. = 0.63, Cronbach’s α = 0.77).
Coalition-level variables. Two coalition-level variables are measured using interview data with coalition leaders. They were corroborated with archival documents requested from the coalitions.First, coalitions that use data to support theories of change and have developed the technical and human resources needed to track and report their data have good data use practices. To determine the quality of coalition practice, we examined two sources of data: transcripts of interviews with coalition leads (specifically related to questions about data collection, data sharing, metrics used, and dissemination of reports to stakeholders; see Appendix A for the list of questions) and archival data. The first step in identifying the coding involved four undergraduate research assistants writing a detailed memo on each network based on the two data sources, following a template to address the following question: What things they are doing well? Where do they fall short, according to our criteria? Given examples and quotes to support your claims). Next, two of the co-authors evaluated all the memos, coded each network individually as ordinal with four magnitudes, discussed the coding, and developed schemes to collectively review and confirm the results.Second, some coalitions received evaluation technical assistance. We operationalized this variable by examining whether a coalition is affiliated with a national organization that promotes strong coalition data use (n = 14). These umbrella networks also provide technical assistance in adopting a particular model and data use practices. The following were identified: Campaign for Grade-Level Reading, Collective Impact Forum, Communities that Care, My Brother’s Keeper, Ready by 21, StriveTogether, Ready by 21, and United Way.We also included several control variables at the coalition level. Coalition age was measured as the years since the coalition was founded (M = 8.85, S.D. = 6.60). Coalition size was measured as the number of organizational members (M = 34.42, S.D. = 24.12). The coalition model was measured as a categorical variable where 1 = collective impact and 0 = non-collective impact.
Organizational-level variables. First, the sector is measured to indicate whether an organization is a nonprofit (n = 283) or a public organization (n = 188). This variable was created to capture the potential differences between public organizations and nonprofits. Second, ego-centric network partner professional use of outcome data is measured by calculating the average score of all the nominated partners from one organization (M = 4.04, S.D. = .30, N = 449). Thus, each organization was given a score based on its portfolio of partners. Each organization in the network completed a survey that recorded its partnerships and level of professional data use. Ego-centric network questions asked the respondents to report with what organizations in the coalition network they collaborated with in the past three years on 7 activities. The ego-centric network data were used to identify which organizations were partners. Then, we averaged the professional data use of partner organizations to predict organizational data use.
Coalition-level variables. Two coalition-level variables are measured using interview data with coalition leaders. They were corroborated with archival documents requested from the coalitions.First, coalitions that use data to support theories of change and have developed the technical and human resources needed to track and report their data have good data use practices. To determine the quality of coalition practice, we examined two sources of data: transcripts of interviews with coalition leads (specifically related to questions about data collection, data sharing, metrics used, and dissemination of reports to stakeholders; see Appendix A for the list of questions) and archival data. The first step in identifying the coding involved four undergraduate research assistants writing a detailed memo on each network based on the two data sources, following a template to address the following question: What things they are doing well? Where do they fall short, according to our criteria? Given examples and quotes to support your claims). Next, two of the co-authors evaluated all the memos, coded each network individually as ordinal with four magnitudes, discussed the coding, and developed schemes to collectively review and confirm the results.Second, some coalitions received evaluation technical assistance. We operationalized this variable by examining whether a coalition is affiliated with a national organization that promotes strong coalition data use (n = 14). These umbrella networks also provide technical assistance in adopting a particular model and data use practices. The following were identified: Campaign for Grade-Level Reading, Collective Impact Forum, Communities that Care, My Brother’s Keeper, Ready by 21, StriveTogether, Ready by 21, and United Way.We also included several control variables at the coalition level. Coalition age was measured as the years since the coalition was founded (M = 8.85, S.D. = 6.60). Coalition size was measured as the number of organizational members (M = 34.42, S.D. = 24.12). The coalition model was measured as a categorical variable where 1 = collective impact and 0 = non-collective impact.
Organizational-level variables. First, the sector is measured to indicate whether an organization is a nonprofit (n = 283) or a public organization (n = 188). This variable was created to capture the potential differences between public organizations and nonprofits. Second, ego-centric network partner professional use of outcome data is measured by calculating the average score of all the nominated partners from one organization (M = 4.04, S.D. = .30, N = 449). Thus, each organization was given a score based on its portfolio of partners. Each organization in the network completed a survey that recorded its partnerships and level of professional data use. Ego-centric network questions asked the respondents to report with what organizations in the coalition network they collaborated with in the past three years on 7 activities. The ego-centric network data were used to identify which organizations were partners. Then, we averaged the professional data use of partner organizations to predict organizational data use.
Unit(s) of Observation:
View help for Unit(s) of Observation
organizations,
coalitions
Related Publications
Published Versions
Report a Problem
Found a serious problem with the data, such as disclosure risk or copyrighted content? Let us know.
This material is distributed exactly as received from the data depositor. As of April 2026, depositors are required to submit study materials in accessible formats. ICPSR has not reviewed, checked, or processed this material. For additional information about the study, please contact the investigator(s) directly. If you have questions about the accessibility of materials distributed by ICPSR or require further assistance, please visit ICPSR's Accessibility Center.