Evaluating Cross-Cutting Approaches to Chronic Disease Prevention and Management: Developing a Comprehensive Evaluation

Evaluating Cross-Cutting Approaches to Chronic Disease Prevention and Management: Developing a Comprehensive Evaluation

SPECIAL TOPIC — Volume 14 — December 7, 2017

Marla Vaughan, MPH1; Jan Jernigan, PhD1; Seraphine Pitt Barnes, PhD, MPH, CHES1; Pat Shea, MPH, MA1; Rachel Davis, MPH1; Stephanie Rutledge, PhD, MA1(View author affiliations)

Suggested citation for this article: Vaughan M, Jernigan J, Barnes SP, Shea P, Davis R, Rutledge S. Evaluating Cross-Cutting Approaches to Chronic Disease Prevention and Management: Developing a Comprehensive Evaluation. Prev Chronic Dis 2017;14:160499. DOI: http://dx.doi.org/10.5888/pcd14.160499.

PEER REVIEWED

On This Page

• Abstract
• Background
• Evaluation Approach
• National Evaluation
• Reporting of Performance Measures
• Evaluations by the States
• Providing Evaluation Technical Assistance
• Dissemination and Use of Evaluation Findings
• Conclusion
• Acknowledgments
• Author Information
• References
• Table
• Appendix

Abstract

We provide an overview of the comprehensive evaluation of State Public Health Actions to Prevent and Control Diabetes, Heart Disease, Obesity and Associated Risk Factors and Promote School Health (State Public Health Actions). State Public Health Actions is a program funded by the Centers for Disease Control and Prevention to support the statewide implementation of cross-cutting approaches to promote health and prevent and control chronic diseases. The evaluation addresses the relevance, quality, and impact of the program by using 4 components: a national evaluation, performance measures, state evaluations, and evaluation technical assistance to states. Challenges of the evaluation included assessing the extent to which the program contributed to changes in the outcomes of interest and the variability in the states’ capacity to conduct evaluations and track performance measures. Given the investment in implementing collaborative approaches at both the state and national level, achieving meaningful findings from the evaluation is critical.

 Top

Background

State Public Health Actions to Prevent and Control Diabetes, Heart Disease, Obesity and Associated Risk Factors and Promote School Health (State Public Health Actions) is a program funded by the Centers for Disease Control and Prevention (CDC) to support the statewide implementation of strategies that promote health and prevent and control multiple chronic diseases and their risk factors (1). In the program, CDC partners with state health departments to address the 4 domains of chronic disease prevention: 1) epidemiology and surveillance, 2) environmental approaches, 3) health care system interventions, and 4) community programs linked to clinical services (2). Four divisions in the National Center for Chronic Disease Prevention and Health Promotion (NCCDPHP) at CDC, the Division of Diabetes Translation (DDT), Division for Heart Disease and Stroke Prevention (DHDSP), Division of Nutrition, Physical Activity, and Obesity (DNPAO), and the School Health Branch (SHB) in the Division of Population Health, have collaborated to fund, implement, and evaluate State Public Health Actions.

Funding from the State Public Health Actions program has provided state health departments with an opportunity to address chronic diseases within their state at the individual level, such as by promoting health care interventions, and at the population level by developing policies and creating environments that promote health. This article is a companion to “Overview of State Public Health Actions to Prevent and Control Diabetes, Heart Disease, Obesity and Associated Risk Factors and Promote School Health,” which was published December 7, 2017, in Preventing Chronic Disease (3). Here we describe the approach taken to evaluate the collaborative, complex State Public Health Actions program to ensure its accountability by demonstrating health outcomes, assisting states and CDC in improving the implementation of programs, and expanding the body of practice-based evidence by identifying successful and replicable strategies.

 Top

Evaluation Approach

Because State Public Health Actions is an innovative, cross-cutting program, it requires robust, multifaceted methods to evaluate it effectively. Although each of the 4 divisions conducted evaluations of their programs before State Public Health Actions, they took different approaches based on various factors, including the size and scale of the programs, the types of strategies being implemented (eg, policy, systems, and environmental changes, community-based and clinical interventions), and types of stakeholders engaged. Although evaluating large, federally funded public health programs is always challenging, the unique approach of State Public Health Actions compounded these challenges. Specifically, for State Public Health Actions there was a need to demonstrate to stakeholders its impact on disease-specific outcomes while implementing cross-cutting activities. Other challenges included coordinating across multiple chronic disease areas at the state and CDC level, accessing new partners and data sources, and the need to report performance measures that focused solely on outcomes.

These complex challenges required evaluators from each division to work together to design a comprehensive, multitiered approach to address the relevance, quality, and impact of State Public Health Actions. To begin, the evaluators followed standard practice by creating a logic model to highlight the inputs, activities, strategies, and outcomes of State Public Health Actions (Figure 1). The evaluators then designed the evaluation to assess and document the processes and outcomes of the program and to highlight how the implementation of the evidence-based strategies would lead to intended outcomes. The evaluation also examines the potential benefits and challenges of State Public Health Action’s approach of improving individual disease outcomes through the use of cross-cutting strategies.

Figure 1.
Program logic model for State Public Health Actions to Prevent and Control Diabetes, Heart Disease, Obesity and Associated Risk Factors and Promote School Health program. Abbreviations: A1c, glycated hemoglobin A1c ; CVD, cardiovascular disease; DDT, Division of Diabetes Translation; DHDSP, Division for Heart Disease and Stroke Prevention; DNPAO, Division of Nutrition, Physical Activity, and Obesity; DPH, Division of Population Health, School Health Branch; DSME, diabetes self-management education; K–12, kindergarten through 12th grade. [A text version of this figure is also available.]

The evaluation approach includes 4 primary components: conducting a national evaluation that assesses progress across all states; reporting by the states of performance measures to track the reach of individual strategies and disease-specific outcomes; conducting evaluations by the states to assess and improve programs at the state level and understand the facilitators of, and barriers to, program implementation; and providing evaluation technical assistance to enhance the capacity for evaluation at the local level and improve the reporting of data. CDC developed a structure to plan and implement the 4 components of the evaluation, which is to be carried out over a 5-year period. DHDSP was chosen to serve as the functional lead for evaluation in the administrative and management structure (3), while all 4 divisions identified a representative to act in a leadership role for evaluation-related decisions and the development of plans, processes, and guidance documents.

Four distinct evaluation workgroups were created to 1) oversee and implement the national evaluation; 2) collect, analyze, report, and provide guidance on performance measures; 3) provide guidance on planning and reporting the individual states’ evaluations; and 4) give technical assistance to build evaluation capacity among the states and ensure successful implementation of the 4 components of the evaluation (Figure 2). For each component, the workgroup members identified and addressed both common and unique challenges to developing and implementing that component.

Figure 2.
Components of state public health actions evaluation, State Public Health Actions to Prevent and Control Diabetes, Heart Disease, Obesity and Associated Risk Factors and Promote School Health (State Public Health Actions). [A text version of this figure is also available.]

 Top

National Evaluation

The national evaluation is the key mechanism for understanding the progress, achievements, and challenges of the overall State Public Health Actions program. This component aims to not only assess the impact, effectiveness, and efficiencies of the program but also to determine the degree to which cross-cutting approaches affect outcomes for health promotion and chronic disease prevention.

Development

The national evaluation workgroup used the CDC evaluation framework (4) to guide the evaluation’s design and methods and to provide context for the findings. The workgroup developed 4 overarching evaluation questions that will be assessed throughout the 5-year span of State Public Health Actions:

1. To what extent has the program been effective, as indicated by progress toward the intended accomplishments and outcomes?
2. To what extent, if any, have state programs gained efficiencies (eg, in infrastructure, management, financial performance) through the implementation of the approach of State Public Health Actions?
3. To what extent, if any, has CDC gained efficiencies by combining the efforts of 4 of its divisions within NCCDPHP?
4. What promising and innovative strategies that could be replicated by state programs have been found effective and efficient?

The 5-year national evaluation plan comprises an examination of the collaborations, efficiencies, activities, and accomplishments of all awardees; an in-depth analysis of the implementation and effectiveness of specific strategies; and an examination of the efficiency of CDC’s internal coordination and the effectiveness of technical assistance to awardees.

Implementation

The national evaluation seeks to assess the implementation and outcomes of the program across all 50 states and the District of Columbia. Because the grantees are at different stages of implementation throughout the program period and because there are several potential focus areas and priorities, CDC evaluators develop an evaluation protocol for each year that incorporates programmatic priorities and subevaluation questions guided by the 4 overarching evaluation questions. Once a protocol is drafted, CDC obtains feedback from evaluators, states, CDC partners, and program staff members to ensure that the protocol is feasible and aligns with stakeholder needs. CDC relies on the primary and secondary collection of both quantitative and qualitative data. Specific data collection and analyses include conducting quantitative analyses of data on state performance measures; fielding surveys to assess the efficiency and collaboration of CDC and the states; implementing focus groups and key informant interviews; and reviewing training and technical assistance notes, state work plans, annual performance reports, and evaluation plans and reports written by the states. While the nature of evaluating a large program conducted by all the states limits the ability to attribute outcomes to the program because of the lack of comparison groups, the multiple sources of data collected allow for data triangulation to identify and assess trends and common themes in state progress. The evaluation of State Public Health Actions strives to show the reach of the program, methods of implementation, its synergy and coordination, and its impact in terms of contributing to improvements in disease-specific outcomes.

 Top

Reporting of Performance Measures

Performance measures provide accountability by answering questions about what was achieved or, conversely, not achieved (5). For State Public Health Actions, performance measures provide key data for reporting outcomes to stakeholders and provide quantitative data that is incorporated into the national evaluation for assessing short-term and intermediate progress across each strategy being implemented by states. There were, however, several challenges to the implementation of reporting on performance measures. For example, previous programs funded by the 4 divisions within NCCDPHP did not require the reporting of outcome performance measures, many of the strategies that states are implementing as part of State Public Health Actions are new, many states were required to engage with new partners (eg, health care systems) and, as a group, states had varying capacity to collect and report measures and access data sources.

Development

To develop the performance measures, leadership from the DHDSP, DDT, DNPAO, and SHB reviewed the purpose and intended outcome of each strategy in the logic model (Figure 1) to determine the areas and type of performance measures needed. Each division pulled most of its measures from previously developed and pilot-tested measures. For example, of the measures selected by the DHDSP, all but one were chosen from a prior multiyear project working with stakeholders to develop a menu of indicators for the control of high blood pressure. For State Public Health Actions, each performance measure aligns with a strategy or intervention that focuses on outcomes relevant to specific disease outcomes and the interests of stakeholders.

To ensure the reporting of high-quality data and to build capacity to collect and report performance measures at the state level, CDC developed guidance documents and provided webinars related to calculating the reach of the intervention and developing baseline and target values. CDC also developed operational definitions, also called profiles, for each of the performance measures; each 2-page profile defines and describes the purpose of the measure, unit of analysis, target population, and setting. It also describes how to calculate the measure, including the data sources to be used and the frequency of data collection, and provides additional resources and references (Appendix). CDC worked with the states to review and finalize the profiles. Once the profiles were disseminated, a tip sheet and considerations for reporting were provided to assist states with the reporting of data on performance measures.

Implementation

In 2013, the states reported initial baselines and targets for strategy-specific performance measures. From 2015 until the end of the program (2018), states are required to report targets and annual progress for performance measures associated with their selected strategies and interventions. The states use a CDC-provided template that includes the measures required for a particular strategy, the prepopulated baseline (based on earlier reporting), targets for the current year and year 5, and actual data for the current year. Depending on the measure, the states report the data as a number, rate, percentage, or numerator and denominator. They also report the data source(s) and, as needed, provide notes that would give context to CDC for understanding the data during its analysis.

Each year, CDC’s performance measure workgroup assesses the quality of the state-reported data on these measures and the appropriateness of the analyses conducted (earlier, CDC had developed criteria for data quality and determined the type of analysis to be used for each performance measure). Data analysts at CDC use the criteria and inclusion and exclusion criteria for final cleaning and analysis of the data. Assessment of data quality also helps determine the performance measures for which the data are of sufficient quality to include them in the national evaluation and identifies measures that have widespread issues with quality. In addition, the process enables the provision of appropriate technical assistance to remedy those quality issues.

 Top

Evaluations by the States

The evaluations performed by the individual states aim to provide data relevant to those states while also contributing to the national evaluation. States use data for purposes such as continuous program improvement and being responsive to local stakeholders. CDC uses these data for purposes such as synthesizing information on common strategies that states are using to identify and engage partners. This information provides a complete picture of progress on the performance measures, aids understanding of facilitators and barriers to implementation, and identifies potential best practices.

Development

Acknowledging the difficulties of aggregating results from evaluations conducted by the states and other challenges in reporting their data, including varying capacities and a lack of standard data collection methods, CDC developed a set of core process evaluation questions and division-specific core outcome evaluation questions to facilitate the aggregation and cross-analysis of findings from the states for the national evaluation. States were also encouraged to develop additional evaluation questions and indicators to meet their own evaluation needs. The core process evaluation questions were related to their coordination with critical partners, their work across areas of chronic disease, their type of organizational structure, and their increased efficiencies obtained. The division-specific core outcome evaluation questions were related to progress made and both the barriers that they encountered and facilitators that aided selected strategies (Table). To reduce their burden and to focus the evaluation, states were required to evaluate only 1 strategy for each CDC division. States could also select whether they were in the adoption or implementation phase of the strategy. CDC designed a template that states could use to provide background information on the particular approach and strategies of the program, the selection of activities implemented, settings and target populations, key stakeholders and partners involved in the program planning and implementation, indicators developed to monitor progress toward achieving an answer to the process evaluation question, and the synergistic approach used to implement the program.

The division-specific outcome evaluation sections of the template included additional information on barriers and facilitators, an indicator table, a findings and results section for each disease-specific core outcome evaluation question, and a plan to disseminate the results of their evaluation to internal and external stakeholders.

Implementation

States annually submit to CDC their plans for evaluation and the evaluation results obtained. Data are stored on an internal SharePoint (Microsoft Corp) site, where CDC evaluators review the data and determine how best to synthesize the data and pull out common themes. The data are triangulated with other data for the national evaluation and summarized. The information provided in the evaluation plans enables CDC evaluation technical assistance providers to understand the proposed methods and, thus, more effectively assist the states in conducting their own evaluations. Technical assistance providers can also provide information to the program team about common barriers and facilitators, which can be used to develop trainings and technical assistance to support and improve program implementation.

 Top

Providing Evaluation Technical Assistance

The national evaluation, reporting of performance measures, and state evaluations all rely on data received by the states. Because the states have varying levels of capacity for evaluation, CDC must provide technical assistance to ensure effective reporting to the agency and to make sure that the state-level evaluations are providing information relevant to improving programs and meeting the needs of stakeholders. Because 4 divisions at CDC support the work of the state health departments, with each bringing its own body of expertise as it pertains to implementing disease-specific interventions, evaluators from all 4 of these divisions have worked collaboratively as part of regional teams that support the states to evaluate various strategies they are implementing.

Development

CDC’s technical assistance plan for the 5-year evaluation consists of evaluation capacity assessments, annual reviews of documents, the development of evaluation tools and resources, and other forms of technical assistance to the states. Evaluation capacity assessments were performed in the first year to understand the capacity of each state to conduct evaluations and to identify needs for technical assistance and types of trainings and resources that were needed for states to meet evaluation requirements. Ongoing assessments are also conducted to identify facilitators of and barriers to developing evaluation plans and tools, identifying appropriate indicators and data sources, and conducting data analysis for annual evaluation reporting. Evaluators at CDC maintain regular communication with evaluators at the state level and assist them with developing their evaluation plans, collecting and reporting performance measures, and reporting the results of their evaluations. CDC evaluators also assist both the states and project officers at CDC through the annual review of work plans, yearly performance reports, and evaluation reports to ensure that states are aligning activities with performance measures and accurately reporting data.

Implementation

Evaluation resources made available to the states by CDC include training opportunities such as cross-state peer-learning communities, evaluation guidance documents, sample data collection tools, and evaluation plan and report templates. The peer-learning communities meet monthly for presentations and facilitated discussion. In addition, there is a listserve on which community members can pose questions to other members about their experiences implementing their evaluations and can share information and documents. Additional evaluation guidance documents and tools developed by CDC include templates and helpful hints documents to support the states’ work throughout various phases of the program.

Consistent and coordinated communication with states and among CDC staff is important to reaching the goal of providing effective technical assistance. To standardize technical assistance, evaluators developed a guide designed to support consistent monitoring and documentation of evaluation technical assistance needs for a state during evaluation plan implementation and performance measure reporting. In collaboration with project officers, evaluators at CDC communicate with states at least monthly through regular calls with the regional team and ad hoc, evaluation-specific follow-up calls and email communication. Internally, CDC uses a performance-monitoring database to document progress on performance and evaluation activities and to track communication and follow-up activities between the states and CDC’s evaluation staff and project officers.

 Top

Dissemination and Use of Evaluation Findings

CDC regularly disseminates findings related to the evaluation of State Public Health Actions to various stakeholders, including internal and external partners as well as the general public, through reports, executive summaries or briefs, presentations, and journal publications. Reports internal to CDC are used to understand how states are implementing programs and how well CDC is providing technical assistance to states and coordinating across divisions. Briefing documents, such as the State Public Health Actions Year 3 Performance Measures Snapshots (6), and the DNPAO state snapshots website (7), which report on highlights at the state level, are used to provide information on the program’s priorities and offer succinct outcomes that are relevant to stakeholders. Findings are also prepared for national partners and Congress to demonstrate accountability and program impact.

Presentations of findings are delivered internally to the CDC staff and externally to state health departments’ staff and other public health practitioners. For example, presentations were made at a meeting of grantees in Atlanta, Georgia, and to various diverse audiences at national conferences, such as those that were held by the American Public Health Association and the American Evaluation Association (8). Evaluation methods and findings obtained are also being shared through journal articles written by the CDC staff and state representatives (3, 9–11). In addition, CDC provides assistance to states in writing journal articles and finding strategies for dissemination.

 Top

Conclusion

The approach to the evaluation of State Public Health Actions is intended to demonstrate the impact of the overall program while capturing unique cross-cutting aspects of the program and the disease-specific outcomes. Lessons learned and key findings from the national evaluation, performance measures data, and evaluations conducted by the states will be summarized throughout the 5 years of the program to assist with ongoing program improvement, report progress to stakeholders, identify successful strategies, and inform future decisions on funding. While the comprehensive evaluation strives to evaluate efficiency, effectiveness, and impact at the state and national levels, it faces numerous challenges.

Evaluations of large public health programs are difficult to conduct, with one of the big challenges being the inability to attribute successes or shortfalls wholly to the program, because there are often confounding factors, a lack of comparison groups, long time frames, or multiple interventions going on at once. The development and use of performance measures to assess outcomes for federal programs is also challenging because of issues such as the complexity of public health problems, which may have multiple determinants or outcomes that may take several years to achieve; the decentralized implementation of public health programs; and measurement issues related to a lack of reliable, timely, and consistent data sources (5). Also, to successfully aggregate standardized measures, it would be ideal, but not realistic, for the states to have similar capacities to access, collect, analyze, and report data. Finally, federal agencies are challenged by the limited resources available to provide state health departments with consistent and intensive technical assistance with evaluation to help them with collecting and reporting performance measures and evaluating their programs.

These common challenges are clearly applicable to State Public Health Actions, with the added complexity of working across multiple topic areas and attempting to evaluate cross-cutting strategies when most state health departments and CDC operate within distinct disease or topic areas. Each topic area has discrete funding streams and must demonstrate effectiveness in achieving outcomes for each of these areas. The State Public Health Actions program also expands funding to more states than were previously funded by each division, and oversight and management requires complex coordination. To accurately describe the implementation and outcomes of State Public Health Actions, assessing collaboration and coordination across topic areas at the state level and at CDC is an important part of the evaluation. Surveys, focus groups, key informant interviews, and results obtained from evaluations conducted by the states using a standard template are employed to highlight this unique aspect of State Public Health Actions. CDC evaluators provide proactive and intensive technical assistance to address challenges, but the complex, cross-topic, structure of technical assistance can be time-consuming.

Although there are challenges and limitations with the evaluation of State Public Health Actions, given CDC’s substantial investment in testing collaborative approaches and working across domains, striving to achieve meaningful findings from the evaluation is critical. Subsequent articles will highlight results achieved by the program and promising practices that can be implemented broadly.

 Top

Acknowledgments

We are grateful for the contributions and hard work provided by members of the CDC evaluation workgroups across the Division for Heart Disease and Stroke Prevention; Division of Diabetes Translation; Division of Nutrition, Physical Activity, and Obesity; and the School Health Branch in the Division of Population Health. We also thank those state health department staff members who contributed to the development of the performance measure profiles. No financial support was received for this work.

The findings and conclusions in this report are those of the authors and do not necessarily represent the official position of CDC.

 Top

Author Information

Corresponding Author: Marla Vaughan, MPH, Centers for Disease Control and Prevention, Division for Heart Disease and Stroke Prevention. 4770 Buford Hwy, Mailstop F-75, Atlanta, GA 30341-3717. Telephone: 770-488-4826. Email: mhv1@cdc.gov.

Author Affiliations: 1Division for Heart Disease and Stroke Prevention, Centers for Disease Control and Prevention, Atlanta, Georgia.

 Top

References

1. Centers for Disease Control and Prevention. State Public Health Actions to Prevent and Control Diabetes, Heart Disease, Obesity and Associated Risk Factors and Promote School Health. https://www.cdc.gov/chronicdisease/about/state-public-health-actions.htm. Accessed May 11, 2017.
2. Bauer UE, Briss PA, Goodman RA, Bowman BA. Prevention of chronic disease in the 21st century: elimination of the leading preventable causes of premature death and disability in the USA. Lancet 2014;384(9937):45–52. CrossRef PubMed
3. Park BZ, Cantrell L, Hunt H, Farris RP, Schumacher P, Bauer UE. Overview of State Public Health Actions to Prevent and Control Diabetes, Heart Disease, Obesity and Associated Risk Factors and Promote School Health. Prev Chronic Dis 2017;14:160437. . http://www.cdc.gov/pcd/issues/2017/16_0437.htm. CrossRef
4. Centers for Disease Control and Prevention. Framework for program evaluation in public health. MMWR Recomm Rep 1999;48(RR-11):1–40. PubMed
5. DeGroff A, Schooley M, Chapel T, Poister TH. Challenges and strategies in applying performance measurement to federal public health programs. Eval Program Plann 2010;33(4):365–72. CrossRef PubMed
6. Centers for Disease Control and Prevention. State Public Health Actions (1305). Year 3 performance measures snapshot. Atlanta (GA): Centers for Disease Control and Prevention. https://wwwdev.cdc.gov/dhdsp/docs/1305-y3-performance-measures-snapshot.pdf. Accessed June 5, 2017.
7. Division of Nutrition, Physical Activity, and Obesity, National Center for Chronic Disease Prevention and Health Promotion, Centers for Disease Control and Prevention. Funding by state. 2017. https://www.cdc.gov/nccdphp/dnpao/state-local-programs/funding.html. Accessed November 13, 2017.
8. Jernigan JD. Evaluating cross-cutting approaches to addressing chronic disease: a comprehensive evaluation of program effectiveness and efficiency. Chicago (IL): American Evaluation Association; 2015.
9. Mensa-Wilmot Y, Bowen S, Rutledge S, Morgan JM, Bonner T, Farris K, et al. Early results of states’ efforts to support, scale, and sustain the National Diabetes Prevention Program. Prev Chronic Dis 2017;14:170478. . http://www.cdc.gov/pcd/issues/2017/17_0478.htm. Accessed December 7, 2017. CrossRef
10. Geary NA, Dooyema CA, Reynolds MA. Supporting obesity prevention in statewide quality rating and improvement systems: a review of state standards. Prev Chronic Dis 2017;14:160518. . http://www.cdc.gov/pcd/issues/2017/16_0518.htm. Accessed December 7, 2017. CrossRef
11. Pitt Barnes S, Skelton-Wilson S, Cooper A, Merlo C, Lee S. Early outcomes of State Public Health Actions’ school nutrition strategies. Prev Chronic Dis 2017;14:170106. . http://www.cdc.gov/pcd/issues/2017/16_0518.htm. Accessed December 7, 2017. CrossRef