miércoles, 31 de julio de 2019

Estimating the Relative Impact of Clinical and Preventive Community-Based Interventions: An Example Based on the Community Transformation Grant Program

Estimating the Relative Impact of Clinical and Preventive Community-Based Interventions: An Example Based on the Community Transformation Grant Program

PCD logo

Estimating the Relative Impact of Clinical and Preventive Community-Based Interventions: An Example Based on the Community Transformation Grant Program

Benjamin Yarnoff, PhD1; Christina Bradley, MA1; Amanda A. Honeycutt, PhD1; Robin E. Soler, PhD2; Diane Orenstein, PhD2 (View author affiliations)

Suggested citation for this article: Yarnoff B, Bradley C, Honeycutt AA, Soler RE, Orenstein D. Estimating the Relative Impact of Clinical and Preventive Community-Based Interventions: An Example Based on the Community Transformation Grant Program. Prev Chronic Dis 2019;16:180594. DOI: http://dx.doi.org/10.5888/pcd16.180594external icon.
PEER REVIEWED
Summary
What is already known on this topic?
Previous work demonstrated the potential long-term impact of clinical and community interventions to prevent chronic disease. However, that work considered only hypothetical interventions that may not accurately reflect the feasibility of implementation in a real-world setting.
What is added by this report?
We examined the potential 10- and 25-year impact of clinical and community interventions to prevent chronic disease as they were implemented under the Community Transformation Grant program.
What are the implications for public health practice?
Results support public health practitioners in strategic planning for chronic disease prevention.

Abstract

Introduction
Public health focuses on a range of evidence-based approaches for addressing chronic conditions, from individual-level clinical interventions to broader changes in policies and environments that protect people’s health and make healthy living easier. This study examined the potential long-term impact of clinical and community interventions as they were implemented by Community Transformation Grant (CTG) program awardees.
Methods
We used the Prevention Impacts Simulation Model, a system dynamics model of cardiovascular disease prevention, to simulate the potential 10-year and 25-year impact of clinical and community interventions implemented by 32 communities receiving a CTG program award, assuming that program interventions were sustained during these periods.
Results
Sustained clinical interventions implemented by CTG awardees could potentially avert more than 36,000 premature deaths and $3.2 billion in discounted direct medical costs (2017 US dollars) over 10 years and 109,000 premature deaths and $8.1 billion in discounted medical costs over 25 years. Sustained community interventions could avert more than 24,000 premature deaths and $3.4 billion in discounted direct medical costs over 10 years and 88,000 premature deaths and $9.1 billion in discounted direct medical costs over 25 years. CTG clinical activities had cost-effectiveness of $302,000 per death averted at the 10-year mark and $188,000 per death averted at the 25-year mark. Community interventions had cost-effectiveness of $169,000 and $57,000 per death averted at the 10- and 25-year marks, respectively.
Conclusion
Clinical interventions have the potential to avert more premature deaths than community interventions. However, community interventions, if sustained over the long term, have better cost-effectiveness.

Introduction

Public and private sector stakeholders have worked together for decades to prevent chronic disease, improve quality of life, and reduce medical costs and death associated with chronic disease. Evidence-based approaches for addressing chronic conditions range from individual-level clinical interventions addressing better identification and control of chronic diseases to broader changes in policies and environments around diet, physical activity, and smoking that make healthy living easier in a community. Public health now focuses on all these areas but recognizes that different interventions may have different potential impacts (1). Assessing the potential impact of interventions is challenging, because interventions take time to affect health and economic outcomes. As a result, only a small part of the impact of these interventions can be quantified in the first few years through observing program reach and initial impact on behaviors. Simulation modeling is a useful tool to extend the time horizon for assessing the potential long-term impact of clinical and community interventions.
Previous comparisons of clinical and community interventions generally considered policy change scenarios that may not have accurately reflected the real-world applications of these interventions (2). In this study, we simulated the potential 10- and 25-year impacts of 2 types of interventions as they were implemented as part of the Community Transformation Grant (CTG) program, a large multicommunity public health program funded by the Centers for Disease Control and Prevention (CDC) from 2011 through 2014.

Methods

The CTG program is a large-scale example of a program that supported the implementation of both clinical and community approaches to address chronic disease (3). CTG awardees were required to address at least one of the following focus areas: 1) increase options for tobacco-free living (eg, smoke-free policies for workplaces or multiunit housing), 2) promote and improve access to opportunities for active living and healthy eating (eg, working with partners to build bike paths and increase the availability of fruits and vegetables at corner stores), 3) increase use of clinical and community preventive services (eg, community health worker initiatives), and/or 4) expand access to healthy and safe physical environments (eg, Safe Streets initiatives) (4). After a competitive application process, CDC allocated $103 million to 61 state and local government agencies, tribes and territories, and nonprofit organizations in 36 states, covering 130 million people (3,5).
We used the CTG program as an example of a chronic disease prevention program to estimate the long-term potential health and economic outcomes of clinical and community interventions if they were sustained at the same level over time. We used information on the classifications of interventions that were conducted as part of the CTG program and their reach as inputs to the Prevention Impacts Simulation Model (PRISM) to estimate the potential long-term impact of clinical and community interventions. In the CTG program, reach was operationalized as the estimated number of people in the target population who had increased access to (eg, those living within 1 mile of a park), are protected by (eg, a workplace smoke-free policy), or are otherwise affected by (eg, patients covered by a community health worker program) an intervention (6).
PRISM is a computer simulation model containing mathematical equations that describe how risk factors interact to produce chronic disease and poor health outcomes and the impacts of various community and clinical interventions. PRISM calculates outcomes annually and cumulatively from 1990 through 2040 (7–10). PRISM was validated in several ways during its development and has been used to estimate the long-term impact of other community health programs, such as the Communities Putting Prevention to Work program (11) and public health prevention activities of the Los Angeles County Public Health Department (12).
PRISM includes a wide range of chronic disease–related intermediate outcomes that can be influenced by clinical and community intervention strategies. These strategies are represented in the model as “levers,” which reflect changes in the numbers of people reached by the strategy. Lever movement provides an estimate of the intent-to-treat population and not the population that changed their health behaviors as a result of lever movement. PRISM simulates the impact of lever movement on cardiovascular disease (CVD) risk behaviors, like smoking and physical activity in the reached population, by applying published estimates of the effect of increased access on health behavior. For example, building a park would increase the lever for access to physical activity spaces; PRISM then simulates the impact on physical activity for the portion of the reached population that used the park and increased their physical activity. These impacts on risk factors, in turn, reduce the prevalence of cardiovascular disease, pulmonary disease, lung cancer, and resulting deaths and costs. PRISM includes levers that address tobacco use; nutrition; physical activity; clinical care for preventing or mitigating hypertension, diabetes, and high cholesterol; and aspirin use. Most PRISM levers are represented by an index ranging from 0 (no implementation of the strategy) to 1 (optimal implementation of the strategy across the entire population). Because PRISM levers represent broad strategies to improve access, each PRISM lever can be moved by one or more evidence-based interventions. For example, the lever “Increasing access to physical activity spaces” can be moved by each of 10 interventions that are expected to produce a positive health outcome, including bike shares (13–17), safe-streets initiatives (18,19), parks (19-21), and joint-use agreements (22–24). Each evidence-based intervention was assigned to an intensity category (minimal, low, medium, and high) that represented its ability to move the lever for those reached by the intervention. The intensity category was assigned primarily on the basis of the impact of the intervention estimated in the literature. A list of all evidence-based interventions that can move each lever and details on the process of generating the list and assigning intensity categories are available in an online supplement (https://forio.com/app/cdc/prism/#/resources).
PRISM simulation outcomes reflect the impact of changes in lever settings compared with baseline trends (ie, no change from the status quo). Baseline PRISM levers were set to reflect a community’s public health environment pre-intervention (ie, before the CTG program began, in 2011). For example, when analyzing the impact of increasing access to physical activity spaces, we did not simply assume that a community started from a baseline access level of zero, but instead we used publicly available information about each community’s policies and environment to estimate the baseline level for each lever. Baseline lever settings were determined by reviewing data and literature on the existing environment for physical activity, nutrition, tobacco, and clinical services policies, such as city, county, and state information from the literature, and secondary data sources, such as the US Census Bureau and the National Health and Nutrition Examination Survey.

Translating CTG activities into PRISM inputs

Building on previous work (11), we used the RE-AIM (reach, effectiveness, adoption, implementation, maintenance) framework to translate CTG activities into PRISM lever inputs for simulation modeling (25–27). The evaluation focused on reach and effectiveness. To assess reach, we used awardee-submitted estimates of the number of people reached by their activities. CDC provided awardees with written guidance on estimating reach, including metrics, definitions, and potential data sources. Awardees were also encouraged to obtain technical assistance from CDC project officers when estimating intervention reach. Reach was operationalized as the estimated number of people in the target population who had increased access to (eg, those living within 1 mile of a park), are protected by (eg, a workplace smoke-free policy), or affected by (eg, patients covered by a community health worker program) an intervention (6). Determining reach included 1) documenting the setting where the intervention was implemented during the funding period, 2) using census data or setting-specific data (eg, school enrollment) to identify the population count for the setting where the intervention was implemented, and 3) aggregating data. If interventions were implemented in settings or populations that potentially overlapped, the overlap was estimated and accounted for in the aggregation process. Submitted reach estimates were reviewed and validated by trained CDC program officers, subject matter experts, and contractors by using census, school enrollment, and other local data sources.
Because reach was an intent-to-treat metric, not all people reached by the intervention will use the intervention or change their behavior as a result of access. The model incorporates effect-size estimates for the proportion reached whose use and behavior changes (ie, the estimated proportion of people in the target population who have increased access to or are protected by an intervention). Because PRISM is a population model representing the entire community, the denominator for proportional reach was the entire adult population, child population, or the total population of the targeted community as indicated by the US Census Bureau.
To assess effectiveness, we used information on the interventions completed by each awardee as reported in the annual reports submitted to CDC. A team of coders reviewed each awardee’s progress reports and determined which evidence-based interventions (https://forio.com/app/cdc/prism/#/resources) were conducted as part of each awardee activity. Each evidence-based intervention was assigned a categorical intensity that was used to determine the PRISM lever movement. For 20% of the awardee activities, a second coder performed a secondary review for quality control, and the 2 coders reconciled differences.
We computed the lever movement for each activity by taking the intensity of the interventions conducted as part of that awardee’s activity and multiplying by proportional reach. We then computed the total lever movement for each awardee by aggregating the lever movements for all of that awardee’s activities that affected each lever.
We estimated the impact of a subset of CTG activities that met our criteria for being evidence-based on premature deaths averted and medical costs saved after 10 and 25 years. The goal of the CTG program was to implement clinical and community interventions that could be sustained into the future with minimal further input, so we assumed that all interventions would be sustained at a constant level and that maintenance costs would be incurred for at least 10 and 25 years. We also examined the projected program implementation costs of awardee activities (including program maintenance costs) and the projected impact on risk factor management costs to calculate the total cost and cost-effectiveness of the CTG program. We constructed cost-effectiveness ratios as the sum of implementation costs and net medical costs (ie, risk factor management costs minus medical cost savings) divided by the incremental health gains of the program (ie, premature deaths prevented). We estimated the impact of each awardee’s activities overall and separately for clinical and community levers. We examined the median and range of the estimated impact across awardees and the aggregate for all CTG awardees. Medical costs were inflated to 2017 dollars by using the medical cost component of the Consumer Price Index (28). Future cost savings were discounted by 3% per year (29).
We conducted a probabilistic sensitivity analysis in which model parameters were varied across a distribution assumed on the basis of the literature (29) to estimate the lower and upper bounds of a 95% confidence interval for premature deaths averted, risk factor management costs, medical costs saved, and cost per premature death averted.

Results

Of the 61 CTG program awardees, 29 worked to build capacity for public health interventions and did not implement any interventions. The remaining 32 awardees implemented interventions that could be translated into PRISM levers and were included in this analysis. These awardees covered a population of 87 million people. They implemented clinical interventions reaching 19 million people, community tobacco interventions reaching 20 million people, community nutrition interventions reaching 37 million people, and community physical activity interventions reaching 26 million people.
CTG awardees worked on interventions that affected 21 different PRISM levers (Table 1). Thirty awardees worked on interventions targeting community PRISM levers (including nutrition, physical activity, and tobacco) and 12 awardees worked on interventions targeting clinical PRISM levers. Physical activity access was the lever addressed by the largest number of CTG awardees (20 awardees) and was increased an average of 20 percentage points across all awardees (ie, a 20 percentage-point increase in the number of people with access to places where they can engage in physical activity). Smoke-free multiunit housing was implemented by 18 awardees, with an average movement of 10 percentage points (ie, a 10 percentage-point decrease in multiunit housing complexes that permit smoking). Other levers moved in our analysis were fruit and vegetable access (12 awardees, average movement = 12 percentage points), physical activity promotion (15 awardees, average movement = 7 percentage points), physical activity requirements in schools (13 awardees, average movement = 11 percentage points), and workplace smoke-free policy (12 awardees, average movement = 23 percentage points). The most frequently implemented clinical interventions were related to improving quality care for people with diabetes (8 awardees, average movement = 12 percentage points), hypertension (7 awardees, average movement = 8 percentage points), and high cholesterol (11 awardees, average movement = 7 percentage points).
Results from PRISM simulations indicate that the projected 10-year impact (from 2015 through 2024) of clinical levers moved by CTG awardee activities would be more than 36,000 premature deaths averted, $3.2 billion in discounted medical cost savings, and $14.2 billion in risk factor management costs incurred (Table 2). The projected 10-year impact of community levers moved by CTG awardee activities would be nearly 25,000 premature deaths averted, $3.4 billion in discounted medical cost savings, and $3.0 billion in risk factor management costs incurred. The 10-year cost-effectiveness of CTG clinical activities was $302,000 per premature death prevented. The estimated cost-effectiveness of CTG community activities was $169,000 per premature death prevented.
The projected 25-year impact (from 2015 through 2039) of clinical levers moved by CTG awardee activities would be more than 109,000 premature deaths averted, $8.1 billion in discounted medical cost savings, and $28.4 billion in risk factor management costs incurred (Table 2). The projected 25-year impact of community levers moved by CTG awardee activities would be more than 88,000 premature deaths averted, $9.1 billion in discounted medical cost savings, and $6.5 billion in risk factor management costs incurred. The 25-year effectiveness of CTG clinical activities was $188,000 per premature death averted, and the 25-year effectiveness of CTG community activities was $57,000 per premature death averted.

Discussion

This analysis provides estimates of the effects of large-scale clinical and community interventions as they were implemented during the CTG program, complementing previous work estimating the impact of hypothetical interventions (2). Results show that CTG clinical activities were projected to avert more premature deaths after 10 years and 25 years than CTG community interventions, but that the gap between the intervention categories shrank from the 10-year mark to the 25-year mark. However, CTG community interventions were projected to save more medical costs after 10 years and 25 years than CTG clinical interventions; this gap increased from the 10-year mark to the 25-year mark. Community interventions in the CTG program had much higher projected program implementation costs than clinical interventions, but led to a much smaller increase in risk factor management costs at the 10-year and 25-year marks. No standard benchmark exists to assess the cost-effectiveness in relation to premature deaths. However, Neumann and colleagues recommended using $100,000 or $150,000 as acceptable amounts to pay per quality-adjusted life year (QALY) gained (30). A cost-effectiveness threshold for premature deaths prevented would be expected to be greater than that for QALYs gained because, on average, preventing a premature death is expected to have a higher value than 1 QALY. Based on this cost-effectiveness threshold, sustained community interventions would likely be considered cost-effective, especially when considered over a period of 10 years or longer.
A previous study using similar methods evaluated another CDC-funded program, Communities Putting Prevention to Work (CPPW), and projected that the program would prevent 14,000 premature deaths in 51 communities during a 10-year period (11). The larger number of premature deaths prevented by the CTG program versus CPPW is likely attributable to the CTG program’s use of clinical interventions, our additional analytic efforts to code evidence-based interventions into PRISM, and the use of existing infrastructure by high-capacity awardees to implement community health interventions.
Our analysis is subject to several limitations. First, all simulation models are approximations to reality and are limited by the evidence of effect sizes that is available. Second, we derived model inputs from awardee progress reports, which may overstate accomplishments. Third, although PRISM is a broad cardiovascular disease model, it accounts for most, but not all, strategies implemented in the CTG program (eg, it does not account for outdoor smoke-free air regulations). Fourth, the analysis assumes that all activities would be sustained for 10 years and 25 years, which is the most optimistic scenario possible. In reality, interventions often lose strength once they are no longer actively promoted. This assumption may be more reasonable for interventions that change policies or the community environment, but may be less realistic for interventions that require regular ongoing support. Fifth, translating programmatic information into any simulation model is challenging, and quantifying community policy and environmental changes introduces aspects of subjectivity. The process used in this analysis was refined from CPPW to reduce subjectivity by focusing on evidence-based interventions from the literature, all of which were assigned to a given category of impact. This approach is consistent with approaches used by others to estimate the “dose” for community health interventions (25–27). Finally, this analysis focused on the aggregate impact of the CTG program and did not address variability in reach and potential health and economic outcomes for specific awardees or target populations.
Study findings suggest that clinical and community interventions, like those implemented in the CTG program, may be expected to have substantial benefits. Clinical interventions have the potential to prevent more premature deaths than community-based interventions in both the intermediate (10 years) and long term (25 years). However, sustaining community-based interventions over the long term may save more in medical costs and have greater cost-effectiveness than investing in only clinical interventions.

Acknowledgments

This research was supported by funding from CDC (contract no. 200-2011-F-42033). The findings and conclusions in this report are those of the authors and do not necessarily represent the official position of CDC. No copyrighted material was used or adapted for this study.

Author Information

Corresponding Author: Benjamin Yarnoff, PhD, RTI International, 3040 E Cornwallis Rd, Research Triangle Park, NC 27709. Telephone: 919-541-6640. Email: byarnoff@rti.org.
Author Affiliations: 1RTI International, Research Triangle Park, North Carolina. 2Centers for Disease Control and Prevention, Atlanta, Georgia.

References

  1. Frieden TR. A framework for public health action: the health impact pyramid. Am J Public Health 2010;100(4):590–5. CrossRefexternal iconPubMedexternal icon
  2. Milstein B, Homer J, Briss P, Burton D, Pechacek T. Why behavioral and environmental interventions are needed to improve health at lower cost. Health Aff (Millwood) 2011;30(5):823–32. CrossRefexternal icon PubMedexternal icon
  3. Centers for Disease Control and Prevention. Strategies snapshot: Community Transformation Grants in action. https://www.cdc.gov/nccdphp/dch/programs/communitytransformation/focus-strategies/index.htm. Updated October 21, 2014. Accessed July 21, 2016.
  4. Centers for Disease Control and Prevention. Community Transformation Grant program FY2013 highlights. https://www.cdc.gov/nccdphp/dch/programs/communitytransformation/accomplishments/index.htm. Updated December 15, 2016. Accessed February 7, 2017.
  5. Centers for Disease Control and Prevention. Investments in community health: Community Transformation Grant Program. https://www.cdc.gov/nccdphp/dch/programs/communitytransformation/funds/index.htm. Updated December 15, 2016. Accessed May 15, 2018.
  6. Bunnell R, O’Neil D, Soler R, Payne R, Giles WH, Collins J, et al. Fifty communities putting prevention to work: accelerating chronic disease prevention through policy, systems and environmental change. J Community Health 2012;37(5):1081–90. CrossRefexternal icon PubMedexternal icon
  7. Homer J, Milstein B, Wile K, Pratibhu P, Farris R, Orenstein DR. Modeling the local dynamics of cardiovascular health: risk factors, context, and capacity. Prev Chronic Dis 2008;5(2):A63. PubMedexternal icon
  8. Hirsch G, Homer J, Evans E, Zielinski A. A system dynamics model for planning cardiovascular disease interventions. Am J Public Health 2010;100(4):616–22. CrossRefexternal icon PubMedexternal icon
  9. Hirsch G, Homer J, Trogdon J, Wile K, Orenstein D. Using simulation to compare 4 categories of intervention for reducing cardiovascular disease risks. Am J Public Health 2014;104(7):1187–95. CrossRefexternal icon PubMedexternal icon
  10. Homer J, Milstein B, Wile K, Trogdon J, Huang P, Labarthe D, et al. Simulating and evaluating local interventions to improve cardiovascular health. Prev Chronic Dis 2010;7(1):A18. PubMedexternal icon
  11. Soler R, Orenstein D, Honeycutt A, Bradley C, Trogdon J, Kent CK, et al. Community-based interventions to decrease obesity and tobacco exposure and reduce health care costs: outcome estimates from Communities Putting Prevention to Work for 2010–2020. Prev Chronic Dis 2016;13:E47. CrossRefexternal icon PubMedexternal icon
  12. Kuo T, Robles B, Trogdon JG, Ferencik R, Simon PA, Fielding JE. Framing the local context and estimating the health impact of CPPW obesity prevention strategies in Los Angeles county, 2010–2012. J Public Health Manag Pract 2016;22(4):360–9. CrossRefexternal icon PubMedexternal icon
  13. Bélanger-Gravel A, Gauvin L, Fuller D, Drouin L. Association of implementation of a public bicycle share program with intention and self-efficacy: the moderating role of socioeconomic status. J Health Psychol 2016;21(6):944–53. CrossRefexternal icon PubMedexternal icon
  14. Fuller D, Gauvin L, Kestens Y, Daniel M, Fournier M, Morency P, et al. Impact evaluation of a public bicycle share program on cycling: a case example of BIXI in Montreal, Quebec. Am J Public Health 2013;103(3):e85–92. CrossRefexternal icon PubMedexternal icon
  15. Fuller D, Gauvin L, Kestens Y, Morency P, Drouin L. The potential modal shift and health benefits of implementing a public bicycle share program in Montreal, Canada. Int J Behav Nutr Phys Act 2013;10(1):66. CrossRefexternal icon PubMedexternal icon
  16. Molina-García J, Castillo I, Queralt A, Sallis JF. Bicycling to university: evaluation of a bicycle-sharing program in Spain. Health Promot Int 2015;30(2):350–8. CrossRefexternal icon PubMedexternal icon
  17. Yang L, Sahlqvist S, McMinn A, Griffin SJ, Ogilvie D. Interventions to promote cycling: systematic review. BMJ 2010;341:c5293. CrossRefexternal icon PubMedexternal icon
  18. Sharpe PA, Granner ML, Hutto B, Ainsworth BE. Association of environmental factors to meeting physical activity recommendations in two South Carolina counties. Am J Health Promot 2004;18(3):251–7. CrossRefexternal icon PubMedexternal icon
  19. Brownson RC, Baker EA, Housemann RA, Brennan LK, Bacak SJ. Environmental and policy determinants of physical activity in the United States. Am J Public Health 2001;91(12):1995–2003. CrossRefexternal icon PubMedexternal icon
  20. Cohen DA, Marsh T, Williamson S, Golinelli D, McKenzie TL. Impact and cost-effectiveness of family Fitness Zones: a natural experiment in urban public parks. Health Place 2012;18(1):39–45. CrossRefexternal icon PubMedexternal icon
  21. Panter JR, Jones AP. Associations between physical activity, perceptions of the neighbourhood environment and access to facilities in an English city. Soc Sci Med 2008;67(11):1917–23. CrossRefexternal icon PubMedexternal icon
  22. Durant N, Harris SK, Doyle S, Person S, Saelens BE, Kerr J, et al. Relation of school environment and policy to adolescent physical activity. J Sch Health 2009;79(4):153–9, quiz 205–6. CrossRefexternal icon PubMedexternal icon
  23. Farley TA, Meriwether RA, Baker ET, Watkins LT, Johnson CC, Webber LS. Safe play spaces to promote physical activity in inner-city children: results from a pilot study of an environmental intervention. Am J Public Health 2007;97(9):1625–31. CrossRefexternal icon PubMedexternal icon
  24. Lafleur M, Gonzalez E, Schwarte L, Banthia R, Kuo T, Verderber J, et al. Increasing physical activity in under-resourced communities through school-based, joint-use agreements, Los Angeles County, 2010–2012. Prev Chronic Dis 2013;10:E89. CrossRefexternal icon PubMedexternal icon
  25. Cheadle A, Schwartz PM, Rauzon S, Beery WL, Gee S, Solomon L. The Kaiser Permanente Community Health Initiative: overview and evaluation design. Am J Public Health 2010;100(11):2111–3. CrossRefexternal icon PubMedexternal icon
  26. Glasgow RE, Vogt TM, Boles SM. Evaluating the public health impact of health promotion interventions: the RE-AIM framework. Am J Public Health 1999;89(9):1322–7. CrossRefexternal icon PubMedexternal icon
  27. King DK, Glasgow RE, Leeman-Castillo B. Reaiming RE-AIM: using the model to plan, implement, and evaluate the effects of environmental change approaches to enhancing population health. Am J Public Health 2010;100(11):2076–84. CrossRefexternal icon PubMedexternal icon
  28. Bureau of Labor Statistics. Consumer price index. 2018. https://www.bls.gov/cpi/tables/supplemental-files/home.htm. Accessed July 17, 2018.
  29. Haddix AC, Teutsch SM, Corso PS. Prevention effectiveness: a guide to decision analysis and economic evaluation. 2nd ed. New York (NY): Oxford University Press; 2003.
  30. Neumann PJ, Cohen JT, Weinstein MC. Updating cost-effectiveness — the curious resilience of the $50,000-per-QALY threshold. N Engl J Med 2014;371(9):796–7. CrossRefexternal icon PubMedexternal icon

A Cohort Review Approach Evaluating Community Health Worker Programs in New York City, 2015–2017

A Cohort Review Approach Evaluating Community Health Worker Programs in New York City, 2015–2017



PCD logo

A Cohort Review Approach Evaluating Community Health Worker Programs in New York City, 2015–2017

Alexis Feinberg, MPH1; Lois Seidl, MPH2; Rachel Dannefer, MPH, MIA2; Katarzyna Wyka, PhD3; Elizabeth Drackett, MPA2; La’Shawn Brown-Dudley, MS2; Nadia Islam, PhD1; Lorna E. Thorpe, PhD1 (View author affiliations)

Suggested citation for this article: Feinberg A, Seidl L, Dannefer R, Wyka K, Drackett E, Brown-Dudley L, et al. A Cohort Review Approach Evaluating Community Health Worker Programs in New York City, 2015–2017. Prev Chronic Dis 2019;16:180623. DOI: http://dx.doi.org/10.5888/pcd16.180623external icon.
PEER REVIEWED
Summary
What is already known about this topic?
Findings from community health worker (CHW) interventions targeting chronic disease prevention and management demonstrate inconsistent results, which may be attributable to funding mechanisms. Monitoring tools developed to address resource constraints, such as the cohort review, have not been used previously to evaluate CHW programs.
What is added by this report?
We applied a cohort review approach as an evaluation framework for a community-focused CHW intervention in New York City. We assessed program implementation and outcomes during the first 2 years of the program. The cohort approach highlighted 6-month outcome successes related to hypertension and diabetes control and identified workload challenges affecting recruitment and retention.
What are the implications for public health practice?
Adapting a cohort monitoring approach can be useful for evaluating the implementation of CHW programs. Such an approach also addresses issues associated with resource constraints and limited program duration.

Abstract

The objective of this study was to describe how a cohort review approach was applied as an evaluation framework for a community health worker intervention among adult residents in 5 public housing developments in New York City in 2015–2017. The cohort review approach involved systematically monitoring participants engaged in the Harlem Health Advocacy Partners program during a given time period (“cohort”) to assess individual outcomes and program performance. We monitored participation status (completed, still active, disengaged, on leave, or died) and health outcomes. In this example of a cohort review, levels of enrollment and program disengagement were higher in cohort 1 than in cohort 2. For 6-month health outcomes, the percentage of participants with hypertension who had controlled blood pressure was static in cohort 1 and improved significantly in cohort 2. The percentage of participants with diabetes who self-reported controlled hemoglobin A1c increased significantly in cohort 1 at 6-month follow-up. The cohort approach highlighted important outcome successes and identified workload challenges affecting recruitment and retention.

Introduction

Although evidence for the effectiveness of community health workers (CHWs) is mounting, reviews of interventions related to chronic disease prevention and management demonstrate inconsistent results (1–3). One key issue is that many CHW programs are funded through grants or operating budgets that are often unpredictable, unstable, and time limited (4). Such funding mechanisms pose unique challenges: with short-term funding, some health outcomes may not emerge within funded evaluation time frames, and positive benefits of programs, including the adoption and maintenance of behavior change, may not have the opportunity to accrue or be sustained. Another problem is inconsistency in how results are reported.
These challenges have affected other public health interventions focused on sustained patient interactions, and monitoring tools developed in response to these challenges can be adapted for CHW program evaluation. For example, the introduction of an annual review process known as a “cohort review” was an important innovation in the monitoring and evaluation of tuberculosis control efforts (5); it involved systematic monitoring of groups of patients beginning treatment within a given period (“cohort”). Structured indicators allowed local and national comparisons, as well as measurement against previous cohorts, to assess improvements in program recruitment, retention, and outcomes.
We adapted cohort review methods to the evaluation of a CHW program. By standardizing participant status definitions and tracking outcome milestones, CHWs and evaluators can develop an analytic framework to better monitor participation status, participant characteristics, and health outcomes. The cohort process also allows for the assessment of trends of program performance indicators that are actionable for decision makers, particularly when comparison groups are unavailable or are no longer supported by funding sources.

Purpose and Objective

The objective of this study was to describe how the cohort review approach was applied as an evaluation framework for a community-focused CHW intervention, the Harlem Health Advocacy Partners (HHAP) program, in New York City. HHAP is an ongoing municipal project that aims to improve the health of adults residing in 5 public housing developments in East/Central Harlem. Despite rich histories of community organizing, East/Central Harlem has been subject to policies and processes such as redlining, broken windows policing, and “benign neglect” that have contributed to high levels of poverty and poor health outcomes. HHAP was launched to address health and social conditions in the neighborhood, with the aim of closing racial/ethnic gaps in health and social outcomes between public housing residents in East/Central Harlem and other New Yorkers (6,7). We developed and applied the cohort review approach to the health coaching component of HHAP to assess program implementation and outcomes during the first 2 years of the program.

Intervention Approach

During the first year of HHAP, 224 participants were enrolled from February through August 2015 (cohort 1), and subsequent cohorts followed an annual enrollment cycle. Cohort 2 enrolled 348 participants from September 2015 through August 2016. Concurrent to cohort 1 enrollment, we recruited a 1-year comparison sample of 176 residents from 5 nearby developments, selected on the basis of frequency-matched sociodemographic characteristics and proximity to the intervention developments (8). After cohort 1, comparison groups were not available.
In addition to a residence requirement, eligibility criteria for health coaching and the comparison group included being aged ≥18 and having at least one of 3 self-reported chronic conditions (asthma, diabetes, or hypertension). Participants who reported ever having received a physician diagnosis of asthma, hypertension, or diabetes were defined as adults with these conditions, on the basis of the following question: “Have you ever been told by a doctor, nurse, or other health professional that you have . . . ?” Both cohort 1 and cohort 2 participants were recruited primarily via community outreach conducted by CHWs, who canvassed the grounds of the selected public housing developments and collaborated with community and senior centers in each development to promote the program. The comparison group was recruited from a random sample telephone survey (9). CHWs attempted to deliver core intervention components within 6 to 12 months of enrollment.
The HHAP intervention includes 4 components: 1) health coaching, 2) navigation of the health care system, 3) wellness activities, including peer support and walking groups, and 4) advocacy to build leadership among residents to address community health needs and improve systems and conditions that influence neighborhood health. The health coaching provided by CHWs also included referrals, emergency interventions during acute-risk situations (eg, morbidly high blood pressure readings, mental health crises), and the setting of one or more SMART (specific, measurable, achievable, results-focused, and time-bound) goals. Additional health care navigation support was available through referrals to a partner organization that assists residents in obtaining medical services and ensures they receive the care to which they are entitled. A full description of the HHAP model is available elsewhere (8).

Evaluation Methods

For cohort 1, CHWs conducted intake assessments as part of participant enrollment (baseline), and an academic research team from the NYU–CUNY Prevention Research Center (PRC) conducted follow-up assessments. The academic research team conducted all comparison group assessments. Cohort 1 and comparison participants received a $20 cash incentive for completing surveys. For cohort 2, CHWs conducted baseline and follow-up assessments. Surveys were conducted at 3 months, 6 months, 9 months, or 12 months after enrollment. Among participants enrolled in cohort 1 and cohort 2, 209 of 224 (93.3%) in cohort 1 and 233 of 348 (67.0%) in cohort 2 completed any follow-up assessment survey. For this analysis, we tabulated data on 6-month follow-up from both years; the response rate was 85.7% (192 of 224) for cohort 1, 92.6% (163 of 176) for the comparison group, and 41.7% (145 of 348) for cohort 2.
We categorized all HHAP participants into mutually exclusive and exhaustive categories of participation in health coaching: completed, enrolled active, disengaged, on leave, or died (Box). CHWs assigned and updated participant status. The NYU-CUNY PRC collected data on health outcomes in the baseline surveys and follow-up surveys. These outcomes were blood pressure control, blood pressure control among participants with hypertension, and self-reported hemoglobin A1c (HbA1c) control among participants with diabetes. Blood pressure was the average of 3 measurements taken at each survey point, and we defined control as systolic blood pressure under 140 mm Hg or diastolic blood pressure under 90 mm Hg (10). We dichotomized self-reported status of glycemic control as controlled if a health professional told a participant their diabetes was within goal and as “uncontrolled or don’t know” if they were told it was not within goal or if they were unaware of their status.
Box. Definition of Each Category of Participation in the Health Coaching Component of the Harlem Health Advocacy Partners Program, New York City, 2015–2017
StatusDefinition of Status
EnrolledCompleted intake
CompletedHealth coaching completed
Enrolled activeStill active in health coaching and have not yet completed
DisengagedNo longer participating in health coaching. Includes people referred out, people lost to follow-up, people unable to fit health coaching into their schedule, and people who request to stop participating
On leaveTemporarily on leave from the program
DiedDied while enrolled active
Using SAS version 9.4 for all analyses (SAS Institute Inc), we compared the baseline characteristics of cohort 1 with the baseline characteristics of cohort 2 and the comparison group with t test for continuous variables and χ2 test for categorical variables. For health outcome variables, we tested significance by cohort between enrollment and 6-month post-enrollment by using the McNemar χ2 test. We chose this test because it is widely used and easy to interpret.

Results

A greater percentage of residents participating in HHAP health coaching than in the comparison group were aged 65 or older and self-reported hypertension (Table 1). Most participants were female and either Hispanic or non-Hispanic black, reflecting the population of the public housing developments (9). Participants were demographically similar to one another across cohorts, except that a greater proportion of cohort 2 participants than cohort 1 or comparison group participants were Hispanic.
Enrollment increased 55.4% from cohort 1 to cohort 2, from 224 to 348 participants. Of the 224 cohort 1 participants, 216 (96.4%) participants were still active in the program after 6 months, 5 (2.2%) had disengaged, 1 (0.4%) was on leave, and 2 (0.9%) had died. Of the 348 participants enrolled in cohort 2, 303 (87.1%) were still active after 6 months, 39 (11.2%) had disengaged, 2 (0.6%) were on leave, and 2 (0.6%) had died.
The percentage of participants with self-reported hypertension in cohort 1 and controlled blood pressure did not change from baseline to 6-month follow-up (58.8% to 60.1%, P = .79) (Table 2). Blood pressure control among residents with hypertension in the comparison group may have worsened from baseline to 6-month follow-up (61.0% to 53.3%, P = .16). In cohort 2, the percentage of participants with diagnosed hypertension and controlled blood pressure increased significantly, from 57.7% to 73.9% (P = .002). The percentage of participants with self-reported diabetes who reported their HbA1c as controlled increased significantly in cohort 1 (50.0% to 64.3%, P = .02), whereas self-reported HbA1c control did not improve among comparison group participants (65.7% to 64.2%, P = .74). Although the change was not significant, we found improvements in HbA1c control among cohort 2 participants (72.3% to 83.0%, P = .20).

Implications for Public Health

Our findings from the first 2 years of HHAP’s health coaching component demonstrate the utility of the cohort review approach in providing a structure for evaluating a multiyear program, particularly when an ongoing comparison group is not available. The approach highlighted successes in health outcomes among participants retained in the program and challenges in program retention.
The assessment showed that more participants in cohort 2 than in cohort 1 disengaged from the program after 6 months. One reason for the higher level of disengagement in cohort 2 could be the challenge of maintaining health-coaching participants carried over from cohort 1 while recruiting for cohort 2, since CHWs in cohort 2 were also responsible for managing participants from the previous year. In addition, in the beginning of cohort 2, programmatic operations were transferred from an external organization to the New York City Department of Health and Mental Hygiene, which may have resulted in a disruption for some participants. Finally, the incentive offered in cohort 1 may have positively influenced program retention and the number of follow-up interviews. Because the cohort process cycle emphasizes continuous monitoring and improvement, the HHAP program addressed retention and workload issues in cohort 3.
Our cohort assessment quantified improvements in key health outcomes shown in previous studies, namely in blood pressure (11) and glycemic control (2,12). The increase from cohort 1 to cohort 2 in the number of participants with controlled blood pressure suggests that the ability of CHWs to enhance care increases over time. This care includes efforts to keep participants connected with their primary care physician and to motivate participants to take all routine tests and medications for their conditions.
In planning for evaluating CHW programs using a cohort approach, metrics for the implementation process should be developed a priori and aligned with program objectives. Our analysis underscored the challenge of defining the participation status of a participant as complete. The definition was challenging because the criteria for program completion changed over 2 cohort years; awareness of this challenge helped formalize the definition of completion. Moreover, the program further disaggregated the disengaged group into 4 new categories: withdrew, lost to follow-up, transferred out of health coaching, and unavailable (ie, unable to fit health coaching into their schedule). It will be important to monitor these categories to assess whether participants are not interested or able to participate in the program, which would reflect a poor fit between the program and a participant’s needs (withdrew), or the program is unable to maintain contact with participants because of other factors (lost to follow-up).
We found the cohort review approach adaptable to new program goals. For example, to better HHAP’s efforts to address the social determinants of health in addition to disease management, we developed outcome metrics for social determinants of health for cohort 3, and the program will continue to monitor other variables that may contribute to health outcomes.
Our study has several limitations. Our findings in part reflect differences in HHAP programmatic operations between cohort 1 and 2. Cohort 1 data were collected by both CHWs and an academic research team and participants in cohort 1 received a cash incentive for completing surveys, whereas cohort 2 data were collected by CHWs only, often with fewer follow-up assessments, and cohort 2 participants did not receive an incentive. Differences in data collection may have biased comparisons between cohort 1 and cohort 2. Some health outcome data were self-reported; however, any bias introduced by self-report is unlikely to be differential across cohorts, except if selection bias was introduced because of higher loss to follow-up in cohort 2. Finally, given the large number of disengaged participants in cohort 2, we are not fully confident that improved outcomes were solely a function of programmatic improvements. Improved outcomes may reflect differential disengagement of participants who would have been less likely to improve.
Although previous CHW evaluations focused on individual-level outcomes, we found the cohort monitoring approach to be an effective method for evaluating the implementation process of CHW programs while also addressing issues associated with resource constraints and limited program duration (13). Adapting a cohort approach can begin to fill this gap (4,14).

Acknowledgments

Research was supported by Harlem Health Advocacy Partners, a project funded and administered by the New York City Department of Health and Mental Hygiene. The efforts of Drs. Thorpe and Islam are supported in part by the Centers for Disease Control and Prevention (CDC) Grant U48DP001904. The contents of this publication are solely the responsibility of the authors and do not necessarily represent the official views of the New York City Department of Health and Mental Hygiene or CDC. The authors declare no conflicts of interest. No copyrighted materials were used.

Author Information

Corresponding Author: Alexis Feinberg, MPH, Department of Population Health, New York University School of Medicine, 180 Madison Ave, New York, NY 10016. Email: Alexis.Feinberg@nyulangone.org.
Author Affiliations: 1Department of Population Health, New York University School of Medicine, New York, New York. 2New York City Department of Health and Mental Hygiene, Long Island City, New York. 3City University of New York Graduate School of Public Health and Health Policy, New York, New York.

References

  1. Kim K, Choi JS, Choi E, Nieman CL, Joo JH, Lin FR, et al. Effects of community-based health worker interventions to improve chronic disease management and care among vulnerable populations: a systematic review. Am J Public Health 2016;106(4):e3–28. CrossRefexternal iconPubMedexternal icon
  2. Palmas W, March D, Darakjy S, Findley SE, Teresi J, Carrasquillo O, et al. Community health worker interventions to improve glycemic control in people with diabetes: a systematic review and meta-analysis. J Gen Intern Med 2015;30(7):1004–12. CrossRefexternal icon PubMedexternal icon
  3. Shah M, Kaselitz E, Heisler M. The role of community health workers in diabetes: update on current literature. Curr Diab Rep 2013;13(2):163–71. CrossRefexternal icon PubMedexternal icon
  4. Albritton E. How states can fund community health workers through Medicaid to improve people’s health, decrease costs, and reduce disparities. 2016.2016 https://familiesusa.org/product/how-states-can-fund-community-health-workers-through-medicaid. Accessed May 13, 2019.
  5. Centers for Disease Control and Prevention. Understanding the TB cohort review process: instruction guide. Atlanta (GA): Centers for Disease Control and Prevention, Division of Tuberculosis Elimination; 2006.
  6. Fullilove MT, Wallace R. Serial forced displacement in American cities, 1916–2010. J Urban Health 2011;88(3):381–9. CrossRefexternal iconPubMedexternal icon
  7. Bailey ZD, Krieger N, Agénor M, Graves J, Linos N, Bassett MT. Structural racism and health inequities in the USA: evidence and interventions. Lancet 2017;389(10077):1453–63. CrossRefexternal icon PubMedexternal icon
  8. Lopez PM, Islam N, Feinberg A, Myers C, Seidl L, Drackett E, et al. A place-based community health worker program: feasibility and early outcomes, New York City, 2015. Am J Prev Med 2017;52(3S3):S284–9. CrossRefexternal icon PubMedexternal icon
  9. Feinberg A, Seidl L, Levanon Seligson A, Pinzon J, Mata A, Gray L, et al. Launching a neighborhood-based community health worker initiative: Harlem Health Advocacy Partner (HHAP) Community Needs Assessment. New York (NY): NYU-CUNY Prevention Research Center, New York City Department of Health and Mental Hygiene, New York City Housing Authority, and Community Service Society; 2015. https://med.nyu.edu/prevention-research/sites/default/files/prevention-research2/FINAL%20MAY%202016%20%20-%20HHAP%20Neighborhood%20Based%20Comm%20Report_0.pdf. Accessed June 3, 2019.
  10. US Department of Health and Human Services Health Resources and Services Administration. Hypertension control. https://www.hrsa.gov/sites/default/files/quality/toolbox/508pdfs/hypertensioncontrol.pdf. Accessed April 4, 2018.
  11. Levine DM, Bone LR, Hill MN, Stallings R, Gelber AC, Barker A, et al. The effectiveness of a community/academic health center partnership in decreasing the level of blood pressure in an urban African-American population. Ethn Dis 2003;13(3):354–61. PubMedexternal icon
  12. Pérez-Escamilla R, Damio G, Chhabra J, Fernandez ML, Segura-Pérez S, Vega-López S, et al. Impact of a community health workers-led structured program on blood glucose control among Latinos with type 2 diabetes: the DIALBEST trial. Diabetes Care 2015;38(2):197–205. CrossRefexternal icon PubMedexternal icon
  13. Katigbak C, Van Devanter N, Islam N, Trinh-Shevrin C. Partners in health: a conceptual framework for the role of community health workers in facilitating patients’ adoption of healthy behaviors. Am J Public Health 2015;105(5):872–80. CrossRefexternal icon PubMedexternal icon
  14. Damschroder LJ, Aron DC, Keith RE, Kirsh SR, Alexander JA, Lowery JC. Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science. Implement Sci 2009;4(1):50. CrossRefexternal icon PubMedexternal icon