sábado, 9 de julio de 2016

Effectiveness of Health Impact Assessments: A Synthesis of Data From Five Impact Evaluation Reports

Effectiveness of Health Impact Assessments: A Synthesis of Data From Five Impact Evaluation Reports

Centers for Disease Control and Prevention. CDC twenty four seven. Saving Lives, Protecting People



PCD logo



Effectiveness of Health Impact Assessments: A Synthesis of Data From Five Impact Evaluation Reports

Andrew L. Dannenberg, MD, MPH

Suggested citation for this article: Dannenberg AL. Effectiveness of Health Impact Assessments: A Synthesis of Data From Five Impact Evaluation Reports. Prev Chronic Dis 2016;13:150559. DOI:http://dx.doi.org/10.5888/pcd13.150559.
PEER REVIEWED

Abstract

Introduction
Since the 1990s, the use of health impact assessments (HIAs) has grown for considering the potential health impacts of proposed policies, plans, programs, and projects in various sectors. Evaluation of HIA impacts is needed for understanding the value of HIAs, improving the methods involved in HIAs, and potentially expanding their application. Impact evaluations examine whether HIAs affect decisions and lead to other effects.
Methods
I reviewed HIA impact evaluations identified by literature review and professional networking. I abstracted and synthesized data on key findings, success factors, and challenges from 5 large evaluations conducted in the United States, Europe, Australia, and New Zealand and published from 2006 through 2015. These studies analyzed impacts of approximately 200 individual HIAs.
Results
Major impacts of HIAs were directly influencing some decisions, improving collaboration among stakeholders, increasing awareness of health issues among decision makers, and giving community members a stronger voice in local decisions. Factors that contributed to successful HIAs included engaging stakeholders, timeliness, policy and systems support for conducting HIAs, having people with appropriate skills on the HIA team, obtaining the support of decision makers, and providing clearly articulated, feasible recommendations. Challenges that may have reduced HIA success were poor timeliness, underestimation of time and resources needed, difficulty in accessing relevant data, use of jargon in HIA reports, difficulty in involving decision makers in the HIA process, and absence of a requirement to conduct HIAs.
Conclusion
HIAs can be useful to promote health and mitigate adverse impacts of decisions made outside of the health sector. Stakeholder interactions and community engagement may be as important as direct impacts of HIAs. Multiple factors are required for HIA success. Further work could strengthen the role of HIAs in promoting equity, examine HIA impacts in specific sectors, and document the role of HIAs in a “health in all policies” approach.
 Top

Introduction

Health impact assessments (HIAs) have been used for 2 decades as a tool to facilitate communication between public health professionals and decision makers in other sectors (1). More than 390 HIAs in the United States have been completed or were in progress as of early 2016 (2). An HIA is a “a systematic process that uses an array of data sources and analytic methods and considers input from stakeholders to determine the potential effects of a proposed policy, plan, program, or project on the health of a population and the distribution of those effects within the population. HIA provides recommendations on monitoring and managing those effects” (3).
Several reports have called for evaluation to determine the impacts and usefulness of HIAs (3,4). Reasons to evaluate HIAs include assessing whether they provide the expected impacts, improving methods, identifying positive and negative unintended consequences, and justifying requests for future resources.
Three types of evaluation can be conducted on HIAs: process, impact, and outcome (3). Process evaluations examine the process followed in conducting an HIA and compares it with the practitioner’s intended plan or with established guidelines for HIA practice (5,6); several process evaluations have been published (7–10). Impact evaluations examine the impacts of an HIA on selected decisions and other events; such information is valuable to those who decide whether or not to conduct HIAs. Outcome evaluations examine changes in health status or health determinants caused by the HIA, but these are rarely conducted. The objective of this study was to review and describe 5 large impact evaluations of HIAs, 3 of which were recently published.
 Top

Methods

I identified HIA impact evaluation reports by networking with HIA colleagues, searching literature in PubMed and Google Scholar, and reviewing references in published studies. I also searched major US and international HIA websites for relevant reports (2,11–15) and reviewed recent HIA textbooks (3,16–19).
This study included HIA evaluation reports available in English that assessed and synthesized data on the impacts of multiple individual HIAs. Of 8 potentially eligible reports identified, 1 unpublished student report (20), 1 unpublished conference presentation (21), and 1 Canadian study now in progress (22) were excluded. I abstracted information about background, methods, and findings, including the impact of HIAs on decisions and on other relevant events, factors that contributed to the success of HIAs, and factors that reduced the likelihood of an HIA having an impact. I reviewed the similarities and differences in impacts, success factors, and challenges found across the various reports. To compare multistudy HIA impact evaluations with individual HIA impact evaluations, I reviewed a convenience sample of 8 impact evaluation reports that described 11 individual HIAs (23–30) selected for topic and geographic diversity.
I invited the investigators of the 5 major HIA evaluation studies to review a draft of this report to ensure their studies were accurately described and to provide additional comments. Investigators who provided comments are named in the Acknowledgments.
 Top

Results

Characteristics of major HIA evaluation reports

Five reports synthesized the impacts of multiple HIAs (31–35) (Table 1). These reports covered HIAs conducted primarily in Europe, Australia, New Zealand, and the United States during various periods from 1996 to 2013. The number of HIAs reviewed per report ranged from 11 to 88 for a total of about 200 HIAs. Ten of the 23 HIAs reviewed by Bourcier et al (31) were among the 81 HIAs conducted in the United States and reviewed by Rhodus et al (34). A small but unknown number of HIAs conducted in Europe may have been reviewed by both Davenport et al (32) and Wismar et al (35).
None of the samples used in the 5 reports were representative of all HIAs. Davenport et al (32) reported on all HIAs identified in multiple databases as of 2004. The other reports purposefully selected HIAs from large databases by using criteria such as having “some potential for effectiveness” (35): reflecting diversity in sector, geography, timing, effectiveness, and funding (31,33): or relating to the agency’s mission (34). HIAs were examined from such sectors as land use, transportation, housing, agriculture, energy, health services, and waste management.
All 5 reports included a review of the written findings and recommendations for each individual HIA. Three reports (31,33,35) included interviews of key stakeholders and decision makers to assess impacts. Davenport et al (32) reviewed commentaries and discussion papers and included an email survey. Rhodus et al (34) used only publicly accessible documentation from the Internet to identify HIA impacts.
Each report sought to document whether HIA recommendations influenced the decisions they were intended to inform. All reports identified impacts on decisions of some HIAs and generally provided examples.

Definition of HIA success

One reason to evaluate an HIA is to determine if it is a “success.” Davenport et al (32) reports that “a successful HIA is one where its findings are considered by decision makers to inform the development and implementation of a [policy, program or project].” Bourcier et al (31) states that “success for HIAs should therefore be defined by both their impacts on decisions and on the environments in which decisions are made.” Haigh et al (33) writes that effectiveness (success) can be defined as “the extent to which the HIA succeeds in bringing about the desired changes to decision-making and implementation . . . [but there are often disagreements] about what constitutes ‘success’ and what constitutes a ‘desired change.’” In an unpublished survey, HIA practitioners defined success as educating decision makers about the health consequences of a policy, influencing the design of a project, creating new partnerships between health and other agencies, facilitating community involvement in a decision, and addressing community concerns (21).

Categories of effectiveness

Effectiveness categories could be helpful to distinguish useful from ineffective HIAs. Wismar et al (35) proposed 4 categories of effectiveness: direct (leads to changes in decision), general (raises awareness but no specific changes are made in decision), opportunistic (favorable decision would have been made anyway), and ineffective (HIA ignored in decision). Haigh et al (36) found these categories difficult to use because different aspects of a single HIA may fall into multiple categories, and judging whether an HIA impact fit into a category often needed a graded characterization rather than a yes-or-no decision.
Wismar et al (35) also proposed measuring effectiveness in 3 dimensions: health effectiveness (avoiding negative health effects and strengthening positive health effects), equity effectiveness(assessing and preventing disproportionate impacts on vulnerable populations), and community effectiveness (incorporating a community’s interests into the decision process). That review found varying levels of these dimensions of effectiveness across the 17 HIAs assessed. The other 4 reports did not use these dimensions in their analyses.

Direct HIA impacts

The 5 reports I studied indicated that many HIAs directly influenced decisions (Table 2). Evidence fulfilling Wismar’s direct effectiveness criteria was found by Rhodus et al (34) in 30 of 81 HIAs (37%) and by Bourcier et al (31) in 11 of 23 HIAs (48%), although the 2 groups overlapped. Examples of decision changes directly attributable to an HIA include those associated with a county bicycle plan (26) and a housing proposal (38). A linear model of decision makers obtaining information, making a decision, and acting accordingly often does not reflect actual circumstances (33), and the impacts of many HIAs may not fit neatly into Wismar’s 4 categories. For some HIAs, other impacts such as community engagement may be more important than direct impacts.
The 5 reports indicated projects or policies are rarely cancelled because of unfavorable HIA recommendations. One HIA about rental housing voucher restrictions and another about a solid waste facility identified adverse health effects and contributed to those proposals not moving forward (34). Occasionally an HIA was favorable but the project was subsequently cancelled for other reasons (39).

Other HIA impacts

Three reports highlighted the raised awareness of health issues among stakeholders and decision makers caused by the HIA (Table 2). Although difficult to quantify, this raised awareness has the potential to influence decisions, thus providing an impact that extends beyond the initial reason for which the HIA was conducted.
Two reports found that an HIA may help create and strengthen interactions among stakeholders and other community members. Such interactions may continue after the HIA is completed and provide ongoing community benefits. Many HIAs helped facilitate the relationships between agencies across sectors (40).
Two reports emphasized the value of community engagement as part of conducting HIAs. The HIA process is designed to incorporate community input and may amplify community member voices in the decision-making process (31,41). Through participation in HIAs, community members may take action, increase contact with decision makers, and acquire skills that could help them influence decisions (42). Community engagement with the goal of empowerment is central to the HIA core value of promoting equity (43,44).
Other potential HIA impacts mentioned in various reports include developing new cross-disciplinary and interagency collaborations, identifying data gaps and questions for future research, establishing a foundation for appropriate monitoring, ensuring the public has accurate information on adverse and beneficial effects, and developing new forecasting methods (3).

Success factors

The 5 reports identified factors that contribute to success (also called “enablers” [32]) (Table 3). The following factors were identified in at least 2 reports. The wording of success factors and of challenges varied widely among reports.
  • Engaging stakeholders. All 5 reports suggested that stakeholder input was valuable during the planning and conducting of the HIA, generally to provide a unique perspective and to increase knowledge and skills among the HIA team and the decision makers.
  • Timeliness. Four reports suggested that an HIA must be conducted early enough in the decision process to have an impact.
  • Policy and systems support for conducting HIA. Three reports indicated the value of having frameworks or legislation supporting HIA conduct and of following established guidelines for conducting HIAs.
  • Engaging the “right” people on the HIA team. Three reports found that team members may be chosen based on their knowledge and experience with HIAs, teamwork skills, knowledge of the subject matter being assessed, and acquaintance with the community involved.
  • Engaging and obtaining support of decision makers. Three reports highlighted the importance of ensuring the willingness of decision makers to receive and consider HIA recommendations.
  • Clearly articulated, feasible recommendations. Three reports highlighted the value of clear, precise language to make it practical for decision makers to act on the HIA recommendations.
  • Tailoring presentation of messages to decision makers and other audiences. Three reports indicated that HIAs with summaries of findings and recommendations written in language that is appropriate for the intended audiences are more likely to achieve the desired impact.
Other success factors included taking advantage of HIA credibility to influence decision makers, quantifying health impacts, clarifying who pays the HIA costs, and using proactive positioning to recognize or create opportunities for HIAs and optimize HIA timing. These factors were each reported in one evaluation. One study reported development of an implementation and monitoring plan for recommendations as a success factor, but another study reported inadequate follow-up of recommendations as a challenge.

Challenges

The reports identified various challenges to an HIA being considered successful (Table 4). The following challenges were identified in at least 2 reports.
  • Timing and timeliness. Three reports indicated that HIAs may have little impact if completed too late to affect a decision. For example, in legislative settings, the pace of decision making is sometimes faster than the pace of the HIA.
  • Adequacy of resources. Three reports found that practitioners often underestimated the time and staff resources needed to conduct an HIA.
  • Accessing relevant data. Two reports found that the absence of sufficiently local health data may reduce the value to decision makers of an HIA’s findings and recommendations.
  • Use of jargon. Two reports mentioned that use of technical terms or inconsistent terminology in HIA reports may reduce their value.
  • Lack of support from, or involvement by, decision makers. Two reports found that HIAs are unlikely to affect decisions when decision makers are uninvolved in the process.
  • Absence of requirement to consider HIA findings. Two reports mentioned that decision makers may give minimal consideration to recommendations by an HIA that was not required.
Other reported challenges included being involved in controversial topics, lack of knowledge of HIA practitioners about policy and of policy makers about health, and disagreements on the breadth of the definition of health and on expert predictions in HIA reports.

HIA evaluation of individual HIAs

A review of impact evaluations of a convenience sample of 11 individual HIAs in 8 reports (Table 5) found impacts, success factors, and challenges similar to those described above. These impact evaluations were done by investigators associated with conducting the original HIAs, except for the 4 HIAs in Ireland (29). A New Zealand website has a list of 13 individual HIA impact evaluations (45), but it does not synthesize the common findings across those evaluations. No similar lists were identified for individual HIA evaluations conducted elsewhere.
 Top

Discussion

The evaluation reports reviewed consistently showed that some HIAs had direct impacts on decisions. Others may have had no direct impact on a decision because the decision maker was not receptive or the decision process was complex and nonlinear. Other impacts were found for most HIAs and may have been as important as direct impacts. Such other impacts included raising awareness of health issues among decision makers, building new partnerships among stakeholders, and engaging community members in issues that affect them. HIAs rarely led to project cancellations, which may be a concern especially for projects in the private sector.
Various success factors and challenges were identified across the studies, but few were consistently found in all reports. No single success factor was identified as necessary and sufficient to ensure the success of an HIA. Even timeliness was found to be not essential; Haigh et al (33) reported examples in which the HIA affected not the decision but the subsequent implementation of the policy.
The findings on HIA success factors and barriers of this report are consistent with the findings of other studies. One survey of 47 HIAs in the United States documented the importance of community participation to increase the likelihood of HIA success (42). A British study based on 14 key informant interviews reported that leadership, integration of the HIA with existing organizational structures, and collaboration among key stakeholder organizations were central to encouraging HIA use in decision making (46). A master’s thesis that reviewed 54 HIAs in the United States reported that time and funding constraints are often barriers to HIA effectiveness and that most HIA practitioners believe that the use of quantitative health data strengthens HIA credibility (20).
A survey of participants in the European Healthy Cities Network reported that the following factors facilitated the use of an HIA: political support, training in HIA, collaboration with a public health institution or agency, a culture of intersectoral collaboration, a supportive national policy, and access to HIA expertise (47). The same survey reported the following barriers to HIA implementation: lack of skill, knowledge, and experience about HIAs; lack of a legal basis for implementation; and lack of political support (47). In a related report, factors enhancing the acceptability of recommended interventions included documented benefits, adaptability for stakeholder needs, an evidence base for the work, engagement of stakeholders, and the use of simple, clear language (48).
Promoting equity and reducing health disparities are often cited as major reasons for conducting an HIA (43,44,49). Equity may have improved as a result of some HIAs examined in this study, but no reports explicitly documented changes in equity or the factors contributing to reduced inequities. This lack of documentation may be due to the time required for documentation or difficulty in measuring such changes.
Process and outcome evaluations are complementary to impact evaluations. Results of several process evaluations of HIAs have been published, either alone (7–10) or as a combined process and impact evaluation (34,35). Guidelines are available for process evaluations (17,50). Most process evaluations have found substantial variability in the extent to which HIA investigators followed guidelines for HIA practice (8,34).
HIA outcome evaluations would be valuable to document whether implementing a recommendation has actual effects on health outcomes (eg, rate of heart disease) or on health determinants (eg, physical activity level) (3,51). Outcome evaluations are difficult to conduct because 1) causal pathways are complex, making it difficult to attribute a specific change in a health outcome or determinant to a specific HIA recommendation, 2) there may be no comparison group to document what would have happened in the absence of the HIA, and 3) they require substantial time and resources (3,19,51,52). No outcome evaluations were found in my literature review. The most relevant example found was a class exercise that predicted dietary changes related to a new supermarket (53).
This analysis synthesized data on the characteristics and findings of multiple HIA impact evaluations. It found compatible results across 5 major reports that evaluated more than 200 individual HIAs. These reports examined HIAs that were conducted in diverse sectors, countries, and years, and by various methods. Particularly valuable were the interviews of stakeholders and decision makers in 3 reports, an approach that increases the likelihood of identifying which factors influence a decision. This analysis also compared various success factors and challenges that affect HIA impacts across multiple studies.
This study has several limitations. The contexts and health systems vary across the countries in which the HIAs were conducted. HIA evaluations may have been missed if unpublished or if published in a language other than English. HIAs conducted in countries whose primary language is not English, such as Iran (54), Mongolia (55), and Thailand (56), may have different impacts on decisions, because of cultural, political, or economic differences. Each of the 5 evaluations used a different strategy for selecting HIAs; none were random samples of all HIAs. Three evaluations included potential effectiveness as part of the sampling strategy, which may have biased results toward finding an HIA to be effective. For some HIAs, decisions were being made as the HIA was being evaluated, so subsequent impacts of the HIA may have been missed. For other HIAs, the decision maker may not have remembered the impact of the HIA on a decision made long ago. Some decision makers may not have been able to state exactly how much weight was given to each of several factors (eg, health, economic, political, and social) in reaching a decision. Some HIAs involved politically sensitive topics about which informants may have been reluctant to provide full information. Some findings were difficult to quantify, such as the extent to which collaboration increased and awareness of health issues among stakeholders was raised. Not all of the evaluators interviewed decision makers; information obtained from HIA investigators or public sources may have not included data on influences of the HIA on the decision-making process. Finally, little information was available about the methods or impacts of HIAs conducted in the private sector (3).
The findings of this review lead to additional questions for which further research would be valuable. Several of these questions are now being researched by working groups of the Society of Practitioners of Health Impact Assessment (57).
First, because the impacts of HIAs are complex, no simple answer exists for policymakers and funders who ask how HIAs make a difference in decision making. HIAs may be more effective in some sectors than in others. Guidelines for best practices for HIAs in various sectors could be developed from reviews and case studies of HIAs in each sector (39,58,59; L.N. Gase et al, unpublished data, 2016).
Second, further work is needed to document the value of HIA in promoting equity and reducing health inequities. Such work requires identifying appropriate indicators of equity that are likely to be affected by decisions and for which changes can be measured (60).
Third, outcome evaluations would be valuable to document the usefulness of HIAs. It may be possible to design outcome evaluations of some HIAs that focus on changes in selected health determinants, such as physical activity or diet.
Fourth, HIAs are a valuable tool in a health-in-all-policies approach to incorporating health into policies in many sectors (61–63). Further work could document examples in which HIAs were used to advance this approach. Impact and outcome evaluations in settings in which health-in-all-policies approaches were implemented could be conducted.
Fifth, HIAs range from rapid desktop analyses to comprehensive studies that require substantial time and resources. Further work could lead to guidelines on the depth and breadth of HIAs needed for projects and policies in various contexts. For some HIAs, such as those for living wage policies in multiple cities (64–66), the findings and recommendations may be generalizable, thereby making further HIAs on the topic unnecessary.
Sixth, the importance of an implementation and monitoring plan in the recommendations of an HIA is often underemphasized by HIA practitioners. Further work could establish best practices to routinely create such plans in HIAs and facilitate follow-up of these plans.
Finally, some HIAs are conducted independently of, or in conjunction with, environmental impact assessments (3,40). Further work could clarify the best means with which to incorporate health issues into environmental impact assessment processes.
HIAs can directly influence decisions outside the health sector and frequently lead to other benefits that may have long-term health promotion value for communities. An aggregation of factors is necessary for HIA success. A central objective of HIAs is to “have health at the table” when decisions are being made; the impact evaluation reports reviewed suggest that this purpose generally is being fulfilled.
 Top

Acknowledgments

This work was supported by the Health Impact Project, a collaboration of the Robert Wood Johnson Foundation and The Pew Charitable Trusts. The initial literature review for this project was supported by a contract on HIA evaluation from the Robert Wood Johnson Foundation to the Center for Community Health and Evaluation of the Group Health Research Institute for which the author served as a consultant. We thank the following colleagues for their helpful comments on a draft of this report: Emily Bourcier, Carol Cahill, Diana Charbonneau, Florence Fulk, Elizabeth Harris, Jonathan Heller, Ruth Lindberg, Arielle McInnis-Simoncelli, Rebecca Morley, Keshia Pollack, and Justicia Rhodus.
 Top

Author Information

Andrew L. Dannenberg, MD, MPH, Affiliate Professor, Department of Environmental and Occupational Health Sciences, University of Washington School of Public Health, Box 357234, Seattle WA 98195-7234. Telephone: 404-272-3978. E-mail: adannen@uw.edu.
 Top

References

  1. Harris-Roxas B, Viliani F, Bond A, Cave B, Divall M, Furu P, et al. Health impact assessment: the state of the art. Impact Assess Proj Apprais 2012;30(1):43–52. CrossRef
  2. The Pew Charitable Trusts and Robert Wood Johnson Foundation. Health Impact Project; 2015. http://www.pewtrusts.org/en/projects/health-impact-project. Accessed April 22, 2016.
  3. National Research Council, Committee on Health Impact Assessment. Improving health in the United States: the role of health impact assessment. Washington (DC): The National Academies Press; 2011. http://www.iom.edu/~/media/Files/Activity%20Files/Environment/EnvironmentalHealthRT/2011-Nov-RT/132291.pdf. Accessed April 22, 2016.
  4. Dannenberg AL, Bhatia R, Cole BL, Dora C, Fielding JE, Kraft K, et al. Growing the field of health impact assessment in the United States: an agenda for research and practice. Am J Public Health 2006;96(2):262–70. CrossRef PubMed
  5. Bhatia R, Farhang L, Heller J, Lee M, Orenstein M, Richardson M, et al. Minimum elements and practice standards for health impact assessment, version 3. http://www.humanimpact.org/doc-lib/finish/11/9. Accessed April 22, 2016.
  6. Hebert KA, Wendel AM, Kennedy SK, Dannenberg AL. Health impact assessment: a comparison of 45 local, national, and international guidelines. Environ Impact Assess Rev 2012;34:74–82.CrossRef
  7. Haigh F, Harris E, Chok HN, Baum F, Harris-Roxas B, Kemp L, et al. Characteristics of health impact assessments reported in Australia and New Zealand 2005–2009. Aust N Z J Public Health 2013;37(6):534–46. CrossRef PubMed
  8. Schuchter J, Bhatia R, Corburn J, Seto E. Health impact assessment in the United States: has practice followed standards? Environ Impact Assess Rev 2014;47:47–53. CrossRef
  9. Hoehner CM, Rios J, Garmendia C, Baldwin S, Kelly CM, Knights DM, et al. Page Avenue health impact assessment: building on diverse partnerships and evidence to promote a healthy community. Health Place 2012;18(1):85–95. CrossRef PubMed
  10. Ali S, O’Callaghan V, Middleton JD, Little R. The challenges of evaluating a health impact assessment. Crit Public Health 2009;19(2):171–80. CrossRef
  11. Human Impact Partners. 2016. http://www.humanimpact.org. Accessed April 22, 2016.
  12. Centers for Disease Control and Prevention. Healthy places; 2016. http://www.cdc.gov/healthyplaces/hia.htm. Accessed April 22, 2016.
  13. UCLA Health Impact Assessment Clearinghouse Learning and Information Center. Links. http://hiaguide.org/methods-resources/links. Accessed April 22, 2016.
  14. Public Health England. HIA gateway; 2007. http://www.apho.org.uk/default.aspx?RID=40141. Accessed April 22, 2016.
  15. World Health Organization. Health impact assessment; 2016. http://www.who.int/hia/en/. Accessed April 22, 2016.
  16. Birley M. Health impact assessment: principles and practice. London (UK): EarthScan/Taylor and Francis; 2011.
  17. Kemm JR, editor. Health impact assessment: past achievement, current understanding, and future progress. Oxford (UK): Oxford University Press; 2013.
  18. O’Mullane M, editor. Integrating health impact assessment with the policy process: lessons and experiences from around the world. Oxford (UK): Oxford University Press; 2013.
  19. Ross CL, Orenstein M, Botchwey N. Health impact assessment in the United States. New York (NY): Springer; 2014.
  20. Singleton-Baldrey L. The impacts of health impact assessment: a review of 54 health impact assessments, 2007–2012 [master’s thesis]. Chapel Hill (NC): University of North Carolina at Chapel Hill, Department of Environmental Sciences and Engineering; 2012. https://cdr.lib.unc.edu/indexablecontent/uuid:02493653-9151-41a7-b98a-c41740bde63a
  21. Pollack KM. Impacts of HIAs: a multiple case study of the US experience. Conference proceeding from 2nd National Health Impact Assessment (HIA) Meeting. 2013 Sep 24–25; Washington, DC. http://www.pewtrusts.org/~/media/assets/2013/09/25/pollack_impacts_hias_multiple_case_study.pdf?la=en. Accessed April 22, 2016.
  22. Nour K, Dutilly-Simard S, Brousselle A, Smits P, Buregeya JM, Loslier J, et al. Evaluation of the effects of health impact assessment practice at the local level in Monteregie. Health Res Policy Syst 2016;14(1):7. CrossRef PubMed
  23. Mindell J, Sheridan L, Joffe M, Samson-Barry H, Atkinson S. Health impact assessment as an agent of policy change: improving the health impacts of the mayor of London’s draft transport strategy. J Epidemiol Community Health 2004;58(3):169–74. CrossRef PubMed
  24. Mathias KR, Harris-Roxas B. Process and impact evaluation of the Greater Christchurch urban development strategy health impact assessment. BMC Public Health 2009;9(1):97. CrossRefPubMed
  25. Quigley and Watts, Ltd. Evaluation of the Whānau Ora HIA of the draft Wairarapa alcohol strategy; 2010. http://www.health.govt.nz/our-work/health-impact-assessment/evaluation-health-impact-assessment/hia-evaluation-reports/evaluation-whanau-ora-hia-draft-wairarapa-alcohol-strategy. Accessed April 22, 2016.
  26. Clark County Public Health. Evaluation of health impact assessment: Clark County bicycle and pedestrian master plan; 2011. http://bikeportland.org/wp-content/uploads/2011/12/HIA_BPplan-copy.pdf. Accessed April 22, 2016.
  27. Harris-Roxas BF, Harris PJ, Harris E, Kemp LA. A rapid equity focused health impact assessment of a policy implementation plan: an Australian case study and impact evaluation. Int J Equity Health 2011;10(1):6. CrossRef PubMed
  28. Ross CL, Leone de Nie K, Dannenberg AL, Beck LF, Marcus MJ, Barringer J. Health impact assessment of the Atlanta BeltLine. Am J Prev Med 2012;42(3):203–13. CrossRef PubMed
  29. O’Mullane M, Quinlivan A. Health impact assessment (HIA) in Ireland and the role of local government. Environ Impact Assess Rev 2012;32(1):181–6. CrossRef
  30. Canterbury District Health Board. Evaluation of the HIA of the Canterbury regional land transport strategy; 2013. http://ecan.govt.nz/publications/General/rlt-strategy-health-impact-190606.pdf. Accessed April 22, 2016.
  31. Bourcier E, Charbonneau D, Cahill C, Dannenberg AL. An evaluation of health impact assessments in the United States, 2011–2014. Prev Chronic Dis 2015;12:E23. CrossRef PubMed
  32. Davenport C, Mathers J, Parry J. Use of health impact assessment in incorporating health considerations in decision making. J Epidemiol Community Health 2006;60(3):196–201. CrossRefPubMed
  33. Haigh F, Harris E, Harris-Roxas B, Baum F, Dannenberg AL, Harris MF, et al. What makes health impact assessments successful? Factors contributing to effectiveness in Australia and New Zealand. BMC Public Health 2015;15(1):1009. CrossRef PubMed
  34. Rhodus J, Fulk F, Autrey B, O’Shea S, Roth A. A review of health impact assessments in the U.S.: current state-of-science, best practices, and areas for improvement. Cincinnati (OH): US Environmental Protection Agency, Office of Research and Development; 2013.
  35. Wismar M, Blau J, Ernst K, Figueras J. The effectiveness of health impact assessment: scope and limitations of supporting decision-making in Europe. Brussels (BE): European Observatory on Health Systems and Policies; 2007. http://www.euro.who.int/_data/assets/pdf_file/0003/98283/E90794.pdf. Accessed April 22, 2016.
  36. Haigh F, Baum F, Dannenberg AL, Harris MF, Harris-Roxas B, Keleher H, et al. The effectiveness of health impact assessment in influencing decision-making in Australia and New Zealand 2005–2009. BMC Public Health 2013;13(1):1188. CrossRef PubMed
  37. Bhatia R, Branscomb J, Farhang L, Lee M, Orenstein M, Richardson M. Minimum elements and practice standards for health impact assessment, version 2. Oakland (CA): North American HIA Practice Standards Working Group; 2010.
  38. Bhatia R. Protecting health using an environmental impact assessment: a case study of San Francisco land use decision making. Am J Public Health 2007;97(3):406–13. CrossRef PubMed
  39. Dannenberg AL, Ricklin A, Ross CL, Schwartz M, West J, White S, et al. Use of health impact assessment for transportation planning: importance of transportation agency involvement in the process. Transp Res Rec 2014;2452:71–80. CrossRef
  40. Bhatia R, Wernham A. Integrating human health into environmental impact assessment: an unrealized opportunity for environmental health and justice. Environ Health Perspect 2008;116(8):991–1000. CrossRef PubMed
  41. Tamburrini A, Gilhuly K, Harris-Roxas B. Enhancing benefits in health impact assessment through stakeholder consultation. Impact Assess Proj Apprais 2011;29(3):195–204. CrossRef
  42. Center for Community Health and Evaluation and Human Impact Partners. Community participation in health impact assessments: a national evaluation; 2016. http://www.humanimpact.org/wp-content/uploads/Full-report_Community-Participation-in-HIA-Evaluation.pdf. Accessed April 22, 2016.
  43. World Health Organization Regional Office for Europe. Gothenburg consensus paper. Health impact assessment: main concepts and suggested approach; 1999. http://www.apho.org.uk/resource/item.aspx?RID=44163. Accessed April 22, 2016.
  44. Heller J, Malekafzali S, Todman LC, Wier M. Promoting equity through the practice of health impact assessment; 2013. http://www.policylink.org/sites/default/files/PROMOTINGEQUITYHIA_FINAL.PDF. Accessed April 22, 2016.
  45. New Zealand Ministry of Health. HIA evaluation reports; 2013. http://www.health.govt.nz/our-work/health-impact-assessment/evaluation-health-impact-assessment/hia-evaluation-reports. Accessed April 22, 2016.
  46. Ahmad B, Chappel D, Pless-Mulloli T, White M. Enabling factors and barriers for the use of health impact assessment in decision-making processes. Public Health 2008;122(5):452–7.CrossRef PubMed
  47. Ison E. Health impact assessment in a network of European cities. J Urban Health 2013;90(Suppl 1):105–15. CrossRef PubMed
  48. Simos J, Spanswick L, Palmer N, Christie D. The role of health impact assessment in Phase V of the Healthy Cities European Network. Health Promot Int 2015;30(Suppl 1):i71–85. CrossRefPubMed
  49. Harris-Roxas B, Haigh F, Travaglia J, Kemp L. Evaluating the impact of equity focused health impact assessment on health service planning: three case studies. BMC Health Serv Res 2014;14(1):371. CrossRef PubMed
  50. Fredsgaard MW, Cave B, Bond A. A review package for health impact assessment reports of development projects. Leeds (UK): Ben Cave Associates Ltd; 2009. http://www.bcahealth.co.uk/pdf/hia_review_package.pdf. Accessed April 22, 2016.
  51. Parry JM, Kemm JR; Evaluation of Health Impact Assessment Workshop. Criteria for use in the evaluation of health impact assessments. Public Health 2005;119(12):1122–9. CrossRefPubMed
  52. Quigley RJ, Taylor LC. Evaluating health impact assessment. Public Health 2004;118(8):544–52. CrossRef PubMed
  53. Petticrew M, Cummins S, Sparks L, Findlay A. Validating health impact assessment: prediction is difficult (especially about the future). Environ Impact Assess Rev 2007;27(1):101–7.CrossRef
  54. Fakhri A, Harris P, Maleki M. Proposing a framework for health impact assessment in Iran. BMC Public Health 2015;15(1):335. CrossRef PubMed
  55. Spickett J, Batmunkh T, Jones S. Health impact assessment in Mongolia: current situation, directions, and challenges. Asia Pac J Public Health 2015;27(2):NP2732–9. CrossRef PubMed
  56. Phoolcharoen W, Sukkumnoed D, Kessomboon P. Development of health impact assessment in Thailand: recent experiences and challenges. Bull World Health Organ 2003;81(6):465–7.PubMed
  57. Society of Practitioners of Health Impact Assessment (SOPHIA). SOPHIA working groups; 2014. http://hiasociety.org/?page_id=576. Accessed April 22, 2016.
  58. The Pew Charitable Trusts. Health impact assessment and housing: guidance for health and housing professionals; 2016. http://www.pewtrusts.org/en/research-and-analysis/collections/2016/03/health-impact-assessment-and-housing. Accessed April 22, 2016.
  59. Sohng HY. Evaluation of health impact assessments related to labor and employment [master’s thesis]. Seattle (WA): University of Washington; 2015. https://digital.lib.washington.edu/researchworks/bitstream/handle/1773/33839/Sohng_washington_0250O_14688.pdf?sequence=1. Accessed April 22, 2016
  60. Heller J, Givens ML, Yuen TK, Gould S, Jandu MB, Bourcier E, et al. Advancing efforts to achieve health equity: equity metrics for health impact assessment practice. Int J Environ Res Public Health 2014;11(11):11054–64. CrossRef PubMed
  61. Gase LN, Pennotti R, Smith KD. “Health in All Policies”: taking stock of emerging practices to incorporate health in decision making in the United States. J Public Health Manag Pract 2013;19(6):529–40. CrossRef PubMed
  62. Rudolph L, Caplan J, Ben-Moshe K, Dillon L. Health in All Policies. A guide for state and local governments. Washington (DC): American Public Health Association. Oakland (CA): Public Health Institute; 2013. https://www.apha.org/~/media/files/pdf/factsheets/health_inall_policies_guide_169pages.ashx. Accessed April 22, 2016.
  63. Wernham A, Teutsch SM. Health in all policies for big cities. J Public Health Manag Pract 2015;21(Suppl 1):S56–65. CrossRef PubMed
  64. Bhatia R, Katz M. Estimation of health benefits from a local living wage ordinance. Am J Public Health 2001;91(9):1398–402. CrossRef PubMed
  65. Cole BL, Shimkhada R, Morgenstern H, Kominski G, Fielding JE, Wu S. Projected health impact of the Los Angeles City living wage ordinance. J Epidemiol Community Health 2005;59(8):645–50. CrossRef PubMed
  66. Tsao TY, Konty KJ, Van Wye G, Barbot O, Hadler JL, Linos N, et al. Estimating potential reductions in premature mortality in New York City from raising the minimum wage to $15. Am J Public Health 2016;106(6):1036–41. CrossRef PubMed
  67. Harris-Roxas B, Harris E. The impact and effectiveness of health impact assessment: a conceptual framework. Environ Impact Assess Rev 2013;42:51–9. CrossRef
 Top

Tables

Return to your place in the textTable 1. Characteristics of 5 Major Health Impact Assessment (HIA) Impact Evaluation Reports, United States, Europe, Australia and New Zealand, 2006–2015
CharacteristicDavenport et al, 2006 (32)Wismar et al, 2007 (35)Rhodus et al, 2013 (34)Haigh et al, 2015 (33); Haigh et al, 2013 (36)Bourcier et al, 2015 (31)
OrganizationUniversity of Birmingham, United KingdomEuropean Observatory on Health Systems and Policies, Brussels, BelgiumUS Environmental Protection Agency, Cincinnati, OhioUniversity of New South Wales, Sydney, AustraliaGroup Health Research Institute, Seattle, Washington
Source of HIAsPrimarily Europe (85 of 88 HIAs [97%])Europe: 19 countriesUnited StatesAustralia and New ZealandUnited States
Years of HIAs reviewed1996–20042002–20062005–20122005–20092005–2013
Sampling strategyAll HIAs found on multiple Web-based databases as of 2004Purposely selected from list of 158 European HIAs completed or ongoing as of 2005 to “have some potential for effectiveness”Used multiple databases of US HIAs completed as of spring 2012; chose all HIAs in 4 sectors related to agency missionPurposely selected from among 55 Australia/New Zealand HIAs completed by 2009 to reflect willingness to participate and diversity in timing, geography, and effectivenessPurposely selected from among all US HIAs completed as of 2013 for diversity in geography, sector, funding; subjectively successful
No. of HIAs reviewed8817 case studies, of which 8 were not HIAs as strictly defined811123
Level of decision makingLocal or regional, 83; national, 4; supranational, 1Local, 10; national, 6; multinational, 1Local or county 63; state 13; national 4; unclear 1Local or regional, 11Local or regional, 17; state, 6
SectorTransportation, 16; housing, 12; regeneration, 11; health care, 11; environment, 7; leisure, 7; industry, 5; other, 19Transportation, 5; urban planning, 5; agriculture, 2; environment, 2; industry, 1; infrastructure, 1; nutrition, 1Land use, 39; transportation, 21; housing/buildings/infrastructure, 17; waste management/site revitalization, 4Land use, 7; health service, 2; housing, 1; transportation, 1Built environment, 11; transportation, 3; natural resources/energy, 3; other, 6
Review methodsReviewed 88 case studies and 32 HIA methods papers; conducted email survey of 10 academicians, practitioners, and policy makersWorked with collaborators in each country to examine dimensions of effectiveness in case studies; included 3–6 interviews with stakeholders and decision makers for each HIAReviewed HIA reports; used minimum elements of HIA as defined by Bhatia et al (37) and 4-cell HIA effectiveness matrix (35); used Internet searches to assess impacts of HIA on decisionsReviewed HIA reports and questionnaires completed by HIA practitioners; conducted 33 semistructured interviews with HIA stakeholdersReviewed HIA reports; conducted 166 semistructured interviews with HIA team members, stakeholders, and decision makers; conducted Web survey of 144 HIA practitioners

Return to your place in the textTable 2. Findings and Impacts of 5 Major Health Impact Assessment (HIA) Impact Evaluation Reports, United States, Europe, Australia, and New Zealand, 2006–2015
Author and Year of PublicationFindings and Impacts
Davenport et al 2006 (32)
  • Important to monitor decisions to determine if impact occurred
  • Engaging decision makers is important but may compromise independence and impartiality
  • HIAs need to fit into the political and administrative environment in which they are being conducted; this fit may be as important as the technical methods used to conduct the HIA
Wismar et al, 2007 (35)
  • Described wide range of HIA methods used in 19 European countries
  • Reported that some complex projects entail a large number of discrete decisions, so effectiveness may vary with different decisions
  • Reported that none of the HIAs reviewed led to complete cancellation of a project
    • Frameworks
  • Defined 4-cell framework for overall effectiveness as direct, general, opportunistic, and none
  • Identified dimensions of effectiveness as health effectiveness, equity effectiveness, and community effectiveness
Rhodus et al, 2013 (34)
  • Raised awareness of health and related issues
  • Introduced health into discussions where health was typically absent (ie, informing decision making)
  • Engaged community members and stakeholders in decisions that affect them
  • Facilitated interdepartmental, interagency, and intersectoral collaborations
  • Built relationships and capacity within the community
  • For 50 of 81 HIAs for which impacts could be ascertained, effectiveness (28) was categorized as direct (60%), general (32%), opportunistic (2%), or none (6%)
  • Only 13 of 81 HIAs (16%) met all the minimum elements as defined by Bhatia et al (37)
Haigh et al, 2015 (33); Haigh et al, 2013 (36)
  • 91% of survey respondents reported that the HIA affected decision making
  • 83% of those with HIA impacts reported that HIA recommendations were easily incorporated into planning process
  • No respondent indicated that the HIA led to proposal postponement or cancellation
  • Some HIAs influenced implementation of proposal after a decision was made
  • Some HIAs helped legitimize involvement of the health sector in nonhealth sector decisions
  • Many HIA participants reported technical, conceptual, and social learning from the HIA process
    • Frameworks
  • Findings generally supported Harris-Roxas and Harris (67) conceptual framework for HIA effectiveness
  • The authors found Wismar’s 4-cell effectiveness framework (35) difficult to use
    • Concepts
  • Introduced concept of “proactive positioning” to recognize or create opportunities for conducting HIAs
Bourcier et al, 2015 (31)
  • 48% of decision makers reported HIA shaped their decision making
  • Made direct and concrete contributions from the recommendations to the decision-making process
  • Facilitated incorporation of health objectives into plans, policies, and programs of nonhealth-related agencies
  • Contributed to longer-term outcomes beyond initial decision targets
  • Institutionalized or strengthened existing relationships between individuals and organizations, or created new and enduring relationships between public health and other agencies such as transportation or planning departments
  • Helped decision makers and stakeholders see how health is connected to seemingly unconnected issues
  • Built consensus around controversial topics
  • Amplified community member voices in the decision-making process

Return to your place in the textTable 3. Success Factors in 5 Major Health Impact Assessment (HIA) Impact Evaluation Reports, United States, Europe, Australia, and New Zealand, 2006–2015
Author and Year of PublicationSuccess Factors
Davenport et al 2006 (32)         Role of decision makers
  • Involvement of decision makers/key stakeholders in the planning and conduct of the HIA (for example, commissioning, steering group, formulation of recommendations)
  • Input from professionals outside of the usual range of people involved in the decision-making process
  • Balance between decision maker ownership and HIA credibility
    • Policy-making process and environment
  • Clear commitment to HIA within organizational decision-making structure
  • Not being a controversial issue
  • Policy support for HIAs (including supporting legislation, promotion of consistency of methods, monitoring, and evaluation)
  • Provision of an enabling structure for HIA (manpower, evidence base, and intersectoral working)
  • Existing statutory frameworks supporting the use of HIAs
  • Recommendations chime with other political drivers
  • Recommendations realistic and can be incorporated into the existing planning process
    • Timing of HIAs
  • Timing of assessment should fit with the decision-making process
  • HIAs need to fit with decision makers’ rules, procedures, and time frames
    • HIA methods
  • Use of a consistent methodological approach
  • Consideration of a broad range of factors that can have an impact on community health and safety
  • Inclusion of empirical evidence relating the effects of a policy, program, or project on health
  • Quantification of impacts
  • Conduct by expert assessors (credibility of results)
    • Methods of reporting HIAs
  • Tailored presentation of information
  • Use insight into organizational concerns and priorities to shape recommendations
Wismar et al, 2007 (35)
  • Capacity to deal with community pressure
  • Timing in relation to the decision-making process
  • Involvement of organizations that can support conduct of the HIA
  • Culture of public health in the country
  • Political leadership
  • Public support
  • Involvement in early stage of proposal development
  • Legal backup for using health determinants in assessment
  • Creation of health systems units to support HIA
  • Clarification of who bears costs of HIA
Rhodus et al, 2013 (34)          HIA best practices listed as
  • Adherence to minimum elements of HIA as defined by Bhatia et al (37) or to National Research Council (3) criteria
  • Use of HIA as a tool for environmental impact assessment
  • Promotion of health equity
  • Documentation of screening and scoping
  • Rules of engagement memos
  • Communication plans
  • Stakeholder involvement
  • Transparent documentation of literature searches/reviews
  • Use of best available qualitative and quantitative data
  • Evaluation of quality of evidence
  • Identification of data gaps
  • Use of existing tools, metrics, methods, and standards
  • Adaptation of existing tools and methods
  • Detailed documentation of data and methods
  • Use of geographic information systems
  • Use of impact pathways and logic frameworks
  • Clear summary of impact assessments
  • Confidence estimates and assessments of uncertainty
  • Documentation of process for prioritizing recommendations
  • Recommendations that meet established feasibility criteria
  • Development of an implementation plan for recommendations
  • Clear and transparent HIA reporting
  • Process evaluation
  • Establishment of monitoring plans for impact and outcome evaluation
Haigh et al, 2015 (33); Haigh et al, 2013 (36)
  • Use of a structured stepwise process
  • Flexibility to adapt process to local context
  • Use of evidence to support recommendations
  • Capacity and experience among practitioners and stakeholders
  • Involvement of decision makers and others who can influence decisions or implement recommendations
  • High-quality relationships across sectors
  • Engagement of community stakeholders
  • Shared goals and values among HIA participants
  • Use of “proactive positioning” to achieve optimal timing
  • Flexibility in time and timeliness to conduct HIA
Bourcier et al, 2015 (31)
  • Method of screening and choosing HIA targets wisely because an HIA is not always the right tool
  • Investment in the right team to conduct HIA
  • Engagement of key stakeholders
  • Engagement of decision makers throughout the process
  • Development of clearly articulated recommendations that spark action
  • Delivery of compelling messages to the right audiences at the right times
  • Use of approach to take advantage of HIA credibility

Return to your place in the textTable 4. Challenges in 5 Major Health Impact Assessment (HIA) Impact Evaluation Reports, United States, Europe, Australia, and New Zealand, 2006–2015
Author and Year of PublicationChallenges
Davenport et al, 2006 (32)          Role of decision makers
  • Limited organizational unique HIA conducted by champions external to the decision-making organization
  • Not having the support of decision makers
    • Policy-making process and environment
  • Lack of awareness of health issues by nonhealth-related sectors
  • Lack of knowledge (on behalf of those conducting HIA) of the policy-making environment
  • HIA not a statutory or policy requirement
    • Conduct and reporting of HIAs
  • Lack of an established standard method for conducting an HIA
  • Time, resources, and staffing
  • Use of jargon
Wismar et al, 2007 (35)
  • HIA timing
  • Quality of communication among stakeholders
  • Quality of HIA predictions
  • Conflicting objectives between health and other sectors in which HIA is done
  • Links among local, national, and international decision making
  • Lack of institutionalization of HIAs
  • Uneven development of HIAs across countries
Rhodus et al, 2013 (34)
  • Ability to discern impact of HIAs on decision-making processes by Internet searches is limited
    • Areas for improvement
  • Increase adherence to the minimum elements of HIA as defined by Bhatia et al (37) or to National Research Council (3) criteria
  • Expand use of HIA to inform decision making at local, state, and national levels
  • Use consistency in HIA terminology
  • Expand use of existing tools and resources for HIAs
  • Identify and close data gaps
Haigh et al, 2015 (33); Haigh et al, 2013 (36)
  • Dealing with problem makers and proposal opponents
  • Responding to unanticipated events such as change in decision maker
  • Identifying effectiveness when goals of HIA were not explicit
Bourcier et al, 2015 (31)
  • Underestimation of overall level of effort
  • Engagement of stakeholders and decision makers
  • Pace of decision making and political administration changes
  • Lack of access to relevant data
  • Incorporation of equity and vulnerable populations consistently and meaningfully
  • Follow-up on recommendations

Return to your place in the textTable 5. Impact Evaluations of Selected Individual Health Impact Assessments (HIAs) in Various Countries, 2004–2013
Author and Year of PublicationName and Location of HIASectorExample of HIA Recommendation Adopted in Final Project or PlanSelected Feature, Finding, or Impact
Mindell et al, 2004 (23)London’s draft transport strategy HIA, United KingdomTransportationGiving priority to infrastructure and services that benefit London’s economically deprived communitiesAchieved policy changes and raised awareness of health issues among policy makers
Mathias and Harris-Roxas, 2009 (24)Christchurch Urban Development Strategy HIA, New ZealandLand use17 actions (not specified) in final strategy addressed HIA recommendationsIncluded process and impact evaluation; strong cross-sector relationships contributed to HIA success
Quigley and Watts, Ltd, 2010 (25)Evaluation of the Whānau Ora HIA of the Draft Wairarapa Alcohol Strategy, New ZealandAlcohol policyHIA recommendations led to revisions of draft strategyPublic health team conducting HIA increased its knowledge and skills with HIA
Clark County, 2011 (26)Clark County Bicycle and Pedestrian Master Plan HIA, Clark County, WashingtonTransportationHealth and equity incorporated as project selection criteriaOf 11 HIA recommendations, 8 were fully adopted and 3 partially adopted
Harris-Roxas et al, 2011 (27)Equity-focused HIA on health promotion policy implementation plan, New South Wales, AustraliaHealth promotion policyChange in resource allocation split between rural and urban servicesFocused on equity aspects of a health-sector plan
Ross et al, 2012 (28)Atlanta BeltLine transit, parks, and redevelopment project HIA, Atlanta, GeorgiaLand use and transportationPublic health professional added to project advisory boardExplicitly tied findings to recommendations and to subsequent impacts
O’Mullane and Quinlivan, 2012 (29)Four policy HIAs on transport, housing, community development, air quality plan, IrelandMultiple sectorsTransport HIA “was used to plan further health promotion and community planning activities”Found local government can be an enabler or barrier to success of HIA
Canterbury District Health Board, 2013 (30)Evaluation of the HIA of the Canterbury Regional Land Transport Strategy, New ZealandTransportationCommitting future funding to policies supporting active transport and public transportNoted that HIA report recommendations did not include effective monitoring for health issues
 Top
The opinions expressed by authors contributing to this journal do not necessarily reflect the opinions of the U.S. Department of Health and Human Services, the Public Health Service, the Centers for Disease Control and Prevention, or the authors' affiliated institutions.

No hay comentarios: