Methods for Evaluating Natural Experiments in Obesity: A Systematic Evidence Review

SYSTEMATIC REVIEW

December 1, 2017

This report is available in PDF only (Full Report [12.2 MB]). People using assistive technology may not be able to fully access information in these files. For additional assistance, please contact us.

Purpose of Review

To characterize studies of programs and policies in obesity prevention and control in terms of data sources, data linkages, measures reported, study designs, and analytic approaches, and identify needed methodological advances.

Key Messages

Relevant programs, policies, or built environment changes were evaluated in 156 natural experiments, 118 experimental studies, and 20 other studies.
Criteria for a data system (source exists, is available for research, is sharable, and has outcomes of interest) were met by 106 data sources.
Thirty-seven percent of U.S. data systems were linked to secondary data.
Outcome measures included dietary behavior (148 studies), physical activity (152 studies), childhood weight (112 studies), and adult weight (32 studies).
Natural experiments most commonly used regression models comparing exposed and unexposed groups at one time.
Natural experiments generally had moderate risk of selection bias and high risk of bias for losses to follow-up.
Research could be advanced by more use of data dictionaries, reporting standards on data linkage, long-term obesity-related outcomes, and study designs with multiple pre- and post-exposure time points.

Structured Abstract

Objectives: Obesity is an enormous public health problem among adults and children. Our objective was to systematically review studies evaluating programs and policies addressing obesity prevention and control in terms of their population-based data sources, use of data linkages, measures reported, study designs, and analytic approaches. The overarching goal of the review was to identify methodological advances that could strengthen research that uses natural experiments to evaluate the effectiveness of policies and programs to prevent and control obesity.

Data sources: We systematically searched PubMed, CINAHL, PsycINFO, and EconLit from 2000 to August 21, 2017 to identify all US and non-US studies of programs or policies targeting obesity prevention and control in people of all ages and in any setting.

Review methods: Two independent reviewers screened abstracts and full-text articles. We required articles to be in English, address a program, policy or built environment change, include 100 or more study subjects, and have a defined comparison or unexposed group. We used the Effective Public Health Practice Project (EPHPP) tool to rate studies for their risk of bias. This tool rates studies for their ability to draw causal inferences about program effectiveness.

Results: The search identified 26,316 unique citations. Of the 294 studies (reported in 312 articles) eligible for inclusion (188 US and 106 non-US), 156 (53 percent) were natural experiment studies, 118 (40 percent) were experimental studies (randomized or non-randomized controlled trials), and 20 (7 percent) had other study designs that did not fall into either of the other categories.

Data Sources: We identified 143 secondary data sources and 26 sharable primary data sources, totaling 116 sharable data sources after duplicates were removed. One hundred and six (71 US and 35 non-US) data sources met criteria for a data system (data source exists, is available for research, is sharable, and contains outcomes of interest). Sixty-two percent of the 71 US data systems contained at least one of the main measures for weight or body mass index (BMI) in adults or children, or dietary or physical activity behaviors. Fifty-three percent of the 71 US data systems included at least one outcome related to the food-environment, physical activity environment, commuting behavior, or purchasing behavior, or included information about a relevant exposure in a policy, program, or built environment change. These 71 US data systems often reported more than one outcome. Thirty-seven percent of the 71 US data systems were linked with a secondary data source or system other than the primary data source. Most studies that linked their data systems with external data systems used an individual-level key or a geographic allocation.

Outcomes/Measures: Of the 294 included studies, we identified 112 studies with childhood weight measures, 32 studies with adult weight measures, 152 studies with physical activity measures, and 148 studies with dietary measures. Thirty-seven of the 294 studies reported on outcomes related to the food environment, physical activity environment, commuting behavior, or purchasing behavior.

Study Design and Methods: Natural experiment studies most commonly used cross-sectional comparisons of exposed and unexposed groups (n=55; 35 percent). Difference-in-differences approaches that compared exposed and unexposed groups before and after an exposure were used in 45 studies (29 percent), while 48 studies (31 percent) used pre/post designs that compared one group before and after an exposure. Most natural experiment studies were rated as having a “weak” global rating (i.e., high risk of bias), with 63 percent having a weak rating for handling of withdrawals and dropouts, 42 percent having a weak rating for study design, 40 percent having a weak rating for confounding, and 26 percent having a weak rating for data collection. Experimental studies were rated as “strong” (low risk of bias) in study design, control of confounding, and data collection methods, but were weaker in blinding and selection bias. We identified methodological and analytic advances that would help to strengthen efforts to estimate the effect of programs, policies, or built environment changes on obesity prevention and control, such as consistent use of data dictionaries, reporting standards on linkage methods of data sources, data sources with long term public health surveillance of obesity and health behavioral outcomes, and use of study designs with multiple pre- and post- exposure time points.

Conclusions: Our systematic review identified numerous natural experiment studies (n=156) and data sources, including sharable and non-sharable (n=216) that have been used to estimate the effect of programs, policies, or built environment changes on obesity prevention and control. The studies used a wide variety of outcome measures and analytic methods, often with substantial risk of bias. The findings reinforce the need for methodological and analytic advances that would strengthen efforts to improve obesity prevention and control.

Citation

Bennet W, Cheskin LJ, Wilson RF, Zhang A, Tseng E, Shogbesan O, Knapp EA, Stuart E, Bass EB, Kharazzi H. Methods for Evaluating Natural Experiments in Obesity: A Systematic Evidence Review. Comparative Effectiveness Review No. 204. (Prepared by the Johns Hopkins University Evidence-based Practice Center under Contract No. 290-2012-00007-I). AHRQ Publication No. 18-EHC006-EF. Rockville, MD: Agency for Healthcare Research and Quality; December 2017. www.effectivehealthcare.ahrq.gov/reports/final.cfm. DOI: http://doi.org/10.23970/AHRQEPCCER204[link is external].