jueves, 6 de abril de 2017

Food Policy Council Self-Assessment Tool: Development, Testing, and Results

Food Policy Council Self-Assessment Tool: Development, Testing, and Results

Centers for Disease Control and Prevention. CDC twenty four seven. Saving Lives, Protecting People



PCD logo



Food Policy Council Self-Assessment Tool: Development, Testing, and Results

TOOLS AND TECHNIQUES — Volume 14 — March 2, 2017

Larissa Calancie, PhD1; Nicole E. Allen, PhD2; Bryan J. Weiner, PhD3; Shu Wen Ng, PhD4; Dianne S. Ward, EdD5; Alice Ammerman, DrPH5,6 (View author affiliations)

Suggested citation for this article: Calancie L, Allen NE, Weiner BJ, Ng SW, Ward DS, Ammerman A. Food Policy Council Self-Assessment Tool: Development, Testing, and Results. Prev Chronic Dis 2017;14:160281. DOI: http://dx.doi.org/10.5888/pcd14.160281.
PEER REVIEWED

Abstract

A large number of food policy councils (FPCs) exist in the United States, Canada, and Tribal Nations (N = 278), yet there are no tools designed to measure their members’ perceptions of organizational capacity, social capital, and council effectiveness. Without such tools, it is challenging to determine best practices for FPCs and to measure change within and across councils over time. This study describes the development, testing, and findings from the Food Policy Council Self-Assessment Tool (FPC-SAT). The assessment measures council practices and council members’ perceptions of the following concepts: leadership, breadth of active membership, council climate, formality of council structure, knowledge sharing, relationships, member empowerment, community context, synergy, and impacts on the food system. All 278 FPCs listed on the Food Policy Network’s Online Directory were recruited to complete the FPC-SAT. Internal reliability (Cronbach’s α) and inter-rater reliability (AD, rWG(J), ICC [intraclass correlations][1], ICC[2]) were calculated, and exploratory and a confirmatory factor analyses were conducted. Responses from 354 FPC members from 94 councils were used to test the assessment. Cronbach’s α ranged from 0.79 to 0.93 for the scales. FPC members reported the lowest mean scores on the breadth of active membership scale (2.49; standard deviation [SD], 0.62), indicating room for improvement, and highest on the leadership scale (3.45; SD, 0.45). The valid FPC-SAT can be used to identify FPC strengths and areas for improvement, measure differences across FPCs, and measure change in FPCs over time.
 Top

Introduction

In a 2011 report by the Centers for Disease Control and Prevention (CDC), the first strategy recommended to increase fruit and vegetable consumption was to establish food policy councils (FPCs) as a way to improve the food environment at state and local levels (1). FPCs are organizations that bring together diverse members of the community to inform food policy and systems change and to coordinate or implement programs that aim to increase food access (2). In 2015, there were 278 FPCs in the United States, Canada, and Tribal Nations (3). FPCs are composed of representatives from many sectors of a local food system: producers, consumers, distributors, retailers, food processors, policy and decision makers, public health practitioners, food waste collectors, and hunger advocacy groups (4). FPCs’ organizational missions vary, but they often aim to inform changes that lead to increased access to nutritious foods for all members of the local food system through changes to agricultural, economic, environmental, and social programs and policies. FPCs may seek to inform policies and programs within organizations, or more broadly through municipal, county, or state-level policies (2).
Peer-reviewed literature, white papers, and guidance reports describe FPC activities, achievements, and challenges (2,5–7). Yet, few measurement tools exist to assess FPCs. Without such tools, best practices cannot be determined, and the mechanisms through which FPCs affect their food systems are difficult to explain. Measurement tools provide an opportunity to determine factors that differ across councils, determine what factors are associated with council effectiveness, and assess change within councils over time. Funders, researchers, FPCs, and the technical assistance groups that work with FPCs can all benefit from FPC-specific evaluation tools. Funders often require funding recipients to evaluate their work to determine the impact of their investments. Researchers also seek to evaluate practical approaches, such as FPCs, to complex, real-world problems. FPCs and the technical assistance groups that work with them can use measurement tools to conduct self-assessments to identify strengths and areas for improvement. No validated FPC-specific tools are available to evaluate councils.
The purpose of this study was to describe the development, testing, and findings from a self-assessment tool that measures FPC members’ perceptions of their council’s organizational capacity, social capital, and council effectiveness. The concepts measured by the self-assessment tool are informed by the literature on health-oriented community coalitions. The self-assessment tool can be used to guide FPC development, tailor capacity-building interventions for FPCs, and measure the internal functioning of FPCs.
 Top

Conceptual Framework

We reviewed the literature on health-oriented community coalitions and FPCs to identify concepts to measure through the Food Policy Council Self-Assessment Tool (FPC-SAT). In reviewing the literature, we identified a parsimonious model explaining how community collaboratives influence institutionalized change in their communities (8). The concepts in the model aligned with the community coalition literature that we reviewed and depicted a set of relationships between concepts that we could test empirically (8–15). We adapted that model to create the FPC Framework (Figure 1) by including a credibility concept and by defining the outcome of interest as council effectiveness. Council effectiveness is defined as synergy, or the power to combine resources and perspectives to create new approaches to complex problems (13), and FPC council members’ perceptions of their councils’ impact on their food system. The concepts included in the FPC Framework are listed in Table 1.
Food Policy Council Framework. Source: Allen NE, Javdani S, Lehrner AL, Walden AL. “Changing the text”: modeling council capacity to produce institutionalized change. Am J Community Psychol 2012;49(3-4):317–31.
Figure 1. Food Policy Council Framework. Source: Allen NE, Javdani S, Lehrner AL, Walden AL. “Changing the text”: modeling council capacity to produce institutionalized change. Am J Community Psychol 2012;49(3-4):317–31. [A text description of this figure is also available]
 Top

Food Policy Council Self-Assessment Tool

Item and scale generation

We drafted a preliminary set of questions for the FPC-SAT Tool based on past studies and the authors’ experience working with FPCs (2,4,8). Nine food council experts who either studied councils or provided technical assistance to councils provided feedback about the relevancy, clarity, and comprehensiveness of each question on the assessment (16). Food council experts were recruited through the Food Policy Council Working Group of the CDC-funded Nutrition and Obesity Policy Research and Evaluation Network (17) and through the North Carolina–based Community Food Strategies group (18). Experts included one who works with Tribal communities and who provided feedback that was specifically relevant to those populations. These experts provided input about the construct validity of the items (19). Two survey methodologists reviewed the questions and recommended simplifying the questions and providing additional question formats. The assessment was revised on the basis of feedback from content and methodology experts and entered into Qualtrics, an online survey platform.

Cognitive response testing

A convenience sample of 4 food council leaders and members was recruited to complete cognitive response interviews. The participants represented a regional council in Washington State, a municipal council in New Jersey, and county-level councils in North Carolina and Kansas. Each participant was sent a link to the Qualtrics FPC-SAT immediately before a telephone interview. The interviewer conducted cognitive response interviews by asking the participants to read and answer the questions aloud to better understand how participants were interpreting the questions (20,21). Specific questions about response options, phrasing, and cognitive burden were also asked. The assessment was revised on the basis of these interviews.

Reliability and validity

Cronbach’s α coefficients were calculated to determine the reliability of assessment items within each scale (22). Average deviation (ADM(J)) (23) and a within-group reliability index (rwg(J)) (24) were calculated to assess inter-rater agreement within FPCs. Interclass correlations (ICC[1], ICC[2]) were calculated from a one-way random effects analysis of variance (ANOVA) to quantify the effect of shared council membership on participants’ scale responses (25,26). ICC(1) estimates the amount of variation that can be explained by council membership and a high ICC(2) indicates that councils can be differentiated in terms of their members’ responses on a scale (25,27).
Exploratory factor analysis was used to examine whether items grouped together as predicted. Items with factor loadings of 0.3 or higher were grouped into that factor (19). Factors loadings were clarified by using an oblique rotation. Scales were created from the average of all the items measuring concept. “Not applicable” responses were considered missing data. Scales were grouped together into the following factors according to the FPC Framework: 1) Organizational capacity — leadership, breadth of active membership, formality of council structure, and inclusivity of council climate; 2) Social capital — knowledge, relationships, credibility, and member empowerment; and 3) Council effectiveness — synergy and perceived impact on the food system. Confirmatory factor analyses (CFA) were used to test whether scales correlated with these hypothesized factors. The CFA model was estimated by using maximum likelihood with missing values method. Bootstrapping was conducted and FPC members were clustered by their council identification during model estimation. CFA model fit was assessed using the Comparative Fit Index (CFI), the Tucker Lewis Index, and the Root Mean Square Error of Approximation (RMSEA) (28). Statistics were calculated by using STATA version 14.0 (StataCorp LLC) and the multilevel package version 2.5 in R (http://CRAN.R-project.org/package=multilevel).

Testing the tool

The 2015 online edition of the Food Policy Network FPC Directory was used to recruit members from all 278 FPCs in the United States, Canada, and Tribal Nations (3). Council members must have attended at least 2 meetings in the past year to be eligible to participate. Council contacts from the FPC Directory were recruited via email and asked to share the FPC-SAT link with their council members. Three reminder emails were sent to council contact persons, indicating how many council members had completed the assessment. Each participant could opt to receive a $5 Amazon e-gift card to incentivize individual members to participate. Councils with 8 or more participants received a feedback report summarizing their council’s aggregate responses as incentives for a high number of participants per council. Data were collected from July 2014 through October 2015. The institutional review board at the University of North Carolina, Chapel Hill, exempted this study.
Participant (N = 354) and council (N = 94) characteristics are listed in Table 2. Most participants were female (n = 240, 74%) and white (n = 271, 84%). Most had been members of their councils for 1 to 5 years (n = 237, 66%). The average age of council members was 6.27 years (standard deviation [SD], 5.10 y), and from 1 to 12 members per council completed the FPC-SAT.
Cognitive response participants said they understood the questions on the FPC-SAT and could easily answer most questions. Participants recommended some word changes (eg, be more specific about “experts,” remove or define “practitioners”) and adding several more response options to the “perceived impact” section. They also had several formatting suggestions: combining blocks of questions on a page so that the progress bar at the bottom of the survey moved more quickly, seeing a question stem at the top of a set of questions rather than re-reading the stem in a related set of questions, and being sure that the “submit” option is very clear on the last page. The 4 cognitive response participants recommended reducing the length if possible, but did not identify any particular questions or sections that seemed superfluous. One participant recommended providing an option to opt out of the gift card incentive because some government employees are not allowed to accept such incentives. We implemented all suggested changes.
Scale properties are listed on Table 3. From 267 to 353 participants selected a response other than “not applicable” to each item. Cronbach α ranged from 0.79 to 0.93, indicating high interrelatedness among items in each scale. Mean within-group agreement (rwg(J)) was above 0.70 for most scales, with the exception of relationships (rwg(J) = 0.69) and member empowerment (rwg(J) = 0.62), indicating that council members within councils generally agreed on their ratings for each scale (Table 3) (27). Mean ADM(J) values below 0.67 indicate agreement among members within FPCs (Table 3) (27). All scales had mean ADM(J) values of 0.67 or lower other than relationships (ADM(J) = 0.69) and member empowerment (ADM(J) = 0.67).
The 1-way ANOVA models for each scale produced a range of ICC(1)s and ICC(2)s, indicating that council membership explained a portion of the variation in most but not all scales (Table 3). Council membership was a significant consideration for the following scales: breadth of active membership (P < .001), formality of council structure (<.001), inclusivity of council climate (P < .001), relationships (P = .01), credibility (P < .001), and synergy (P = .01).
Exploratory factor analysis indicated that the items grouped together within scales as expected with factor loadings of 0.3 or higher for each item. There was minimal cross loading (factor loadings of 0.3 or higher on more than 1 factor) between items from different scales, with the exception of some cross-loading between leadership and council climate items. Leadership sets the tone for whether a council has an inclusive climate and therefore may be difficult to parse in this assessment. The CFA was a good fit with the data (χ2 = 76.146, df = 32, P = <.001, CFI = 0.970, TLI = 0.958, RMSEA = 0.062, p-close = 0.121). These results indicate that the observed variables (the scales) were good measures of the hypothesized factors. The covariances between each factor ranged from 0.60 to 0.71 and were significant (P < .001), indicating that the factors are related yet distinct.
We calculated concept averages from FPC leaders and members (Figure 2). Approximately half of participants reported “member” as their position within their councils (n = 172, 49%). The average concept scores were similar for the 2 groups. In the averaged combined scores of leaders and members, the leadership scale had the highest mean (mean, 3.45; SD, 0.45), followed by formality of council structure (mean, 3.25 SD, 0.60), and synergy (mean, 3.17; SD, 0.51). Breadth of active membership had the lowest scale average (mean, 2.49; SD, 0.62) followed by credibility (mean, 2.58; SD, 0.80, and impact (mean, 2.76; SD, 0.52) (Table 3). Table 4 shows the number of respondents, mean, and standard deviation for each of the 59 separate items in the scales.
Concept means and standard deviations measured by the Food Policy Council Self-Assessment Tool (FPC-SAT) (range 1–4) for a sample (N = 354) of food policy council leaders and members. Leaders (formal or informal) (n = 51, 15%), administration or staff (secretary, coordinator) (n = 49, 14%), and working group chairs or members of steering committee (n = 77, 22%) were grouped together as leaders because of their additional investment in the councils.
Figure 2. Concept means and standard deviations measured by the Food Policy Council Self-Assessment Tool (FPC-SAT) (range 1–4) for a sample (N = 354) of food policy council leaders and members. Leaders (formal or informal) (n = 51, 15%), administration or staff (secretary, coordinator) (n = 49, 14%), and working group chairs or members of steering committee (n = 77, 22%) were grouped together as leaders because of their additional investment in the councils. [A tabular version of this figure is also available.]
The FPC-SAT asked participants to indicate their level of agreement with statements about their FPC’s impact on the food system in which they work (Table 4). Participants reported the areas of most impact as the following: facilitating changes in policy or practice that will promote our council's mission (mean, 3.13; SD, 0.62), increasing opportunities to purchase locally produced agricultural products (mean, 3.05; SD, 0.71), promoting social justice in the food system (mean, 3.03; SD, 0.69), and increasing access to healthy food in our community (mean, 3.02; SD, 0.70). Participants reported the lowest impact in the following areas: promoting occupational safety in the agricultural sector (mean, 2.22; SD, 0.71) and promoting humane treatment of animals in the agricultural sector (mean, 2.25; SD, 0.67). Additional research is needed to understand how FPCs prioritize which impact areas to pursue.
To make the tool more practical for repeated use by FPCs, we suggest abbreviating the tool. Table 4 shows suggested items to remove based on their effect on scale α values and their conceptual contribution to the concept scale. Table 5 shows the reliability of the abbreviated tool. The reliability measures are very similar in the original and abbreviated tool.
 Top

Summary and Applications

This study described the development and testing of the first self-assessment tool adapted for FPCs to measure their members’ perceptions of their council’s organizational capacity, social capital, synergy, and impact on their food system. The FPC-SAT was adapted from Allen and colleague’s work and informed by literature on FPCs and community-based collaborations (4,8,15). Feedback from FPC members and experts was incorporated into the assessment. A final version of the assessment tool was tested with 354 council members from 94 councils in the United States, Canada, and Tribal Nations. Item reliability was high for all concept scales.
Researchers and practitioners can use the FPC-SAT to explore how FPCs function and why some may be more successful than others. Findings from the assessment tool may be useful in monitoring FPCs over time, especially before and after capacity-building interventions. The instrument could also reveal whether council members’ perceptions about their councils are similar or discordant and why that might be. Moreover, the FPC-SAT can help researchers identify the mechanism by which FPCs influence their food systems.
The FPC-SAT is a tool that council members and technical assistance groups can use to strengthen councils. In one study, authors interviewed and surveyed participants in an FPC and then provided feedback to the FPC regarding organizational capacity factors such as leadership (29). They reported an increase in council activity (eg, produced more media releases) following the provision of feedback (29). Training and technical assistance should be tailored to collaboratives’ needs (30), which can be illuminated by the FPC-SAT. For example, FPCs with high scores in certain areas on the FPC-SAT can focus improvement efforts on other areas that have more room for improvement. The FPC-SAT can also be used to measure the impact of training and technical assistance on specific items within scales or on scale averages.
In testing this assessment tool, we found several important patterns. Most FPC members are white women aged 35 to 65 years. A study of the relationship between community coalition factors and community impact found that greater racial diversity was associated with coalitions’ ability to change public policy and improve community prevention systems (31). FPCs may consider strategies for increasing diversity among their members. More research is needed to determine what member engagement strategies are effective at increasing member diversity. When looking at the mean scores for each concept scale, breadth of active membership, credibility, and impact had the lowest means. These are areas where councils may choose to direct more energy or seek external support such as training, technical support, and resources. Councils with low breadth of active membership scores, for example, may consider engaging potential members in a variety of settings. Councils can hold public forums where community members can provide input on the councils’ activities, or they can boost active participation by aligning council priorities with members’ goals and values.
Another interesting finding was the variability in inter-rater agreement and interclass correlation statistics among the scales. Inter-rater agreement and interclass correlations were both low for the member empowerment scale. Therefore, the member empowerment scale appears to capture perceptions that vary more among individuals, independent of their council affiliation, than the perceptions captured in the other scales. Researchers have found that community collaboratives can be empowering environments for their members but that members’ perceptions of empowerment can be influenced by other factors as well, such as the climate in their home organizations (32). Additional research into what factors explain variation in participants’ perceptions of empowerment in FPCs is warranted.
To our knowledge, this is the first study to recruit all members from all FPCs listed in the Food Policy Network’s FPC Directory (3). However, members from one-third of all councils listed in the directory submitted a survey. More than one member responded from only 35 councils, or 12% of total councils. FPC members who completed the assessment may be more invested in their councils or more motivated to see evaluation findings than those who did not complete the survey. The limited response rate may indicate that the survey was too long or that FPC members did not see the value in completing the assessment. The abbreviated tool may be more appealing to these members. Future work will test the use of the abbreviated FPC-SAT.
In conclusion, this study describes the Food Policy Council Self-Assessment Tool development, testing, and findings from council members in the United States, Canada, and Tribal Nations. Researchers, public health practitioners, and FPC members can use the tool to identify strengths and areas for improvement within councils and to measure change in these areas over time. Using the tool to understand council members’ perceptions of their councils’ organizational capacity, social capital, and council effectiveness can provide insight for researchers trying to determine how councils affect change in their food system. More research is needed to explore whether factors associated with FPC impact are not captured in the FPC-SAT. Empirical research is needed to test the relationships between the factors measured in the assessment to develop a theory or framework explaining how FPCs function.
 Top

Acknowledgments

We thank CDC’s Nutrition and Obesity Policy Research and Evaluation Network for its support (3-U48-DP001944-05S1). The project described was also supported by the National Center for Advancing Translational Sciences, National Institutes of Health (NIH), through grant award no. UL1TR001111. The content is solely the responsibility of the authors and does not necessarily represent the official views of NIH. This work was conducted at the Center for Health Promotion and Disease Prevention at the University of North Carolina at Chapel Hill, a Prevention Research Center (5U48DP001944).
 Top

Author Information

Corresponding Author: Larissa Calancie, PhD, Center for Health Equity Research, Department of Social Medicine, School of Medicine, University of North Carolina at Chapel Hill, Chapel Hill, NC 27599. Telephone: 919-843-1349. Email: lcalancie@unc.edu.
Author Affiliations: 1Center for Health Equity Research, Department of Social Medicine, School of Medicine, University of North Carolina at Chapel Hill, Chapel Hill, North Carolina. 2Department of Psychology, University of Illinois at Urbana-Champaign, Champaign, Illinois. 3Department of Global Health, School of Public Health, University of Washington, Seattle, Washington. 4Carolina Population Center, University of North Carolina at Chapel Hill, Chapel Hill, North Carolina. 5Department of Nutrition, Gillings School of Global Public Health, University of North Carolina at Chapel Hill, Chapel Hill, North Carolina. 6Center for Health Promotion and Disease Prevention, University of North Carolina at Chapel Hill, 1700 Martin Luther King Jr. Blvd., CB# 7426, Chapel Hill, NC, 27599-7426.
 Top

References

  1. Centers for Disease Control and Prevention. Strategies to prevent obesity and other chronic diseases: the CDC guide to strategies to increase the consumption of fruits and vegetables. Atlanta (GA): US. Department of Health and Human Services; 2011. http://www.cdc.gov/obesity/downloads/fandv_2011_web_tag508.pdf. Accessed February 4, 2017.
  2. Harper A, Alkon A, Shattuck A, Holt-Giménez E, Lambrick F. Food policy councils: lessons learned. Inst Food Dev Policy; 2009;1–63. https://foodfirst.org/publication/food-policy-councils-lessons-learned/. Accessed February 4, 2017.
  3. Food Policy Network. Food Policy Council Directory. Johns Hopkins Bloomberg School of Public Health; 2015.http://www.jhsph.edu/research/centers-and-institutes/johns-hopkins-center-for-a-livable-future/projects/FPN/directory/online/. Accessed January 5, 2016.
  4. Scherb A, Palmer A, Frattaroli S, Pollack K. Exploring food system policy: a survey of food policy councils in the United States. J Agric Food Syst Community Dev 2012;2(4):1–12.
  5. Ventura S. A working white paper: Wisconsin Food Systems Council; 2013 p. 1–23.
  6. DiLisio C. Food Policy Councils: helping local, regional, and state governments address food system challenges. American Planning Association; 2011. http://ucanr.edu/sites/MarinFoodPolicyCouncil/files/178441.pdf. Accessed February 4, 2017.
  7. Walsh CC, Taggart M, Freedman DA, Trapl ES, Borawski EA. The Cleveland-Cuyahoga County Food Policy Coalition: “we have evolved.” Prev Chronic Dis 2015;12:E86. CrossRef PubMed
  8. Allen NE, Javdani S, Lehrner AL, Walden AL. “Changing the text”: modeling council capacity to produce institutionalized change. Am J Community Psychol 2012;49(3-4):317–31. CrossRefPubMed
  9. Butterfoss F, Kegler M. The community coalition action theory. In: Emerging theories in health promotion practice and research. 2nd edition. Hoboken (NJ): John Wiley and Sons; 2009.
  10. Goodman RM, Speers MA, McLeroy K, Fawcett S, Kegler M, Parker E, et al. Identifying and defining the dimensions of community capacity to provide a basis for measurement. Health Educ Behav 1998;25(3):258–78. CrossRef PubMed
  11. Granner ML, Sharpe PA. Evaluating community coalition characteristics and functioning: a summary of measurement tools. Health Educ Res 2004;19(5):514–32. CrossRef PubMed
  12. Kegler MC, Steckler A, McLeroy K, Malek SH; American Stop Smoking Intervention Study for Cancer Prevention. Factors that contribute to effective community health promotion coalitions: a study of 10 Project ASSIST coalitions in North Carolina. Health Educ Behav 1998;25(3):338–53. CrossRef PubMed
  13. Lasker RD, Weiss ES, Miller R. Partnership synergy: a practical framework for studying and strengthening the collaborative advantage. Milbank Q 2001;79(2):179–205, III–IV. CrossRefPubMed
  14. Roussos ST, Fawcett SB. A review of collaborative partnerships as a strategy for improving community health. Annu Rev Public Health 2000;21(1):369–402. CrossRef PubMed
  15. Zakocs RC, Edwards EM. What explains community coalition effectiveness? A review of the literature. Am J Prev Med 2006;30(4):351–61. CrossRef PubMed
  16. DeVellis RF. Scale development: theory and applications. Vol. 26. Sage Publications; 2011.
  17. Blanck HM, Kim SA. Creating supportive nutrition environments for population health impact and health equity: an overview of the Nutrition and Obesity Policy Research and Evaluation Network’s efforts. Am J Prev Med 2012;43(3, Suppl 2):S85–90. . Accessed January 5, 2016. CrossRef PubMed
  18. Community food strategies. 2014. http://communityfoodstrategies.com/. Accessed January 5, 2016.
  19. Kline RB. Principles and practice of structural equation modeling. Guilford press; 2011.
  20. Carbone ET, Campbell MK, Honess-Morreale L. Use of cognitive interview techniques in the development of nutrition surveys and interactive nutrition messages for low-income populations. J Am Diet Assoc 2002;102(5):690–6. CrossRef PubMed
  21. Willis GB. Cognitive interviewing and questionnaire design: a training manual. US Department of Health and Human Services, Centers for Disease Control and Prevention, National Center for Health Statistics; 1994.
  22. Nunnally J. Psychometric methods. New York (NY): McGraw; 1978.
  23. Burke MJ, Finkelstein LM, Dusig MS. On average deviation indices for estimating interrater agreement. Organ Res Methods 1999;2(1):49–68. CrossRef
  24. James LR, Demaree RG, Wolf G. Estimating within-group interrater reliability with and without response bias. J Appl Psychol 1984;69(1):85–98. CrossRef
  25. Bliese PD. Within-group agreement, non-independence, and reliability: implications for data aggregation and analysis. In: Klein KJ, Kozlowski SW. Multilevel theory, research, and methods in organizations: foundations, extensions, and new directions. New York (NY): Jossey-Bass; 2000. p. 349–381.
  26. Shea CM, Jacobs SR, Esserman DA, Bruce K, Weiner BJ. Organizational readiness for implementing change: a psychometric assessment of a new measure. Implement Sci 2014;9(1):7.CrossRef PubMed
  27. Bliese P. Multilevel Modeling in R (2.5) — A Brief Introduction to R, the multilevel package and the nlme package. 2013. http://www.oalib.com/references/19892921. Accessed February 4, 2017.
  28. Hu L, Bentler PM. Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Struct Equ Model Multidiscip J. 1999;6(1):1–55. CrossRef
  29. Hawe P, Stickney EK. Developing the effectiveness of an intersectoral food policy coalition through formative evaluation. Health Educ Res 1997;12(2):213–25. CrossRef PubMed
  30. Wandersman A, Duffy J, Flaspohler P, Noonan R, Lubell K, Stillman L, et al. Bridging the gap between prevention research and practice: the interactive systems framework for dissemination and implementation. Am J Community Psychol 2008;41(3-4):171–81. CrossRef PubMed
  31. Hays C, Hays S, DeVille J, Mulhall P. Capacity for effectiveness: the relationship between coalition structure and community impact. Eval Program Plann 2000;23(3):373–9. CrossRef
  32. Javdani S, Allen NE. Councils as empowering contexts: mobilizing the front line to foster systems change in the response to intimate partner violence. Am J Community Psychol 2011;48(3-4):208–21. CrossRef PubMed
Publicado por en

No hay comentarios:

Publicar un comentario

Suscribirse a: Enviar comentarios (Atom)