Accelerating the Dissemination and Implementation of PCOR Findings Into Primary Care Practice (R18) | Agency for Healthcare Research & Quality (AHRQ)

Accelerating the Dissemination and Implementation of PCOR Findings Into Primary Care Practice (R18)

Technical Assistance Conference Call Transcript

Transcript of a technical assistance conference call related to Funding Opportunity Announcement (FOA), "Accelerating the Dissemination and Implementation of PCOR Findings Into Primary Care Practice (R18)".

Moderator: Bob McNellis
April 24, 2014, 2:00 p.m. ET

Please note: The FOA should be considered the source document for applications.

Coordinator: Welcome, and thank you for standing by. I'd like to remind the participants that their lines have been placed on a listen-only mode until today's question-and-answer session. Today's conference is being recorded. If anyone has objections, please disconnect at this time. I will turn the conference call now to your speaker for today, Bob McNellis. Sir, you may begin.

Bob McNellis: Thank you. Good afternoon, everyone, or good morning to those of you joining from the west coast. My name is Bob McNellis. I am the senior advisor for primary care at AHRQ. It's my pleasure to welcome you to today's technical assistance call for RFA-HS-14-008, "Accelerating the Dissemination and Implementation PCOR in Primary Care Practice."

We're really excited about this FOA and looking forward to telling you more about it. Let's go over what we're going to cover today.

Here's what we have planned. For the next 75 minutes we're going to focus on this R18 FOA. Many of you may be interested in a followup call that we have starting at 3:30 p.m. eastern time. We'll talk about the companion, RO1 evaluation FOA.

First, what we'll do is we'll start with introductions. I'll then provide some brief background of what is behind the scenes of the creation of this FOA. We'll review some of the highlights of it; share a few of the frequently asked questions that we've received from you. And then we'll open it up for more questions and answers.

Also, I want to make sure you know that we're going to post the transcript from this call on the AHRQ Web site in 2 weeks. What I'd like to do now is turn to the AHRQ team in Rockville who is working on this initiative and supporting today's call. First I'm going to turn it over to Dr. David Meyers to start the introductions from Rockville.

David Meyers: Thanks so much, Bob. I'm joined here in Rockville by a large team of people who are behind Bob in making this FOA happen. I'm David Meyers. I direct the Center for Primary Care, and we're going to go around the room here and ask folks to introduce themselves.

Tess Miller: Hi, I'm Tess Miller. I'm a member of the Center for Primary Care.

Debbie Rothstein: Debbie Rothstein. I'm a member of the Office of Extramural Research and Priority Populations at AHRQ.

Rebecca Roper: Rebecca Roper. I'm a member of the Center for Primary Care.

Janice Genevro: This is Janice Genevro. I'm a member of the Center for Primary Care.

Phillip Jordan: Hello. Phillip Jordan. I'm a member of the Center for Primary Care.

Bill Borden: Hi. Bill Borden. I'm a member of the Center for Outcomes and Evidence.

Kishena Wadhwani: Kish Wadhwani. I'm a member of the Office of Extramural Research and Priority Populations at AHRQ.

Galen Gregor: Hi, I'm Galen Gregor. I'm the grants management specialist, and I'm listed on the FOA.

Michelle Burr: I'm Michelle Burr, and I am with grants management as well.

Cecilia Casale: And I'm Cecilia Casale, and I'm with the Division of Priority Populations.

David Meyers: Take it away, Bob.

Bob McNellis: Great, thank you all. And as you can hear, we have a fantastic team behind this FOA. We certainly look forward to working with all of you.

I'm going to provide a brief background of what's behind this FOA. This FOA is at the nexus of AHRQ's mission, priorities, research interests, and previous investments. Like you, AHRQ recognizes that revitalizing our Nation's primary care system is foundational to achieving high-quality, accessible, and efficient health care for all Americans.

Over the past several years, AHRQ has invested in research to understand the process of primary care redesign and transformation, and through its work on the patient-centered medical home has developed resources to help clinicians, researchers, decision makers, and policymakers to better evaluate methods for improving primary care practice and tools to help facilitate it.

Much of primary care is still delivered in small- and medium-sized practices scattered throughout the country. Often these practices do not have access to quality improvement infrastructure. AHRQ is particularly interested in providing support for quality improvement to these practices.

You will have noticed that the evaluation of the clinical effectiveness of the intervention is focused on performance measures for heart health. AHRQ selected heart health for a number of reasons. First, heart disease is the leading cause of death in the United States. Second, there is patient-centered outcomes research with evidence on effectively preventing heart disease and managing those risk factors. We know what to do; the challenge is to help get it done in practice. Finally, this initiative intersects with other efforts being undertaken at the Federal level, including the Million Hearts campaign, CDC's Winnable Battles, and other public health interventions that address heart disease throughout the country.

Lastly, as noted in the FOA, AHRQ has been a leader in fostering comparative effectiveness research with a focus on patient-centered outcomes for many, many years. With the establishment of the Patient-Centered Outcomes Research Trust Fund, Congress charged AHRQ with assuring dissemination and implementation of Patient-Centered Outcomes Research findings.

In summary, this FOA aligns AHRQ's primary care quality improvement research with efforts to address a leading cause of death and disease by accelerating the uptake of PCOR findings in practice. With that as a brief background, let's go ahead.

The FOA outlines its purpose very well. But I'll briefly review this. The initiative will fund cooperatives to provide quality improvement support to small- and medium-sized primary care practices within a defined geographic service area. We'll talk a little bit more about what that means with the goal of disseminating and implementing patient-centered outcomes research, both clinical and organizational findings, into primary care practice to improve healthcare quality with an initial focus on cardiovascular care. Also, the intent is to build primary care practices' capacity to receive and incorporate other PCOR clinical and organizational findings in the future.

AHRQ is interested in understanding if and how externally provided quality improvement support can accelerate the dissemination and implementation of PCOR findings in primary care practice to guide future initiatives.

There are several important terms that have been defined in the FOA. Many of the questions that you've asked over the last couple of weeks have been around these terms. Let's review the definitions of these terms so they are very clear.

The FOA defines "service area." What we expect is that applicants must define a discrete and contiguous geographical region for their work. It is expected, although not required, that the area will include a population of approximately 2 to 5 million people, and 500 to 1,000 or more primary care practices.

AHRQ encourages applications to propose regions that are made up of whole States, combinations of whole contiguous States, or local contiguous portions of large States. This is important. We'll talk more about it later.

The FOA also defines what we mean by "cooperative." While applications must be submitted by a single eligible organization, AHRQ recognizes the scope and breadth of this project is likely best carried out through partnerships with multiple entities. Any eligible organization willing and able to be responsible for the conduct of this project may submit the application on behalf of the cooperative. We expect the cooperatives to include a wide range of different types of organizations, and we can talk more about those later.

We strongly encourage applications that bring together the skills, experience, and resources of local organizations, such as local, State, or regional primary care organizations dedicated to improving care. Public health agencies and community-based organizations, private or public payors in the region, as well as consumer, patient advisory, and/or advocacy groups—all will be important partners in the cooperative.

We define "primary care" in the FOA. And you'll recognize this from the 1996 IOM definition. AHRQ defines primary care as the provision of integrated, accessible health care services by clinicians who are accountable for addressing a large majority of personal health care needs, including prevention and health promotion, developing a sustained partnership with patients, and practicing in the context of family and community.

We define what we mean by "primary care practice." Primary care practices, from our perspective, are health care organizations that are dedicated to the provision of primary care as we defined it, utilizing lead clinicians. "Lead clinicians" is the term that we used to describe clinicians such as family medicine physicians, general internal medicine physicians, general practice physicians, general pediatric physicians, geriatric physicians, and nurse practitioners and physician assistants who work in primary care settings.

We define what we mean by "small- to medium-sized primary care practices." This is important because this is really where our focus is, as I mentioned in the introduction. The intent of this initiative is to provide dissemination and implementation support and capacity building to small and medium primary care practices that typically do not have internal resources for quality improvement and change management.

For the purpose of this initiative, AHRQ defines small- and medium-sized practices as organizations or offices that consist of 10 or fewer lead clinicians. That's based on the definition I just gave you—the physicians, physician assistants, and nurse practitioners who work in primary care.

Community health centers with 10 or fewer lead clinicians may be included. Multi-specialty practices that provide primary care which have 10 or fewer total lead clinicians may also be included. We'll talk more about this later as well.

Finally, I want to describe and define what we mean by "Patient-Centered Outcomes Research." Patient-Centered Outcomes Research (PCOR) is defined as comparative clinical effectiveness research on the impact of health outcomes of two or more preventative diagnostic treatment or health care delivery approaches. PCOR produces not only clinical findings but also evidence about the effectiveness of how health care is delivered, referred to in this FOA as organizational practices.

There are many sources of PCOR findings, including the three listed here which include AHRQ's Effective Health Care Program, which has dozens of Comparative Effectiveness Reports; Patient Centered Outcomes Research Institute (PCORI) is beginning many of their research initiatives—may have findings available now or in the very near future; and the U.S. Preventive Services Task Force is another source of PCOR findings.

Next I'll talk a little bit about guidance for applicants. This section in the FOA begins with what applicants should do. I've hit the highlights here—important things that we think you should know and to make sure we help you understand.

Applicants should first, of course, convene an experienced team. That team should definitely have some experience in doing this kind of work previously. Second, it's important to define a contiguous geographic region and develop a plan for recruiting and working with at least 250 primary care practices. Next, develop a process and criteria for identifying PCOR findings and determine what findings to disseminate to primary care practices.

Applicants should define a comprehensive, multi-component, evidenced-based approach to provide quality improvement support to small- and medium-sized primary care practices. AHRQ strongly encourages applications to use practice facilitation as a central and unifying strategy within the comprehensive approach.

Let me highlight a variety of other things applicants should do that includes planning for a rigorous evaluation. Part of that evaluation is going to be making sure that you have a method to control for secular trends. This will influence the effect of the intervention on the practices. We'll talk more about that later.

Assess practice level performance of the ABCS measures. These should be measured at the baseline prior to beginning work with the practice, then every three months during the intervention and for at least 6 months following the intervention. We also would like you to measure the practice's capacity for quality improvement. This should be done at baseline, at the end of the intervention, and 6 months post-intervention.

Applicants should measure implementation of the approach to QI support at the aggregate and the practice level across all the practices you're working with as well as individual practices. In addition, collect and analyze internal and external contextual data. Many things are going on in the environment. It's important to identify what those are and be able to determine what type of effect they've had on your practices and the interventions that you've done with them.

Your methods should be rigorous. Mixed methods are encouraged. Also, we suggest you consider examining the impact on health disparities and health outcomes. Although not required, this is the ultimate goal of these interventions that is to improve health and decrease health disparities.

In our guidance for applicants, we say applicants should describe a data collection and analysis plan for all aspects of evaluation. As part of that plan, it's important that applicants be prepared to participate in the separate overarching evaluation, the RO1 evaluation that we'll discuss on a later call.

And as part of that participation in the evaluation, appropriate staff need to be identified in the application, including at least 20 percent FTE of a senior evaluator and 40 percent FTE of a research assistant. It is useful for everyone who's interested in applying to this RFA to review the evaluation, RFA-HS-14-009, to understand what some of the potential roles those staff will be playing in terms of the overarching evaluation.

Applicants should plan to complete work within 36 months of the start date. They should also plan to start work with practices within 6 months of award. We recognize that it'll take a little bit of time for you to identify practices. Much of that time should be spent now identifying as many as you can. It'll take a little time to get things geared up, but we expect within 6 months of award, applicants will begin work with practices.

We want to make sure that you are providing support to 250 practices. You should have provided that support to all 250 practices within 30 months of

award; 24 months is encouraged. This is a relatively short timeframe for the work. It's important to emphasize this—we want the work to be done as quickly as possible.

The timeframe is somewhere between rapid cycle evaluation time and longer research cycles that you may be familiar with. As you can see, we're really interested in moving this work very quickly.

You should plan and budget for three team members to travel to Washington, DC, once a year for each year of the grant to participate in an annual grantee meeting. The first meeting we anticipate will be held in spring 2015, shortly after the awards are made. There were some typos in FOA and it showed up as fall 2014 in at least one place. But just to be clear, it's not going to be until spring 2015, after the awards are made, when we'll have that first grantee meeting.

Now, moving on, let's talk a little bit about some of the basics around the FOA. AHRQ is utilizing an R18 mechanism as we've described. We anticipate making up to eight awards. Grants are limited to $5 million in total cost per year. The project may not exceed 3 years.

The FOA describes the organizations that are eligible to apply for this award. Grants are made to organizations, not individuals. Eligible organizations that may submit and lead applications include these: public, nonprofit private institutions, units of local or State government, eligible Agencies of the Federal Government, and Indian-Native American tribal governments and designated organizations. For-profit organizations and foreign institutions are not eligible to lead these applications.

The FOA also talks about the program director/principal investigator (PD/PI). Any individual with the knowledge, skills, and experience required to carry out the proposed research is eligible to serve as the project's program director/principal investigator. There are no degree requirements.

The PD/PI must be accountable to the organization submitting the application. AHRQ requires that there be one—and only one—program director/principal investigator identified on the application. No individual may be listed as the PD/PI on applications to this FOA (HS-14-008) and the evaluation FOA (HS-14-009). An individual cannot serve as the principal investigator/program director on applications for both of these FOAs.

The research strategy section of the application itself is limited to 35 pages. This is a little longer than many typical research strategies. We recognize this is a large project. People will need a little extra room to talk about it.

This section itself needs to contain the following sections: talk about the cooperative and the region which the initiative will serve; the approach to disseminating and implementing PCOR evidence through your quality improvement support intervention; what the evaluation plan is; and the project timeline; and the dissemination.

And if you look at those sections, you'll notice that if you add up all the numbers related to the individual sections it comes up to 40 pages. We only want an application with a Research Strategy section that is a maximum of 35 pages long. So, if you go the maximum possible length in one section, you'll need to cut back on others. Thirty-five pages is going to be the maximum, although we've given you a little latitude as to how you divide up those 35 pages within the sections.

We emphasize the FOA is extremely detailed. A lot of work went into creating this. Make sure you read the entire FOA completely and carefully.

Briefly about the budget: AHRQ does not accept modular budgets. We only use the detailed Research and Related Budget. The budget ceiling is for total cost, which consists of direct and indirect cost. Matching funds are welcome and encouraged, but they are certainly not required.

Next, we're going to briefly talk about the review criteria to make sure you have an idea of how the review process will go and what reviewers will be looking for. The review criteria provides an outline of what AHRQ is seeking and questions peer reviewers will be asked to consider.

The significance of the application will be based upon several criteria. First is its ability to disseminate PCOR findings to primary care practices. Next, provide support for implementation in practices. Third, build the capacity of practices to implement new PCOR findings. And lastly, to understand how external quality improvement support accelerates dissemination of PCOR findings into primary care practice.

Pay special attention to the research approach. The reviewers will be considering three areas—the approach to PCOR dissemination and implementation through quality improvement support, your approach to the evaluation, and your approach to dissemination.

AHRQ will consider the following selection criteria when making award decisions. First, of course, is the scientific and technical merit of the proposed project as determined by peer review. Next is the availability of funds. Third is the responsiveness of the application to the goals and objectives of the FOA.

Fourth, this is an important one, is the relevance and fit within AHRQ's portfolio priorities, as well as overall programmatic balance including balance across funded studies to achieve geographic diversity, adequate inclusion of safety net providers, AHRQ priority populations, and the uniformity of evaluation. Finally, the independence of grantee teams under this FOA and the related evaluation FOA is going to be critical.

Important dates for this FOA: Letters of intent are due on May 23. Of course, letters of intent are not required. We'll talk about that more in a moment. The earliest submission date for this application will be June 3. The latest submission date will be July 3. All applications must be received by July 3.

We anticipate that the peer review process will happen in September and/or October of this year. The anticipated grant start date is going to be for February 2015.

Let me provide a brief clarification about letters of intent. They are highly encouraged but not binding and not required. It should include some very specific information, including the number and title of this funding opportunity; a descriptive title of what your proposed activity is; the name, address, and telephone number of the PD/PI who is proposed; and names and institutions of other key personnel and participating institutions should be included.

These non-binding letters are extremely helpful to the AHRQ staff and assist us in ensuring that your application gets the highest quality review. We strongly encourage anyone who is considering submitting an application to submit a letter of intent. You can send that letter electronically to Phillip Jordan. His email is in the FOA.

For additional help, we have a variety of folks who you might reach out to: For content in the FOA itself, Mr. Phillip Jordan. For the peer review process, Mr. Kishena Wadhwani. And for financial matters, Ms. Galen Gregor is the contact person.

That completes our review of the FOA and some of the things that we think are important for applicants to know. We have received a variety of questions from you and so I put together just a few, six or seven, frequently asked questions here that we're going to go ahead and raise and answer, and see if that covers some of the questions you have. Once we've gone through those, we'll have the operator begin to open up the lines and take some questions from you that haven't been answered.

The first frequently asked question is—may applicants propose to serve small- and medium-sized primary care offices of larger organizations, including outpatient primary care offices of hospital systems or academic health centers, larger community health center networks, or multi-office groups? We've heard this question quite a few times.

The answer is this: It's important. If an applicant is interested in serving practices within those larger systems, it is going to be important for the application to demonstrate, to justify, and to discuss how the practices do not receive significant quality improvement support.

Again, just to reiterate, our focus is on small and medium-sized practices that don't have the resources internally for that support. And, if within the context of a larger health care system, those practices that don't have support, it needs to be carefully explained why that is.

Can the applicant work with larger primary care practices that have greater than 10 lead clinicians? Well, certainly as we've described, this is not the intended focus of the FOA. If the applicant proposes larger organizations with more lead clinicians as a target for their work, they need to justify those types of inclusions.

We've seen some corollary questions about this, such as how we define 10 clinicians—as FTEs, as individuals, as NPI numbers, etc. We can discuss that further if needed, but in general, we think about it as individuals or FTEs. You have to make the case for the larger number being something that is worth working with.

We're interested in smaller practices that have smaller than 10 lead clinicians. If they have 15 folks who are all part-time, you just have to talk about that and about why that group is important to work with.

May an applicant organization submit applications to both this FOA and the related FOA (HS-14-009) on the evaluation component? Yes, that is possible. Applicant organizations may submit applications to both FOAs. But, as we mentioned earlier, an individual may not be the principal investigator or program director on applications submitted for both FOAs. They need to be independent.

We've also heard questions about whether Accountable Care Organizations can apply to this FOA. Yes, and typically fewer than two conditions. First, they must serve or partner with organizations that serve at least 250 primary care practices. Second, they must be a nonprofit institution.

Can applicants recruit primary care practices that are scattered across multiple regions of the country? The answer is no. Our intent here is that practices must be from a discrete, contiguous geographic area. You should be able to draw lines between practices in that contiguous area and not leave it. It needs to be very well defined. We're looking for regional approaches to practice improvement.

Can an applicant recruit practices which have not yet adopted electronic health records? Yes. We talk a little about electronic health records in the FOA. In general, this work will be easier through the work of EHR-enabled practices. What we've set up is a criterion that at least 60 percent of the practices you work with must be EHR enabled.

If you intend to work with fewer than that number, a justification must be provided. We recognize fully that some of the practices that may need this type of support the most, those that are under-resourced and underserved—addressing the needs of underserved populations—may not have EHRs in place. We want to make sure that there are opportunities to provide support to those practices. But we'd like to have some justification of how you'll work within that kind of system when you have a larger number of practices that do not have EHRs.

Our last question that we'll offer up is this: can applicants propose clinical measures other than the six heart health measures listed in the FOA? As we've described, applicants are required to collect those six measures around heart health as listed in the FOA. Applicants may propose additional types of measures. But, at a minimum those six measures need to be done.

We're also very much aware that the science is still emerging through some recent publications. We know that the measures around blood pressure and lipid management, for example, are evolving. It'll be useful for the applicants to be mindful of this as they look at those measures.

So there you go. There are a handful of FAQs, a brief description of the FOA and AHRQ's interest in putting it out there. What I'd like to do now, in the approximately 45 minute we have left, is have the operator open up your lines to some questions for us. And then between myself and our folks in Rockville, we'll see if we can come up with some answers.

I do need to say one thing. Some of the questions may be so specific that we might need some time to think on them and we might have to follow up with you on those. So be prepared for that. If we're not able to answer your question today, we'll certainly get to it in the coming days as we get you a very specific answer.

Bob McNellis: Before we go to questions, I'd like to give an opportunity to other AHRQ staff if they have any comments or things that they want to reinforce or chime in from the presentation. Let me just give an opportunity for my colleagues to jump in as well before we take some questions.

David Meyers: Thanks, Bob. This is David Meyers here at AHRQ. And we did want to add one clarification. One FAQ was may an applicant organization submit applications to this FOA and the related FOA HHS-14-009. And we completely agree that the answer is yes.

But we wanted to point out another thing. In addition to the PD/PI not being able to be the same, we recommend that folks take a look at the selection criteria that AHRQ will use for picking awardees for the dissemination and implementation grantees and the evaluation. And if you remember, one of those were that we are committed to ensuring that the evaluation is independent from the work being done in each of the implementation groups.

So in preparing applications, organizations need to carefully consider how they would maintain independence of the implementation and the evaluation team, and how they can demonstrate that both to the peer reviewers and AHRQ. If the PI on one grant is not the PI but is a main person on another grant, it makes it hard to understand how they wouldn't be potentially biased, or make it look as if they were biasing the overall evaluation because the evaluation is looking at how successful they were in their own work.

On the other hand, a group—a large institution, a University of California system who might have a department of family medicine who wanted to work with a region of California in the implementation, and a public health school that wanted to do the evaluation, and there was no staff overlap from that—could easily, I think, make a case for applying for both FOAs because they could show that while they were of the same institution, they were very different teams that were independent from each other.

Somebody who wants to bring those teams very close together has to think very, very carefully about this selection criterion and demonstrate that they're independent. And if anybody has questions about that, I'd be happy to try to answer them now or you're welcome to contact us afterwards.

Is there anybody else in the room that had something they wanted to jump in with? Okay, Bob, we are ready to turn it over to our participants. Thanks.

Coordinator: QUESTION/ANSWER PORTION OF CALL BEGINS.

Question: Hi. I wanted to confirm the 35-page limitation is single-spaced, not double-spaced.

Answer: It's single-spaced.

Additional AHRQ Response: Do remember that your colleagues, your peers will be reading this. And, if you make it so tiny that they don't want to, that they get bored, or can't easily read it, that could be a problem for you. But it definitely is expecting a single-spaced application.

Question: Hi. Thanks for the great presentation. My question is if the requirement is for 250 practices, but if you are proposing a randomized controlled trial, does that mean you have to have 500 processes or an active control? Are you expecting 250 practices to be in an intervention arm? Can you clarify that?

Answer: Sure can. I'll take the first stab at it, and then I'll turn to my colleagues. The goal is to have the intervention in 250 practices. We don't necessarily say it needs to be a randomized trial. And, even if you wanted to do that, you don't necessarily have to have equal numbers in the control group. And there are other ways that you might be able to control for the intervention.

So I think the main goal is to have 250 practices for intervention and some method of control. I think that is our main goal. Other thoughts from our colleagues?

Additional AHRQ Response: No, I think that's perfect, Bob. Although, we do in the FOA talk about methods other than randomized control trials. Some methods may allow practices to serve partially as their own control. Two hundred and fifty, however, must receive the full intervention and have a minimum of 6 months followup by the end.

Question: I guess the question we have is we were interested in actually comparing two different levels of facilitation. And so we're thinking that both arms are somewhat active, but different. I mean, just different. So an active control, not even a control, but basically the comparing two different types of QI interventions among 250 sites was one of the ways we were thinking about doing this. Would that be responsive?

Answer: I think if you had no control, if everybody got something—you'd have to think about whether you're being fully responsive. But a very small secular control group and then two larger interventions of two different things that had more than 250 practices would seem like it could be responsive.

Question: Yes, hi. Thank you for the presentation. I actually have a two-pronged question. Can one individual be included on multiple R-18 applications?

Answer: So I'll start. My thinking is that I think they can be included on multiple applications. Other thoughts from the home office?

Additional AHRQ Response: Everybody here was nodding their heads in agreement, Bob. You can be listed on multiple applications. Of course, the person needs to be able to demonstrate that if they received them, they wouldn't be going over 100 percent effort.

Question: And what about—can individuals from a single institution be included then on the multiple R-18 applications?

Answer: Yes.

Question: In Oklahoma, we have a number of Federal initiatives, including the comprehensive primary care initiative and some ACOs taking place. Would any of those preclude or would any of those—the practices participating in those—be precluded from participating in this program … sites?

AHRQ Response: I don't think so. But actually, David, I'm going to turn that one to you.

Answer: Thanks Bob. And, I agree with you on that as well. So absolutely. We expect that applications will be done in areas with varied and changing landscapes, that there are many kinds of initiatives. And there is no exclusion automatically from participating in this AHRQ grant.

However, the grant does talk quite a bit about the need for applicants to describe the environment, the external context in which their intervention will be performed. So being able to explain how an ACO in a region or another Federal initiative may effect, may augment, how that information may be collected during the evaluation, is important.

We expect you to do that kind of deep thinking and explain that in your application. But there is no required exclusion, automatic exclusion due to participation in other programs. Going one step further though, I want to point out that it also says in the application that you shouldn't be working only with—and I believe the word is "cutting-edge"—practices or innovative practices.

So say that of my 250 practices, 150 of them have received millions of dollars of support in the last 5 years from this ACO and through this other demonstration project, and now we're going to go in and help them. This would not be the spirit of this FOA.

On the other hand, if you say they've been participating in this thing to improve asthma care and got a little bit of help in a collaborative with HRSA 3 years ago, that would be a very different, I think, level of support.

Additional AHRQ Response: One other thing—I think it would be very important to include a description of how inclusion in other initiatives in the control group and the intervention group would affect the interpretation of the evaluation finding.

Additional AHRQ Response: Yes, a good question. We talk about exemplars and early adopters, even, in the FOA. So I think that's an important consideration. Thank you, David.

Question: I have a question about the service region or how you define the regions. It's a point of further clarification. If two States share a border and in one State the intervention would be statewide, but in the neighboring State the intervention would be—could the intervention be in specific urban areas within the State, but not covering the entire State. Would you consider that still contiguous?

Answer: I think so. So, if I understand what you're saying—and we'll get some other opinions as well—if you've covered a complete state and some of the border areas in an adjoining State, especially around the urban areas, I think that's—if you can define a discreet, contiguous area, if you go from covering the eastern part of the State that borders the State of where you're covering the most, and you're jumping all the way over to the western part of the State, I think you've lost some of the contiguousness.

Unless you're making a case that you're serving all of the practices in between. So can you help clarify a little bit more? Do you see the continuity there between the entire State that you're covering and the region of the other State?

Response: Right, yes.

Additional AHRQ Response: It doesn't have to be all of both States, I don't think. That would be my take on it. So other opinions on that one?

Additional AHRQ Response: Yes, I think most folks here were nodding with you, Bob. For folks who—just to make it specific—if somebody was doing the up-country of South Carolina and then included Charlotte, North Carolina, which is, you know, the big city in that region, but it's in a different State but geographically, it is part of the same area. Yes, that is totally, you know, within contiguousness.

On the other hand, if they wanted to do upstate South Carolina and they wanted to include Jacksonville, Florida, or Atlanta, you know, there's 100 miles that they're just skipping over that's actually very similar to South Carolina but they want a different population, that would not do it.

Question: So can I give you a specific example of, let's say, Pennsylvania and the city of Trenton, and the city of Newark.

AHRQ Response: My geography is not great. But my memory is they're both on the border with Pennsylvania. And so yes…

Response: Trenton is, but Newark is not.

AHRQ Response: Right, I'm sorry. So that everybody in the room who is better at geography than I am said yes, for Trenton, that is. But no, Newark, you couldn't do unless you were doing all of New Jersey, some larger part of New Jersey.

Response: Yes, but Trenton and Camden, which Camden touches Pennsylvania, would be?

Answer: Right. So that would—does that make—because really, for everybody on the phone, so let's generalize a bit. The concept of contiguous areas here, and the size of the areas, why we were talking about this is then you have enough saturation in a community. Community can be very large. That you're really changing things.

If you—if somebody tries to say, "Oh, we're contiguous because we have everything from Pennsylvania to Oklahoma, and therefore we can pick out all these different towns that we like"—no. Yes, it's contiguous, but you're not thinking about that as one region. Somebody else just says we're doing Wyoming, Nevada, and Colorado—I hope those are contiguous—but some big States, and it's a big area. But the reason you're doing it is because really that's how you get 250 primary care practices. That's fine too.

Somebody comes in and says we're doing Brooklyn because we've got 500 primary care practices, small primary care practices in the borough of Brooklyn, that's fine because we are trying to saturate this region. But we also want you to consider having a large enough region that you can work with at least 250 practices and some controls and still have some left over, because we know that not everybody is going to join right away.

So that's where you get to a region that had 1,000 might give you practices for control and primary practices that you approach, but for whatever reason choose not to join eventually. But that you're still at a high enough saturation that you really have the potential to change the whole region, to serve it.

And if we gave you another few years, you know, you might eventually get all of them. So somebody comes in with the region with 25 million people and 3,000 practices, and says we're going to work with 250, they're not meeting the spirit.

Somebody else who comes in and says, "We've got this big geographic area where there are only 300 practices and we can still do it. We're going to work with 250 of them over these next 3 months. And we'll have a few for control." That would meet the spirit.

Question: Hi, I had questions about attachments. Whether letters of support or other attachments could be included? And then I have a second question after that.

Answer: Yes. I think we actually encourage it. Especially when you're looking at defining the cooperative and who the partners are, I think letters of support will be really important. Letters of support from practices, if you have them, that's great. But we also recognize that you may not be able to pull 250 practices to write letters of support and in time for the application. That's highly unlikely.

But I think from the cooperative, I think that's going to be an important thing to have those kinds of letters of support. Other thoughts from AHRQ?

Additional AHRQ Response: One thing I would recommend, and that is make sure you submit the support letter with the application. You submit at the same time. You cannot submit any materials within 45 days before the review meeting. So please be aware of that.

Question: Is there any page limit on those support letters? And then my second question relates to the SF-424, and which version? Because there's a variety of versions out there. Even under the R&R designation there's lettered versions. So I just wanted to make sure we use the appropriate version.

AHRQ Response: So if you go into the FOA and you click where it says "Apply for Grant Electronically" that should be attached to the version that would be appropriate to use.

Response: Okay, I'll double check. But I didn't think it was. Okay.

Answer: We will also double check that. But officially the version you're supposed to use is linked to the FOA.

AHRQ Response: And one other comment is the FOA contains specific instructions that may be different from what is in the general instructions in the SF-424. The FOA supersedes the general instructions. But if they're not specific instructions, there are lots of them in the SF-424.

Answer: There are specific instructions that you have to follow. And in terms of the page limit, well technically, there are no page limits on things in the appendix. The application has to stand on its own without the appendix. So using the appendix to get around the page limits is a really bad idea, in fact, it's not allowed.

Response: Yes, I'm just thinking about getting 250 letters from practitioners.

Answer: Yes, but if you need a lot of them for the people—if you're able to—one of the reasons we don't expect, and we didn't ask, there are other applications where we would expect letters of interest or support from the practices. You're not going to hear about this until next February. You probably won't touch these practices until July or sometime in the spring of 2015.

Getting them to say we're going to do something a year from now is like getting my 15-year-old to tell me he'll clean his room. Yes, maybe. But our goal for the letters, the thing we hope you will spend more time on is who your partners are. How are you going to reach out to the practices? Show us your capability. Because you're going to be doing this for years. We want to see your capacity.

Now, if you're in a good place and you have a relationship and you can get some, and you think that strengthens your application, that's absolutely allowed. Trying to count and say, "Oh, my gosh, we only have 240," that doesn't matter. It's your plan about how you're going to reach 250, which generally is going to say we'll reach out to more than that. And here's why we expect X percent to say yes, you know. That would be the way we expected people to approach this.

Question: Hi, guys. We're still struggling with demonstrating the practices don't have quality improvement support. In the FOA, the language is that they do not have internal resources for quality improvement. And you've talked about practices do not receive significant quality improvement support, which implies external support. And how do we demonstrate that?

And let me just give you an example. A lot of FQHCs do receive quality improvement support from their State primary care association from HRSA and have documented requirements around doing QI within the FQHC, which in my mind would preclude them from being involved in the project. But I wondered if you could just maybe expand a little bit on this issue around limiting the sites to practices around either not having internal QI support or not receiving external QI support.

AHRQ Response: You're talking about large practices, right? Practices that have more than 10 clinicians?

Response: No. We're talking about small...

Answer: Thanks for the question. I'm going to ask you to take a half-step back and think more about the application. And so if your application says we are going to be reaching out to 15 FQHCs, across the State of Washington, then some explanation about what is the level of support that the PCA of the State of Washington gives and why you believe giving them more is needed to help them succeed would do it.

You don't have to give us a thoroughly documented case, "Clinic A received 2.7 hours of a practice facilitator in 2014, and 4 hours in 2014." It's not that level. It's explaining in general why these practices and why in general the level of support that they're currently getting really could use more support.

On the other hand, if you want to tell me that you're going to the University of Washington's outpatient clinic, then I need to hear about, well, what does the University of Washington provide? And if they've got a staff of 22 quality improvement people, three of which are assigned to work weekly with the practice, they're part of a large system getting regular support. You would really need to explain why this extra support from this AHRQ grant will make a difference.

It is not a clear bar here. It's not a clear thing. We don't expect that you will go in and report on all 250 small and independent practices exactly. Or how much QI support they have internally, how many times they have meetings. If they're small, especially really small, we're just accepting the fact that they could use some external support. It's for the larger ones, including FQHCs, as you pointed out. Even a moderate-sized FQHC, six clinicians, might get some built in support from the state PCA, or the QIO. This is true for everybody around the country.

Understanding what kind of support the practices you're going to be reaching out to currently receive through the QIO program, which will soon be the QIN program, would strengthen your application. And if you're able to say in general they haven't been receiving a lot of support or practice level improvement or using evidence the way we're going to help them, that's great.

But if you're in a State with a dynamo QIN, or better yet, you are a great QIO, you're going to have to explain what's different and why practices are getting something really special based on this project. I hope that's helpful.

Question: Thank you for the presentation, by the way. It was very, very good. My first question is for the south Texas region, we have two large metropolitan cities that service quite a large rural population. And most of those have very limited primary care physicians in those areas. So they're serviced more so in the metropolitan areas, if you will.

So in defining our region, would we not define it according to the counties or more so in the region encompassing the rural areas that utilize the metropolitan areas for service and care?

Answer: So very quickly on that, the region is based on the physician location region. Now, you're obviously going to talk about the population then of that region, but you've done a very nice job already describing it. Describe this region, the docs and the nurses you'll be working with, and how they form the community and who do they serve. That's what works. And that's what should be contiguous.

And if they go out more to the right and the south because of where the populations come from, you don't need to keep going and expanding your region to the area where the people come from. Unless you plan to reach out to practices in those rural areas.

Question: We have a question about the evaluator support and wondering if it can be split between a qualitative and a quantitative researcher? Or if it should just be a single evaluator, lead evaluator?

Answer: Bob, let me take this first. The 20 percent time of the senior and the 40 percent time of the research assistant are meant to build this into your application since you don't exactly know what you're going to be doing right now. But you know that the evaluation will be asking for things. The one part you do know is turning in your data on an every 3-month basis, giving lots of data to AHRQ and the central evaluator. And so that time can be put into these people's time. But there will be other things. Other requests that come in. And if you haven't built in people's time to help answer them, the project is going to fail. So we're requiring you to do that.

If you think the best way to do that is to put in 10 percent time in your qualitative and 10 percent time of the senior quantitative person, plus the 40 percent research time, that's fine. But it's not that these are the people doing your part of the evaluation. And we're thrilled to hear people are thinking we'd like to have a mixed-method evaluation with experts in both sides. Those people's time for your internal evaluation is separate from the required time to participate and coordinate with the AHRQ evaluation.

Question: I think my question's been answered. But I'll reiterate it here. Our research approach will be to compare methods of practice support appropriate for this FOA. And from the earlier response it sounds like that would be appropriate, but you still have to have some other non-intervention comparison group. Is that correct?

Answer: Yes.

Question: I gather that the lead physician includes only primary care PAs and NPs. But if the practice has specialists, then that is not counted in the maximum of 10? Is that correct?

Answer: Great question. We said the practice size is defined by the number of lead clinicians, period. So if you had a group with four cardiologists and three primary care doctors and two NPs and a PA, they're in because they're still under 10 even though they're multispecialty. But if you had nine primary care folks, and nine non-primary care folks in multispecialty group, that we would not consider small to medium.

Because in general, our thinking is that if a practice with 18 lead clinicians has the infrastructure to start doing quality improvement, standardization, and support services, then make it different than the small group. If there are a couple of large practices supported over the 3 years of the grant, nobody's going to come out, by the way, and try to count at the end. We expect most of you will go way past 250. And if there's some on the border that made sense for you at the time to provide support for, great. But in your planning, if you're saying we're reaching out to the multispecialty groups of south Texas, they're all got 20 or so lead clinicians, and these would be a large percentage of the practices you were planning on working with, you would need to explain why you were working with larger practices than the FOA call for.

Question: Hi. Thank you very much for the call. It's been great. I have a pragmatic question which is: what's the study section for this? Who's going to be reviewing this?

Answer: There will be a special emphasis panel that will be set up to review the set of applications.

Question: And then I guess our second question is the expectation of the intervention that we are using facilitation to intervene in full for all four areas simultaneously? In other words, is the intervention only responsive if we deal with A, B, C, and S? Or could we choose to focus on A and C?

Answer: Glad you asked this because it is very, very important. It's all. And not only is it all, it's really more than all. It's about changing the way practices are able to incorporate new evidence. And in doing that, you're going to start with the evidence around ABCS. And you're going to measure around ABCS and help them. But it's more than just helping people improve blood pressure. Or even improve aspirin, blood pressure, cholesterol, and smoking.

It's about building their improvement capacity, their ability to incorporate new evidence. And using the ABCS whole package as a way of building capacity. So it's a two-pronged approach. All of the ABCS and capacity.

Question: Are you expecting that the intervention will use AHRQ published and recommended ABCS interventions?

Answer: Fine question. No. We're very proud that AHRQ has done many of the efforts and has put forth many PCOR findings. But it's up to the applicants to describe in the whole world of PCOR findings—from PubMed, from PCORI, from other sources, from AHRQ sources—how you will prioritize what specific pieces of evidence you're going to be helping practices implement.

And that's actually called out. It's not just saying you'll do it. You have to show us how you'll think about where you find PCOR. How do you prioritize and evolve that over time?

Question: Hi. I had a question. It might have been answered before. But I wanted to know about some specific examples of quality improvement support. You mentioned some work with QIN and QIOs, but I wanted some specific examples that would exclude a practice from participating.

AHRQ Response: We're thinking about this. It wouldn't be exclusion, per se. What we're looking for is to identify what the other quality improvement opportunities practices may have had. Particularly around large practices, I think, is where those need to be defined carefully.

All practices need to define what kind of contextual things are going on, that external environment where they may have had support. Other thoughts from the (AHRQ) team?

Answer: Well, as I heard it, Bob, there are potentially two different ways to interpret the question. And the first was about, well, what is a comprehensive quality improvement support? And that's what we're asking the applicant to describe what they're going to deliver.

And if you're looking for examples of different ways of doing that, the FOA includes some direct links and references to AHRQ guidance on those topics. And I'd recommend you look at that. But then at the end of your question it switched, and it seemed to be not what the applicant might put forward, but could you teach us more or say more about the description of what the environment is, and I think that's what you were saying, Bob.

Nothing would exclude a practice automatically if it's been done before. Except that if they've already shown they're the world leader in the ABCS. Helping them doesn't make a lot of sense.

But the kinds of things—programs that exist are also called out in the FOA. We did as you said, QINs do this. PCAs do this, demonstration projects, work from AHECs, work from private payors such as a Wellpoint or a Blue Cross State initiative. All of those kinds of things are contextual factors that are changing the landscape in which practices are deliving care.

Are you going to be able to detail perfectly everything that's happening in your region? Probably not. But if you do none of it and don't describe it at all, that would weaken an application significantly.

Question: Thank you. Do we need (IRB) approval for the research portion of the grant application?

Answer: Terrific question. But unfortunately I'm going to have to bat it back to you. If your application creates a human subject's protection issue, then yes, it would need to be reviewed by an IRB. And given that collecting data from practices—some people like to think that doctors and nurses are not people, not human, and therefore not needing a protection—you probably are going to collect some data that requires protection.

But it's really your application that says this. If somehow you were doing it all using some amazing secondary data source that they were already providing the data from, theoretically it's possible that you wouldn't. But it's up to you based on your application to define if it's necessary. And if so, then demonstrate where you're going to get that and how you're going to do that.

AHRQ Response: And I'd like to add to that too. And that is, please read the additional review criteria in the FOA. Describing in detail about how you go about protection of the human subject if the human subject involves IRB protocol.

Additional AHRQ Response: One other comment is that IRB approval is not required at the time of application submission or review.

Question: Okay, great. I've got two quick questions. One is nitpicky on count and practices again. So I have a rural health group with six sites and one tax ID. There are only three physicians at each office and they don't do quality improvement, and they don't really support each other in any way, systems-wise, except for an EHR and billing. Can I count them as six practices?

Answer: Yes. Six.

Question: And the other question involves the data systems. There is a comment in the FOA about agreeing to coordinate data elements, standardization, quality assurance, and share required data including the measures, capacity measures, et cetera. And I'm assuming that in creating the master file that the evaluator will bear those expenses. Do I need to have an official data sharing plan and budget for all that?

Answer: Very, very perceptive set of questions. And it's somewhere in-between. All eight awardees, assuming eight grants are given, will in that first meeting in the spring of 2015 be talking with the lead evaluation team and AHRQ about data protocols and standardization. Some of you may be asked to change your measures to better align with the group. Others may not. And together you will discuss how to move forward.

It is expected that the individual grantees under this FOA, once they have the basic structure, each quarter will create their data and move it. So absolutely the time, resources, or putting data in a way that isn't exactly maybe how you're going to use it, but that is necessary for the AHRQ evaluation is something you must plan on doing.

Some of that discussion, though, and the brain power and maybe creating a file that is easy for you all to put your data into will come from AHRQ and this evaluator. So it's a shared responsibility and you should be putting definitely 20 percent time of the senior person, 40 percent time of research assistant. And then if you think it's more, it's absolutely okay to budget some additional time.

Response: Yes, I'm thinking about data analysts.

Additional AHRQ Response: Yes. When we said research assistant, a data analyst, I think, is in that same category.

Question: So I run the California Right Care Initiative. And we have been working on heart attack and stroke prevention since 2008, both statewide in our large 38 million population. And also in three regional collaboratives in three major metro areas. What we see in the quality performance is that in the independent practice associations, we consistently have lower performance.

So although you've addressed the issue of contiguousness, my question to you is this—we see those practices that are organized around the concept of independent practice associations to fulfill the California unique structure of delegated managed care model. Those practices that are associated through that contracting mechanism, the small, independent practices, have a real problem with their performance relative to the integrated medical groups.

So, our key research question is to look at a statewide approach to support those small independent practices that are affiliated through the IPA model. And I'm wondering whether we can address that subset of the population in the whole State?

AHRQ Response: That's a great question. I think you sent that one to us in advance, and I know the team has been noodling on that one. So, thoughts from the AHRQ team?

Additional AHRQ Response: I think it's a great question that we should take offline. So, why don't we try to follow up with you to discuss it? It's not a clear-cut answer. Certain parts of it you did a great job of explaining why though they're in an association with each other, they're really small and independent. That speaks to the yes.

But the idea of saying we're going to take the entire 38 million population of California and call it one region, and then pick off practices. Even though they have a similarity, we're going to have to think through it a little bit more with you. The goal really was to change a whole community and so how do we define that.

The last thing I'd say on this issue more broadly is; we're giving you guidance. But you decide the application. And you're convincing your peers of the merit.

So we may say, you know, small is 10. And if your application is saying you got around 15 and here's our thinking and here's why, and you convince the folks, you can get funded. If you say, well, you said 30 pages, but we want you to do 40 because you were wrong, yeah, we'll win on that one.

But for a lot of these other things, it's up to you to make the case. We'll give you our guidance, but it then goes to you.

Question: Yes, hi. Couple of questions. First one is around the timeline. This is a very restrictive timeline. Do all 250 practices need to complete the 1-year intervention and the 6-months maintenance period, and the 6 months of followup? If that's true, then all 250 will have to be signed on and working by the end of Year 1. Is that the intent? Or can a majority be in the pipeline and some come on later?

So do you have to complete the work by 30 months or can you have a majority completed and some started later?

Answer: Thanks for the question. It is an aggressive timeline. All 250 intervention practices must have received the complete proposed intervention and had 6 months post-intervention data collection by the end of 3 years.

Response: So for all intents and purposes then, everyone has to start by Year 1?

Answer: Actually, though, that is not true, because interventions may vary. There may be somebody who has an 8-month intensive intervention. There may be somebody else that proposes a 4-month intensive intervention. There may be other groups who say, "We've got to diffuse 18-month intervention." All of them would have different requirements about when people would have to start to finish. So there are some design possibilities.

Question: The other question I have is a contextual one. Here in upstate New York, we've done a lot of work with a lot of practices, mostly small. But it's been the starting and stopping. Like having a Beacon grant or a (youth) grant or a PCMH. Where they've had practice facilitators in, and then out, and then in, and then out.

How is that counted in terms of quality improvement if they have no active QI—they've had previous exposure but no active QI now?

Answer: In general, if they're small that's absolutely fine. You might want to explain it in the background about the context of your community that you're working with. But even if they had continued to have a practice facilitator and a small-sized practice, if they hadn't been doing ABCS recently I think that would still be consistent with our intent.

And as the question earlier, it's for larger practices where you'd have to show if they've got ongoing internal infrastructure or externally through their larger infrastructure, that's different. For your example, you're okay.

Question: Quick question about the multiple PI issue. So you're not allowing that? It seems to me it would lend itself beautifully to a multiple PI application. Is that an absolute no for this?

Answer: It's a great question. You can call them Czars, co-gods, whatever you want. But you get one and only one PD/PI on the official application. It is an AHRQ-wide thing.

Question: Hi. The State of Oklahoma has 3.8 million people in it. And the population centers between Oklahoma City and Tulsa, which covers essentially 2 million of that 3 million. But they're roughly 2 hours from each other. And then other major metropolitan areas are 2 hours in any direction from either one of those cities.

If we were to create a collaborative across the State, is that acceptable to get to the 250 practices? Just given the size of the population of the State, it may potentially limit our ability to meet the requirements.

Answer: I would have said your numbers sounded perfect for the State. In fact, that's exactly the kind of State we were thinking could come in as a whole State. So it's a great way to do it. If there are ways to divide Oklahoma in half and still meet 250 practices that might be possible. But one of—this gives me a great chance at the—and we are about to wrap up to say, though, that while you're thinking in Oklahoma and California and in Texas, New York, Pennsylvania, all the places we've heard. Recognize there may be other people in your regions that are having similar ideas to you.

And the most successful applicants are going to be those who reach out now and come in with really great collaborations. Three different groups in Oklahoma could potentially submit applications, one that's only doing Tulsa. One that's doing the whole State. And one that's doing the panhandle would look at and consider each of those applications. But they would be competing against each other, instead of gaining strength through collaboration.

David Meyers: So thank you so much, Bob. If there's anybody who still has questions that we didn't get to today and you still have them, the team here at AHRQ is available to answer them. The FOA lists the contact information of Phillip Jordan for anything related to the contents of the FOA. Dr. Kishena Wadhwani for anything related to the peer review process. Galen Gregor for anything related to budgeting and financial kinds of things. And if you can't remember any of that, go to Phillip, and he'll get you to the right person. So we encourage you to continue submitting those questions. Bob, final words?

Bob McNellis: Yes. No, thank you, David. And thank you for the team at AHRQ for answering all those great questions. And thank you everybody who joined us today. This is an exciting initiative. We really can't wait to get it kicked off. And look forward to talking with you in the future. And thank you, operator, for organizing our Q&A.

Coordinator: You're welcome. Thank you. That does conclude today's conference. You may all disconnect at this time.

Current as of May 2014

Internet Citation: Accelerating the Dissemination and Implementation of PCOR Findings Into Primary Care Practice (R18): Technical Assistance Conference Call Transcript. May 2014. Agency for Healthcare Research and Quality, Rockville, MD. http://www.ahrq.gov/funding/fund-opps/rfahs14008-transcript.html