Stephanie Fry: Good afternoon Good morning to you on the West Coast. This is Stephanie Fry, and I wanted to welcome you to our Webcast today from the Agency for Healthcare Research and Quality, or AHRQ. Today we’re going to provide you with an overview of the CAHPS surveys, or a Primer for New Users, as we’ve called it. Our speakers today will explain how the CAHPS surveys focus on patient priorities and build on current research in survey development and administration We’re also going to talk to you a little bit about how surveys are administered and discuss ways in which organizations are using survey results Our focus today is pretty broad. We want to really give you an introduction to the CAHPS program, tell you what the surveys are and what their purpose is, talk a little bit about how they’re developed and administered, and give you a bit of a primer about how survey results are used to improve patient experience And more than anything we want to provide answers to your questions, and thank you for all of you who posed questions in advance. We’re hoping to get through as many of those as possible today We have a great lineup of speakers. We’ll be starting with Dr. Caren Ginsberg. Dr. Ginsberg serves as the Division Director for AHRQ’s work on CAHPS and also the Surveys on Patient Safety Culture. An anthropologist and demographer, Dr. Ginsberg has broad-based experience in patient experience, patient safety and public health. In her position at AHRQ, she focuses on program development, implementation, operations and evaluation, with a specialty in survey design and development and qualitative evaluation and assessment. We’ll also be hearing from Dale Shaller Dale Shaller is a Principal of Shaller Consulting Group, a health policy analysis and management consulting practice based in Stillwater, Minnesota. He has devoted over three decades to the design, implementation and evaluation of health care quality measurement and improvement programs, with a special focus on listening to the voice of the patient and promoting methods for engaging consumers in managing their health care He has served as a member of the CAHPS Research Team for 20 years and has directed the National CAHPS Database since its inception back in 1998 I’m Stephanie Fry, and you will be hearing my voice today, as well I’m a survey methodologist and Senior Study Director at Westat, which is the research organization that supports the CAHPS program under contract to AHRQ I am the survey instrument lead at Westat and have supported the CAHPS program for nearly 15 years I will serve as your moderator for the next hour or so Before we begin and jump into our content, just a couple of quick housekeeping details. If you’re not getting sound from your computer speakers, please do join is by phone. And there is a telephone number up here on screen. If you’re having trouble with your connection or your slides not moving forward you could try selecting F5 on your keyboard, or at worst you could log out and log back in Please be aware that sometimes there is a little bit of a delay due to internet speed, so bear with us and we’ll try to make sure that you get the full content. And if you have other questions, please do use the Q&A feature and ask for help and we’ll do our best to get you set up with whatever you may need We will be taking questions at the end of the Webcast, so as we go along if you have questions that you want to pose, or certainly at the end, and we’ll remind you again, there is the Q&A box. We’ll ask you to type your question here, and then we will discuss them aloud and try and get answers to as many questions as we possibly can in the next hour There are presentation materials, including these slides, available to you now, and again after the Webcast, so please feel free to download those and use them And one more piece of resource for you, we do have several documents and Web links available to you to support your understanding of the CAHPS program and resources that we will discuss a little bit in today’s Webinar, and these are available to you through the Web console, the green icon there. So, with those housekeeping details behind us, I will turn it over to Dr. Caren Ginsberg Caren Ginsberg: Thank you, Stephanie, and welcome, everyone. Thank you for joining this Webcast on CAHPS surveys. And I want to reiterate Stephanie’s thanks to you for providing such valuable input when you registered on what you wanted to learn from this Webcast. And we hope to answer many of your questions, but if we don’t, please email us. You can find the email address in the Resources tab or on the last slide

Those comments really helped us understand what your concerns are and what your interests are and helped us think about this presentation. So thanks again. So, I’d like to tell you a little bit about the Agency for Healthcare Research and Quality, or AHRQ AHRQ is essentially a research and development agency. It’s one of the federal agencies of the U.S. Department of Health and Human Services, and we are the lead agency charged with improving the safety and the quality of America’s health care system As a research and development agency we invest in research on the nation’s health delivery system to understand how to make health care safer and improve quality, and we develop knowledge and tools and data for improving the U.S health care system and health care consumers and professionals and policymakers make informed decisions. I would like to point out that AHRQ is not a regulatory agency We don’t mandate the use of CAHPS surveys So if you see requirements to use CAHPS for other purposes, those requirements come from other federal agencies or outside organizations So that’s an important point, by the way, and we’ll be coming back to that again in this next hour. So, CAHPS stands for Consumer Assessment of Healthcare Providers and Systems, and it’s really one of AHRQ’s flagship programs We’ve been around for 22 years now And the CAHPS program advances the understanding and measurement of patients’ experiences with their health care We do this through a research program that explores all aspects of patient experience understanding as well as create tools and materials to measure and report patient experience. So the CAHPS Consortium provides oversight to the AHRQ CAHPS program, and the Consortium consists of staff from AHRQ; our two funded grants – the grantees are RAND and Yale University; and our support contractor, Westat So one of the tools that I mentioned are the CAHPS surveys. And you’ll hear more today about everything on this slide, so I’m just going to give you the preview. So, the surveys measure patient experience of care in different settings or with health plans or with health care providers. And we use a standardized methodology to develop the surveys, and we base the final survey on extensive testing and research findings. The CAHPS surveys are trademarked, and the trademark is held by AHRQ. The trademark means that the survey development process, testing and final survey all adhere to the CAHPS standard design principles. I want to point out that CAHPS is recognized now as the gold standard for patient experience of care measurement, and, even more importantly, the CAHPS program is committed to capturing the patient’s voice in our surveys. The surveys reflect what patients tell us is important to measure So, you can see on this slide some of the uses of CAHPS surveys. They’re used in reimbursement programs such as value-based purchasing, public reporting, accreditation, quality improvement, health services research. And, as I said earlier, AHRQ does not require the use of CAHPS for any purposes such as the reimbursement program. The organizations and agencies that require the administration of a CAHPS survey will have their own requirements for participating in the programs and how to administer the survey, so it’s best to check with them for their requirements Now, that said, AHRQ does have databases for survey data from our health plan and clinician and group survey data that are voluntarily submitted to us, and you’ll hear more about that later. And, finally, I just want to tell you a little bit about patient experience of care research at AHRQ. CAHPS is part of a larger patient experience of care program at our agency, and aside from the CAHPS surveys and tools we maintain an active research program with patient experience, understanding and measurement. Our current research includes understanding the relationships between patient experiences and care coordination, shared decisionmaking, patient engagement, patient safety. We’re also conducting research on using patient narrative information as part of Clinician & Group CAHPS Surveys and other surveys, CAHPS surveys, and on the effectiveness of different survey administration modes for collecting CAHPS data So, thank you. Stephanie, back to you Stephanie Fry: Thank you, Caren. So, with that background about AHRQ and their stewardship of the CAHPS program, I want to talk a little bit about what is patient experience and how does CAHPS measure it. So, here you’ll see a definition of patient experience And there are many definitions of patient experience out there. This is the AHRQ’s CAHPS program definition. And you’ll see that it really focuses

on the range of interaction that patients have through the health care system And we’ll talk a little bit about how that gets implemented into different surveys. And it really focuses on those aspects of health care that patients value highly So, this here forms the basis of the CAHPS program Before we jump into talking any more about the CAHPS program and measurement of patient experience, I wanted to take just a moment and talk about the difference between patient experience and patient satisfaction. Those are terms that are sometimes used interchangeably or concepts that can be conflated, and they really do have different meanings and they serve different purposes. So, the CAHPS program really looks at patient experience, and what patient experience is it’s a focus on patient reports We’re asking patients whether something that should’ve happened actually did happen and how often that thing happened We use a frequency scale to determine that, and we’re looking for objective assessments of that patient experience, of those factual elements of that health care delivery. Patient satisfaction, on the other hand, looks more at patient ratings and whether patient expectations were met, how they felt about their care. There’s a reliance on rating scales, and it’s more of a subjective assessment. It’s that individual’s report on how satisfied they were, whether that particular interaction met their personal expectations So, the CAHPS survey, which, again, looks at patient experience, has some really core principles, principles both in terms of how surveys are developed and what they include, and how they are made available thereafter. The CAHPS surveys have this core principle that they focus only on those elements of that health care interaction for which patients are the best or only source of information. If there’s information available through medical records or through some other source of information, then we needn’t take patient time to respond to them in a CAHPS survey. So the CAHPS surveys are really just those elements that we need to hear from patients about. There is a rigorous scientific background for all the development and testing, as Caren already mentioned, and there is extensive stakeholder input into what are those most important elements of that patient experience for that particular setting or with that type of doctor or health care provider All of the CAHPS surveys are also in the public domain, so they are available to be used free of charge They’re on the AHRQ Web site, and we encourage you all to take a look at them and access them as you need One of the reasons that patient experience is so important is that patient experience is strongly correlated with other important outcomes There is much testing and research that has been done that links positive patient experience with positive health outcomes. So through various factors that we may be looking to measure, from patient adherence to specific clinical outcomes, there is a strong correlation. As patient experience improves, so, too, do these positive health outcomes There are also business outcomes that have been linked to positive patient experience scores in terms of malpractice risk, employee satisfaction, and also financial performance. There are some follow-on materials that you will see linked at the bottom of this page that are also available to you on our resources, should you want to get more information about the connection between patient experience and these key outcomes. I would say, however, while there is linkage to many different positive health outcomes and financial outcomes, really patient experience is important, and it’s critical to measure as part of our health care experience because it matters to patients and families. It is important in its own right. And I think it warrants taking a moment to just make sure that that fact is quite clear In terms of the CAHPS survey content, as we’ll get into in a moment here, there is a whole family of CAHPS surveys, and it begins with the CAHPS Core Questionnaire. To that Core Questionnaire supplemental items can be added. So there are CAHPS

supplemental items that have been tested and that are made available through the AHRQ CAHPS Web site. There is also some information provided to users about how to incorporate your own individual questions, should you feel you need them. And so through both the core survey plus the supplemental items users get to their own customized CAHPS survey, which represents the rigor and survey science that AHRQ uses in development and allows people to customize those surveys to ensure that they get the information that they need to answer their research questions or to support their initiatives of a variety of kinds CAHPS Core Questionnaires include a common set of questions that look at particular aspects of care that we have found to be important to patients, and these include access to care, so getting the care you need when you need it; communication with providers or health care teams; coordination of health care across those providers and teams and settings; customer service in terms of interactions with the individuals that you may need as you first come into a doctor’s office or a practice of some kind; and an overall rating of a provider or a health care experience. Those form the basis of the core CAHPS questionnaires, and, again, then there is the flexibility to add supplemental items as needed to fit your individual needs CAHPS supplemental items have been developed to address some common areas of health care that we’ve heard from users are really important, and these include shared decisionmaking, health literacy, interpreter services, health information technology, and new to the CAHPS family of supplemental items is the narrative elicitation, where patients are requested to use their own words to describe their health care experiences There are other supplemental items also available, and these all are on the AHRQ CAHPS Web site available through a searchable database Some examples of the core surveys are listed here for you. As you see, there are various different levels of surveys, from provider-level surveys – the most familiar may be the Clinician & Group Survey – and then there are other surveys that look at individual condition specifics, such as the Cancer Care Survey, again, a new addition to the CAHPS family of surveys. There are facility-based surveys. Many of you may be familiar with the hospital CAHPS survey, or H-CAHPS, and there are others as well here. And then there are surveys that look also at health plans and at programs These all make up the CAHPS family of surveys To get to these surveys they’re all available through the AHRQ CAHPS Web site And so there’s a screenshot here for you to take a look at. There are surveys and some information available for all the surveys you see listed here. As well, you will find a link to some surveys that are administered or required by CMS, and we provide you a link to the CMS site in the event that you would be interested in seeing their requirements for administration, which extend beyond the research and science that AHRQ provides, and we do want to make sure that people get access to all of that content here easily, all in one place. So, with that background about the CAHPS survey, I will turn it over to my colleague, Dale Shaller, to tell you some of the key things that you should really know about CAHPS surveys Dale? Dale Shaller: Yes, hey, thank you, Stephanie. And this segment will do that. We will focus on several important things that you should know that are fundamental and true about all CAHPS surveys And the first thing is that not only are patients often the best and sometimes the only source of information regarding patient experience, they are always themselves the central source of information about what CAHPS surveys will focus on. So the topics and the features of care and service delivery that are included in the various CAHPS surveys are derived from patients themselves. And we use mechanisms

such as focus groups, and we’ve done hundreds of focus groups over the span of the CAHPS program; interviews individually with respondents to get their perceptions about how well questions perform; and ongoing input that we receive from patient and consumer advisors through advocacy group relationships that we have through the CAHPS program So I think it’s very safe to say that there’s no measurement content included in CAHPS surveys that has not originated in some way from patients or consumers regarding the aspects of care that they want and need to know about The second thing is that second only to patient input and survey development is the involvement of stakeholders and users in the process. And by stakeholders we refer to organizations or groups that have a vested interest in the surveys, because they are often the subject of the measurement, whether they’re providers, hospitals, health plans, or groups that will be using the survey results for various applications, such as reporting or payment certification or improvement, or even consumer choice and decisionmaking. And the mechanisms that we use for gathering stakeholder and user input include advisory panels Every survey developed through the CAHPS development process includes a TEP, or a technical expert panel We use a public comment period through the Federal Register to solicit comments and recommendations, questions and input from stakeholders and users. We request feedback on an ongoing basis after surveys have been released And the practical experience we gain through the field testing process with our partners is really fundamental to understanding how the surveys are actually playing out in a real sense among stakeholders and users Another important thing to know about CAHPS surveys is that they are all based on the latest research evidence that relates to survey methodology The CAHPS team members include some of the country’s most experienced survey scientists and psychometricians that are engaged in ongoing research and testing regarding sampling and administration methods, analysis methods, reporting methods. And some examples will be provided a little bit later in our Webcast in terms of, for example, on survey administration, the next phase of CAHPS research underway now we’ll be looking, for example, at SMS text messaging as one possible form of improving response rates through better invitation methodologies, etc. And Stephanie mentioned the narrative supplemental item set that has been developed with CAHPS, and so we’re looking at ways of gaining better insights in how feedback reporting can be used for disseminating survey results back to the providers for their use Another thing that you should know about CAHPS surveys is that the survey content and the administration methods are extensively field tested so that we can learn how these surveys actually perform in real-worid applications And this isn’t done just once, but often through multiple rounds of testing to learn about the psychometric properties of surveys, such as their internal reliability within the survey instrument itself, their site-level reliability, the extent of missing data that shows up, or sealing effects of the various questions, looking at response rates and ways of improving the representativeness of the sample of patients that are surveyed So all of this is intended to assure that the finished product performs as intended. And it’s important to note that these products are never finally finished, in the sense that we’re constantly evolving, we’re refining them through innovation to improve the product over time, and that’s why you will see that we have various versions of a number of the CAHPS surveys, such as a 3.0 version of CG-CAHPS and a 5.0 version of the CAHPS Health Plan Survey. So it’s an ongoing process Another thing to know is that standardization of CAHPS surveys actually is the key to their comparability across different applications and users, and comparison data are essential for improvement. The CAHPS Database is available as one resource to provide comparative data for selected surveys, and we’ll talk a little bit

more about that in a moment. But the idea here is that the core survey content that Stephanie described does support the standardization that’s essential for comparison even while supplemental items are available to support customization for specific applications So, the sixth and final thing to say, I think, about CAHPS surveys, is that all the survey questionnaires, the tools, the resources that are developed through the CAHPS program, are publicly and freely available. The CAHPS program is a totally open-book enterprise There is no black box here. Everything is open to inspection and free to all users So, that’s a quick spin through six important things to know about all CAHPS surveys, and, Stephanie, I’ll turn it back to you Stephanie Fry: Thanks, Dale. I’m going to take just a moment now and talk a little bit about how CAHPS surveys are administered So, Caren mentioned this as she began her discussion, but I think it warrants just recalling it, that AHRQ is a research and development agency, so AHRQ doesn’t require any type of survey administration, but, rather, AHRQ provides information about best survey science and develops surveys with this really high research bar. And so this information, as Dale just mentioned, is made available wide and open There are other organizations, then, that may pick up the CAHPS surveys because of their rigor and because they are, as Caren mentioned, the gold standard in patient experience measurement, and they may pull them in as part of their organization’s approach to measurement. So, for example, CMS requires some CAHPS surveys to be fielded, and they have very specific requirements about how that must be done. Those are requirements that come from CMS, not from AHRQ, who develops the CAHPS surveys, so just something to keep in mind, that the AHRQ information is there to help you and to provide the scientific backbone for all of the CAHPS program. So, in terms of sampling, starting right from the beginning, how many surveys do you need to put out in the field in order to answer your questions? So there is extensive testing that is conducted by the CAHPS Consortium to assess reliability and validity of CAHPS items. And really what that means is how many CAHPS survey completes do you need in order to get good data that will allow you to report on either individual survey questions or on groups of questions that we call composites. So what number do you need? And there are a number of factors to consider, and we provide this information to you to help you think through what level of reporting do you anticipate. Are you reporting at a facility level, so something like a hospital or a nursing home, or are you trying to provide data well down into the weeds at a doctor or provider level? All of these things will impact how many questionnaires you would need to put in the field, how many completes you need and how you would want to draw your sample. The other thing to consider in sampling is how are you planning to use the data? If you are using the data to support public reporting, you will want one type of sample that may be somewhat larger. If you are using it for your internal quality improvement purposes, perhaps a smaller sample size may be needed. So these are all factors that AHRQ tries to help you to think through as you are embarking in a CAHPS survey In terms of data collection modes, or ways to collect the data, there are a whole host of different ways that the CAHPS Consortium has tested to try to figure out what is an efficient way to get good coverage for surveys. And so you’ll see on the screen here a whole range of different ways that we’ve tried to collect survey data And Dale mentioned there is ongoing testing that continues to this day And so there’s all of these ways that you can collect data, and as testing is concluded the CAHPS Consortium tries to publish papers to make all of this publicly available to people to support other users in their measurement of patient experience

What we’ve found to this point is that in order to achieve the highest response rates, mail or telephone or a mixed-mode approach with mail and telephone, or email with mail or telephone as a follow-up, these are the methods of data collection that are going to yield you the highest response rates, that is, the greatest number of completes from the surveys that you mail. It’s not to say that there is a requirement, as I mentioned before, to do anything, but rather just a recommendation that to yield the highest response rates these are methods of data collection that we would have you consider. That said, we are aware that the environment is rapidly evolving in terms of how people communicate, how they expect to receive information, so the CAHPS Consortium continues extensive testing and providing Webinars and publications about findings of that testing. So stay tuned as the worid continues to evolve So now that you have your sample, you’ve put your surveys out, you have all of your completes, then what do you do with them? Again, there is information provided to help support users with the analysis and reporting portions of their patient experience journey. And there is information out there about how to group questions into composites, which really are just groups of like questions that are all measuring a similar topic – for example, communication or access to care. So we help you in getting those questions grouped together as they were intended and as they have been tested. There is also support out there for case mix adjustment. And what that really looks at is if you want to level the playing field for providers that have different kinds of patients, there’s the CAHPS macro that helps you to adjust your survey data for respondent age, education and health status. And what this adjustment does is that it makes it more likely that any differences that you would see in comparisons are due to real differences rather than just differences in the characteristics of patients in particular groups or practices or at different facilities So that is available to users also free on the CAHPS Web site. And then the other piece that is provided is information about reporting. So, depending on what your reporting goals are – they may be internal quality improvement, they may be public reporting – there is a wealth of information provided to you about different strategies to approach that reporting to achieve different goals So, we’ll talk a little bit more about resources in Dale’s section, and, as I mentioned at the beginning of this Webcast, there is a whole host of resources available to you, including these ones here through this Webinar resource function But the AHRQ CAHPS Web site is really the place that we would send users to begin with to get most of everything that they could need in terms of looking at the surveys themselves, identifying what the measures are, and seeing the research on survey administration and reporting. There’s also frequently asked questions and answers there that should address many questions that particularly new users may have There is a searchable bibliography, and the CAHPS Ambulatory Care Improvement Guide, which provides information on how to improve in domains of patient experience We’ll also talk a little bit more about the CAHPS Database Web site, where you can submit data and receive comparative results. And there is the resource of TalkingQuality, which provides information on reporting CAHPS data. We also have ongoing technical assistance for users by email, by phone. We’re happy to support you as you take on your measurement of health care and patient experience. So with that I’m going to turn it back to you, Dale, to talk a little bit about use of CAHPS survey results Dale Shaller: Okay, thank you, Stephanie And so we’ve already touched on some of the key uses of CAHPS survey results already, and this is the last segment before we stop and do our Q&A portion of the Webcast The major uses of CAHPS survey results are noted here in terms of public reporting that can support transparency and consumer

decisionmaking and choice or selection among different options; value-based payment that financially rewards providers for superior performance on patient experience measures; recognition and certification that helps to assess and maintain a high standard of performance among organizations that seek recognition, certification or accreditation from various organizations. And these first three uses of survey results actually are levers or important forces that drive the next one, which is quality improvement, which is really the end result that we’re all hoping to achieve anyway through the development and use of CAHPS surveys And research applications support all of the above in terms of improvements to reporting, payment, certification and improvement processes So just briefly, to give a few examples, there’s a lot of public reporting happening in the country today, and actually internationally, using patient experience measures at the federal level, state initiatives, regional collaboratives, and health care providers and systems themselves that are collecting and reporting their own performance score So there are many examples. I’m just going to show you two. This first one is from the CMS Physician Compare Web site, which shows a display of patient survey scores for – this is Allina Health System, which is a major health system in Minnesota that happens to have an accountable care organization. You can see there’s three summary measures shown here related to the access composite, provider communication, and the health promotion and education, which is a summary measure within the ACO CAHPS survey. This particular display shows the percent most positive, which is often referred to as a top box score, so that you can kind of see at a glance how this system is performing in the upper trinity to drill down for further information if you’d like. This second example is drawn from the Minnesota HealthScores Web site, which is maintained by Minnesota Community Measurement in Minnesota. And this shows clinic-level comparisons for three medical groups that operate in the state. Again, the focus here is on top-box scores for several of the CAHPS measures. The first one is the provider rating, then the next is care coordination composite and office staff. And it does kind of show you a different way of displaying side-by-side comparisons, what the actual score is, and then some kind of icons that indicate whether the scores for a particular group are at, below or above average And so it is clear that there are certain differences that exist among medical groups, and this is the kind of thing that public reporting efforts are intended to show There’s been a lot of activity over the last decade in the use of CAHPS survey results for value-based payment Probably the most well-established program in the country today is the Hospital Value-Based Purchasing program for H- CAHPS that’s managed and operated by CMS. It started back in 2008 as kind of a pay-for-reporting program, and it evolved into a pay-for-performance, where today on the order of 30 percent of a hospital’s payment is pegged to their performance on H- CAHPS scores in terms of the 30 percent of the formula that’s used to calculate payment adjustments to hospitals. So CAHPS is playing a major role in the hospital use of patient experience for value-based reimbursement With the Affordable Care Act, the Medicare Shared Savings program evolved, and so concurrent with that was the development of the CAHPS for ACO, the accountable care organization survey that’s now a required element of the ACO program in terms of being able to publicly report and receive payment for performance within the Medicare Shared Savings program The MACRA legislation, which is the Medicare Access and Chip Reauthorization Act, which totally reformed

how Medicare handles payment to physicians and physician organizations, now includes what’s called a quality payment program, which has two components that replace what used to be the PQRS, the Physician Quality Reporting System and value-based modifier, with what’s called a Merit-Based Incentive Program, which allows practices to use the Clinician & Group CAHPS Survey as an optional way to achieve credit for quality improvement activity, and also in terms of another program called Alternative Payment Models, which gives organizations a little bit more flexibility in adopting new delivery approaches to improving quality and efficiency as measured in several ways, including through the CAHPS program There’s another large program underway called Comprehensive Primary Care that’s maintained by the innovation arm of CMS, the Centers for Medicare and Medicaid Innovation This is a program that basically is looking at ways of advancing primary care and uses the CAHPS survey as both an evaluation tool and as a tool for payment among a very large number of primary care practices, a total of almost 3,000 across the country in 14 different regions. And this is a program CMS is conducting in collaboration with many other payers. So it’s not just CMS. It’s a combined kind of collaboration among other public and private payers throughout the country And there are commercial pay-for-performance programs that have been underway for quite a long time in various states, various health plans using CAHPS survey results to basically reward performance within their networks And internally to many health care organizations are provider compensation programs that have begun to peg reimbursement or compensation to health care professionals on the basis of their own individual CAHPS survey scores. So a lot of activity underway with the use of CAHPS survey results for value-based payment. The use of survey results in recognition and certification is also pretty widespread for health plans, hospitals, medical homes. A couple of logos here for NCQA, the National Committee for Quality Assurance, and URAC, which are two prominent organizations that do accreditation and recognition for various health care delivery programs. And the medical board certification noted here is an activity that is managed by the American Board of Medical Specialties, which is kind of an umbrella group consisting of 24 member boards across all the major medical specialties that they maintain a program called maintenance of certification, and they’ve been looking over many years now at the use of CAHPS surveys in some of these programs for maintenance certification for individual physicians. So, as I mentioned, many of the uses of CAHPS surveys for reporting and payment and accreditation are actually helping to drive this increasing focus within health care organizations on improvement across health plans, within health care provider practices and health systems There’s also a sense of growing demand among the patient population within consumers, with sort of an escalating expectation of improved service and sort of their interaction with the health care system. The CAHPS program maintains a number of resources to support improvement. We’ve noted already the CAHPS Ambulatory Care Improvement Guide as one resource. It’s been developed over the last 10 years and has been updated on various points within the last decade, made available as a free resource on the CAHPS site, augmented by a number of Webcasts we’ve done over the years and podcasts and case studies that document specific approaches to quality improvement by different health care organizations to give users examples of how this has actually been applied, challenges that have been encountered, how those have been overcome, and the results that have been achieved So, finally, CAHPS survey results can be used in research, and there is a lot of that underway. We’ve noted already within the CAHPS program itself, looking at

best practices for survey design and administration, but also a number of demonstration programs and tests underway to look at ways of improving public and private feedback reporting, value-based payment programs, and quality improvement programs. Within the CAHPS program we maintain several sources that can be useful for supporting research. The CAHPS Database has a set of research files that are available upon request that are reviewed, and if accepted as a research project a data use agreement is put together. And these are de-identified respondent-level files available currently for the Health Plan and CAHPS Clinician & Group Surveys. CMS, for its part, within its programs that it maintains, does provide a set of data files Sometimes they are at the patient level. This depends on the program and the survey. I know for the Health Plan Survey and the ACO Survey there are datasets available through CMS through a process of a data use agreement that allows the availability of those files for research. Some programs that CMS maintains do not provide patient-level data, but rather downloadable datasets for, for example, for the H-CAHPS Survey directly from the H-CAHPS Web site that CMS maintains. And one further resource available for research is what’s called the SEER, which stands for Surveillance, Epidemiology and End Results, and Medicare CAHPS Linked Dataset This is a program maintained by the National Cancer Institutes which provides researchers datasets that can be used for analyzing patient experience among cancer patients, at least within the Medicare program So, finally, just a quick note on the CAHPS Database We’ve talked about it a couple of times throughout the Webcast today. This is a voluntary program. We invite users of the CAHPS Health Plan Survey within Medicaid and CHIP populations and the CAHPS Clinician & Group Survey to submit data voluntarily. We have an annual cycle of submissions and reporting coming up in this year for the CAHPS Clinician & Group Survey We will be opening submission in May and likely reporting results in August And for the Health Plan Survey, submission opens in July and we report results generally by October of the year The applications for the database that we maintain are primarily for comparative data that we talked about one of the features of CAHPS surveys is having the standardization that allows comparability across survey applications, and so we have several products that the CAHPS Database maintains for providing comparative data through an online reporting system and through an annual set of chart books that we produce with summary-level data that is available for all users in the public. And as I just was mentioning, we have several de-identified datasets for research purposes So, this is all a voluntary process. It is free and open to all users of CAHPS surveys. And we encourage users to check out the CAHPS Database, the features that are available, the fairly simple approaches to follow for participation. And there’s much further information regarding the CAHPS Database and all of the resources that we’ve described today on the CAHPS Web site. So we have some time remaining. Stephanie, I’m going to turn it back to you and see what kinds of questions we might’ve had at this point Stephanie Fry: Thanks, Dale. So, we have just shy of 10 minutes now to answer some of the questions that were submitted in advance and questions that attendees to this Webinar have posed using the question-and-answer box. So I will just remind you that you can use the question-and-answer function to type in a question and hit Enter, or hit Submit, and we will do our best to respond to as many questions as we possibly can in our remaining time here. So I’m going to begin with covering off one broad topic that seems to have

popped up through many of the questions so far. There’s a fair number of questions about CMS requirements, either what surveys are required or what surveys is CMS going to be requiring in future or using for reimbursement or other CMS programs in future. And I will let you know that AHRQ cannot respond to those questions Those are questions that will need to be posed to CMS. There is a wealth of information on CMS Web pages about the surveys that they require and how they require those surveys to be administered. So if you begin from the AHRQ CAHPS Web pages you can navigate to the various survey-specific pages to get information about what CMS does require. And, unfortunately, in terms of what CMS is planning to do down the road, we do not have that information to share with you. So I will just sort of cover off the questions with regard to CMS with that broad response of unfortunately we’re going to need to point you to CMS for that. But AHRQ can answer any questions you may have about the science or development or background testing or any of those sorts of things about the CAHPS program. So, with that said, I’m going to turn to a couple of questions that we have received here. So, Dale, maybe I will get you to jump in as our database expert here and explain a little bit about top-box results and the differences between top-box and summary star scores. And that meanders into CMS territory in terms of the scoring that they report there, but if you could just speak a little bit about top-box and reporting on that front that would be great Dale Shaller: Sure Top-box is a term that’s used to describe within a set of possible survey response options that are scaled. For example, CAHPS surveys, as Stephanie mentioned, we use a frequency scale most often, which basically has options such as always, usually, sometimes and never, which are four response options. The top box is the top most positive part of that survey scale So in a never to always scale, the top box would be always responds, and so top-box scores are basically the percent of survey respondents responding always. It’s used because there are different ways to score and report survey results, but top box is, what we’ve found through consumer testing, is a pretty easily understood concept. You can express it as a percentage. It’s a simple, single number One could also use mean scoring, which is average. Instead of reporting just the top box you could average the response values across all four of those response options and come up with a mean value But what we’ve also found in testing is that’s a little bit more difficult concept to explain. So that’s essentially what top-box results are about Star rating is a much more – is different. It uses survey results along with fairly – depending on who’s doing the scoring, to get you a star-based result can look at other factors and weights to come up with what is, for example, a four-star or a five-star result It’s not at all the same as a top-box score unless the reporter is using the top-box score to equate to a star Generally, through star rating, it’s a much more complicated and involved kind of methodology to get to what’s actually a five – or four-star rating Stephanie Fry: Thanks, Dale. I’m going to take another question now that’s asking about modes for data collection And I just want to clarify a couple of things with regard to mode here. We listed, as one of our participants here noted, basically every possible mode there is out there And that in fact we did. And those are all of the modes that we have tested or are in the process of testing. As the worid continues to evolve and people receive information differently, AHRQ wants to ensure that

the CAHPS surveys stay abreast of that and really understand the science behind getting people to respond to surveys and collecting this really important information. So, yes, we have tested just about every imaginable mode. What we are finding at this moment there’s a question about trying to get higher response rates. In terms of achieving high response rates, the modes that we have found that achieve those best response rates are either telephone or the good old-fashioned mail-out survey, which strikes people as unusual, but it really does achieve the goal, or a mixed mode, where you use a combination of mail first with a telephone follow-up for people who don’t respond, or where you use email first if in your particular setting you have good email addresses. However, we do really recommend that there be a follow-up either with mail or telephone to ensure that you’re getting good coverage across the population with those modes of administration. So that’s where the state of science is at this moment. And, again, there is testing that continues, so we will update information as we learn more and as people respond to surveys and to questions in different ways. So, Dale, I will pass another one over to you about how can health plans, and I think we could probably address this more broadly, use CAHPS results to encourage or incentivize patient experience improvements among providers and in clinical settings? So how can CAHPS results be used to achieve better patient experiences? Dale Shaller: Well, that’s a great question, and there are different ways in which that can happen, and there are different examples of it being done. One example that comes to mind is a longstanding program in California called the Health Insurance Association. It’s a group of health plans that use standardized CAHPS measures collected at the medical group level and collaboratively use that information to set pay-for-performance incentives within the health plans that they contract with. There are other plans – I can think of examples – I won’t name them – but in other states that use CAHPS survey results collected within a hospital setting as well as the medical group and practice setting, again, as part of a reimbursement methodology that basically rewards the medical groups that they contract with with superior payments if certain target goals are met. So the use of value-based payment again in the context of health plan contracting with providers and clinics is one important way that plans can encourage improvements. Another way is through actually sharing results through feedback reports that provide – for example, a provider network that a health plan uses could extend to a set of data on patient experience survey scores that a single provider group itself may not have access to, so that the health plan, in a sense, could become a source of comparative data and feedback reports that identify areas that providers can actually focus on. And this is an activity that’s been undertaken without the use of financial incentives in many regional collaboratives that involve both health plan and provider organizations and make that information available The incentive becomes the information itself, and the ability for providers to understand where they stand compared to their peers, because oftentimes that information in itself is enough motivation for improvement activity to begin. The next step, of course, is then to provide resources and guidance on how actually to undertake the improvement activities, and that’s, again, where some of the resources that the CAHPS program maintains can come into play, for example, the CAHPS Improvement Guide So that actually, so we talked about that resource once already, and that’s actually a very important source of information not only

about program activities but specific interventions that both health plans and provider groups can use to improve their scores. Stephanie Fry: Thanks, Dale And I know we are at the very tail end of our Webinar, so to those of you who asked questions and didn’t get responses, we will ask you to please follow up with us through the AHRQ CAHPS Survey User Network. We’re happy to respond to any further questions that you have Thank you all for attending this Webinar. I hope you have taken some useful information about the CAHPS program and about the CAHPS surveys and AHRQ’s role as research and development partner and ensuring that the patient voice continues to be important There is a survey that we would request for you to fill out to give us some feedback about this Webinar and its utility to you, so please take a moment to fill out that survey and submit the survey when you are done with it We also want to remind you that there are many resources available to you. You can join the CAHPS GovDelivery listserv to stay abreast of all new things unfolding in the worid of CAHPS. And we have many additional resources available to you, as well, that you will find on this Webcast console and through the AHRQ CAHPS Web site If you have questions or comments, please do follow up with us. And we thank you again for attending this Webcast. Have a wonderful day