Key Considerations in Using Real-World Evidence to Support Drug Development(Draft for Public Review)Center for Drug Evaluation, NMPAMay, 2019Table of Contents1.INTRODUCTION (1)1.Background and Purpose (1)2.Progress in the development of related regulations or guidelines by domestic and foreign regulatory agencies (3)2.Relevant Definitions of Real-World Research (4)1.Real-World Data (5)2.Real-World Evidence (9)3.Scenarios where real-world evidence supports drug development and regulatory decisions (9)1.Treatment for rare diseases (9)2.Revision of indications or drug combination labeling (10)3.Post-marketing evaluation (11)4.Clinical development of traditional Chinese medicine hospital preparations (12)5.Guiding clinical trial design (13)6.Identify the target population (14)4.The Basics of Real-World Research Design (14)1.Pragmatic clinical trials (14)2.Single-arm trial using real world data as control (15)3.Observational studies (16)5.Evaluation of Real-World Evidence (16)1.Real world evidence and the scientific questions it supports (17)2.How to transform real-world data to real-world evidence (17)References (19)Appendix 1: Glossary (21)Appendix 2: Common Statistical Methods for Real-World Research (24)Appendix 3: Chinese-English Vocabulary (42)Key Considerations in Using Real-World Evidence 1to Support Drug Development231.INTRODUCTION41.Background and Purpose5Randomized Controlled Trials (RCTs) are considered the "gold 6standard" for evaluating drug efficacy and are widely used in clinical 7trials. With strictly controlled trial eligibility criteria and the utilization of 8randomization, RCTs minimize the impact of factors that potentially 9affect the causal inference, and hence result in more definitive 10conclusions and derive more reliable evidence. However, RCTs also have 11limitations: stringent entry criteria may reduce the representativeness of 12the trial population to the target population, the standard trial 13interventions used may not be completely consistent with real world 14clinical practice, the limited sample size and short follow-up time leads to 15insufficient evaluation of rare adverse events. These limitations bring 16challenges when extrapolating the RCT conclusions to real world clinical 17practice. In addition, for some rare and major life-threatening diseases 18that lack effective treatments, conventional RCTs may be difficult to 19implement, require substantial time costs, or raise ethical issues.20Therefore, how to use real-world evidence (RWE) during drug R&D, 21especially as complementary evidence to RCTs in evaluating the efficacy 22and safety of drugs, has become a common and challenging question for 23global regulatory agencies, the pharmaceutical industry and academia.24First, we need to clarify the definition and scope of real-world 25evidence on a conceptual level.26Secondly, can and how will real-world data (RWD), as the 27fundamental basis of real-world evidence, provide sufficient support will 28face many questions that need to be discussed, including data sources, 29data standards, data quality, data sharing mechanism, data infrastructure 30and so on.31Third, the lack of regulatory guidance. At present, there are no 32mature and relevant regulations worldwide. Without sufficient experience, 33how to formulate guidelines that fit the reality of China's pharmaceutical 34industry requires active exploration and innovation.35Fourth, the methodologies for evaluating real-world evidence needs 36to be streamlined. Real-world evidence stems from the correct and 37adequate analysis of real-world data. The analysis methods used are 38mainly for causal inference, which often requires more complex models 39and assumptions, screening of corresponding covariates, identification of 40confounding factors, definition of intermediate variables and instrumental 41variables, etc., All these will put forward higher requirements for 42statistical analysts as well as the urgent needs for regulatory guidelines.43Fifth, the scope of real-world evidence application remains to be 44determined. The main role of real-world evidence is to complement, 45instead of substitute, the evidence provided by conventional clinical trials, 46and to form a complete and rigorous chain of evidence to further improve 47the efficiency and scientific validity of drug development. Therefore, it is 48necessary to clearly define the scope of application of real-world 49evidence according to the stage of drug development, and in the 50meanwhile adopt appropriate adjustment as the actual conditions evolve 51over time.52In light of the above, this guideline aims to provide clarity on the 53definition of real-world research, outline the use and scope of real-world 54evidence in drug R&D, explore the basic principles for the evaluation of 55real-world evidence, and consequently provide scientific and practical 56guidance for the industry to consider when utilizing real-world evidence 57to support drug development.582.Progress in the development of related regulations or guidelines 59by domestic and foreign regulatory agencies60In February 2009, the American Recovery and Reinfection Act 61played a significant role in promoting Comparative Effectiveness 62Research (CER).Accordingly, the concept of real-world research (RWR, 63or real-world study RWS) was proposed given the context of the real 64world environment of CER.65In December 2016, the United States passed the 21st Century Cures 66Act (the Act), encouraging the Food and Drug Administration (FDA) to 67accelerate the development of pharmaceutical products by conducting 68research in the use of real-world evidence. Under the support of the Act, 69during 2017-2018 the FDA issued a series of guidelines, namely "Use of 70Real World Evidence to Support Medical Device Regulatory Decisions", 71"Guidelines for the Use of Electronic Health Record Data in Clinical 72Research" and "Framework for Real World Evidence Solutions".73In 2013, the European Medicines Agency (EMA) released the 74"Qualification opinion of a novel data driven model of disease 75progression and trial evaluation in mild and moderate Alzheimer’s 76disease", discussing the technical details in using real-world observational 77data to establish disease progression models. In 2014, EMA also launched 78the Adaptive Licensing Pilot to assess the feasibility of using 79observational study data to assist decision-making. Later in 2016, the 80“Scientific Guidance on Post-authorisation Efficacy Studies”was 81released.82At the International Council for Harmonisation of Technical 83Requirements for Medicinal Products for Human Use (ICH), Japan’s 84Pharmaceuticals and Medical Devices Agency (PMDA), proposed a 85strategic approach for pharmacoepidemiology studies submitted to 86regulatory agencies to advance more effective utilization of real-world 87data.88The systematic use of real-world evidence to support drug 89development and regulatory decision-making in China is still under 90development. However, the national drug regulatory agencies have 91already begun to utilize real-world evidence in the review practices. For 92example, the extended Bevacizumab treatment regimen in combination 93with platinum-based chemotherapies was approved in 2018, using 94real-world evidence from three retrospective studies. In another case, a 95drug was further evaluated, after marketing, through a prospective, 96observational real-world study to provide additional evidence on efficacy 97and safety.982.Relevant Definitions of Real-World Research99Generally speaking, real-world research includes both research on 100natural populations and on clinical populations; the latter yields 101real-world evidence that can be used both to support medical product 102development and regulatory decisions, as well as for other scientific 103purposes. For that reason, this guidance focuses on real-world research 104that supports healthcare product development and regulatory decisions 105(see figure below).106107Figure 1 The path from RWD to RWE, which supports regulatory108decisions for medical products109We define real-world research as: collecting patient-related data in a 110real-world environment (real-world data), and obtaining clinical evidence 111(real-world evidence) of the value and potential benefits or risks of the 112medical products through analysis. The primary research type is 113observational, but it can also be pragmatic clinical trials.1141.Real-World Data115（1）D efinition116Section 505F (b) of the Federal Food, Drug, and Cosmetics Act (FD&C 117Act) defines real-world data as "data regarding the usage, or the potential 118benefits or risks, of a drug derived from sources other than traditional 119clinical trials". In “Framework for FDA’s Real-World Evidence Program”120and the “Use of Real World Evidence to Support Medical Device 121Regulatory Decisions.", the FDA defines real-world data as "data relating 122to patient health status and/or the delivery of health care routinely 123collected from a variety of sources”. For example, Electronic Health 124Record (EHR) data, Electronic Medical Record (EMR) data, medical 125insurance data, product and disease registry data, patient report data 126(including home environment), and other health tests (such as mobile 127devices) data.128We define real world data as: data collected from patients’ 129medications and health status, and/or derived from various daily medical 130processes.131（2）S ource of real-world data132Common sources of real-world data in China include:1331) Health Information System (HIS): similar to EMR/HER, digital 134patient records including structured and unstructured data fields, such as 135patient demographics, clinical characteristics, diagnosis, treatment, 136laboratory tests, safety and clinical outcomes.1372) Medicare system: structured data such as basic patient 138information, medical service utilization, prescriptions, billing, medical 139claims, and planned health care.1403) Disease Registry System: a database of patients with specific 141(usually chronic) diseases, often derived from a cohort registry of the 142disease population in the hospital.1434) China ADR Sentinel Surveillance Alliance (CASSA): the use of 144electronic data from medical institutions to establish an active monitoring 145and evaluation system for the safety of drugs and medical devices.1465) Natural population cohort database: the (to be) established 147natural population cohort and special disease cohort database.1486) Omics-related databases: databases that collect information on 149the physiology, biology, health, behavior, and possible environmental 150interactions of patients, such as pharmacogenomics, metabolomics, and 151proteomics.1527) Death registration database: a database formed by death 153registries jointly confirmed by hospitals, centers for disease control and 154prevention (CDC), and department of household registration.1558) Mobile devices: mobile devices such as wearable devices that 156measure relevant data.1579) Other special data sources: databases created for special purposes, 158such as national immunization program databases.159（3）D ata Quality Evaluation160The quality of real-world data is mainly assessed by its relevance 161and reliability.1621) Relevance: Important relevant factors to assess the suitability of 163real-world data for regulatory use include, but are not limited to:164①the inclusion of important variables and information related to 165clinical outcomes, such as drug use, patient demographic and clinical 166characteristics, covariates, outcome variables, follow-up duration, sample 167size, etc.;168②whether the definition of clinical outcome is accurate and the 169corresponding clinical significance is meaningful;170③Accurate and representative definition of target population;171④The study hypothesis can be evaluated through the study 172protocol and statistical analysis plan.1732) Reliability: The reliability of real-world data is mainly evaluated 174by data integrity, accuracy, quality assurance, and quality control.175①Integrity: missing data problems are inevitable in the real-world 176setting, but the amount of missing should have a certain limit. For 177different studies, the degree of missing data may vary. When the 178proportion of missing data within a specific study exceeds a certain limit, 179there is a great deal of uncertainty about its impact on the study 180conclusion. At this time, it will be necessary to carefully assess whether 181the data can be used as real-world data that produce real-world evidence. 182②Accuracy: the accuracy of the data is critically important and 183needs to be identified and verified against authoritative sources of 184reference. For example, the measurement of blood pressure requires the 185use of a calibrated sphygmomanometer, for which and the measurement 186process is subject to the operating specifications; whether the endpoint 187event is determined by an independent endpoint event committee, etc.188③Quality Assurance: quality assurance refers to the prevention, 189identification, and correction of data errors that occur during the course of 190the research. Quality assurance is closely related to regulatory compliance 191and should run through every aspect of data management that needs to 192have a corresponding Standard Operating Procedures (SOPs).193④Quality Control: data collection, modification, transmission, 194storage, and archiving, as well as data processing, analysis, and 195submission, are all subject to quality control to ensure that the real-world 196data are accurate and reliable. It is necessary to develop a complete, 197normative and reliable data management process or protocol.198（4）D ata criteria199Data standards, in the form that information technology systems or 200scientific tools can use, help ensure that the submitted data are predictable 201and consistent. In order to manage real-world data from multiple sources, 202it is necessary to convert the data into a common format with a generic 203formulation (e.g., terminology, vocabulary, coding scheme, etc.).204In addition, whether the quality of real-world data can support drug 205development depend on key factors including (but not limited to): 206whether there is a clear process and qualified personnel for data 207collection; whether a common defining framework, i.e., the data 208dictionary, is used; whether the common time frame for key data points 209collection is followed; whether a study plan, protocol and/or analysis plan 210related to the collection of real-world data have been established; whether 211the technical approach used for data element capture, including 212integration of data from various sources, data records of drug use, links to 213claims data etc., is adequate; whether patient recruitment minimizes the 214bias and reflects the true target population; whether data entry and 215transfer are useable and timely; and whether adequate and necessary 216patient protection measures such as patient privacy protection and 217regulatory compliance with informed consent are in place.2182.Real-World Evidence219Real-world evidence is clinical evidence about the use and potential 220benefits or risks of medical products, obtained through the analysis of 221real-world data. This definition is not limited in concept to obtaining 222evidence through retrospective observational studies, but also allows 223prospective access to a wider range of data to form evidence, through 224particular study designs including pragmatic clinical trials (PCTs).2253.Scenarios where real-world evidence supports drug 226development and regulatory decisions227Real world evidence may support drug development through a variety of 228ways, covering pre-marketing clinical development and post-marketing 229evaluation. Any use of real-world evidence for the purpose of product 230registration will require adequate communication in advance with231regulatory authorities to ensure alignment on the study objectives and232methodology.2331.Treatment for rare diseases234In addition to the challenges in subject recruitment, clinical trials for 235rare disease also face difficulties in the choice of control arm, given the 236few or lack of treatment options. Therefore, external controls established 237based on real world data in natural disease cohorts can be considered.238External controls are primarily used for non-randomized single-arm 239trials, as a historical or in-parallel control. Historical external controls are 240based on real-world data obtained earlier; parallel external controls are 241based on data from disease registries constructed simultaneously with the 242single-arm trial. The use of external controls should take into account the 243impact of the heterogeneity and comparability of the target population on 244the corresponding real-world evidence.2452.Revision of indications or drug combination labeling246For drugs that are already marketed, long-term clinical practice may 247find it necessary to expand the indication, and RCTs are often utilized to 248support the indication expansion. When an RCT is not feasible or when 249evidence it generates is not optimal, a PCT could be a reasonable choice. 250For example, clinical practice may find that a new drug for diabetes can 251potentially benefit patients with cardiovascular diseases (such as heart 252failure). In that case the subject recruitment into an RCT will be difficult 253with potential ethical issues and therefore the use of a PCT design may be 254more feasible.255In terms of pediatrics medication, there are often cases of off-label 256usage in clinical practice. For that reason, the use of RWE in supporting 257the expansion of targeted population is also a viable strategy in drug 258development.259A typical use of real-world evidence to support the development of 260Bevacizumab, a humanized monoclonal antibody of the vascular 261endothelial growth factor (VEGF). In 2015, Bevacizumab was approved 262in China in combination with chemotherapy (carboplatin and paclitaxel) 263for the first-line treatment of late stage unresectable advanced, metastatic 264or recurrent squamous non-small cell lung cancer. However, the 265real-world use of chemotherapy with Bevacizumab also includes 266Pemetrexed in combination with platinum, Gemcitabine and Cisplatin. In 267October 2018, Bevacizumab was approved to expand the treatment 268regimen with a combination of platinum-based chemotherapy, based on 269the strong supporting evidence from three real-world studies. These 270studies retrospectively analyzed patient data from three hospitals and 271showed that the combination of Bevacizumab with platinum-based 272chemotherapy significantly prolonged PFS and OS compared with 273chemotherapy alone, and no new safety issues were identified. This 274finding was consistent with global population data. In addition, relevant 275real-world studies have also provided data in different patient subgroups 276such as those with EGFR mutations or brain metastases, confirming the 277efficacy and safety of Bevacizumab combination therapy from multiple 278perspectives.2793.Post-marketing evaluation280Due to factors such as limited sample size, short study duration, 281strict enrollment criteria, and standardization of intervention, drugs 282approved based on RCTs usually have limited safety information, lack of 283generalization of efficacy conclusions, less optimal drug regimen, and 284insufficient health economic benefits. As a result, there is a need to use 285real-world data for more comprehensive assessment of these aspects of 286the approved drugs, and to refine the decision making based on the 287real-world evidence from natural populations on a continuous basis.288For example, a drug for cardiovascular diseases has been approved 289in more than 50 countries/regions worldwide. In the multi-regional 290clinical trials that supported it approval, small number of Chinese 291subjects resulted in limited number of cardiovascular events and short 292drug exposure in the Chinese subgroup. This has led to greater variability 293in the efficacy results in the Chinese population. As an overseas marketed 294drug with clinically urgent needs in China, to further evaluate the efficacy 295of this compound in Chinese patients, the applicant plans to conduct a 296prospective, observational, post-marketing real-world study to evaluate 297the combination of the compound with standard treatment versus standard 298treatment alone, in the prevention of major adverse cardiovascular events 299(MACE) in Chinese patients with cardiovascular disease.3004.Clinical development of traditional Chinese medicine hospital 301preparations302Traditional Chinese medicine prepared and used in hospitals have 303been widely used clinically for a long time without being approved for 304marketing. This is a unique phenomenon in China. For the clinical 305research and development of such drugs, if real-world research and 306randomized controlled clinical trials can be combined, scientific and 307feasible clinical R&D and regulatory decision-making pathways can be 308further explored.309For the development of traditional Chinese medicine hospital 310preparation, there exist multiple R&D strategies that utilize real-world 311evidence. Figures 2 and 3 outline two potentially possible pathways. The 312pathway that combines observational studies and RCTs is illustrated in 313Figure 2. Specifically, stage 1 starts with retrospective observational 314studies. At this stage effort should be made to collect as much as possible 315existing real-world data related to the use of the product including all 316possible covariates, develop data cleaning rules, identify possible controls, 317assess data quality, and conduct comprehensive and detailed analyses 318using appropriate statistical methods. If the retrospective observational 319studies show that the drug has potential benefits for patients in clinical 320use, it may proceed to the next stage of the development, otherwise the 321process should be terminated. In stage 2, prospective observational 322studies can be conducted. Based on the stage 1 research, this second stage 323can be more carefully designed in terms of several aspects, including data 324acquisition and its system, data quality control, data cleaning rules, and 325clearer definition of controls. Once this stage 2 prospective observational 326research has progressed to certain phase, and if the data are consistent 327with the results of stage 1 retrospective observational studies by 328continuing to show clinically meaningful benefits, a third stage of RCT 329can be conducted in parallel. If needed, a pilot RCT may be conducted 330first to acquire sufficient information to support the design of the primary 331RCT. However, if existing evidence from previous observational studies 332is deemed sufficient, a confirmatory RCT may be designed and 333conducted directly. In terms of timing, the duration of the RCT may be 334covered by the stage 2 prospective observational studies, which can be 335completed at the same time as the RCT or extended for some time after 336the end of the RCT, depending on the maturity of the real-world 337evidence.338339Figure 2 Potential development pathway for traditional Chinese medicine 340hospital preparations341Another potentially possible pathway, which combines observational 342studies with PCTs, is outlined in Figure 3. In the first stage, retrospective 343observational studies are conducted first. If it is concluded that the drug 344has potential benefits in clinical practice, it may proceed to the second 345stage, otherwise the process should be terminated. The second offstage 346consists of a PCT research, which provides evidence that can be used to 347support the evaluation of the drug’s clinical efficacy and safety.348349Figure 3 Potential development pathway for traditional Chinese medicine 350hospital preparations3515.Guiding clinical trial design352Compared with other potential applications, using real-world 353evidence to guide clinical trial design has more practical utilization. For 354example, the two potential pathways for the development of 355hospital-prepared traditional Chinese medicines described in the previous 356section have used the real-world evidence generated by retrospective 357observational studies, including for example the disease natural history, 358the disease prevalence in the target population, the effectiveness of 359standardized treatments, and the distribution and variation of key related 360covariates, to provide a basis for the next stage study design. More 361generally, real-world evidence can provide valid reference for inclusion 362and exclusion criteria, parameters for sample size estimation, and 363determination of non-inferiority margins, etc.3646.Identify the target population365Precision medicine aims to better predict the therapeutic benefits and 366risks of drugs to specific populations (subgroups), and real-world 367evidence based on real-world data provides the possibility for the 368development of precision medicine. For example, due to the limited 369sample size, regular clinical trials often ignore or have limited power to 370consider subgroup effects in the research plan. This prevents important 371information on potential treatment responders or high-risk populations 372with serious side effects from being fully recognized. Through a thorough 373analysis of real-world data, the treatment benefits and risks in different 374subgroups can be more adequately assessed, and hence real-world 375evidence can be obtained to support more precise identification of the 376target population.377The identification of biomarker is critical for preclinical and early 378clinical studies of targeted therapies. Using real-world information such 379as omics data, public gene bank information, and related clinical data in 380population cohorts, real-world evidence can be generated through various 381contemporary data mining techniques such as machine learning, which 382can in consequence support the precise identification of population for 383targeted therapies.3844.The Basics of Real-World Research Design3851.Pragmatic clinical trials386Pragmatic Clinical Trials (PCT), also known as practical clinical 387trials, refer to clinical trials that are designed and conducted in an 388environment close to the real-world clinical practice. They represent a 389type of study between RCTs and observational studies. Unlike RCTs, 390PCT interventions can be either standardized or non-standardized; 391subjects in the PCTs can be randomized or allocated per pre-defined 392criteria; the inclusion criteria for the subjects are often less restrict and 393considered more representative of the target population, and the 394evaluation of intervention outcomes may not be limited to clinical 395。
