4 Crucial challenges in your real-world evidence (RWE) study

4 Crucial challenges in your RWE study - and how to overcome them

Planning and conducting a real-world evidence study can be very challenging. In this article, SDS consultant Dr. Robert Szulkin breaks down four crucial challenges when conducting an RWE study.

The first step, that is more difficult than it sounds, is to specify valid research questions for the problem at hand. These questions are then translated into statistically testable hypotheses.  To test these hypotheses, the next step is to choose an appropriate study design, analysis data from the health registers, and correct statistical methods. In this article, we list four crucial challenges and how to overcome them when conducting a successful RWE study. 

1. Choosing the correct study design

Depending on the research question, choosing the correct study design can be quite challenging. In general, a prospective follow-up study is preferable due to showing higher scientific evidence than other study designs. In so-called cohort studies individuals are followed prospectively from the time they become exposed (for example, get ill or start a new treatment) until they experience an event of interest (for example, recovery or death). However, sometimes it is more feasible to use other study designs. For example, a retrospective case-control study can be useful for studying incidence of a rare disease.

It may also be beneficial to use either a prospective or retrospective matched cohort study design, to have comparable groups of individuals if the research question is about comparing outcomes (such as mortality rates). This usually requires careful consideration and choosing the correct matching method so that the groups are matched on the right set of variables. Otherwise, the groups may become incomparable, or even worse, the groups become so equal (“overmatched”) that it is no longer possible to assess the relative difference between the groups.

Furthermore, planning a RWE study usually requires a combination of clinical understanding of the disease in question with underlying biological mechanisms as well as good knowledge of epidemiological study designs and statistical methods. Therefore, to conduct a successful RWE project, it is vital to have a broad range of competencies and a good collaboration between clinicians and statisticians.

2. Using the right data

In the RWE studies, patients are followed in the real clinical world. More specifically, their health-related data, known as real world data (RWD) appear in different electronically generated health-records registers, such as hospital registers, national healthcare, and quality registers, etc. It is important to evaluate properly which type of data is needed for a specific research question. A population-based national register may be required if the study needs to be representative of the whole population. However, the national registers are not always as detailed as one would wish. Hence, it may be preferable to have data from smaller clinics, quality registers or university research data. A great advantage in Sweden and other Nordic countries is the possibility to link all registers via the national identification numbers.

Where to find the right data 

There are numerous data sources in the world that could be used to address RWE problems. Unfortunately, registers are not standardized which means that they are all different in how they are organized and which information they contain. Furthermore, the register holders have different cumbersome application processes for researches seeking to apply for the register data. Therefore, a good comprehensive study plan that includes the correct study design and a well-outlined statistical analysis plan, along with a clear ethical application is a good start. For this purpose, it is essential to work in close collaboration with a RWE project manager who is experienced in interacting with the register holders. This can save both time and money for a project as any essential details that were not specified in the application, will require a supplementary application, which can take several months to be processed.


At SDS Life Science, our several consultants have previously worked directly with the Swedish governmental authorities who are also the register holders, such as the Regional Cancer Centre (Regionalt cancercentrum), the National Board of Health and Welfare (Socialstyrelsen) and the Public Health Agency (Folkhälsomyndigheten). Furthermore, we have a close collaboration with several university research groups, and we often work as a direct link between our academic and industry partners.  

3. Confounding – a crucial challenge in RWE!

A clinical trial comparing treatment A to treatment B has the advantage of allocating individuals randomly (e.g. by tossing a coin) to the treatment groups. Another appealing feature of the randomization is that it balances the two treatment groups on underlying factors, i.e. groups A and B will have the same age and gender distribution, equal disease severity, etc. Of course, if chance would have it, the groups may end up being unbalanced especially if the study is small. Nevertheless, if in a randomized clinical trial, individuals receiving treatment A show a 30% reduction in the progression of the disease compared to individuals with treatment B within the given period, then the conclusion is that treatment A is better since the two treatment groups are comparable. However, in the “real world”, such results can hardly apply, as the groups would not be comparable. This is due to a multitude of factors. In particular, in regular clinical practice, the choice of treatment is not random. Most often, it is based on the assessment of the treating doctor (in agreement with the patient) according to the disease severity, the patient’s general condition, concomitant medications, etc.

Overcoming issues with confounding

The imbalance between treatment groups leads to a systematic error (bias) which, somewhat simplified, in epidemiological terms is described as confounding. Proper adjustment for important variables (confounders) is fundamental in a RWE analysis. Directed Acyclic Graphs (DAGs) are helpful tools for this purpose. To minimize residual confounding, it is crucial at the planning stage to decide on the set of variables required for the analysis. If the information regarding essential confounders is not collected, then it will not be possible to adjust for them in the statistical models. Hence, an appropriate data source should be chosen accordingly.

4. Choosing the right statistical method

Proper adjustment for confounding is central in the choice of statistical model in the RWE studies. Another difficulty that often arises in “the real world” is that patients switch treatments. For example, a patient can start with treatment A, then get a side effect, and therefore, switch to treatment B (in agreement with a doctor), and finally, switch back to treatment A (which is more effective). To draw conclusions about the treatment effect in such situations is more challenging than in a clinical trial, where patients are expected to follow a clinical protocol more stringently during a specified follow-up time.

Choose methods for both confounding and time-varying exposures

Fortunately, statistical methods, which can cope with both confounding and time-varying exposures (switching treatments) have been developed for many years within the field of epidemiologic research. Many of the applications stem from a branch of epidemiology and biostatistics, called causal inference. For example, marginal structural Cox models could be applied in studies of comparative effectiveness (treatment effect in the real world). Under certain assumptions, these models estimate valid causal effects, i.e. they mimic clinical trials. 


We use cookies to ensure that we give you the best experience on our website. Read our cookie policy here.