Editorial Review

Observational studies: Are they significant in the current evidence-based trial era?

S Chandrashekara¹, R Misra²

Author Affiliations

¹Managing Director and Consultant Rheumatologist, Chanre Rheumatology and Immunology Center, Basaweshwaranagar, Bangalore, India

²Professor and Head of Clinical immunology, Sanjay Gandhi Post Graduate Institute of Medical Sciences, Lucknow-226 014, India

IJRCI. 2013;1(S1):SR1

Received: 30 October 2013, Published: 1 November 2013

2013

© IJRCI

Introduction

In the current era of evidence-based medicine, a comprehensive range of well-planned studies plays a crucial role in making prescribing decisions and developing effective patient management strategies. Clinical trials are significant to prove the safety and efficacy of treatments, whereas observational studies provide valuable additional data on how these therapies perform in real-life scenarios. This article focuses on the significance, advantages, and limitations of observational studies. The authors have also tried to explain how well-planned observational studies may help in obtaining the desired outcome by citing relevant examples from rheumatology clinical practice and research.

Defining observational studies

As per the definition put forth by Cochran, ‘Observational study is an empiric comparison of treated and control groups in which: the objective is to elucidate cause-and-effect relationships [. . . in which it] is not feasible to use controlled experimentation, in the sense of being able to impose the procedures or treatments whose effects it is desired to discover, or to assign subjects at random to different procedures’.¹

During observational studies, the researchers directly examine and document the events in clinical settings without relying on experimental interventions. Such studies are mainly employed for the characterization of prognostic indicators and risk factors, and also in scenarios where randomized controlled studies are not feasible or ethical.² Choice of the appropriate observational study type is based on the timeline, data needed, and/or practical aspects of collecting data. Researchers are increasingly using observational studies to estimate the effect of treatment outcomes.³ Observational studies are useful in investigating a wide range of objectives such as identifying the disease etiology, verifying the previously reported associations, or obtaining biologic insight into disease pathogenesis. Such studies also hold greater significance in confirming or refuting initial clinical observations or ideas obtained from routine practice. In certain instances, specifically designed observational studies are carried out to overcome potential limitations of previous studies. These studies may also help to gather newer data and to obtain various viewpoints for e.g. the benefits of evaluating subgroups or the significance of a predetermined sample size.⁴

There are three main types of observational study designs that are distinguished by the objective of the research study, how subjects are sampled, and the timeline of data collection. The three types are cross-sectional study, case-control study, and cohort study. The table 1 provides a comparison of these study categories.⁵

Table 1: Comparison of the three main observational study types

Variables	Case-control studies	Cross-sectional studies	Cohort studies
Design	The association between exposure and a disease of interest, and other covariates are evaluated at a given point of time in the sample cohort selected from the targeted population.	The association of the disease and exposure is investigated in case and control groups chosen from the same population.	The assessment includes the subsequent development of an outcome in subjects exposed or not exposed to a factor.
Primary use	Employed to evaluate both exposure and outcome simultaneously mainly for screening hypotheses and assessing prevalence.	Used in cases where matching between the groups is required to assess the association.	Used to investigate the association between exposures and rare outcomes prospectively or retrospectively.
Strengths	Beneficial in studying exposures not changing with time Cost effective and time efficient Employed commonly in generating novel hypotheses for further research Often used before conducting longitudinal study or clinical trial	Can be completed in a short time frame Since the subjects are sampled with regard to their disease status, it is beneficial to study chronic or latent diseases with transient latent periods	Since the exposure is documented before the occurrence of the disease, temporal relation is apparent The effect of multiple exposures on the outcome can be evaluated in a single study It is possible to calculate the disease incidence
Limitations	Unable to identify the temporality of events assessed May not be possible to differentiate between prognostic and etiological roles of the exposure of interest Not suitable for measuring the incidence of rare diseases Length-time bias has to be considered while interpreting the findings	Difficulty in choosing the subjects for both the case and control groups from the same study base Finding a good match control is a time consuming and expensive procedure Recall bias occurring due to the assessment of exposure after the disease occurrence may lead to overestimation of the association If the outcome is known, then observer or interviewer bias can arise	Expensive and time consuming Bias could arise if the exposure results in loss of follow-up in prospective studies Not very useful in evaluating the outcome of rare diseases Retrospective study depends purely on the already collected information. Hence, there is a chance for missing data or records. Observer’s bias may be more significant if the outcome is not clearly described

For observational studies, investigators use data gathered exclusively for the study purpose (primary data) or those collected previously for another study, but it is used in the current observational study to evaluate a novel research query (secondary data). Sources commonly used for retrieving data for such studies include data from medical chart review or previously conducted research studies and hospital administrative databases.⁶ Large-scale observational studies are usually carried out by regulatory organizations for risk assessment/management, pharmacovigilance, and evaluating risk-benefit profiles.

Controlled vs. observational studies

Observational studies are touted to possess several advantages in contrast to controlled and randomized trials such as reduced cost, time adherence, and the ability to evaluate a broad range of subjects. The main concern against the wider use of observation study is that it over estimates the treatments effects compared to randomized controlled trials. A previous research by Concato and colleagues noted that the observational studies possess less heterogeneity in point estimates when compared to randomized, controlled trials. A plausible explanation provided by the researchers for this finding is that observational studies usually involve a broad representation of the patient population. Moreover, there is less chance for differences in patient management. On contrary, experimental trials may have a distinct patients group owing to the precise exclusion and inclusion criteria, and the investigational therapy protocol may not represent a treatment in practice.⁷

However, comparison of the results of the study types by Benson et al. concluded that there is lack of evidence to validate that the treatment effect estimates reported in observational studies from 1984 were varying significantly with that of experimental studies.²

Systematic differences in both measured and unmeasured baseline characteristics between treated and untreated subjects is considered as one of the major limitations of observational studies.⁸ Therefore, it is crucial to adjust such variations using statistical methods while using observational data for evaluating the effect of treatment on outcomes.

Meticulous patient selection and careful study designing are the other criteria that should be considered while conducting observational studies. In well-planned observational studies, there are fewer chances for differences in patient management. On contrary, experimental trials may have a distinct patients group owing to the precise exclusion and inclusion criteria. For instances, in RA trials, inclusion of patients non-responsive to only methotrexate will be technically different from patients who were exposed and non-responsive to multiple DMARDs. In an observational study, where such specific inclusion criteria are not considered, the possible group of patients included will be broader representatives of the subjects seen in real-time clinical practice. Experimental studies are highly impracticable in certain circumstances owing to the increased cost, complex ethical issues, and difficulties associated with patient enrollment.⁹ Due to these reasons, observational studies are increasingly used by investigators as supportive data to develop clinical trial protocols and policy statements.¹⁰

In addition, large observational or case control studies are reported to be ideal for evaluating adverse events, as the patients will be selected randomly without knowing the risk of adverse outcome. In such cases, clinical trial is not preferred as the follow-up period is too short for evaluation. The real-time clinical issues are often not likely be addressed by the randomized clinical trials. For instances, the dose adjustment based on the occurrence of adverse events. In the management of RA, there is no clear consensus on dose adjustments for methotrexate, the widely used DMARD, upon elevation of liver enzyme levels. Observational studies are useful in exploring the effects of a wider range of exposures such as prevention, treatments, and possible disease causes. The information obtained from such studies may help explaining the causes of disease incidence, identifying the determinants of disease progression, to foresee the future healthcare requirements of a population, and to develop strategies to prevent or control disease.⁹

Observational studies are preferred for conducting most of the epidemiologic research owing to the certain limitations of experimental studies such as their inapplicability in certain scenarios, difficulty in enrolling subjects, increased expenses, and stringent ethical issues.⁹ For example, in order to evaluate the association of lifestyle factors like smoking on the development of RA, it is neither possible nor ethical to conduct the study by exposing the subjects to smoking. Suitable use of observational studies may aid in investigating incidence prevalence, associations, causes, and outcomes.¹¹

Table 2: Advantages and disadvantages of observational studies

Advantages

Disadvantages

Easy, less expensive and they can be performed within the natural clinical setting Denotes a real-world clinical setting Ethical issues related to the manipulation of study variables can be avoided Behavior of the participants could be directly observed Easy to carry out in normal clinical care settings

Bias in selection of patients and treatment regimen, since it is often guided by circumstance rather than specified guidelines. Difficulty in reproducing the study results Inability to obtain information regarding the subjects’ emotional status or thoughts Since the study is not blinded, there is increased chance for observer bias. However, this bias does not occur, in scenarios where observer cannot influence the study objective; e.g. smoking habit, clearly defined diagnostic criteria. Cause and effect statements cannot be derived due to the inability to control extraneous variables Requirement of extensive time and trained investigators as observers

Major limitations of observational studies

In spite of the careful study design, implementation, and analysis; there is an increased possibility of random variation (chance) or bias (selection, information) or confounding to influence the observational study results. If the patients are not randomized to corresponding groups, or if the treatment is selected by the subjects themselves or if it is imposed by their environment, the variation in the study outcomes observed may indicate the initial differences than the actual treatment effects. These selection biases or pretreatment differences can be classified as ‘Overt biases’ (which can be measured precisely and controlled) and ‘Hidden biases’ (that are speculated to exist, but cannot be measured or called random bias). The two key challenges associated with observational studies are eliminating overt biases and resolving the uncertainty linked to hidden biases. Skowronski and coworkers indicated that maintenance of consistency in study results across various populations and study designs, and larger sample sizes may help in limiting the random variation to some extent. Whereas, the researchers reported that, owing to the possibility of occurrence of bias or confounding, it is difficult to arrive at a conclusion on previous observational studies, which had suggested that seasonal vaccination increases the risk of contracting pandemic influenza A (H1N1).¹²

In contrast to experimental trials, observational studies help in evaluating the association of events and in providing assessments in the natural settings.¹¹ However, it may not be feasible to eliminate the source of biases in observational studies using the standard analytic techniques.¹³ Inherent bias associated with observational studies restricts its application in treatment comparison. In this regard, controlled studies are more reliable.² Furthermore, accounting of various confounding factors such as age, sex, and race during the study design is possible in experimental trials.

The influence of pre-selection bias is suspected in a study by Bhardwa and Harron (2005).The study reported an increased prevalence of association between steroid use and the development of extra-articular features.¹⁴ Although, these finding are statistically true, pre-selection bias is speculated considering steroid as a commonly prescribed RA treatment in severe cases and the frequent occurrence of extra-articular manifestations. To exclude pre-selection biases, it is advocated to perform patient regrouping based on the severity features of RA, prior to the reanalysis of the data. The association needs to be considered as significant, if it holds good even after the regrouping or adjusting for risk-associated factors. However, it may not be feasible in such observational studies, since the number of patients classified into severe disease but not on steroid will be relatively small. Hence, adjusting of key variables is significant to ascertain the comparability between cohorts, while evaluating the association between a specific treatment or exposure and an outcome using observational studies.

Even after adjusting for the confounding factors and the non-random allocation of the treatment, there is an increased possibility for all the study subjects having a specific type of treatment. The mechanism of treatment assignment followed in most of the observational studies is without evaluating the subsequent data analyses. Hence, there is an increased probability that both measured and unmeasured covariates may affect the study outcome.

Selection bias: The major drawback of observational studies

Selection bias due to the lack of randomization has been identified as another major limitation of observational studies. Although the selection bias cannot be eliminated fully, several strategies are employed to restrict or reduce it such as propensity analysis, instrumental variables, and risk adjustment through regression or analysis of variance methods.

The propensity score method involves the substitution of all confounding covariates with one single function called propensity score and to use the same as sole confounding covariate. Another way of performing propensity analysis is by developing a logistic regression model considering choice of treatment as the binary dependent variable and the characteristics related to treatment of choice as the independent variables.¹⁵ However, the conclusions drawn from observational studies employing propensity score is not as significant as those obtained from randomized control trials, owing to the inability to adjust for unknown or unmeasured confounding variables.¹⁶

Propensity score method can be used in observational analyses to reduce the influence of treatment-selection bias. Using propensity score for evaluation of absolute risk reduction and the number needed to treat (NNT) will help in deriving more clinically significant results. Multivariable models are commonly used in observational research to assess the relationship between a certain exposure or treatment and an outcome, while adjusting for important variables necessary to ensure comparability between the groups.¹³

Interference of several biases may limit the validity of data. Hence, it is crucial to remember the fact that many research questions on treatment and other related issues can be answered only by meticulously designed and executed observational studies.

Planning and writing observational studies

Appropriate designing as well as efficient reporting is crucial, while conducting an observational study. The Meta-analyses of Observational Studies (MOOSE) checklist for authors, editors, and reviewers highlighted the significance of including 35 necessary components, while preparing the manuscript or planning an observational study. The checklist is given below: ^{17, 18}

Table 3: The MOOSE checklist for planning and writing observational studies

Titles	Components
Background	Problem definition Hypothesis statement Type of exposure or intervention used Type of study designs used Study population
Methods	Description of relevance or appropriateness of subjects and methods in assessing the hypothesis to be tested Rationale for the selection and coding of data (e.g. sound clinical principles or convenience) Documentation of data classified and coded (e.g. multiple raters, blinding, and interrater reliability) Assessment of confounding variables (e.g. comparability of cases and controls in studies where appropriate) Assessment of study quality, including blinding of quality assessors; stratification or regression on possible predictors of study results Assessment of heterogeneity Description of statistical methods (e.g. complete description of fixed or random effects models, justification of whether the chosen models account for predictors of study results, dose-response models, or cumulative meta-analysis) in sufficient detail to be replicated
Results	Provision of appropriate tables and graphics Graphic summarizing individual study estimates and overall estimate Table giving descriptive information for each study included Results of sensitivity testing (e.g. subgroup analysis) Indication of statistical uncertainty of findings
Discussion	Quantitative assessment of bias (e.g. publication bias) Justification for exclusion (e.g. exclusion of non-English-language citations) Assessment of quality of included studies
Conclusion	Consideration of alternative explanations for observed results Generalization of the conclusions (i.e. appropriate for the data presented and within the domain of the literature review)
Guidelines for future research
Disclosure of funding source

A brief description on some of the major components of the checklist is provided below:

Background

The theoretical framework and the hypothesis to be tested through observational study should be precise and clear. This assists in meticulous planning of the study. Exploratory research is generally employed if there is little or limited information is available about the problem. For example, in order to evaluate the strategies to be used to measure the day-to- day pain and symptoms in RA patients when they stay out of the hospital, exploratory research should be performed to figure out how patient might identify and simultaneously document their pain within the scope of their understanding. Such exploratory research will help in gathering data, and creating initial hypothesis and theories on how best the measures of RA activity can be captured on daily basis. The primary objective of exploratory research is to provide researchers relevant information and to assist them in framing initial hypothesis on the topic of evaluation.

Descriptive study is done with a specific research query in mind. For instance, in a survey analyzing the utilization and validation of questionnaire, specific information captured using diaries, pictures, or internet-based questionnaires can be represented as descriptive data or quantitative data. Information such as age, gender breakdown, and the literacy status of patients, which influence the accuracy of the recording and their preference of methods is a quantitative data; whereas the response to questionnaire and problems faced by different groups, while using different methods is presented as descriptive data. This assists in choosing right instrument and methodology in further research. Thus, a well-planned descriptive research aids in obtaining a clear understanding about research questions, the populations, or the methods of analysis prior to the initiation of the study. Similarly, in scenarios where sufficient theoretical information is available, adequate background work help in planning the study. For example, in order to evaluate the prognostic factors, which can predict the outcome of a disease; it is significant to map the probable candidate that can influence the disease process. For instance, in systemic lupus erythematosus (SLE), C-reactive protein (CRP) may not be a right candidate to choose, since the changes in CRP levels do not clearly reflect the disease process. But the protein may serve as good marker for infection outcome in SLE. The hypothesis and the theoretical framework should be conceived after a careful literature search, and if possible, following a careful pre-study observations. Thus the objectives should be precise and focused on the hypothetical questions.

Descriptive observational studies, data of rare or even common diseases or audited reports with extensive perspective may have broader objectives on the basis of the nature of the studies. The aim and scope of such studies should be defined clearly. In order to analyze the clinical as well as the laboratory features of a specified disease, for example male SLE, the study objective should be clear and specific. If the objective of the study is vague, it may be difficult to achieve the desired results. For instance, if the objective of a study is to compare male and female lupus, and to distinguish features between the two groups, the specific objective, i.e. the incidence of nephritis, should be evaluated through non-descriptive research and the differences in the laboratory and clinical features as descriptive study.

The selection bias can be generally avoided by enrolling all eligible patients within a specified time interval, not influenced by investigator bias. For example, if the patients admitted due to SLE is only included in a study, there will be a bias, as most of the subjects will be having more aggressive disease.

Methods

After careful consideration of hypothesis or research question to be addressed, a greater emphasis should be given to study methods. In any human-based studies, selection of eligible and appropriate number of subjects is significant. This can be achieved by proper understanding of hypothesis and possible variables associated with the subject selection. For instance, if the study is proposed to investigate the impact of socio-economic status on the outcome of SLE, the study population should represent subjects belonging to all socioeconomic levels. The prognosis of SLE in subjects belonging to the black race is reported to be poor. If the number of black subjects likely to be recruited is less because of geographic location of the study, it is preferable to exclude such patients. If the recruited subjects do not belong to socio-economic groups, the patient data obtained may skew the findings. Hence, it could be construed that drawing a clear definition of inclusion and exclusion criteria, at the very beginning of study, can eliminate bias related to subject selection to a major extent. Period of recruitment and recruitment plan may also influence the study success.

In an observational study, the occurrence of hidden biases remains a major challenge, since the interventions are not chosen randomly. Planning an early arthritis study in a tertiary care center to estimate the community prevalence will not be relevant, since it will invariably have a referral bias. The data proposed to be obtained through the study should be useful for evaluating the hypothesis. In addition to the capturing of relevant data with regard to the study, the other possible variables, which are likely to influence the observation either in the form of bias or covariate should also be obtained. An appropriate statistical method is one of the key to derive the correct inference in any study and to justify or discard a hypothesis.

Reporting an observational study

The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) initiative have put forth an extensive guideline for the accurate and complete reporting of observational studies. The recommendations for all the three main observational study types could be accessed at : von Elm E, Altman DG, Egger M, Pocock SJ, Gøtzsche PC, et al.. (2007) The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) Statement: Guidelines for Reporting Observational Studies. PLoS Med 4(10): e296. 19

Future perspectives

Well-designed, non-experimental studies that mimic clinical trials do not persistently overestimate the treatment effectiveness. In early phase of drug evaluation, randomized controlled trials can yield reliable findings. However, it is recommended to consider observational study as complementary to resolve challenges in medical practice. Often the journals and peer-reviewing process consider observational studies as inferior to randomized experiments. But a merited consideration will help in adding significant information, especially in real-time clinical management. Additionally, if stringent standards of randomized controlled trials are utilized while conducting observational studies, they will enhance the clarity and strength of evidence. Large-scale observational studies are also best suited for risk-to-benefit analysis, pharmacovigilance, and risk assessment/management.

Statement by Mervyn Susser highlights the importance of observational studies: “Observational studies have a place in the epidemiological armament no less necessary and valid than controlled trials; they take second place in the hierarchy of rigor but not in practicability and generalizability. . . . Even when trials are possible, observational studies may yield more of the truth than randomized trials.”⁹

Competing interests

The author declares that he has no competing interests.

References

1.Rosenbaum PR. Observational studies [Internet]. Springer; 2002 [cited 2013 Apr 8]. Available from: http://www.springer.com/cda/content/document/cda_downloaddocument/9780387989679-c1.pdf?SGWID=0-0-45-101690-p1458307.

2. Benson K, Hartz AJ. A Comparison of Observational Studies and Randomized, Controlled Trials. New England Journal of Medicine. 2000;342(25):1878–86.

3.Austin PC. The performance of different propensity-score methods for estimating differences in proportions (risk differences or absolute risk reductions) in observational studies. Stat Med. 2010 Sep 10;29(20):2137–48.

4.Vandenbroucke JP, von Elm E, Altman DG, Gøtzsche PC, Mulrow CD, Pocock SJ, et al.. Strengthening the Reporting of Observational Studies in Epidemiology (STROBE): Explanation and Elaboration. PLoS Med. 2007 Oct 16;4(10):e297.

5.Setia M. Observational studies: How to go about them? Indian Journal of Dermatology, Venereology and Leprology. 2008;74(3):288.

6.Carlson MDA, Morrison RS. Study design, precision, and validity in observational studies. J Palliat Med. 2009 Jan;12(1):77–82.

7. Concato J, Shah N, Horwitz RI. Randomized, Controlled Trials, Observational Studies, and the Hierarchy of Research Designs. New England Journal of Medicine.  2000;342(25):1887–92.

8. Austin PC. Optimal caliper widths for propensity-score matching when estimating differences in means and differences in proportions in observational studies. Pharm Stat. 2011 Mar;10(2):150–61.

9. Aschengrau A, III GRS. Essentials of Epidemiology in Public Health. Jones & Bartlett Publishers; 2008.

10. Stukel TA, Fisher ES, Wennberg DE, Alter DA, Gottlieb DJ, Vermeulen MJ. Analysis of observational studies in the presence of treatment selection bias: effects of invasive cardiac management on AMI survival using propensity score and instrumental variable methods. JAMA. 2007 Jan 17;297(3):278–85.

11. Mann CJ. Observational research methods. Research design II: cohort, cross sectional, and case-control studies. Emerg Med J. 2003 Jan 1;20(1):54–60.

12.Skowronski DM, De Serres G, Crowcroft NS, Janjua NZ, Boulianne N, Hottes TS, et al.. Association between the 2008-09 seasonal influenza vaccine and pandemic H1N1 illness during Spring-Summer 2009: four observational studies from Canada. PLoS Med. 2010 Apr;7(4):e1000258.

13. Newgard CD, Hedges JR, Arthur M, Mullins RJ. Advanced statistics: the propensity score--a method for estimating treatment effect in observational research. Acad Emerg Med. 2004 Sep;11(9):953–61.

14. Biswas M, Chatterjee A, Ghosh SK, Dasgupta S, Ghosh K, Ganguly PK. Prevalence, types, clinical associations, and determinants of peripheral neuropathy in rheumatoid patients. Ann Indian Acad Neurol. 2011;14(3):194–7.

15. Hannan EL. Randomized Clinical Trials and Observational StudiesGuidelines for Assessing Respective Strengths and Limitations. J Am Coll Cardiol Intv. 2008 Jun 1;1(3):211–7.

16.Trojano M, Pellegrini F, Paolicelli D, Fuiani A, Di Renzo V. observational studies: propensity score analysis of non-randomized data. Int MS J. 2009 Sep;16(3):90–7.

17. Bhandari M. Advanced Concepts in Surgical Research. Thieme; 2012.

18. Sanderson S, Tatt ID, Higgins JPT. Tools for assessing quality and susceptibility to bias in observational studies in epidemiology: a systematic review and annotated bibliography. Int J Epidemiol. 2007 Jun;36(3):666–76.

19. Von Elm E, Altman DG, Egger M, Pocock SJ, Gøtzsche PC, Vandenbroucke JP. Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement: guidelines for reporting observational studies. BMJ. 2007 Oct 20;335(7624):806–8.