Method evaluation - OHDSI

Method evaluation - OHDSI

Method evaluation Martijn Schuemie Disclaimer I am a full-time employee of Janssen R&D This talk represents my own opinion At my current location it is already Halloween-eve. 2 Horrible news! 3 Evidence Danish study National cardiac arrest registry (n = 28,947) Case-time-control design Ibuprofen cardiac arrest: OR = 1.31 (1.14-1.51)

4 How reliable is this? How reliable is a case-time-control design? How reliable are observational studies in general? What about Confounding? Measurement error? Selection bias? 5 Measuring performance of observational study designs OBSERVATIONAL MEDICAL OUTCOMES PARTNERSHIP Original Originalreason reasonfor fordeveloping developingthe

theOMOP OMOPCDM! CDM! 6 Measuring performance of observational study designs Drug Gold standard Outcome True effect size A X 1.0 B Y

2.0 Methods Case-control Cohort method Are Areestimates estimatesininline linewith withthe thetrue trueeffect effectsize? size? Databases Database 1

Database 2 OBSERVATIONAL MEDICAL OUTCOMES PARTNERSHIP 7 Issues with previous experiments Positive controls: Effect size not known with certainty Known to doctors Methods: Cohort method didnt use survival model Self-controlled cohort didnt truncate to observation period MSCCS applied shrinkage to drug of interest

8 OHDSI Methods Benchmark Real negative controls (n = 200) Pick 4 outcomes and 4 exposures of interest Acute pancreatitis Diclofenac GI bleeding Ciprofloxacin Stroke Metformin IBD Sertraline Use LAERTES to identify potential negative controls Use clinicaltrials.gov + ATC to find potential comparator exposures Rank by prevalence Manual review, up to 25 per outcome or exposure of interest Target Eszopiclone Eszopiclone Eszopiclone Eszopiclone Ciprofloxacin

Ciprofloxacin Ciprofloxacin Ciprofloxacin Comparator Triazolam Triazolam Triazolam Triazolam Azithromycin Azithromycin Azithromycin Azithromycin Nesting Insomnia Insomnia Insomnia Insomnia Otitis media Otitis media Otitis media Otitis media Synthesized positive controls (n = 600)

Based on real negative controls (where true RR = 1) Fit predictive models for each outcome in exposed population Sample simulated additional outcomes during exposure based on predicted probability until True RR = desired RR (1.5, 2, and 4) Outcome True effect size Acute pancreatitis 1 Acute pancreatitis, RR=1.5 1.5 Acute pancreatitis, RR=2 2 Acute pancreatitis, RR=4 4 Alcohol abuse 1 Alcohol abuse, RR=1.5 1.5 Alcohol abuse, RR=2 2 Alcohol abuse, RR=4 4

9 OHDSI Methods Benchmark Can Canbe beused usedboth bothfor for Effect estimation: Effect of Target on Outcome Effect estimation: Effect of Target on Outcome Target Eszopiclone Eszopiclone Eszopiclone Eszopiclone Ciprofloxacin Ciprofloxacin Ciprofloxacin Ciprofloxacin Comparator

Triazolam Triazolam Triazolam Triazolam Azithromycin Azithromycin Azithromycin Azithromycin Nesting Insomnia Insomnia Insomnia Insomnia Otitis media Otitis media Otitis media Otitis media Outcome True effect size Acute pancreatitis 1 Acute pancreatitis, RR=1.5 1.5

Acute pancreatitis, RR=2 2 Acute pancreatitis, RR=4 4 Alcohol abuse 1 Alcohol abuse, RR=1.5 1.5 Alcohol abuse, RR=2 2 Alcohol abuse, RR=4 4 10 OHDSI Methods Benchmark Can Canbe beused usedboth bothfor for Effect estimation: Effect of Target on Outcome Effect estimation: Effect of Target on Outcome

Comparative effect estimation: Effect of Target on Outcome compared to Comparative effect estimation: Effect of Target on Outcome compared to Comparator Comparator Target Eszopiclone Eszopiclone Eszopiclone Eszopiclone Ciprofloxacin Ciprofloxacin Ciprofloxacin Ciprofloxacin Comparator Triazolam Triazolam Triazolam Triazolam Azithromycin Azithromycin Azithromycin Azithromycin

Nesting Insomnia Insomnia Insomnia Insomnia Otitis media Otitis media Otitis media Otitis media Outcome True effect size Acute pancreatitis 1 Acute pancreatitis, RR=1.5 1.5 Acute pancreatitis, RR=2 2 Acute pancreatitis, RR=4 4 Alcohol abuse 1 Alcohol abuse, RR=1.5 1.5 Alcohol abuse, RR=2

2 Alcohol abuse, RR=4 4 11 OHDSI Methods Library Set of R packages Implementing most well-known designs CDM to estimate (+ diagnostics) Are used in OHDSI studies Cohort Method New-user cohort studies using large-scale regression for s propensity and outcome models

Self-Controlled Case Series Self-Controlled Case Series analysis using few s or many predictors, includes splines for age and seasonality. Self-Controlled Cohort A self-controlled cohort design, where time preceding s exposure is used as control. Case-control Case-control studies, matching controls on age,s gender, provider, and visit date. Allows nesting of the study in another cohort.

Case-crossover Case-crossover design including the option to adjust s for time-trends in exposures (so-called case-time-control). 12 Databases Truven MDCD Truven CCAE ? (US Insurance claims) (US Insurance claims) (Other OHDSI dbs)

13 Metrics Coverage: how often is the truth in the 95% confidence interval? Type 1 error: how often is the null rejected when the null is true? Type 2 error: how often is the null not rejected when the null is not true? 14 Results! http://data.ohdsi.org/MethodEvalViewer/

15 Overall conclusions The good Cohort method using PS matching has high coverage but relatively low power. The bad (Nested) case-control has low coverage, often overestimating the true effect size with very narrow confidence intervals. 16 Evidence Danish study National cardiac arrest registry (n = 28,947) Case-time-control design Ibuprofen cardiac arrest: OR = 1.31 (1.14-1.51) 17

Does Ibuprofen cause heart attacks? I have no idea whether that drug is dangerous or not, But I sure as hell know that particular analysis didnt shed any light on the question. David Madigan in Nature 513, 472 (25 September 2014) 18 Questions? 19

Recently Viewed Presentations

  • A changing approach to geodetic networks and standards

    A changing approach to geodetic networks and standards

    A changing approach to geodetic networks and standards MATT AMOS and DON GRANT Overview Historical Based on method of observation Current Based on accuracy achieved Future NZ Geodetic Datum 1949 Limitations of NZGD1949 "New" technology EDM, GPS, etc Public access...
  • The British Restoration Period 1660-1798

    The British Restoration Period 1660-1798

    The period begins with the RESTORATION of the Monarchy in 1660 bringing Charles II from his exile in France. 1. Brings with him the indulgent and artistic ways of Louis XIV's court ... and small middle-class societies. ... The British...
  • Three Runtimes, one standard… .NET Standard: All in Visual ...

    Three Runtimes, one standard… .NET Standard: All in Visual ...

    Open source momentum "Samsung is embracing .NET because it is a completely open source project." "ASP.NET is open source, that allows us to contribute back to it if we have any performance issues which Microsoft review and together we make...
  • LWW PPT Slide Template Master

    LWW PPT Slide Template Master

    Consensual light reflex. Shine penlight into right eye . Watch left pupil. Normal brisk constriction of both pupils. Abnormal pupillary reactions. Stroke, head trauma, eye injury, artificial eye. Unequal size. Optic nerve trauma. Opposite eye constriction with assessed eye dilation.
  • Complex Equilibria General Approach to solving complex equilibria

    Complex Equilibria General Approach to solving complex equilibria

    Redox titration example. How many grams of oxalic acid should be dissolved in 500.0mL of water to make a 0.100N solution for titration with MnO. 4-? Coloumetric Titration. Cyclohexene is titrated with Br. 2. that has been electrolytically oxidized from...
  • C-FOAM and the 2011 World Conference on Natural

    C-FOAM and the 2011 World Conference on Natural

    55 Laurier Avenue East, Ottawa The Earth Reserve Currency Exchange Indexing System is an innovative project to resolve criticisms of earlier designs by Ben Graham (1937), Nicholas Kaldor (1964) and others, for money based on intrinsic biological and physical value.
  • ADL CODING IMPACTS THE MDS FOCUSED SURVEY TERRY

    ADL CODING IMPACTS THE MDS FOCUSED SURVEY TERRY

    I wanted to talk a little about the financial importance of ADL coding. Both Medicare and Medicaid have RUG classification systems in Pennsylvania. The Medicare is a 66 RUG system and Medicaid is a 44 RUG system. They both have...
  • The Different Voices of Morality: Carol Gilligan's Theory of ...

    The Different Voices of Morality: Carol Gilligan's Theory of ...

    Critical Terms. Ethic of Care- A conception of morality involving an awareness of the crucial nature of human relationships and a need to respect your responsibilities to other people.For example, a young girl might express concern about "leaving out" a...