Transcription

The Calibration Verification/Linearity Program:Meetingg Regulatorygy RequirementsqandImproving Laboratory QualityAnthony Killeen, MD, PhDPatricia StyerStyer, PhDWilliam Castellani, MDMay 6, 2011www.cap.org

Speakers (in order of presentation) Anthonyy Killeen,, MD,, PhD,, FCAPo Chair, Instrumentation Resource Committee William Castellani,Castellani MD,MD FCAPo Inter-Regional Commissioner for the CAPLaboratory Accreditation Program and Chair ofthe CAP ISO15189 Committee Patricia Styer, PhDo Biostatistics Consultant, College of AmericanPathologists 2011 College of American Pathologists. All rights reserved.2

Outline Introduction ((Dr. Killeen)) CLIA and LAP Regulations for Calibration Verificationand AMR Validation ((Dr. Castellani)) Overview of the CAP CalibrationVerification/Linearityy Programg((Dr. Styer)y ) Examples and Troubleshooting (Dr. Killeen) 2011 College of American Pathologists. All rights reserved.3

CLIA Calibration Verification Periodic verification that the calibration of theanalytical system remains validqbyy Clinical Laboratoryy Improvementp RequiredAmendment (CLIA) if the test system has not beenrecalibrated for 6 months Typically assessed by comparing test results fromsamples with those samples’ expected targetvalues If the calibration changes, then patient test resultvalues will also change 2011 College of American Pathologists. All rights reserved.4

Linearity From Clinical Laboratory Standards Institute (CLSI)document EP6EP6-AA (2003)o A quantitative analytical method is linear whenthere exists a mathematically verified straightline relationship between the observed valuesand the true concentrations or activities of theyanalyte.o The linearity of a system is measured by testinglevels of an analyte which are known byfformulationl tior knownkrelativel ti tot eachh otherth (not( tnecessarily known absolutely).CLSI. Evaluation of the Linearity of Quantitative Measurement Procedures: A Statistical Approach; Approved Guideline. CLSIdocument EP6-A (ISBN 1-56238-498-8). CLSI, 940 West Valley Road, Suite 1400, Wayne, Pennsylvania 19087-1898 USA,2003. 2011 College of American Pathologists. All rights reserved. 5

CLIA Calibration Verification vsvs. Linearity Calibration verification is the pprocess of verifyingy gagreement between calibrators (or other materialsof known analyte concentrations) and measuredvalues Calibrators should ideally be traceable to areference method to ensure accuracy Linearity evaluation does not require knowledge ofthe “true”true analyte concentration “Linearity” does not appear in CLIA 2011 College of American Pathologists. All rights reserved.6

Linearity vsvs. Instrument Response Linearityy typicallyypy refers to the final analyticalyresult,,not the raw instrument outputo A plot of analyte concentration vs. the rawi tinstrumentt outputt t may nott beb linearli(e.g.,(competitive immunoassay) “Li“Linearity”it ” as usedd ini thisthi contextt t means a straightt i htline relationship between “true” analyteconcentrations and measured concentrations 2011 College of American Pathologists. All rights reserved.7

Linearity vsvs. Instrument Response 2011 College of American Pathologists. All rights reserved.8

Linearity and the Analytical Measurement Range The analyticalymeasurement rangeg is the rangeg ofconcentrations that an instrument can measurewithout any pretreatment of the sample (e.g.,concentration dilution) that would change theconcentration,concentration of an analyte An analytical system should show linearity over itsanalytical measurement range 2011 College of American Pathologists. All rights reserved.9

Linearity and the Analytical Measurement Range 2011 College of American Pathologists. All rights reserved.10

Advantages to Participating in the CAPCalibration Verification/Linearity Program CVL pprogramgpprovides test samplespand dataanalysis to assist laboratories in meeting CLIA andLAP requirements Samples are prepared to challenge the fullanalytical measurement range Linearity testing often has smaller absolute limits forerror, based on medically or analytically relevantcriteria than does PTcriteria, Can detect problems earlier than QC or PT 2011 College of American Pathologists. All rights reserved.11

CLIA and LAP Requirements for CalibrationVerification and AMR ValidationPresented byy William Castellani,, MD 2011 College of American Pathologists. All rights reserved.12

Calibration and Calibration Verification “Calibration” means a process of testing andadjustingdj tian instrumenti tt or testt t systemttot establisht bli h acorrelation between the measurement response andthe concentration or amount of the substance that isbeing measured by the test procedure.procedure “Calibration verification” means the assaying ofmaterials of known concentration in the samemanner as patient specimens to substantiate theinstrument or test system’s calibration throughout thereportable range for patient test results.- CentersC tforf MedicareM diandd MedicaidM di id SServices,iState Operations Manual, Appendix C 2011 College of American Pathologists. All rights reserved.13

Reportable Range Reportableprangeg means the spanpof test resultvalues over which the laboratory can establish orverify the accuracy of the instrument or test systemmeasurement response.response- CLIA ’88, Sec. 493.2, Definitions 2011 College of American Pathologists. All rights reserved.14

Reportable Range Continued Two components:po The primary range of measurement Analytical measurement range “Linear” rangeo Anything done to the system to expand this range “Clinical“Cli i l reportablet bl range”” Reportable range 2011 College of American Pathologists. All rights reserved.15

Requirements for Compliance Validate or verifyyo Reportable range: as part of method validationo Analytical measurement range: as part of methodvalidation and every six months thereafter (whennecessary) 2011 College of American Pathologists. All rights reserved.16

Other Considerations Set criteria of acceptance Established protocol Medical relevanceo All of this should be established byy the laboratoryydirectoro All of this should be documented formallyo The actual review may be delegated, though finalauthorization may be reserved for the director 2011 College of American Pathologists. All rights reserved.17

General Principles Establish a targetg valueo May use a patient sample’s result as the “target”o May use peer group mean of PT materialo MayMbeb establishedt bli h d byb theth providerid off ththe materialt i l Establish an acceptable range around the targeto MayMbeb a laboratory-assignedl btid range [10%]o May use precision data for control material near thetargeto May be provided by the manufacturer Document yyour pprotocol ((approvedppbyy director)) 2011 College of American Pathologists. All rights reserved.18

CLIA Requirement for Calibration and CalibrationVerification 493.1255: Calibration and calibration verificationprocedures are required to substantiate thecontinued accuracy of the test system throughoutthe laboratory’slaboratory s reportable range 2011 College of American Pathologists. All rights reserved.19

Calibration and Calibration Verification Calibration: Establishes the relationshipp betweenanalyte content and instrument measurement signal Calibration verification: Confirms that the currentcalibration settings remain valid 2011 College of American Pathologists. All rights reserved.20

CAP Interpretation of CLIA Calibration Verification The Laboratory Accreditation Program considersCLIA calibrationlib tiverificationifi titot beb secondarydtotcalibrationo If calibration satisfies the CLIA requirements for calibrationverification [i.e., calibrated at least every six months withappropriate calibrators], no further action is necessary The CAP also separates CLIA calibration verification(when required) into two parts:o Prove the calibration still is valid (CAP CalibrationVerification)o Prove response over the entire analytical measurementrange (CAP AMR validation) 2011 College of American Pathologists. All rights reserved.21

CLIA Calibration Verification Requirements Sec. 493.1255(b)(2)( )( ) [Perform[and documentcalibration verification procedures] Using the criteriaverified or established by the laboratory o (i) Including the number,number type,type and concentration of thematerials, as well as acceptable limits for calibrationverification; ando (ii) IIncludingl diatt leastlt a minimali il (or( zero)) value,la mid-pointidi tvalue, and a maximum value near the upper limit of therange to verify the laboratory's reportable range of testresults for the test system 2011 College of American Pathologists. All rights reserved.22

CAP Requirements for Calibration Verification Targetg values Appropriate MatrixoCalibrators used to calibrate the analytical measurementsystem (different lot)oMaterials provided by the analytical measurementsystem vendor for the purpose of calibration verificationoPreviously tested unaltered patient/client specimensoPrimary or secondary standards or reference materialswith matrix characteristics and target values appropriatefor the method 2011 College of American Pathologists. All rights reserved.23

CAP Requirements for Calibration VerificationContinued Appropriatepp pMatrix ((continued))oThird party general purpose reference materials ifcommutableoProficiency testing material or proficiency testingvalidated material with matrix characteristics and targetvalues appropriate for the method.oQC Material if: appropriate matrix and a peer groupinterlaboratory mean value based on at least 10 differentlaboratories using comparable method. In general,general routine control materials are not suitable forcalibration verification, except in situations where thematerial is specifically designated by the methodmanufacturer as suitable for verification of the method'scalibration process. 2011 College of American Pathologists. All rights reserved.24

CAP Requirements for the Verification of theAnalytical Measurement Range Targetg values Sufficient samples (as discussed later) AppropriateAi t matrixtio Linearity material of appropriate matrixy testingg surveyy materialo Proficiencyo Previously tested patient specimens, unalteredo Previously tested patient specimens, altered by admixture withother specimens, dilution, spiking or other techniqueo Primary or secondary standards or reference materials with matrixcharacteristics and target values appropriate for the methodo Calibrators used to calibrate the analytic measurement systemo Control materials, if they adequately span the AMR. 2011 College of American Pathologists. All rights reserved.25

Implementation of LAP Calibration Verificationand AMR Validation “Trueness” assumes that there is a value that theinstrument should report for a specific sampleo Calibration establishes this assignment; calibrationverification shows that this is still trueo ControlsC t l dod nott usuallyll come withith assignedid valueslthatth t arevalid for the instrument unless the manufacturer provesthese values “Li“Linearity”it ” demonstratesdt t a fixedfi d relationshipl tihibetween two valueso A doubling of a value indicates twice as much analyte In this case, the actual “values” don’t matter, only therelationshipo The relationship between results must hold throughout theanalytical measurement range,range including when the rangeextends beyond the calibrator values 2011 College of American Pathologists. All rights reserved.26

Implementation of LAP Calibration Verificationand AMR Validation Continued If LAP calibration verification is needed:o EstablishE t bli h theth “trueness”“t” off theth methodth d Most often easiest to perform at the calibration point(s) If AMR validation is required:o If you have established “trueness” (by calibrationor calibration verification), verify that a linearrelationship holds throughout the instrument AMRo Establish “trueness” throughout the AMR bycomparing results to established target valueso Use a combination of both comparison to targetyvalues and verification of linearity 2011 College of American Pathologists. All rights reserved.27

Number of Samples Required for AMR Validation Threeo CLIA minimal requirement (low, mid-point, high) Fouro Various opinions Fiveo What I was taught as a resident More?o The more the points, the greater your confidence thatany value actually reflects the concentration in thepatient sample,sample but practical considerations (cost(cost, time)constrain the laboratory 2011 College of American Pathologists. All rights reserved.28

Samples for Analytical Measurement RangeValidation – How Low and How High? “Guidelines for analyte levels near the low and high range ofthe AMR should be determined by the laboratory director.Factors to consider are the expected analytic imprecision nearthe limits, the clinical impact of errors near the limits, and theavailabilityil bilit off testt t specimensinear theth limits.li it It may beb difficultdiffi ltto obtain specimens with values near the limits for someanalytes (e.g., T-uptake, free thyroxine, free phenytoin,prolactin FSHprolactin,FSH, troponintroponin, pO2).) In such cases,cases reasonableprocedures should be adopted based on available specimenmaterials.”- Chemistry and Toxicology checklist,checklist 6/17/10 2011 College of American Pathologists. All rights reserved.29

Samples for Analytical Measurement RangeValidation – How low and how high? Determined by available material:o Define the linear range as going from the low to the hightarget sample Fixed range:o Within 10% of the top end and 1% of the bottom end Clinical use and decision pointsThe ability of commercial “available material” to span theentire range of an instrument is constrained by the cost ofmakingg samplespwith extremelyy highg concentrations 2011 College of American Pathologists. All rights reserved.30

Extending the Verification Range Available material withtarget values may notreach the upper limit ofthe analyticalmeasurement range Example: urine sodiumo manufacturer’srange: 0 – 200 mmol/L 2011 College of American Pathologists. All rights reserved.31

Extending the Verification Range Continued Preparepa stocksodium solution of 200mmol/L Do two serial x 2dilutions (100 and 50mmol/L target) Assay each level andplot 2011 College of American Pathologists. All rights reserved.32

Measurement of Results Beyond the AMR Mayy decrease the lower limit of the analyticalymeasurement range by:o Concentrating the sample AmiconA iconcentratort t Extraction and resuspensiono Increasingg the ratio of samplep to reagentg Altering the programming of the instrument 2011 College of American Pathologists. All rights reserved.33

Measurement of Results Beyond the AMRContinued More commonly, may increase the upper limit of theanalyticall ti l measurementt range by:bo Decreasing the ratio of sample to reagento Diluting the sample before analysis Most often, the manufacturer provides theinformation or mechanism for this modificationo Autodilution/autoconcentrationo Dilution protocolo Concentration protocol Good laboratory practice would include verifyingthat these modifications work 2011 College of American Pathologists. All rights reserved.34

Overview of the CAP CalibrationVerification/Linearity Survey EvaluationsPresented byy Patricia Styer,y , PhD 2011 College of American Pathologists. All rights reserved.35

Purpose of the CAP CalibrationVerification/Linearity Survey Provide test samples and analysis for AMRvalidationlid tio Exceed the minimal requirements for the numberof specimens and possible analyseso Review and modify material specifications foroptimal AMR coverage Provide information for ongoing quality monitoringo Performance criteria are usuallyy more stringentgthan proficiency testingo Detect possible problems before they impact PTor patientti t testingt ti 2011 College of American Pathologists. All rights reserved.36

Another Use of the Term “CalibrationCalibration VerificationVerification” Previous slides have defined CLIA calibrationverification and LAP calibration verification We also have the calibration verification evaluation inthe CAP CVL Program In the CVL Program, the calibration verificationevaluation compares participant results to targetvalues 2011 College of American Pathologists. All rights reserved.37

Components of a CAP CalibrationVerification/Linearity Survey Participants receive a set of vials with varyingconcentrationst tioff analyte(s)l t ( ) Participants submit results for two assays from eachvial,i l withinithi theth same run if possibleibl The CAP provides two individual evaluations andse eral peer groupseveralgro p summariess mmarieso Calibration verification evaluationo Linearity evaluationo Peer group summary statisticso Peer group performance summaries 2011 College of American Pathologists. All rights reserved.38

Participant Data Input Participants receive a set of numbered vials and aresultlt form.f Participants specify an instrument, method, and/orreagentt ffor eachh analyte.l tSerum Ethanol Survey 2011 College of American Pathologists. All rights reserved.Automated Hematology39

Participant Data Input Continued Participants perform two assays from each vialwithin the same run.run 2011 College of American Pathologists. All rights reserved.40

Calibration Verification Example forSerum Ethanol mg/dL Assay meanscompared to targetvalues Differences evaluatedusing allowable errorlimits by specimen level Allowable errors can belarger on percentagescale for lowerconcentrations Result is Verified overfull range 2011 College of American Pathologists. All rights reserved.41

Linearity Example for Serum Ethanol mg/dL Results compared to fittedstraight line X-axis shows relativeconcentrations (frommaterial production) Evaluation based onaverage deviations fromfitted straight line Evaluation can beo Linearo Nonlinearp((Pooro ImpreciseRepeatability and/or Fit) 2011 College of American Pathologists. All rights reserved.42

Serum Ethanol Example – Interpretation of Results Evaluation resultso VerifiedV ifi d fromf13.2513 25 tot 521.40521 40 ((goodd agreementtwith peer-based target values)o Linear from 1313.2525 to 521.40521 40 (expected linearrelationship is confirmed) Sometimes evaluation results will not agreeo Review peer group data and summarieso Matrix effects can cause linearityypproblemso Mixed reagent lots can cause calibrationverification problems 2011 College of American Pathologists. All rights reserved.43

Calibration Verification Example forHemoglobin A1c1 % Participant meanscompared to accuracybased target values Peer groups forperformancesummaries All other componentsof evaluation are thesame 2011 College of American Pathologists. All rights reserved.44

Extended/diluted Linearity Example forWhite Blood Cells 109/L Extended range specimens are indicated in the linearity evaluationsummaryy table. We fit a line to the non-extended range specimens. The non-extended range specimens must be linear for theevaluation to continue.continue Means of the extended range specimens are compared to theextrapolated line (Extended Range Specimen Analysis – next slide). 2011 College of American Pathologists. All rights reserved.45

Extended/diluted Linearity Example forWhite Blood Cells 109/L Continued The plot on the left is the same; the difference plot shows allowableerror bars for the extended rangeg specimenpresults. We complete the same analysis for diluted specimens when wehave at least five undiluted specimens to fit the initial line. 2011 College of American Pathologists. All rights reserved.46

Interpreting CVL Evaluations WithProblematic Results Participants have many pieces of inf