The life and times of Coretta Scott King

The life and times of Coretta Scott King

Grade 3 FCAT Test Construction & Equating June 1, 2007 Cornelia S. Orr, Assistant Deputy Commissioner of Accountability, Research, and Measurement (ARM) Office of Assessment and School Performance Florida Department of Education Experience teaches only the teachable. Aldus Huxley (1894-1963) Topics The Grade 3 Test in 2006 Test Construction Process and Product Science and Art Psychometric Primer Test Calibration and Equating 2 The Grade 3 Test in 2006 Passages Questions Forms Student scores based on 5 passages & 45 questions 30 different forms, each with 1 passage & 7-8 questions Forms are used for anchor and field test questions

One of the 6 passage positions is used for anchor and field test questions 2006 Grade 3 FCAT Test Passages and Positions Day 1 Session 1 1 2 3 Day 2 Session 2 4 5 6 3 The Grade 3 Test in 2006 2006 Grade 3 FCAT Test Passages and Positions Day/ Session 1 2 Passage Position Number of Questions Passage Description 1

8 2 7 or 8 3 10 A Gift of Trees (Inform.) 4 13 Swim, Baby, Swim (Lit.) 5 8 Slip, Slop, Slap/Sunny Sidebar (Inform.) 6 6 Making Spring (Lit.) TOTAL Ladybird, Ladybird, Fly Away Home (Lit.) Anchor and Field Test Passages (Varies)

52-53 4 Test Construction Process of building the test Occurs the summer before a test Based on available passages, questions, and statistics Guidelines for building the test Test Construction Specifications Building the test is an iterative process 5 Test Construction Cycles 1999 2000 2001 2002 2003 Oct/Nov Item Review Meetings Oct/Nov Item

Review Meetings Oct/Nov Item Review Meetings Oct/Nov Item Review Meetings Nov/Dec Item Review Meetings May/Aug Item Review Meetnigs Sept/Oct Field Test Construction June Field Test Construction June Test Construction June Field Test Construction June Test Construction June Field Test Construction June Test Construction

June Field Test Construction June Test Construction 2004 June Field Test Construction June Test Construction 2005 2006 June Test Construction 2006 Test Admin. March Field Test 2005 Test Admin. March Field Test 2004 Test Admin. March Field Test 2003 Test Admin.

March Field Test 2002 Test Admin. March Field Test 2001 Test Admin. March Field Test 6 Test Construction Specifications - 1 Guidelines for building the test Ranges for each category Iterative process Content Guidelines Reading Passages (type and word counts) Benchmark Coverage Reporting Category (Strand) Coverage

Multicultural & Gender Representation Cognitive Level Guidelines 7 Test Construction Specifications 2 Statistical Guidelines for Questions Classical Item Difficulty and Discrimination IRT Difficulty, Discrimination, and Guessing Differential Item Functioning (DIF) IRT Model Fit Statistics Statistical Guidelines for Tests Test Characteristic Curves Test Information Functions Standard Error Curves 8 Test Construction Specifications 3 Anchor Item Guidelines Number and position of questions Content Representation Mini Test Performance Characteristics (range of difficulty)

Previous use as a Core or Anchor No change in wording Passage position 9 Test Construction Review and Approval Process 1st Draft of Content Harcourt Content Staff Review of Content DOE Content Staff Review of Statistics Harcourt Psychometric Staff Review of Statistics DOE Psychometric Staff Approval by DOE FCAT team leadership 10 Psychometric Primer 1 Classical Item Statistics: P-value or difficulty the percent (P) who answer the question correctly. Discrimination (point-biserial) the degree to which students who get high scores answer the question correctly and vice versa (similar to correlation). 11 Psychometric Primer 2 Item Response Theory (IRT) Statistics A-parameter discrimination or how well the question differentiates between lower and higher performing

students. B-parameter difficulty or the level of ability on the 100500 scale required to answer the question correctly. Guessing the probability of examinees with extremely low ability levels getting a correct answer. FIT how well the scores for a given item fit, or match, the expected distribution for the model. DIF (Differential Item Functioning) the degree to which the question performs similarly for all demographic groups based on ability. 12 0.8 0.6 Discrimination=1 Difficulty =0.5 0.2 0.4 Pseudo-Guess. =0.13 0.0 P robability of C orrect Re sponse 1.0 Item Characteristic Curve Figure 1 -4 -2 0

2 4 Achievement Index (Theta) 13 Test Characteristic Curve Figure 2 % corr TCC: Reading 2006 Gr. 3 Core ver. 7 .... Base Form (2005) ____ New Form Expected Pct-Correct Score 100.0 75.0 50.0 259 284 332 394 25.0 0.0 100 200 300 400

500 Scale Score 14 Standard Error Curve Figure 3 SEM: Reading 2006 Gr. 3 Core ver. 7 .... Base Form (2005) ____ New Form 100 75 50 259 284 332 394 25 0 100 200 300 400 500 Scale Score

15 Test Calibration and Equating Calibration Converting from Raw Scores to IRT scores Equating Making Scores Comparable Across Years Florida uses Item Response Theory (IRT) to score and equate FCAT results from year to year. 16 FCAT Calibration and Equating Process Step 1: Determine unscaled item parameters for core & anchor together. Current Year Core Test New Values ion bratt Cali ll of a Step 5: Generate Student Scores Using Equated Score Scale Step 2: Analyze the change in the anchor Items Current Year Anchor Item

New Values Prior Year Anchor Item Values 1 1 2 2 3 3 4 4 1 2 3 4 5 6 7 8 9 10 11 12 s) 13 eter ram 14

c pa & , ,b 15 er (a 16 s togeth em it17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 5

5 6 7 pare Com nt Curre lues or Va to Pri 6 7 8 8 9 9 10 10 11 11 12 12

13 13 14 14 15 15 16 16 17 17 18 18 19 19 20 20 21 21 Step 4: Apply the Equating Adjustments and Convert to Scaled Item Parameters

Step 3: Identify Equating Adjustments (M1 & M2) 17 Equating Solutions 2006 equating solution anchor questions ??? Identify a better equating solution Define better Process considerations Select anchor questions Follow the guidelines Evaluate the quality of the anchor 18

Recently Viewed Presentations

  • U.S. General Services Administration Federal Acquisition Service System

    U.S. General Services Administration Federal Acquisition Service System

    How the system will address orders in transition. Background. ... If the winning quote/offer does not reference an existing contract number and instead is listed as 'Open Market'; you will be given the opportunity to select from a different list...
  • Marine Geology Using GEODE  Welcome!  Please get a

    Marine Geology Using GEODE Welcome! Please get a

    Explore marine sediments using Google Earth, including a "ready to go" student exercise. Explore the ocean basins using Google Earth. Outline ideas for how these resources could be used in their teaching. Prepare sample requests to IODP for physical samples...
  • Practicing Similes and Metaphors - SharpSchool Redirect

    Practicing Similes and Metaphors - SharpSchool Redirect

    Practicing Similes and Metaphors Click anywhere on this page to begin. Created in 2005 By Cheri Sandford Liberty MS Directions: Read the sentence on each slide. Then click on the button that tells what kind of comparison is being made....
  • The US in World War I - McGrath's Classroom

    The US in World War I - McGrath's Classroom

    African-Americans in WW I. The Great Migration (pg 308) Women During World WAR I. ... Run by George Creel during World War I. ... all taken during war, colonies, Alsace-Lorraine. Military restrictions - no making/importing weapons, limits on size of...
  • USENIX Windows NT Workshop - kls2.com

    USENIX Windows NT Workshop - kls2.com

    Adding Response Time Measurement of CIFS File Server Performance to NetBench Karl L. Swartz [email protected] Large Scale System Administration of Windows NT Workshop Overview NetBench only measures throughput Hacks to measure response time Comparing different CIFS dialects and file locking...
  • Indexing with Elasticsearch

    Indexing with Elasticsearch

    Open browser (from within VLabs or ISTNetwork wired computer) ... e.g., for team 1 → ist441giles.ist.psu.edu:5601. for team2 → ist441giles.ist.psu.edu:5602 … and so forth. Defining Text Preprocessing. How to create a new index that filters out HTML tags and extracts/indexes...
  • University Application Seminar

    University Application Seminar

    University Application Seminar Fall 2018 Scholarships Entrance scholarships Automatic…based on admission average University scholarships needing application Check the university web site or look on e-!NFO Foundation scholarships (ie Terry Fox, Loran) Usually need high marks and outstanding community service.
  • www.erhsnyc.org

    www.erhsnyc.org

    Vocabulary. Budget: A financial plan used to forecast and track income and expenses. Surplus/Deficit: _____ Expense: Use of finances to fulfill a one-time or ongoing obligation; can be fixed or variable. Fixed Expense: A regular, predictable expense such as a...