Development of an Uncertainty Tool to Assess Model Forecast ...

Development of an Uncertainty Tool to Assess Model Forecast ...

Development of an Uncertainty Tool to Assess Model Forecast Parameters Taylor Mandelbaum1, Brian Colle1, Trevor Alcott2 1 Stony Brook University School of Marine and Atmospheric Sciences, Stony Brook, NY 2 Earth Systems Research Laboratory, Boulder, CO * This work is supported by NOAA-CSTAR The Ensemble Situational Ensemble Table (ESAT) plots

anomalies in multiple formats (using NAEFS and GEFS). The goal of ESAT is to provide a tool to assess anomalies in ensemble forecasts. DSS and forecasters can determine how anomalous a forecast is relative to previous events. Background

Terminology The ensemble mean is often regarded as the best guess forecast Conventional spread can provide uncertainty information to a forecaster There is a lot of information that is

compressed in ensemble spread The Motivation The North American Ensemble Forecast System (NAEFS, GEFS + CMC) 96 hour mean, valid February 11, 2010 at 00z. R-Climate and MClimate metrics on the ESAT show the anomaly of the forecast (in both reanalysis and model relative formats) but not the anomaly of uncertainty.

The Motivation The spread of a forecast for a similar anomaly can be compared to M-Climate to assess anomaly. If the anomaly is less than normal, it should translate to less uncertainty. ERA Interim analysis and mean absolute error, valid February 11, 2010 at 00z.

The Motivation The goal is to create a standardized spread anomaly which can be used operationally but The reliability of spread in relation to the skill or error of a forecast is key to moving forward. In order for spread to have a value to forecasters, it must provide a snapshot of the likely magnitude and/or location of forecast error. Otherwise, what use is it? Previous studies (Barker 1991, Whitaker & Loughe 1998, Hopson 2014 Scherrer et al. 2004) utilized either perfect ensemble models (no IC errors) or did not approach the problem from a purely operational standpoint. The results are mixed. Other studies have been done using 500mb heights (Toth et al. 2001) based on R-Climate. The Questions Is there a relationship between spread and mean absolute error? Does the NAEFS offer an improvement over single

model systems for spread/error relationship? Does the subset of spread based on similar forecasts (M-Climate) provide context to the uncertainty of an event? Is the tool useful can it be applied to operations similar to ESAT? The Data M-Climate 1985-2017 GEFS Reforecast 1 degree x 1 degree for DJF, 10 members MSLP ensemble mean, spread Verification Data ERA-Interim 1 degree x 1 degree Ensemble Data

DJF 2007-2014 cyclone storm tracks (Korfe, personal correspondence) for storms with >=8 verifying ensemble members @ hr 96 GEFS and CMC MSLP perturbed members (20 members each) DJF 2007-2014 Verification 2007-2014 Eastern US Cyclones A verified cyclone must have passed through the region within 72W 68W, 34N 46N Each forecast must contain >= 8 ensemble members with cyclones identified 96 hour forecast minimum low pressures

are IDd in GEFS data (1x1 grid) 312 identified 96 hour forecasts Storm tracks for all 2007-2014 storms identified by Nathan Korfes tracker data (blue, green, purple lines denote month) and 96 hour IDd ensemble mean cyclone (blue points) Cyclone Relative Error and Spread Low pressure centers are identified within bounding box and centered within a 10 degree by 10 degree grid (~500km radius) Spread, ensemble mean SLP, and mean absolute error are averaged and plotted for GEFS, CMC, and NAEFS. GEFS (61)

CMC (54) Error, MSLP (hPa) Cyclone Relative Error and Spread Results NAEFS (56) Spread, MSLP (hPa) Cyclone Relative Error GEFS Spread (Difference) between MAE and spread for n=61 Difference each set of cyclones. Red hatching at 95% CMC denotes statistical significance confidence (t-test) n=54

NAEFS n=56 Cyclone Relative Error Spread, Separated by Spread Magnitude GEFS Separating the GEFS and n=30,31 NAEFS n=28,28 NAEFS by ranked mean spread of each event GEFS shows greater error near the center of the low for the bottom half of spread cases; lows have slightly less magnitude (top left). NAEFS handles lower

spread storms better (bottom left) but has higher spread than error along the east, center, and southern flank (bottom right). Cyclone Mean Error Separating the events in half by ranked spread magnitude GEFS n=30,31 Slight dipole between NAEFS n=28,28

center of storm and north of storm for GEFS lower half Dipole pattern for the GEFS upper half Slight positive error in NAEFS lower half Larger positive error north and slightly west of NAEFS upper half GEFS NAEFS Error, MSLP (hPa) valid December 28, 2012 x

Spread, MSLP (hPa) More than one way to look at similarities between spread and error, ideally the error and spread correlate well with a linear pattern. When looking at the MAE and Spread in a joint plot, correlations appear low in both GEFS and NAEFS. Theres more to do there is value to looking at spread in the context of both position and error, but as always it cant be used as the end-all product. Standardized Spread Anomaly Method Standardized Spread Anomaly Sample Case: February 10-11, 2010 Synopsis A strong mid-latitude low

brought blizzard conditions to the Mid-Atlantic and Northeast, where snowfall amounts topped 12 inches into much of Pennsylvania, New Jersey, and into the New York Metro region. Results The SSA indicates a 3-4 standard deviation anomaly (non-normal distribution) offshore Cape Cod during the latter stages of the event, coinciding w/ the 10-12 hPa spread visible in overlapping locations.

Standardized Spread Anomaly Sample Case: February 10-11, 2010 Synopsis A strong mid-latitude low brought blizzard conditions to the Mid-Atlantic and Northeast, where snowfall amounts topped 12 inches into much of Pennsylvania, New Jersey, and into the New York Metro region. Results The SSA indicates a 3-4 standard deviation anomaly (non-normal

distribution) offshore Cape Cod during the latter stages of the event, coinciding w/ the 10-12 hPa spread visible in overlapping locations. Standardized Spread Anomaly Sample Case: February 10-11, 2010 Synopsis A strong mid-latitude low brought blizzard conditions to the Mid-Atlantic and Northeast, where snowfall amounts topped 12 inches into much of Pennsylvania, New Jersey, and into the New

York Metro region. Results The SSA indicates a 3-4 standard deviation anomaly (non-normal distribution) offshore Cape Cod during the latter stages of the event, coinciding w/ the 10-12 hPa spread visible in overlapping locations. Standardized Spread Anomaly Sample Case: February 9-10, 2013 Synopsis A significant cyclone caused snowfall totals of

over 2 feet in Long Island and New England, with some locations in Long Island and Connecticut reaching 34-40 inches. Results The storm on the GEFS at 96 hours was widely dispersed but spread only peaking at 8-10 hPa near the Gulf of Maine. Neither the location nor magnitude was apparent. The SSA indicates at least a larger anomaly wrt magnitude. Standardized Spread Anomaly

Sample Case: February 9-10, 2013 Synopsis A significant cyclone caused snowfall totals of over 2 feet in Long Island and New England, with some locations in Long Island and Connecticut reaching 34-40 inches. Results The storm on the GEFS at 96 hours was widely dispersed but spread only peaking at 8-10 hPa near the Gulf of Maine. Neither the location nor magnitude was apparent. The SSA indicates at least a larger

anomaly wrt magnitude. Standardized Spread Anomaly Sample Case: February 9-10, 2013 Synopsis A significant cyclone caused snowfall totals of over 2 feet in Long Island and New England, with some locations in Long Island and Connecticut reaching 34-40 inches. Results The storm on the GEFS at 96 hours was widely dispersed but spread only peaking at 8-10 hPa near

the Gulf of Maine. Neither the location nor magnitude was apparent. The SSA indicates at least a larger anomaly wrt magnitude. Conclusions Verification GEFS has no statistically significant difference (95%) between mean absolute error and spread with n=61 NAEFS (and CMC) has statistically significant differences on the southern (and south, center, east) flank of cyclones with n=56 (and n=54) Using mean absolute error as a first order verification metric is promising for cyclone relative SLP, other variables should be possible (QPF, Z, 850T, etc) Spread Anomaly The tool shows promise in terms of identifying magnitude of spread

anomaly but is only as good as the model its looking at. Standardized anomalies when looking at a non-normal distribution is less than optimal; percentiles wash out the events and dont specify location as well. Return intervals are a potential alternative. Future Work Website A website is currently being built for spread anomaly and verification for real time usage this winter Verification via backtesting the previous year using plots similar to the ESAT verification (Spread vs MAE) will be tested for GEFS and NAEFS Variables Other variables will be used including QPF and 850mb temperature similar to ESAT Verification of Spread Anomaly The verification procedure looking at spread/error will also be used for SSA.

GEFS Reforecast will be tested for similarity w/ the operational GEFS Utilizing probability matched mean to account for magnitude errors in error/spread calculations References Barker, T. W. (1991). The Relationship between Spread and Forecast Error in Extended-range Forecasts. Journal of Climate. https://doi.org/10.1175/15200442(1991)004<0733:TRBSAF>2.0.CO;2 Buizza, R. (1997). Potential Forecast Skill of Ensemble Prediction and Spread and Skill Distributions of the ECMWF Ensemble Prediction System. Graham, R. A., & Grumm, R. H. (2010). Utilizing Normalized Anomalies to Assess SynopticScale Weather Events in the Western United States. Weather and Forecasting, 25(2), 428445. https://doi.org/10.1175/2009WAF2222273.1 Hamill, T. M., G.T. Bates, J. S. Whitaker, D. R. Murray, M. Fiorino, T. J. Galarneau, Y. Zhu, and W. Lapenta. (2013). NOAAs Second Generation Global Medium Range Ensemble Forecast Dataset. Bull. Amer. Meteor. Soc., 94, 15531565. Hopson, T. M. (2001). Assessing the Ensemble SpreadError Relationship. https://doi.org/10.1175/MWR-D-12-00111.1 Scherrer, S. C., Appenzeller, C., Eckert, P., & Cattani, D. (2004). Analysis of the SpreadSkill Relations Using the ECMWF Ensemble Prediction System over Europe. Toth, Z., Zhu, Y., & Marchok, T. (2001). The Use of Ensembles to Identify Forecasts with Small

and Large Uncertainty. Weather and Forecasting, 16(4), 463477. https://doi.org/10.1175/1520-0434(2001)016<0463:TUOETI>2.0.CO;2 Whitaker, J. S., & Loughe, A. F. (1998). The Relationship between Ensemble Spread and Ensemble Mean Skill. Questions? Email: [email protected]

Recently Viewed Presentations

  • Unit 27: Network Operating Systems - James Tedder

    Unit 27: Network Operating Systems - James Tedder

    Unit 27 - Network Operating SystemsAssignment 1 - Task 1.1. You are required to evaluate types of NOS and NOS services of a selection of Network Operating Systems such as standalone, infrastructure based and cluster based systems.
  • Chapter 3: Probability

    Chapter 3: Probability

    Chapter 3 Probability § 3.2 Conditional Probability and the Multiplication Rule Conditional Probability Conditional Probability Independent Events Probability § 3.2 Conditional Probability and the Multiplication Rule Conditional Probability Conditional Probability Independent Events A conditional probability is the probability of an...
  • 2009- 2010 Cheerleading Parent Meeting

    2009- 2010 Cheerleading Parent Meeting

    Krispy Kreme Pre-Sale: Dates: Money due 20th ; Order arrival 22nd Flap Jack Applebee's- September Split the Pot(Varsity Only) Candy Sale (Basketball Season) Great American Fundraising Scratch cards Raffle Tickets All cheerleaders are strongly encouraged to participate in the fundraiser.
  • Jeopardy - MRS. KNIGHT&#x27;S 7TH GRADE MATH CLASS 2017-2018

    Jeopardy - MRS. KNIGHT'S 7TH GRADE MATH CLASS 2017-2018

    Review Miscellaneous Probability Value of X Scale factor Linear Q $100 Q $100 Q $100 Q $100 Q $100 Q $200 Q $200 Q $200 Q $200 Q $200 Q $300 Q $300 Q $300 Q $300
  • Chapter Three: Designing a Personal Fitness Program

    Chapter Three: Designing a Personal Fitness Program

    ffecting Progression. Your initial fitness level (the lower you start, the more quickly you usually improve) Your heredity. The rate at which you overload your body or change your FITT. Your specific goals (health or performance)
  • Poe&#x27;s Recurring Themes, Symbols, and Motifs

    Poe's Recurring Themes, Symbols, and Motifs

    Poe's Recurring Themes, Symbols, and Motifs Master of the Macabre HORROR Gothic = literature characterized by a gloomy setting, mysterious or violent events, and an atmosphere of degeneration and decay Where's a great place to get ideas?
  • Mutualism & Commensalism Photo of hawk moth potentially

    Mutualism & Commensalism Photo of hawk moth potentially

    Facilitation - in other words, "+" means benefits outweigh costs. Commensalism = +/0. Mutualism = +/+ Positive Interactions. Photo of hawk moth potentially pollinating . Dianthus. from Wikimedia Commons . What might the benefits and costs be to
  • NC Teach Ag Ambassador Program Introductions Name Year

    NC Teach Ag Ambassador Program Introductions Name Year

    Stop by the booth and pick up your regional S/D profile. 2013/2008 Ag Ed Summit R/R ideas. Teach Ag owl. List of resources and more. If we are out of anything go to the Teach Ag website and print your...