Evaluation of individual and ensemble probabilistic forecasts of COVID-19 mortality in the United States

• Authors: CRAMER EY; RAY EL; LOPEZ VK; BRACHER J.; BRENNEN A.; RIVADENEIRA AJC; GERDING A.; GNEITING T.; HOUSE KH; HUANG YX; JAYAWARDENA D.; KANJI AH; KHANDELWAL A.; LE K.; MUHLEMANN A.; NIEMI J.; SHAH A.; STARK A.; WANG YJ; WATTANACHIT N.; ZORN MW; YY Gu; JAIN S.; BANNUR N.; DEVA A.; KULKARNI M.; MERUGU S.; RAVAL A.; SHINGI S.; TIWARI A.; WHITE J.; ABERNETHY NF; WOODY S.; DAHAN M.; FOX S.; GAITHER K.; LACHMANN M.; MEYERS LA; SCOTT JG; TEC M.; SRIVASTAVA A.; GEORGE GE; CEGAN JC; DETTWILLER ID; ENGLAND WP; FARTHING MW; HUNTER RH; LAFFERTY B.; LINKOV I.; MAYO ML; PARNO MD; ROWLAND MA; TRUMP BD; ZHANG-JAMES Y.; CHEN S.; FARAONE SV; HESS J.; MORLEY CP; SALEKIN A.; WANG DL; CORSETTI SM; BAER TM; EISENBERG MC; FALB K.; HUANG YT; ET Martin; MCCAULEY E.; MYERS RL; SCHWARZ T.; SHELDON D.; GIBSON GC; YU R.; GAO LY; MA Y.; WU DX; YAN XF; JIN XY; WANG YX; CHEN YQ; GUO LH; ZHAO YT; GU QQ; CHEN JH; WANG LX; XU P.; ZHANG WT; ZOU DF; BIEGEL H.; LEGA J.; MCCONNELL S.; NAGRAJ VP; GUERTIN SL; HULME-LOWE C.; TURNER SD; SHI YF; BAN XG; WALRAVEN R.; HONG QJ; KONG S.; VAN DE WALLE A.; TURTLE JA; BEN-NUN M.; RILEY S.; RILEY P.; KOYLUOGLU U.; DESROCHES D.; FORLI P.; HAMORY B.; KYRIAKIDES C.; LEIS H.; MILLIKEN J.; MOLONEY M.; MORGAN J.; NIRGUDKAR N.; OZCAN G.; PIWONKA N.; RAVI M.; SCHRADER C.; SHAKHNOVICH E.; SIEGEL D.; SPATZ R.; STIEFELING C.; WILKINSON B.; WONG A.; CAVANY S.; ESPANA G.; MOORE S.; OIDTMAN R.; PERKINS A.; KRAUS David; KRAUS Andrea; GAO ZF; BIAN J.; CAO W.; FERRES JL; LI CZ; LIU TY; XIE X.; ZHANG S.; ZHENG S.; VESPIGNANI A.; CHINAZZI M.; DAVIS JT; MU K.; PIONTTI APY; XIONG XY; ZHENG A.; BAEK J.; FARIAS V.; GEORGESCU A.; LEVI R.; SINHA D.; WILDE J.; PERAKIS G.; BENNOUNA MA; NZE-NDONG D.; SINGHVI D.; SPANTIDAKIS I.; THAYAPARAN L.; TSIOURVAS A.; SARKER A.; JADBABAIE A.; SHAH D.; DELLA PENNA N.; CELI LA; SUNDAR S.; WOLFINGER R.; OSTHUS D.; CASTRO L.; FAIRCHILD G.; MICHAUD I.; KARLEN D.; KINSEY M.; MULLANY LC; RAINWATER-LOVETT K.; SHIN L.; TALLAKSEN K.; WILSON S.; LEE EC; DENT J.; GRANTZ KH; HILL AL; KAMINSKY J.; KAMINSKY K.; KEEGAN LT; LAUER SA; LEMAITRE JC; LESSLER J.; MEREDITH HR; PEREZ-SAEZ J.; SHAH S.; SMITH CP; TRUELOVE SA; WILLS J.; MARSHALL M.; GARDNER L.; NIXON K.; BURANT JC; WANG L.; GAO L.; ZL Gu; KIM M.; LI XY; WANG GN; WANG YY; YU S.; REINER RC; BARBER R.; GAKIDOU E.; SI Hay; LIM S.; MURRAY C.; PIGOTT D.; GURUNG HL; BACCAM P.; STAGE SA; SUCHOSKI BT; PRAKASH BA; ADHIKARI B.; CUI JM; RODRIGUEZ A.; TABASSUM A.; XIE JJ; KESKINOCAK P.; ASPLUND J.; BAXTER A.; ORUC BE; SERBAN N.; ARIK SO; DUSENBERRY M.; EPSHTEYN A.; KANAL E.; LT Le; LI CL; PFISTER T.; SAVA D.; SINHA R.; TSAI T.; YODER N.; YOON J.; ZHANG LY; ABBOTT S.; BOSSE NI; FUNK S.; HELLEWELL J.; MEAKIN SR; SHERRATT K.; ZHOU MY; KALANTARI R.; YAMANA TK; PEI S.; SHAMAN J.; LI ML; BERTSIMAS D.; LAMI OS; SONI S.; BOUARDI HT; AYER T.; ADEE M.; CHHATWAL J.; DALGIC OO; LADD MA; LINAS BP; MUELLER P.; XIAO J.; WANG YJ; WANG QX; XIE SH; ZENG DL; GREEN A.; BIEN J.; BROOKS L.; HU AJ; JAHJA M.; MCDONALD D.; NARASIMHAN B.; POLITSCH C.; RAJANALA S.; RUMACK A.; SIMON N.; TIBSHIRANI RJ; TIBSHIRANI R.; VENTURA V.; WASSERMAN L.; O'DEA EB; DRAKE JM; PAGANO R.; TRAN QT; HO LST; HUYNH H.; WALKER JW; SLAYTON RB; JOHANSSON MA; BIGGERSTAFF M.; REICH NG
• Year of publication: 2022
• Type: Article in Periodical
• Magazine / Source: Proceedings of the National Academy of Sciences of the United States of America
• MU Faculty or unit: Faculty of Science
• web: http://dx.doi.org/10.1073/pnas.2113561119
• Doi: http://dx.doi.org/10.1073/pnas.2113561119
• Keywords: forecasting; COVID-19; ensemble forecast; model evaluation
• Description: Short-term probabilistic forecasts of the trajectory of the COVID-19 pandemic in the United States have served as a visible and important communication channel between the scientific modeling community and both the general public and decision-makers. Forecasting models provide specific, quantitative, and evaluable predictions that inform short-term decisions such as healthcare staffing needs, school closures, and allocation of medical supplies. Starting in April 2020, the US COVID-19 Forecast Hub (https:// covid19forecasthub.org/) collected, disseminated, and synthesized tens of millions of specific predictions from more than 90 different academic, industry, and independent research groups. A multimodel ensemble forecast that combined predictions from dozens of groups every week provided the most consistently accurate probabilistic forecasts of incident deaths due to COVID-19 at the state and national level from April 2020 through October 2021. The performance of 27 individual models that submitted complete forecasts of COVID-19 deaths consistently throughout this year showed high variability in forecast skill across time, geospatial units, and forecast horizons. Two-thirds of the models evaluated showed better accuracy than a naive baseline model. Forecast accuracy degraded as models made predictions further into the future, with probabilistic error at a 20-wk horizon three to five times larger than when predicting at a 1-wk horizon. This project underscores the role that collaboration and active coordination between governmental public-health agencies, academic modeling teams, and industry partners can play in developing modern modeling capabilities to support local, state, and federal response to outbreaks.