Homeland Security Grant Program:

Additional Actions Could Further Enhance FEMA's Risk-Based Grant Assessment Model

GAO-18-354: Published: Sep 6, 2018. Publicly Released: Sep 6, 2018.

Additional Materials:

Contact:

Chris P. Currie
(202) 512-8777
CurrieC@gao.gov

 

Office of Public Affairs
(202) 512-4800
youngc1@gao.gov

The Federal Emergency Management Agency distributes money to states, territories, and urban areas to help them get ready for potential terrorist attacks. The amounts provided are based in part on a model FEMA uses to rank the risk posed to each area. Locations deemed at higher risk of an attack with major consequences, for example, may receive more money.

We looked at FEMA's main grant programs and what it has done to improve its risk model. We recommended FEMA take additional steps to bolster the model's quality and credibility, including fully documenting the assumptions on which the model is based and subjecting it to external review.

 

This photo shows FEMA headquarters.

This photo shows FEMA headquarters.

Additional Materials:

Contact:

Chris P. Currie
(202) 512-8777
CurrieC@gao.gov

 

Office of Public Affairs
(202) 512-4800
youngc1@gao.gov

What GAO Found

GAO found that various factors affected Federal Emergency Management Agency (FEMA) State Homeland Security Program (SHSP) and Urban Area Security Initiative (UASI) grant awards from fiscal year 2008 through 2018. SHSP grant awards to states were based on two factors—(1) minimum amounts set in law each year, and (2) FEMA's risk model. For example, in fiscal year 2012, each state was to receive a minimum of approximately $2.74 million, with each state receiving additional funds based on its relative risk score. Conversely, UASI grant awards are made based on its FEMA's risk-based grant assessment model, which ranks each urban area relative to others in that year, and Department of Homeland Security (DHS) leadership decisions on how funding should be allocated. From fiscal year 2008 through 2018, the number of USAI grantees varied from year to year (see figure below).

Annual Number of Urban Area Security Initiative (UASI) Grantees for Fiscal Years 2008 through 2018

Annual Number of Urban Area Security Initiative (UASI) Grantees for Fiscal Years 2008 through 2018

Since 2008, FEMA has taken steps to strengthen its risk-based grant assessment model, but has not incorporated additional scientific practices into its model. For example, in 2011 FEMA included more information in its model on potential targets and their vulnerability in each state and urban area, addressing a prior GAO recommendation. More recently in 2018, FEMA added additional factors to better assess vulnerability in each state and urban area, such as the number of special events where large crowds gather and soft targets susceptible to lone wolf attacks, among other things. However, GAO found that FEMA does not fully utilize scientific practices recognized by the National Research Council and the Office of Management and Budget as best practices. Specifically, FEMA did not fully document its model's underlying assumptions, such as the weights in its model or the justification for changes to these weights. FEMA also did not perform the level of analysis needed to determine how changes to its model could affect the resulting risk scores. Finally, FEMA has not coordinated an independent external peer review of its model. Applying such scientific practices could assist FEMA in further strengthening its model.

Why GAO Did This Study

FEMA, a component of DHS, provides preparedness grants to state, local, tribal, and territorial governments to help prepare for, prevent, protect against, respond to, recover from and mitigate terrorist attacks or other disasters. SHSP grants fund the nation's 56 states and territories, while UASI grants fund eligible urban areas. Grant allocations have been based, in part, on FEMA's risk-based grant assessment model, with states and urban areas deemed to be at higher risk receiving more grant dollars than those deemed at lower risk. Since 2008, GAO and others have assessed the model and made recommendations to strengthen it.

This report 1) describes SHSP and UASI grant awards during fiscal years 2008 through 2018, and factors affecting grant distributions; and 2) examines the steps that FEMA has taken to strengthen its risk assessment model for allocating preparedness grants, and any additional opportunities to improve the model. GAO analyzed the information in FEMA's model, and data on SHSP and UASI grant awards for fiscal years 2008 through 2018. GAO also interviewed FEMA and DHS officials and collected documents.

What GAO Recommends

GAO is making three recommendations to FEMA to further strengthen its risk-based grant assessment model by (1) fully documenting the model's assumptions and justifications, (2) performing additional in-depth analyses, and (3) coordinating an external peer review. FEMA concurred with our recommendations.

For more information, contact Chris P. Currie at (202) 512-8777 or 20CurrieC@gao.gov.

Recommendations for Executive Action

  1. Status: Open

    Comments: Following the release or this GAO report, FEMA officials provided additional documentation, noting that FEMA maintains specific risk methodology folders and files for Fiscal Year 2012 to Fiscal Year 2019 (in preparation for the next grant year). According to FEMA officials, these grant program sub folders for each year contain files showing the weight makeup of the data elements, data sources documents, as well as information on the allocations for each year and MSA footprint information, and historical folders of previous allocations, model structures, risk profiles, and other risk methodology summary documents. We recognized FEMA's stakeholder feedback efforts in this report. However, as we noted, FEMA's documentation on the sources of data used for the model's calculations does not include information that would enable a reviewer to understand the underlying assumptions that form the basis for its risk-based grant assessment model. Pending issuance of additional information from FEMA, this recommendation will remain open. In order to fully implement this recommendation, documenting how subject matter expert assumptions are made would help FEMA increase the transparency of the model for key internal and external stakeholders, and will further support the efforts of an independent external peer review of FEMA's risk-based assessment model. As a result, this recommendation will remain open.

    Recommendation: The FEMA Administrator should fully document the underlying assumptions and justifications that form the basis of the risk-based grant assessment model, such as the size of the weights assigned to Threat, Vulnerability, and Consequence, or the justification for changes to these weights from one year to the next.

    Agency Affected: Department of Homeland Security: Directorate of Emergency Preparedness and Response: Federal Emergency Management Agency

  2. Status: Open

    Comments: FEMA concurred with our recommendation, stating that the agency will expand the use of sensitivity analysis to review the entire risk methodology, and will also document these results for leadership review, as appropriate. Understanding the extent that uncertainty has on the results of the model can be especially important if the model produces materially different results in response to even small changes in assumptions-often referred to as the "sensitivity" or "robustness" of a model's assumptions and results. Developing a greater understanding of the how uncertainty affects its risk-based grant assessment model's results helps achieve the objectives of risk management. This recommendation will remain open.

    Recommendation: The FEMA Administrator should perform sensitivity analyses to verify how changes to the risk-based grant assessment model could affect the resulting risk scores, and document the results.

    Agency Affected: Department of Homeland Security: Directorate of Emergency Preparedness and Response: Federal Emergency Management Agency

  3. Status: Open

    Comments: FEMA concurred with our report recommendation, stating that they will coordinate an independent external peer review and develop a detailed written response to leadership for further appropriate action. This recommendation will remain open.

    Recommendation: The FEMA Administrator should take steps to coordinate an independent, external peer review of its risk-based grant assessment model.

    Agency Affected: Department of Homeland Security: Directorate of Emergency Preparedness and Response: Federal Emergency Management Agency

 

Explore the full database of GAO's Open Recommendations »

Dec 13, 2018

Dec 6, 2018

Nov 29, 2018

Nov 28, 2018

Oct 11, 2018

Sep 21, 2018

Sep 19, 2018

Sep 4, 2018

Aug 16, 2018

Looking for more? Browse all our products here