Forest Service releases Aerial Firefighting Use and Effectiveness Report

The AFUE study began in 2012

Tanker 02, a BAe-146, dropping on the Elephant Butte Fire
Tanker 02, a BAe-146, drops on the Elephant Butte Fire southwest of Denver, July 13, 2020. Photo by skippyscage.com.

In 2012 the U.S. Forest Service began a study in 2012 to quantify the effectiveness of the various types of fixed and rotor wing aircraft in order to better justify the hundreds of millions of dollars spent annually by the Forest Service on firefighting aircraft. Their stated objective was to answer the question: “What are the best mixes of aircraft to do any fire suppression job?”

After missing all of their self-imposed deadlines to release annual “detailed fire suppression aircraft use summaries” beginning in 2017, the Forest Service only released two “Fact Sheets” of one to two pages each in 2017 and 2019.

Yesterday the Forest Service released what they are calling the final report. The 46-page “Aerial Firefighting Use and Effectiveness (AFUE) Report” claims to provide “[A] first-of-its-kind baseline information gathering and analysis effort regarding when and why aviation resources are utilized in responding to wildfires while evaluating mission completion performance. This information will be used to inform future analyses on operational use and fleet investment decisions.”

Pressure to collect firefighting aircraft effectiveness data

report released by the Government Accountability Office in 2013 about air tankers pointed out some of the same issues in a 2009 audit by the USDA’s Office of Inspector General. Both reports emphasized that the U.S. Forest Service and the Department of the Interior need to collect data about the effectiveness of air tankers and put together a coherent plan on the management of the fleet, as well as a plan for the acquisition and justification of additional aircraft.

In a 2019 Congressional hearing in which Forest Service personnel were asked about the delays in releasing the AFUE reports, it was explained that data was being collected by four “observation modules,” each comprised of three qualified firefighters and a dedicated aircraft, to collect ground and aerial data at wildfires throughout the nation during fire season. In addition to the 12 firefighters, 3 analysts/managers were tasked to evaluate the data. Christine Schuldheisz, a spokesperson for the USFS, has said the annual cost of the project is approximately $1,300,000.

In 2017 the agency spent over half a billion dollars on fire aviation — $507,000,000.

Objectives of the study

The Forest Service had the following goals for the AFUE when it began:

  • Determine the best mixes of aircraft to do any fire suppression job.
  • Determine the composition of the interagency wildland firefighting aircraft fleet.
  • Track the performance of specific aircraft types.
  • Assess the influence of the operational missions that drops supported and environmental factors that influenced outcomes.

One of the guiding principles in the final report was, “Transparency: AFUE reports clearly identified study strengths and limitations, including potential sources of error and bias.”

The timing of the report’s release

We have been asking the Forest Service about the report regularly. Stanton Florea, Fire Communications Specialist for the agency, told us four days ago on August 17, “The AFUE study remains in the review process with the Office of Management and Budget.” The Forest Service released the report dated “March 2020” this week, in the middle of a very busy fire season while Colorado has several large fires and California is experiencing what may be one of the busiest periods of firefighting in the state’s recorded history, based on daily active fire acres.

What is in the report released yesterday?

I don’t have time to study the nuances of the report now, and neither do many of the people from whom I would like to solicit their analysis of the document. So we will save the detailed impressions for later. In the meantime, here a few details.

The report is highly technical and uses jargon apparently invented for the study, or at least used in very narrow, specific ways. Here is how they introduced two terms which are ubiquitous throughout the 42 pages:

  • “Interaction Percentage: For a variety of reasons, not every drop interacts with a fire; AFUE only evaluated effectiveness for drops that did interact with fire. IP quantifies the proportion of drops that did interact with fire. Many drops provide utility and insurance by increasing line width or to anchor burnout operations, but do not end up visibly interacting with the fire. IP is computed as the number of drops with known outcomes that interacted with the main fire divided by the total number of drops with known outcomes.
  • “Probability of Success: POS is computed as number of effective drops divided by the total number of drops with known and interacting outcomes. This measure can be calculated for any set of conditions to see how success likelihood can vary with factors such as drop objectives, aircraft type, and fire type.”

The study broke down firefighting aircraft into the following categories

  • Helicopter: Types 1, 2, and 3
  • Water scooping airtankers: single engine and multiengine
  • Airtankers: single engine, large, and very large

(Note: the U.S. Forest Service for some reason has always chosen to spell “air tanker” as one word. Yet, they abbreviate Large Airtanker with “LAT”, and Very Large Airtanker with “VLAT”)

Here are two of many graphics in the document.

Aerial Firefighting Use and Effectiveness study report
Figure 17 — AFUE sample interaction percentage (IP) and probability of success (POS) results by aircraft in initial attack, 2015 to 2018. The IP for each aircraft is the result of dividing the sample counts of interacting drops by total of interacting (effective plus ineffective) plus those with no fire interaction, in other words the proportion of drops interacting with the main fire to all drops with known outcomes. Bands indicate the range between the worst and best cases possible, assigning all unknown outcomes as no fire interacting or fire interaction. The POS for each aircraft is the result of dividing the sample counts of effective drops by the total of effective plus ineffective drops, in other words the proportion of effective to all interacting drops. Bands indicate the range between the worst and best cases possible, assigning all unknown outcomes as either ineffective or effective.
Aerial Firefighting Use and Effectiveness study report
Figure 19 — AFUE sample interaction percentage (IP) and probability of success (POS) results by aircraft in large fire, 2015 to 2018. The IP for each aircraft is the result of dividing the sample counts of interacting drops by total of interacting (effective plus ineffective) plus those with no fire interaction, in other words the proportion of drops interacting with the main fire to all drops with known outcomes. Bands indicate the range between the worst and best cases possible, assigning all unknown outcomes as no fire interacting or fire interaction. The POS for each aircraft is the result of dividing the sample counts of effective drops by the total of effective plus ineffective drops, in other words the proportion of effective to all interacting drops. Bands indicate the range between the worst and best cases possible, assigning all unknown outcomes as either ineffective or effective.

That was a very, very brief glance at a small part of the AFUE report. Let us know what you think.

Forest Service needs to be more transparent while spending hundreds of millions contracting for firefighting aircraft

Fifteen large federal air tankers is not enough for the United States

CAL Fire air tanker 118 C-130
CAL FIRE air tanker 118 at Sacramento McClellan Airport. The Forest Service was given seven of these HC-130Hs formerly owned by the Coast Guard in order to convert them to air tankers, but lost interest and regifted them to the California Department of Forestry and Fire Protection. Photo by John Vogel March 4, 2020. CAL FIRE will be getting them in the air over the next couple of years.

–Opinion–

Now that the U.S. Forest Service has activated two Call When Needed air tankers, there will be 15 large and very large federal air tankers on duty.

For the United States.

In 2002 there were 44 on exclusive use contracts. After two air tankers crashed that year killing the five that were on board, the Forest Service weeded out the World War II aircraft and beefed up the safety standards. During the next three years the numbers dropped from 44 to 18, and kept falling until the fleet barely existed in 2013, leaving only 9.  The air tanker fleet has not been rebuilt — 18 years should have been sufficient time.

Usage of large air tankers, 2000-2019
Usage of large air tankers, 2000-2019. Revised 2-24-2020. Fire Aviation.

It is possible that the Forest Service will bring on more CWN tankers in the next month, but this year the agency will not disclose any information publicly about their aerial firefighting contracts that consume hundreds of millions of taxpayer dollars. Fire Director Shawna Legarza (during her last month in the job) and Forest Service Chief Vicki Christiansen need to shift out of their secret mode and be far more transparent. If they were proud of what they were doing it would be logical to make their decisions public. I would recommend an investigation by the Department’s Inspector General, but recently five IGs in the federal government have been fired and replaced with political lap dogs.

There needs to be accountability for how these huge decisions are made and how taxpayers’ dollars are being used. Are they being spent wisely? When will they release the Aerial Firefighting Use and Effectiveness Study that has been going on for eight years? Launched in 2012 at a cost of about $1.3 million annually, the study is supposed to quantify the effectiveness of the various types of fixed and rotor wing aircraft used on wildfires. In FY 2017 for example, the most recent year with exact numbers available, the agency spent over half a billion dollars on fire aviation; $507,000,000. If ever completed the AFUE study could make it possible to answer the question: “What are the best mixes of aircraft to do any fire suppression job?” Data collected from this study and other sources would be used to inform decisions about the composition of the interagency wildland firefighting aircraft fleet — to use the best, most efficient, and effective tools for the job.

In hearings before the Senate Energy and Natural Resources Committee in both 2018 and 2019 the Forest Service told the Senators the results of the study would be released “soon”. In another hearing in February of this year after Colorado Senator Cory Gardner, a Republican, asked when it would be made public, Chief Christiansen at first said “soon”, and when pressed by the Senator said it would be released before June, 2020.

This week I asked Forest Service spokesperson Stanton Florea when it would be released, and he said “soon”.  When I asked him again for a date, he said, “We expect to have it available soon, Bill.” They have learned they can get away with stonewalling Congress and taxpayers –and don’t care.

One knowledgeable person I talked with in D.C. thinks AFUE may never be released, which would not be without precedent. When the Forest Service did not like the recommendations in an air tanker study conducted by the Rand Corporation in 2012, they refused to release it, even after Wildfire Today filed a Freedom of Information Act request. Eventually the Rand Corporation made it public. If it is not released, Chief Christiansen and Director Legarza would be following the example set by former Fire and Aviation Director Tom Harbour about refusing to make taxpayer-funded air tanker studies public.

The leaders in the Forest Service, Senators, Representatives, and the personnel in the White House need to accept responsibility for the sorry state of our fixed wing air tanker fleet. They are the ones that introduce and pass legislation, or allow it to be introduced, that determines the amount of funding allocated for fire aviation. When they write letters, little is accomplished. Actions speak louder than a written word.

You can't fight wildfires on the cheap.

During the COVID-19 pandemic while our firefighters have one hand tied behind their backs, it is important to spend our money wisely and support our firefighters on the ground with rapid attacks on emerging wildfires using overwhelming force from both the air and the ground. (see Dr. Gabbert’s Prescription , June 26, 2012)

Here is an excerpt from an article I wrote March 19, 2020 in an article titled, “Fighting wildfires during a pandemic.”


In 2002 there were 44 large air tankers on federal exclusive use (EU) contracts. Last year and at the beginning of this year there are only 13. That is a ridiculous number even in a slow fire season like last year when 20 percent of the requests for large air tankers were unfilled. The number of acres burned in the lower 48 states in 2019 was the least since 2004.

There are so few large airtankers on EU contracts that dispatchers have to guess where fires will erupt and move the aircraft around, like whack-a-mole.

The U.S. Forest Service says they can have “up to” 18 large air tankers on EU contract, but that will only be possible if and when they finally make awards based on the Next-Generation 3.0 exclusive use air tanker solicitation that was first published November 19, 2018. There are an additional 17 large air tankers on call when needed (CWN) contracts that can be activated, but at hourly and daily rates much higher than those on EU.

If multiple large air tankers and helicopters could attack new fires within 20 to 30 minutes we would have fewer large fires.

40 Large Air Tankers

Congress needs to appropriate enough funding to have 40 large air tankers on exclusive use contracts. Until that takes place and the aircraft are sitting on ramps at air tanker bases, all 17 of the large air tankers on call when needed contracts need to be activated this summer. Right now, only one large air tanker is working.

50 Type 1 Helicopters

Several years ago the number of the largest helicopters on EU contracts, Type 1, were cut from 34 to 28. This number needs to be increased to 50. Until that happens 22 additional CWN Type 1 helicopters should be activated this summer.

We often say, “air tankers don’t put out fires”. Under ideal conditions they can slow the spread which allows firefighters on the ground the opportunity to move in and suppress the fire in that area. If firefighters are not nearby, in most cases the flames will eventually burn through or around the retardant. During these unprecedented circumstances brought on by the pandemic, we may at times need to rely much more on aerial firefighting than in the past. And there must be an adequate number of firefighters available to supplement the work done from the air.  It must go both ways. Firefighters in the air and the ground support each other.

Study shows correlation between rapid dispatch of air tankers and duration of wildfire

Data suggests duration of fires is shorter when air tankers are deployed early

C-130 air tanker retardant drop Canyon Fire California
A C-130 makes a retardant drop on the Canyon Fire in Napa County, California July 22, 2019. Photo by Kent Porter used with permission.

A study conducted by university researchers found that the speed of arrival of air tankers at a new fire is correlated with fires of shorter duration. Firefighters have known this for decades, but the use of data to confirm it has been lacking. It is a small step, until the eight-year Aerial Firefighting Use and Effectiveness study is released.

The research was commissioned by Global SuperTanker Services, the company that operates the 747 SuperTanker that can carry up to 19,200 gallons of fire retardant. Raw data about air tankers that were dispatched to 11,655 fires from 2014 through 2018 was acquired from the U.S. Forest Service by means of a Freedom of Information Act Request.

Keith L. Waters, Ph.D. and Stephen S. Fuller, Ph.D., of George Mason University who specialize in public policy and statistics, conducted the study. The factors they considered included the elapsed time between the first report of a wildfire and the arrival of air tankers at the fire. The duration of the fire was defined as the time between the first report and the arrival of the last air tanker over the fire.

Number Of AT Assignments Duration Of Fires air tankers

The study concluded, for example, that among 11,655 fires in which large air tankers were deployed, fires burned on average for less than one day when tankers were deployed in the first 4-6 hours of a reported fire. Fires in which tankers were deployed after 72 hours burned on average for more than 20 days.

wildfires Initial air tanker arrival

The researchers also analyzed “fires fought by the State of California”, and found that on the 6,278 fires, the California Department of Forestry and Fire Protection had air tankers over the fires within one hour of the first report 96.7 percent of the time. That compares to 37.9 percent of “fires not fought by the State of California”.

This is not a perfect study, of course, just considering fire start times and the arrival of air tankers at the scene, but the researchers were dealing with the limited information produced by the Forest Service as a result of the FOIA. It does not consider the fuels at the point of origin, the weather, availability of air tankers, time of day, ground forces assigned, helicopters working the fire, and other factors. But it does provide food for thought and a category of air tanker data that is not normally seen.

More detailed conclusions could be reached if, for example there were a dozen data collectors on the ground and in the air at numerous fires for eight years observing objectives and outcomes for individual retardant/water drops; terrain, slope, fuel type; fire spread characteristics; weather conditions and other environmental factors that may influence retardant drop effectiveness. In other words, exactly what the Forest Service has been doing in the still secret Aerial Firefighting Use and Effectiveness (AFUE) study that began in 2012.

The AFUE study is supposed to quantify the effectiveness of the various types of fixed and rotor wing aircraft when they are used on wildfires, in order to better justify the hundreds of millions of dollars spent by the Forest Service on firefighting aircraft.

In hearings before the Senate Energy and Natural Resources Committee in both 2018 and 2019 the Forest Service told the Senators the results of the study would be released “soon”. In another hearing this week before the Committee Forest Service Chief Vicki Christiansen again said it would be released “soon”. When pressed by Colorado Senator Sen. Cory Gardner, who last year made his opinion about the delay very clear, she said it would be released “this Spring”. Senator Gardner said, “Before June?”. She said, “Yes”.  A clip from that exchange is below.

Link to the entire hearing

If detailed, unfiltered, and unbiased results of the study are not released in June, the Committee could subpoena the information.


Dr. Gabbert’s prescription for keeping new fires from becoming megafires:

Rapid initial attack with overwhelming force using both ground and air resources, arriving within the first 10 to 30 minutes when possible.

Senators question why results from air tanker study have not been released

U.S. Forest Service Chief Vicki Christiansen testified before the Senate Energy and Natural Resources Committee

CL-415. Photo by LA County Fire Department.
CL-415, October, 2013. Photo by LA County Fire Department.

In addition to grilling the Chief of the Forest Service about hostile workplaces, several other issues were covered in a hearing Tuesday before the Senate Energy and Natural Resources Committee.

A video recording of the hearing is available at the Committee’s website. It begins at 19:48.

Senator Lisa Murkowski, Chairperson of the committee, said (at 1:39:30 in the video) that a year ago the committee was told by the Forest Service that results from the Aerial Firefighting Use and Effectiveness (AFUE) study would be released “soon”. Launched in 2012, the study is supposed to quantify the effectiveness of the various types of fixed and rotor wing aircraft when they are used on wildfires, in order to better justify the hundreds of millions of dollars spent by the Forest Service on firefighting aircraft. In FY 2017 for example, the most recent year with exact numbers available, the agency spent over half a billion dollars on fire aviation; $507,000,000. If ever completed and the results implemented, the study could make it possible to answer the question: “What are the best mixes of aircraft to do any fire suppression job?” Data collected from this study and other sources would be used to inform decisions about the composition of the interagency wildland firefighting aircraft fleet — to use the best, most efficient tools for the job.

However, to date no detailed reports have been released from the AFUE.

The Senator asked about the results of the study, now entering its eighth year. The data is being collected by four “observation modules,” each comprised of three qualified firefighters and a dedicated aircraft, to collect ground and aerial data at wildfires throughout the nation during fire season. In addition to the 12 firefighters, 3 analysts/managers were tasked to evaluate the data. Christine Schuldheisz, a spokesperson for the USFS, has said the annual cost of the project is approximately $1,300,000.

Chief Christiansen, referring to the lack of any detailed results being released, said, “I absolutely share your concern and your question….. I am low on patience as well, Senator. This is a complex and labor intensive endeavor.”

Senator Murkowski: “But should it really require seven years to get a report like this?”

Chief Christiansen: “To have enough, when you have to take these assessment teams and have to be on the fire scene and to get enough data to get what the trend line is, it does take some time.”

The Chief then referred to a very small amount of preliminary data that was released in a two-page document in March which in a vague manner referred to the probability of success of direct vs. indirect attack by aircraft. This was was reported by Fire Aviation April 8, 2019.

Senator Murkowski asked the Chief to have more details from the AFUE study when the Committee holds their annual fire outlook hearing in about a month.

Since after seven years the Forest Service has not released any significant data about the study, a person has to wonder what they have found that is so embarrassing, controversial, or perhaps critical of specific models of aircraft, retardant products, or vendors?

Some people think the Forest Service will never release the full results of the AFUE study.

The Committee might have to subpoena the data.

Later in the hearing (at 1:43:30) Colorado Senator Sen. Cory Gardner referred to the study, saying in his rapid-fire speaking style: “There is a technical term I want to use to describe the length of time it is taking to get that study done, and it is bunk! I’m sorry, it’s just a bunch of bunk that it has taken seven years to get this done. We fought a world war in four years, we built the Pentagon in 16 months, we can’t do a study in 2 years, 1 year, 3 years, 4 years, maybe 5 years? It has taken seven years to do this? In the meantime we have western states that have had significant and catastrophic fires. I understand it’s important to get the information right. But doggonnit, someone needs to get a fire lit underneath them to get something done on this study.”

Washington Senator Maria Cantwell mentioned very briefly in the hearing (at 59:00 in the video) the availability of CL-415 water scooping air tankers but the issue was not discussed. The Forest Service, even though funds are available and a vendor offered the usually very expensive aircraft at a greatly reduced rate this year in a meeting with Chief Christiansen and Fire Director Shawna Legarza (according to our sources), the agency does not plan to have any scoopers on exclusive use contracts for the second year in a row. Historically the FS does not hold scoopers in high esteem even though they are used extensively in Canada and Europe. The 2012 Rand Study, which the agency attempted to keep secret (and did so successfully for two years), recommended a heavy emphasis on water-scooping air tankers and fewer conventional air tankers which would have been a monumental shift in the paradigm.

On another subject, New Mexico Senator Martin Heinrich expressed concern that the Administration intends for both the Collaborative Forest Landscape Restoration Program (part of the Forest Service) and the Land and Water Conservation Fund (under the Department of the Interior) to be unfunded beginning in October. Again, the Chief mentioned that the White House directed the Forest Service to cut its overall budget by five percent.

Referring to the fact that the “fire fix” has reduced the necessity for the Forest Service to borrow funds from unrelated accounts to pay for fire suppression, Senator Heinrich said, “We’re giving you the tools, you’re not using the tools we are giving you.”

At 56:30 in the video Senator Cantwell asked Forest Service Chief Vicki Christiansen about the $545 million that was appropriated for fuel management in the recent omnibus legislation but was not mentioned in the administration’s proposed budget for FY 2020 which begins October 1. The Senator asked for assurances that the funds would still be available and would be used for that purpose. The Chief would not commit to the funds still being available, saying, “We will use whatever resources are given to the agency”.

The Chief reminded the Senator that the White House directed the Forest Service to cut its overall budget by five percent.

After seven years of the air tanker effectiveness study, what have we learned?

In FY 2017 over half a billion dollars was spent by the U.S. Forest Service on firefighting aircraft

A K-MAX helicopter drops water on the Comet Fire north of Salmon, Idaho July 28, 2016. Photo by Bill Gabbert.

This year the Aerial Firefighting Use and Effectiveness (AFUE) study that started in 2012 will begin its eighth season.

The Government Accountability Office in 2013, a 2009 audit by the USDA’s Office of Inspector General, and Senators and Congressmen have asked questions about justifying the taxpayer’s funds that are annually allocated for firefighting aircraft by the federal government. When asked if aircraft were worth the cost and if they were effective the answers from the land management agencies have been, “Yes”. How do you know? “We just do”. (I’m paraphrasing here).

According to the Administration’s FY 2020 budget summary, over half a billion dollars was spent on fire aviation in FY 2017; $507,000,000.

The U.S. Forest Service started the AFUE in an effort to answer the question: “What are the best mixes of aircraft to do any fire suppression job?” Data collected from this study and other sources would be used to inform decisions about the composition of the interagency wildland firefighting aircraft fleet.

After seven years of the study, which costs about $1.3 million annually, very little information has been released about the status of the effort or any detailed findings that have been developed. It is almost as if the Forest Service is less than enthusiastic about what they have discovered so far. In fact, a reliable source told us that one or more high-ranking folks in the agency want it to “go away” and that detailed findings would never be released. The USFS refused to release the $840,092 RAND air tanker study completed in 2012 even after we filed a Freedom of Information Act Request. Finally RAND released it two years after it was completed, but as far as we know the USFS never did. That study recommended a heavy emphasis on water-scooping air tankers and fewer conventional air tankers, which would have been a monumental shift in the paradigm.

When we asked Christine Schuldheisz, a Forest Service spokesperson, when a report from the AFUE study would be released, she said, “The USDA Forest Service has not released a report and currently the agency does not have a timeframe to release a report. The Forest Service is collecting data to provide adequate information for a report that will be released in the future.”

We asked if the Forest Service wanted the study to “go away”, and she said, “USDA Forest Service has no plans to discontinue the AFUE program at this time.”

In stories like this, we often include the disclaimer that air tankers do not put out fires. Under ideal conditions aircraft can slow a fire enough to allow ground based firefighters an opportunity to contain sections of the fire’s edge by constructing a fireline. Strong winds or dense smoke can make it impossible for aircraft to operate safely or effectively.

The data is being collected by four “observation modules,” each comprised of three qualified firefighters, as well as a dedicated aircraft, to collect ground and aerial data at wildfires throughout the nation during fire season. In addition to the 12 firefighters, 3 analysts/managers evaluate the data. Ms. Schuldheisz said the annual cost of the project is approximately $1,300,000.

After the first three years the AFUE, the Forest Service found that the data collected from 2012 through 2014 could not be used to provide statistically defensible analysis and results. After making the necessary adjustments to their procedures, they made a commitment to begin releasing detailed annual aircraft use summaries several months after each collection season. The annual reports were scheduled to begin in the early months of 2017 for 2015-2016. By now three reports should have been issued.

Ms. Schuldheisz told us that annual reports have not been released but she sent us a copy of a two-page “Fact Sheet” about the program that she said was sent to Congress in March, 2019. (Another one-page “Fact Sheet” was released in 2017.) The recent document includes information about data collection and the preliminary information shown below about probability of success.

AFUE air tanker study
* Direct: Any treatment applied directly to burning fuel such as wetting, smothering, or chemically quenching the fire. This includes drops adjacent to the active fire or with limited unburned fuels between the drop and fire edge. Whenever you hear the requestor suggest that the intent of the drops was half in and half out, select direct for tactic.
* Indirect: A method of suppression in which the control line is located some distance away from the fire’s active edge. Generally done in the case of a fast-spreading or high-intensity fire and to utilize natural or constructed firebreaks or fuel breaks and favorable breaks in the topography. The intervening fuel is usually backfired; but occasionally the main fire is allowed to burn to the line, depending on conditions. Source: US Forest Service.

The two-page Fact Sheet has some preliminary information from 2015 to 2017 with enough data to report with high confidence, Ms. Schuldheisz said.

  • Rotor-wing aircraft data indicates an 87% probability of success in direct attack drops, and 62% in indirect attack drops.
  • Fixed-wing aircraft data indicates a 74% probability of success in direct attack drops, and 56% in indirect attack drops.
  • Rotor-wing and fixed-wing have different mission profiles with a varying degrees of complexity. Both aircraft types fly direct attack missions the majority of the time.

When we asked how the researchers defined “success”, Ms. Schuldheisz replied:

Data is collected in multiple, nested scales which account for requestor objectives and then compare those to outcomes achieved at each scale and across various resource configurations.  The AFUE developed hierarchical data groupings of: Resource Actions, capturing information about individual drops; Tasks, to aggregate multiple, coordinated individual resource actions, over the course of one shift or less, in support of the task work assignment; and Campaigns, to group multiple aerial and ground tasks, working in concert, for a measurable amount of time, in a defined geographic area, supporting incident objectives. By documenting outcomes independently of objectives, effectiveness can be accurately determined.  To translate effectiveness into the observed probability of success, we divide the effective outcomes by the sum of effective and ineffective. Observed probability of success shows how often drops tested by fire meet or exceed their intended objective.

The Forest Service AFUE webpage includes these questions they hope to answer:

  • The best mixes of aircraft to do any fire suppression job.
  • Composition of the interagency wildland firefighting aircraft fleet.
  • Track the performance of specific aircraft types.
  • Assess the influence of the operational missions that drops supported and environmental factors that influenced outcomes.

We assume, with few details having been released by the Forest Service, that the study will collect data about four to five types of fixed wing aircraft (single engine, scooper, large, and very large) and at least three types of helicopters, Types 1, 2 and 3. Breaking it down by aircraft model and vendor would also be helpful. The type of fluid that is dropped should be recorded: water, long term fire retardant, or water with some other enhancement product.

If the study can determine the effectiveness of each of these seven types of firefighting aircraft, it should not only lead to answers about which ones are most effective, but also under what conditions of wind, terrain, fire behavior and fuel types they should be used.

Hopefully it will lead to answers to the questions from the GAO, Inspector General, and Congressmen about justifying the half billion dollars of taxpayer funds spent each year.

If the study can actually quantify the on-the-ground effective production rates of each type of firefighting aircraft, an analyst should then be able to develop a recommendation for how many of each type are needed nationwide and where they should be based.

And beyond that, algorithms or artificial intelligence could eventually, based on scientific data, make on-the-fly recommendations for which aircraft should be dispatched after a report of a new fire, based on availability of aircraft, aircraft production rates, location of the fire, fuel type, fuel moisture, terrain, scooping sites, location of reload bases, congestion at reload bases, weather, and predicted fire behavior.

At 10 a.m. EDT on April 9 the Senate Energy and Natural Resources Committee will hold a hearing “to examine the President’s budget request for the USDA Forest Service for Fiscal Year 2020.” It will be interesting to see if Chief Vicki Christiansen is asked questions about the AFUE. These hearings are usually live-streamed.

Gary (Bean) Barrett, a frequent contributor to the discussions on Fire Aviation, spent a career in U.S. Naval  Aviation as a fighter pilot and served on the Navy Staff as a program sponsor responsible for planning, programming, and budgeting. Here are some of his thoughts about the information that has been released so far about the AFUE:

This Australian air tanker effectiveness study defined results in terms of probability of success in meeting a common first [initial] attack objective of containment within 8 hours of detection. They didn’t try to differentiate between success and effectiveness in their report. They produced an excellent operationally useful study based on probability of suppression that begs for a follow-on study to compare different tanker types.

“Maybe the AFUE effort is suffering from excessive complexity by trying to address all the air tanker success, effectiveness, efficiency, and use questions on the first report. I would think it might be useful to get out an initial AFUE report with less complexity, get feedback, and then refine and expand it as more data becomes available each year. Here’s an example of an excellent report using partial data that produced an operationally useful document.

“AFUE might consider coming out with a partial report by focusing on IA objectives with basic variables. Simplify the process. If they are trying to get everything done in the first report, that just might be a bridge too far and the reason we haven’t seen any reports yet.”

Hard data on air tanker effectiveness still not available

Results have not been released from the Aerial Firefighting Use and Effectiveness study that began in 2012

BAe-146 drops on Devore Fire
BAe-146 drops on the Devore Fire, November 5, 2012. Photo by Rick McClure.

The federal government spends around $100 million each year on large air tankers. A reasonable person would hope that the results of a very careful analysis determine the performance specifications and effectiveness of aircraft that are needed to assist firefighters on the ground to the greatest extent possible while still being a careful steward of taxpayers’ money. If the effectiveness of air tankers can’t be quantified, how to spend that $100 million is left up to the gut feelings of decision makers.

In stories like this, we often include the disclaimer that air tankers do not put out fires. Under ideal conditions aircraft can slow a fire enough to allow ground based firefighters an opportunity to contain sections of the fire’s edge by constructing a fireline.

Government officials often mention the 2012 Large Airtanker Modernization Strategy as a blueprint. However, that document does not make any independent conclusion about the number or types of air tankers. If you wade through the footnotes it actually refers to a 2009 study that recommended increasing the number of large air tankers on exclusive use contracts from 19 in 2008 to 32 in 2018. In addition there would be three water scooping air tankers by 2018, bringing the total up to 35.

report released by the Government Accountability Office in 2013 about air tankers pointed out some of the same issues that were in a 2009 audit by the USDA’s Office of Inspector General. Both reports emphasized that the U.S. Forest Service and the Department of Interior need to collect data about the effectiveness of air tankers and put together a coherent plan on the management of the fleet, and a plan for the acquisition and justification of additional aircraft.

Due to these reports and repeated questions over the years by Senators and Congressmen, in 2012 the Forest Service began an effort to answer the question: “What are the best mixes of aircraft to do any fire suppression job?” Data collected from this study and other sources would be used to inform decisions about the composition of the interagency wildland firefighting aircraft fleet. The study was named the Aerial Firefighting Use and Effectiveness (AFUE) Study.

On a Forest Service web page the agency describes when data would be released:

AFUE study
Screenshot from www.fs.fed.us

Now that it is June, 2018 we should have seen the data from at least 2015, 2016, and 2017. But, it is not available.

When we asked Vicki Christiansen, the Interim Chief of the Forest Service, when the study’s results would be released, she responded by email, “AFUE personnel have been making excellent progress and continue to engage agency leadership on performance metrics, data collection, analysis and tech transfer processes to support a transition to an operational performance reporting system. Currently the program is funded until 2022.”

After we asked for more information, she wrote, “Summaries of the Aerial Firefighting Use and Effectiveness (AFUE) Study where planned for release in 2017. However, the summaries are not currently available. Unforeseen delays with staffing changes, retrieving aviation use data, and completing final reviews has delayed their overall schedule. The AFUE work group is continuing their work to complete the summaries and they will be provided as soon as they become available.”

Some would think that developing actual data to determine how to spend $100 million, year after year, should be a very high priority and would lead to finding solutions to staffing changes, retrieving aircraft use data, and completing reviews.

To our knowledge the Forest Service never did release the findings of an air tanker study conducted by the Rand Corporation in 2012 even after we filed a formal Freedom of Information Act Request. The report was finally released by Rand two years after it was completed. When awarding the $840,092 contract the Forest Service told the company to not consider Very Large Air Tankers at all in making their recommendations for how the air tanker fleet should be configured. The study found that “the most cost-effective fleet of initial attack aircraft is dominated by scoopers, but airtankers play a niche role, particularly in fires that are not close to appropriate water sources.” In one variant, Rand said, “the optimal fleet is composed of eight 3,000-gallon airtankers and 48 1,600-gallon scoopers”.

We heard from sources that the Forest Service was not pleased with Rand’s recommendations. It remains to be seen if the agency will release all of the data and conclusions from the AFUE study that is now in its seventh year.

Gary (Bean) Barrett, a frequent contributor to the discussions on Fire Aviation, spent a career in U.S. Naval  Aviation as a fighter pilot and served on the Navy Staff as a program sponsor, responsible for planning, programming, and budgeting. Here are some of his thoughts about determining the composition of a fleet of aircraft:


“It’s been my observation that if you don’t know how to derive your asset inventory objectives then you can’t explain or defend why you have chosen today’s particular inventory objectives.

“If you can’t determine the effectiveness of each type of asset, you cannot explain why today’s particular inventory mix was chosen or why certain trade-offs were made due to budget cuts.

“If you don’t know whether your primary mission requirement is Initial Attack or Extended Attack you cannot determine the mix of Exclusive Use contracts that can support IA, and Call When Needed contracts that take longer to get an asset on scene and would best support EA.

“Not much will change until:

1. The mission is clearly defined.
2. The effectiveness of each type of asset utilized for mission execution is known.

“At that point, budget impacts can be defined and dealt with objectively, contracts can be written that will provide the most mission effectiveness for the least cost, and the USFS will have definitive answers to questions about asset inventory, asset mix, and EU vs CWN contract mix.

“I still believe AFUE is the key to getting all of this off of bureaucratic dead center. Until you understand tanker effectiveness you cannot determine and justify inventory objectives.”

Video: Aerial Firefighting Use and Effectiveness study

Myrtle Fire air tanker p2v

Above: Air tanker 07, a P2V, on the Myrtle Fire, July 9, 2012. Photo by Bill Gabbert.

In 2015 and 2016 the Aerial Firefighting Use and Effectiveness (AFUE) study collected data on 7,000 drops of water or fire retardant on wildfires. This is partially in response to demands by Congress and the Government Accountability Office to provide actual data to justify the huge expense of using helicopters and air tankers on fires. Anecdotal information generally indicates that aircraft are effective under certain conditions to slow but not extinguish fires, however something more conclusive is needed when answering questions in front of a Congressional committee making decisions about allocating hundreds of millions of dollars.

And, we have not been able to determine, definitively, how many air tankers and helicopters are actually needed. The Forest Service likes to point to one of more than a dozen air tanker studies, the 2012 Large Airtanker Modernization Strategy, to answer this question, but it does not address the quantity of aircraft that are needed.

This video released today is an overview of the AFUE, which began in 2012.

For some reason NIFC never tells you what the acronym seen in the film, “WFSTAR”, stands for, but it is “Wildland Fire Safety Training Annual Refresher”.

Articles on Fire Aviation tagged “AFUE”.

Art Prints

Air tanker study has collected data on 7,000 drops

It will be several years before data is released about the effectiveness of aerial resources.

P2V air tankers

Above: P2V air tankers at Rapid City Air Tanker Base during the Myrtle Fire, July 21, 2012. Photo by Bill Gabbert.

The federal land management agencies spend many millions of dollars flying aircraft over fires dropping retardant or water. When Congress and the Government Accountability Office (GAO) ask how the numbers in the annual request for aerial firefighting funds were determined, they are often not satisfied with the answers, which may appear to come off the top of someone’s head. How many air tankers and helicopters do you need? How did you come up with those numbers? Are air tankers effective? How do you know?

After at least 14 studies on the use of air tankers since 1995, are the answers to these questions still scribbled on the back of an envelope?

The number of large air tankers on exclusive use contracts with the U.S. Forest Service has varied substantially in the last 15 years, from 44 in 2002 to 9 in 2013. In 2015 and 2016 there were 21 when the seasons started, plus approximately half a dozen or so on Call When Needed contracts in 2016. This year the numbers will not change much except for a few more that could be on CWN when a new round of contracts is awarded in a few months.

As we often say, aircraft do not put out fires. Under ideal conditions they can slow or temporarily stop the spread of a portion of a fire to enable ground personnel to move in and establish a fireline on the perimeter. Most wildland firefighters believe aircraft are an essential tool in their toolbox and can be very effective if used correctly. Those opinions are based on their experience on the fireground, however it is difficult to transfer that knowledge to decision-makers in Washington.

A report issued by the GAO in 2013 titled, Improvements Needed in Information, Collaboration, and Planning to Enhance Federal Fire Aviation Program Success (it is a very large file), included three recommendations:

  1. Expand efforts to collect information on aircraft performance and effectiveness to include all types of firefighting aircraft in the federal fleet;
  2. Enhance collaboration between the agencies and with stakeholders in the fire aviation community to help ensure that agency efforts to identify the number and type of firefighting aircraft they need reflect the input of all stakeholders in the fire aviation community; and
  3. Subsequent to the completion of the first two recommendations, update the agencies’ strategy documents for providing a national firefighting aircraft fleet to include analysis based on information on aircraft performance and effectiveness and to reflect input from stakeholders throughout the fire aviation community.

Under pressure from Congress and the GAO to justify the aerial firefighting program, in 2012 the U.S. Forest Service began a program to develop metrics and collect data to document and quantify the effectiveness of aircraft in assisting firefighters on the ground.

The new Aerial Firefighting Use and Effectiveness (AFUE) program gathered data from 2012 to 2014. Those first two years, USFS spokesperson Jennifer Jones told us, were preliminary to the full study:

That was done mainly as a methods development process and is not sufficient to provide statistically defensible analysis and results supporting the objectives identified by senior U.S. Forest Service leadership or GAO.

During the next two-year period, 2015 through 2016, data was collected on approximately 7,000 drops from more than 130 fires.

Mrs. Jones explained:

Since this data includes fires from many jurisdictions, fuel, weather and terrain conditions, the process of statistically characterizing the sample in terms of the population it represents requires merging data from many different sources. This work is ongoing, even for the 2015 data, but study management expects the process to be much quicker for subsequent years.

The USFS claims they have accomplished the first two items on GAO’s list regarding collaboration between the agencies. The last task is years away from completion. They plan to publish a peer-reviewed paper soon that will detail the methodology being used. Some early results of the study are expected later this year when they expect to release annual use summaries for 2015 and 2016 during 2017. Additional use summaries will come out several months following each data collection season.

After several more years when the sample size and statistical confidence increases, Mrs. Jones said, they expect to release findings related to the effectiveness and probability of success of aerial resources.

We asked Gary Barrett for his opinion about the AFUE study. Known as “Bean” to our readers, he is a former naval aviator and has contributed articles to this website. He brings a different background and point of view to the air tanker issue. Below are his comments:

****

“With data on 7000 drops on 130 fires over 4 years perhaps the AFUE program could have produced a report like this one [from Australia]. Or this one in the International Journal of Wildland Fire [from the US]  . Or this one [from Australia].

And if wildfire fighting in the US is being done by a combined integrated air ground team, why aren’t reports like this one available after our big fires?

Why is it that Australia seems to encourage ops analysis and its application to firefighting and here in the US we haven’t caught up with the concept. Until US ops analysis gets going, there will be no definitive answers on the utility of US air tankers and how they are utilized.

Even New South Wales in Australia has an opinion on the utility of heavy air tankers and has initiated a study on large air tankers operating in Australia.”

Sell Art Online