American Airlines flights came to a halt today due to computer issues. After talking to a few people at American, here’s my understanding of what happened.

• At 11am Central American began having computer connectivity issues.
  
• At 11:30am central they asked the FAA to initiate a ground stop at their Chicago, Dallas, and Miami hubs.
  
• By 12:30pm central the connectivity issue was resolved and systems began to come back online
  
• The ground stop was lifted at 1:15pm central

American Airlines Boeing 787 at Chicago O’Hare Airport

It wasn’t just flights that were affected. Check-in systems were affected. American’s website was affected.

Ultimately American delayed about 300 flights on average of an hour during the outage, and then ripple effects the rest of the day as the operation worked to catch up. Six flights were cancelled (these were regional jets out of Chicago, the airport gets congested and American had a lot of flights wanting to take off at the same time when systems came back online).

Miami International Airport Can Be Busy… But Not Outside Banks of Flights

Some flights did take off during the ground stop that were already in queue to depart. But for the most part once the systems started going down, you had to wait it out.

The issue was network capacity, an intermittent problem of bandwidth. It’s not clear as I write this what caused the slowdown, but with everyone trying to check in at the same time, board flights at the same time, in the places where that was happening the most the system clogged up. The American Airlines computer network started acting like it was running off of a single gogo inflight internet connection, at least at the three airports that were trying to push the most data at the same time.

The shutdown occurred in American’s busiest hubs, which are also the ones that have “re-banked” their flights meaning that they have arrivals coming in during a short period of time, and departures leaving at around the same time as well, to facilitate efficient connections. Their busiest hubs, with the busiest activity happening all at once, data slowed down and requests timed out.

• Legacy US Airways hubs weren’t affected other than flights from those airports to Chicago/Dallas/Miami.

• Other American hubs (Los Angeles and New York JFK) didn’t experience the problem – the timing of the issue occurred during relative lulls in activity at those two airports. (e.g. New York JFK is busy in the morning and between 3 and 8pm eastern)

• Other American airports weren’t affected, other than flights from those airports to Chicago/Dallas/Miami.

American Airlines Terminal at Miami International Airport

As best as I can tell at this point it wasn’t a problem with American’s reservation system, so unrelated to the pending combination of operations of US Airways and American coming October 17.

The challenge now is to determine what caused the bandwidth issue – hardware, programming error, etc.