IdeaWorks and Switchfly are out again with their study of award availability. They purport to show which airline frequent flyer programs are offering the best award availability. And they’re getting lots of attention for it, such as this Wall Street Journal coverage.
This year the study even makes bolder claims. Here’s their press release title:
Value Airlines Fill Top 7 Slots with Best Reward Seat Availability, and Among Global Airlines Singapore is Best
Nowhere in the study do they account for the value or quality of what the low cost carriers like Southwest Airlines are getting you.
And while Singapore Airlines does have excellent award availability for members of their own Krisflyer program, there’s no comparison of award chart pricing (Singapore’s chart is certainly more expensive for comparable awards than most US carriers) and no comparison of out of pocket cash costs — like many Asian and European airlines they add fuel surcharges onto award tickets that add several hundred dollars to cost. Just because a seat is available more often doesn’t mean it’s a better value.
More fundamentally, though — and I wrote about the flaws in their study last year, too (and the year before) — they don’t seem to make their flawed study materially better, and every year the media still eats it up.
But the results are dead wrong, because the methodology is dead wrong. In fact, consumers will be worse off if they pay any attention to it.
Here’s how they go about their study:
Booking queries for a party of two travelers were made at frequent flier program websites during March 2013. Some airlines require a Saturday night stay for reward travel; all of the queries used date pairings that included a Saturday night stay. While the city pairs varied for each frequent flier program, the travel dates queried did not. 280 specific dates were selected for survey queries and only reward seat availability for travel on the date specified was recorded; any departure time was acceptable. Furthermore, reward travel had to be available on the outbound and return dates queried. Overly circuitous routings and layovers longer than 4 hours were not accepted.
The top 10 routes longer than 2,500 miles and the top 10 shorter routes were selected for each airline. Alaska Airlines was switched to this methodology for 2013 due to the carrier’s increased emphasis on the Hawaii market. Due to a lack of long-haul routes, the top 20 overall routes were queried for these airlines: Air Asia, AirTran, GOL, JetBlue, Southwest, and Virgin Australia. Ten top Europe – Palma de Mallorca city pairs (out of 20) were substituted for Air Berlin to reflect the carrier’s major Mediterranean emphasis on holiday flights. When offered, online reward availability for partner airlines was always requested; rewards fulfilled by calling the airline were not. Online access is important for consumers; a major US carrier disclosed more than 90 percent of its domestic reward bookings are made online. The survey is designed to focus on this important consumer attribute
This seems really problematic, as though it will yield strange results (which it does, as we’ll see in a moment):
- They searched airline websites only. So even though US Airways and United have access to the same award space, United offers online booking of many partners while US Airways does not. Miles in each airline’s programs can access the exact same saver award seats (with an immaterial number of technical differences, such as regarding Lufthansa and Singapore space).
- They’re searching different routes for each airline but over the same dates, which ignores the effects of high and low seasons. Since they’re looking at fixed months and days prior to departure for each airline’s most popular routes, airlines whose routes fall into high season during that date range are disadvantaged.
- They’re making subjective judgments about ‘overly circuitous routes’ but not about departure time, consistently offering 6am flights or redeyes counts just as much as offering times many consumers would find more desirable.
Their methodology also:
- Ignores cost of acquiring the miles. It may be really easy to earn Skymiles, so it could even make sense to spend twice as many Delta points. But Delta’s any seat availability for extra points doesn’t count, while Southwest’s does.
- Ignores the value of a given redemption. Greyhound Road Rewards may give you a free bus trip every 10 trips, and if those bus seats aren’t capacity controlled then they satisfy their riders every time. But that doesn’t make Greyhound Road Rewards a more lucrative, rewarding, satisfying program than United Mileage Plus or American AAdvantage which all you to see the world, in a premium cabin no less (a travel style many would never be able to afford but for the points!).
The conclusion that the low cost carriers with points-based programs where those points are good for any seat is misleading. The methodology focuses only on short haul routes for those carriers, while focusing on long haul routes for other airlines. And since they don’t face the same capacity controls their availability (“100%” for Southwest!) is compared against saver awards only at the legacy carriers — even though for additional miles you can book any seat using the American AAdvantage program. That’s just not comparing apples to apples. (Which doesn’t even get into the minutiae of replacing Air Berlin’s routes in the study with Europe-Palma Mallorca, if their ’emphasis’ on these routes were significant enough for the study why would they not be among the airline’s top 10 routes? And how would Air Berlin have done if 10 routes suggested by the study’s methodology hadn’t been replaced?)
The bias towards online only may not be surprising. The study is sponsored by Switchfly which sells online services to frequent flyer programs with a focus on using points as currency. Curious then that the study’s methodology focuses only on online redemptions, and (cough) concludes that airlines that are using their points as currency are among the best?
To be sure, if you want domestic coach redemptions,, then you might well be attracted to a points program. But then that implies other things as well – that you shouldn’t have a points-earning credit card, for instance, since a 2% cash back card will suit you best.
Ultimately, though, the study tries to prove too much with a methodology that doesn’t come close to supporting its conclusions. They go looking for strong value in low cost revenue-based frequent flyer programs and find it, without regard to what ‘value’ for the consumer really is. Folks following the advice of this study will find themselves with tickets to Florida and Ohio, while the legacy frequent flyer programs that score less well continue to offer greater options to their members. Don’t be fooled.