IdeaWorks and Switchfly are out again with their study of award availability. They purport to show which airline frequent flyer programs are offering the best award availability. And they’re getting lots of attention for it, such as this Wall Street Journal coverage.
This year the study even makes bolder claims. Here’s their press release title:
Value Airlines Fill Top 7 Slots with Best Reward Seat Availability, and Among Global Airlines Singapore is Best
Nowhere in the study do they account for the value or quality of what the low cost carriers like Southwest Airlines are getting you.
And while Singapore Airlines does have excellent award availability for members of their own Krisflyer program, there’s no comparison of award chart pricing (Singapore’s chart is certainly more expensive for comparable awards than most US carriers) and no comparison of out of pocket cash costs — like many Asian and European airlines they add fuel surcharges onto award tickets that add several hundred dollars to cost. Just because a seat is available more often doesn’t mean it’s a better value.
More fundamentally, though — and I wrote about the flaws in their study last year, too (and the year before) — they don’t seem to make their flawed study materially better, and every year the media still eats it up.
But the results are dead wrong, because the methodology is dead wrong. In fact, consumers will be worse off if they pay any attention to it.
Here’s how they go about their study:
Booking queries for a party of two travelers were made at frequent flier program websites during March 2013. Some airlines require a Saturday night stay for reward travel; all of the queries used date pairings that included a Saturday night stay. While the city pairs varied for each frequent flier program, the travel dates queried did not. 280 specific dates were selected for survey queries and only reward seat availability for travel on the date specified was recorded; any departure time was acceptable. Furthermore, reward travel had to be available on the outbound and return dates queried. Overly circuitous routings and layovers longer than 4 hours were not accepted.
The top 10 routes longer than 2,500 miles and the top 10 shorter routes were selected for each airline. Alaska Airlines was switched to this methodology for 2013 due to the carrier’s increased emphasis on the Hawaii market. Due to a lack of long-haul routes, the top 20 overall routes were queried for these airlines: Air Asia, AirTran, GOL, JetBlue, Southwest, and Virgin Australia. Ten top Europe – Palma de Mallorca city pairs (out of 20) were substituted for Air Berlin to reflect the carrier’s major Mediterranean emphasis on holiday flights. When offered, online reward availability for partner airlines was always requested; rewards fulfilled by calling the airline were not. Online access is important for consumers; a major US carrier disclosed more than 90 percent of its domestic reward bookings are made online. The survey is designed to focus on this important consumer attribute
This seems really problematic, as though it will yield strange results (which it does, as we’ll see in a moment):
- They searched airline websites only. So even though US Airways and United have access to the same award space, United offers online booking of many partners while US Airways does not. Miles in each airline’s programs can access the exact same saver award seats (with an immaterial number of technical differences, such as regarding Lufthansa and Singapore space).
- They’re searching different routes for each airline but over the same dates, which ignores the effects of high and low seasons. Since they’re looking at fixed months and days prior to departure for each airline’s most popular routes, airlines whose routes fall into high season during that date range are disadvantaged.
- They’re making subjective judgments about ‘overly circuitous routes’ but not about departure time, consistently offering 6am flights or redeyes counts just as much as offering times many consumers would find more desirable.
Their methodology also:
- Ignores cost of acquiring the miles. It may be really easy to earn Skymiles, so it could even make sense to spend twice as many Delta points. But Delta’s any seat availability for extra points doesn’t count, while Southwest’s does.
- Ignores the value of a given redemption. Greyhound Road Rewards may give you a free bus trip every 10 trips, and if those bus seats aren’t capacity controlled then they satisfy their riders every time. But that doesn’t make Greyhound Road Rewards a more lucrative, rewarding, satisfying program than United Mileage Plus or American AAdvantage which all you to see the world, in a premium cabin no less (a travel style many would never be able to afford but for the points!).
The conclusion that the low cost carriers with points-based programs where those points are good for any seat is misleading. The methodology focuses only on short haul routes for those carriers, while focusing on long haul routes for other airlines. And since they don’t face the same capacity controls their availability (“100%” for Southwest!) is compared against saver awards only at the legacy carriers — even though for additional miles you can book any seat using the American AAdvantage program. That’s just not comparing apples to apples. (Which doesn’t even get into the minutiae of replacing Air Berlin’s routes in the study with Europe-Palma Mallorca, if their ’emphasis’ on these routes were significant enough for the study why would they not be among the airline’s top 10 routes? And how would Air Berlin have done if 10 routes suggested by the study’s methodology hadn’t been replaced?)
The bias towards online only may not be surprising. The study is sponsored by Switchfly which sells online services to frequent flyer programs with a focus on using points as currency. Curious then that the study’s methodology focuses only on online redemptions, and (cough) concludes that airlines that are using their points as currency are among the best?
To be sure, if you want domestic coach redemptions,, then you might well be attracted to a points program. But then that implies other things as well – that you shouldn’t have a points-earning credit card, for instance, since a 2% cash back card will suit you best.
Ultimately, though, the study tries to prove too much with a methodology that doesn’t come close to supporting its conclusions. They go looking for strong value in low cost revenue-based frequent flyer programs and find it, without regard to what ‘value’ for the consumer really is. Folks following the advice of this study will find themselves with tickets to Florida and Ohio, while the legacy frequent flyer programs that score less well continue to offer greater options to their members. Don’t be fooled.
Clowns.
Does the WSJ guy still applaud their conclusions, the way he did a couple years ago when he sent me emails telling me how dumb I am for pointing out, especially, the WN errors?
This is just another example that you can use statistics to tell whatever story you want. Maybe that’s OK, more aspirational awards for us to utilize versus FLL or CLE. Agree with 2many books, McCartney is a clown.
CNBC aired a piece on this today as well. Very misleading.
I’m not fooled. As for SW, the crude analysis I would perform is as follows: (1) choose perhaps 6, 7 or 8 specific r/t flights using a mix of leisure type fares and their business type fares, calculating the number of RR points which would be generated from those flights. Then test 100 r/t flights, determining if the number of points earned above would be sufficient to get a “wanna get away ticket.”
The claim of no black out dates is really a bit preposterous, when, in fact, there are high demand dates which deplete an RR account as much as a “standard” award booking would deplete a more conventional FF program account.
Sounds like they need a few frequent/award flyers to advise them on what people in the know look for when they need award space. But I expect most people reading the WSJ article or watching CNBC will take it as gospel. I am sending this to my husband who on occasion advises students on survey methodology. Your critique is a great example of how assumptions can drive the conclusions.
It’s like saying that restaurant is great because the portion size is so big….even though it tastes horrible!
It’s a shame that so many outlets buy into this “study” – it’s only a marketing product designed to sell more of their consulting services to the points-as-currency programs.
As for Greyhound, there could be some real potential. Imagine, if you do cheap 10 r/t’s from Boston to Providence – the functional equivalent of a segment run, you can get an r/t Greyhound ticket from Boston to Los Angeles!!!!
I was a little surprised Scott McCartney from the WSJ did not point out that most of the top airlines for availability have revenue based programs– so there will always be seats available. I also don’t understand how Jetblue only had seats available 88% of the time since they have a revenue based program as well– unless they tested itineraries that were simply sold out.
If I was running the Southwest program, I’d be delighted by these results. It shows you the limits of “consumer intelligence.” If you package your offerings correctly, you may be able to offer worse value (and make more money) but make your customers happier. They should teach this in biz school.
We can all talk about the flaws in the study’s methodology, but the reason Southwest scores higher than the legacy carriers in this study is because Southwest’s program is simple and it’s competitors programs are not. For the folks reading this blog, we’re willing to put up with the complexity for the much better rewards. But are most travelers willing to do so? Probably not. For the major airlines, then, it comes down to the desires of the high revenue (aka “profitable”) passengers. If this group also prefers simplicity to generosity-with-complexity, they should also convert to a revenue-based frequent flyer program.
Phil LeBeau just did a quick CNBC story about this study. As is typical with TV news, there was no time to discuss methodology. He just reported that “Southwest had a good frequent flyer program, and the legacy carriers have bad ones.” Obviously, no one who disagrees with that opinion was offered a chance at rebuttal.
Pretty easy way to mislead the public, no?
Flawed testing methodology or self serving results for Switchfly does not matter too much. If the mainstream press believes it is representative of John Q. Flyer’s experience, it will be accepted as the gospel.
The average person earning a saver type coach ticket or two in a year (due to credit cards and occasional flights), will probably live the type of results the survey reported. Those who know how to game the system (readers of this blog, people on Flyertalk, etc.) can dismiss it, but it’s perceived as “real” to 85% of the public.
“but it’s perceived as “real” to 85% of the public”
The 85% was rounded from the “true” number: 84.693%.
😉
it’s important to rank them without each group – LCCs with variable redemption in one group, and long-haul carrier in typical fixed-miles redemption in another.
it’s a known fact that UA is the easiest, AA is middle, and DL/US are both pesos.
Given the fact that UA uses a traditional saver-award model, their rating of 80% of very impressive. Most of the time, even near-last-seat “Standard” awards have better cpm value than ones like Southwest or jetBlue.
The biggest surprise is how ridiculously generous SQ is to their own members (i’ve verified it with multiple TPAC searches in J/F/R), but impossibly stingy with partners (maybe it’s a causation effect – no awards for US-based mileage millionaires mean tons for themselves)
I don’t like how people bash on SW program because it doesn’t fit their needs or they don’t know the ins and outs of it. I personally have companion pass and don’t mind traveling locally in coach(hey, we weren’t born into flying first class only), and I find that most of their flights are cheaper to redeem, and they have way more availability than their competitors.
I consider AA to be the top program for me because it’s easy to go to SE Asia over Pacific, and the redemptions cost less in # of miles than UA which is a close second.
Delta is horrific whether study methodology is right or wrong, I like that message getting out any way possible
Too many factors in these programs to have a real “winner”. Every flyer has different priorities. No one mentioned that one can cancel a SWA reservation (paid or with points or Award) with no fee or penalty. I make reservations far ahead even if only a 20% chance I will go – then change or cancel if need be. Try that with other airlines.
I’m sure we’ll see these silly conclusions cited all over if and when Delta and the other majors gut their award charts with a “we heard you and we’re responding!” Of course this sort of free publicity is likely what the study is designed for in the first place.
Mainstream America generally prefers quantity over quality. Obviously Red Carpet Clubs are superior to The Wing in HKG or The Turkish Lounge in IST; there are more of them!
Hey, Gary, why don’t you do your own study? Just design it and then ask Steve B. to find some college students to do the leg work. Release a summary free and charge clients for the details and your recommendations.
You’re correct that this question is so complex that it will take someone of your caliber to identify the crucial variables and to obtain genuinely useful results.
Sure, the study may have flaws which you highlight well, but an apples-to-apples comparison rings true for me: UA has excellent availability (especially last-minute), while AA does not. I simply can’t use my AA miles, and won’t redeem for AAnytime Awards.
“Just because a seat is available more often doesn’t mean it’s a better value.” Totally agree. Actually, I do think it is because some US legacy airlines like AA and UA bring so much potential value to award tickets that made them less available. Considering CC bonus is not even close in most countries other than US, some awards availability from US carriers are so good and Miles and Points community has been so active that the awards seems relatively harder to get. But that is def not the carriers’ fault. I think the guys who wrote the report are just too greedy.
I worry that this kind of press and the general public’s ignorance to the power miles have if used properly will only encourage airlines to switch to a revenue model. This is easier for the consumer to understand but much less rewarding, im sure the airlines would be only too happy to oblige. Its will be a sad world when we are using Airpoint Dollars like they do in NZ
Now we know Delta’s motive for going revenue based; to move up this list!
NSX started the best conversation-when will someone take on reporting on availabilty using a valid measurement process. Virtually every imaginable cottage industry segment of FF programs has been tackled, so why not this. Probably because it is hard to do and to keep current, and the method to monetize is uncertain.
Not that I totally agree with the report. But as another poster stated, the vast majority of travelers (most whom are leisure) want simplicity when redeeming miles. Most don’t have the know how or time to learn the ins and outs of redeeming for long haul business or first class on the legacy airlines or its partners. And frankly most don’t care. What SW and the other low cost carriers offer is a pretty straight forward redemption process. Want to fly your family tommorrow, its going to cost you this much, want to fly to NY in September, it is going to cost this much more. Leisure travelers just want to be able to get the free ride without any complications and do it at a reasonable date and time. Seasoned travelers who value miles differently would not hesitate to cough up a $1000 fuel charge to be able to fly on LH’s or BA’s first class. Try convincing the average Joe that the quality of that first class experience is worth paying $1000 over his or hers $5 RT economy ticket to Florida. I agree that the report is simplistic in its approach but it is meant for an audience that is never going to read this blog or any other frequent flyer blog. It is meant for my neighbor who redeem all of his JetBlue points to fly himself and his family to Orlando last summer. He was surprised at the ease of redeeming the points and was equally delighted that he had only spent $20 instead of $1600.
The next time we have trouble with a gift card, VISA prepaid or Vanilla we might find the reason behind it in this story
http://www.nytimes.com/2013/05/10/nyregion/eight-charged-in-45-million-global-cyber-bank-thefts.html?smid=pl-share
The problem is that the companies that don’t follow suite get bad press. I would rather Delta just left their SkyMiles program as is. I can find reasonable availability to JNB in C, I’m OK with the fact that their domestic coach award availability is not good.
Can you point me to a “correct” study?
A slightly flawed study is better than no study at all, since they’re very transparent as to how they did it.