Most DVD-by-Mail Comparisons Are Flawed

Almost every week, some well-intentioned but clueless journalist releases the obligatory DVD-by-mail comparison story. The story is generally titled something like "Customers Win in War of Online DVD Rental Firms", “Who’s Winning the DVD War?”, "In the Competition for DVD Rentals by Mail, Two Empires Strike Back", “Netflix vs. Blockbuster”, etc.

The titles of the articles don’t matter, because the stories are all the same. The brilliantly original, hard-hitting investigative reporter always signs up with Blockbuster Online and Netflix and tracks the turnaround of DVDs for one month. The reporter then compares the numbers and draws conclusions about who is better. The results are almost always like this:

• Netflix has a better Web site
• Netflix processes DVDs a little faster
• Netflix carries NC-17 titles
• Blockbuster Online has more new releases
• Blockbuster Online processes DVDs on weekends
• Blockbuster Online provides free coupons for in-store rentals
  

The comparison stories always seem to wrap things up so cleanly and finally get to the bottom of the issue about who is better in the DVD-by-mail industry.

The problem, however, is that anyone who has tested Netflix and at least one other rental company knows the methodology used for most of these investigations is flawed. Anyone who has tried renting a lot of DVDs from Netflix for more than a few weeks in a row knows that Netflix begins applying throttling measures to limit the flow of DVDs after as little as one month of heavy use. Since Netflix throttles users, no comparison story is complete without explaining this concept.

The failure to address throttling is the major flaw in most of the comparison stories; however, the stories often have other major errors and glaring omissions that make the results useless. Please read these comparison stories with a critical eye, and keep in mind that no DVD-by-mail comparison story is complete unless it meets the following criteria:

1. The study covers a single time period lasting at least three continuous months.
2. The investigator took full advantage of each rental plan by returning DVDs as quickly as possible.
3. The investigator lives near distribution centers for all companies studied.
4. The investigator rents DVDs that are representative of the readers’ viewing habits. (For example, an investigation conducted for a mainstream newspaper should include a sample of DVDs that represents mainstream viewing tastes.)
5. The investigator details the fine points of each rental plan (e.g., throttling, plan limits, poor inventory quality, number of shipping days per week, etc.)
   

DVD-by-mail comparison stories have become a popular diversion for lazy journalists and provide the readers with misleading or useless information. As readers, we need to be demanding more. When you see these incomplete comparison stories, consider contacting the authors to let them know what they missed. Hopefully, journalists will start conducting some thorough investigations of these companies, and we can finally reach some real conclusions about where consumers can get the best value.