Droid The Winner? Not So Fast, Now

Ars Technica just did a review of one of the latest Android OS mobile phones, Motorola's Droid. However, in their haste to praise the phone as not just an improvement over previous Android phones, like the HTC G1, they highlighted a performance comparison to Apple's iPhone 3GS-- and declared the Droid the winner.

The problem?

The data says otherwise. They assigned the phones average times for loading a series of pages, on which the iPhone averaged 8.0 seconds and the Droid averaged 9.3. Then they drew a graph, noted "longer bars are better" and claimed the Droid's score of 9.3 was better than the iPhone's 8.0.

Even in the article it says that the Droid is only faster on some mobile-optimized pages, but you have to jump through some hoops to see that result, too:

The article states:

The results are pretty obvious, and match the results of the synthetic benchmarks. The Droid is slower than the 3GS in Javascript-heavy pages, but is as fast if not faster on the mobile-optimized pages.

The problem is that this is unsupported by the data unless you are only considering the m.digg.com page "mobile-optimized".

On the averages of all loads, cached and uncached, of all pages, the iPhone comes out faster, 7.94 seconds to 9.31 seconds.

When you consider only the uncached loads, which would be a comparison not only of the two handsets but also of the networks they run on, the gap widens-- 10.49 for Droid and 8.56 for the iPhone, despite the general (and justified) belief that AT&T's network is inferior.

Even on the cached loads, though, the iPhone 3GS comes out ahead, averaging 7.33 seconds to the Droid's 8.122. In other words, leaving out the network and comparing only the handsets, the difference is narrower, but the iPhone is still faster. The Droid isn't "as fast or faster". It is, at best, as fast. Except when it isn't. When the Droid was faster it was almost imperceptibly faster, and when it was slower, it was a lot slower.

Even taking out those slow-loading sites, though, doesn't push the Droid over the top.

Exclude the Chicago Times and the Droid's average load drops from 9.31 seconds to 7.17. However, the iPhone 3GS' average drops from 7.94 to 6.95, still a tiny bit lower than the Droid's.

Take out the NYT also and Droid's average drops to 5.75 seconds-- and the iPhone's drops to 5.66.

This can in no way be accurately characterized as "as fast if not faster" even with the qualification that it refers to "mobile optimized sites"-- especially when the article does not define which ones those are. Perhaps the inference is that when the iPhone is faster, that's a bad, "javascript-heavy" site, and when the Droid is faster, that's a "mobile-optimized" site!

You have to remove both newspapers and Reddit before Droid's average pulls ahead-- and when it does, it's not by much. With all sites included the iPhone's advantage is about 1.3 seconds. With the newspapers removed, it's a dead heat, with a difference of less than 0.1 seconds. With the papers, reddit, and ars removed, Droid ekes out an advantage of .46 seconds, less than half of the iPhone's advantage over all sites loaded.

In other words, the Droid is only faster when it doesn't matter: when the two phones both load a site so fast that the difference is barely perceptible, the Droid might-- sometimes-- be a tiny bit faster (especially when loading from the cache).

Data from the Ars article plus additional calculations as cited here are in the attached Excel file.