Time-Warner cable and the value of reliability

Time-Warner cable experienced an outage in Internet access two weeks ago in New York that lasted almost a full day. The service is a joint-offering with Earthlink, so it is not clear where the blame goes. Such large scale service failures can happen: a number of undersea cables were cut in the Middle East, affecting net access in Egypt among other countries. The fact that this can happen in Manhattan of all places is another story. But even more disconcerting is the way Time-Warner believe customers should be compensated: by offering a refund for a single day which amounts to roughly 3% off the monthly bill. TW/Earthlink is trying to price reliability here, and that have significantly undervalued it.

No service can guarantee 24/7 uptime. But a service that is advertised with availability  99.999% of the time is not simply worth just another 2.999% over one that only works 97% of the time. It is far more valuable because at that limit diminishing returns have kicked in. Adding one more nine to the availability number requires a lot of investment. As the service-level guarantee increases, the system designers must contend with increasingly esoteric and improbable events. A very simplified example: a RAID array can ensure that a computer will survive a single drive failure– an event that happens with disturbingly high frequency for machines that are running under load all the time– by using multiple drives as redundancy. So if disks are fail 1% of the time and this is the most likely problem, 99% uptime is achieved by investing in improved storage solutions. But suppose there is a smaller 0.1% chance that the entire data-center can go up in smoke or the power can fail longer than the on-site generators can compensate. This is a lower probability event but being prepared is more difficult. Adding more drives  does not help because their failures are correlated. The same fire will take out all of them. Dealing with the less likely but more catastrophic event calls for building a brand new data center some place else and adding software logic to handle fail-over in case of an outage in the primary site, a much more expensive proposition.

Time-Warner assumed  that if customers are paying $30 for an almost always reliable service, they should have no problem paying a few percent less for one that experiences a massive outage every month. In fact Internet access advertised up front as working only 97% of the time would be worth much less and provide stronger incentives for customers to switch to an alternative such as fiber to the home.

Update: TW/Earthlink experienced another outage on Friday. This time they were apparently prepared: customers calling the support number were greeted with an automated recording announcing that New York was experiencing service problems. Meanwhile the otherwise reliable Verizon wireless access card had crawled to a halt when this blogger pressed it into service as a back-up, probably because other users had the same idea and Verizon did not expect to become the alternative broadband provider for a chunk of Manhattan.

cemp

Security, excuses and hidden agendas

Bruce Schneier has often commented on the tendency for hidden agendas to masquarade behind excuses for security. “For security reasons, we must do …” or “due to security concerns, we do not alow…” The classic example in Beyond Fear was the prohibition against bringing beverages into a baseball park: is it really about safety inside the park in the heightened awareness of 9/11 or a boost to the soft-drinks sales inside which goes to lining the club’s pocket at the end of the day?

The latest MSFT one-eighty around virtualization is starting to look like another one. To recap, in June last year MSFT announced that it was expanding virtualization options for Vista to allow Home Basic and Home Premium skews to run in a VM. This was shortly reversed by a change of course, now requiring users to fork for the more expensive business editions due to unstated security reasons.  More recently MSFT announced that it is again allowing  virtualization of the less expensive varieties. What to make of this? If this was a politician running for a coveted nomination on super primary Tuesday this type of change in policy would be understandable. Ruling that out, two other options remain:

  1. It was decided that customers can live with lower security assurances for the scenario. That is to say, after spending 5 years to ship the most secure version of Windows to date in Vista, break backwards compatibility and even sink untold amounts of R&D into inane, useless features such as UAC to prove this commitment, Microsoft is now letting go of a strategic advantage by allowing the operating system to be run in a vulnerable configuration.
  2. Security excuse was a ruse all along, intended to push customers towards more expensive Vista skews until the company itself could develop a proper response to the disruptive nature of virtualization.

#2 is looking like the smarter bet at the moment. It is not clear that virtualization is necessarily a short term revenue threat. Virtualized or not those copies of Windows must still be licensed. In other words the Mac user running Vista under Parallels of VMware Fusion is still paying for a full-license as if they had installed it natively. (Granted there might be a small uptick in piracy since pre-activated/genuine-advantage-validated VM images make for a convenient way to distribute pirated copies.) This scenario might be of greater concern to Dell or HP since it means that consumers have the option to purchase a Mac instead of a PC. Meanwhile server consolidation, the other major business case for virtualization is not affected by the Vista licensing arrangements because Vista is a client OS. Windows Server 2003 and 2008 are the relevant products for virtualized data-center environments, and it’s primarily the virtualization policies around these products that have to be carefully crafted to protect server business revenue.

Long term however there is a strategic threat. Parallels and VMware might be great for getting the best of both worlds from Linux/Mac + Windows but if Vista is increasingly seen as a “secondary” OS to run alongside a primary, purely for compatibility with applications written for the venerable Win32/64 API, it raises the question of how long before those applications can be finally ported to the other platforms so they do not need virtualization as a crutch. More than any short term risks around piracy or missed revenue from consumers opting for the inexpensive Vista skews, this is the great danger of undercutting the platform that MSFT has to contend with.

cemp

Blogging from an XO

The machine had arrived unceremoniously after New Year’s– it was sitting there in a box when we came back from vacation. Intended as a gift for my better half that arrived just a few days too late for the holidays, it was in time for all the controversy surrounding Intel parting ways with the OLPC group.

After installing the battery and charging for the first time, we had a chance to experiment briefly. Her initial impressions were that it was surprisingly unintuitive as far as user interface goes. This blogger agrees: after being used to a standard Windows/OS-X/Ubuntu system, the XO involves a steep learning curve. And that may be perfectly reasonable beause the true target audience for this laptop will be coming to the table with no pre-conceived notions of what a personal computer ought to look like. In that sense the XO is that rare opportunity for system designers: a chance to start with a clean slate, no backwards compatibility, not even the faintest worry about “sideways compatibility” to interop with the applications rest of the world is using, except for the ubiquitious web itself. Perhaps the only familiar moment aside from a stripped down web-browser was launching a command line shell to see which standard utilities were available. Python, ssh, grep: check. ifconfig, emacs, gcc: no dice.

One big problem initially was getting wireless networking. The graphical “neighborhood view” is a great way to visualize other peers and infrastructure access points but the XO could not associate with our DLink draft-N router. A quick Google search revealed that the particular build that ships with this version does not support WPA out of the box. Luckily a work-around was available in the form of a shell script that manually adds the information to config files.

After getting net access and trying out the other included applications, the XO sat on the shelf for a while until the blogger decided to borrow it for a test-drive today. Writing this post can be described in one word as frustrating. The keyboard is dimunitive, which is understandable considering it is designed for children. But it also lacks feedback because of the water-proofing  covering the entire layout in an uninterrupted sheet of plastic.  Biggest challenge to text-editing is that the system is awfully slow: it makes Vista feel like a streamlined catamaran by comparion. Of all things simple UI tasks such as typing and clicking should be the times when CPU speed does not matter. After all a user can’t “outclick” or “outtype” a modern CPU running at hundreds of megahertz. Apparently on the XO they can: there is noticeable delay between typing and having the words  appear in the WordPress edit box. (A problem aggravated by the fact that on an unfamiliar keyboard  half the time the first attempt at typing contains a typo.)

There is a lot more to write about the XO but it is clear that these future posts are best not authored on the XO itself.

cemp

Penny-slots and DRE voting machines

The recent debacle over allowing voting in casinos at the Nevada democratic primary brings to mind an earlier comparison between gambling and voting. During the height of the Diebold controversy, one of the computer scientists speaking at CFP 2004 pointed out that users of slot-machines have more confidence in the integrity of the machine that users of voting machines. The reason is that gaming commissions in gambling centers such as Las Vegas require the machines to be licensed and certified. This does not make the odds any better necessarily for the players but it means that the machine is designed to deliver exactly those odds posted consistently. No cheating by the house to skew bets, decrease probability of winning for larger amounts, different days of the week etc.

Until recently direct-recording electronic machines had very little oversight and the certification program only provided a cursory look. Case in point: Diebold was de-certified in California just in time for CFP2004 as the above parallel was being drawn.

Putting voting machines in the midst of slots is a fitting juxtaposition.

cemp

Self-negating advice on privacy

This suggestion from LifeHacker is unlikely to work. First it’s not all clear that the DNS names in question are affiliated with Google. The mappings can change and sending search-queries to random third party is hardly conducive to privacy. Second the threat model assumed here is a lost cause. Most enterprises control the computing environment used by their employees, right down to the software for web browsing. That means web history can be ferreted out of the client side, without having to sift through proxy logs or network traces. (Home user vs. over-reaching ISP is a better example.)

But there is another reason for the overwhelming futility of the idea: even if it were useful against the current crop of Big-Brother-ware because of an oversight in the URLs it logs, publicizing that blind-spot only ensures that the next versions are likely to fix the problem.

cemp

NYT on shoulder-surfing at Grand Central ATMs

Catching up to past Sunday papers; this one from the Metro section dated 1/12/2008: the Times reports on a series of ATM fraud incidents at Grand Central station. A series of unauthorized withdrawals are traced to individuals lingering around the bank of ATMs, shoulder-surfing for PIN numbers and then stepping over to the ATM when the legitimate owner walks away without ending the session.

Not surprisingly the root-cause is bad usability: it is not intutitive to the user when they have “logged out” of the ATM for lack of better expression. There are two main design options: In the first case the card is inserted and stays inside the ATM until the transaction is complete. End of the session is signaled by the machine spitting out the card, letting the customer know it is safe to walk away. (For quick withdrawals there is even an additional forcing factor to guide users: the ATM will first return the card and wait to dispense cash until the user has taken the card back.)  The Grand Central ATMs used a different model: the card is swiped and in order to end the session the user has to answer the question “Do you want another transaction?” The problem is that question stays up for 17 seconds according to the article, enough time for a crook to walk up to the ATM and dip into the other fellow’s funds. As for location, Grand Central is the perfect setting.  Chances are people are hurrying to get some place or catch a train, making it even more likely they will not notice the ATM asking a question after the primary task is complete.

cemp

Netflix, Apple and movie distribution over the Internet

Couple of announcements on movie distribution made the headlines recently:

  • At CES 2008 in Vegas, Netflix announced a partnership with LG to build a set-top box for streaming movies to consumers over the Internet. Movies will be free to existing Netflix subscribers, the only additional cost being the hardware.
  • Not to be outdone, Apple took the opportunity and preaching-to-the-choir environment of MacWorld to make a splash with its own take-two attempt at movie distribution, iTunes Movie Rentals.
  • Not to be outdone Netflix announced it was removing existing limits on streaming for subscribers– Netflix already had boasted a “watch now” feature where subscribers

What to make of these developments?

For Netflix users, it’s business as usual. This blogger’s account had its streaming limits lifted before the announcement, at least if the Netflix web page was correct about describing the program. Streaming works fine on a decent broadband connection already,  although the image quality is sub-par when projected on a TV and the software requires a Windows operating system because of its dependence on the DRM platform. (Also worked fine under Parallels on this bloggers’s Macbook Pro with Tiger.) Long-term trends in increasing bandwidth as well as availability of new options such fiber-to-the home means that the quality may improve to the point of being competitive with existing high-definition content options. Given that an average PC or laptop can easily feed a high-quality digital via DVI interface today (and some even boast HDMI output) the set-top has questionable value. At best it may be an all-in-one solution for consumers who are not tech-savvy but it’s hard to argue that learning how to connect a DVI cable to the TV is not worth the $$$ for the device. In all likelihood the hardware will be subsidized by Netflix and given away for free in exchange for binding contracts on an extended Netflix subscription– similar to the cell-phone/wireless plan model.

The main challenge for Netflix is the limited selection. While the main catalog for physical DVD distribution boasts tens of thousands of titles and current new releases, the “Watch Now” option limits viewers to 6000 titles, most of them ancient. It’s as if a record label decided to experiment with DRM-free downloads and started with the Perry Komo collection.

As for Apple, this is the second foray into movie downloads. Jobs admitted that the first time around was not very successful:

“We learned what people wanted was movies, movies, movies. […] We weren’t delivering that, so we’re back with Apple TV, take two.”

iTunes will charge $4 for new releases and $3 for the euphemistically named “library titles”(translation: dated junk featuring washed out movie stars from the 1980s) As with Netflix everything comes with the inane DRM baggage. Apple gives viewers 24 hours to finish the movie once downloaded, terms comparable to XBox Live movie downloads. At 640×480 image quality is hardly stellar but again there is room for improvement with an eye toward HD-quality in the future. Another significant disadvantage: iTunes requires download of the entire movie before it can be played. Netflix solution allows for streaming with intelligent buffering.

Ultimately the choice comes down to pricing models: Netflix is flat fee for all-you-can-download over a limited catalog that is likely to work better for independent film, documentaries and rehashed TV-series, as well as shoring up gaps in movie background– in case there is a friend who has not yet seen “The Clockwork Orange.” As back-up there is always the DVD arriving in the mail. iTunes is optimized for instant gratification over a more updated selection and a correspondingly higher price tag.

cemp

MSFT and One-Laptop-Per-Child

OLPC project is showing a pattern of tumultuous relationships with leading IT companies. In the wake of a widely publicized fall-out with Intel comes a disagreement with Microsoft over the meaning of “dual-boot laptops.” To recap:  news reports suggested that OLPC and MSFT were working on models of the XO that could run both the custom Linux operating system and garden-variety Windows. Later Microsoft firmly denied these rumors and suggested the company had a different vision than Negroponte for integrating the Windows platform into the XO system.

Hardly any surprises here because XO laptop and Windows are ultimately irreconcilable concepts. There is no question that earning the loyalty of future PC users in emerging markets is critical for the long-term success in the platform battle. It is important enough to justify giving away copies of an operating system at a loss or trying to co-exist in an open-source ecosystem. But this is going to be a difficult balancing act.

One-Laptop-Per-Child project started out with the goal of producing $100 devices at scale. Some SKUs of Vista cost more than that already. This is a glimpse into the  impending reality check for Windows: as the price of hardware drops and the licensing costs for the operating system begin to constitute ever increasing shares of that price, vendors and customers are increasingly motivated to search for alternatives. Cost is a huge factor for OLPC but so is energy consumption and CPU/memory resources– two things that Vista has a voracious appetite for. That’s good news for Intel, AMD and for that matter any company supplying PC components: as long as the software continues to peg capabilities of the hardware, improvements in hardware can make a meaningful impact on the overall user experience and justify the investment.  But the target audience for OLPC is not subject to the standard hardware upgrade cycles, nor expected to meet the minimum recommended specs for Vista.

Even if copies of a highly stripped down version of Windows could be made to run efficiently in the highly minimalist specs of the XO and given away for free (similar to the Starter Edition sold at a significant discount at emerging markets where even the basic SKUs are very expensive compared to standard earnings) it will not create a sustainable advantage. Converting those free copies into full-paying licenses down the road will be a challenge to the extent that the premium for a Windows PC over an open-source one is appreciable– exactly the situation guaranteed by Moore’s law and dropping hardware prices.

cemp

From the digital media front

Starting the year on a positive note:

  • On the last day of 2007, New York Times published an article about the University of Oregon resisting RIAA’s subpoena requests. In the Fight Over Piracy, a Rare Stand for Privacy points to the opposition from Oregon state Attorney General’s to RIAA request for student information. RIAA has been aggressively going after P2P file-sharing in higher-education. Quoting the article:

The recording industry may not be selling as much music these days, but it has built a pretty impressive and innovative litigation subsidiary.

Oregon AG is not taking a stand on the principle that file sharing should be legalized in all forms– that more extreme position, while espoused by EFF is unlikely to hold sway with the courts. Instead this is a more focused, tactical battle against the questionable approach used by RIAA in going after suspected file-sharers by pressuring colleges to work around due-process and presumption of innocence.

  • More labels announced support for publishing their catalog without DRM. Sony/BMG is the last label to get on the bandwagon; still a long way for a company that once root-kitted user machines in the name of content protection.
  • Better technology can succeed in the market: Warner may just have delivered the fatal hit to HD-DVD by throwing its weight behind Blu-Ray format pioneered by Sony. This new alignment brings everyone one step closer to the anticipated end of the high-definition DVD format wars. The 3% decline in DVD sales for the past year was in part being attributed to consumer reluctance to buy into a new format until the dust settled. Some companies such as Samsung tried capitalizing on the confusion by building dual-mode HD/Blu-Ray players but consumers balked at the price. Sony may have its revenge for losing the VCR format with BetaMax, which provided a textbook example of how a better technology (similar to BluRay having more storage capacity than HD-DVD) does not necessarily succeed in the marketplace against savvy deal-making. It sounds like Sony learned the lesson and aggressively pursued studios with heavy incentives for exclusive commitment to its favored format this time around.

cemp

Comcast: 350 is the magic number

Postscript to  a two part series on Comcast throttling upstream bandwidth [1, 2]: a quick email exchange and Google search suggests that other subscribers noticed similar problems. Trevin,  a former colleague at MSFT wrote in a private communication that when using a remote backup service in the cloud, bandwidth is also capped around ~350kbps. Aside from an isolated thread on Slingbox community forums, this does not appear to have been publicized widely.

cemp