Author: Cem Paya

Cem Payahttps://randomoracle.wordpress.com/Building & breaking systems.
Cem Paya's avatar

Email storage and a lump of coal

TreeHugger is not the first to notice that computing technology can have environmental impact and different “systems” can be greener than others. In an invited talk at Microsoft Research in 2004, Andrew Shapiro from the Berkman Center and author of The Control Revolution raised the question of whether Linux could be deemed more environmentally friendly because it ran on lower-end hardware that would not meet the base requirements for modern Windows SKUs. (He was polite enough not to answer this question given the audience.) Similarly it is widely acknowledged that data centers today are gated by cooling and power consumption– air conditioning being one of the prime resource hogs– and availability of power generation is a significant factor in selecting “hot-spot” locations for building them.

TreeHugger post frets over the cost of email storage and wonders whether deleting email will curb carbon emissions. Good intentions for sure but the calculation may have been slightly off base for several reasons. First the bad news: storage in large-scale services like the one cites in the article are replicated. There can’t be just one copy of the message sitting around. Try explaining to a user that you lost all of their vacation pictures because drive #3385 failed– the so-called “we blame Seagate” approach.  That implies the figures are underestimating the true impact. That would be true only in a simplistic model where  power consumption scales with amount of data stored. Transaction capacity is often the determining factor for data center design. If one million people are checking email at the same time, enough servers have to be up and running to process those requests with tolerable latency. That’s true even if everyone keeps an empty inbox.

Similarly different storage architectures can lead to very different resource consumption patterns. If drives are directly attached to server, then more storage means more servers even if the servers sit idle CPU-wise. If the service uses a storage array network (SAN) then only drives are being powered and not all the extra baggage that would come with a full-fledged server. This is similar to the difference between using a networked drive at home verses another general purpose PC for handling backups. Finally there is the storage corollary to Moore’s law: disk sizes increase, price drops and so does power consumption per GB. (Unfortunately there is also a storage corollary to Peterson’s principle which states that data expands so as to fill the drive available.) It’s true that less storage will achieve some reduction but the Treehugger article probably overestimates this by several orders of magnitude. And if hosted cloud service were comapred to storing the same amount of data at home, there would be no contest: those massive data-centers achieve economies of scale and corresponding eco-efficiency not available to the average consumer not living off-the-grid with solar panels.

cemp

Old-school voting machines

Reminiscences from Robert Holt Jr.  who has been working as a voting-machine technician in New York city for over 20 yeras, as quoted by the New York Times:

“To tell you the truth, I like these machines. With all the problems they’re having with the computerized machines, these are solid. You can’t tamper with them.” 

“The ones who lost canvass the machines and see how many votes they lost by, machine by machine. Sometimes they come in angry. They’re upset they lost, but there’s nothing they can do. A loss is a loss. These machines don’t lie. What you see is what you get.”

Here is to machines that don’t lie and more importantly are transparent in allowing that alleged honesty to be verified by everyone.

cemp

NDSS, final day: “minding the gap”

(Trying to write about the conference before the recollections fade.)

Dan Kaminsky was scheduled to be the invited speaker on Wednesday morning , tentatively titled “On breaking stuff” but he was held up by consulting work at IOActive. Fortunately for the conference program committee, Paul van Oorschot volunteered to give a talk on short-notice and the result was the highly engaging “Security and usability: mind the gap” presentation.

He first started with some anecdotal evidence on the sad state of affairs in what should have been the poster child for usable security: online banking. One of the largest banks in Canada promised to refund 100% of losses resulting from unauthorized transactions– provided the user lived up to their side of the agreement. This fine-print in the customer agreement (granted nobody pays attention to that) makes for entertaining reading:

  • Select unique and not easy to guess password– and user will judge the quality of their password how? Windows Live ID has a password quality meter but this is far from being a standard feature.
  • Sign-out, logoff, disconnect and close the browser when done (What is the difference between first two? Disconnect means yank the network cable?)
  • Implement [sic] firewalls, a browser with 128-bit encryption and virus-scanning. As van Oorschot pointed out, the bank probably means “deploy” rather than “implement”– otherwise they would drastically narrow the potential customer base to developers with copious spare time for writing code from scratch for commodity purposes.

It only gets worse from there. The general pattern is promises of security and reassurance that damages will be covered in exchange for vague expectations of “secure behavior” form users who are often not in a position to accurately judge risks associated with their use of technology. Case in point: one study on malware found that 95% of users had heard of the word “spyware” and 70% banked online– yet some assumed that spyware was a good thing– 45% did not look at URLs and 35% could not explain what HTTPS meant. The status quo for online banking is not an isolated incident, as other case studies drawn from two recent publications van Oorschot coauthored:

  • An evaluation of Tor/Vidalia/Privoxy for anonymous browsing, which concluded that Tor is not ready from prime-time use by a novice even with the supposedly user-friendly Vidalia UI. (Given its remarkably low bandwidth and high latency reminiscent of the early “world-wide-wait” days of dial up, you have to wonder if a usability study was necessary to reach that conclusion.)
  • Usability study of two password managers with 26 non-technical users that found several problems, including situations where users falsely concluded a security feature was functioning when it was not– the very dangerous “false success” scenario. [Full disclousure: this blogger had reviewed and broke an earlier version of one, PwdHash.]

If poor usability is a security vulnerability as much as a flaw in a cryptographic protocol, what is the prescription? This is where the information security community is now wrestling with its collective conscience. van Oorschot made the frank observation that usability and HCI issues are routinely looked down upon by CS culture, not included in the traditional curriculum because they are  easy/trivial and better left to “people who can’t write code” to sort out. He raised the possibility that we had it wrong all the time: cryptography is the easy bit, secure system implementation is far more challenging and the hardest task is building usable secure systems.

cemp

NDSS, day II: Virtualization and security panel

The highly anticipated panel ended up taking a different turn. My colleague Tavis from Google could not attend, leaving DJ Capelis from UCSD as the only other skeptical voice to point out the risks from virtualization. (Recap: Last year Tavis found several problems in qemu, Virtual PC/Server and VMware virtualization platforms.) Intel was represented, and so was AMD with John Wiederhirn and Tal Garfinkel attended for VMware completing the  viewpoints at the table: hardware, virtualization platform, security research.

Most of the discussion implicitly focused on the server consolidation scenario, without spelling out the other uses of virtualization. Briefly consolidation scenario is about replacing multiple physical machines by a single powerful box that runs a VM with the equivalent OS/software configuration for each PC displaced. It sounds like re-arranging deck chairs but in fact this is a major cost saving opportunity for enterprise IT departments. A single powerful, expensive server hosting N virtual machines is by far easier to maintain than N low-end servers each running a different configuration. And in the long run the cost of maintenance dominates the original purchase cost of the hardware. Full machine virtualization creates new opportunities because it allows very clean consolidation between applications that could not otherwise live on the same bare metal: for example a legacy W2K3 line-of-business app alongside a new W2K8 terminal server or even Linux and Windows coexisting side-by-side.

This is the most commercially viable market for virtualization. VMware has been leading the charge and MSFT giving chase with Virtual Server R2 and the upcoming hypervisor in W2K8.  But focusing on it alone skewed the discussion, setting the stage for a predictable debate around trade-offs. Separate hardware is an isolation boundary: it keeps different applications from interfering with each other accidentally or by malicious logic. Virtualization is another one, as are operating system processes, BSD jails etc. Each one has an assurance level from security perspective or equivalently an attack surface. Server consolidation with a VMM involves changing the isolation boundary and creating new attack surface. There may be new channels for one VM to attack another when running on the same bare-metal, while separate boxes would have been confined to the network or shared storage etc. Quantifying that incremental risk and sharing opinions on whether it is a reasonable trade-off fueled much of the debate.

This is a comparison virtualization can not win on the single dimension of risk. Considering the extent VMs are used for malware research and quarantining untrusted code, it’s surprising that other applications were not considere. The flip-side of consolidation is sand-boxing: moving applications running in the same trust boundary to different VMs is a corresponding improvement– although the extent that it improve security is again debatable and dependent on the quality of implementation.

As a side note, the moderator raised a point about reduced customer choice: with individual machines one has a choice of different vendors to buy a network switch from. With the functionality of the switch subsumed in the software stack that choice goes away.

cemp

Dispatches from NDSS: Day I, breaking online games

Gary McGraw gave the talk “Breaking Online Games” at other conferences before, so this may be repeat material for some who have attended BlackHat or CSS in Washington earlier this year. (One difference is that apparently few security researches play World of Warcraft in the NDSS audience, neutralizing some of the gamer jokes.) At first the concept of cheating at online games seems out of place at a conference focused on fundamental security problems with a pragmatic bent: phishing, botnets, spyware, vulnerability research etc. But as McGraw pointed out there are two key observations making this topic very relevant:

1. MMORPGs foreshadow the future of massively distributed systems. World of Warcraft recently cracked 10M users (the slides had 8M, demonstrating how rapidly presentation material becomes outdated in this field) with up to half million online simultaneously.

2. There is real dollars at stake. Games like Linden’s Second Life– much smaller than WoW but far more visible in the media– have spawned a virtual economy that maps to transactions in the bricks-and-mortar economy complete with lawsuits. Even the devaluation of the dollar against foreign currencies such as the euro has a parallel in the going rate for gold coins. Cheating at online games then is about ill-gotten gains, a familiar theme for cybercrime.

The presentation itself was a broad overview of the security challenges in online games and stories of organized “exploit” opportunities it has given rise to, with references to the accompanying book. (There were also interesting digressions into eggregious EULAs, because it turns out World of Warcraft includes one to cover an ineffective anti-cheating solution that functions like spyware.) One implied conclusion is that designers for online games don’t in general grok the concept of security: traditionally it meant protecting the game against cracking and pirated distribution. The problem of contending with untrusted clients “outside the trust boundary” as McGraw puts it has not made it into the design philosophy.

cemp

Time-Warner cable and the value of reliability

Time-Warner cable experienced an outage in Internet access two weeks ago in New York that lasted almost a full day. The service is a joint-offering with Earthlink, so it is not clear where the blame goes. Such large scale service failures can happen: a number of undersea cables were cut in the Middle East, affecting net access in Egypt among other countries. The fact that this can happen in Manhattan of all places is another story. But even more disconcerting is the way Time-Warner believe customers should be compensated: by offering a refund for a single day which amounts to roughly 3% off the monthly bill. TW/Earthlink is trying to price reliability here, and that have significantly undervalued it.

No service can guarantee 24/7 uptime. But a service that is advertised with availability  99.999% of the time is not simply worth just another 2.999% over one that only works 97% of the time. It is far more valuable because at that limit diminishing returns have kicked in. Adding one more nine to the availability number requires a lot of investment. As the service-level guarantee increases, the system designers must contend with increasingly esoteric and improbable events. A very simplified example: a RAID array can ensure that a computer will survive a single drive failure– an event that happens with disturbingly high frequency for machines that are running under load all the time– by using multiple drives as redundancy. So if disks are fail 1% of the time and this is the most likely problem, 99% uptime is achieved by investing in improved storage solutions. But suppose there is a smaller 0.1% chance that the entire data-center can go up in smoke or the power can fail longer than the on-site generators can compensate. This is a lower probability event but being prepared is more difficult. Adding more drives  does not help because their failures are correlated. The same fire will take out all of them. Dealing with the less likely but more catastrophic event calls for building a brand new data center some place else and adding software logic to handle fail-over in case of an outage in the primary site, a much more expensive proposition.

Time-Warner assumed  that if customers are paying $30 for an almost always reliable service, they should have no problem paying a few percent less for one that experiences a massive outage every month. In fact Internet access advertised up front as working only 97% of the time would be worth much less and provide stronger incentives for customers to switch to an alternative such as fiber to the home.

Update: TW/Earthlink experienced another outage on Friday. This time they were apparently prepared: customers calling the support number were greeted with an automated recording announcing that New York was experiencing service problems. Meanwhile the otherwise reliable Verizon wireless access card had crawled to a halt when this blogger pressed it into service as a back-up, probably because other users had the same idea and Verizon did not expect to become the alternative broadband provider for a chunk of Manhattan.

cemp

Security, excuses and hidden agendas

Bruce Schneier has often commented on the tendency for hidden agendas to masquarade behind excuses for security. “For security reasons, we must do …” or “due to security concerns, we do not alow…” The classic example in Beyond Fear was the prohibition against bringing beverages into a baseball park: is it really about safety inside the park in the heightened awareness of 9/11 or a boost to the soft-drinks sales inside which goes to lining the club’s pocket at the end of the day?

The latest MSFT one-eighty around virtualization is starting to look like another one. To recap, in June last year MSFT announced that it was expanding virtualization options for Vista to allow Home Basic and Home Premium skews to run in a VM. This was shortly reversed by a change of course, now requiring users to fork for the more expensive business editions due to unstated security reasons.  More recently MSFT announced that it is again allowing  virtualization of the less expensive varieties. What to make of this? If this was a politician running for a coveted nomination on super primary Tuesday this type of change in policy would be understandable. Ruling that out, two other options remain:

  1. It was decided that customers can live with lower security assurances for the scenario. That is to say, after spending 5 years to ship the most secure version of Windows to date in Vista, break backwards compatibility and even sink untold amounts of R&D into inane, useless features such as UAC to prove this commitment, Microsoft is now letting go of a strategic advantage by allowing the operating system to be run in a vulnerable configuration.
  2. Security excuse was a ruse all along, intended to push customers towards more expensive Vista skews until the company itself could develop a proper response to the disruptive nature of virtualization.

#2 is looking like the smarter bet at the moment. It is not clear that virtualization is necessarily a short term revenue threat. Virtualized or not those copies of Windows must still be licensed. In other words the Mac user running Vista under Parallels of VMware Fusion is still paying for a full-license as if they had installed it natively. (Granted there might be a small uptick in piracy since pre-activated/genuine-advantage-validated VM images make for a convenient way to distribute pirated copies.) This scenario might be of greater concern to Dell or HP since it means that consumers have the option to purchase a Mac instead of a PC. Meanwhile server consolidation, the other major business case for virtualization is not affected by the Vista licensing arrangements because Vista is a client OS. Windows Server 2003 and 2008 are the relevant products for virtualized data-center environments, and it’s primarily the virtualization policies around these products that have to be carefully crafted to protect server business revenue.

Long term however there is a strategic threat. Parallels and VMware might be great for getting the best of both worlds from Linux/Mac + Windows but if Vista is increasingly seen as a “secondary” OS to run alongside a primary, purely for compatibility with applications written for the venerable Win32/64 API, it raises the question of how long before those applications can be finally ported to the other platforms so they do not need virtualization as a crutch. More than any short term risks around piracy or missed revenue from consumers opting for the inexpensive Vista skews, this is the great danger of undercutting the platform that MSFT has to contend with.

cemp

Blogging from an XO

The machine had arrived unceremoniously after New Year’s– it was sitting there in a box when we came back from vacation. Intended as a gift for my better half that arrived just a few days too late for the holidays, it was in time for all the controversy surrounding Intel parting ways with the OLPC group.

After installing the battery and charging for the first time, we had a chance to experiment briefly. Her initial impressions were that it was surprisingly unintuitive as far as user interface goes. This blogger agrees: after being used to a standard Windows/OS-X/Ubuntu system, the XO involves a steep learning curve. And that may be perfectly reasonable beause the true target audience for this laptop will be coming to the table with no pre-conceived notions of what a personal computer ought to look like. In that sense the XO is that rare opportunity for system designers: a chance to start with a clean slate, no backwards compatibility, not even the faintest worry about “sideways compatibility” to interop with the applications rest of the world is using, except for the ubiquitious web itself. Perhaps the only familiar moment aside from a stripped down web-browser was launching a command line shell to see which standard utilities were available. Python, ssh, grep: check. ifconfig, emacs, gcc: no dice.

One big problem initially was getting wireless networking. The graphical “neighborhood view” is a great way to visualize other peers and infrastructure access points but the XO could not associate with our DLink draft-N router. A quick Google search revealed that the particular build that ships with this version does not support WPA out of the box. Luckily a work-around was available in the form of a shell script that manually adds the information to config files.

After getting net access and trying out the other included applications, the XO sat on the shelf for a while until the blogger decided to borrow it for a test-drive today. Writing this post can be described in one word as frustrating. The keyboard is dimunitive, which is understandable considering it is designed for children. But it also lacks feedback because of the water-proofing  covering the entire layout in an uninterrupted sheet of plastic.  Biggest challenge to text-editing is that the system is awfully slow: it makes Vista feel like a streamlined catamaran by comparion. Of all things simple UI tasks such as typing and clicking should be the times when CPU speed does not matter. After all a user can’t “outclick” or “outtype” a modern CPU running at hundreds of megahertz. Apparently on the XO they can: there is noticeable delay between typing and having the words  appear in the WordPress edit box. (A problem aggravated by the fact that on an unfamiliar keyboard  half the time the first attempt at typing contains a typo.)

There is a lot more to write about the XO but it is clear that these future posts are best not authored on the XO itself.

cemp

Penny-slots and DRE voting machines

The recent debacle over allowing voting in casinos at the Nevada democratic primary brings to mind an earlier comparison between gambling and voting. During the height of the Diebold controversy, one of the computer scientists speaking at CFP 2004 pointed out that users of slot-machines have more confidence in the integrity of the machine that users of voting machines. The reason is that gaming commissions in gambling centers such as Las Vegas require the machines to be licensed and certified. This does not make the odds any better necessarily for the players but it means that the machine is designed to deliver exactly those odds posted consistently. No cheating by the house to skew bets, decrease probability of winning for larger amounts, different days of the week etc.

Until recently direct-recording electronic machines had very little oversight and the certification program only provided a cursory look. Case in point: Diebold was de-certified in California just in time for CFP2004 as the above parallel was being drawn.

Putting voting machines in the midst of slots is a fitting juxtaposition.

cemp

Self-negating advice on privacy

This suggestion from LifeHacker is unlikely to work. First it’s not all clear that the DNS names in question are affiliated with Google. The mappings can change and sending search-queries to random third party is hardly conducive to privacy. Second the threat model assumed here is a lost cause. Most enterprises control the computing environment used by their employees, right down to the software for web browsing. That means web history can be ferreted out of the client side, without having to sift through proxy logs or network traces. (Home user vs. over-reaching ISP is a better example.)

But there is another reason for the overwhelming futility of the idea: even if it were useful against the current crop of Big-Brother-ware because of an oversight in the URLs it logs, publicizing that blind-spot only ensures that the next versions are likely to fix the problem.

cemp