Credit rating system and meaningful choice

A story from NYT Real Estate section about a British expat’s search for an apartment in Manhattan reads on different levels. Describing the interaction with a real estate agent:

“Almost by way of small talk, she said ‘Where are you from’  and I said ‘I’ve just come over from London yesterday,’ … She asked whether he had a credit history in the Unites States or a bank account or a Social security number, all of which he would need to rent an apartment. No, no, no. … But his employer would provide initial financing and act as guarantor.”

What would be the expected response from the realtor? In this case walking out on the client:

“She completely lost interest and just left,” leaving him standing on the pavement.

Welcome to the Big Apple. It would be easy to dismiss this as yet another rude-awakening in the ways of Manhattan for a new arrival– an experience this blogger can relate to. But there is a more subtle point about the pernicious growth of credit rating systems here. It’s not an oversimplification to say that without a social security number, a US consumer is just a nebulous and largely invisible presence in the eyes of lenders.  Most of the data compiled by data-brokers such as Acxiom, Choicepoint and the more familiar credit-reporting bureaus such as Experian and TransUnion are indexed by the SSN. To oversimplify in database terms one could say SSN is the primary key to the database. In this case the expression “key” is quite appropriate because it unlocks all the reputation information required for a significant transaction: buying a car, leasing an apartment, even getting a cell-phone contract. With the credit history available, consumers stop being blanks faces, they acquire useful numbers: Alice has 700 FICO score, Bob has an 8-year mortgage in good standing etc. Everyone is now a three-dimensional character jumping out of the page, shrouded in precise numbers.

One of the arguments in defense of massive data collection is that it enables credit: individuals can go anywhere around the country and still enjoy the same access to credit as if they lived in a small-town where everyone knew first-hand about their impeccable track record in paying back debts. (The flip-side, never mentioned in the same sentence, is that nobody can start over: the scarlet letter of bankruptcy or foreclosure also follows people around. It is true that in this case there are no second acts in American life.)  The more wide-spread and inflexible our reliance on credit history, the more difficult it is to get started and the greater discrimination between those who have an extensive dossier verses those with a blank slate. NYC may be an extreme example. In keeping with its completely ludicrous and preposterous state of affairs, some landlords demand to see bank statements,  employment verification on official company letterhead and even past tax returns before approving a lease. But stories like the one above are far from unique: if the agent had any shred of common sense, she would have realized that a decent sized company–implied by having offices in London and New York– as a guarantor is much better than one would expect to get from most consumers: while individuals can go bankrupt or disappear, a company with deep pockets can be litigated to the last penny. The story did have a happy ending because at least one rental agency was sane enough to accept his application with six month deposit– but only after running a credit check on this person’s manager. There is no escaping the system.

cemp

Choosing the wrong side in a format war

MSFT finds itself in this situation after the HD-DVD format it backed was finally consigned to the dustbin of history after Toshiba announced that it will stop producing the players. This was a domino effect, starting with the studious announcing Blu-Ray exclusive production, Netflix switching and finally WalMart saying the last word.

That leaves the question of what to do with all those XBox 360s with HD-DVD drives which are going to be about as useful as a brick in a few years. In fact the decisive and abrupt BluRay victory has just created a large collection of expensive and useless gadgetry overnight. Consider the dual-mode Samsung players that could play both HD-DVD and BluRay, in an uneasy truce to allow customers to hedge their bets on the war. With a clear winner emerging from the format war, all of the effort goes out the door. On the bright side Samsung will fare better than the HD-DVD camp because the company itself hedged its bets.

There is going to be frustration among the early adopters who guessed wrong– but that’s the cost of doing business on the leading edge. Just ask the initial round of iPhone buyers after the price drop. Long term consumers are probably better off because standardization will increase sales of players by removing the cloud of uncertainty. More players will drive down costs, and increase availability of content. It may also cement Sony as the new hegemon unseating the reigning oligarchy of the DVD Forum, depending on how the licensing around patents and royalties for use of BluRay technology are structured.

cemp

The conscience of a mutual-fund manager

“Upon reflection it doesn’t take long to realize that we were living for more than two decades in the Age of Decadence. This decadence was so prevalent that everyone from the government down to the regular citizen was an accomplice. During this period we saw America continually make the wrong decisions, lose its industrial might, damage its national balance sheet, and erode the reserve status of its currency.”

This could have passed for a stump speech by an aspiring politician sharpening his/her rhetorical skills for November. Instead it comes from the opening paragraph to the annual report for a mutual fund. The private Swiss bank Julius Baer is more likely to make the headlines these days because its role in shutting down the controversial Wikileaks website than any flourish with prose. Yet a quick peek at the report covering the period ending 10/31/07 reveals a different side of the culture.

Mutual fund reports and statements are invariably written in a dry, legalistic language designed with only one purpose in mind: minimizing liability to the company from a litigation-happy client who is looking for a scapegoat to blame after losing their shirt on trading straddle options on the Zimbabwe stock exchange. Disclaimers about past performance not being an indication of future results are everywhere, as are doom-and-gloom, danger-Will-Robinson caveat abouts the risks of non-diversification, short-term fluctations, exposure to emerging markets and the health hazards of consuming trans-fats. At least one section of the Julius-Baer report is a far cry from this content-free boilerplate:

“We also created structural imbalances and excesses in our economy that led to one bubble then another—the least painful way to contain one bubble is to create another; hence postponing the day of reckoning. In this period, we made useless financiers fly-by-night billionaires, destroyed most American’s living standards by depressing their wages and sinking the dollar against most currencies known to man—with few exceptions such as the Zimbabwe dollar. ”

Such moral outrage and indignation against incompetent fiscal policy and income inequity can’t be a very common sentiment in the financial sector. Penned by Rudolph-Riad Younes, long-time manager for the successful International Equity Fund, ticker symbol BJBIX, now closed to new investors, these words carry a strange sense of gravity more appropriate to an oped column than an announcement of financial results. (Full disclosure: this blogger owns shares in the fund.) It only gets better as Younes takes aim at other sacred cows:

“The Fed has shirked many of its responsibilities: by allowing asset bubbles to form unfettered; by maintaining ultra-lax monetary policies; by neglecting its regulatory oversight authority; and, by succumbing easily to the faintest political pressure. […]
The rampant decadence at the top trickled, as expected, all the way to the bottom resulting in two major bubbles while laying the foundation for future ones.” 

What follows are brief retrospectives on the tech bubble and the more recent housing bubble. One of the most interesting arguments is in the section labelled “The Cardinal Sin: Believing in Santa Claus.” Here he argues that a good deal of the problem originated with the Federal government revising its inflation measure to a completely different benchmark which made the figures come out significantly lower, very conveniently thank you– the equivalent of tampering with the speedometer as a way to speed up the car. A dangerous implication is that “true” inflation rates driving economic forces stand at 4-6% above stated numbers.

Finally throwing in a simple metaphor to emphasize the folly in case it was lost on the reader:

“In short, the government (the parents) invented Santa Claus in order to cheer up pensioners and laborers (the children) who were worried about their parent’s ability to pay for their entitlements (gifts). The whole family was happy with Santa Claus. The children were happy with the yearly gifts and parents were satisfied that their children were buying the fairly tale and able to rein in spending. But as in real life, it is a blessing only when children believe in Santa Claus and a tragedy when parents do!” 

No happy endings here though. The report concludes with predictions of more decadence and bubbles. Great reading overall.

cemp

RIP Cafe La Fortuna– one last cup of coffee

This is a good time to take a break from writing about work and observe the end of a New York City institution. Cafe La Fortuna. Yes the oldest cafe on the Upper West Side and a favorite haunt to its residents– John Lennon and  Yoko Ono included– is closing today.

According to the New York Times article the culprit is the same one that made New York claim the #4 spot on the list of most miserable cities in the US: ludicrous real estate situation. A change in the ownership of the building meant transition from an almost rent-stabilized situation to completely insane market prices. The milestone is also covered by CNN, Gothamist and amNY.

To get a sense of the history here: the table Lennon used to sit at and featured on the cover of the single Nobody Told Me was retired by the owner “Uncle Vinny” after his death in 1980, but remained stacked with memorabilia in the front window. It was recently gifted to Yoko Ono. So surprised was she that they had kept it for 25+ years that she wrote a letter thanking him, which hangs framed on the wall.

One long-time customer quoted by the NYT put it very concisely:

“I’ve told many people,” he said. “When this place closes, it’s time to leave New York.” 

This blogger could not agree more.

cemp

Rumors of Windows server platform “failure” slightly exaggerated

This article which made it to Slashdot recently and the linked postback from CNN/Money could use an application or two of Occam’s Razor. It stipulates that the MSFT bid for Yahoo is prompted by an internal recognition that the Windows server platform has failed. The company having seen the light, according to this commentator, is going after systems built on the Linux/Apache platform instead.

“Microsoft runs on the Windows platform and it has proved inadequate to run big Internet companies. There is not one big Internet company – and I mean “BIG” like Google Inc. (GOOG), Yahoo, Amazon.com Inc. (AMZN), eBay Inc. (EBAY) and such – that runs on Windows besides Microsoft. Its software platform has been a disaster supporting its search engine, email and other free services.”

It only takes a second to recognize this as uninformed drivel: Hotmail/Windows Live Mail is the world’s largest email service period. Passport/Windows Live ID is the largest online authentication system. When it comes to instant messaging, MSN/Live Messenger is not to far behind Yahoo and AIM– never mind the branding confusion between MSN verses Live. All of them run on W2K3, IIS , SQL Server and the accompanying much criticized baggage. It’s not a recent phenomenon either: in the late 90s MSR built TerraServer— long before viewing satellite imagery was an everyday activity– to showcase the scalability of a massive data warehouse running on Windows.

Yet the quote above does raise an interesting question about why more large scale web services are not built on top of Windows. The obvious reason is easy to shoot-down: the difference between shelling out $$$ for W2K3/W2K8 or getting Linux for free. It’s true that a single license for server can run into the hundreds of dollars depending on the particular SKU and thousands of dollars for the more esoteric 64-bit variants. This is why hobbyist sites, non-profits and small-businesses (as well as the virtual hosting companies catering to them) are more likely to prefer open-source software, because of the extreme price sensitivity in the market segment. Assuming that the distribution of internet facing websites has a very large tail fitting that category, this would explain why Netcraft surveys continue to show Apache leading IIS 50% to 35%, in spite of huge jumps in April ’06 and September ’07 that narrowed the gap from previous 3x difference.

But in the enterprise context, the gating factor becomes recurring costs for running a data-center: all of that IT staff, leasing the space and power used adds up. The upfront purchase price of hardware and software is dwarfed by operational costs– and that’s one reason why Windows server platform continues to make inroads into this segment, joining Linux in slowly chipping away at the market share of the more expensive UN*X variants that once dominated the server business. Nowadays it is not rare to see entire IT infrastructures of companies run on Windows and developed using .NET programming models.

What about large scale Internet services? This is the mystery: the existence of very large-scale (in at least two cases cited above, the largest period) services running on Win32 and Win64 proves it can be very competitive. In that case the nagging question remains, why are there are so few examples outside Microsoft?

cemp

Email storage and a lump of coal

TreeHugger is not the first to notice that computing technology can have environmental impact and different “systems” can be greener than others. In an invited talk at Microsoft Research in 2004, Andrew Shapiro from the Berkman Center and author of The Control Revolution raised the question of whether Linux could be deemed more environmentally friendly because it ran on lower-end hardware that would not meet the base requirements for modern Windows SKUs. (He was polite enough not to answer this question given the audience.) Similarly it is widely acknowledged that data centers today are gated by cooling and power consumption– air conditioning being one of the prime resource hogs– and availability of power generation is a significant factor in selecting “hot-spot” locations for building them.

TreeHugger post frets over the cost of email storage and wonders whether deleting email will curb carbon emissions. Good intentions for sure but the calculation may have been slightly off base for several reasons. First the bad news: storage in large-scale services like the one cites in the article are replicated. There can’t be just one copy of the message sitting around. Try explaining to a user that you lost all of their vacation pictures because drive #3385 failed– the so-called “we blame Seagate” approach.  That implies the figures are underestimating the true impact. That would be true only in a simplistic model where  power consumption scales with amount of data stored. Transaction capacity is often the determining factor for data center design. If one million people are checking email at the same time, enough servers have to be up and running to process those requests with tolerable latency. That’s true even if everyone keeps an empty inbox.

Similarly different storage architectures can lead to very different resource consumption patterns. If drives are directly attached to server, then more storage means more servers even if the servers sit idle CPU-wise. If the service uses a storage array network (SAN) then only drives are being powered and not all the extra baggage that would come with a full-fledged server. This is similar to the difference between using a networked drive at home verses another general purpose PC for handling backups. Finally there is the storage corollary to Moore’s law: disk sizes increase, price drops and so does power consumption per GB. (Unfortunately there is also a storage corollary to Peterson’s principle which states that data expands so as to fill the drive available.) It’s true that less storage will achieve some reduction but the Treehugger article probably overestimates this by several orders of magnitude. And if hosted cloud service were comapred to storing the same amount of data at home, there would be no contest: those massive data-centers achieve economies of scale and corresponding eco-efficiency not available to the average consumer not living off-the-grid with solar panels.

cemp

Old-school voting machines

Reminiscences from Robert Holt Jr.  who has been working as a voting-machine technician in New York city for over 20 yeras, as quoted by the New York Times:

“To tell you the truth, I like these machines. With all the problems they’re having with the computerized machines, these are solid. You can’t tamper with them.” 

“The ones who lost canvass the machines and see how many votes they lost by, machine by machine. Sometimes they come in angry. They’re upset they lost, but there’s nothing they can do. A loss is a loss. These machines don’t lie. What you see is what you get.”

Here is to machines that don’t lie and more importantly are transparent in allowing that alleged honesty to be verified by everyone.

cemp

NDSS, final day: “minding the gap”

(Trying to write about the conference before the recollections fade.)

Dan Kaminsky was scheduled to be the invited speaker on Wednesday morning , tentatively titled “On breaking stuff” but he was held up by consulting work at IOActive. Fortunately for the conference program committee, Paul van Oorschot volunteered to give a talk on short-notice and the result was the highly engaging “Security and usability: mind the gap” presentation.

He first started with some anecdotal evidence on the sad state of affairs in what should have been the poster child for usable security: online banking. One of the largest banks in Canada promised to refund 100% of losses resulting from unauthorized transactions– provided the user lived up to their side of the agreement. This fine-print in the customer agreement (granted nobody pays attention to that) makes for entertaining reading:

  • Select unique and not easy to guess password– and user will judge the quality of their password how? Windows Live ID has a password quality meter but this is far from being a standard feature.
  • Sign-out, logoff, disconnect and close the browser when done (What is the difference between first two? Disconnect means yank the network cable?)
  • Implement [sic] firewalls, a browser with 128-bit encryption and virus-scanning. As van Oorschot pointed out, the bank probably means “deploy” rather than “implement”– otherwise they would drastically narrow the potential customer base to developers with copious spare time for writing code from scratch for commodity purposes.

It only gets worse from there. The general pattern is promises of security and reassurance that damages will be covered in exchange for vague expectations of “secure behavior” form users who are often not in a position to accurately judge risks associated with their use of technology. Case in point: one study on malware found that 95% of users had heard of the word “spyware” and 70% banked online– yet some assumed that spyware was a good thing– 45% did not look at URLs and 35% could not explain what HTTPS meant. The status quo for online banking is not an isolated incident, as other case studies drawn from two recent publications van Oorschot coauthored:

  • An evaluation of Tor/Vidalia/Privoxy for anonymous browsing, which concluded that Tor is not ready from prime-time use by a novice even with the supposedly user-friendly Vidalia UI. (Given its remarkably low bandwidth and high latency reminiscent of the early “world-wide-wait” days of dial up, you have to wonder if a usability study was necessary to reach that conclusion.)
  • Usability study of two password managers with 26 non-technical users that found several problems, including situations where users falsely concluded a security feature was functioning when it was not– the very dangerous “false success” scenario. [Full disclousure: this blogger had reviewed and broke an earlier version of one, PwdHash.]

If poor usability is a security vulnerability as much as a flaw in a cryptographic protocol, what is the prescription? This is where the information security community is now wrestling with its collective conscience. van Oorschot made the frank observation that usability and HCI issues are routinely looked down upon by CS culture, not included in the traditional curriculum because they are  easy/trivial and better left to “people who can’t write code” to sort out. He raised the possibility that we had it wrong all the time: cryptography is the easy bit, secure system implementation is far more challenging and the hardest task is building usable secure systems.

cemp

NDSS, day II: Virtualization and security panel

The highly anticipated panel ended up taking a different turn. My colleague Tavis from Google could not attend, leaving DJ Capelis from UCSD as the only other skeptical voice to point out the risks from virtualization. (Recap: Last year Tavis found several problems in qemu, Virtual PC/Server and VMware virtualization platforms.) Intel was represented, and so was AMD with John Wiederhirn and Tal Garfinkel attended for VMware completing the  viewpoints at the table: hardware, virtualization platform, security research.

Most of the discussion implicitly focused on the server consolidation scenario, without spelling out the other uses of virtualization. Briefly consolidation scenario is about replacing multiple physical machines by a single powerful box that runs a VM with the equivalent OS/software configuration for each PC displaced. It sounds like re-arranging deck chairs but in fact this is a major cost saving opportunity for enterprise IT departments. A single powerful, expensive server hosting N virtual machines is by far easier to maintain than N low-end servers each running a different configuration. And in the long run the cost of maintenance dominates the original purchase cost of the hardware. Full machine virtualization creates new opportunities because it allows very clean consolidation between applications that could not otherwise live on the same bare metal: for example a legacy W2K3 line-of-business app alongside a new W2K8 terminal server or even Linux and Windows coexisting side-by-side.

This is the most commercially viable market for virtualization. VMware has been leading the charge and MSFT giving chase with Virtual Server R2 and the upcoming hypervisor in W2K8.  But focusing on it alone skewed the discussion, setting the stage for a predictable debate around trade-offs. Separate hardware is an isolation boundary: it keeps different applications from interfering with each other accidentally or by malicious logic. Virtualization is another one, as are operating system processes, BSD jails etc. Each one has an assurance level from security perspective or equivalently an attack surface. Server consolidation with a VMM involves changing the isolation boundary and creating new attack surface. There may be new channels for one VM to attack another when running on the same bare-metal, while separate boxes would have been confined to the network or shared storage etc. Quantifying that incremental risk and sharing opinions on whether it is a reasonable trade-off fueled much of the debate.

This is a comparison virtualization can not win on the single dimension of risk. Considering the extent VMs are used for malware research and quarantining untrusted code, it’s surprising that other applications were not considere. The flip-side of consolidation is sand-boxing: moving applications running in the same trust boundary to different VMs is a corresponding improvement– although the extent that it improve security is again debatable and dependent on the quality of implementation.

As a side note, the moderator raised a point about reduced customer choice: with individual machines one has a choice of different vendors to buy a network switch from. With the functionality of the switch subsumed in the software stack that choice goes away.

cemp

Dispatches from NDSS: Day I, breaking online games

Gary McGraw gave the talk “Breaking Online Games” at other conferences before, so this may be repeat material for some who have attended BlackHat or CSS in Washington earlier this year. (One difference is that apparently few security researches play World of Warcraft in the NDSS audience, neutralizing some of the gamer jokes.) At first the concept of cheating at online games seems out of place at a conference focused on fundamental security problems with a pragmatic bent: phishing, botnets, spyware, vulnerability research etc. But as McGraw pointed out there are two key observations making this topic very relevant:

1. MMORPGs foreshadow the future of massively distributed systems. World of Warcraft recently cracked 10M users (the slides had 8M, demonstrating how rapidly presentation material becomes outdated in this field) with up to half million online simultaneously.

2. There is real dollars at stake. Games like Linden’s Second Life– much smaller than WoW but far more visible in the media– have spawned a virtual economy that maps to transactions in the bricks-and-mortar economy complete with lawsuits. Even the devaluation of the dollar against foreign currencies such as the euro has a parallel in the going rate for gold coins. Cheating at online games then is about ill-gotten gains, a familiar theme for cybercrime.

The presentation itself was a broad overview of the security challenges in online games and stories of organized “exploit” opportunities it has given rise to, with references to the accompanying book. (There were also interesting digressions into eggregious EULAs, because it turns out World of Warcraft includes one to cover an ineffective anti-cheating solution that functions like spyware.) One implied conclusion is that designers for online games don’t in general grok the concept of security: traditionally it meant protecting the game against cracking and pirated distribution. The problem of contending with untrusted clients “outside the trust boundary” as McGraw puts it has not made it into the design philosophy.

cemp