Google has recently open sourced  protocol buffers, one of the building blocks used for building large-scale systems. This is a good time to recap the controversies over data formats. In this debate, opinion has been sacred and facts largely open to interpretation.

Most recent iteration from May: XML: the angle bracket tax takes issue with the inefficiency and developer-unfriendliness of XML as data format. First point is hard to dispute: all of those tags and attributes decorating the data add weight. (On the other hand, all of that redundancy means XML documents compress very well but that adds another step.) Compared to the YAML equivalent on the page, the XML variant looks manifestly verbose. In all fairness the example picked here (SOAP) is probably one of the worst case scenarios for bloat, short of even more tchotckied-up format such as WS-Trust/WS-Security/WS-anything. The jury is out on the second point. XML is not necesarily intended for direct presentation to users– that is why there is XSL as stylesheet language to add presentation. For example Internet Explorer’s default tree view for XML files with collapsible nodes is actually generated by applying a default built-in stylesheet.  Manually tweaking XML files is never intended for end users. As for developers and tinkerers stuck with that problem, there is help in the form of XML editors that allow for directly working with the high-level structure instead of low-level syntax. But even with plain-text editing, consider that XHTML is a formulation of HTML as well-formed XML and not particularly more challenging than vanilla HTML/4.0 to work with.

Skepticism against XML runs deep, including entire websites devoted to cataloging its deficiencies. But strong opinions run both ways. Inflammatory arguments bashing other formats as unnecessary reinventions of XML are also abundant; here is one comparing JSON vs XML.

In to this mess Google throws a curve-ball with protocol buffers which define a binary serialization format. The overview includes the expected comparison against XML as a data point:

Protocol buffers have many advantages over XML for serializing structured data. Protocol buffers:

• are simpler
• are 3 to 10 times smaller
• are 20 to 100 times faster
• are less ambiguous
• generate data access classes that are easier to use programmatically

If efficiency were the primary constraint, a binary format is the way to go. The challenge then is making that format extensible. This is the catch: if a data format were known to be immutable forever, everything would be hand-coded to use up the fewest number of bits. Reality is that systems evolve, protocols change. Version #2 of the application introduces extensions and yet must remain backwards compatible in a way that it can still communicate with version #1. (Sometimes even in the opposite direction) Building this level of flexibility without rewriting code for each version is the challenge.

Strangely there is already a widely used binary serialization format: ASN1. Virtually any large scale web site is likely using it because the X509 certificates required for the SSL/TLS security protocol are encoded this way. It is more complex, slightly more powerful than protocol buffers but boasts the same level of extensibility since new optional fields can be added in such a way that the structure remains a superset of its previous version. But greatest challenge for ASN1 is ease of use. There are ASN1 compilers but the interface is nowhere as polished as the cross-language programming model for protocol buffers. Parsing ASN1 is so complex that integer overflows in the parser used on Windows featured in a remote-code execution vulnerability back in 2004.

One of the more interesting properties of ASN1 is that the abstract structure of data and its encoding are clearly separated. The same logical structure can be expressed in different ways on the wire and while the most common variant “basic encoding rules” or BER is binary, a different version called XER allows outputting directly to XML. Small step for reconciliation in the format wars.

cemp

From relative obscurity, Nebuad has emerged as the star witness in the Senate hearings last Wednesday that also included representatives from Microsoft and Google.

WSJ points out that many current Nebuad executives previously worked for Gator, which almost single-handedly defined the “spyware” category earlier in this decade. Gator was one of the first examples of software bundling useful functionality appreciated by users with with intrusive, privacy invading and unwanted “features” designed to monetize that user for the benefit of advertising networks. On the surface Gator was a form-fill assistant; it helped users with the repetitive task of completing forms on websites. Web browser including Internet Explorer had an autocomplete / intelliforms feature for some time, but it was fairly primitive compared to what Gator could do. IE remembered that the user typed in some string on a particular field on a particular form on one web page; it had no concept of recognizing that as an address or recognizing other forms on other websites also asking for the address. So far so good– even Microsoft Passport was taking out full-page advertisements in the New York Times about its ability to save time by avoiding forms, because the profile data would be included in the authentication process.

But Gator had also had a dark side and it was hidden in plain sight, buried in the terms-of-use / click-wrap agreement that users seldom read. It collected information about the user’s navigation history and called home with the information. The resulting profile was used for taking targeted advertising to new levels: Gator replaced existing banner ads on websites with ads of its own choosing, disrupting the business model for many ad-funded free websites. This over-zealous tinkering with other people’s advertising brought Gator a lawsuit from several publishers including the New York Times and Washington Post.

Worse there was no meaningful opt-out possibility: no way to retain the useful functionality and opting out of the invasive tracking, other than uninstalling the app completely. And that is exactly what users decided soon after it was branded spyware (or “adware” as the preferred expression in polite company) in the public perception. On balance the inconvenience of taking two additional to provide your address one more time to another website did not outweight the potential privacy invasion from the tracking. Making matters worse, while it was possible to verify what Gator collected, users had no way to verify how the data was used once it is uploaded to the service in the cloud.

Seeing the writing on the wall, Gator soon found a more appealing application to latch on to: P2P. The popular file sharing program Kazaa was bundled with Gator (and soon several other variants of spyware) P2P created a dilemma for users who wanted to tap into the global jukebox but avoid the dubious spyware that often came bundled with the free software. Volunteer programmers responded by creating Kazaa Lite and other derivative “unauthorized” versions stripping out the dubious functionality. As for Gator, when the backlash became widespread, to the point that today Symantec has anti-spyware application that will remove Gator, the company reacted in typical fashion: by changing its name to “Claria” and hoping that will white-wash any previous associations.

Nebuad is an unmistakable here-we-go-again moment for the privacy advocacy community:

1. Company decides to push the boundaries of accepted data collection and user tracking with new “creative scheme,” crossing the line from dubious into nefarious
2. Technology press gets wind of the idea, at first as a curiousity, later with growing skepticism and apprehension, feeding the blogosphere.
3. Mainstream media picks up on the story
4. A public relations crisis results, indignant pundits demand that the company change its ways, high-level executive begin complicated song and dance for damage control.

At this point, the story normally continues with the dust settling down, contrite executives offering token changes to appease the privacy wonks and all concerned individuals move on to the next crisis-of-the-day. This time it is different. Charter canceled plans to pilot Nebuad, other ISPs are backing out and the Senate has taken an interest in the problem. Quiet possibly Nebuad picked the worst possible timing: with concerns about monitoring Internet traffic and ongoing FISA discussion around retroactive immunity for carriers, the technology community has been collectively primed to watch for the slightest incursion into the privacy of electronic communications. CDT released an interesting report suggesting that an ISP contracting with Nebuad would be violating a provision of the Wiretapping Act around unauthorized disclosure of private communications, a possibility hinted at earlier by Peter Wu.

Predictions? Keeping in mind Yogi Berra’s warning here, it is a reasonable bet that high burden of opt-in proof (as opposed to the current opt-out structure or simple click-through agreement) will be required if Nebuad-type systems are to be operational in the US. These barriers will make it unlikely that many subscribers will in fact participate voluntarily, unless the ISP offers heavy discounts on the Internet service, which is going to defeat the purpose of collecting additional revenue from Nebuad. Even if the economics worked out, the participation incentives will skew the data towards customers that were willing to make that trade-off and it is unlikely that this demographic will be very interesting for many advertisers. Bottom line is that Nebuad and its ilk are currently sailing into terra incognita with an extremely shaky business model.

cemp