Author: Cem Paya

Cem Payahttps://randomoracle.wordpress.com/Building & breaking systems.
Cem Paya's avatar

Comcast support– undoing the damage (3/3)

The next day two technicians showed up and this time the blogger was in attendance to witness the damage from previous day. Technician #1 was the same person who tried to fix the problem earlier and made no progress after several hours, leaving behind a completely disabled network and a tangle of cables, including a new heavy, 5-piece component + AV cable as a souvenir. (There was already a component cable around but apparently he brought in a brand new one when at some point he could not get any image on the screen at all.) Technician #2 was the more senior employee and he was the only one working on the systes this time while his colleague went to check the connections outside. Observations from this round:

  • After the initial visit, the network was down. In particular the standard-issue Linksys WRT-54GS wireless router appeared bricked.
  • The technician claimed that Comcast does not touch customer-owned equipment and therefore the malfunction of the router could not have been as a result of their trouble-shooting. This was in direct contradiction of what was observed the previous day, when technician #1 repeatedly power-cycled routers to get the network back up. The wireless router was still unresponsive when they left. Quick check afterwards showed that they had the power adapters mixed up. Easy enough, considering both were Linksys devices with compatible connectors. Cable modem still ran fine on the higher-amperage from the router’s power source but not the other way around.
  • He first attempted to isolate the problem by “hard-wiring” to the cable modem, which worked fine on his laptop. But a different laptop running Vista refused to cooperate with the ritual of powering on/off the routers in a specific sequence.
  • UAC once again got in the way: when the technician attempted to coax the laptop into getting an IP address from the cable modem, he was stumped. His standard trick of running “ipconfig /renew” returned an access denied error message. No problem– this blogger helped out by launching an elevated command prompt as admin where work could be done again. (If this experience is representative, Vista is still not very popular on customer systems or Comcast has not bothered to update their trouble-shooting diagnostics.)
  • Comcast uses an unaffiliated, third-party websites for speed tests, to measure effective bandwidth available to the customer. The technician suggested Googling for “bandwidth test” and following several links from there including the Speakeasy page. None of them were particularly conclusive: on a 10MB/s line one of them returned almost twice that speed. (A telco giving customers more bandwidth than they are paying for?)
  • Eventually the speed-tests revealed that downstream bandwidth is within spec but there were problems in upstream bandwidth. Since the problem originated in the fiber running to the area, nothing could be done here other than placing a call to the service center to check on the lines.
  • He also discovered that the DVR was defective– this was the replacement DVR provided in exchange for the original equipment Comcast dropped off, which also did not function correctly.

After three attempts we finally had a functioning network. The experience added another chapter to the “inept telco-monopoly indifferent to customer experience” image. A few days afterwards news reports surfaced that Comcast had been downgrading  BitTorrent traffic. Finally this was one explanation: it’s hard to focus on customer problems when you are too busy interfering with their service.

cemp

Comcast support– diagnostic rituals 2/3

Day #2: After failing to show up on the appointed day, Comcast unilaterally “reschedules” to Thursday.

(Note: the following sequence of events can only be inferred from first-hand experience of the resulting destruction afterwards, as the blogger was not present during this stage.)

Friendly support representative shows up this time. Unable to diagnose the problem with broadband speed, he tries a series of rituals and incantations instead. First up is the usual “power cycle everything” trick where devices are unplugged and replugged in all possible sequences on the chance that one of them will trigger the correct sequence of events– that or spark the electronics inside. Next the cable modem/wireless router combination is moved upstairs to a different cable outlet, in case that one happens to have the magic pixie dust required for reliable broadband. No dice.

Why the wireless router? Fair queston and a different technician the next day insisted that they never touch customer owned wireless networks and only diagnose using a CAT5 cable to the modem directly. So the first support person was not even following their own stated policy. After failing to get any broadband working, a different problem is tacked: the case of the DVR that refuses to record. That would the Comcast provided Motorola DVR/cable-box installed by the original crew.

The DVR is not healed but the TV acquires a series of new cables hanging out of it. The blogger arrives to discover an all-in-five RGB component video cable plus analog audio cable, as well as an DVI-to-HDMI conversion cable. Apparently at one point nothing was working, no image on TV etc. and new connections were tried between the cable-box and TV.

After 4 hours, at least cable picture is restored. Internet access appears to be completely toast. This is actually a step backwards because the previous day evening at least there was intermittent but slow connectivity.

[continued]

cemp

Comcast support experience– yet another telco story (1/3)

No wonder customers are going ballistic after dealing with Comcast. After experiencing 48-hours of Comcast support, it is not difficult to understand why the spirit of telco ineptitude evokes such strong reactions. (Full disclosure: the bits comprising this blog post were carried upstream on a Comcast cable connection.)  Cable was not necessarily the first choice for broadband in our Philadelphia residence. But Verizon did not help its own case when different support reps on the phone had  inconsistent stories about the availability of their fiber-to-the-home (FiOS) service in the  neighborhood.  Final answer appeared to be negative, and since cable is faster than DSL around the same price points (despite asymmetry between upstream/downstream speeds) the decision was easy.

Setup was anything but straightforward. On initial connection, all requests are routed to a Comcast page and the only download allowed is the setup software. This when problems started because the link advertised as 6Mbps began acting more like 6kbps. Several times the download stalled, speed dropped to zero inexplicably after starting out with  familiar burst of broad-band quickness. After more random behavior and several attempts at power-cycling the cable modem, the infrastructure decided to give the blogger a break, turning on the bit tap just long enough to get the setup. (The Comcast technician who performed the installation could have dropped off a CD.) Enrollment process required two full runs,  requiring  either the account number or address to “activate” according to instructions. Still two tries is not particularly bad for a technology where “if at first you don’t succeed, reboot and try again” is the trouble-shooting mantra.

Problems started soon after. Initially the connection appeared to work and putting a WRT54GS wireless router in front of the cable-model was the first step to restoring a home office environment. But the connections were flaky, web browsing had frequent random pauses and interruptions. Investigating this is tough because the cable-modem must be power cycled before directly connecting, otherwise the IP address remains tied up. A call to Comcast revealed that the online activation in fact did not activate anything and that an old-fashioned phone call would still have been necessary to unleash the bits. (She was surprised that we were able to get anything working before Comcast threw the switch on their side.)

Meanwhile the DVR did not work correctly and refused to record. No problem, we are assured by Comcast representative. An appointment is made for Wednesday for a trained expert to fix the problem. Only the technician never showed up, in spite of assurance on the phone that the crew is on the way. The next day Comcast modifies the story with a twist: the appointment had been canceled (by unknown actors, one assumes) and they were never scheduled to visit in the first place.

[continued] 

cemp

“OS-X and Windows are fighting over my laptop”

Tiger wins this round. For the second time in two months, a Windows image in Parallels was corrupted, leading this blogger down a winding path of frequent visits to the friendly tech-stop at the Google NYC offices. Unlike the last time when the entire machine was corrupted (score +1 for Windows) this time the damage was limited to the Windows XP side, with the host OS-X showing no signs of malfunction. (score +1 for Mac, previous corruption avenged.)

Meanwhile Windows started to blue-screen very early on in the boot process, while loading the mup.sys driver, complaining about missing registry hive. Since that is a fairly vital piece of the system,  booting into safe-mode or last known-good configuration wouldn’t help. It took some time to track down an XP image to launch into recovery console– first sign that one is no longer working at MSFT where there are copies of every SKU of Windows imaginable floating around, just in case you needed to boot Server 2003 R2 Enterprise edition on x64 out of curiosity. But recovery console required the local administrator password, and not just any user in the admins group. (Since AD domain logon is not available in the limited recovery environment.) Neither the blogger nor tech-stop folks could track that down. No problem, since many live Linux CDs allow mounting NTFS and scrambling the SAM to blank-out admin password.  That didn’t work either when the first one tried complained about file-system corruption and mounting in read-only mode only.

At this point, re-installing the OS was starting to become the path of least resistance when considering that Parallels ships with an application that can mount the Windows image offline as a drive and recovery any files. (This is similar to vhdmount utility from Virtual Server R2 pack.) One problem there– the BIOS emulated by Parallels can not do PXE boot. This is a Parallels limitation since the MSFT offerings Virtual PC and Virtual Server installed on a desktop had no problem starting with a blank image and booting into a PE environment used at Google for  network installs. The missing piece of the puzzle had to be provided by tech-stop folks, a floppy image to bootstrap the PXE process. (Virtualization has its ironies– remember how Apple was skewered for not including floppy drives with the iMac in 1998?)

After that the installation more or less proceeded on automatic pilot, although host integration was missing until installing Parallel tools later.

Time wasted: Roughly four hours, spread across two days.
Current standings: Macintosh evens up the score against Windows, tied game.  It is time to reconsider that “snapshot” feature in Parallels. Comparison point: in 2+ years of using MSFT virtualization platforms, not a single image has been corrupted or hosting machines inexplicably toast in the process.

cemp

Throwing fuel on the fuel-economy debate

How often do GM and Toyota get into a public argument with a Pulitzer-prize winning author, using the blogosphere as their battle-ground? It all started when Thomas Friedman, author of the globalization classics Lexus and The Olive Tree and The World Is Flat, wrote an op-ed piece for the New York Times titled Et tu Toyota?, taking the company to task for its duplicity in joining the Detroit big-three for lobbying against higher fuel-economy standards in the US, while publicly cloaking itself in the language of eco-friendliness when it comes time to hawk hybrids on TV.

Toyota PR machinery kicked into high-hear and soon Irv Miller, group VP of communications had a response posted on the company’s external facing blog. General Motors also took offense at the allegations, and joined in the fray with a post of their own on the GM blog, appropriately borrowing Shakespearean title from Julius Ceasar: Beware the Ideas of Friedman. (Perhaps they could have waited until March in deference to the theme?) And there are just the “official” participants– bloggers have been actively writing about the problem.

Here is the quick run-down of the argument:

  • Friedman questions why Toyota is fighting against fuel-economy standards in the US, since their fleet already complies with the higher ones in Europe and Japan. Detroit is in a different boat, as their primary market is US and their production is  heavily weighted towards light trucks. Precisely for that reason, higher CAFE standards place GM/Ford/Chrysler at a disadvantage while favoring the imports which do need to costly adjustments to the new regime. The puzzle is why sheer self-interest did not lead Toyota to lobby in favor of higher standards.
  • The answer implied in the article: because that would leave significant revenue on the table since large-trucks and SUVs constitute a big slice of the US market. It’s not uncommon for a large company with diversified product lines to demonstrate schizophrenic behavior– one side going after the “green” niche while another seeks to capitalize on gas-guzzlers. No surprises there.
  • Irv Miller counters that Toyota is pushing for higher standards but not the most aggressive version described in the senate bill because it is unrealistic:

“It’s because there’s a point at which the bar is set too high for all competitors.”

  • Both the Toyota and GM responses counter that the reason large trucks are built is because the large trucks are bought by consumers- effectively a syllogism that amounts to “we sold them because they bought them.”
  • Similarly this line makes no sense:

It’s why our full-size pickups are the fuel economy leaders. It’s why our new Chevy Tahoe and GMC Yukon Hybrids match the city fuel economy of a Toyota Camry.

The fact that one model can beat a competitor doesn’t give GM a “green heritage” anymore than the fact that the Viper can hang with a Ferrari give Chrysler a “Formula 1 heritage” across the line up. Existential proofs are useless because environmental impact is about total emissions across the board. The “A” in CAFE stands for average, not some best-case scenario achieved by prototypes in a controlled lab experiment or niche model driven by a few hundred people.

  • There is a deeper concern raised by Friedman which is not answered in the GM retort. NYT article refers to Michigan reps’ attempt to lobby against CAFE standards on behalf of auto-manufacturers a case of “empty-barrel politics” and corporate euthanasia– effectively hastening the decline of the US industry. This is a far more damning and bold accusation. Putting on the McKinsey consultant hat, Friedman is dropping a hint that management has been clueless and their long-established strategy of abandoning the small car segment to imports has driven the industry into the ground. (The reasons for the decline may be debatable but its existence is certain. Last year Toyota quietly surpassed GM to become the world’s #1 manufacturer.)
  • Finally the engineering creed of “doing more with less” is missing from the whole debate. There is undeniably a trade-off between vehicle size and fuel efficiency, but there is nothing that precludes improvements across the board. Even today wide difference exist in the fuel efficiency for vehicles in same size, weight and performance categories. In fact one could argue there is a greater burden to improve fuel economy in that segment. There is no reason that tricks applied to  optimize small cars today (multi-valve engines, variable timing, use of lighter metals in construction, aerodynamics, hybrid drive-trains etc.) could not be employed elsewhere.

cemp

Firefox IE-tab extension

(Second part of an earlier post)

Firefox 2.0 makes a great contender against IE to become the default web browser for Internet scenarios. Full disclosure: this blogger was an Internet Explorer developer in a former lifetime. But it is still not mature enough for enterprise market, which understandably is not glamorous enough for Mozilla foundation to target. This is almost the reverse of the IIS-Apache split: IIS/Sharepoint is very well optimized for corporate intranet use, while Apache continues to dominate the market for Internet facing websites as evidenced by over a decade of Netcraft surveys.

Firefox’s main limitations include handling Kerberos authentication transparently and running ActiveX controls, both of which can be significant for large enterprises using AD and custom line-of-business applications. Firefox does have the equivalent of the Negotiate package but it is not integrated as well: it prompts the user, while IE going through the SSP simply picks up the credentials already registered with the logon session. (This could have been easily fixed by using the SSPI itself, at the cost of portability. The interface is generic and not bound to any articular user experience.) ActiveX may be the greatest challenge because implementing that may  require wasting years of developer time in the depths of COM and OLE.

IE-tab extension provides an easier way out. By clicking on the Firefox icon on the status bar, the extension allows switching to use IE for rendering. That means all IE components come into play, including wininet for network downloads, urlmon for binding, mshtml/Trident for HTML rendering, the Windows javascript engine for active content etc. The only pieces missing are the so-called “chrome” associated with IE: toolbars, IE status bar, window menus. (Right-click inside the page will still bring up the context sensitive menu from Internet Explorer.) No more error messages about needing to view the page in IE. In fact the operation is transparent because websites for the most part can’t peek outside the IE control to see if it is being hosted. Windows Update just works and so does Genuine Advantage validation if the operating system was licensed appropriately.

That means Firefox can for the first time become a full-time IE replacement in all scenarios– but only by hosting IE components inside the Firefox user interface. If that sounds like the embrace-and-extend strategy employed by Microsoft in its battle against Netscape, the irony is well deserved. For all its faults and awful security track record, one of the things IE designers did get right is an architecture that allows easy embedding in other applications. This feature apparently cuts both ways: during the browser wars it helped a great deal by pushing IE into every other application, but in the hands of Firefox developers it can help a competitor  supplant IE as the default browser of choice.

cemp

Blodget on socially responsible investing

Another great issue from the Atlantic Monthly for October 2007. Subtitled “The Values Issue,” the journal contains a trio of articles on the subject of philantropy and altruism. One of these is The Conscientious Investor, by Henry Blodget, focusing on the emerging field of socially responsible investing or SRI for short. In case the name rings familiar: Blodget made his name during he dot-com era with the unlikely prediction that Amazon stock price would hit $400. It did but his resultant status as celebrity financial analyst for Merill-Lynch ended amidst revelations that he’d long been expressing doubts in private about the companies he was raving about publicy in order to drum up banking business for Merill.  The irony of Blodget writing about SRI is inescapable. What is next– Britney Spears on good parenting? Luckily the author has a good sense of irony as well and acknowledge this strange twist in a parenthetical remark alluding to his own run-in with the SEC.

Blodget is no doubt very knowledgeable and in a great place to write an engaging article. It begins by comparing two hypothetical portfolios: both invest in the S&P 500 index from 1957-2003 except that one of them leaves out Philip Morris, since tobacco companies are verboten by most screening criteria used for SRI. Sadly for the second investor, it turns out that PM was in fact the equity appreciating the most during that time– a staggering 19.75% compared to a mild 10.85% for the broader stock index. The net result of leaving out just this one stock out of a group of five-hundred is 5% over the five-decade span. Even more poignantly, investing in just PM instead of the diversified portfolio would have multiplied the returns by a factor of 36.

The article is full of these hard data points. For example we learn that SRI investing accounts for almost a tenth of all professionally managed assets but most of this is institutional investors. The 200+ mutual funds make up a small fraction overall of the sum and for that matter, of all the assets invested in equity funds. The punch-line still remains the qualitative argument around the conceptual hand-waving surrounding the definition of “socially responsible.” Screening criteria used by different SRI funds is all over the map and full of internal contradictions, easy targets for picking. Not all the criteria makes sense: while shunning tobacco or coal-fired power generation is understandable, the jury is out on whether nuclear energy is good on balance for the short-term until carbon emissions are under control. Similarly, some criteria can already find expression in everyday decision without being elevated to investment strategy. Consumers have little choice about the local utility building a coal-powered plant or dumping waste into the river. In these cases, voting with the portfolio may be the only response because individuals can’t influence the outcome and the collective bargaining process through politics is inefficient. But individuals can opt out of gambling, drinking and tobacco, so it’s not clear that investment decisions need to be tweaked. Even the question of alcohol is ambiguous: PAX’s decision to divest Starbucks for lending its brand-name to a Godiva liquor is the reduction-to-absurdity of the criteria. (By all definitions, SBUX has one of the more socially responsible operations.) The article raises a more disturbing question about market response to SRI. If the approach goes mainstream, and by all indications it may be on the verge, companies will  mount PR campaigns to create the appearance of satisfying SRI criteria while conducting business-as-usual. This will confuse the screening criteria further, since the model does not yet include companies trying to game the system.

[continued]

cemp

Putting the Gap into “security gap”

From Money/CNN:

“Gap Inc. announced Friday that a laptop containing the personal information of about 800,000 job applicants was stolen from the offices of one of its vendors that manages data for the company.”

The laptop was not encrypted. Way to go. Part of the reason for the impressive total is Gap also owns Old Navy and Banana Republic. (The latter brand may inspire a few cracks about the information security approach used by the retailer.) Invariably adding insult to injury in these cases is the PR response:

“Gap (Charts, Fortune 500) said it has no reason to believe the data contained on the computer was the target of the theft or that the personal information has been accessed or used improperly.”

The data on the laptop may not have been the motivation (unless there was a big sticker exclaiming “Danger! Confidential data, property of Gap Inc.”) but that does not help because: Either the criminals is 1. clueless and will simply resell the machine without peeking, in which case it is up to the next person to see what he/she just inherited, OR 2. is a professional and will try to sanitize the drive prior to resale, in order to hide the fact that it is stolen merchandise. It’s not a stretch to say that prior to wiping out the drive he/she may take a peek. In any case, how does the Gap conclude there has not been accessed or used improperly? Is there a new, alien technology in their possession to monitor ongoing access to a stolen drive? Did they run credit checks on all 800K applicants to check for signs of new -account fraud using the compromised identities?

Another fine addition to the excellent Hall of Shame maintained by PrivacyRights,  the Chronology of Data Breaches going back to ChoicePoint in 2005. Current count stands at 166M– half the population of the US, including every child if there had been no overlaps.  That web page could well serve as mandatory web-browser start page for CISOs and other influential officers of any organization tasked with protecting large amounts of user data.

cemp

Reversing the flow– when cars power houses

“You don’t have to plug it in” Toyota keeps insisting in their commercials about the Prius. A conservative company confronted with the problem of marketing a disruptive technology  to a conservative audience, opted for doing the exact opposite of what every other automobile manufacturer is doing: convincing you that their product is just like any other car. While every one is spinning tales of personal liberation and spiritual enlightenment, Toyota says this is just another car. You drive it like any other car, never mind this “hybrid” mumbo-jumbo and there is no fussing with a power cord.

The problem is owners want to mess with power cords. Toyota management may want to brush up on their copy of Crossing the Chasm, because the people buying into the technology are still early-adopters at this stage and they are interested in pushing the limits of the technology. Already CalCars demonstrated plug-in conversions and Google has a fleet of converted Priuses drawing power from the solar panels lining the roof of the parking lot for buildings 45-46 in Mountain View campus.

The capability goes both ways: they can either draw power from the grid (or off the grid from renewable sources, in the case of Google) or they can supply the grid. The economic viability rests on a type of “price arbitrage”– charge at night when electricity is cheaper due to lower demand, then supply the grid during the day when it is more expensive with all those air-conditioning units whirring away.

But that vision is a few years out for a vehicle rolling off the assembly line in bone-stock condition. Without voiding the warranty or paying for a conversion that costs more than the car itself, there is no easy way to tap into this resource. Undeterred, a community of tinkerers continue to push the limits while holding on to the warranty. One of the more interesting projects is PriUPS, a pun on UPS for Uninterrupted Power Supply.

The standard answer to unreliable utilities and hurricane-prone regions is a portable generator. But a car is effectively a portable generator. When the engine is running, the alternator supplies current. (Nevermind the battery commercials showing sports cars equipped with their brand shredding rubber: the battery starts the engine but is largely out of the picture under normal driving.) Tapping into this with an off-the shelf inverter is standard. Up to ~200W can be drawn directly from the cigarette-lighter adapter– is that an anachronism now that few people use it for that purpose?– and upwards of 2000W by hooking up to the battery itself. This means that in an emergency or power-loss, a car could power basic house-hold appliances.

In principle the Prius is great for this application. First it is equipped with a large bank of batteries distinct from the ordinary one (called “traction battery” in Toyota language) that can supply more juice. More importantly, it has automatic power management to sense when the battery is running low and run the engine to recharge. This is a big improvement over running the engine constantly or manually stopping/starting. Ordinary car batteries are not designed for deep-cycles so they can’t handle being nearly discharged completely and come back to life. The Prius traction battery has no such problem.

But the engineering department had other ideas and drawing power from the traction battery was not high on their list of priorities for the car. (Besides– what would happen if potential buyers got wind of that capability? It’s no longer “just another car” and they would run away screaming, intimated by all this technological complexity, according to Toyota thinking.) PriUPS project details all the hoops to jump through to get this working. Detailed descriptions makes for good reading in their own right, but the short version is this gentleman managed to get 2400W while the engine was running 40% of the time in burts of 3 minutes. 5-6KW is the upper limit extrapolated from there, more than enough to keep a residence fully functional, minus air conditioning.

Interesting enough Honda, Toyota’s biggest competitor at home, has a healthy business in small, portable generators from home back-up power. Exactly the type that could be replaced with a Prius in the garage.

cemp

So dark the con of baseball: home run #756

The fans have spoken. In restoring a measure of respectability to the blemished game, they voted to get the #756 home-run ball branded with an asterisk before its ultimate journey to Cooperstown. Creating a PR spectacle around the fate of an infamous ball is not original, (Chi-town residents can indignantly point to the decision by online vote to destroy the Bartman ball after the Cubs’ ill-fated 2003 post season run) and the accuracy of the election about as reliable as Diebold touch-screen voting devices, it appeals to an abstract sense of justice.

This is far from compensating for the failures of “Bud-Lite” Selig to keep steroid use in check during the late 90s. Outside the city limits of San Francisco, there were few who were seriously in doubt that this record is not tainted. As for the Giants front office, it was a good move because the inevitable run-up to the record and the many milestones on the way (500, 600, then passing Willie Mays and Babe Ruth) made for great drama, packing the seats at a stadium better known for its Bay views than the quality of baseball on the field.

The good news is 2007 will also be remembered as the season Alex Rodriguez reached the 500 home-run mark, the youngest for any player to reach that total. He has easily another decade of playing time. Assuming he can hit at least thirty HR every year during the rest of his career– easy enough, he averaged over 40HR for the past ten seasons– Rodriguez will eclipse the current record and erase a disgraceful chapter from the history of the favorite pastime. Until then, the most pithy description for the state of baseball comes from the Onion: “Destruction Of National Pastime Given Two-Minute Standing Ovation”

cemp