Month: December 2021

Logical access and the security theater of data-nativism

Data-center address as security guarantee

WSJ recently quoted a spokesman for Binance.US stating that all US customer data is stored on servers located in the US. The subtext of this remark is that by exclusion, customer information is not stored in China, an attempt to distance the company from concerns around the safety of customer information. Such new-found obsession with “data terroire” is a common interpretation of the data-sovereignty doctrine, which holds that information collected from citizens of a particular country must remain both geographically and legally subject to its privacy regulations. While the concept predates the Snowden revelations of 2013, it was given renewed urgency after disclosures of US surveillance programs leveraging massive data collections hoarded by private companies including Google, MSFT and Yahoo among others named as participants in the mysterious PRISM program of “upstream” collection. [Full disclosure: this blogger was a member of the Google security team from 2007-2013] 

Data-sovereignty is a deceptively simple solution: If Google is forced to store private information of German citizens on servers physically located in Germany, the argument goes, then NSA— or its counterparts in China, Russia or whichever foreign policy boogeyman looms large in the imagination on a given day— can not unilaterally seize that data without going through the legal framework mandated by German law. This comforting narrative makes no sense from a technology perspective. (If it ever made sense in other terms, including lawful access frameworks. The NSA is legally barred from conducting surveillance on US soil. Moving data out of US into foreign countries amounts to declaring open season on those assets.) To explain why, we need to distinguish between two types of access: physical and logical.

One note about the hypotheticals explored here: the identity of the private companies hoarding sensitive customer information and the alleged boogeyman going after that stash varies according to the geopolitical flavor of the day. After Snowden, US tech giants were cast as either  hapless victims or turncoat collaborators depending on your interpretation, while the NSA conveniently assumed the role of the arch-villain. For the purpose of this blog post we will use Binance/US and China as the stand in for these actors, with the full expectation that in a few years these examples will appear quite dated.

Physical access vs logical access

Imagine there is a server located inside a data-center in the middle of nowhere, as most datacenter are bound to be for proximity to cheap hydropower and low real-estate costs. This server is storing some important information you need to access. What are your options?

1. You can travel to the datacenter and walk up to the server directly. This is physical access. It unlocks some very convenient options. Server is stuck, not responding to requests? Press the power button and power-cycle it. Typical rack-mounted servers do not have a monitor, keyboard, mouse or any other peripherals attached for ease of use. But when you are standing next to the machine, you can connect anything you want. This allows getting an interactive shell and using it as a glorified workstation. One could even attach removable storage such as a USB thumb-drive for conveniently copying files. In a pinch, you could crack-open the server chassis and pocket one of the disk drives to hoover up its contents. As an added bonus: if you walk out of the datacenter with that drive, the challenge of reading its contents can be done later from the comfort of your office. (Incidentally the redundancy in most servers these days means that they will continue ticking on as if nothing happened after the removal of the drive, since they are designed to tolerate failure of individual components and “hot-swapping” of storage.) But all of this flexibility comes at a high cost. First you have to travel to the middle of nowhere which will likely involve a combination of flying and driving, then get past the access controls instituted by the DC operator. For the highest level of security in tier-4 datacenter that typically involves both an ID badge and biometrics such as palm scans for access to restricted areas. Incidentally the facility is covered with cameras everywhere, resulting in a permanent visual record of your presence, lest there be any doubt on what happened later. 

2. Alternatively you can access the server remotely over a network using a widely deployed protocol such as SSH, RDP or IPMI. This is logical access. For all intents and purposes, the user experience is one of standing next to the machine staring at a console, minus the inconvenience of standing in the uncomfortable noisy, over-air-conditioned, florescent-lit datacenter aisle. Your display shows exactly the same thing you would see if you were logged into the machine with a monitor attached, modulo some lag in the display due to the time it takes for the signal to travel over a network. You can type commands and run applications exactly as if you had jury-rigged that keyboard/mouse/monitor setup with physical access. Less obvious is that many actions that we typically associate with physical access can be done remotely. Need to connect an exotic USB gadget to the remote server? Being thousands of miles away from the USB port may look like a deal-breaker but it turns out modern operating systems have the ability to virtually “transport” USB devices over a network. USB forwarding has been supported by Windows Remote Desktop Protocol (RDP) for over a decade, while the usbip package provides a comparable solution on Linux. Need to power-on a server that has mysteriously shutdown or reset one that has gotten wedged, not responding to network requests? There is a protocol for that too: IPMI. (IPMI runs on a different chip called the “baseboard management controller” or BMC located inside the server, so the server must still be connected to power and have a functioning network connection for its BMC which happens to be the usual state of affairs in a data-center.) Need to tweak some firmware options or temporarily boot into a different operating system from a removable drive? IPMI makes that possible too.

The only prerequisite for having all these capabilities at your fingertips from anywhere in the world is the foresight to have configure the system for remote access ahead of time. Logical access controls define which services are available remotely (eg SSH vs IPMI), who is allowed to connect, what hoops they jump through in order to authenticate— there is likely going to a be VPN or Virtual Private Network at the front door— and finally what privileges these individuals attain once authenticated. The company running that server gets to define these rules. They are completely independent of the physical access rules enforced by the datacenter, which may or may not even the same company. Those permitted to remotely access servers over a network could be a completely different set of individuals than those permitted to step inside the datacenter floor and walk up to that same server in real life.

Attack surface of logical access

Logical access is almost as powerful as physical access when it comes to accessing data while having the convenience of working form anywhere in the world. In some cases it is even more convenient. Let’s revisit the example from the previous section, of walking into a datacenter and physically extracting a couple of disk drives from a server, with the intention of reading their contents. (We assume the visitor resorted to this smash-and-grab option because they did not have the necessary credentials to login to the server and access the same data the easy way even while they were standing right next to it.) There are scenarios where that problem is not straightforward, such as when disk encryption is used or the volumes are part of a RAID array that must be reconstructed in a particular manner. Another challenge is retrieving transient data that is only available in memory, never persisted to disk. There are ways to do forensic memory acquisition from live systems, but the outcome is a snapshot that requires painstaking work to locate the proverbial needle in the haystack of a full memory dump. By comparison, if one could login to the server as a privileged user, with a few commands the running application could be reconfigured to start logging the additional information somewhere for easy retrieval.

There is another reason logical access beats physical access: it’s easier to hide. Logical access operates independently of physical access: there is no record of anyone getting on an airplane, driving up to the datacenter gates, pressing their palm on the biometric scanner or appearing on surveillance video wondering the aisles. The only audit trails are those implemented by the software running on those servers, easily subject to tampering once the uninvited visitors have full control over the system.

Data-nativism as security theater

This distinction between physical and logical access explains why the emphasis on datacenter location is a distraction. Situating servers in one location or another may influence physical access patterns but has no bearing on the far more important dimension of logical access. Revisiting the Binance/US example from the introduction to illustrate this, there are three threat models depending on the relationship between the company and alleged threat actor.

  1. Dishonest, outright colluding with the adversary to siphon US customer data
  2. Honest but helpless in the face of coercion from a foreign government to hand-over customer data
  3. Honest but clueless, unaware that APT associated with a foreign nation has breached its infrastructure for collecting customer data in an unauthorized manner

In the first case it is clear that the location of data-centers is irrelevant. Binance/US employees collectively have all necessary physical and logical access to copy whatever customer information is requested and turn it over to the authorities.

The second case is identical from capability standpoint. Binance/US employees are still in a position to retrieve customer data from any system under their control, regardless of its geographic location. The only difference is a legal norm that such requests be channeled through US authorities, under an existing Mutual Legal Assistance Treaty (MLAT) agreement. If China seeks information from a US company, the theory goes, it will route the request through DOJ who is responsible for applying appropriate safe-guards under the 4th amendment before forwarding the request to the eventual recipient. This is at best wishful thinking under the assumptions of our scenario— a rogue regime threatening private companies with retaliation if they do not comply with requests for access to customer information. Such threats are likely to bypass official diplomatic channels and be addressed to the target directly. (“It would be unfortunate if our regulators cracked down on your highly profitable cryptocurrency exchange.”) For-profit organizations on the receiving end of such threats will be disinclined to take a stand on principle or argue the nuances of due process. The relevant question is not whether data is hosted in a particular country of concern, but whether the company and/or its employees have significant ties to that country such that they could be coerced into releasing customer information through extra-judicial requests.

A direct attack on Binance infrastructure is one where geography would most likely come into play. Local jurisdiction certainly make it easier to stage an all-out assault on a data-center and walk out with any desired piece of hardware. But as the preceding comparison of physical and logical access risks indicate, remote attacks using software exploits are a far more promising avenue of attack than kicking in the door.  If the government of China wanted to size information from Binance, it is extremely unlikely to involve a SWAT-style smash-and-grab raid. Such overt actions are impossible to conceal; data-center facilities are some of the most tightly controlled and carefully monitored locations on the planet. Even if target is greatly motivated by PR concerns to conceal news of such raids, even limited knowledge of the incident breaks a cardinal rule of intelligence collection: not letting the adversary realize they are being surveilled. If nothing else, the company may think twice about placing additional infrastructure in the hostile country after the first raid. By comparison, pure digital attacks exploiting logical access can go undetected for a long time, even indefinitely depending on the relative level of sophistication between attacker vs defender. With the victim none the wiser, compromised systems continue running unimpeded, providing attackers an uninterrupted stream of intelligence.

Physical to logical escalation: attacker & defender view

This is not say that location is relevant. Putting servers into hostile territory can amplify risks involving logical access. One of the more disturbing allegations from the Snowden disclosures involve Google getting sold out by Level3, the ISP hired to provide network service to Google data-centers. Since Google at the time relied on a very naive model of internal security and traffic inside the perimeter was considered safe to transit without encryption, this would have given the NSA access to confidential information bouncing around the supposedly “trusted” internal network. Presumably a compliant ISP in China will be similarly willing to arrange for access to its customers’ private fiber connections than one located overseas. Other examples involve insider risks and more subtle acts of sabotage. For example the Soviet Union was able to hide listening devices within the structure of the US embassy in Moscow, not to mention backdoor typewriters sent for repair. Facilities located on foreign soil are more likely to have employees and contractors acting at the behest of local intelligence agencies. These agents need not even have any formal role that grants them access; recall the old adage that at 4AM the most privileged user on any computing system is the janitor. 

One silver lining here is that risks involving pure physical access have become increasingly manageable with additional security technologies. Full-disk encryption means the janitor can walk away with a bundle of disk drives, but not read their contents. Encryption in transit means attackers tapping network connections will only observe ciphertext instead of the original data. Firmware controls such as secure boot and measured boot make it difficult to install rogue software undetected, while special-purpose hardware such as hardware security modules and TPMs prevent even authorized users from walking away with high-value cryptographic keys.

Confidential computing takes this model to its extreme conclusion. In this vision customers can enlist run their applications on distant cloud service providers and process sensitive data, all the while being confident that the  cloud provider can not peek into that data or tamper with application logic— even when that application is running on servers owned by that provider with firmware and hypervisors again in the control of the same untrusted party. This was not possible using vanilla infrastructure providers such as AWS or Azure. Only the introduction of new CPU-level isolation models such as Intel SGX enclaves or AMD SEV virtual machines has made it possible to ask whether trust in the integrity of a server can be decoupled from physical access. Neither has achieved clear market dominance, but both approaches point towards a future where customers can locate servers anywhere in the world— including hostile countries where local authorities are actively seeking to compromise those devices— and still achieve some confidence that software running on those machines continues to follow expected security properties. Incidentally, this is a very challenging threat-model. It is no wonder both Intel and AMD have stumbled in their initial attempts. SGX has been riddled with vulnerabilities. (In a potential sign of retreat, Intel is now following in AMD’s path with an SEV competitor called Trust Domain Extensions or TDX.) Earlier iterations of SEV have not fared any better. Still it is worth remembering that Intel and AMD are trying to solve a far more challenging security problem than the ones facing by companies who operate data-centers in hostile countries, as in the case of Apple and China. Apple is not hosting its services out of some AWS-style service managed by CCCP in a mysterious building. While a recent NYT investigation revealed Apple made accommodations for guanxi, the company retains extensive control over their operational environment. Hardware configured by Apple is located inside a facility operated by Apple, managed by employees hand-picked by Apple, working according to rules laid down by Apple, monitored 24/7 by security systems overseen by Apple. That’s a far cry from trying to ascertain whether a blackbox in a remote Amazon AWS datacenter you can not see or touch— much less have any say in the initial configuration— is working as promised.

Beyond geography

Regulators dictating where citizens’ private information must be stored and companies defending their privacy record by stressing where customer data is not stored both share in the same flawed logic. Equating geography with data security reflects a fundamental misunderstanding of the threat model, focusing on physical access while neglecting the far more complex problems raised by the possibility of remote logical access to the same information from anywhere in the world.

CP

Flash loans and the democratization of market manipulation

Borrowed guns

Imagine an enterprising criminal out to rob a well-defended gold vault in the middle of nowhere. Unfortunately for his burgeoning career, he has neither the command of a private army of mercenaries or any tactical gear required for the plan. Nor does our hypothetical crook at the beginning of a life of crime have the funds to acquire them yet. He could try buying those resources on credit, with the promise to pay the lender back with the proceeds from the successful heist. But most honest financial institutions have been getting gun-shy about lending to criminals and even the loan-sharks require some type of collateral— which, again our man does not have.

Luckily the neighborhood aviation company is running a special: anyone can walk-in and rent an Apache AH-64 gunship for a very low price, no questions asked. But the offer comes with a few strings attached:

  • This bird is programmed to return to its original take-off point after an hour.
  • It can not refuel. You get exactly one tank of gas to work with.

Borrowers can take off and do whatever they want with the helicopter— including a detour to rob the gold vault— but must return to the designated landing area. If they run out of fuel and crash land in the middle of nowhere, they will have to walk away from the spoils and watch as the stolen loot is recovered by its rightful owners. The world reverts to its previous state, as if the heist never happened.

DeFi exploits in the wild

This is one perspective on the concept of flash loans in decentralized finance. Anyone can initiate an Ethereum transaction and borrow funds which must be paid back at the end of that transaction. There is no collateral or credit-check required because it is not possible for the borrower to default. The immutable logic of smart-contracts enforced by the blockchain that guarantees this. If the loan is not paid back by the end of the transaction, the transaction “reverts”— it is still recorded on the blockchain and fees paid to miners for their effort, but nothing has changed. But if all goes well and the loan is paid, the changes that occurred within the span of that transaction— money changing hands, someone making a killing, someone else losing their shirt— is committed to the blockchain. Those possibilities are only limited by the maximum gas that can be consumed in a transaction, the virtual equivalent of the AH-64 fuel tank.

Not surprisingly, flash loans have been used for attacking DeFi exchanges and lending pools by manipulating the price signals those applications rely on. The attacks are complex and necessarily involve multiple defi contracts (exchanges such as Uniswap or lending pools such as Compound) and trading in/out of multiple assets. Here is a very simplified example of how such an attack can be executed:

  1. Flash-borrow a large amount of Ether
  2. Divide the ETH into two chunks of capital
  3. Convert the first chunk into token A, using a decentralized exchange. Now recall that DEXes do not have traditional order-books with ask/bid offers that can be matched when they cross. Instead they use automated market-makers (AMMs) which set the price based on the total amount of funds available on either side. More importantly, the liquidity available on these exchanges is often razor thin. It does not require a lot of capital to cause massive change in price. The result of this large, single “buy” order to convert ETH → A is that the “price” of A goes way up on the decentralized exchange. In other words, there is massive slippage. This type of trade is normally a terrible idea—the buyer effectively overpaid for A when they could have gotten a much better deal if they traded on a centralized exchange. So how can an “attacker” make up lost ground if they are starting out with such a lousy trade?
  4. Convert the second chunk of ETH into asset A. The trick is using a different venue for this than #3. Goal is for this trade to execute at close to fair market price and avoid slippage.
  5. Time to visit the real victim, yet another defi application. There are specific criteria for selecting this target:it must be using the venue from step #3 as its price oracle. In other words, when the attacker tries to trade A or borrow using A as collateral on the target venue, that venue will rely on faulty price signals from #3 which has been artificially manipulated by the attacker. (Recall that everything is executing inside a single ethereum transaction orchestrated by the attacker; no other trades that could interfere with this mispricing can occur.)
  6. This time the attacker has a favorable trade from A → B.The target venue is working with an overinflated price for A, because the last A-for-ETH transaction artificially inflated the price of A relative to ETH. The market maker is willing to swap/lend an outsized quantity of some other token B in exchange for a small amount of A. This is the crucial step. While the attacker lost money on the first chunk and ended up with a deficit in asset A, they aim for a killing on the second chunk, ending up with a surplus of asset B relative to the value of A exchanged.
  7. Time to pay back the flash loan. The attacker converts enough of their holdings in A and B back to ETH to cover the original loan, again using a venue where price indications are not distorted. The proceeds are used to close out the flash loan and complete the transaction successfully.
  8. Assuming the profit from B exceeds the losses on A, the attacker comes out ahead. (What if the math did not work out? No harm, no foul. The transaction will revert. So the attacker does not stand to lose any money beyond the Ethereum gas fees paid for the attempt.)

This is a highly simplified view; actual attacks can be far more complicated. Any asset can be flash-loaned, so the starting point need not be ether. However the loan has to be paid back in kind, so the attacker is still on the hook for returning the identical amount. The exchange process may involve multiple hops such as ETH → A- → B → C → … → ETH before the cycle is completed. For more concrete examples, see this 2021 paper or breakdown of the recent attack on CREAM which involved dozens of steps within a single Ethereum transaction. That paper also poses the question of whether attacks in the wild were being “optimal” in how they divided up the total amount borrowed into two chunks. The surprising answer is they are far from optimal: in each case, a different allocation between different assets A and B would have resulted in a more profitable heist. The crooks left money on the table. (Incidentally, you have to wonder about the ethics of academic research that doubles as a handbook on committing more optimal robberies and leaving less money behind in the virtual vault.)

Root causes

With this background on how flash-loans are leveraged in recent attacks, we can revisit the original question: were flash loans the root cause? The answer is clearly no. Weak connection between prices on decentralized exchanges and the “real” market prices elsewhere is the real culprit. By definition blockchains are isolated systems: they can not interact with the outside world. A smart contract can not sidle up to a Bloomberg terminal and request a fresh quote on current commodity prices. It must rely on indirect indications, such as trusted pricing oracles maintained by others on-chain or observed actions of participants interacting with the contract when trading an asset. Multiple DeFi exploits have demonstrated that these signals are surprisingly easy to manipulate given enough capital. When taken in isolation, each such instance of manipulation looks self-defeating: “attacker” gets the price of an asset completely out-of-whack on one particular exchange but only by making a lousy trade. Whatever distortion is achieved will be short lived, as other investors take note of the mispricing and jump-in to quickly arbitrage away the difference. Why would any rational actor engage in this meaningless gesture? Because other venues rely on the same distorted price signal and create profit opportunities far exceeding the loss on the original trade. This is an intrinsic structural weakness for some— but not all— decentralized application with flawed pricing signals.

From this perspective, flash-loans did not enable a new class of attacks that were impossible before. The sequence of actions depicted in the previous section could have skipped the first step — flash borrowing— and start out with an existing pool of capital already in the hands of the perpetrator. Even the most extreme case of the recent CREAM attack involved a $500MM USD flash loan. There are many hedge-funds and high net-worth individuals in possession of amounts in that neighborhood. Every one of them could have executed the exact same transaction without borrowing a single wei. Seen in this light, flash loans democratize the possibility of market manipulation.

This episode has parallels in a story covered in an episode related by Michael Lewis in Flash Boys. Goldman Sachs argued that high-frequency trading source code allegedly stolen by its one-time employee Aleynikov could be used for “unfair market manipulation.” To which Lewis effectively retorted: If such code exists, is the real problem that Aleynikov had possession of it? Is market manipulation “fair” when the same algorithm is wielded by Goldman? To the extent DeFi applications are built on flawed pricing signals, they are vulnerable to manipulation. Whether the manipulation is done with institutional capital on-hand or aided by no-questions-asked flash-loans seems irrelevant.

Deterrence at the margins

One counter-argument is that reputable market participants with large concentrations of are unlikely to attack smart-contracts regardless of profit opportunity, for fear of legal and reputational risks. This is complicated by the ambiguity of what qualifies as attack. It is not clear that what happened to CREAM and others is a traditional “hack” in any sense. There were no logic bugs in the contract. There was no compromise of a secret key held by the CREAM team. Other smart-contracts such as DAO or the Parity multi-sig wallet suffered massive losses due to logic flaws in their implementation. In both of those cases, the smart-contract had a glaring programming error such that its behavior diverged from their intended behavior, however informally specified that may have been. Compare these two cases:

  • In the case of Parity, the expectation is that only the owner of the wallet can withdraw funds form their wallet. If everyone in the world can take money out of your wallet, there is no ambiguity: the contract has failed to implement the intended policy. Anyone taking advantage of that flaw is exploiting a security vulnerability and committing theft.
  • In the case of CREAM the contract worked exactly as intended, using precisely the price signals it was expected to consume. But the designers did not look far enough ahead to understand how their creation would behave in extreme circumstances when those signals become wildly distorted. If the casino designs a game such that clever players can inflict massive losses on the house while playing by the rules, is it an “attack” to implement that strategy?

If this is not a breach in the traditional sense, one could at least hope that it qualifies as market manipulation. (Standard disclaimer: the author is not a lawyer and none of this should be construed as legal advice.) At least that categorization could serve as a deterrent for participants interested in staying on the right side of the law. But it is unclear how existing statutes for trading securities or commodities apply in the context of blockchain assets. While this post liberally uses the term “market manipulation,” not every instance of buying up large quantities of something to profit from the artificial scarcity is necessarily criminal. Not every scalper hoarding Hamilton tickets for resale merits an SEC investigation. Even if the perpetrators of these attacks were identified and prosecuted— unlikely given the relative anonymity of blockchain transactions— they may well rest their defense on the claim that “manipulation” is impossible when dealing with a system that is defined by immutable rules implemented in code.

 On the other extreme, if we posit that what happened to CREAM constitutes criminal activity that falls under SEC or CFTC jurisdiction, some troubling questions are raised about the venues providing the flash-loans. Is there liability? Did they aid and abet theft? Returning to the opening hypothetical about the helicopter available for anyone to borrow: if that craft turned up as the get-away vehicle for an actual robbery, surely the owners would have some explaining to do. Were they aware that this customer intended to commit criminal activity? Did they conduct any due diligence? Saying that the business has a policy of not asking any questions reeks of willful blindness. Virtually all flash-loans on Ethereum today follow this model— since the loan is guaranteed to be repaid, the lender does not have to care about the creditworthiness of the borrower. But that narrow focus on avoiding defaults misses the negative externalities created by (temporarily) arming random people with large amounts of capital to wreak havoc on other blockchain applications. Did Maker aid and abet criminal activity in providing the half billion dollars in capital used to drain CREAM? In the same way that Aave is contemplating the creation of a permissioned lending pools subject to Know-Your-Customer rules, flash-loan providers may need to revisit their strategy around doing business with anyone.

CP