Can our current crypto infrastructure support the decentralized future of, anything?

On 12/14 we saw an apparent supply chain attack on Ledger connect software using the Ledger connect-kit affecting various dApps and wallets. This is extremely unfortunate and we always hate to see this happen in the space. Based on the information available it seems that this scenario was avoidable across various security and architecture domains. Our deep dive into this scenario also poses greater questions around our decentralized architecture for crypto assets and NFTs. If one dependency, system, or application can have such an impact on so many individual users, how decentralized are we really?

Below we break down the chain of events, how they happened, and how they could have been prevented. It is our hope that continued lessons learned can lead to safer and better Web3 for all of us. The decentralized future and arguably the future of finance, art, and self-sovereign distribution models depends on it.

This tweet (and others) serve as the sources of information received from Ledger and other affected wallets and entities. We used the tweet excerpts highlighted to explain the series of unfortunate events that occurred.

“This morning CET, a former Ledger Employee fell victim to a phishing attack that gained access to their NPMJS account”

In the first of this series of events ex- Ledger employee Jun Sugiura was targeted as part of a spear phishing exercise to his personal Gmail account. Once the attacker gained access, they were able to access their NPMJS account that had the ability to access Ledger content and then modify the contents of the Ledger connect-kit.

For additional context, NPMJS is a package manager for JavaScript and allows developers to install software dependencies. The Ledger connect-kit is a library that is used by multiple dApps to communicate with a Ledger device.

Typically, when an employee leaves a company their access is revoked enterprise wide and those accounts are unable to access any company network, software, or tools.

Twitter user @0xSentry captured some of the settings that show users/maintainers to the library in a now deleted tweet that indicates that it is likely that external Ledger accounts (contractors) and personal accounts retained access that is possibly unauthorized. Accounts for terminated users should have been removed to avoid this scenario and access to NPM contents should be limited and restricted to authorized personnel. There should also be logging and monitoring to ensure published versions are indeed made by appropriate personnel and are not malicious.

The accounts are NPM accounts and we do know that NPM supports a security key for two-factor authentication. If configured, this could have prevented the user’s account from being compromised.

Next, “The attacker published a malicious version of the Ledger connect-kit (affecting versions 1.1.5, 1.1.6, and 1.1.7). The malicious code used a rogue WalletConnect project to reroute funds to a hacker wallet.”

Once the attacker had NPM account access they were able to unilaterally upload a malicious version of the Ledger connect-kit package. This malicious version included code that would drain users’ wallets once they signed the message in the wallet.

Upon further investigation, no updates were made to Ledger’s official Git repo which indicates that the NPM repo was a single point of a failure in the SDLC (Software Development Life Cycle) architecture. As a best practice it should never be possible for a single user to publish production software at any point in the SDLC.

So how was an NPM account a single point of failure?

This excerpt was taken directly from the Ledger Github read me:

The @ledgerhq/connect-kit-loader allows dApps to load Connect Kit at runtime from a CDN so that we can improve the logic and UI without users having to wait for wallet libraries and dApps updating package versions and releasing new builds.

Ledger deployed the connect-kit across a CDN. For context, a CDN is a network of servers which distributes data across multiple regions in order to reduce the response time. The CDN being used is jsDelivr. jsDelivr can instantly serve any file from any NPM package in the public registry. New versions pushed to the NPM are instantly available via the CDN with no required maintenance.

In this architecture, users of the connect-kit are downloading the package directly from the web link without any verification of the content. jsDelivr retrieves the latest version of the NPM package for the Connect-Kit and publishes it across the CDN for use.

According to the jsDelivr documentation the @1 in the CDN link referenced by the connect-kit-loader package ("https://cdn.jsdelivr.net/npm/@ledgerhq/connect-kit@1") would automatically point to the latest release published on NPM that was major version 1, allowing the attacker’s malicious package to be seamlessly hot-loaded by every application containing the connect-kit-loader package.

This means that dApps are dynamically loading code from the CDN/web without being able to properly verify the contents.

Additionally the connect-kit-loader package does not perform any integrity checks of the contents it downloads from the CDN, which could have prevented the malicious package from being loaded even if the CDN was compromised.

While using a pinned version of a package and reviewing changes before upgrading is the preferred method of ensuring dependency integrity, especially for such a sensitive application, in reality many developers do not abide by this procedure. It appears that Ledger was likely trying to improve the security of the ecosystem by implementing hot-loading functionality, as this design allows for the immediate propagation of critical security updates across all dependent applications without requiring action on the part of their respective developers. However, because of severe deficiencies in the security of their release pipeline/process and the complete absence of integrity checks of the hot-loaded package, this became a massive single point of failure ripe for exploitation.

It is well known that the JavaScript ecosystem is prone to supply-chain attacks because of large numbers of dependencies and sub-dependencies, and we’ve seen attacks that target obscure and little-scrutinized packages that are depended upon by larger frequently used packages (see the flatmap-stream incident, amongst many others). However, this attack is particularly galling as it directly compromised the package of well-known and generally trusted company in the crypto space. This should not be considered solely a result of the known limitations of the JS dependency-management ecosystem, but a failure to implement basic security best practices.

A better design to support this architecture would be to have a separate server from the CDN that publishes the hash of the latest legitimate version of the Ledger connect-kit and the loader package should call that endpoint and compare the hashed versions. No one should be required or encouraged to blindly load code without pinning a specific version and checksum. dApps and developers also had the option to download the library directly to ensure they are using the latest and verified packages.

The design and architecture of the connect-kit highlights a broader issue in the industry around the vast amounts of third party dependencies that custody and wallets ultimately have. While this is similar to many Web2 applications and historically has not been as much of a threat due to the closed ecosystems of those applications, it presents a very different problem in a decentralized ecosystem that affects millions of users.

As we saw, this did not just affect Ledger wallets, and the impact spread to other wallet providers and connectors that relied on this specific software. Any process that fully trusts a third party such as NPM, Github, Amazon, etc is inherently flawed. Third party tools and providers should be configured appropriately for security at minimum, but the reality is that all wallets, and subsequent connections pose a single point of failure risk.

“We remind you to always Clear Sign with your Ledger. What you see on the Ledger screen is what you actually sign. If you still need to blind sign, use an additional Ledger mint wallet or parse your transaction manually.”

The malicious code injected a popup that prompts a user to sign a transaction which would drain the wallet if the user executed a signature. The private key of the wallet is unaffected and instead a malicious front end is presented to trick the user into signing a transaction. In other words, not your keys not your crypto and not your code not your crypto.

We don’t believe in blaming users for security failures, however a crucial part of the attack is dependent on the user signing a malicious transaction unknowingly. So, how does this happen? How can this be improved, and eventually prevented?

In terms of signing transactions there are three levels of transaction signing; clear, transparent, and blind.

In blind singing, users are presented with a message that is designed for computers vs humans and takes an indecipherable sequence of bytes and blindly signs it with their private key, which allows for whatever it contains to be executed on-chain. In this case, a transaction that drains the wallet is executed. Users are basically looking at pure raw data.

Blind signing has become the norm in crypto and too often users will sign anything they are prompted for. SO, we have blind downloads and blind signing for the future of finance. Sick.

Transparent signing parses the data and information using the contracts ABI (Application Binary Interface) making it more human readable. This allows a user to verify the contract they are interacting with, type of function, and amounts and gives them transaction information to verify vs bytes. This is still a transformed version of the raw data and the conversion needs to be properly secured or else an attacker can modify this also.

Clear signing takes the raw data and the ABI information and presents a summary of transactions in human readable form that is easy to read and understand. For example. Swapping x USDC for x ETH on address xyz. THIS should be the standard.

In short, we all have the responsibility to know what we are signing, but as an industry we should not expect the vast majority of users to understand the data and information presented to them as bits and byte. Even the flyest of internet girlies can’t read and understand what they are signing when its in code. End user security AND usability should be the norm, not a net new feature and devices should support screens that can present the user with actual text about their transaction that is human readable. Shout out to all the developers working on tools that help with malicious transaction identification and all the work being done in that domain, as this will also need to be the norm and set the bar for transaction review.

The state of security across both Web2 and Web3 infrastructures and architectures will heavily influence the success or failure of crypto being able to become the decentralized future of money, art, or anything.

Special s/o to @realmatteobruni for his contributions to this entry.

xo, xo