Incident Summary

At 10:09AM UTC, October 18, 2024, Tapioca DAO suffered a security breach as a result of a social engineering attack targeted at a core contributor, which resulted in the loss of approximately 605 ETH and 3.1M USDC totaling $4.65M USD. These stolen funds are currently being held across several known addresses, with the primary address of the hacker being 0x69d91e56ca80f2a4d7b808b59053ea5c5505ffe2, where (as of time of writing) $3.915 million USD in cryptographic assets are being held on Binance Smart Chain.

The security breach was made possible via the attacker(s) compromising the private keys of a core contributor responsible for smart contract development. The compromised engineer had been a trusted lead smart contract engineer for Pearl Labs since inception, or for approximately two and a half years. SEAL911 identified the attackers as a North Korean group, used a "contagious interview" attack methodology to inject malware on the contributor’s computer in order to gain control of the private keys of his address to carry out the theft. The attacker(s) have been linked to several similar incidents involving contagious interview attacks by respected individuals in the space. The Contagious Interview is a tactic used by attackers, who pose either as a job seeker or recruiter, to trick the target into downloading files that appear legitimate but actually contain malware. The goal of this method is to exploit the trust typically associated with the hiring process.

The attacker utilized the control of the engineer’s private keys in order to:

• Mint 315.5 trillion USDO stablecoins to drain the USDO/USDC liquidity pool.

• Steal 29,669,866 TAP tokens from the vesting contract, valued at approximately $38 million at the time of the attack, to drain the TAP/ETH liquidity pool.

The 315.5 trillion USDO tokens were exchanged for roughly 3.1 million USDC from the USDO/USDC liquidity pool, and the 29.6M TAP tokens were exchanged for additional $1.6 million USD in ETH from the TAP/ETH liquidity pool, totaling approximately $4.75 million stolen, a supermajority of which are DAO owned funds.

This attack was made possible due to the admin controls of both token contracts being controlled by a single signer, with this signer address being a hot wallet. Security personnel and other contributors had previously repeatedly directed the compromised engineer to transfer the admin role of the smart contracts to the DAO’s 4-of-7 multisignature wallet after the June 2024 deployment of the Tapioca protocol. The engineer was only directed to implement single-signer controls for pause functionality to enable swift reaction to any security events, and to integrate Hypernative security monitoring. Additionally, the DAO provided the compromised engineer with a cold storage wallet, hardware 2FA key, and other security devices. Despite these directives and precautions, the engineer failed to follow or implement the directives, use any of the provided devices, and further additionally received and ignored a direct warning about “contagious” interview attacks less than three months prior.

The engineer, who was compromised, both stated that the directives were properly completed and created the appearance of completion by closing task(s) on ClickUp. This was mistakenly taken as fact by the DAO due to his seniority and the absence of formal compliance measures.

Attack Methodology

The social engineering attack started when the compromised engineer was targeted through LinkedIn, and further communicated through Telegram by and with an individual posing as “Alberto Flores Galvan,” who was offering the engineer a fake job opportunity.

The engineer was asked to download a malicious .env file containing malware, leading to the compromise.

Once the attacker(s) obtained the compromised engineer’s private key, they transferred the admin ownership of the TAP token’s vesting contract, stole 29.6M TAP tokens held within it via a “rescue” function, then liquidated these stolen tokens to steal the majority DAO owned ETH liquidity held within the TAP/ETH liquidity pool. Admin Ownership within token vesting contracts in of itself is a common practice within DeFi to modify vesting recipient addresses.

The attacker then added a new address to USDO’s whitelist of minters, minted USDO stablecoins, and stole the DAO owned USDC liquidity in the USDO/USDC liquidity pool with the illegitimately procured USDO. The purpose of this whitelist was similar to Aave’s GHO “facilitators”, wherein if the Tapioca DAO elected to create new methodologies to mint USDO, they could be directly added as a minter to the contract.

Attached below are the recoverable screenshots of the conversation between the engineer and attacker on LinkedIn & Telegram. The only information redacted is the first name and handle of the engineer.

To provide context for the compromised engineer’s actions, Pearl Labs began restructuring the engineering staff in June 2024, prompted by the twAML economic issue which caused the Tapioca protocol to be paused. EnigmaDarkLabs was brought in at this point to advise Pearl Labs and oversee development. Enigma then performed an analysis of development processes and recommended that both senior engineers be terminated for lack of expertise, sluggishness, and recklessness. Based on this recommendation, Pearl Labs leadership reached a compromise, one of the two engineers would be terminated, while the other, would be demoted to a junior developer role while new experienced engineers would assume the CTO and Senior engineer roles.

However, due to the timelines to employ new engineers to develop and deploy Tapioca V1.1, both engineers were retained in their existing positions to complete V1.1 under the supervision of internal security personnel while leadership carried out the restructure. As both of the original engineers' existing issues worsened, one of the two aforementioned engineers was terminated before the release of V1.1- just two days prior to this incident taking place. This termination prompted the compromised engineer to rapidly seek employment without anyone’s knowledge, making them especially vulnerable to the contagious interview attack. This, in concert with the compromised engineer’s insubordination of directives regarding contract admin controls and lack of utilization of any of the provided security devices, provided the attacker an easy pathway to carry out this simple attack.

Incident Response

The security breach was reported at 11:18 AM UTC, and within minutes, security personnel engaged SEAL911. A war room was established by 11:22 AM UTC, and the root cause was identified within five minutes: the attacker had full control of the admin role, preventing any immediate countermeasures to stop the theft. The team quickly removed liquidity from the two key Uniswap V3 pools, recovering approximately $700,000 USD.

Further investigation revealed that the attacker had 996 ETH ($2.65 million USD) in collateral under their control. With the assistance of EnigmaDarkLabs, the security team utilized an exploit to recover the ETH from the attacker before he could launder it. As a result, the DAO recovered $2.65M USD from the hacker’s control. At this time it has been advised to withhold the specifics of this exploit due to ongoing circumstances. However, as soon as possible, this will be provided as an Addendum to the Post Mortem.

Following the attack, the DAO's treasury, excluding TAP tokens, stands at ~$4.8 million, representing a 45% loss from where it stood before the social engineering attack.

Detailed Events Timeline

Arbitrum: Ownership of Tapioca Multicall (Deployer) contract gets transferred to attacker:

Oct-18-2024 10:09:07 AM +UTC

Arbitrum: Attacker adds new minter to USDO:

Oct-18-2024 10:09:27 AM +UTC

Arbitrum: Attacker mints USDO:

Oct-18-2024 10:09:49 AM +UTC

Arbitrum: Attacker swaps newly minted USDO for USDC, emptying the Uniswap liquidity pool:

Oct-18-2024 10:11:00 AM +UTC

Arbitrum: Ownership of TAP token vesting contracts transferred:

Oct-18-2024 10:56:05 AM +UTC

Oct-18-2024 10:57:07 AM +UTC

Oct-18-2024 11:05:24 AM +UTC

Arbitrum: Emergency Rescue of TAP token vesting contract called, draining vested TAP:

Oct-18-2024 10:56:37 AM +UTC

Oct-18-2024 10:58:11 AM +UTC

Oct-18-2024 11:07:28 AM +UTC

Arbitrum: Swapping stolen TAP & USDO tokens on Uniswap:

Oct-18-2024 11:00:45 AM +UTC

Oct-18-2024 11:15:48 AM +UTC

Moves stolen funds to Binance Smart Chain:

TAP -> WETH -> USDT (Stargate transactions)

Oct-18-2024 11:04:19 AM +UTC

Oct-18-2024 11:09:35 AM +UTC

Oct-18-2024 09:26:03 PM +UTC

USDO -> USDC (swapped to USDT) via Stargate:

Oct-18-2024 10:13:39 AM +UTC

Oct-18-2024 10:15:58 AM +UTC

Oct-18-2024 10:21:19 AM +UTC

Oct-18-2024 10:28:39 AM +UTC

Oct-18-2024 10:29:42 AM +UTC

Compromised Addresses:

0x30EC55E70282AC3A032A3C7003Ead8B79c76Fde6 (developer address whose private key was compromised)

Security Measures

The DAO has implemented necessary actions to ensure that such issues do not arise in the future. The engineer compromised in this social engineering attack has been terminated and removed from all sensitive services, channels, and controls. Moving forward, the following changes and improvements will be enforced to strengthen the security and governance of the protocol:

Web3 Security Measures:

1. No Direct Access to Core Protocol: No individual contributor or individual will have direct access to core protocol functions or critical contract roles. This measure ensures that no single member can compromise the system through social engineering or direct attacks.

2. Separation of Duties: Contributors responsible for deployments or on-chain actions will not hold ownership roles in the protocol. The protocol's ownership will default to a Governance Executor that is controlled by the DAO or a TimelockController managed by the Tapioca multisig.

3. Multi-Party Review Process: All on-chain actions, including deployments and configurations, will be thoroughly documented, simulated, and reviewed by at least two independent parties. Once validated, the action will be queued in the TimelockController, with a mandatory two-day delay before execution. This ensures any vulnerabilities, bugs, or malicious actions can be addressed before finalization.

4. Real-Time Observability Portal: A dedicated portal will monitor the protocol in real time, displaying current roles, ownership, and token balances across all smart contracts. This portal will ensure transparency and provide immediate detection of any unauthorized changes.

5. Anomaly Detection Systems: Automated systems will continuously monitor for suspicious activity in smart contracts, bridges, oracles, and the underlying blockchain. If anomalies are detected, the protocol will automatically pause, preventing further damage.

Multisig Security Measures:

Even though the attack did not target the multisig directly, the DAO is enhancing security protocols for multisig signers:

1. Isolated Signing Environment: All signers must use isolated machines equipped with hardware wallets which will be provided by the DAO, and these machines will be reset to their original state after every reboot.

2. Transaction Simulation and Verification: Signers must simulate each transaction using tools like Tenderly or Foundry and ensure the results match across all simulations. Any inconsistencies must be reported to the security team before signing.

3. Payload Data Verification: The signer must verify that the data payload in their hardware wallet matches the transaction. Any discrepancies will prompt immediate contact with the security team.

4. Continuous Availability: Signers must be reachable at all assigned times to ensure prompt handling of urgent transactions, with direct lines of communication to other signers.

   Conclusion

   In summary, this incident was equally as unacceptable as it was avoidable, and the DAO takes full responsibility for this theft. While there was no malicious intent involved, the compromised engineer’s failure to follow basic directives combined with leadership’s misplaced trust, led to this breach.

   The two fatal and key assumptions were:

   • The compromised developer was using the provided cold storage wallet for critical actions, such as deployments and managing "pausable" roles.

   • The ADMIN role had been correctly transferred to the 4-of-7 multisig, as instructed.

   Moving forward, the DAO is committed to strengthening its security measures and ensuring much stricter compliance to prevent such breaches in the future.

   Debrief

   As of this post-mortem release, we have identified the location of all stolen funds, and relevant authorities have been notified. The Tapioca DAO team, BNB Chain security, and zeroShadow are closely monitoring the situation in real-time.

   All of the DAO contributors are deeply devastated by this attack, and are continuing to work 24/7 to assist the relevant agencies and outside collaborators to identify the exploiter and recover the funds as soon as possible. DAO Contributors are also working tirelessly on a Token Migration Plan & Mainnet Plan to be released as quickly as possible.

   Company wide punitive measures have been taken, such as the compromised engineer was terminated effective immediately, and the engineer’s contributor token allocation has been revoked. All Pearl Labs founders contributor token allocation(s) were also revoked in a self imposed fashion, to potentially be earned back over time from the Tapioca DAO. Combined, this totals 9M TAP or nearly 10% of the supply of the 15% or 15M TAP. Additionally, Tapioca Foundation’s vesting schedule will be reset on the day of the migration of the token. Lastly, further details regarding the token will be documented in the upcoming Token Migration Plan. Restructuring is actively underway of the Solidity engineering roster.

   The hacker was given the chance of returning $3.7M in exchange for a $1M USDT whitehat bounty, this offer has since elapsed and been revoked on October 22nd, 2024 at 4pm UTC, and now a 10% bounty is available for anyone who provides information which leads to the recovery of the stolen funds.

   This attack, while severe, was not sophisticated and could have been prevented by adhering to basic operational security practices, smart contract development standards, and compliance measures. The breach occurred due to shortcomings in these critical areas within the DAO and is not any one individual’s fault. Moving forward, the only path is to acknowledge these failures and focus on improving our processes. By doing so, we aim to rebuild trust and become a stronger organization simultaneously.