Introduction
With the recent major protocol launches in the Cosmos ecosystem bringing massive amount of capital, such as Celestia, dYdX, and Noble, among others that are soon to hit mainnet (Namada, Berachain, Penumbra..), it has now become an attractive ecosystem for extracting MEV.
What makes this extraction unique is that it is no longer focused on a single-ledger, but on multiple and interconnected blockchains leveraging a standardized cross-chain communication protocol. This opens up arbitrage opportunities across different chains.
Unlike EVM-chains, where MEV extraction can be easily measured due to their single-chain architecture, cross-domain MEV lead to challenges related to transaction ordering on asynchronics networks, centralization forces and latency wars that incentivize colocation, among others.
MEV on CometBFT-chains
Low-fee chains and CometBFT (prev. Tendermint) blockchains in the Cosmos ecosystem face specific challenges related to MEV extraction. The first-come-first-serve (FCFS) mempool order and fixed gas prices in these chains create negative externalities such as network spamming resulting in wasted block space and centralizating profits across third parties who can gain an advantage due to information discrepancy. These challenges intensify as transaction volume increases across chains, adversely affecting the overall user experience.
Transaction ordering relies on the order of their arrival at a specific node. Without a mechanism to prioritize transactions, searchers resort to strategies like colocation and are incentivize to get involve in latency wars to gain an advantage. Colocation involves positioning arbitrage bots close to validators and full nodes to minimize latency and increase the chances of transaction inclusion. This practice centralizes profits on HFT firms, which are economically incentivized to do so. This disadvantages users who are geographically distant from the majority of nodes, the users who are farthest away from the majority of nodes will have a substantial disadvantage compared to others, even if their transaction was sent first.
Currently, in Ethereum, users participate in some kind of ‘auction’ in which they bid the highest gas fee they are willing to pay the validator to have their transaction processed and included above the others on the network. However, in Cosmos blockchains this is not the case: users have no way to communicate to the current validator that they want their transaction processed first. This lead to searchers spamming their transactions to the nodes to increase their chances of probabilistically capturing MEV. If you’re interested, I also wrote a blog with some thoughts about why ‘fair-ordering’ does not exist
Spamming networks is a dominant strategy for extracting MEV on low-fee, low-latency chains. Transactions are cheap, and the marginal cost of spamming the network is negligible. Although the gas price cannot be increased, the strategy of sending the same transaction repeatedly until it is finally processed consumes unnecessary space in the block, increase nodes computation congesting the network, and creates a bottleneck between normal users and bots that attempt to extract MEV.
Cross-chain MEV on interconnected networks
Cosmos is a network of many IBC-connected blockchains rather than a single blockchain isolated. This opens a wide range of opportunities for MEV to be captured in the Interchain space, such as cross-chain arbitrage taking place on different chains leveraging the IBC through which they are connected. For example, an arbitrage opportunity between Osmosis DEX on Osmosis and Diffusion on Evmos.
Borrowing the cross-domain MEV explanation from Unity is Strength, ‘This extraction requires moving or bridging assets across three domains. The opportunity now depends on the ordering of transactions in multiple domains rather than a single one and the mechanics by which this opportunity is seized seems to involve transferring assets across domains, a non-atomic operation that breaks the atomicity’.
Entities with sufficient capital and infrastructure on both ends are likely to capture the majority of cross-domain MEV from arbitrage.
Cosmos chains rely on dPoS and typically have a capped validator set of 150/175 at most, which can lead to heavy centralization with the same actors controlling each blockchain. As a result, it is well-known that a few entities will control block production within these blockchains. The MEV extraction is centralized in the upper part of the validator set, this results in a high success rate for the operator who has a monopoly on block production in both chains when their proposer slots overlap.. This MEV is only accesible by well-capitalized entities running validators in the upper-part of this blockchains.
In scenarios where a single entity controls a substantial stake in two different chains, cross-chain multi-block MEV strategies become possible. By manipulating the ordering of blocks in both chains, this entity can execute probabilistic strategies that others cannot. Controlling the simultaneous ordering of transactions in multiple chains enables the execution of complex MEV strategies and amplifying the potential for profits.
With the launch of Replicated Security, entities will now have to run more than one or two nodes. In fact, they may need to run multiple nodes if they want to participate in running consumer-chain nodes that are secured by the Hub. This completely changes the current MEV landscape, as not all entities running validators in the Hub will have the infrastructure to run other nodes, leaving MEV extraction on the table to other well-capitalized players. The bottom set of validators is generally unprofitable for the vast majority, with many of them barely breaking even. Now, having to secure multiple networks, this issue can become even more problematic.
This combination of features creates an ideal environment for MEV extraction. Only well-capitalized entities with sufficient infrastructure, the know-how, and willingness to take on additional slashing risks can effectively compete. Consequently, only a specialized subset of validators is likely to earn the majority of the Interchain MEV, leaving many other validators (particularly those in the bottom set) without the opportunity to access these rewards. This is essentially the 'the rich get richer' problem.
Analyzing the Interchain MEV landscape
Atomicity is necessary to guarantee MEV extraction. For MEV opportunities to occur the cycle must be atomic: something happened, or it doesn’t. However, IBC logic does not support this by design. All communication must happen asynchronously on each single blockchain.
Theoretically, atomic transactions that involve IBC communication would require a shared state machine where synchronicity is shared between nodes, e.g. a network state that leverage Heterogeneous Paxos as a consensus protocol. This would enable users to ensure that all of those atomic transactions are included in the app-chain state or none of them are included. The main benefit of this approach is that it would allow transactions to run atomically in one block across multiple chains.
Let's explore the trick that resolves this complex cross-communication mess. The likelihood of a validator having simultaneous slots on multiple blockchains depends on their stake and the way block building is organized on CometBFT chains. Validators with a significant stake in multiple chains can access exclusive MEV opportunities by having the monopoly of block production.
The table from NumiaData shows the overlap of voting power between validator sets in Cosmos chains. It shows how often single entities get overlapping slots across chains. For example, 93% of the voting power of validators in Passage are also currently validating in Juno, while 42% of the voting power in Juno is also currently present in Passage.
Since a single entity can control a significant stake in two different chains, they can gain an advantage in cross-chain MEV extraction by controlling the ordering of two blocks simultaneously. However, when two chains commit blocks at different blocktimes, it introduces latency between them because networks are asynchronous. Therefore, cross-chain MEV extraction is probabilistic and depends on the timing of the blocks on each chain.
Now, a new player is introduced into the MEV supply chain: the relayer.* The classic MEV supply chain in a single-domain looks something like this:*
To extract this MEV within the Interchain, validators on Chain A must need to connect to other blockchains leveraging IBC connections/channels. In this scenario, a new player gets involved into the MEV supply chain: the Relayer. Relayers enter the MEV Supply Chain by acting as a middleware between validator sets, and this posses a potential centralizing flywheel to their agnostic purpose.
The Cosmos ecosystem differs substantially from other ecosystems in which there are many bridges connecting different blockchains, their market share is divided among them, resulting in liquidity fragmentation between wrapper token versions of a single token of different bridge providers. The Cosmos ecosystem async communication is lead entirely by IBC, so whoever controls relayers between chains will also potentially control ordering.
IBC itself moves around ~$3B in volume per month without any incentives nor speculation from end-users due to its agnostic approach, making it a credibly neutral protocol that attracts organic usage, even without being connected to EVM-chains.
Relayers play a role in facilitating cross-chain communication, they even might be colluding with the entity controlling multiple chains. Although there’re plans to implement a relayer fee incentivization, currently they are entities that operate at a loss. As a result, only financially motivated and well-capitalized entities, such as those connected to the same ~5 validators, run these relayers because they also know how to maintain the infrastructure. These validators also happen to handle the highest transaction volumes, which further strengthens their position in the MEV landscape.
The potential trade-off of relayers entering the MEV Supply Chain is that they are designed to be agnostic and perform a single function: to relay the packets, which contributes to their ease of use and permissionlessness. However, if they become part of it, their original purpose may be compromised and friction and complexity may be introduced into the system.
Larger validator sets in Cosmos aren't sustainable
It's a common belief that the more validators join a network as block builders, the more divided the market share will be, leading to greater decentralization. However, this is a misconception. The upper-bottom of the set is not profitable, and many validators are just breaking even. The belief that more actors equal more decentralization is not always true.
While expanding the validator set allows validators to enter, it has not shown evidence of shifting the overall distribution of stake in the long run. A mechanism to address the lack of profitability for small validators running multiple consumer chains must be considered.
The smallest validator in the active set faces significant challenges when we compare their delegation numbers with those of the largest validators. While their delegation may be lower, the costs associated with running even a single consumer chain (running a single consumer chain can cost $200 to $600 monthly in expenses) can outweigh their earnings. This lack of profitability poses a risk to their sustainability.
Larger validators in the top enjoy more substantial delegations, which result in significantly higher earnings. With a stronger financial foundation, these validators can more comfortably operate multiple consumer chain nodes.
Approaches to solve cross-chain MEV centralization
Interchain Scheduler
Although is no longer in maintenance, the general idea around the Scheduler is interesting: In an effort to address the MEV cartelization problem, some of the main actors in the ecosystem have developed a cross-domain blockspace market for Cosmos chains called the "Interchain Scheduler," as described in the ATOM 2.0 paper. In this mechanism, the auction facilitates cross-chain synchronization by allowing block space to be reserved simultaneously across several blockchains. The builder reserving this block space can then create temporary synchronous and atomic cross-chain transactions. On each participating blockchain, the block allocation design must allow the winner of the auction to control the ordering and inclusion of transactions in the block space it has won.
Validators can anticipate slot overlap between chains and purchase a slot far in the future to later sell at a just-in-time block auction. This allows them to choose the most profitable block at the time of commitment and benefit from time arbitrage. They buy a slot cheaply, expecting demand to increase later on.
Voting power hard-cap
dPoS systems often suffer from the "rich get richer problem", where rewards are heavily skewed towards the top validators. This means that validators with more stake receive significantly larger rewards compared to others. The top validators may receive rewards much higher than what is needed to cover their costs of operating multiple consumer-chains. This imbalance may lead to centralization tendencies within the system, favoring larger validators and potentially discouraging smaller operators from participating.
One of the potential solutions to address this is to implement a voting power cap. This cap may encourage entities to run multiple validators, but since the validator set in Cosmos ecosystem is capped at most 175/200, this can be avoided as the entities running them are all well-known. By setting a cap, all validators can expect to receive a similar share of rewards. However, this solution may come with an obvious trade-off as validators will no longer be as profitable as before, reducing their capitalization and rewards from the services they provide.
The voting power cap can even be dynamic, adjusting based on the overall network conditions. Factors such as the total stake in the system, the number of active validators, or the average rewards earned by validators can determine the cap. This approach ensures that the voting power cap remains responsive to the network's dynamics and can adapt to changes in validator participation and behavior.
Internalizing the profit: ABCI++ & Protorev
ABCI++
One advantage of Cosmos chains over other ecosystems is that, with fewer validators, sovereign chains can implement custom logic built in-house. This eliminates the burden of managing hundreds of validators and the associated overhead of block construction.
This is where ABCI++ comes into play. ABCI++ is an interface that enhances consensus among validators over block construction constraints. It allows validators to effectively implement specific rules on how their blocks should be built. This way, they can distribute the MEV generated among stakeholders instead of keeping it for themselves or leaving it at the mercy of searchers. This also helps prevent harmful MEV flows such as sandwiching and front-running.
Taking the background explanation from x/builder blog: the full node of a Cosmos chain consists of two components:
-
CometBFT (previously Tendermint): Provides a mempool, which gathers unconfirmed transactions, and the consensus algorithm, which collects transactions into blocks, shares blocks with other nodes, verifies them, and adds them to the chain. Today chain developers mostly treat this as a black box providing application input. CometBFT acts as the client.
-
Cosmos SDK application: Provides a key-value store representing the state of the chain and transaction handlers representing state transition functions, which take transactions as input. This is primarily where Cosmos devs do their magic, defining the unique functionality of their chain. The Cosmos SDK application acts as the server.
ABCI++ transfers control from the consensus layer (CometBFT) to the application layer. This allows the application to define the logic they want to apply over block building.
Formerly, transactions would be collected in Tendermint's mempool, passed to the current proposer where a proposal (pre-commit block) was created. This proposal would go to consensus and the proposer would finalize the block itself. This way, they effectively have full control over transaction ordering/inclusion, and other validators don’t have a way to express their preferences over block building.
This flow changes with ABCI++: with this interface, the consensus layer (CometBFT) create a proposal and then it calls a PreparedProposal function to pass it to application layer. The application reviews this proposal and verifies if the block is built correctly according to custom-logic enforced in a prior block. If the block doesn't adhere to previously established rules, the application can modify the RawProposal, by changing transactions included or their order. If valid, it then returns the modified proposal to the consensus layer.
Then, the entire validator set executes a validity check over the block built by calling the ProcessProposal function. This means that all validators can participate in decision-making about the transaction order within a single block. They can also determine if the block was built according to the previously agreed logic and whether it includes any undesirable transactions. If proposal is valid, it then broadcast their vote to their peers. This step essentially allows the application to reject invalid blocks. Is important to note that at this point the application cannot modify the proposal but can reject it if invalid.
The ExtendVote function allows proposers and non-proposer validators to add extra data to their votes (votes are cryptographic signatures over other validator’s blocks). This data will be broadcast together as a package, each with its own separate signature. It will be received together with the vote it is extending and made available to the application in the next block.
TL;DR: ABCI++ empowers Cosmos chains to enhance their consensus mechanisms. Previously, validators had full rights over block building, without input from others. Now, non-proposing validators can provide additional validity checks, leveraging collaboration over transaction inclusion and block construction. With this, Appchains can:
-
Enforce transaction ordering and inclusion preferences
-
Reduce MEV with threshold encryption
-
Empower the community with control over MEV distribution
-
Allow and/or reject certain types of MEV extraction
This is an ABCI++ overview. For a deep understanding of it, check out x/builder blog!
Protorev
Protorev is a module developed by Skip Protocol, a team analogous to Flashbots on Ethereum. Protorev turns Osmosis' validators into searchers that identify trade opportunities to execute backrunning transactions in the Osmosis mempool. This change turns validators into a distributed builder operating under the same underlying consensus. Note that toxic MEV, such as front-running and sandwiching, is not allowed.
Basically, this module enshrines Osmosis’ execution preference within the protocol itself and enforces transactions ordering beforehand, allowing harmless MEV extraction in benefit of the overall community and effectively re-capturating the MEV profit that previously was left to third parties such as builders or searchers. The protocol as a whole reaches a consensus on how the blocks should be constructed and how and what type of the MEV should be captured.
This process executes in the background by flash minting any IBC/denom token. This is permitted by using the x/bank module from the Cosmos SDK to mint and burn capital. For a better understanding of how the bank module works, we can compare it to the ERC20 standard. Unlike Ethereum, where each asset has a separate registry (i.e. single-separated smart contract deployments for each asset), all assets on Cosmos chains are managed by a single, centralized ledger; all that Protorev has to do is index the bank module with just one simple RPC call and flash-mint the necessary tokens itself.
At the end of the transaction, the module atomically burns the token and sends the profit to an Osmosis account controlled by the DAO. Protorev is the first implementation of Protocol Owned Builder, a fully permissionless plug-and-play mev-infra that any Cosmos chain can implement to perform top-of-block auctions leveraging ABCI++.
Unlike the hooks in Uniswap v4, this module can enforce in-protocol validator logic, instead of relying on the application level as in the case of Uniswap on Ethereum. This comes with several advantages, including collaborative block-building rather than depending on the underlying transaction inclusion/ordering on the validator set on different networks where Uniswap is deployed. It also allows community decision-making about revenue usage instead of relying on the hook pool deployer, where UNI holders have no voice or vote.
It also contrasts with Uni v4 because in Cosmos, the module can flash-mint the required tokens to execute the backrun. So essentially, it has access to unlimited-volume liquidity that would otherwise be inaccessible.
Protorev allows the network to internalize this profit and let the governance community decide what to do with it, included but not limited to:
-
Cover protocol expenses, self-funded
-
OSMO burn mechanism
-
Increase stakers APR
-
Reduce protocol fees
-
Grant rebates based on community’s rationale - recently, an interesting topic around MEV rebates had emerge into Osmosis forum, discussing giving a refund to a user that lost ~$250k due to price impact in an axlUSDC/USDC iliquid pool.
Since the implementation of Protorev, more than $2.8M in MEV revenue has been captured from the Osmosis community; the module is extracting an average of ~$60k per day, resulting in a revenue of almost $22M per year.
This post initially aimed to study the theoretical aspects of cross-domain, fully on-chain MEV within the Cosmos ecosystem. However, it has become apparent that CEX/DEX arbitrage (EV ordering / EV signal) is significantly more profitable. There are comprehensive reports that cover this topic in depth, including:
-
A new game in town by Frontier