The Making of The Smurfs’ Society - Episode III: Reflecting on the foundational collaboration between Alchemy and The Smurfs’ Society

The Smurfs’ Society is the official Web3 project of the Smurfs. The project is an invite-only, fully on-chain gamification platform that was launched in December 2022 and has generated 83,000+ wallet connections and 3 million on-chain transfers in just two months, making it one of the TOP100 most used Dapps according to DappRadar. In this article, our CTO and co-founder Sebastien Bastard tackles the technical challenges of scaling high volumes of on-chain transactions, and how our team is collaborating with Alchemy to deliver a smooth and consistent experience to tens of thousands of players.

The relativity of ‘scalability’ in the context of Blockchain

Most of us wouldn’t consider 100 transactions per minute an “at scale” or volume problem, myself included. In the past, I've collaborated with many companies to optimize Cassandra deployments running clusters in a multi data-center world with terabytes of data, low latency & high volume. Back then, I would never have thought that this low transaction volume could be hard to process when working with blockchain infrastructure. But the challenge is twofold: 

1. We need to be able to physically execute all the transactions 

2. We need to ensure the correctness of the transaction and more precisely, its eventual data consistency and accuracy. 

To address the first challenge, this is pretty much the same as sizing any platform cluster deployed in k8s with auto-scaling capabilities. One small differentiation would be on the way we sharded the load using an HD wallet in a StateFulSet to provide each pod with its own wallet to manage directly. Keep in mind that blockchain mandates transactions be executed in order (for a specific order wallet). 

For the second challenge, think about a transaction in the mempool but not getting mined (gas). One could (a) just wait for the network to cooldown (if possible) for the transaction to just be mined, or (b) overwrite the transaction with a new one (forcing nonce) and increased gas settings, or (c) simply use a different wallet. In our case, scenarios (a) and (c) didn’t make sense. So scenario (b) was our only option, which created its own complexities.

This article provides a quick look at our experience, partnering with Alchemy and moving from Reinforced Transactions towards Gas Optimized Transactions to overcome our main challenges.

Mining 20,000 transactions per day while maximizing correctness

Why is it so hard to scale an application or game that makes heavy use of the blockchain as a backbone? (According to polygon stats, 20,000 transactions represent 2.4% of the total Polygon traffic).

The main issues come down to limitations:

• Are we limited by the number of transactions that can be mined in a single block?

• Are we limited by the capacity of the network as a whole? 

• Are we limited by network congestion and specific intraday usage patterns?

• Are we limited by the way the transactions on the blockchain get executed? 

• Are we limited by gas price volatility? 

After several weeks of operations and a lot of iterating all our controls, we now have a platform for the game to operate in optimal conditions. But before I explain all the obstacles we had to overcome to get here, let me introduce you to the game dynamics and give an overview of the actions we automate from a blockchain standpoint and its challenge. Then I will explain how we solved the problems by partnering with Alchemy.

The game dynamics of The Smurfs’ on-chain game

The game as it stands right now is a disruption of the normal concept of a“waitlist,” where the players mix ingredients to try to find recipes that create potions, some of which can give access to a crystal (guaranteed mint access with a discount). All the assets we use on the game are represented as NFTs on the Polygon network. These are either ERC-1155 tokens for items we would consider as fungible and consumable (items used for mixing like ingredients, potions, blue clay, and statuettes) or ERC-721 tokens for items that aren’t fungible (the tickets to the mint, i.e. crystals).

Below are the interactions our platform has with the Polygon network for the game to run:

• drop box(s) composed of ingredients: players are invited to claim their box of 3 ingredients on a daily basis

• mint statuette: main component of the game. All players are required to possess one to be able to complete all actions on the platform. The statuette is dropped for free when the player logs into the game for the first time. After the initial game, the statue’s additional role will be announced.

• burn clay: when the player attempts a recipe that fails (i.e. no potions attached to that recipe), the blue clay is burned (x1)**
  **

All of these operations are performed by our platform, meaning we cover the gas fees of each transaction.

Only one operation is actually performed by the player on the Polygon network 

• mix recipe: the player successfully mixes 3 ingredients to create a recipe for a potion. In this case, the player is the one paying the gas fees.

For the first two functions that the platform performs, we can work in a 100% asynchronous fashion without real impact to the game if a transaction isn’t mined for a long time. Obviously, our preference from an experience point of view would be to have these times as low as possible. 

On the other hand, burning clay is a bit more challenging, and implementation choices were mainly driven by the game itself. When a player actually proposes a recipe, they do so by signing an EIP-712 struct containing all the ingredients they want to use to mix it with a 3-hour deadline, as shown below.

This signature will be used as authorization to burn if no recipe is found with the associated mix. We execute this burnClay transaction on behalf of the player to absorb the transactional fees associated. At this point, we have to manage several states:

• transaction is transit: stored off-chain -> moving to on-chain. Used for reconciliations.

• transaction is in the mempool and waits to be mined: transition state requires us to reconcile what we believe the user has in flight versus what’s being processed - all with no guarantee on how long it would take

• transaction is mined: the blue clay quantity displayed on screen is now accurate.  This state is the final state and does not represent any challenge - blockchain is source of truth

• transaction is mined but considered as aborted.

In an ideal world, we expect all states to converge in 1 minute or less to guarantee the correctness of operations and game experience. For this use case, we decided to use the Polygon network for the gamification phase. Below are some of the most important criteria that helped us choose Polygon as our network:

• fast finalization times for reactiveness

• reduced transaction fees for operational costs management

• Ethereum interoperability for long-term vision**
  **

Our challenge: mine 20,000 transactions per day maximizing consistency

Reducing Polygon gas volatility impact while delivering a high success rate

Like any other network, Polygon can suffer from congestion and can directly affect us by adding delays on the different actions we described earlier. There are basic actions we can take to address these:

• sharding: have multiple wallets interacting with the blockchain. As long as there is an operational wallet (similar to a connection pool for DB), transactions can be done. 

• over pricing: to guarantee our transactions are processed in a timely manner, we can boost the incentive on priority fees (EIP-1559).

Our initial setup was done using the Reinforced transactions solutions and over-pricing the market on gas fees to optimize the transaction finalization times.

The two screenshots were taken within a few minutes' interval (~ 5’).  You can see the big variation in gas in a very short time frame! This had a direct impact on us: any rapid surge in the market volatility for gas price, and our transaction would be stuck, blocking one of our wallets!

This is exactly what happened to us early on. So, we came up with several ways to optimize it:

• Adding resiliency to our application and tracking the state of transactions better for longer periods of time

• Tracking and alerting when symptoms were exhibited - we did this by tracking the time it took a message to get processed

When everything looks good in micro, we see this:

And this is from a macro level when things get more interesting:

Occasionally, (and the example above is from one of our hardest days,) some messages were nearly 200” old. When this gap of time grows too much, players can start to notice it: the biggest flag being that the blue clay counter doesn’t reduce when a recipe is mixed. On our worst day, post-Polygon upgrade, we had to stop the game for 30 minutes because some transactions could not be mined. 

At this time, we decided to more actively collaborate with Alchemy to attack this challenge with one of their new offerings. So we entered the beta for the Gas *Optimized transactions. * 

The promise of this new solution was to manage these unpredictable outliers. Imagine if instead of signing a single transaction with some hard coded kind of logic to determine the gas, you could sign 5 or 10 with different gas price conditions and leave it up to Alchemy to do the magic and escalate nonce pricing automatically! 

In our case, we decided to opt for the following escalation policy (1.25, 1.5, 1.75, 2, 2.25, 2.5). We instantly saw a reduction in the number of outliers where transactions were stuck for a long period of time. This created a much smoother experience for our users, which in a gaming environment is hugely important. 

Overall, partnering with Alchemy and testing the Gas-Optimized Transaction Api was a win-win-win. Our user experience was improved, our transaction costs were optimized, and Alchemy got to test its new service with a high-volume platform. 

Alchemy themselves recognized the benefits of the partnership and the success of the game as a milestone: 'It's been great partnering with The Smurfs' Society on optimizing and scaling their on-chain gamification platform. Its unique characteristics and volume made it the perfect use case for our product." JT Hayden, lead Engineer @ Alchemy.

That’s all folks

By the end of the game, we had successfully run several optimizations and continued to see improvement in the system. We’re excited to see where the next phase takes us and look forward to continuing our collaboration with the Alchemy team, whom I want to personally thank for the reactivity the day shit hit the fan and for the quick turnaround response time. It’s all about choosing the right partners; Alchemy lived up to our expectations and helped us deliver an incredible experience to our users.**
**

Smurfily,
Sebastien Bastard
CTO & Co-Founder of The Smurfs’ Society.