Attestations are the blood cells of Ethereum. Validators continuously pump them through the system in dazzling numbers. Without them, Ethereum would become pale, faint, and unable to move its muscles. In this post, we analyze some blood samples in order to learn more about how our patient works on the inside and how healthy they are, in particular after their recent heart transplant surgery known as The Merge. The diagnosis: Ethereum is fit as a fiddle.

Theory

First, some background information. Time in Ethereum is split into regular intervals called slots of 12 seconds each. 32 of these slots (that is 6.4 minutes) make up one epoch. In each slot, a single validator is selected to propose a block. In addition, each epoch all validators are asked to send one attestation. To make sure this proceeds in an orderly fashion, each validator is assigned one slot to do so. If everything goes smoothly, these attestations will be included in the next slot’s block.

In proof of work, blocks are the only consensus message: Blocks build on top of other blocks and in this way represent a vote by the block proposer on what should be considered the canonical chain. In proof of stake, however, attestations are the votes. This allows a much larger number of consensus participants (and with them a much larger stake) to express their view and, thus, makes the protocol much more secure.

Data

Let’s now explore the attestations we can find in the chain. We’ll look at different metrics to see if they match our understanding and check if the protocol performs as we expect it to. The time frame we analyze is 24 hours before and after the merge — just because in addition to the normal behavior we might detect some interesting changes at the time of the upgrade.

You can find the source code of the analysis as well as the data set on GitHub.

Inclusion Rates

Let’s start with the easiest question to ask: How many attestations per second are there? Here’s the answer:

On average, there’s about 13200 attestations per slot, or 423000 per epoch, or 1100 per second. Recall that each validator is supposed to send one attestation per epoch. At a validator count of 427192, this corresponds to a participation rate of 99.0%. At the time of the merge, it decreased only slightly by 1.1%, likely due to validators not preparing their system for the fork. Still, all in all, an excellent job!

As we can see in the diagram, there are some slots without any attestations immediately followed by big spikes of about twice as many. What is going on there? In all likelihood, the block proposers assigned to the empty slots are to blame: They either refused to include any attestations in their block or didn’t propose a valid block at all. In these cases, the next proposer picks up the missed attestations. Of the 14400 slots analyzed, 174 (1.2%) were missed entirely and 15 (0.1%) contained a block with no attestations. At the time of the merge, the rate of missed slots increased by 68%, again likely due to validators failing to prepare their system to follow the fork. But the absolute numbers are still very small and don’t constitute a threat to the network.

Besides the holes and spikes, the attestation inclusion rate is fairly constant, but not perfectly so. In particular after the merge, it becomes somewhat jittery. To be able to see this more clearly, here’s a diagram of the distribution:

The standard deviation, a measure for the width of the distribution, almost triples at the merge from 60.7 to 173 attestations per slot. Why is that? One explanation could be the increased network traffic: Not only do validators now have to handle consensus messages, they also need to sync execution level data, most importantly transactions. Validators thus will have less bandwidth available for attestations, meaning they can download fewer of them and have to publish their blocks emptier. In turn, the next proposer can draw on a larger set of unused attestation and is therefore able to build fuller blocks, explaining the right side of the distribution. However, there is a flaw in this argument: Most validators didn’t just turn on their execution node at the time of the merge, but some time in advance. While execution and consensus node didn’t have to interact pre-merge, they were still competing on bandwidth.

A more likely explanation is that consensus nodes became busier. Not only do validators have to download execution data, they also have to verify it, i.e., execute all transactions in each block. This costs time and may lead to some validators sending their attestations later than they did before. As a result, they also arrive later at the block proposer — and sometimes too late. This theory should be easy be confirm by client developers with performance metrics on live nodes, but in this post we are unfortunately limited to on-chain data.

Inclusion Latencies

In trying to make sense of the data above, we alluded to the fact that attestations might be delayed. It would be great if we could measure these delays directly. Unfortunately, on-chain we don’t have access to network level data, but there is a rough metric: Each attestation records in which slot it was created. We can compare this creation slot number with the slot number in which the attestation was eventually included. This is a very imprecise way of measuring latency, but we’re making up for that by sampling a huge number of attestations, so we can still learn something.

Good news first: Almost all attestations get included immediately in the next slot, as they should be: 98.9% before the merge, 97.8% after. The average number of slots it takes to include an attestation is 1.03 before and 1.05 after. Apparently, the network is functioning quite efficiently. Let’s check the few attestations that don’t get included immediately:

Nothing too surprising, but another sign of slightly degraded performance after the merge: The number of attestations with a delay of 5 or more slots increased from almost zero to a few.

Disagreement

Attestations are messages by which validators express their local perception of the chain in order to achieve consensus. In concrete terms, they contain two votes: A fork choice vote and a finalization vote. Without going into too much detail, with the fork choice vote the validators set down which block to consider as the new head of the chain. The second vote is used to finalize previous epochs, i.e., make them forever impossible to revert.

Since attestations are votes, it might be insightful not to just look at the winners, but also at the margin with which they won. Is it a landslide or very close? Let’s define disagreement as the percentage of attestations for a particular slot that voted against the majority. Both as time-series and histogram, here’s what it looks like:

The disagreement before the merge is almost negligible at 0.02%. After the merge, it increases tenfold to 0.2%. Here, we used the median instead of the mean as the average because the distribution especially after the merge is highly skewed with a long tail.

Noteworthily, the disagreement is purely in the fork choice vote. The whole data set contains not a single disagreement in the finalization vote. While the extremity of the effect is surprising, its existence is not: The latest block is very volatile and validators often change it with a single newly received message. Past epochs are much more stable. Therefore, the network participants generally have the same view on past epochs, but less often on the latest block.

Fun fact: While disagreeing attestations vote against the majority, they still almost always vote for blocks in the canonical chain (typically they just lag behind a little). Only 0.02% of all attestations vote for reorgs, a testament to the robustness of the Ethereum’s consensus protocol.

What could be reasons for the increase in disagreement after the merge? As always, worse network latencies may be to blame: During the time a block propagates through the network, the participants have a different view on the head block and subsequently will disagree in the votes they send. An increase in block propagation time consequently means an increase in disagreement.

Aggregation

Earlier we saw that Ethereum handles a staggering number of attestations, more than 1000 per second. To be able to continuously sustain such a high load, it uses a neat trick: Aggregation.

In the previous section we noticed that most of the time validators have the same view of the chain and as a result their attestations are the same. In these cases, we can condense the attestations into a single message. To facilitate this (among other reasons), Ethereum randomly groups validators into committees of about 200 members. Attestations among one committee can be aggregated, but not across them.

The only obstacle to aggregation are the attesters’ signatures. Being from different private keys, they are of course always different. For this reason, Ethereum uses a fancy signing scheme called BLS. It allows us to aggregate many different signatures into one, as long as they are for the same message. All we have to do is remember who the signers are, which Ethereum does by means of a bitfield: Each bit in the string corresponds to one validator. If the bit is 1, the validator’s signature is included, if it’s 0, it isn’t.

Attestations can be aggregated repeatedly: Let’s say there’s attestations by four validators A, B, C, D. One way to aggregate them would be all at once: A + B + C + D -> ABCD. But we can also do it in two sequential steps: First A + B -> AB, C + D -> CD and then AB + CD -> ABCD, giving us the same result. However, aggregation would not be possible if there’s an overlap: AB + BCD can not be aggregated because B is included in both parts.

Aggregation is performed by randomly selected nodes in the network, while the attestations are still propagating and before they are included in a block. There’s a spectrum of aggregation strategies nodes can follow: On one extreme, they can wait for as many attestations as possible before aggregating (A + B + C + D → ABCD). On the other, they can aggregate as early as possible and in smaller chunks (A + B → AB, C + D → CD, AB + CD → ABCD). The latter has the advantage of cutting down network traffic early. But it runs the risk of creating overlaps, leading to less efficient aggregations and more messages that eventually have to be stored in the chain.

Let’s check how well Ethereum is aggregating and how much potential for improvement there is. To start, let’s look at the most basic measure of aggregation efficiency: The number of aggregated attestations we manage to end up with. Ideally, we’d always get down to a single one, but due to overlaps between partially aggregated attestations as well as simply bad timing, we won’t always get there.

As we can see, before the merge, aggregation worked reasonably well — on average attestations were aggregated into 1.2 messages. In 88% of the cases, aggregation was perfect but there’s a long tail of less impressive work. After the merge, the average performance almost didn’t change. But perfect results were achieved in only 80% of the cases. To compensate, the tail almost disappeared. All in all we can say that aggregation works ok. We could improve it by up to about 25% to increase the chain’s efficiency, potentially through some engineering efforts or network protocol changes.

We can still dig a little deeper. To do so, we look at a slightly more complicated metric we call aggregation effectiveness that we define it as follows: Consider a set of partially aggregated attestations that (if not overlapping) could be aggregated even further. The effectiveness of aggregation is the weight of the heaviest attestation (i.e., the number of signers of the attestation with the most signers) divided by the total number of signers of any of the attestations.

As an example, say there’s a committee of seven signers, five of whom attest to the same thing. The attestations have been aggregated as follows: AB, BCE, AD. The effectiveness of aggregation is 3/5 or 60% as the heaviest attestation (BCE) contains three signers of five in total.

Here’s a histogram of aggregation effectiveness before and after the merge:

As we can see, aggregation effectiveness is pretty good. The heaviest attestation almost always contains more than 99% of the signatures. There’s a slight drop in performance at the merge, but nothing alarming.

Lastly, it might be interesting to look at the overlap of attestations. Say there’s an aggregation process that didn’t work perfectly but instead resulted in more than one attestation. Take the two heaviest attestations of that process and count the number of signers who are included in both. If this number is high, it means we may be aggregating too aggressively. We should hold off for longer and collect more individual attestations in order to make smarter aggregating decisions. A low overlap is a sign we’re doing a good job.

As always, there’s too many attestations to look at individually, so here’s a histogram of all of them:

Attestations either overlap almost fully (> 90%) or very little (< 20%), there’s not much of a middle-ground. At the merge, the number of highly overlapping attestations decreased in favor of ones with little overlap. On the surface that’s a positive development. However, the relevance of this should not be overrated as it might be caused by the increase in inclusion delay we saw earlier: Attestations that arrive less frequently are easier to aggregate efficiently.

Conclusion

In this post, we listened to the attestations in Ethereum through the stethoscope of several different metrics: Throughputs, latencies, disagreements, aggregation efficiencies. We showed that for the most part Ethereum is in very good shape, but also detected some light bruises caused by the merge. Still, it appears very possible to make our chain work a bit harder, for instance by decreasing slot and epoch intervals or by reducing validator staking requirements.