The review process of RetroPGF3 was a titanic task, not only because of the sheer number of projects but by the format and tooling provided at the onset of this process.

My process entailed a couple of steps, collaborating with different people in the ecosystem, from fellow Badgeholders (Badgies) to contributors of the Optimism Collective, and using diverse information sources through a custom-made GPT agent I trained: GPT-Sunny.

TLDR;

LauNaMu’s steps to evaluation design:

1. Recategorize and subcategorize all projects

2. Select evaluation criteria per category

3. Bounce off and determine level definitions with GPT-Sunny

4. Determine weights for each criterion with support from GPT-Sunny, fine-tune based on personal criteria

5. Bucket creation based on points received from criteria

6. Reward specific behavior - yes/no fields

7. Setup rails: 2 fundamental constraints

LauNaMu’s steps to evaluation:

1. Decide if the project has generated impact for the Collective: Yes/No

2. Decide if I will vote on this project: Yes/No

3. Evaluate based on Category-criteria: Low, Mid, High

4. Evaluate for specific attributes: Yes/No

5. Add comments per project

Getting started

To make reviewing the volume of projects easier and reduce the cognitive load of shifting from one type of contribution to another, I used the re-categorization of the projects generated for the List-making workshops available here. Big shoutout to @Billy191 from RetroPGFHub and to the Retrolist.app team for including this categorization into their websites. These tools made the exploration of projects much easier.

I used primarily the Retrolist.app site for project discovery and evaluation. Having non-overlapping sub-categories enabled me to review smaller chunks of similar contributions. There was room for improvement in the sub-categories but this split made it incredibly easier for me to get closer to comparing apples to apples.

The Categories

I created one evaluation framework per category based on the Impact Evaluation Framework I generated and that was available for all Badgeholders to use in the Badgeholder Hub. I used the “relevant terms'' as evaluation criteria. Each of these criteria had three possible levels: Low, Mid, and High.

For example, for Collective Governance I had:

1. Impact: Low, Mid, High

2. Accessibility: Low, Mid, High

3. Participation: Low, Mid, High

4. Understandability: Low, Mid, High

5. Accountability: Low, Mid, High

6. Capture Resistance: Low Mid, High

GPT-Sunny

To enhance my workflow and effectively integrate the Impact Evaluation Framework and the Metrics Garden, I trained a specialized GPT for Impact Evaluation in the Optimism Ecosystem: GPT-Sunny. We worked together to:

1. Create the segmentation of what would be Low, Mid, or High within each of the evaluation criteria per subcategory.

   Example of segmentation in OP Stack:

   0 for no identifiable impact; Low for minor enhancements or indirect benefits; Mid for notable improvements or integrations; High for significant, direct contributions to the OP Stack.

2. Create the weight each criterion would have in my evaluation based on my chosen relative importance and information from the Impact Evaluation Framework around Ecosystem context and the impact of each item relative to the definition of Impact in the category.

   Example of weighting in Collective Governance:

   • Impact: 27.7% (impact was included in all categories)

   • Accessibility: 25%

   • Participation: 20.45%

   • Understandability: 13.64%

   • Accountability: 6.82%

   • Capture Resistance: 6.82%

Buckets

After determining the weights, I simplified the process using a point-based system aligned with the criterion. I used the points to place evaluated projects into buckets. Projects falling within the same bucket, defined by a specific range of points, would be allocated an equal amount of tokens.

I calculated the number of points each level within each criterion would be awarded so that if a project were to score High in all criteria it would receive 100 points.

Example for Collective Governance:

• A project with a "Low" Impact rating receives a score of 4.17 points

• A project with a "Medium" Impact rating receives a score of 12.50 points (4.17 for Low + 8.33 for Medium).

• A project with a "High" Impact rating receives a score of 25 points (4.17 for Low + 8.33 for Medium + 12.50 for High)

Incentivize specific behaviours

Across all categories, there were specific behaviors/attributes that I rewarded because I believe they contribute to a more resilient Ecosystem. These were binary criteria (yes/no).

Each criterion was assigned a fixed amount of OP tokens that the project would be allocated on top of their point-based allocation. This means projects that do not fulfill the criteria wouldn’t be affected in their baseline allocation and those that do, would be better off.

1. Does the contribution promote a Superchain future/promotion of multiple OP Chains?

2. Is this an innovative use case built on an OP Chain?

3. Are they OG Optimism builders? (Received RetroPGF previously or have been developing in the Optimism Ecosystem for more than one year?)

4. Did they use Optimism-specific impact metrics in their application?

5. Can I personally attest to their impact on the Optimism Ecosystem?

6. Do they have a geographical regional focus?

7. Are they women-led projects?

8. Is this a contribution by an individual?

Token allocation

I assigned a pre-set amount of tokens per Category based on what I considered contributed to the sustainability of RetroPGF and knowing there was an inclination towards technical contributions from other Badgeholders based on their backgrounds:

OP Stack: 6 million OP

Collective Governance: 4 million OP

Developer Ecosystem: 8 million OP

User Experience + Adoption: 7 million OP

These amounts don’t add up to 30 million as I was allocating additional funds based on the specific behaviors mentioned above.

These amounts were then split between the subcategories, and after the evaluation of projects, token allocation was assigned based on the number of projects per bucket in each subcategory. Projects with a higher score, and therefore in a higher level bucket, were allocated higher OP token amounts.

Wrapping up!

I revisited the final amounts and made tweaks and corrections where I felt were needed and then subtracted previous funding received by the projects where I considered necessary.

For example: Some projects have received funding from the Governance Fund, but the funding is locked for a year, so I only discounted a percentage of the allocation to consider that these funds cannot be accessed by projects. Reported revenue was discounted in full.

Allocation for projects that were included in Lists, by Badgeholders whose expertise I trusted and that included a clear methodology was also reviewed against my allocations.

This process was incredibly interesting and provided an incredible learning ground for hands-on Impact Evaluation. Having specific definitions for impact and knowing what the perceived main barriers in each category as informed by the Impact Evaluation Framework contributed vastly to reducing bias. It opened up opportunities for me to discover numerous new projects and contributions within the Optimism Ecosystem, which I might not have encountered otherwise.

I'm excited to apply the insights gained in the next iteration, employing a revised methodology that more systematically incorporates new data points. This approach will be informed by improved Impact Metrics provided by contributors.

There are more learnings and thoughts on improvements to come from my end.

🔴✨I’m Optimistic that we are headed to a more data-driven and systematic reporting and evaluation of the Impact generated by contributors 🔴✨