This is Part I in a series of three articles introducing cadCAD GPT, an open-source Large Language Model (LLM) framework to support token system simulations based on radCAD or cadCAD Python models.

Read more:

• Part II: This is me, cadCAD GPT! A deep-dive into cadCAD GPT’s powerful, customizable components

• Part III: Let’s chat! Experiments and further development of cadCAD GPT

Introduction

Large Language Models (LLMs), like OpenAI GPT-3 and its successors, have demonstrated remarkable potential in supporting humans in solving complex tasks. Token engineers invest substantial time and effort in constructing models and running system simulations to find an optimal token design. Imagine, token engineers could access token system simulations just by prompting a Python model with a natural language request.

And imagine that AI agents could not only answer questions but change parameter settings, analyze, and visualize results.

In fact, AI agents can do this - and much more!

cadCAD GPT is an open-source LLM agents framework to support token systems simulations. cadCAD GPT can be used on top of any Python model following the cadCAD/radCAD model structure. Via Python functions, it can integrate today’s most powerful data analysis and machine learning libraries (Tools) and can access data in various formats (Memory). Finally, cadCAD GPT allows users to control and track the agents’ workflow for verifiable and reproducible results.

In this article, we derive requirements for LLM agents in a token engineering context and present the generalized framework. Part II provides a detailed overview of the open-source cadCAD GPT components and shows how to apply and further develop cadCAD GPT to make it cater to any complex system modeling task. In Part III, we present a range of experiments demonstrating interactions with cadCAD GPT in two simulation use cases.

The Token Engineering Design Process

Token engineering is the design, verification, and optimization of token-based crypto-economic systems. The token engineering design process involves several steps, from discovering and collecting system requirements to designing incentive mechanisms to implementing the algorithms in smart contracts. In the design phase, token engineers test if the desired properties of the system hold under a certain parameter setting, stress test the system, or run parameter sweeps to find the optimal setting. Here, a Python model of the token system serves as a digital twin to run these experiments.

The modeling and simulations phase requires a lot of experience and includes the following steps:

a) Collect the questions to ask and prioritize;b) Derive the metrics to answer these questions and build a model of the token system; c) Verify the model ("did we build the model right?") and validate the model ("did we build the right model to represent reality to answer our questions?") Only then are we ready to d) start running experiments, collect insights, draw conclusions, and e) take decisions.

With cadCAD GPT, we focus on step d) to help token engineers and their teams iterate on the design, become faster in deriving conclusions, and, ultimately, gain confidence in making decisions.

Use Case Example: Building up a Token Sale Proposal

Designing a token sale is a challenge that every token engineer is tasked with at some point. The main goal of a token sale is to raise the funding needed to develop a protocol or application by selling tokens to investors. Tokens are typically sold in several sales rounds to different stakeholder groups. They are usually not immediately distributed when sold, though. Instead, a vesting schedule defines unlock dates and rates for investor groups.

In a token sale round, token issuers and investors aim to reach equilibrium via several iterations of one-to-one negotiations. A token price is typically a function of the amount a startup aims to raise and the token supply, or the token amount sold, respectively. As such, it is a simple calculation. However, any investor group seeks to achieve a fair deal compared to any other investor group. Thus, the number of tokens and conditions sold in prior and potential future rounds impact any token sales offer.

The fact that a token sales offer is influenced a) by the parameters supply and price in a particular round and b) by the same parameters across all other rounds, and c) by comparing other token sale events in the same sector, makes the optimization of a token sale offer a multidimensional optimization problem. We see such multidimensional optimization problems most frequently in token engineering. Designing a token sales offer requires a comparably simple model since most values (such as the token price or supply) are deterministic.

Token engineers build more complex models simulating the interplay between various token mechanisms and uncontrollable external factors such as token market price or human behavior. We hypothesize that LLM agents can provide massive value in solving this class of problems and support simulating complex, emerging, dynamical systems. In Part III, we’ll demonstrate cadCAD GPT’s results in a token sales use case and showcase its application to a typical dynamical systems model. For now, let’s look first at the requirements for LLMs in a token engineering context.

Requirements

LLM agents have not been built to support systems simulations initially. They are incapable of performing reliable calculations and lack intrinsic mathematical skills. Moreover, there are plenty of experiments showing their tendency to generate false information or hallucinations.

To provide value to token engineers, we have to satisfy the following requirements:

• cadCAD GPT agents take user input in natural language, change parameters accordingly, run simulations, analyze results, and give back answers in natural language.

• cadCAD GPT agents access and integrate data and information in the simulation, such as transaction data or benchmarking information, and a user should be able to control what information is included or excluded in running a simulation.

• cadCAD GPT agents access, modify, and run a Python token system model. They provide further analysis of simulation outputs and apply typical statistical methods, testing and optimization approaches (such as parameter sweeps and machine learning) to find the best parameter setting.

• cadCAD GPT agents run these methods; they should be able to determine when to use a certain method best. In case several are needed, LLM agents must determine the ordering of methods – all based on a user’s question in natural language.

• Finally, cadCAD GPT outputs have to satisfy criteria of credibility, whether the model’s results are ‘correct’ for each question addressed. Any cadCAD GPT output has to be correct, verifiable, and reproducible.

Introducing the cadCAD GPT framework

The following section introduces the conceptual design of cadCAD GPT.

cadCAD GPT can run simulations on any Python system model adhering to the cadCAD/radCAD system model structure. Through access to the model’s objects experiment, model, and simulation, cadCAD GPT can interact with the model to run simulations.

Users can communicate with agents in natural language (User Input) and receive the results of a simulation experiment in natural language (User Output) provided through the cadCAD GPT chatbot.

cadCAD GPT agents are equipped with Toolkits. Tools are typically Python functions to run experiments and execute parameter changes, A/B tests, or Monte Carlo simulations. Via libraries like Pandas, token engineers can gear agents with the ability to read simulations output data, clean, reformat and analyze output data, and visualize results. In Part III, we show examples and how they are made available to the agents. The modular setup of toolkits allows token engineers to customize and expand the toolkit according to the needs of their particular project. Additionally, we enable maximum control and verifiability of the simulation's outcome.

Memory is any information relevant to a simulation task. Examples include

• The documentation of the Python system model, the model parameters, or metrics to observe

• Protocol information, such as developer docs or whitepapers

• Market data about fundraising or token unlock events

• Blockchain transaction data

• Research papers, and much more.

cadCAD GPT can process a wide variety of data formats, from HTML, XML, and Markdown text to tabular data via API calls. In Part II, we provide a detailed introduction to integrating information in cadCAD GPT Memory.

The core innovation of cadCAD GPT is its orchestration components.To provide value in the simulation process, agents must make sense of user input and decide what tools and/or memory to use and in what sequence.This is where they take over the role of a token engineering copilot. Here, we apply the concepts from projects like BabyAGI and AI-powered task management. Its core idea is to separate the planning and execution steps to allow agents to prioritize steps and improve task completion autonomously.

cadCAD GPT provides a Planner Agent that creates a plan and task list based on the user’s natural language prompt. This task list points to the toolkits, tools, and information needed in the simulation.

This task list is then handed over to the Executor Agent, who goes over the tasklist, reasons about the steps needed, selects and executes actions, and observes the results toward the final output.

The actions of both the Planner Agent and Executor Agent are collected and made printable to allow users to monitor and verify cadCAD GPT’s reasoning process and results delivered.

Finally, the cadCAD GPT chatbot is where all components are weaved together. It provides the UI for natural language inputs and outputs. It imports the radCAD/cadCAD system model objects. Most importantly, it constructs the Planner and Executor Agents and triggers the interplay between agents and the tools and memory available.

Summary

cadCAD GPT equips token engineers with powerful tools to execute simulation tasks via natural language interfaces. With this article, we embark on a journey to establish the essential requirements for LLM agents within complex systems engineering. By deriving these requirements, we pave the way for a deeper understanding of cadCAD GPT’s technical implementation and practical applications. Read on in Part II, where we show how cadCAD GPT components are constructed in detail. We demonstrate how to connect cadCAD GPT with any radCAD or cadCAD model and how additional tools and memory can be added and further customized. Part III of this series showcases cadCAD GPT’s results in typical simulation use cases.

cadCAD GPT Demo

cadCAD GPT will be available on Thursday, Nov 30, 3:00pm UTCSign up for the demo and be the first to get access!

Acknowledgements

cadCAD GPT was kickstarted by funding received from Token Engineering Commons. We thank the TE Commons community, and Gideon Rosenblatt in particular, who encouraged us to embark on this exciting journey. Big thank you to our advisors Roderick McKinley, Richard Blythman, and Robert Koschig for ongoing support and feedback. Shoutout to Dr. Achim Struve, Dimitrios Chatzianagnostou, Stephanie Tramicheck, Ivan Bermejo, Rohan Sundar, and Lukasz Szymanski for the most valuable alpha user feedback and insights, and Kaidlyne Neukam for her tireless support in publishing this work.

Links

The token sales spreadsheet model that informed our token sales experiments is available online, along with a comprehensive online course by Roderick McKinley.

TE Academy is the home for the token engineering community. Learn how to design token systems with rigor and responsibility! Sign up for our newsletter to receive the latest token engineering trends, tools, job offers and ecosystem news.