This is Part III in a series of three articles introducing cadCAD GPT, an open-source Large Language Model (LLM) framework to support token system simulations based on radCAD or cadCAD Python models.

Read more:

• Part I: Hello, cadCAD GPT! Requirements and conceptual design of LLMs to support token system simulations

• Part II: This is me, cadCAD GPT! A deep-dive into cadCAD GPT’s powerful, customizable components

cadCAD GPT is a new LLM agent framework to support token systems simulations, built by Token Engineering Academy’s Rohan Mehta and Angela Kreitenweis. cadCAD GPT can be used on top of any Python model following the cadCAD/radCAD model structure. It is made for token engineers, and their project stakeholders. By offering a natural language interface, it makes it easy to iterate and establish simulation routines, freeing up bandwidth for reasoning about the system design.

Following the core design requirements introduced in Part I of this series, we demonstrate how cadCAD GPT can support typical token engineering tasks. For demonstration purposes, we go over a token sales use case first and showcase the same types of queries for a Lotka–Volterra predator–prey model (system dynamics model).

Accessing Memory

As outlined in Part II, Memory is the capability of agents to access sources of knowledge relevant to a simulation task. Examples in token engineering include on-chain transaction data, token fundraising data, or the documentation of a cadCAD model. To learn more about the concept of Memory in cadCAD GPT, check out Part II.

Accessing tabular, numerical data

In a token sales use case, the goal is to design an attractive offer to token investors. In this context, it's crucial to consider current market conditions and examine the accomplishments of comparable projects within the same sector and sales rounds. cadCAD GPT can process such benchmarking data and create token sales offers.

In our example, we search for data in an external database. We ask cadCAD GPT to find benchmarking data (pink).

cadCAD GPT provides the following result:

Here’s what happens behind the scenes: The cadCAD GPT chatbot receives the user query and triggers the Planner Agent, who submits the plan, including the tool to use (long_term_memory). Then, the Executor Agent reasons about this plan, runs long_term_memory, and includes the SQL query to filter the database according to the user query.

Accessing Memory with cadCAD GPT is easy. Search for data, run API calls, and access SQL tables or CSV files hosted anywhere. All you have to do is ask cadCAD GPT in natural language.

Moreover, token engineers can customize cadCAD GPT and add tools according to their needs. It only requires a Python function, the data source, and a well-formulated docstring. cadCAD GPT agents use these docstrings to figure out when to use these tools and how to use them. Learn more about adding tools and memory to cadCAD GPT in Part II of this series.

Accessing semantically searchable data

One of the first steps in a simulation project is familiarizing yourself with the token system model in use, its purpose, the metrics it can observe, or any other information included in the model documentation. In the following example, we explore a predator-prey model. We ask cadCAD GPT for the assumptions in the model.

cadCAD provides the following answer:

Here’s what happens behind the scenes: cadCAD GPT has access to a Markdown file in the model Github. The tool model_documentation makes the document available to agents and creates a semantically searchable vector database to allow LLMs to search for relevant information based on the user question. cadCAD GPT can process any text source, like PDF files or webpages. We equipped cadCAD GPT with a Retrieval Augmented Generation framework for semantic searching and enable token engineers to specify data sources for a maximum level of control over cadCAD GPT’s knowledge. Learn more about the concept of Memory in Part II.

In a final example, we ask for the parameters and metrics in the model. The screenshot below shows the full flow of user input and cadCAD GPT output:

Fetching simulation output

In a simulation project, token engineers typically iterate on a design and create a range of variants to find the best solution. This often requires numerous simulation runs with different parameter settings and simulation outputs.In the context of a token sale, designing a token supply scheme with total supply and supply unlocked over time is a typical token engineering task. cadCAD GPT makes it easy to keep track of current values by simply querying it in human language.

Equivalently, we can ask for the current values in the predator-prey model.

Behind the scenes, cadCAD GPT’s Planner agent picks the model_info tool to access the current parameter settings and simulation output (available via the cadCAD model objects). Today, cadCAD GPT allows access to current values only – which are updated in every simulation run. In future versions, we plan to include tools for version control and comparing simulation results easily. Learn more in Part II.

Analyzing and changing parameters

While the examples above demonstrate cadCAD GPT’s capabilities in simple queries, let’s now look at more complex tasks that demand a sequence of intermediate steps to be executed.

Improving a token sales offer

We start with the case of a token sale again. In Part I, we introduce the iterative process of designing a token sale offer, searching for a solution that is considered fair to any investor group, like participants in a private or public sale, and the startup team. Part of this offer is vesting terms, which are usually time-based to unlock tokens for particular stakeholders at a certain point in time. In negotiations, investors might ask for different distribution terms, and token engineers must analyze the consequences, e.g., the need for a larger total supply. cadCAD GPT can support this process. Let’s see if cadCAD GPT can find the following values for us:

In this case, the Planner Agent creates a task list with multiple ordered steps, the tools to pick from cadCAD GPT’s Toolkit, and the arguments to calculate the results - per step.

The Executor Agent takes these instructions and reruns the simulation. It loops over Thoughts, Actions, and Observations and runs data analysis via the analysis_agent, utilizing Python Pandas methods. The analysis agent is available in cadCAD GPT’s open-source Toolkit.

Providing simulation output as charts

On top of the features presented above, cadCAD GPT can plot charts and visualize simulation outcomes. Let’s assume we need a diagram showing the evolution of the prey population based on a new reproduction rate.

We ask cadCAD GPT:

cadCAD GPT provides the result in a graphical form by picking the right tool - plotter - without further instructions. The Planner Agent’s task list and the Executor agent’s messages are shown below:

Summary

cadCAD GPT is able to support a wide array of use cases in a token engineering context. Its intelligent agents are not limited to answer questions but reason about a system design, seek out the best parameter settings, conduct analyses, and visualize results. Remarkably, without the need for explicit instructions, they handle non-deterministic task sequencing.

Via Toolkits, cadCAD GPT can be equipped with the most cutting-edge data analysis, diagramming, and machine-learning Python libraries available today. Toolkits are modular and can be customized and expanded to serve any simulation project’s needs. Similarly, cadCAD GPT can access information and data in diverse formats stored in Memory, offering full control over which sources are incorporated in the solution of a given task.

Open research questions

This first release of cadCAD GPT marks an important milestone in developing AI-powered token engineering tools. When navigating the early stage of LLMs applied to the needs of our discipline, many intriguing questions emerge.

Open research questions include:

• Incorporating novel workflows: Data science workflows sometimes involve re-prioritizing tasks based on observations from the tool outputs. The Executor agent currently does not change its tracks once the Planner Agent makes the initial plan list. To solve this, we could get the Planner agent to flag certain tasks as required while leaving others to the discretion of the Executor to re-prioritize based on answers received by the tools. We would like to know if this can be done in a generalized way without introducing failure modes.

• Adding short-term memory to the Planner Agent: The Planner agent currently does not remember previous conversations; we need to add a dynamic message history to the Planner Agent for it to remember important details of earlier conversations in the same session. Since we will never have an infinite context window with any LLM, we must delete message history. What framework can we use to decide on the processing of conversation history? Ideas from cognitive architecture research should aid us in this aspect.

• Retrieval Augmented Generation to improve few-shot prompts: As tools increase in number and complexity, it will become increasingly likely that the plans developed by the Planner Agent may miss some steps or choose the wrong tools. Few-shot prompting is the best way to solve it, but there are only so many examples we can give before we run out of context length. We wonder if building dynamic system prompts with a retrieval augmented generation over a dataset of correct examples will be the right approach for the future.

cadCAD GPT Demo

cadCAD GPT will be available on Thursday, Nov 30, 3:00pm UTC
Sign up for the demo and be the first to get access!

Acknowlegements

cadCAD GPT was kickstarted by funding received from Token Engineering Commons. We thank the TE Commons community, and Gideon Rosenblatt in particular, who encouraged us to embark on this exciting journey. Big thank you to our advisors Roderick McKinley, Richard Blythman, and Robert Koschig for ongoing support and feedback. Shoutout to Dr. Achim Struve, Dimitrios Chatzianagnostou, Stephanie Tramicheck, Ivan Bermejo, Rohan Sundar, and Lukasz Szymanski for the most valuable alpha user feedback and insights, and Kaidlyne Neukam for her tireless support in publishing this work.

Links

The token sales spreadsheet model that informed our token sales experiments is available online, along with a comprehensive online course by Roderick McKinley.

TE Academy is the home for the token engineering community. Learn how to design token systems with rigor and responsibility! Sign up for our newsletter to receive the latest token engineering trends, tools, job offers and ecosystem news.