The Together platform, the Gensyn protocol and the future of machine learning
Intro
I recently published a podcast episode called “Decentralized Compute for Machine Learning,” which focused on a decentralized protocol for machine learning called Gensyn.
This episode marked a first look into the developing market for decentralized AI. In this article, I want to build on that work by introducing a platform called Together alongside Gensyn, which is using a different strategy and technological approach to work towards similar goals. This article aims to provide an overview of both initiatives and how they compare in the overall landscape of decentralizing access to the resources needed to build the world’s most capable ML.
—————————————————————————————————————————
Gensyn recently completed a $43M Series A funding round led by a16z and Together recently announced a $20M seed funding round led by Lux Capital. These funding rounds confirm that venture is attracted to the thesis of decentralized AI and betting on it from a commercial perspective in alignment with its obvious benefits for sovereignty and free society. Things are moving fast, so let’s dive in!
—————————————————————————————————————————
The Closed API Paradigm
Today’s most powerful AI models are either completely closed or gated behind commercial APIs that grant limited access to select partners on preferred terms. API gatekeeping can happen in several ways. First, it usually locks in certain aspects of the model — which can’t be changed, for example on the basis of “safety” or some other political decision. The company can also restrict access up to a certain level of compute and/ or elect to grant access to select applicants for specific and usually non-commercial use.
This is the paradigm of centralized AI development that companies like OpenAI and Google are operating under today and both Gensyn and Together aim to disrupt.
If our current economic model has led to the development of centralized AI systems, then it means we will need to create a new financial incentive structure that supports AI systems that are owned by no one and accessible to everyone, instead.
—————————————————————————————————————————
The Together Platform
Together has created a decentralized cloud platform to train and publish large open-source models. It began by using aggregating resources from five large compute providers— Stanford, ETH Zurich, Open Science Grid, University of Wisconsin-Madison, and CrusoeCloud — into a decentralized cloud. It then reproduced 1.2 trillion tokens that were used to train Meta’s LLaMA to create a new base dataset and a group of base and instruction tuned models that are collectively referred to as The RedPajama Project. LLaMa is Meta’s foundational large language model that the company released earlier this year as open-source.
Together’s approach here was rooted in a research paper called Chinchilla suggesting that if you spend more time (and tokens) training a smaller model (with less parameters) you can get the same quality as one that is much larger.
RedPajama aims not only to produce state-of-the-art open models, but also to make the whole process radically transparent by including its creation recipe and training process along with the base dataset. The goal of these efforts is to accelerate innovation around LLMs, in part by creating a high-performing model that can be run on a device as small as a home computer or even a phone (like the RedPajama base model can).
A key characteristic of open-source AI is allowing developers to own their model weights. This is in contrast to the closed source big tech models, which are built around protecting the weights. This design for protection means that providers host their own models and provide access on an API basis.
API access for model training in turn raises privacy issues, which makes privacy difficult for parties downstream who are hoping to use their own data to further train models. These privacy concerns pose prohibitive limitations for many companies — which drives a greater commercial interest in advancing industry-leading open-source to allow these companies to build in-house ML systems in-house that steer clear of data privacy and related compliance concerns.
”AI is going to be a compute disaggregation event.”
Vipul Ved Prakash, Co-founder and CEO of Together
—————————————————————————
This might seem counterintuitive, but as you dive further into the momentum of decentralized machine learning, it starts to make sense. It is a powerful statement that invites us to think about AI more in terms of opportunity than risk and situates the present reality more as a reflection of the past than an indication of the future. On a technical level, this statement refers to how AI workloads are stateless, which means they do not need anchors like databases and file systems but rather, use a form of compressed data that can migrate to the compute.
RedPajama has been trained on repurposed compute from several sources (including GPU crypto mining farms) but they are currently all large data centers. In the future, Together hopes to use crypto tokens to incentivize a wider set of compute providers, like you and I with our devices, at home. If you listened to the podcast, this should sound familiar… because it would mirror the crypto-native compute-driven approach that Gensyn is leading with today.
—————————————————————————————————————————
The Gensyn Protocol
While Together is focusing on techniques to make high-performing open-source models that are more transparent and accessible, Gensyn is (arguably) disrupting the current AI paradigm further by creating the infrastructure for anyone to access large-scale compute and train their own models. Eventually this training will take place on top of a Gensyn-native global foundation model that many of the network’s participants have contributed to training.
“Everyone, train this foundation model. Design it together, we train it together on an infrastructure that nobody owns and at the end of the day we have a model we can all use that is global and hasn’t been biased by a specific company’s cache of data.”
—————————————
At this point, anyone can then find the hash of this model on-chain and continue training at a specific point on their dataset for their task. Gensyn’s approach centers on the question and challenges of decentralized verification of ML work. The core tension of blockchains (interwoven with what makes them so valuable) is that they are slow and expensive for multiple nodes in a network to run every instruction or every computation to ensure that transactions are accurate. This is also what makes the technology incredibly secure, and that security (when effectively implemented) is what allows them to operate as permission-less, decentralized, censorship-resistant…etc.
As a crypto protocol, a core challenge for Gensyn is to scale decentralized verification. This involves ensuring that both 1) a diversity of hardware devices will be compatible with providing compute to the network and 2) off-chain ML computations can be proven on-chain.
“The fundamental challenge in building this network is the verification of completed ML work. This is a highly complex problem that sits at the intersection of complexity theory, game theory, cryptography, and optimization.”
————————
Gensyn begins with securing a global network of compute that is capable of supplying power for the most computationally intensive machine learning models. One of the protocol’s core objectives is to be compatible with all of the world’s hardware devices that wish to contribute computing power to the network. This includes custom ASICs, consumer GPUs and even computer chips like Apple’s M1 and M2.
Once this size has been secured, the protocol ensures that computation can be verified without the need for any third party by using crypto to achieve consensus. It is on these two foundational components of computational liberty that the magic begins.
Its current approach uses several concepts to delegate verification tasks amongst different types of participants — Submitters, Solvers, Verifiers and Whistleblowers. Submitters are the users that provide tasks for completion and pay for units of completed work. The Solver does the work, the Verifier shows whether it’s correct and then the Whistleblower can confirm whether it’s correct. The Verifier will at times show an error to the Whistleblower on purpose just to keep them on their toes— this is all off-chain work completed by off-chain participants who are rewarded when this work is checked and pushed to the chain. The idea is that rewards are distributed to incentive financial rationality over the entire system.
There is a lot of development that is currently happening to advance zero-knowledge techniques across crypto. This is a cryptographic solution that works by generating smaller proofs to submit to the chain that we can rely on to represent the entire, compute-heavy transaction off-chain. ZK solutions also foster a critical measure of privacy, by being able to reveal that the transaction is valid without revealing the specific data that has been used for calculation.
As ZK techniques advance, Gensyn will be able to continue optimizing its approach to verification. For example, one of Gensyn’s co-founders postulates that they might be able to eliminate the Whistleblower role as ZK technology advances in its ability to securely represent computational work performed off-chain.
“Having the Whistleblower is annoying because it’s overcomplicated but it is crucial given the size of the computations to ensure the honesty of the verifier. It is not certain we would always have to have that, there are certain ZK techniques that mean we could but we don’t want to get ahead of ourselves.”
——————
Fin
One useful, albeit simple distinction that emerges for me at this initial stage of the analysis is that a platform is not a protocol.
Together is designed as a platform. They have aggregated a selected set of decentralized compute sources to create a specialized cloud platform for efficiently training large models that are then made available with the base datasets and detailed explanations of process. They have started by releasing open-source models that are size-optimized for people to download, run and fine-tune on their home devices or commodity GPUs. Soon, they plan to open up access to their platform. Together’s roadmap reflects its central role in scaling access to provide compute and directly train models using their cloud.
Gensyn, by contrast, is building a protocol. They are working at the very base of the stack to create a decentralized layer of machine learning compute that anyone can access at any time. The vision is for everyone to participate in creating the global foundation models that live on Gensyn by contributing work that is verified through consensus. Developers can then uses these models to create further specialized models that are nonetheless powered by the generalized intelligence of the whole.
So far, Together has focused on efficiency in training its RedPajama models and its approach could drive innovation in techniques for achieving greater performance with less compute. Gensyn is focused on ensuring that everyone has access to a maximum quotient of compute to equalize the playing field for training models on the ground floor. Both approaches aim to provide developers with friction-less and foundational access to training the most advanced AI.
I’m sure my framing of the issues will change as I continue to spend more time with these concepts. Even in the week spent between releasing my solo pod on decentralized compute and today, it feels like my understanding of the field has grown in many ways. So here’s to an evergreen beginner’s mindset, learning in public and enjoying the ride.
Signing off for now! I hope you leave reading this more informed and slightly unhinged with excitement about the future of open source AI and decentralized compute.
—————————————————————————————————————————
Indivisible is a media outlet focused on sovereignty and culture at the AI - crypto frontier. It is home to the Indivisible podcast and newsletter.
Twitter | Farcaster | Apple | Spotify
#npub15a570wxs8acaa8p6efh7fyuf40ynplzc7ufvlfk39nq285h46a9slced8v
—————————————————————————————————————————