by Doug Petkanics, Livepeer Co-founder and CEO.

I’m excited to introduce a vision for the next phase of the Livepeer Network - Cascade. This post will aim to:

• Provide a brief background on the 7+ year history of Livepeer from a product perspective.

• Introduce Cascade, a vision for how Livepeer can be the leading platform and community for the future of real-time AI video.

• Explain why this is a unique opportunity for Livepeer that is well suited to the project’s strengths.

• Outline the focus this can bring to the community and lay out practical next steps.

A Brief Journey Through A 7 Year Livepeer History

In the nearly 7 years that the Livepeer Network has been live, it’s gone through many successive phases that have increased the capabilities of the project, scaled the project, furthered decentralization of the project, and enabled sustainable public goods funding for the project. Through this iteration, Livepeer has delivered many core utilities to build the world’s open video infrastructure.  This has been evidenced by the millions of minutes/week of video transcoding and AI media inference it performs across hundreds of applications built by developers building toward the future of AI media. It generates fees in both ETH and LPT split across hundreds of hardware contributors who run nodes and GPUs on the network. It is one of the most widely used practical projects that pioneered the DePIN category.

While the raw utility, strong community, and demonstrated use cases on top of the network are all positive signals relative to many other early decentralized tech projects, it is still fair to say that the Livepeer Network as a product hasn’t yet found that clear breakout fit that drives growth momentum and inbound interest at an exponential rate. Transcoding and generic AI inference alone, with their value props of cost-effectiveness and open access, remain critical elements of the video infrastructure of the future but haven’t yet been bundled into the must-have-hair-on-fire-solutions that send hundreds of potential users at once clamoring for direct integration with the network. As such, it’s critically important that the Livepeer community continue to iterate and experiment with various productization theses that ensure that the network delivers must-have-utility for a broad and fast-growing market - one that can provide orders-of-magnitude fee growth on the network and a thriving community coming together to build the project toward the future to support this vision.

I’m excited to introduce this proposal for the next phase of the Livepeer Network: Livepeer Cascade. At its core, Cascade proposes that during this phase, the project focuses on building the leading platform and community for the future of real-time AI video pipelines.

Livepeer Cascade Vision - The Leading Platform and Community for Real-Time AI Video Pipelines

For those paying attention, it’s no surprise that the rise of artificial intelligence impacts every vertical across technology, and media is no exception. ChatGPT showcased the power of text-based LLMs two years ago, and it wasn’t long before richer forms of media in the form of image generation models like Dall-E, Midjourney, and Stable Diffusion became consumer accessible and became built into some of the fastest-growing consumer products and open models of all time. In 2024, the world is beginning to experience the power of AI as it relates to video, with the Sora demonstrations and new models emerging each week showing how videos can be generated, enhanced, and analyzed using AI.

While these sorts of batch media generation are handy for creating files or analyzing data asynchronously, a new, exciting, cutting-edge wave of technologies is coming that operate not just on prompts or files but also on livestreams of media or data. They can generate or modify outputs in real-time to keep up with the speed of livestreams, and they can perform analyses that output metadata that can be used in real-time decision-making.

Cascade - any situation where something flows or occurs in succession, often building in intensity or effect.

This area of real-time AI video is at the forefront of innovation. It requires a very different technical stack and approach than the very saturated batch AI processing market that is so prevalent today. Fortunately, one of Livepeer’s key strengths and assets is its real-time streaming stack. Every week between 78% and 92% of the usage on the Livepeer network is powering livestream transcoding, which can be broadly applied to other forms of computing on livestreams, including running AI pipelines.

What is a real-time AI video pipeline? Many AI models perform a single processing function on media. A chained together set of AI models and processing steps that operate on video, and execute many different steps to accomplish a use case or business goal can be called a Pipeline. A pipeline consisting of many steps can be far more powerful than just a single model.

What sorts of things are possible via real-time AI video? Some exciting new use cases range from creator- and entertainment-focused to real-time analysis of investment decisions. See the following examples

Content Creation

See this example using a livestream video input to animate and direct an avatar within live content.

Video Agents

AI-powered video agents can be deployed to identify objects, assess environments, or generate insights directly from live video streams, reducing the need for manual labor and on-site assessments. The diagram below illustrates a range of applications for video agents. Each use case showcases the versatility and efficiency gains possible through AI-driven video agents operating in real-time, providing scalable, dynamic solutions to industries reliant on live data analysis and decision-making.

Real-Time Analysis

Real-time analysis of streams can be applied across countless business use cases. See this sports example, which can feed real-time data to coaches making decisions. Previously, this data was only accessible to top-level professionals with big budgets, but it can now be accessible via open-source models to any high school coach with access to a camera.

Analysis use cases can be extended to any sort of monitoring use case, be it traffic, safety, security, business intelligence leading to real-time trading or investment actions, or more.

Livepeer as the Leading Platform AND Community

As mentioned, Livepeer’s stack is already very well suited to perform real-time processing on video streams. However, as we’ve seen with transcoding and generic AI inference, it’s not enough just to have technical capabilities on a network. That also needs to be coupled with a solid go-to-market motion to introduce demand to the network, and in those particular areas in crowded market segments, demand generation took the form of B2B sales and BD. While leading to growth over time, it was competitive and a bit slow going.

In cutting-edge areas of niche AI experimentation, the early momentum is typically far more community-oriented. Before knowing exactly how all of these technologies will be used at scale within applications, the most vibrant activity is within communities of enthusiasts - tinkerers, hackers, creatives, and engineers who are playing with new models and pipelines, sharing and showing off their outputs, discussing techniques for optimization and performance all evening in chat rooms, and building clout and reputation through sharing the latest innovations. The momentum often follows things that may look on the surface like fun toys to play with. We see this through examples like Civitai and Huggingface spaces, where the browsable galleries of potential outputs are just as inspiring to the communities and users as the underlying technical innovations are.

These communities and popular tools-based communities like ComfyUI are thriving but not built and designed around real-time AI video. The tools accept static inputs like prompts or files. The outputs are similarly static. They don’t offer a playground and browse experience to hook up existing livestreams or let tinkerers go live from their webcam. And perhaps more importantly, they aren’t coupled with infrastructure stacks that can keep up with the real-time latency and scale demands for ingesting, computing, and serving back AI-enabled video in real-time.

Livepeer’s community is already an excellent seed for what could be the center of innovation in this new real-time AI area. The tech stack, Livepeer network, and the incredibly knowledgeable base of node operators and video developers present a powerful starting point for a community to rally around showcasing, sharing, playing around with, and innovating on real-time specific AI models. This community growth is where momentum can be built and picked up significantly for the Livepeer project.

As the technology matures and developers are already innovating and playing with these inspirational real-time AI technologies within the community, it’s only natural that many will begin to bring them into more scalable business and application use cases down the line. Because real-time AI video needs to be paired with an infrastructure that can handle the scale and latency, unlike text and image batch processing that can be done closer to the edge, the Livepeer network itself will stand to benefit and grow from the fees paid in not only through the initial community playground experience, but eventually from the integrations into these scaled applications.

A relentless focus on providing this infrastructure and cultivating and growing the leading community product and platform for playing with these technologies will be central to Livepeer’s growth and success.

The Unique Superpowers of Livepeer and Opportunities This Creates

There are a few potential reasons why pursuing the goal of being the leader in real-time AI video would produce excellent outcomes for the Livepeer project.

The Superpowers that make Livepeer well suited to succeed at this effort

1. Live video is a differentiator that aligns with Livepeer’s tech stack and expertise. Most generic AI platforms don’t have the tech, expertise, or inclination to spend years building a global video ingestion, serving, and processing infrastructure.

2. Cryptoeconomic incentives and public goods funding present opportunities to make AI developers stakeholders in the network and its future use. Centralized platforms may be able to use fundraising capital to build out capabilities, but they can’t easily make the contributors stakeholders in the future growth of the networks the way token-coordinated networks can.

3. Community-Oriented. As an open and community-oriented project, Livepeer is well suited to pursue an effort that requires and benefits from community engagement, support, and growth. To date, this has been very effective on the supply side of the network, but with this real-time AI video community approach, the network's usage will also be reinforced by a demand-side community of developers.

4. Cost-effective infrastructure. Video-specific AI computing requires 10-1000x the compute of text or images, and GPUs come at a cost. Livepeer has already shown the ability to provide cost-effective GPUs through crypto-economic bootstrapping incentives and an open marketplace.

The winning outcomes for Livepeer

1. Opportunity to be the market leader. Going after real-time AI video pipelines is a bet on the future. Not many other tech platforms are trying to be this, and it’s still unclear what killer use cases will emerge. By leaning in early, and leveraging the above strengths, Livepeer can be a market leader rather than a competitor in a crowded space where it may struggle to catch up.

2. A network effect. Each new real-time AI video pipeline or plugin added to the network is open and composable with other pipelines also on the network. This means that every new pipeline makes the network MORE useful than before, and the more useful this network is, the better alternative it is to DIY cloud platforms, which should attract more developers. Unlike transcoding, where each user doesn’t add any more utility to the network, for the first time, this presents an opportunity for an actual network effect that can create a defensible moat around Livepeer as the most useful real-time AI platform. This is exciting.

3. Fee growth, community growth, decentralization. Ultimately, a growing community that is adding and playing with real-time AI pipelines leads to increased utilization of the Livepeer network. Those pipelines inspire application developers and businesses, leading to even more scale. Fees into the network accrue value to the stakeholders powering the network, and ultimately, Livepeer sees the demand growth that keeps those GPUs humming and the live video flowing across a decentralized network of node operators serving diverse AI video pipelines and use cases.

Focus

Previous network phase summaries have often listed out a number of protocol changes to support scaling, decentralization, or new capabilities. We’ve laid the groundwork with a robust and decentralized video infrastructure network, various core teams and SPEs that contribute development to different vital areas of the stack, and public goods funding sources.

The Cascade vision in this proposal is far more about focusing the entire ecosystem on one significant opportunity. While every stakeholder can use and build on the open Livepeer network as they see fit according to the protocol, we’ll be far more likely to succeed if we can marshal the above stakeholders, node operators, SPEs, community admins, and more to lean into this opportunity and spend the next 6 months validating that Livepeer can become the leading platform and community in real-time AI video pipelines. This may manifest in some dramatic changes to how the community messages Livepeer to the world, the tools we use, the metrics and stats we look at, and the product interfaces for both node operators and users of the network.

What sort of things does the community need to come together to build?

Practical Next Steps and Roadmap

At the moment, there are some early prototypes and demos that show real-time AI workflows being run in the context of Livepeer software. See this StreamDiffusion demo, which takes prompts and converts mouse movements in real-time to visual outputs.

Several elements must be built to deliver an initial product that makes these types of pipelines usable and contributable to the network. Here’s a quick glimpse at some of the components and stakeholders who will be focused on this in the coming months.

Phase 1 - Initial Real-Time AI Video Pipeline Playground - December 2024

The initial phase of delivering on this vision includes demonstrating and inspiring the world with the early possibilities of real-time AI video pipelines. To do this, the Livepeer community will enable developers to discover, test, and deploy AI-enhanced real-time video experiences. The first milestone will feature:

• A playground to experiment with real-time AI video pipelines in your browser, letting creators and developers transform live video streams with just a few clicks and share the results with collaborators.

• Livepeer network support for these pipelines, including payments, on our orchestrator Livepeer nodes.

• Self-serve live Pipeline deployment for ComfyUI workflow creators. This will allow you to convert your ComfyUI workflows into live video pipelines and deploy them on the Livepeer network.

• A community experience designed around onboarding, engaging, and retaining real-time AI pipeline developers. Let’s start cultivating the home for this thriving community, which may require a bit of a revamp of Livepeer’s existing community platforms.

Phase 2 - Standardized Pipeline Contributor Platform - Q1 2025

The second phase goes from provable and inspirational demonstration to permissionless contribution of custom pipelines. It allows unbridled community growth and the creative use cases enabled by real-time AI pipelines to flourish. It includes potentially:

• Standardized pipeline contribution framework or SDK.

• Curation, reputation, and social elements amongst pipeline developer community in the AI pipeline playground and community.

• Growth hacking to accelerate the community growth

Phase 3 - Scalable real-time AI Developer platform - Post-community growth

When Livepeer is a thriving and growing community of real-time AI pipeline contributors and a growing number of inspirational pipelines, it becomes time to focus more on the tools that enable developers to integrate these pipelines into their applications and businesses at scale. This means there’s an increased focus on:

• Network reliability and cost

• Developer SDKs and product interfaces for integration

• Community-driven support and resources for application builders

While phase 3 represents significant growth and scaled usage, phase 2 alone can dramatically increase the usage and fees on the Livepeer network. All of these cutting-edge technologies have begun to be adopted by communities tinkering with them, and the pipeline showcase and playground allow significant usage amongst a growing community. If there is growth and momentum there, those developers will inevitably take the pipelines into applications at scale, the same way we’ve seen image and text-based AI models be adopted into countless products directly out of the experimentation playgrounds.

In Summary

This post contained lots of information about the potential of Cascade and this new opportunity. But in summary, if the Livepeer community can come together to do the following things in the next 6 months, then we’ll collectively be off and running with a shot at achieving this exciting goal:

1. Add real-time AI video pipeline support to the Livepeer Network and surrounding tools.

2. Build a great pipeline showcase and playground UI that lets people show up to discover these pipelines, run them, and view and share the outputs to inspire others.

3. Update our community experience to be an excellent home for those looking to experiment and push the boundaries of what’s possible with real-time AI video pipelines.

4. Leverage our public goods funding programs to create the right incentives to make builders in this space stakeholders in the Livepeer network.

This new and exciting opportunity energizes me. For all those in the community or looking to join the community to get involved in this mission, the best starting points are:

1. Get early access to real-time AI video pipelines.

2. Stay up to speed with development by joining the Livepeer’s Discord community.