COPIUMS stands for Centralized Orange Pi Utilization Management System which helps you track and manage inventory of memory in your running Orange Pi devices. With COPIUMS, you can easily see which devices are being used, which models are running on them, and how much they are costing you to inference. You can also use COPIUMS to manage your Orange Pi devices remotely, making it easy to load hundreds of models into your Mixture of Experts LLM at home, in the garage, batteries included (convert hf models to rknn models as fast as TheBloke releases them with an LLM-assisted seamless CI/CD bulk model inventory and management chat widget tool that’s self-hosted at home).

COPIUMS is the perfect solution for businesses and individuals who are looking for a tiny box… but like smaller or bigger, you know? I call it the “tinierbox” (the size doesn’t matter in a COPIUMS, because you can pick how many Orange Pi units you buy)

With COPIUMS, you can save money, run local and BIG, and I’m talking silly BIG models, how does about 3.2TB of RAM sound to you?

3200 GB of RAM. That’s what I’m talking about. tinybox comes with 144GB GPU RAM for $15,000? This is a whole order of magnitude apart, granted it costs twice as much, but let’s do some quick math here

$ 104.17 per GB - Tinybox

Cost per GB= 144 GB → $15,000 ​ =$104.17 per GB

$ 9.375 per GB - Tinierbox (running COPIUMS)

Cost per GB= 3200 GB → $30,000 ​ =$9.375 per GB

About 11 times cheaper 🫡

That’s like 450 parallel 7B models that your MoE expert can pick from that all just happened to be already loaded into memory, ready for you to inference, a planet’s corpus in-memory in the palm of your hands. AGI scale - At home. (Internally)

It’ll be slower, I think right now we’re doing about 10 tokens / second, 10 is still plenty good for me to not think about as I am reading a sentence coming from an LLM, but what we lack in speed, we can make up for in sheer parallelism!

Back COPIUMS today and help me make it a reality!

What is COPIUMS?

Harness the power of centralized management for multiple Orange Pi units using the reliability of a single Raspberry Pi 4B 8GB and MEGA4 hubs. Elevate your garage-tuned LLM lab startup with simplified, remote control over flashing and management of a theoretical maximum of 100 Orange Pi units.

I’m sold! What am I backing?

Develop and deploy a comprehensive software stack - CLI + WebApp + Server Daemon that lives on the Raspberry Pi which allows users to seamlessly manage, monitor, and flash (sequentially) multiple Orange Pi units from a centralized Raspberry Pi 4 interface.Once the management plane is brought up, we’ll get to control planes that can CI/CD models converted from onnx/huggingface format to rknn format so they can run efficiently on the NPU, and possibly on the GPU.

Components & Setup:

• Primary Node: Raspberry Pi 4B 8GB

  • BMC - Ubuntu

  • Secure and remote SSH access.

  • Flash operating systems on the Orange Pis remotely with a CLI + WebApp that can remotely install models or experts / export metrics / debug Pis remotely

• Connection Hubs:

  • Main Connector: MEGA4 - 4-port USB 3.1 hub tailored for Raspberry Pi 4B.

  • Additional Connectivity: Standard USB 3.1 hubs.

  • Every hub comes with an independent power source for optimum stability.

• Network Management:

  • 4 x Aruba Instant On 1930 52-Port Gb Ethernet 48xGE, 4X 1G/10G SFP+, L2+ Smart Switch

• Power Management:

  • Dedicated power supply for each hub.

Software Framework

• (Proposed)

  • UUGear Web Interface (UWI): Power control for individual MEGA4 ports.

  • Flashing Tool: Sequential flashing for Orange Pi(s).

  • CLI for basic workflows/troubleshooting

  • The web app should allow you to

    • poke at rough metrics around bmc health, uptime, nodes statistics

    • remotely upload new images and models to the registry (loader)

    • report in on usage metrics / cost of operations

  • CI/CD framework that uses nodes in the cluster to bring in huggingface models and convert them into rknn compatible 1-1 models with LLM assisted tooling (pick them up from loader and slowly steadily convert them into rknn compatible models, adding them to the roster of available models overtime that the MoE can consult against)

What are the benefits of using COPIUMS?

There are many benefits to using COPIUMS, including:

• Updates: COPIUMS and the problems under this arc like batching inference using vllm on embedded compute devices is a problem close to metal and I love the challenge, so you can rest assured knowing you will get updates from me, it’s very likely this project will succeed (it’ll just take longer) because it’s poised to just be a repo with scalable distributed embedded computing paradigms baked in for the joy of being able to parallelize generic compute inexpensively using RK3588’s NPU, (we’ll explore adding GPU to the mix as well, since it’s a powerful one)

• Local: None of your data leaves your garage

• Powerful: Each Orange Pi comes with upto 32GB of memory, and 2 x 2.5Gbps ethernet ports for some serious inter-node ethernet bandwidth

• BMC: It’s practical to have just one Raspberry Pi operating the switches on the MEGA4 hubs even with various USB host ports required to initiate and sequence flashing all of the connected Orange Pis, if you get into “Oh I need to be flashing them in parallel, because acceleration, I dont have time to sequentially flash 100 Pis”, you have bigger problems than me and I don’t know how to help you.

  Consider though that once you’ve flashed them once there’s barely any reason to flash them again so even if it takes an hour that’s only 100 hours (4 days) to bootstrap all the nodes remotely (no sd card flashing business, we have per port usb port switching, so everything will be remote and automatic, just need to wire it up), then it’s all state management, and occasional rolling release software upgrades for which usually you can have a few pods that bring in most of the ubuntu packages locally and update them in a decentralized p2p way from each other node, though let me take this opportunity to assure you, that you’ll never have to think about it again, I just explained it because I’ll admit it brings me incredible joy to discuss such systems in detail, because as an aspiring neochibi web3 niche internet microcelebrity status esoteric hipster sys admin indie maker building large language model tooling in open source, I will solve for such silly software package/model distribution issues so you can focus on running LLMs in your garage, with such incredible thought and attention to detail around developer happiness tooling thrown in that in a hypothetical conversation about COPIUMS with Steve Jobs, Iron Man, Christopher Nolan, Richard Hammond, James May, and Jeremy Clarkson, the group will nod in agreement, united by the idea that a man's garage, no matter its purpose, is his sanctuary. A temple of creation, a place for passionately sharpening and fine-tuning tools, and that the COPIUMS 100 was the most practical self-sufficient self-healing garage LLM cluster companion, that was available on the market, easily decades ahead of its time.

  Growth Blueprint: Need more Orange Pi units? It’s just like Factorio! Just paste more blueprints! No more “oh man that new 180B is too big to fit in my garage rig man, I need to upgrade again, im thinking 2 x A6000s 96GB VRAM or Mac Studios 192GB RAM or tinybox 144GB RAM oughta cover me till January whatdyu think man, do you think I’ll make it till December or is it over for me? oh man” Just connect more Powered USB Hubs and connect MEGA4s on the other end. Daisy chain Orange Pis till USB limits of 127 devices.

  You can start with as little as 1 Raspberry Pi and 1 Orange Pi and keep scaling them as you go! <3

Since this will come with a lot of individually addressable RAM sequences, load a Dask cluster operator right off the bat and have a langchain agent ready that can load data frames across the cluster and treat the whole thing as one big memory unit

How can I get COPIUMS?

You can get COPIUMS by backing this mirror mint. I am offering a variety of rewards for backers, including early access to COPIUMS local, if you hold this commemorative piece on Optimism, I will make way for you ssh into this beast / chat with the MoE monstrosity I cook and run in a shared decentralized first come first serve basis, subject to other jobs people have already started on it, likely forever, and holding this will grant you the ability to vote on the names I use for the daemons that will be written for various systems I’ll end up needing to create in the software stack as well!

The current family: Edith - Image Registry (usually containers)Tripsitter - ML Job Provisioner and Scheduler

What are the risks and challenges?

There are always risks and challenges associated with any new project. However, I believe that COPIUMS is a very low-risk project with a huge potential for success. As a cult member of both cute/acc milady, I am extremely passionate about choosing this highly specific hill to die on, the software I’ll write is free and open source of course <3

Proof of work: Raspberry Pi 65B Cluster

Remember this guy? Can you imagine each Raspberry Pi talking to 100 Orange Pis? 600 Orange Pis would be 19.2 TB of RAM plenty of room for 2-3 AGIs in there, this is my Temple OS arc- we’ll call it “Milady OS” 🤍

-Loki (X)

How can I help?

You can help by minting this article, and spreading COPIUMS on the timeline!

Thank you for your support!