PCHQ is an informal testbed/research facility, where the software and to an extent hardware that will make up EHI and the Estuary V2 initiative is being benchmarked, tested and proven out. Read this for an exclusive inside look!

In 7-10 days, we hope to provide a similar look behind the scenes at our upcoming production hardware refresh - EHI - which will be real hardware in a colocation facility, to an enterprise grade standard (unlike the "just for fun and research" WingsLab @ PCHQ).

We have 4x EPYC CPUs, spread across 4x 4U boxes, each with either 128 or 256G of RAM. There is also a couple of NUC(alike)s (some AMD and an Intel NUC). We're adding a 10G switch soon. There's a pebibyte of hard drives spread across the infrastructure, glued together by a high availability MooseFS Pro cluster, the same that will power EHI.

As the hardware is very much experimental in nature, unlike EHI which is all production enterprise grade and well tested, the PCHQ location is an unofficial "home lab" style deployment, which I personally maintain and build out for myself but run a significant and growing number of test and research workloads on for Outercore Engineering.

Everything is metered to account for power, and most of the equipment runs on a dedicated UPS to prevent outages due to temporary power glitches. The internet connection is provided by a Ubiquiti router and an enterprise fibre 1Gbps connection.

Here is the first photo of the "completed" lab at full power. We will be relabelling things soon, as well as putting everything on true rails and production-ising over the next 3 months or so as I prepare to leave it behind where it will be maintained for at least the next 3 years.

Two NetApp units, a DS4246 and DS4243, provide cool, quiet and cheap expansion of up to 48 hard drives, allowing us to run up to 1.5PiB of capacity+ in the lab if and when needed, when combined with the hard drives already in the rest of the rack. Currently this top unit is offline.

A $150 fan from Bunnings provides enterprise grade cooling, pushing bulk hot air into the Australian heat. For days when that's not enough, we have a whole room cooling solution which can efficiently drop the temperature of the entire room to keep the drives safe in the event that the room becomes way too hot.

We are using this fairly unconventional setup - a home lab - to benchmark and test while we wait for the real EHI hardware to be installed and racked and setup in the lab, to provide a very cheap and affordable and most importantly - immediately accessible way to handle the tasks we need to test for.

Two AMD EPYC 7V13s are being procured for the lab (or rather Wings is buying them) in the near future, which will add a 5th machine with a total of 512GB of RAM, 16TB of SATA enterprise grade SSDs, and 128 cores of processing power, which will be used for testing Lotus mining on Filecoin (via testnet) as well as other tasks, with an eye to testing out MooseFS Pro and other distributed filesystems to find a great way to run a distributed and highly available Lotus setup.

Finally, an Ubiquiti UniFi Dream Machine SE provides core networking and internet access and routing, as well as a specially modified cabinet that securely provides a pseudo-offsite backup (fire resistant cabinet) and additional, unrelated lab for other workloads (unaffiliated with but connected to the WingsLab portion of PCHQ).

It should be noted, for anyone concerned, PCHQ is largely "for fun" - no critical part of our infrastructure will ever run here, but it's a nice experimental lab for testing out and proving concepts, training up in a low risk environment on skills necessary to operate the true EHI environment, and testing theories and benchmarking things.

We'll find ways over time to offload some processing to the WingsLab @ PCHQ in a way that won't matter if it falls over, allowing us to make more effective use of the processing power of EHI proper by offloading low importance or low risk/non-time-sensitive tasks to PCHQ.

The ambient temp in the room in summer sits close to 38°C, the noise levels normally sit at about 43dB.

We hope you enjoyed this inside look at PCHQ!