HollowDB is a decentralized key-value database on Arweave network, developed by FirstBatch. The goal of HollowDB is to provide a database for applications with actual production-level stability and feasibility for projects of any scale, while also being privacy-preserving.

Why?

Collective data shouldn’t be owned by a single entity.

A database consists of fragments from each user/entity. Therefore, each user should own their part of the database. This invokes the question of how we determine if a request to change a fragment of data came from the actual owner. The question of who demands the storage of personal data on centralized or decentralized servers, and in both cases, we can't deliver a trustworthy experience. As a result, we need to have a DB where:

• Only the owner of the data fragment can control it

• Ownership is provable without revealing sensitive information like wallet addresses, emails, and usernames

• Data is persistent and reachable without any dependencies

• Read and write times are acceptable from a user's perspective.

We've built HollowDB on these fundamentals as a decentralized key-value database that preserves privacy. It's a key-value DB because, as FirstBatch, we needed a performant decentralized DB method with simple read and write and fast retrievals.

Key Takeaways

• A fully decentralized DB for applications

• Actual production-level stability and feasibility for projects at any scale.

• A decentralized database with UX-wise acceptable read & write times

  • Write interaction ~1.5s on average, read interaction ~300ms on average with gRPC (bundled data).

• Almost constant read/write times, regardless of the database size.

• Permissionless and privacy-preserving write operations on DB using Zero-Knowledge Proofs

• Permanent storage with low write costs

• A few lines of code are needed to integrate into your full-stack application.

• All available on GitHub

Building on Permaweb

The permaweb is a permanent and decentralized web built on top of the Arweave, a permanent information storage network trusted by outstanding companies like Meta.

Permaweb allows permanent storage, fast retrieval, flexible querying, and smart contracts, making it a robust blockchain with extra utilities. HollowDB is built on top of Permaweb to deliver permanent data storage with a single upfront fee, which is significantly low compared to other blockchains, and provides programmability with SmartWeave contracts on stored data.

Read more on Arweave Wiki.

Warp Contracts

Warp by RedStone wraps SmartWeave contracts to boost overall delivery to a great extent. Supporting multi-language, wasm development of smart contracts that execute code off-chain and transfer the computational load of contracts to the client side.

Besides unparalleled scalability with client-side execution, Warp also bundles interactions and guarantees transaction success. In Ethereum terms, Warp by RedStone hands out an L2 and a Relayer directly integrated into contracts for scalability, finality, and availability.

HollowDB: Key-value DB

So with all that reasoning, we've built HollowDB on the Permaweb.

HollowDB utilizes Lazy Evaluation, KV-Storage (by Warp), and Zero-Knowledge proofs to provide its full functionality.

HollowDB allows the storage of complex objects as key-value pairs on the blockchain. An AR wallet is sufficient to start developing with HollowDB.

KV Storage

HollowDB heavily relies on the powerful feature of KV Storage introduced by Warp contracts. SmartWeave contracts need to evaluate the whole state before write and read interactions with the contract. As DB stores the data in the contract state, databases with large numbers of entries become infeasible in terms of interaction. KV Storage solves the problem by evaluating/retrieving only the required parts of the state, which is essential for HollowDBs functionality.

Bundled Data

Keys can be strings of arbitrary length, while values can be objects with various layouts. HollowDB enables the functionality to bundle large values with Bundlr and store only the txID of the bundled interaction. This adds an extra call but significantly reduces the size of the DB.

Zero-Knowledge Proofs

HollowDB presents a configuration that makes data update and remove operations gated with zero-knowledge proofs. Users can only update key-value pairs by providing a zero-knowledge proof that they know the pre-image of that specific key. This configuration requires keys to be Poseidon hashes of arbitrary strings. Proof acts as a JWT to validate the requested change without revealing the pre-image. Using this configuration through gRPC nodes allows apps to preserve users' privacy while remaining fully decentralized.

Whitelists

HollowDB can be used without zk proofs. To protect against unwanted data manipulation, one can add desired wallet addresses to the whitelist, granting them the role to read & write to HollowDB.

gRPC Nodes

The power of decentralized key-value storage can truly scale with gRPC nodes. HollowDB provides plug-and-play gRPC nodes acting as decentralized clusters syncing with blockchain states. gRPC unlocks:

• True privacy preservation

• Web2-like read/write on large-scale data

• Gassless experience for entries with large payloads

More on HollowDB gRPC.

Using Hollow

There are two distinct designs applicable to using HollowDB.

Using HollowDB: Vanilla

Takeaways:

• No backend, operational with client-side code

• Gasless interactions for data size < 2KB

• Transactions are linked to the user's Arweave wallets, therefore, can't be fully anonymous.

• Users must download the cache (the whole DB) to their devices.

After deploying the contract, users can directly start reading and writing data from/to HollowDB on the client side with their wallet. Devs can attach HollowDB to their applications with a few lines of code and without a backend.

The vanilla version is best suitable for projects that store small data on a single HollowDB contract. Any user trying to read a single key from the DB must download the whole contract state and cache it locally on the first interaction. After the first interaction, Warp downloads only the non-existent data to the local cache, syncing the DB with the blockchain.

Vanilla usage is a perfect fit for a design choice of deploying unique contracts for every user or applications with a small DB size. For instance, let's think of a calendar application where each user has their HollowDB contract, a personal DB where they can directly control their calendar in a decentralized manner. HollowDB implements an LMDB cache for vanilla usage.

Thanks to Warp, a single transaction smaller than 2KB is free. For larger sizes, bundle data with Bundlr and store only the TxID on HollowDB.

One downside with the vanilla version is that users create transactions with their Arweave wallets, linking the key-value pairs to an Arweave address. Therefore one must consider the specifications of DB and data in the vanilla version.

Using HollowDB: gRPC

Takeaways:

• The best option for scalability; can scale as much as you need

• Gasless for any data size

• It can be fully privacy-preserving, anonymous writing operations

• Faster read/write times compared to vanilla

• It needs a backend to run gRPC Server

• gRPC clusters are always in sync using the blockchain state

• The user doesn't have to download data

By enabling gRPC nodes, HollowDB transfers the burden of downloading caches from the user side to a distributed set of nodes. These nodes act as APIs and can be queried by gRPC stubs attached to the client side.

HollowDB gRPC nodes utilize Redis cache to store contract state efficiently with multiple concurrent read and write operations. Each gRPC server acts as an API in sync with the shared state on the blockchain. The fact that nodes don't have to communicate with each other makes large-scale operations painless. In addition, HollowDB implements both server-side and client-side load balancing for multiple clusters creating a highly scaleable system for production-grade products with high uptimes.

HollowDB gRPC transactions are created by single or multiple wallets provided by the developer, allowing:

• Developers to fund DB, enabling a gasless experience without data size limits.

• A privacy-preserving pipeline with Zero-Knowledge proofs and independent wallets

In this case, it is safe to say gRPC acts as a mixer for update and remove transactions.

The main downside is that gRPC nodes require a backend, a set of dedicated machines to run. Though it may seem this defeats the purpose of decentralization, there are two things to consider.

First, the gRPC structure can also be decentralized, where developers around the globe can run your DBs node to provide decentralization and better locality distribution.

Second, in a case where all your dedicated machines fail, one can retrieve data directly from your HollowDB contract.

Remarks

HollowDB by FirstBatch is an effort to make decentralized, privacy-preserving databases a reality all web, including web3. Our efforts heavily focused on providing the best user experience for large-scale systems while remaining loyal to our core values. We hope this public repository will be a good starting point for a collaborative effort to make decentralization and privacy feasible for all.

Repo:

gRPC:

For inquiries: developer@firstbatch.xyz

Note: We named it after the wonderful indie game, Hollow Knight :)