The Tableland team has been doing deep research the past few months. We’re amidst some new protocol capabilities and recently wrote about how DIMO and WeatherXM plan to use it—check out the blog post we wrote for a little alpha! Throughout this process, we looked at a number of SQLite tools and wanted to pass along some high-level research as we found them quite interesting!
The following list defines the projects we came across and various notes about them, including consensus type (Raft, Consul, or N/A), features (sync, replication, recovery, hosted, publish, or GUI), whether or not it’s open sourced, and potential usage highlights/limitations for our use case. It’s not meant to be fully comprehensive, but it just might be useful to folks looking for database products/solutions.
LiteSync allows your applications to easily keep their SQLite databases synchronized.
Features: replication, sync
Open Source?: no
Consensus: Raft
Usage: Simple to integrate, does replication, sync, and encryption. Full sync between replicas. Directly operate via SQLite.
Marmot is a distributed SQLite replicator.
Features: replication, sync
Open Source?: yes
Consensus: Raft
Usage: Change detection running on NATS (https://nats.io). Appears to support sharding. Uses RAFT via NATS for conflict resolution. Last writer wins. NATS appears to be something like LibP2P, but with connection and scalability tooling built in.
rqlite is an easy-to-use, lightweight, distributed relational database, which uses SQLite as its storage engine.
Features: replication, sync
Open Source?: yes
Consensus: Raft
Usage: Full replication across replicas. Each replica can write, uses RAFT for consistency. However, the API wraps SQLite.
Dqlite is a fast, embedded, persistent SQL database with Raft consensus that is perfect for fault-tolerant IoT and Edge devices.
Features: replication, sync
Open Source?: yes
Consensus: Raft
Usage: Supported by Canonical (folks behind Ubuntu). Wraps SQLite and includes networking and RAFT consensus. While it does sync, it isn’t bi-directional like other projects. This is purely for scaling and still uses a client-server architecture. However, the API wraps SQLite.
LiteFS is a FUSE-based file system for replicating SQLite databases across a cluster of machines.
Features: replication, sync
Open Source?:
Consensus: Raft, Consul
Usage: Replicating database state across a cluster of replicas. Uses Consul for leader election, and projects like litefs-raft provide things like RAFT.
Litestream is a standalone disaster recovery tool for SQLite.
Features: replication, recovery
Open Source?: yes
Consensus: N/A
Usage: Litestream is a streaming replication tool for SQLite databases. It runs as a separate background process and continuously copies write-ahead log pages from disk to one or more replicas. It does not do sync.
SQLite3 Sync is a live SQLite3 database master-slave replication with sqlite3-rdiff using rsync over SSH.
Features: replication, recovery, sync
Open Source?: yes
Consensus: N/A
Usage: Live SQLite3 database master-slave replication using rsync over SSH.
Litereplica allows your applications to easily keep replicas of SQLite databases
Features: replication, recovery
Open Source?:
Consensus: no
Usage: Litereplica is a library that provides master-slave replication with automatic failover. Appears to be related to the Litesync project.
SymmetricDS is open source database replication software that focuses on features and cross platform compatibility.
Features: replication, sync
Open Source?: yes
Consensus: N/A
Usage: Generic database replication software that focuses on features and cross platform compatibility. This isn’t specific to SQLite per se, but does replicate SQLite as a feature.
Mycelial is the one-click data backbone for Edge AI/ML, Cloud ETL, and everything in between.
Features: replication, sync
Open Source?: yes
Consensus: N/A
Usage: SQLite extension that connects a remote hub that can sync many readers on your DB.
SQLite Cloud is a is a distributed relational database system that ensures strong consistency across all nodes in a cluster.
Features: replication, hosted
Open Source?: no
Consensus: N/A
Usage: Fully hosted SQLite DBs with some auto-scaling, auto-replication things built into the dash.
DBHub is a Cloud for SQLite databases.
Features: publish
Open Source?: yes
Consensus: N/A
Usage: Create private and public datasets. Integration for DB Browser for SQLite. CLI interface. Automatic alerts for data changes. Collaborative development.
SQLite Browser is a high quality, visual, open source tool to create, design, and edit database files compatible with SQLite.
Features: GUI, hosted
Open Source?: yes
Consensus: N/A
Usage: Online viewer that integrates with DB Hub. Simple visualization tool for SQLite databases.
Turso is an edge-hosted, distributed database based on libSQL, an open-source and open-contribution fork of SQLite.
Features: replication, hosted
Open Source?: yes
Consensus: N/A
Usage: Turso is an edge-hosted, distributed database based on SQLite. Designed to minimize query latency via intelligent geo-replication. A major selling feature is its integration with platforms like CloudFlare, Netlify, and Vercel. Some of it is open source.
Cloudflare D1 is Cloudflare’s native serverless SQL database.
Features: replication, hosted
Open Source?: no
Consensus: N/A
Usage: D1 is Cloudflare’s native serverless database. It is based on SQLite, and integrates with Cloudflare Workers. (In fact, Tableland’s SDK is fully D1-compatible!)
Awesome Decentralized Database is a curated list of the best decentralized database solutions, tools, resources, and learning materials that empower developers to build data-driven applications on the decentralized web. (Note: if you see anything missing, PRs are welcome!)****
You can check out the repo linked above for the details, but a current snapshot of what we’ve put together is outlined below.
AergoLite - AergoLite: SQLite with Blockchain.
Catena - Catena is a distributed database based on a blockchain, accessible using SQL.
Daisy - Private proof of authority blockchain where blocks are SQLite databases, in Go.
Tableland - Decentralized SQLite database that provides provides wallet-based authentication, row-level permissions, and high performance at a low cost.
go-bs-sqlite3 - IPFS blockstore.
ipfs-sqlite-block-store - IPFS blockstore.
ipfs-sqlite-cli - CLI utility to backup and restore a database from IPFS. Requires local instance of GO-IPFS running.
CR-SQLite - CR-SQLite is a run-time loadable extension for SQLite and libSQL. It allows merging different SQLite databases together that have taken independent writes.
IceFireDB - IceFireDB-SQLite database is a decentralized SQLite database.
ChiselStore - ChiselStore is an embeddable, distributed SQLite for Rust, powered by Little Raft.
ha-sqlite - High-availability sqlite database service based on raft.
raft-sqlite - Raft backend using SQLite.
ReSqlite - ReSqlite is an extension of Sqlite that aims to add basic replication functionality to Sqlite database.
tqlite - Distributed SQL database with replication, fault-tolerance, tunable consistency and leader election.
Fireproof - Fireproof is a decentralized realtime database that stores documents using prolly-trees.
Redwood - Highly-configurable, distributed, realtime database that manages a state tree shared among many peers.
BigchainDB Blockchain database.
Bluzelle - Decentralized database for Web 3.0 using blockchain and edge computing for greater performance and security.
ComposeDB - Decentralized, composable graph database to build interoperable applications on Ceramic.
The Graph - Indexing protocol for querying networks like Ethereum, IPFS, Polygon, and other blockchains.
db3 - Open-source and decentralized firebase firestore alternative for building fully decentralized dApps quickly with minimal engineering effort.
OrbitDB - Serverless, distributed, peer-to-peer database.
WeaveDB - NoSQL database powered by Warp Contracts (SmartWeave) on the Arweave blockchain.
FreeSQLiteEncryption - Add-on to the public domain version of SQLite that allows an application to read and write encrypted database files.
Lit - Create and manage distributed cryptographic key-pairs for condition-based encryption and programmatic signing.
Threshold Network - Access control with end-to-end encrypted data sharing and communication without the need to trust a centralized authority.
sqlcipher - Fork of the SQLite database library that adds 256 bit AES encryption of database files and other security features.
sqlite_codec - sqlite3 with built-in transparent encryption based on mbedtls library.
Awesome Database Learning - Another awesome list for database learning.
Learn GraphQL - Learn real world GraphQL, quickly.
Learn SQL - Practice and learn SQL from the basics.
If you found this interesting, give out a follow on Twitter, check out our GitHub, and drop into our Discord. We’re always down to discuss the latest and greatest in database and blockchain tech! And soon, we’ll be launching our Studio web application to make it easier to develop with Tableland, so sign up for the waitlist if you’re interested in trying it out.