LumoSQL ↗ protects data on mobile phones using a new data storage technology which is highly compatible with most existing devices. With LumoSQL, the device owner has ultimate right to decide who can read or change their data… and this decision continues to be enforced even after it has been copied off the phone to (for example) a bank or insurance company for processing with their in-house database software. Today, device owners are rarely in control of the privacy of their own data, despite many laws relating to privacy.

If someone takes a phone away from its owner, LumoSQL data cannot be read without the consent of either the phone owner or someone to whom the phone owner has granted access. These controls are fine-grained, meaning different levels of permission can be granted.

Why this matters

A mobile phone stores all its non-streaming data — contacts, messages, health records, browsing history, app settings — in hundreds of SQLite ↗ databases. SQLite is also found in web browsers, operating systems and vehicles, and it is a standard data format ↗ relied on by companies such as Airbus whose products last decades. All this makes SQLite very likely the most-deployed software in the world, by a factor of at least three zeros if not four, and probably the only trillion-scale software in existence.

And yet SQLite stores all of this data in plain text. Whole-device encryption does not address the problem, because once the phone is unlocked, any process that can open a database file can read every row in it. It is unacceptable to store personal data this way, but SQLite works so well and is so ubiquitous that it is hard imagine an alternative.

LumoSQL does not try to be an alternative, because it modifies SQLite (rather than replacing it) so that existing apps continue to work without change, but the data they store gains protection. At-rest encryption of the whole database file works today, so that even an unlocked phone cannot open a database unless the user gives a key. The larger idea is a Lumion, a row of data that carries its own rules with it, like a letter in a locked envelope where only the people named on the envelope can open it, even after it’s been forwarded. A Lumion still enforces its owner’s permissions no matter where it goes by email or in the cloud, or copied to a corporate database or some other device. The phone owner decides who can read or change each piece of data, or at least set default policies.

The detailed strategy document LumoSQLMotivation-1.0.pdf explains the social, business and technical pressures behind LumoSQL. As the strategy document says, “LumoSQL assumes software development should never be relied on and is getting worse”.

Surprising facts about SQLite

  • SQLite is a full-featured database, supporting the standard SQL language despite being tiny compared to all the other mainstream SQL databases.
  • SQLite is open source, exceptionally well-maintained, mostly by just 3 people. Many more people contribute occasionally to SQLite, and the community of deeply technical users is very large.
  • SQLite is exceptionally reliable, given the policy decisions not to change certain fundamentals.

The corollaries to these unusual facts are significant:

  • This is uncharted territory for Computer Science: is SQLite’s ultra-conservative compatibility commitment to its (at least) hundreds of billions of installations the right choice? Is SQLite’s fast-moving support of formal database standards the best way forward? It does certainly work very well.
  • Of the many forks of SQLite, none seem to have more than a relatively trivial deployed base (likely at most a few handfuls of millions.) This includes forks with seemingly very useful features, so it appears the nature of the SQLite project ensures its success and discourages replacements. This does constrains its future in important ways too, especially that it cannot introduce encryption.
  • These obvious strategic problems with SQLite around privacy do not seem widely discussed. But why? Perhaps LumoSQL will help make these discussions possible.

Lumions

If combined even to some extent with the immense scale of SQLite, a new concept for encryption is potentially transformational for all device users. Lumions are just such a new concept, being a row of data that carries its own rules within it.

Wherever a Lumion goes, any user can prove what the data is, who wrote it, and who is allowed to read or change each part of it, without contacting a server or an authority. That is the opposite of the usual arrangement, where the rules live in a program: one developer’s code decides that you may not read someone’s salary, and if the code is wrong, or doesn’t know the salary is your salary, there is nobody to argue with.

A Lumion takes a plain row of information (think of a line in a CSV file) and adds encryption, a global name, selective disclosure of individual fields, and a way for one row to point at another. It is closed under copy: everything it needs to work travels inside it, so it keeps working after being copied to the cloud, to another device, or into someone else’s database. Lumions wouldn’t be needed if JSON already had these ideas.

Two draft RFCs ↗ are complete: the Lumion object itself, and its byte encoding. Between them they contain everything needed to write a conformant implementation. A Python demo shows how to build and use Lumions in a plain text file from the commandline.

Funding and contributors

LumoSQL exists due to the volunteer effort contributed by many skilled people. The Vrije Universiteit Brussel ↗ funded valuable contributions in cryptography and mathematical analysis. The NLnet Foundation ↗ supported LumoSQL from 2020 to 2022. The pandemic years took their toll and the project was dormant from the end of 2022 until 2026. The timing of the reboot turned out well, because Howard Chu was readying LMDBv1.0 for release and LumoSQL inherits a great deal from it. The project history ↗ names everyone.

Information for developers

For much more detail see the code development page ↗ .

In technical summary: LumoSQL is a modification (not a fork) of the SQLite embedded data storage library. v0.83 is a preview release for curious technical developers wondering why anyone would want to touch trillion-scale software. Production use is not yet recommended, although we are reasonably confident LumoSQL will not eat your data.

You can compile LumoSQL against applications that already use SQLite, because from the application’s point of view it is libsqlite3. The tree maintains working examples for Fossil and libfossil ↗ , rpm, and the better-sqlite3 Node.js binding ↗ . Fossil is the demanding case: it is developed alongside SQLite and often needs SQLite versions ahead of the latest release, so LumoSQL follows the tip of SQLite’s development branch to build against it. If LumoSQL can run Fossil, there is a good chance it can run millions of other applications.

LumoSQL offers multiple key-value backend storage systems selectable by the user, and features not found in any other mainstream database:

  • ability to checksum every row on write and verify on read, stored in an ordinary sqlite3 file that any standard SQLite binary can still read and write
  • ability to trigger arbitrary functions on per-row read and write
  • a general test suite for benchmarking precisely how LumoSQL (or SQLite) is performing and the full context of that benchmark run. Results land in an SQLite database and can be merged between users, so runs on different hardware and different architectures can be compared. Database benchmarking is very poorly done, including by the TPC, founded in 1988 for that purpose.
  • a general build system able to mix and match multiple versions of the database with multiple versions of multiple backends. Of the order of a thousand binaries can be built from a decade of SQLite and LMDB releases, and the test suite includes cross-binary interoperability checks. Never before has it been possible to compare the different strategies of various Key-Value stores with the same database frontend.

Notable outcomes from LumoSQL already include:

  • The only mainstream database with swappable Key-Value stores, where all stores are peers rather than one store having special knowledge that gives it technical advantages. Three ship today: the existing SQLite btree with optional binary-compatible modifications such as rowsums; LMDB 0.9.x ↗ , long stable and now in maintenance; and LMDBv1.0, which brings page-based encryption, page checksums and incremental backup, all of which LumoSQL inherits without further work. We look forward to integrating other stores, and are interested in 21st-century designs such as Adaptive Radix Trees.
  • At-rest encryption for an ordinary SQLite application, open source, via libsodium and ChaCha20-Poly1305. Key management is deliberately rudimentary so far.
  • The only mainstream database optionally without a Write-ahead Log ↗ , when used with an LMDB storage backend. LMDB hands memory management to the operating system’s virtual memory system through mmap, which is tuned harder than any userspace pager can be.
  • Benchmarking results ↗ comparing LMDB-backed against native SQLite, and SQLite’s own performance across 22 releases.
  • The Not Forking tool, which avoids forks in both simple and complicated source code, and which is what makes all of the above possible without carrying the burden of forking SQLite.

Exciting things in progress:

  • Row-level encryption, built on rowsums. This is a separate mechanism from the page-level encryption that already works.
  • Documented API for arbitrary key-value stores
  • Documented API for accessing the key-value stores via the SQLite library, instantly making the SQLite key-value store the most widely-distributed key-value store. Nothing calls the SQLite key-value store today except SQLite