Introducing SurrealMX: SurrealDB’s new in-memory storage engine
Author: Tobie Morgan Hitchcock
Today, we are introducing SurrealMX 1.0, our new high-performance in-memory storage engine for SurrealDB.
SurrealMX is an embedded, transactional key-value engine with lock-free reads, designed for workloads where latency and throughput matter most. By keeping the working dataset in memory and combining multi-version concurrency control with a highly concurrent architecture, SurrealMX can process millions of operations per second while maintaining transactional guarantees.
It gives developers a new storage option for low-latency applications, high-concurrency workloads, caching, temporary data, and applications where the dataset can comfortably reside in memory.
Why we built SurrealMX
Not every workload needs the same storage model. For applications where durability and datasets larger than memory are the priority, disk-backed storage makes sense. But there is another class of workloads where the main requirement is simple: make reads and writes as fast as possible (e.g. real-time applications, caching layers, ephemeral state, simulations, agent workloads, testing environments, and applications serving large numbers of concurrent requests). SurrealMX is designed specifically for these workloads.
Instead of optimising a disk-backed database for in-memory use, SurrealMX starts with memory as the primary storage medium. This lets us remove much of the overhead that comes with disk-oriented architectures and design the engine around fast concurrent access.
Built for performance
SurrealMX combines a concurrent adaptive radix tree with multi-version concurrency control and a lock-free circular commit ring. The result is high throughput across reads, writes, updates, deletes, and scans.
In our latest bare-metal benchmarks, SurrealMX delivered:
| Engine | Point reads (OPS) | Creates (OPS) | Updates (OPS) | Deletes (OPS) | Scans (OPS) |
|---|---:|---:|---:|---:|---:|
| SurrealMX | 21,506,711 | 1,887,149 | 1,595,009 | 1,814,037 | 18,031,692 |
| LMDB (in-memory) | 1,226,897 | 232,805 | 241,431 | 177,198 | 1,188,365 |
| RocksDB (in-memory) | 759,060 | 616,649 | 572,246 | 705,773 | 836,666 |
That translates to more than 21.5 million point reads per second, nearly 1.9 million creates per second, and over 18 million range scans per second.
Compared with RocksDB configured for in-memory workloads, SurrealMX achieved approximately:
- 28× higher point-read throughput
- 3.1× higher create throughput
- 2.8× higher update throughput
- 2.6× higher delete throughput
- 22× higher range-scan throughput
The benchmark was run on bare metal using an AMD Ryzen Threadripper 9970X with 128 GB DDR5 RAM, across 15 million keys, 48 concurrent worker threads and 128 clients. Each figure is the best of three runs of that benchmark, for every engine.
Designed for concurrency
Modern applications rarely execute one operation at a time. SurrealMX is designed around multiple concurrent readers and writers without requiring a global lock. Its lock-free circular commit ring provides constant-time atomic slot claiming, while a sharded active-reader registry reduces contention between CPU cores. Reads can inspect version chains without taking locks, allowing the engine to scale more effectively as concurrency increases.
For point reads, SurrealMX can also bypass transaction-pool checkout and reader registration entirely, returning the latest committed value directly. This architecture is particularly useful for applications where many threads or clients are continuously accessing and changing the same in-memory dataset.
Transactions without giving up speed
Being an in-memory database does not mean sacrificing transactional behaviour. SurrealMX supports full transactions, including rollbacks and savepoints, as well as both snapshot isolation and serializable snapshot isolation.
Multi-version concurrency control allows transactions to operate against consistent snapshots while concurrent changes continue in parallel. For simpler workloads, direct operations such as get, set, put, delete and exists provide a lower-overhead path without requiring developers to manually manage transactions. That means applications can choose between convenience and more advanced transactional control while using the same engine underneath.
Fast scans, without the allocation overhead
In-memory analytical and retrieval workloads often involve more than point lookups. SurrealMX provides zero-allocation range scanning, allowing applications to inspect borrowed keys and values directly rather than repeatedly creating and copying objects. In our benchmarks, this helped SurrealMX exceed 18 million range scans per second.
For workloads involving repeated retrieval over subsets of an in-memory dataset, reducing this allocation overhead can make a significant difference both to latency and CPU utilisation.
In-memory first, with optional persistence
SurrealMX is fundamentally an in-memory engine, and its fastest configuration keeps data entirely in memory. In this mode, data disappears when the process terminates. This makes it particularly well suited to workloads such as:
- Caching and temporary state
- Real-time application data
- Development and testing
- Simulations
- Reconstructable datasets
- Low-latency application state
But some workloads need the speed of an in-memory database while still requiring recovery after a restart. For those cases, SurrealMX also supports optional persistence. Developers can configure append-only logging, periodic snapshots, asynchronous persistence, configurable fsync behaviour, and LZ4 snapshot compression.
This makes durability a spectrum rather than an all-or-nothing decision. Applications can optimise for maximum throughput, maximum durability, or somewhere in between.
Reliability under concurrency
Highly concurrent database engines are difficult to test using conventional unit and integration tests alone. SurrealMX uses deterministic simulation testing: seeded, reproducible runs compared against a sequential reference model, to validate its behaviour across many interleavings of open transactions.
The engine has been tested across 25 million simulated operations with zero divergences. Its persistence layer also protects transactions using checksummed append-only log frames. If a crash interrupts a write, recovery can distinguish incomplete data at the end of the log from valid committed transactions, ensuring transactions are recovered atomically rather than partially applied.
Built for embedded and browser applications
Like SurrealDB itself, SurrealMX is designed to operate in more than traditional server environments. It is an embedded Rust engine, meaning applications can run it directly inside their own process rather than communicating with a separate database server.
SurrealMX also supports WebAssembly, opening the door to running an extremely fast transactional database directly inside browser-based applications. This makes it useful across everything from high-performance backend services to desktop applications and browser environments.
SurrealKV and SurrealMX
With SurrealKV and SurrealMX, SurrealDB now has storage engines designed for two different priorities. SurrealKV is our durable storage engine, designed for high-performance persistent workloads where data lives on disk. SurrealMX is our in-memory engine, designed for workloads where the entire dataset can remain in memory and minimum latency is the priority.
Both are being built around the same principle: rather than treating storage as an external dependency, we can design the underlying engines around the workloads SurrealDB needs to support. That lets us optimise more of the stack ourselves, from the query layer all the way down to how data is stored and accessed. For more information, see our Deployment guide.
A new option for low-latency SurrealDB workloads
SurrealMX gives SurrealDB a storage engine designed specifically for the fastest end of the performance spectrum.
With more than 21 million reads per second, millions of writes per second, lock-free reads, transactional guarantees, and configurable persistence, it provides a new foundation for applications where every microsecond counts.
SurrealMX brings the full power of an in-memory transactional storage engine to SurrealDB - built for speed, concurrency, and the next generation of real-time applications.
Go to the SurrealMX GitHub repository, read the whitepaper, or check out our Deployment guide for more information.
22 reactions
Mentions
Introducing SurrealMX: SurrealDB’s new in-memory storage engine Author: Tobie Morgan Hitchcock Today, we are introducing SurrealMX 1.0, our new high-performance in-memory storage engine for SurrealDB. SurrealMX is an embedded, transactional key-value engine with lock-free reads, designed for workloads where latency and throughput matter most. By keeping the working dataset in memory and combining multi-version concurrency control with a highly concurrent architecture, SurrealMX can process millions of operations per second while maintaining transactional guarantees. It gives developers a new storage option for low-latency applications, high-concurrency workloads, caching, temporary data, and applications where the dataset can comfortably reside in memory. Why we built SurrealMX Not every workload needs the same storage model. For applications where durability and datasets larger than memory are the priority, disk-backed storage makes sense. But there is another class of workloads where the main requirement is simple: make reads and writes as fast as possible (e.g. real-time applications, caching layers, ephemeral state, simulations, agent workloads, testing environments, and applications serving large numbers of concurrent requests). SurrealMX is designed specifically for these workloads. Instead of optimising a disk-backed database for in-memory use, SurrealMX starts with memory as the primary storage medium. This lets us remove much of the overhead that comes with disk-oriented architectures and design the engine around fast concurrent access. Built for performance SurrealMX combines a concurrent adaptive radix tree with multi-version concurrency control and a lock-free circular commit ring. The result is high throughput across reads, writes, updates, deletes, and scans. In our latest bare-metal benchmarks, SurrealMX delivered: | Engine | Point reads (OPS) | Creates (OPS) | Updates (OPS) | Deletes (OPS) | Scans (OPS) | |---|---:|---:|---:|---:|---:| | SurrealMX | 21,506,711 | 1,887,149 | 1,595,009 | 1,814,037 | 18,031,692 | | LMDB (in-memory) | 1,226,897 | 232,805 | 241,431 | 177,198 | 1,188,365 | | RocksDB (in-memory) | 759,060 | 616,649 | 572,246 | 705,773 | 836,666 | That translates to more than 21.5 million point reads per second, nearly 1.9 million creates per second, and over 18 million range scans per second. Compared with RocksDB configured for in-memory workloads, SurrealMX achieved approximately: - 28× higher point-read throughput - 3.1× higher create throughput - 2.8× higher update throughput - 2.6× higher delete throughput - 22× higher range-scan throughput The benchmark was run on bare metal using an AMD Ryzen Threadripper 9970X with 128 GB DDR5 RAM, across 15 million keys, 48 concurrent worker threads and 128 clients. Each figure is the best of three runs of that benchmark, for every engine. Designed for concurrency Modern applications rarely execute one operation at a time. SurrealMX is designed around multiple concurrent readers and writers without requiring a global lock. Its lock-free circular commit ring provides constant-time atomic slot claiming, while a sharded active-reader registry reduces contention between CPU cores. Reads can inspect version chains without taking locks, allowing the engine to scale more effectively as concurrency increases. For point reads, SurrealMX can also bypass transaction-pool checkout and reader registration entirely, returning the latest committed value directly. This architecture is particularly useful for applications where many threads or clients are continuously accessing and changing the same in-memory dataset. Transactions without giving up speed Being an in-memory database does not mean sacrificing transactional behaviour. SurrealMX supports full transactions, including rollbacks and savepoints, as well as both snapshot isolation and serializable snapshot isolation. Multi-version concurrency control allows transactions to operate against consistent snapshots while concurrent changes continue in parallel. For simpler workloads, direct operations such as get, set, put, delete and exists provide a lower-overhead path without requiring developers to manually manage transactions. That means applications can choose between convenience and more advanced transactional control while using the same engine underneath. Fast scans, without the allocation overhead In-memory analytical and retrieval workloads often involve more than point lookups. SurrealMX provides zero-allocation range scanning, allowing applications to inspect borrowed keys and values directly rather than repeatedly creating and copying objects. In our benchmarks, this helped SurrealMX exceed 18 million range scans per second. For workloads involving repeated retrieval over subsets of an in-memory dataset, reducing this allocation overhead can make a significant difference both to latency and CPU utilisation. In-memory first, with optional persistence SurrealMX is fundamentally an in-memory engine, and its fastest configuration keeps data entirely in memory. In this mode, data disappears when the process terminates. This makes it particularly well suited to workloads such as: - Caching and temporary state - Real-time application data - Development and testing - Simulations - Reconstructable datasets - Low-latency application state But some workloads need the speed of an in-memory database while still requiring recovery after a restart. For those cases, SurrealMX also supports optional persistence. Developers can configure append-only logging, periodic snapshots, asynchronous persistence, configurable fsync behaviour, and LZ4 snapshot compression. This makes durability a spectrum rather than an all-or-nothing decision. Applications can optimise for maximum throughput, maximum durability, or somewhere in between. Reliability under concurrency Highly concurrent database engines are difficult to test using conventional unit and integration tests alone. SurrealMX uses deterministic simulation testing: seeded, reproducible runs compared against a sequential reference model, to validate its behaviour across many interleavings of open transactions. The engine has been tested across 25 million simulated operations with zero divergences. Its persistence layer also protects transactions using checksummed append-only log frames. If a crash interrupts a write, recovery can distinguish incomplete data at the end of the log from valid committed transactions, ensuring transactions are recovered atomically rather than partially applied. Built for embedded and browser applications Like SurrealDB itself, SurrealMX is designed to operate in more than traditional server environments. It is an embedded Rust engine, meaning applications can run it directly inside their own process rather than communicating with a separate database server. SurrealMX also supports WebAssembly, opening the door to running an extremely fast transactional database directly inside browser-based applications. This makes it useful across everything from high-performance backend services to desktop applications and browser environments. SurrealKV and SurrealMX With SurrealKV and SurrealMX, SurrealDB now has storage engines designed for two different priorities. SurrealKV is our durable storage engine, designed for high-performance persistent workloads where data lives on disk. SurrealMX is our in-memory engine, designed for workloads where the entire dataset can remain in memory and minimum latency is the priority. Both are being built around the same principle: rather than treating storage as an external dependency, we can design the underlying engines around the workloads SurrealDB needs to support. That lets us optimise more of the stack ourselves, from the query layer all the way down to how data is stored and accessed. For more information, see our Deployment guide. A new option for low-latency SurrealDB workloads SurrealMX gives SurrealDB a storage engine designed specifically for the fastest end of the performance spectrum. With more than 21 million reads per second, millions of writes per second, lock-free reads, transactional guarantees, and configurable persistence, it provides a new foundation for applications where every microsecond counts. SurrealMX brings the full power of an in-memory transactional storage engine to SurrealDB - built for speed, concurrency, and the next generation of real-time applications. Go to the SurrealMX GitHub repository, read the whitepaper, or check out our Deployment guide for more information.
SurrealMX: a lock-free in-memory MVCC engine over a concurrent adaptive radix tree SurrealMX will be the in-memory engine in the next release of SurrealDB, and it is written in Rust. Reads never write to shared memory, so they keep scaling as you add cores. It supports serializable and snapshot-isolated transactions, can persist to disk, and runs embedded, in the browser and on a server. With both engines running in memory, it reads 28.3x faster than RocksDB and 17.5x faster than LMDB, and updates 6.6x faster than LMDB. On a 32-core, 64-thread AMD Ryzen Threadripper 9970X with 128 GB of RAM, that is 23 million point reads per second across 15 million keys. The absolute number moves with the hardware. Repo, benchmark commands and whitepaper: Github. Questions and feedback welcome.
28x faster than RocksDB on in-memory reads. Introducing SurrealMX v1 from @SurrealDB. A lock-free, transactional KV engine built for serious performance. Fully open source. Check out the benchmarks: surrealdb/surrealmx
SurrealMX: a lock-free in-memory MVCC engine over a concurrent adaptive radix tree SurrealMX will be the in-memory engine in SurrealDB 4, and it is written in Rust. Reads never write to shared memory, so they keep scaling as you add cores. It supports serializable and snapshot-isolated transactions, can persist to disk, and runs embedded, in the browser and on a server. With both engines running in memory, it reads 28.3x faster than RocksDB and 17.5x faster than LMDB, and updates 6.6x faster than LMDB. On a 32-core, 64-thread AMD Ryzen Threadripper 9970X with 128 GB of RAM, that is 23 million point reads per second across 15 million keys. The absolute number moves with the hardware. Repo, benchmark commands and whitepaper: Github. Questions and feedback welcome.