Our Technology

Speedb Data Engine removes the capacity, scale, and performance limitations of existing KVS solutions.

Leveraging our vast experience and expertise in the storage and data management domain, we redesigned the RocksDB internal data structure to fit the performance and scalability requirements of modern, hyperscale data operations.

By redesigning key parts of RocksDB, we were able to develop a first-of-a-kind embedded KVS that outperforms RocksDB. Speedb supports petabyte scaling of datasets with billions of objects while maintaining high performance, and low hardware requirements.

Ask us anything

LSM

Speedb’s Log-Structured Merge (LSM)-based KVS introduces a new architecture that supports petabyte scaling of datasets with billions of objects while maintaining high performance and low hardware requirements.

The Speedb Data Engine is based on a revolutionary compaction method that reduces the write amplification factor (WAF) from ~30 to ~5.

As a result, Speedb eliminates processing latency issues and throughput drops, which are frequently encountered when using traditional LSM-trees, while signficantly reducing CPU utilization and memory consumption.

By redesigning the RocksDB I/O and job schedulers we were able to further improve performance stability, and reduce stalls. We have also redesigned RocksDB’s flow control mechanism to eliminate spikes in user latency.

Based on these technological breakthroughs, Speedb supports unprecedently fast writes even on large datasets while keeping a B-Tree like read performance.

Probabilistic Index

Speedb features an innovative indexing method that supports the storing of index together with data for extreme performance at scale.

Our probabilistic index consumes less than 3 bytes per object, regardless of object and key size. This way, indexes for large datasets can reside in DRAM, allowing for the mapping of hundreds of billions of objects while ensuring nearly one media access per read. To accelerate search time, the Speedb index uses a hierarchical data map.

Enterprise-Grade Capabilities

On top of Speedb’s core, we developed additional capabilities to make our data engine suitable for enterprise use.

For example, we redesigned RocksDB’s flow control mechanism to eliminate spikes in user latency and prevent stalling. Real-time monitoring and adaptive auto-tuning of system parameters allow for ensuring high performance for any workload. We also offer advanced reporting capabilities that turn RocksDB log data into actionable information, as well as enterprise-grade support and customization for specific use-cases.