Add SlateDB - #1490
Open
alexey-milovidov wants to merge 8 commits into
Open
Conversation
SlateDB is an embedded LSM-tree key-value store on object storage (the local filesystem here). One key-value pair per row; the 43 SQL queries run through embedded Apache DataFusion with a custom TableProvider doing parallel SlateDB range scans. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- glibc malloc fragmentation grew load RSS unboundedly (9.4 GB at
4 cores); jemalloc keeps it flat.
- v0.15.0's embedded compactor intermittently panics on large bulk
loads ("compaction source view not found in L0"); pin the upstream
fix (slatedb/slatedb#2002) until it lands in a release.
- Adapt to the post-0.15.0 write API (WriteHandle instead of
WriteOptions::await_durable).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Contributor
|
Results for Logs:
|
The background GC never deletes objects younger than 5 minutes, so a fast load reported ~72 GB in data-size while only ~24 GB were live SSTs. Run an immediate GC pass after close (6 s on the full dataset). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Contributor
|
Results for Logs:
|
The first end-of-load GC pass freed nothing: closing mid-compaction pins the collector's low watermark at the oldest pending compaction, and the 74 checkpoints left behind by compaction workers pin old manifests and every SST they reference. Wait for the compactor to drain, delete the checkpoints, then collect — data-size now matches live SSTs (15.8 GB instead of 71 GB), and queries scan 3 merged runs instead of dozens. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Contributor
|
Results for Logs:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds SlateDB — an embedded LSM-tree key-value store (Rust) that keeps all state in an object store; here the local filesystem via
object_store::LocalFileSystem.SlateDB has no query language, so as usual for key-value stores the storage layout and query execution live in the client harness (
slatedb/hits-slatedb/):queries.sqlas thedatafusionentry) through embedded Apache DataFusion with a customTableProviderwhose partitions are parallel SlateDB range scans, with projection pushdown at the row-decode level. Comparing against thedatafusionentry isolates what the KV storage layer costs.Validation: all 43 queries diffed against DataFusion reading
hits.parquetdirectly, at full 100M-row scale — identical results modulo float last-digit jitter and tie order underLIMIT(sort-key sequences verified equal).Notes:
🤖 Generated with Claude Code