You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@bookkeeper.apache.org by "merlimat (via GitHub)" <gi...@apache.org> on 2026/10/15 19:58:39 UTC
[PR] Partition index and filter blocks of the entry location RocksDB [bookkeeper]
merlimat opened a new pull request, #4909:
URL: https://github.com/apache/bookkeeper/pull/4909
### Problem
`DbLedgerStorage`'s entry location index (`KeyValueStorageRocksDB`, `DbConfigType.EntryLocation`) keeps index and
filter blocks in the block cache (`cache_index_and_filter_blocks=true`). It uses the default index type and one
full Bloom filter (10 bits/key) per SST, with a 64 MiB target file size. A full filter covers the whole file: about
8.7 MiB for a 64 MiB SST of `(ledgerId, entryId)` keys.
The block cache is `new LRUCache(size)`. RocksDB splits any cache of 32 MiB or more into 64 shards, and a block
lives in one shard. The default size is `maxDirectMemory / ledgerDirs / 10`, so with the default 2 GiB of direct
memory and one ledger directory it is 205 MiB and each shard holds 3.2 MiB. An 8.7 MiB filter block needs a shard
of at least 8.7 MiB, which means a cache of about 560 MiB per ledger directory.
In RocksDB 10.10.1's `LRUCacheShard`, inserting a block bigger than its shard first evicts every unreferenced
entry in the shard, then inserts the block anyway. `Release()` frees it immediately because the shard is over
capacity. The filter is not pinned, so every `Get()` on every overlapping SST:
- reads the whole multi-MiB filter again,
- verifies its checksum and parses it,
- and empties that shard of the cache.
With buffered I/O, BookKeeper's default, these reads come from the page cache and lookups become CPU-bound.
With direct I/O, or when the page cache is tight, they go to disk. `EntryLocationIndex.getLocation()` runs on
every read-cache miss of a bookie, so this limits bookie read throughput.
### Example
Current code (`KeyValueStorageRocksDB.initializeRocksDBWithBookieConf`):
```java
this.cache = new LRUCache(blockCacheSize); // 64 shards when >= 32 MiB
...
tableOptions.setFilterPolicy(new BloomFilter(bloomFilterBitsPerKey, false)); // one full filter per SST
tableOptions.setCacheIndexAndFilterBlocks(true); // read through the cache, not pinned
```
The benchmark drives `EntryLocationIndex` through the default bookie-conf path:
- 50M keys: 1,000 ledgers × 50,000 entries, written in sorted, flush-like batches.
- Natural leveled compaction left 10 SSTs: 2 in L0, 3 in L5, 5 in L6. Data is 459 MiB and the largest filter is 8.67 MiB.
- Random `getLocation()` from 4 threads: 5 s warm-up, then 20 s measured.
- Buffered I/O with all files in the page cache, on an Apple M5 Pro.
- Counters come from the RocksDB PerfContext.
| block cache (shard) | gets/s before → after | filter block reads per get | bytes read per get | CPU per get |
|---|---|---|---|---|
| 64 MiB (1 MiB) | 1,746 → 176,144 (×101) | 3.58 → 1.47 | 16.2 MB → 34 KB | 2,275 µs → 23 µs |
| 256 MiB (4 MiB) | 2,182 → 216,277 (×99) | 1.44 → 0.74 | 12.8 MB → 26 KB | 1,814 µs → 19 µs |
| 1 GiB (16 MiB) | 834,446 → 757,691 (−9%) | 0 → 0.06 | 2.4 KB → 1.8 KB | 4.7 µs → 5.2 µs |
- Before the change, at 64 and 256 MiB the reader threads copied 27 GB/s out of the page cache, almost all of it filter blocks.
- At 64 MiB the block cache held no filter block at the end; at 256 MiB it held one.
- With partitioning, filter reads are 4 KiB partitions, and 2,600 to 16,000 of them stay cached.
- A third run used the new code with partitioning disabled on the old DB: 2,226 gets/s, the same as before.
`getLastEntryInLedger()` (`getFloor`) doesn't use Bloom filters, so it doesn't change at 64 MiB: about 77K/s
before and after. With 256 MiB and 1 GiB, everything it touches is cached, and the extra top-level index lookup
makes it about 5% slower (271K → 258K/s and 275K → 260K/s).
### Change
```java
tableOptions.setCacheIndexAndFilterBlocks(true);
if (partitionedIndexAndFiltersEnabled) {
tableOptions.setIndexType(IndexType.kTwoLevelIndexSearch);
tableOptions.setPartitionFilters(true);
tableOptions.setPinTopLevelIndexAndFilter(true);
}
```
This is the setup RocksDB recommends when index and filter blocks live in the block cache
(https://github.com/facebook/rocksdb/wiki/Partitioned-Index-Filters):
- Index and filter blocks are split into partitions of about 4 KiB (`metadata_block_size`).
- Only the top-level index and filter blocks are pinned: about 47 KiB per 64 MiB SST in the benchmark, charged to the block cache.
- The partitions are cached like data blocks.
Other changes:
- The new setting `dbStorage_rocksDB_partitionedIndexAndFiltersEnabled` defaults to `true` and turns this off. It is documented in `bk_server.conf` and `reference/config.md`.
- `conf/entry_location_rocksdb.conf.default` and its test copy set `index_type=kTwoLevelIndexSearch`, `partition_filters=true` and `pin_top_level_index_and_filter=true`.
Compatibility:
- SSTs keep `format_version=5`.
- Partitioned indexes and filters are readable by RocksDB 7.9.2 (BookKeeper 4.17 / Pulsar 4.0), so downgrades still work. 10.10's default `decouple_partitioned_filters=true` is readable by every version that reads partitioned filters.
- Tested by reading the SSTs from the benchmark with rocksdbjni 7.9.2: checksums of all 10 SSTs verify, 100K gets of present keys and 100K of absent keys return correct results with the partitioned Bloom filters in use, and a full scan returns all 50M keys.
- Only SSTs written after the upgrade use the new layout; old and new SSTs can be read side by side. Existing SSTs are rewritten by normal compaction, or right away through `/api/v1/bookie/entry_location_compact` or `entryLocationCompactionInterval`.
Trade-off: when the whole index fits in the block cache with room to spare (the 1 GiB row), point gets are about
9% slower and floor lookups about 5% slower, because of the extra top-level lookup. Real location indexes are
usually much larger than the block cache.
### Testing
- New test `KeyValueStorageRocksDBTest#testEntryLocationPartitionedIndexAndFilters`:
- Writes and compacts an SST three ways: the default bookie-conf path, the same path with the setting off, and the shipped options file.
- Checks the index type with `SstFileReader` (index partitions) and the filter type from the SST's metaindex key (`partitionedfilter.` vs `fullfilter.`).
- Fails without the change, on both the bookie-conf and options-file paths.
- `org.apache.bookkeeper.bookie.storage.ldb` tests pass, except `DbLedgerStorageWithDirectEntryLoggerTest`. It fails locally because there is no native-io JNI library for macOS/aarch64; this change doesn't touch it.
- checkstyle, spotbugs and spotless are clean.
- The benchmark and the 7.9.2 read check above are scratch programs and are not committed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@bookkeeper.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Partition index and filter blocks of the entry location RocksDB [bookkeeper]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on code in PR #4909:
URL: https://github.com/apache/bookkeeper/pull/4909#discussion_r4288650441
##########
bookkeeper-server/src/test/java/org/apache/bookkeeper/bookie/storage/ldb/KeyValueStorageRocksDBTest.java:
##########
@@ -97,6 +99,47 @@ public void testCompatibleSstFormatWithAndWithoutConfigurationFiles() throws Exc
}
}
+ @Test
+ public void testEntryLocationPartitionedIndexAndFilters() throws Exception {
+ File directory = temporaryFolder.newFolder();
+ ServerConfiguration configuration = new ServerConfiguration();
+ configuration.setEntryLocationRocksdbConf(new File(directory, "entry.conf").toString());
+ assertEntryLocationSstLayout(configuration, true);
+
+ configuration.setProperty("dbStorage_rocksDB_partitionedIndexAndFiltersEnabled", false);
+ assertEntryLocationSstLayout(configuration, false);
+
+ configuration = new ServerConfiguration();
+ configuration.setEntryLocationRocksdbConf(
+ getClass().getClassLoader().getResource("conf/entry_location_rocksdb.conf").getPath());
+ assertEntryLocationSstLayout(configuration, true);
+ }
+
+ private void assertEntryLocationSstLayout(ServerConfiguration configuration, boolean partitioned)
+ throws Exception {
+ File directory = temporaryFolder.newFolder();
+ try (KeyValueStorageRocksDB storage = new KeyValueStorageRocksDB(
+ directory.toString(), "db", KeyValueStorageFactory.DbConfigType.EntryLocation, configuration)) {
+ storage.put(new byte[]{1}, new byte[]{2});
Review Comment:
**[QUALITY] layout test only exercises a single-partition SST**
Optional: with one key the SST has a single index/filter partition, so this checks that the layout is selected but not that lookups work across several partitions. Writing a few thousand keys (enough to span multiple 4 KiB metadata blocks), compacting, then reading some of them back (hits and misses) would cover the two-level search path too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@bookkeeper.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Partition index and filter blocks of the entry location RocksDB [bookkeeper]
Posted by "merlimat (via GitHub)" <gi...@apache.org>.
merlimat commented on code in PR #4909:
URL: https://github.com/apache/bookkeeper/pull/4909#discussion_r4288679956
##########
bookkeeper-server/src/test/java/org/apache/bookkeeper/bookie/storage/ldb/KeyValueStorageRocksDBTest.java:
##########
@@ -97,6 +99,47 @@ public void testCompatibleSstFormatWithAndWithoutConfigurationFiles() throws Exc
}
}
+ @Test
+ public void testEntryLocationPartitionedIndexAndFilters() throws Exception {
+ File directory = temporaryFolder.newFolder();
+ ServerConfiguration configuration = new ServerConfiguration();
+ configuration.setEntryLocationRocksdbConf(new File(directory, "entry.conf").toString());
+ assertEntryLocationSstLayout(configuration, true);
+
+ configuration.setProperty("dbStorage_rocksDB_partitionedIndexAndFiltersEnabled", false);
+ assertEntryLocationSstLayout(configuration, false);
+
+ configuration = new ServerConfiguration();
+ configuration.setEntryLocationRocksdbConf(
+ getClass().getClassLoader().getResource("conf/entry_location_rocksdb.conf").getPath());
+ assertEntryLocationSstLayout(configuration, true);
+ }
+
+ private void assertEntryLocationSstLayout(ServerConfiguration configuration, boolean partitioned)
+ throws Exception {
+ File directory = temporaryFolder.newFolder();
+ try (KeyValueStorageRocksDB storage = new KeyValueStorageRocksDB(
+ directory.toString(), "db", KeyValueStorageFactory.DbConfigType.EntryLocation, configuration)) {
+ storage.put(new byte[]{1}, new byte[]{2});
Review Comment:
Done in a6b6b4bd20: the test now writes 10 ledgers × 1000 entries and compacts them. With 256-byte data blocks the SST has 6 index partitions; the 12.4 KB filter spans about 4 partitions of 4 KiB. It then reads back every entry, entries that are absent, and each ledger's last entry via `getFloor`. All three cases are covered: default config, partitioning disabled, and the options file.
_🤖 Addressed by [Claude Code](https://claude.com/claude-code)_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@bookkeeper.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Partition index and filter blocks of the entry location RocksDB [bookkeeper]
Posted by "merlimat (via GitHub)" <gi...@apache.org>.
merlimat merged PR #4909:
URL: https://github.com/apache/bookkeeper/pull/4909
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@bookkeeper.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org