You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@bookkeeper.apache.org by Andrey Yegorov <an...@datastax.com> on 2023/02/08 16:44:27 UTC
Re: [Discuss] BookKeeper upgrade the RocksDB version to 7.9.x
+1 to the plan.
We can configure RocksDb to not use kXXH3 in all branches and provide a
config option for the user to enable it when needed.
On Sun, Jan 29, 2023 at 2:54 AM Hang Chen <ch...@apache.org> wrote:
> Hi guys,
> When we introduced `deleteRange` for ledger index deletion, we
> found a segment fault in ledger index deletion [1]. I suggest
> upgrading the RocksDB version to the latest 7.9.x to see if the
> RocksDB deleteRange segment fault bug has been fixed. I have tested
> upgrading the RocksDB version from 6.10.2 to 7.9.2 and rollback to
> 6.10.2. This is the testing result.
>
> - The upgrade process works fine in the following cases
> - Pulsar producer keeps producing messages to Pulsar topic
> - Pulsar consumer consumes messages from the Pulsar topic, and the
> Pulsar broker fetches messages from the BookKeeper cluster
> - Trigger compaction to cleanup expired ledgers
>
> - Rollback RocksDB version from 7.9.2 to 6.10.2, and bookie failed to
> startup with the following exception.
>
> ```java
> 2023-01-29T17:09:27,794+0800 [main] ERROR
> org.apache.bookkeeper.server.Main - Failed to build bookie server
> java.io.IOException: Error open RocksDB database
> at
> org.apache.bookkeeper.bookie.storage.ldb.KeyValueStorageRocksDB.<init>(KeyValueStorageRocksDB.java:200)
> ~[org.apache.bookkeeper-bookkeeper-server-4.14.6.jar:4.14.6]
> at
> org.apache.bookkeeper.bookie.storage.ldb.KeyValueStorageRocksDB.<init>(KeyValueStorageRocksDB.java:89)
> ~[org.apache.bookkeeper-bookkeeper-server-4.14.6.jar:4.14.6]
> at
> org.apache.bookkeeper.bookie.storage.ldb.KeyValueStorageRocksDB.lambda$static$0(KeyValueStorageRocksDB.java:63)
> ~[org.apache.bookkeeper-bookkeeper-server-4.14.6.jar:4.14.6]
> at
> org.apache.bookkeeper.bookie.storage.ldb.LedgerMetadataIndex.<init>(LedgerMetadataIndex.java:68)
> ~[org.apache.bookkeeper-bookkeeper-server-4.14.6.jar:4.14.6]
> at
> org.apache.bookkeeper.bookie.storage.ldb.SingleDirectoryDbLedgerStorage.<init>(SingleDirectoryDbLedgerStorage.java:170)
> ~[org.apache.bookkeeper-bookkeeper-server-4.14.6.jar:4.14.6]
> at
> org.apache.bookkeeper.bookie.storage.ldb.DbLedgerStorage.newSingleDirectoryDbLedgerStorage(DbLedgerStorage.java:150)
> ~[org.apache.bookkeeper-bookkeeper-server-4.14.6.jar:4.14.6]
> at
> org.apache.bookkeeper.bookie.storage.ldb.DbLedgerStorage.initialize(DbLedgerStorage.java:129)
> ~[org.apache.bookkeeper-bookkeeper-server-4.14.6.jar:4.14.6]
> at org.apache.bookkeeper.bookie.Bookie.<init>(Bookie.java:819)
> ~[org.apache.bookkeeper-bookkeeper-server-4.14.6.jar:4.14.6]
> at
> org.apache.bookkeeper.proto.BookieServer.newBookie(BookieServer.java:152)
> ~[org.apache.bookkeeper-bookkeeper-server-4.14.6.jar:4.14.6]
> at
> org.apache.bookkeeper.proto.BookieServer.<init>(BookieServer.java:120)
> ~[org.apache.bookkeeper-bookkeeper-server-4.14.6.jar:4.14.6]
> at
> org.apache.bookkeeper.server.service.BookieService.<init>(BookieService.java:52)
> ~[org.apache.bookkeeper-bookkeeper-server-4.14.6.jar:4.14.6]
> at
> org.apache.bookkeeper.server.Main.buildBookieServer(Main.java:304)
> ~[org.apache.bookkeeper-bookkeeper-server-4.14.6.jar:4.14.6]
> at org.apache.bookkeeper.server.Main.doMain(Main.java:226)
> ~[org.apache.bookkeeper-bookkeeper-server-4.14.6.jar:4.14.6]
> at org.apache.bookkeeper.server.Main.main(Main.java:208)
> ~[org.apache.bookkeeper-bookkeeper-server-4.14.6.jar:4.14.6]
> Caused by: org.rocksdb.RocksDBException: unknown checksum type 4 in
> data/bookkeeper/ledgers/current/ledgers/000025.sst offset 1078 size 33
> at org.rocksdb.RocksDB.open(Native Method)
> ~[org.rocksdb-rocksdbjni-6.10.2.jar:?]
> at org.rocksdb.RocksDB.open(RocksDB.java:239)
> ~[org.rocksdb-rocksdbjni-6.10.2.jar:?]
> at
> org.apache.bookkeeper.bookie.storage.ldb.KeyValueStorageRocksDB.<init>(KeyValueStorageRocksDB.java:197)
> ~[org.apache.bookkeeper-bookkeeper-server-4.14.6.jar:4.14.6]
> ... 13 more
> ```
> The root cause of this exception is that RocksDB 7.9.2 uses `kXXH3`
> by default, and `kXXH3` is only supported since RocksDB 6.27
>
> https://github.com/facebook/rocksdb/blob/79e57a39a33dbe17c8f51167e40e66d6c91f8eb4/include/rocksdb/table.h#L56
>
> For the BookKeeper master branch, we have upgraded the RocksDB to
> `6.29.4.1`, which can support RocksDB upgrade to 7.9.2 and rollback to
> 6.29.4.1.
> For the RocksDB < 6.27, we can push a fix to ensure RocksDB 7.9.2 does
> not use the latest checksum type `kXXH3`
>
> I suggest doing this upgradation. Do you have any concerns?
>
> [1] https://github.com/apache/bookkeeper/issues/3734
>
> Thanks,
> Hang
>
--
Andrey Yegorov
Re: [Discuss] BookKeeper upgrade the RocksDB version to 7.9.x
Posted by Enrico Olivelli <eo...@gmail.com>.
+1
Enrico
Il giorno mer 8 feb 2023 alle ore 17:44 Andrey Yegorov
<an...@datastax.com> ha scritto:
>
> +1 to the plan.
> We can configure RocksDb to not use kXXH3 in all branches and provide a
> config option for the user to enable it when needed.
>
> On Sun, Jan 29, 2023 at 2:54 AM Hang Chen <ch...@apache.org> wrote:
>
> > Hi guys,
> > When we introduced `deleteRange` for ledger index deletion, we
> > found a segment fault in ledger index deletion [1]. I suggest
> > upgrading the RocksDB version to the latest 7.9.x to see if the
> > RocksDB deleteRange segment fault bug has been fixed. I have tested
> > upgrading the RocksDB version from 6.10.2 to 7.9.2 and rollback to
> > 6.10.2. This is the testing result.
> >
> > - The upgrade process works fine in the following cases
> > - Pulsar producer keeps producing messages to Pulsar topic
> > - Pulsar consumer consumes messages from the Pulsar topic, and the
> > Pulsar broker fetches messages from the BookKeeper cluster
> > - Trigger compaction to cleanup expired ledgers
> >
> > - Rollback RocksDB version from 7.9.2 to 6.10.2, and bookie failed to
> > startup with the following exception.
> >
> > ```java
> > 2023-01-29T17:09:27,794+0800 [main] ERROR
> > org.apache.bookkeeper.server.Main - Failed to build bookie server
> > java.io.IOException: Error open RocksDB database
> > at
> > org.apache.bookkeeper.bookie.storage.ldb.KeyValueStorageRocksDB.<init>(KeyValueStorageRocksDB.java:200)
> > ~[org.apache.bookkeeper-bookkeeper-server-4.14.6.jar:4.14.6]
> > at
> > org.apache.bookkeeper.bookie.storage.ldb.KeyValueStorageRocksDB.<init>(KeyValueStorageRocksDB.java:89)
> > ~[org.apache.bookkeeper-bookkeeper-server-4.14.6.jar:4.14.6]
> > at
> > org.apache.bookkeeper.bookie.storage.ldb.KeyValueStorageRocksDB.lambda$static$0(KeyValueStorageRocksDB.java:63)
> > ~[org.apache.bookkeeper-bookkeeper-server-4.14.6.jar:4.14.6]
> > at
> > org.apache.bookkeeper.bookie.storage.ldb.LedgerMetadataIndex.<init>(LedgerMetadataIndex.java:68)
> > ~[org.apache.bookkeeper-bookkeeper-server-4.14.6.jar:4.14.6]
> > at
> > org.apache.bookkeeper.bookie.storage.ldb.SingleDirectoryDbLedgerStorage.<init>(SingleDirectoryDbLedgerStorage.java:170)
> > ~[org.apache.bookkeeper-bookkeeper-server-4.14.6.jar:4.14.6]
> > at
> > org.apache.bookkeeper.bookie.storage.ldb.DbLedgerStorage.newSingleDirectoryDbLedgerStorage(DbLedgerStorage.java:150)
> > ~[org.apache.bookkeeper-bookkeeper-server-4.14.6.jar:4.14.6]
> > at
> > org.apache.bookkeeper.bookie.storage.ldb.DbLedgerStorage.initialize(DbLedgerStorage.java:129)
> > ~[org.apache.bookkeeper-bookkeeper-server-4.14.6.jar:4.14.6]
> > at org.apache.bookkeeper.bookie.Bookie.<init>(Bookie.java:819)
> > ~[org.apache.bookkeeper-bookkeeper-server-4.14.6.jar:4.14.6]
> > at
> > org.apache.bookkeeper.proto.BookieServer.newBookie(BookieServer.java:152)
> > ~[org.apache.bookkeeper-bookkeeper-server-4.14.6.jar:4.14.6]
> > at
> > org.apache.bookkeeper.proto.BookieServer.<init>(BookieServer.java:120)
> > ~[org.apache.bookkeeper-bookkeeper-server-4.14.6.jar:4.14.6]
> > at
> > org.apache.bookkeeper.server.service.BookieService.<init>(BookieService.java:52)
> > ~[org.apache.bookkeeper-bookkeeper-server-4.14.6.jar:4.14.6]
> > at
> > org.apache.bookkeeper.server.Main.buildBookieServer(Main.java:304)
> > ~[org.apache.bookkeeper-bookkeeper-server-4.14.6.jar:4.14.6]
> > at org.apache.bookkeeper.server.Main.doMain(Main.java:226)
> > ~[org.apache.bookkeeper-bookkeeper-server-4.14.6.jar:4.14.6]
> > at org.apache.bookkeeper.server.Main.main(Main.java:208)
> > ~[org.apache.bookkeeper-bookkeeper-server-4.14.6.jar:4.14.6]
> > Caused by: org.rocksdb.RocksDBException: unknown checksum type 4 in
> > data/bookkeeper/ledgers/current/ledgers/000025.sst offset 1078 size 33
> > at org.rocksdb.RocksDB.open(Native Method)
> > ~[org.rocksdb-rocksdbjni-6.10.2.jar:?]
> > at org.rocksdb.RocksDB.open(RocksDB.java:239)
> > ~[org.rocksdb-rocksdbjni-6.10.2.jar:?]
> > at
> > org.apache.bookkeeper.bookie.storage.ldb.KeyValueStorageRocksDB.<init>(KeyValueStorageRocksDB.java:197)
> > ~[org.apache.bookkeeper-bookkeeper-server-4.14.6.jar:4.14.6]
> > ... 13 more
> > ```
> > The root cause of this exception is that RocksDB 7.9.2 uses `kXXH3`
> > by default, and `kXXH3` is only supported since RocksDB 6.27
> >
> > https://github.com/facebook/rocksdb/blob/79e57a39a33dbe17c8f51167e40e66d6c91f8eb4/include/rocksdb/table.h#L56
> >
> > For the BookKeeper master branch, we have upgraded the RocksDB to
> > `6.29.4.1`, which can support RocksDB upgrade to 7.9.2 and rollback to
> > 6.29.4.1.
> > For the RocksDB < 6.27, we can push a fix to ensure RocksDB 7.9.2 does
> > not use the latest checksum type `kXXH3`
> >
> > I suggest doing this upgradation. Do you have any concerns?
> >
> > [1] https://github.com/apache/bookkeeper/issues/3734
> >
> > Thanks,
> > Hang
> >
>
>
> --
> Andrey Yegorov