You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by "MonicaMagoniCom (via GitHub)" <gi...@apache.org> on 2024/02/13 16:12:00 UTC
[I] [Bug] Downgrade issue [pulsar]
MonicaMagoniCom opened a new issue, #22051:
URL: https://github.com/apache/pulsar/issues/22051
### Search before asking
- [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar.
### Version
I'm trying to downgrade from 3.0.2 to 2.10.4
### Minimal reproduce step
I'm using Kubernetes and I've deployed Pulsar with Helm. Pulsar works fine with 3.0.2 but I want to try the procedure to downgrade to check if it possible to go back to 2.10.4 (in case of emergency).
### What did you expect to see?
I expect that Pulsar downgrade works
### What did you see instead?
Bookkeepers cannot start and from the logs I see:
java.io.IOException: Error open RocksDB database
Caused by: org.rocksdb.RocksDBException: **unknown checksum type 4** in /pulsar/data/bookkeeper/ledgers/current/ledgers/000006.sst offset 929 size 32
### Anything else?
Zookeeper and brokers seem to run correctly
### Are you willing to submit a PR?
- [ ] I'm willing to submit a PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] Downgrade issue [pulsar]
Posted by "nicoloboschi (via GitHub)" <gi...@apache.org>.
nicoloboschi commented on issue #22051:
URL: https://github.com/apache/pulsar/issues/22051#issuecomment-1964171900
Hi @MonicaMagoniCom @lhotari
Even if we ensure that the bookies are still on BK 4.14, at some points user will have to perform the upgrade to newer Pulsar/BK versions.
IIUC the write format used by BK 4.15, is not compatible with BK 4.14 (I'm referring to server versions). I don't think this is acceptable neither for Pulsar and BookKeeper.
I believe @hangc0276 found the proper way to upgrade/downgrade without issues in https://github.com/apache/bookkeeper/pull/3793
From my understanding it should be done like this:
- before upgrading to 3.0, set ´dbStorage_rocksDB_checksum_type=kCRC32c´ in the bookies
- upgrade bookies -> now the bookies should continue writing in format `kCRC32c`, avoiding the new default `kXXH3`
- downgrade bookies should be fine since the format didn't change
But at the same time, BK uses two different formats for different purposes
- ledger index -> kCRC32c
- entry log location-> kxxHash
but the new setting would set both. Since they're different that would end up in a breaking change in one of the two.
So there's no really way at the moment to keep full compatibility
@hangc0276 Am I correct?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] Downgrade issue [pulsar]
Posted by "merlimat (via GitHub)" <gi...@apache.org>.
merlimat closed issue #22051: [Bug] Downgrade issue
URL: https://github.com/apache/pulsar/issues/22051
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] Downgrade issue [pulsar]
Posted by "MonicaMagoniCom (via GitHub)" <gi...@apache.org>.
MonicaMagoniCom commented on issue #22051:
URL: https://github.com/apache/pulsar/issues/22051#issuecomment-1943297221
It seems to be related to this issue:
https://github.com/apache/bookkeeper/issues/3734
As far as I understand it should be fixed. But in which version of Pulsar?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] Downgrade issue [pulsar]
Posted by "MonicaMagoniCom (via GitHub)" <gi...@apache.org>.
MonicaMagoniCom commented on issue #22051:
URL: https://github.com/apache/pulsar/issues/22051#issuecomment-1944183801
Is it possible to run all the components except for the bookies with image apachepulsar/pulsar-all:3.0.2 and the bookies with image apache/bookkeeper:4.14.8 through the Helm chart? Maybe I’m just saying a silly thing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] Downgrade issue [pulsar]
Posted by "MonicaMagoniCom (via GitHub)" <gi...@apache.org>.
MonicaMagoniCom commented on issue #22051:
URL: https://github.com/apache/pulsar/issues/22051#issuecomment-1959595546
With the image 2.10.6-SNAPSHOT, everything works fine :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] Downgrade issue [pulsar]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on issue #22051:
URL: https://github.com/apache/pulsar/issues/22051#issuecomment-1944241164
> Is it possible to run all the components except for the bookies with image apachepulsar/pulsar-all:3.0.2 and the bookies with image apache/bookkeeper:4.14.8 through the Helm chart? Maybe I’m just saying a silly thing.
the `apache/bookkeeper:4.14.8` image isn't compatible with the Helm chart. It might also be unusable since it wasn't until recently that I fixed the Dockerfiles in apache/bookkeeper.
It's better to create a custom docker image based on apache/pulsar's branch-2.10 instead (since 2.10.6 isn't available).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] Downgrade issue [pulsar]
Posted by "MonicaMagoniCom (via GitHub)" <gi...@apache.org>.
MonicaMagoniCom commented on issue #22051:
URL: https://github.com/apache/pulsar/issues/22051#issuecomment-1948872897
I just created the custom image. I will try it next week and let you know if it works correctly. Thank you
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] Downgrade issue [pulsar]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on issue #22051:
URL: https://github.com/apache/pulsar/issues/22051#issuecomment-1944119150
> It seems to be related to this issue: [apache/bookkeeper#3734](https://github.com/apache/bookkeeper/issues/3734)
>
> As far as I understand it should be fixed in Bookkeeper 4.14.8: [apache/bookkeeper#3947](https://github.com/apache/bookkeeper/pull/3947)
>
> So if I am right, I can't rollback from 3.0.2 to 2.10.4...
Yes, that's right. Thanks for pointing that out and sharing the issue and PR links.
The newest 2.10.x is 2.10.5, and it doesn't yet include the change. The upgrade to 4.14.8 / RocksDB 6.29.4.1 has been done in branch-2.10 (#21148), but there's no release. It would be helpful to bring this issue up for discussion in the dev@pulsar.apache.org mailing list thread https://lists.apache.org/thread/n5zfhohvk7olodfx2gksxk9n5f2pt3h8 . This could be a justification for 2.10.6 release.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] Downgrade issue [pulsar]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on issue #22051:
URL: https://github.com/apache/pulsar/issues/22051#issuecomment-1942114429
> Bookkeepers cannot start and from the logs I see:
> java.io.IOException: Error open RocksDB database
> Caused by: org.rocksdb.RocksDBException: unknown checksum type 4 in /pulsar/data/bookkeeper/ledgers/current/ledgers/000006.sst offset 929 size 32
Good question. This might be something to ask on user@bookkeeper.apache.org mailing list.
Pulsar 2.10.4 uses RocksDB 6.29.4.1 and Pulsar 3.0.2 uses RocksDB 7.9.2.
https://github.com/facebook/rocksdb/wiki/RocksDB-Compatibility-Between-Different-Releases
> Backward compatible: newer version of RocksDB should be able to open DBs generated by all previous releases for normal configuration.
It seems that RocksDB doesn't provide out-of-the-box tooling to migrate from newer version to the older one.
There would have to be a tool to export the data from the newer DB and import it with older DB version. I'm not aware of such tool for Bookkeeper.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] Downgrade issue [pulsar]
Posted by "merlimat (via GitHub)" <gi...@apache.org>.
merlimat commented on issue #22051:
URL: https://github.com/apache/pulsar/issues/22051#issuecomment-6376817448
Thanks @MonicaMagoniCom for the report and for confirming the workaround. The RocksDB "unknown checksum type 4" failure when downgrading from 3.0.x to 2.10.x was fixed by upgrading BookKeeper to 4.14.8 / RocksDB 6.29.4.1 in branch-2.10 (#21148, released in 2.10.6). #22988 also added an upgrade/downgrade integration test to CI to catch this kind of regression. Closing; please open a new issue if you hit a downgrade problem between currently supported releases.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org