You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by "thetumbled (via GitHub)" <gi...@apache.org> on 2024/02/23 06:57:25 UTC
[I] [Bug] [Broker] Entry digest does not match [pulsar]
thetumbled opened a new issue, #22103:
URL: https://github.com/apache/pulsar/issues/22103
### Search before asking
- [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar.
### Version
2.9
### Minimal reproduce step
We use transactinal feature and start some tasks.
### What did you expect to see?
.
### What did you see instead?
One entry can't be read out because entry digest does not match, we have meet this problem serveral times, but the frequency of occurrency is pretty low.
The replication factor is `3 3 2`, so there are 3 copy of this entry. Further, i find that all of these three copy are same, which exclude the possibility of disk error. There must be something wrong with the code.
### Anything else?
_No response_
### Are you willing to submit a PR?
- [ ] I'm willing to submit a PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [Broker] Entry digest does not match [pulsar]
Posted by "merlimat (via GitHub)" <gi...@apache.org>.
merlimat commented on issue #22103:
URL: https://github.com/apache/pulsar/issues/22103#issuecomment-6381527768
Thanks for the report and the investigation, @thetumbled. This was seen on 2.9.x with cherry-picked patches and BookKeeper 4.14.1, which are no longer supported. Several fixes for this kind of corruption have landed since then, including https://github.com/apache/bookkeeper/pull/4196 (checksums over CompositeByteBuf) and the fix for #21933 (corrupt messages after producer SSL errors). If you still see entry digest mismatches on a supported release (4.0.x, 4.2.x or 5.0.x), please share the broker and BookKeeper versions, whether transactions/TLS/BrokerEntryMetadata are enabled, and broker logs around the write, especially any `IllegalReferenceCountException` or "Failed to release" messages. Without an update, this issue will be closed in a few weeks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [Broker] Entry digest does not match [pulsar]
Posted by "thetumbled (via GitHub)" <gi...@apache.org>.
thetumbled commented on issue #22103:
URL: https://github.com/apache/pulsar/issues/22103#issuecomment-1963497376
> > Yes, there is the bug need to be fixed by [apache/bookkeeper#4196](https://github.com/apache/bookkeeper/pull/4196). I have a quetion, which config is used to enable the tls feature between Pulsar & BookKeeper?
>
> @thetumbled One possible reference to using TLS between Pulsar Broker & BookKeeper is to check how the Apache Pulsar Helm Chart handles the TLS configuration.
>
> Regarding TLS, do you use TLS between clients and the broker? I guess that's the common configuration. I think that CompositeByteBufs could be created occasionally by Netty TLS handling also in that case, even without having BrokerEntryMetadata.
I add some log to detect whether the buffer is of type `CompositeByteBuf`:
```
if (unwrapped instanceof CompositeByteBuf) {
if (logger.isDebugEnabled()) {
logger.debug("test Digesting a composite buffer for ledger-id: {}, entry-id: {}",
ledgerId, entryId);
}
((CompositeByteBuf) unwrapped).forEach(this::update);
} else {
update(unwrapped);
}
```
And i seems not, we don't use a CompositeByteBuf object to calculate checksum, so the problem may not related with https://github.com/apache/bookkeeper/pull/4196.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [Broker] Entry digest does not match [pulsar]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on issue #22103:
URL: https://github.com/apache/pulsar/issues/22103#issuecomment-1960951910
This might be related to https://github.com/apache/bookkeeper/pull/4196 . Which BookKeeper version (client & server) do you have?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [Broker] Entry digest does not match [pulsar]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on issue #22103:
URL: https://github.com/apache/pulsar/issues/22103#issuecomment-1961293702
This might be related to #21421
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [Broker] Entry digest does not match [pulsar]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on issue #22103:
URL: https://github.com/apache/pulsar/issues/22103#issuecomment-1960954220
Do you have TLS enabled between Pulsar & BookKeeper? In the checksum calculation on the bookkeeper server side it has an impact since Netty's TLS implementation would use CompositeByteBufs in some cases.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [Broker] Entry digest does not match [pulsar]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on issue #22103:
URL: https://github.com/apache/pulsar/issues/22103#issuecomment-1963536992
@thetumbled Is there a specific reason to continue to use Pulsar 2.9.x ? The support for 2.9.x has ended a long time ago. Apache Pulsar PMC currently only supports officially 3.0.x and 3.2.x at the moment: https://pulsar.apache.org/contribute/release-policy/#supported-versions
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [Broker] Entry digest does not match [pulsar]
Posted by "thetumbled (via GitHub)" <gi...@apache.org>.
thetumbled commented on issue #22103:
URL: https://github.com/apache/pulsar/issues/22103#issuecomment-1963549021
> @thetumbled Is there a specific reason to continue to use Pulsar 2.9.x ? The support for 2.9.x has ended a long time ago. Apache Pulsar PMC currently only supports officially 3.0.x and 3.2.x at the moment: https://pulsar.apache.org/contribute/release-policy/#supported-versions
We upgrade to Pulsar 2.9.x in 2022, and don't upgrade the version frequently but cherry pick some usefully pr to fix the problem we meet.😂
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [Broker] Entry digest does not match [pulsar]
Posted by "thetumbled (via GitHub)" <gi...@apache.org>.
thetumbled commented on issue #22103:
URL: https://github.com/apache/pulsar/issues/22103#issuecomment-1960965294
> This might be related to [apache/bookkeeper#4196](https://github.com/apache/bookkeeper/pull/4196) . Which BookKeeper version (client & server) do you have?
version 4.14.1, we are study this pr too, but we are not sure the problem is related with it yet.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [Broker] Entry digest does not match [pulsar]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on issue #22103:
URL: https://github.com/apache/pulsar/issues/22103#issuecomment-1961030550
> > This might be related to [apache/bookkeeper#4196](https://github.com/apache/bookkeeper/pull/4196) . Which BookKeeper version (client & server) do you have?
>
> version 4.14.1, we are studying this pr too, but we are not sure the problem is related with it yet.
https://github.com/apache/bookkeeper/pull/2701 is included in 4.14.1 so there's a clear problem in checksum calculation in 4.14.1 . The problem is fixed by https://github.com/apache/bookkeeper/pull/4196 .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [Broker] Entry digest does not match [pulsar]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on issue #22103:
URL: https://github.com/apache/pulsar/issues/22103#issuecomment-1961295150
There's also #21933
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [Broker] Entry digest does not match [pulsar]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on issue #22103:
URL: https://github.com/apache/pulsar/issues/22103#issuecomment-1963553079
> > @thetumbled Is there a specific reason to continue to use Pulsar 2.9.x ? The support for 2.9.x has ended a long time ago. Apache Pulsar PMC currently only supports officially 3.0.x and 3.2.x at the moment: https://pulsar.apache.org/contribute/release-policy/#supported-versions
>
> We upgrade to Pulsar 2.9.x in 2022, and don't upgrade the version frequently but cherry pick some useful pr to fix the problem we meet.😂
@thetumbled is there a specific reason to follow this practice?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [Broker] Entry digest does not match [pulsar]
Posted by "thetumbled (via GitHub)" <gi...@apache.org>.
thetumbled commented on issue #22103:
URL: https://github.com/apache/pulsar/issues/22103#issuecomment-1961057586
> > > This might be related to [apache/bookkeeper#4196](https://github.com/apache/bookkeeper/pull/4196) . Which BookKeeper version (client & server) do you have?
> >
> >
> > version 4.14.1, we are studying this pr too, but we are not sure the problem is related with it yet.
>
> [apache/bookkeeper#2701](https://github.com/apache/bookkeeper/pull/2701) is included in 4.14.1 so there's a clear problem in checksum calculation in 4.14.1 . The problem is fixed by [apache/bookkeeper#4196](https://github.com/apache/bookkeeper/pull/4196) .
Yes, there is the bug fixed by https://github.com/apache/bookkeeper/pull/4196.
I have a quetion, which config is used to enable the tls feature between Pulsar & BookKeeper?
And i find that `org.apache.pulsar.common.protocol.Commands#addBrokerEntryMetadata` may transform the payload into `CompositeByteBuf`, i guess this is the trigger?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [Broker] Entry digest does not match [pulsar]
Posted by "thetumbled (via GitHub)" <gi...@apache.org>.
thetumbled commented on issue #22103:
URL: https://github.com/apache/pulsar/issues/22103#issuecomment-2034202011
We found that the corrupted entry is composed by two corrupted messages, https://github.com/apache/bookkeeper/pull/4196 may not be able to result into such phenomenon.
Do you have any other clues about this problem? thanks. @lhotari
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [Broker] Entry digest does not match [pulsar]
Posted by "thetumbled (via GitHub)" <gi...@apache.org>.
thetumbled commented on issue #22103:
URL: https://github.com/apache/pulsar/issues/22103#issuecomment-1963565008
> > > @thetumbled Is there a specific reason to continue to use Pulsar 2.9.x ? The support for 2.9.x has ended a long time ago. Apache Pulsar PMC currently only supports officially 3.0.x and 3.2.x at the moment: https://pulsar.apache.org/contribute/release-policy/#supported-versions
> >
> >
> > We upgrade to Pulsar 2.9.x in 2022, and don't upgrade the version frequently but cherry pick some useful pr to fix the problem we meet.😂
>
> @thetumbled is there a specific reason to follow this practice?
Upgrading pulsar version require lots of verifying work, there are multiple types of client and complcated production enviroment, we can't upgrade the version frequently. Some client may even use 2.8 version sdk.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [Broker] Entry digest does not match [pulsar]
Posted by "zhanglistar (via GitHub)" <gi...@apache.org>.
zhanglistar commented on issue #22103:
URL: https://github.com/apache/pulsar/issues/22103#issuecomment-2003168608
https://github.com/apache/bookkeeper/pull/4194
https://github.com/apache/bookkeeper/pull/4196
Note
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [Broker] Entry digest does not match [pulsar]
Posted by "thetumbled (via GitHub)" <gi...@apache.org>.
thetumbled commented on issue #22103:
URL: https://github.com/apache/pulsar/issues/22103#issuecomment-1960966016
> Do you have TLS enabled between Pulsar & BookKeeper? In the checksum calculation on the bookkeeper server side it has an impact since Netty's TLS implementation would use CompositeByteBufs in some cases.
No, we don't set any tls config.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [Broker] Entry digest does not match [pulsar]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on issue #22103:
URL: https://github.com/apache/pulsar/issues/22103#issuecomment-1961290076
> Yes, there is the bug need to be fixed by [apache/bookkeeper#4196](https://github.com/apache/bookkeeper/pull/4196). I have a quetion, which config is used to enable the tls feature between Pulsar & BookKeeper?
@thetumbled One possible reference to using TLS between Pulsar Broker & BookKeeper is to check how the Apache Pulsar Helm Chart handles the TLS configuration.
Regarding TLS, do you use TLS between clients and the broker? I guess that's the common configuration. I think that CompositeByteBufs could be created also in that case, even without having BrokerEntryMetadata.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [Broker] Entry digest does not match [pulsar]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on issue #22103:
URL: https://github.com/apache/pulsar/issues/22103#issuecomment-1961589216
A "double release" of a Netty ByteBuf or a thread safety issue could also cause such issues that the buffer's memory space is already used for other purposes causing the checksum calculation to fail. Netty ByteBufs aren't thread safe alone and it's possible that there could be an unsafe publication of the ByteBuf instance across thread boundaries (safe publication concept is explained [here](https://shipilev.net/blog/2014/safe-public-construction/#_safe_publication)).
This type of issues are hard to debug and it's very hard to reason about the correctness in the Pulsar code base.
Passing an mutable object instance across thread boundaries is fine as long as it happens with one of the methods listed in "safe publication" (which is a concept to make it easier to reason about Java Memory Model's "happens before" rules).
In the "double release" issue the ByteBuf instance would get released twice so that it's already in use by another thread when it is released the second time. There's `io.netty.util.ReferenceCountUtil#safeRelease(java.lang.Object)` which should be called "unsafe release". It hides the problems, although it does log issues.
In the case of "double release" issues there should be either "Failed to release a message" or IllegalReferenceCountException in logs at some point when the double release results in a failure.
So there's a high chance that checksum errors could happen even after the https://github.com/apache/bookkeeper/pull/4196 fix has been applied.
When looking into the "double release" problem, I did find bad code in the Pulsar code base that hides the problem:
https://github.com/apache/pulsar/blob/82237d3684fe506bcb6426b3b23f413422e6e4fb/pulsar-common/src/main/java/org/apache/pulsar/common/api/raw/ReferenceCountedMessageMetadata.java#L52
I have a draft PR to address the `setRefCnt(1)` and adding a solution to catch the double release bugs.
https://github.com/apache/pulsar/pull/22110
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [Broker] Entry digest does not match [pulsar]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on issue #22103:
URL: https://github.com/apache/pulsar/issues/22103#issuecomment-1963738805
> Upgrading pulsar version require lots of verifying work, there are multiple types of client and complcated production enviroment, we can't upgrade the version frequently. Some client may even use 2.8 version sdk.
Is there a way that Pulsar upgrades could be made easier? This is something that has been discussed on Pulsar Slack too.
It's not a sustainable solution that the Apache Pulsar project tries to support users on huge variations of software. It would be much more productive for everyone if the amount of configurations could be reduced so that we work on stabilizing and improving Pulsar.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [Broker] Entry digest does not match [pulsar]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on issue #22103:
URL: https://github.com/apache/pulsar/issues/22103#issuecomment-2034227824
> We found that the corrupted entry is composed by two corrupted messages, [apache/bookkeeper#4196](https://github.com/apache/bookkeeper/pull/4196) may not be able to result into such phenomenon. Do you have any other clues about this problem? thanks. @lhotari
@thetumbled I think @poorbarcode was looking into a similar problem recently and worked on some fixes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org