You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by "TakaHiR07 (via GitHub)" <gi...@apache.org> on 2024/09/23 03:17:17 UTC
[I] [Bug] Another issue result in broker lost bookie rack information in pulsar new version [pulsar]
TakaHiR07 opened a new issue, #23330:
URL: https://github.com/apache/pulsar/issues/23330
### Search before asking
- [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar.
### Read release policy
- [X] I understand that unsupported versions don't get bug fixes. I will attempt to reproduce the issue on a supported version of Pulsar client and Pulsar broker.
### Version
pulsar-3.0.6
### Minimal reproduce step
restart broker many times
### What did you expect to see?
when broker restart, can load rack information correctly
### What did you see instead?
This is a supplement of https://github.com/apache/pulsar/issues/23282. After fix the previous issue, we found another issue would cause broker lost rack information. When broker restart many times, some broker may generate the following log, illustrate that it has lost rack information.
```
11:52:24.768 [ForkJoinPool.commonPool-worker-1] INFO org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl - Initialize rackaware ensemble placement policy @ null @ Unknown : org.apache.pulsar.bookie.rackawareness.BookieRackAffinityMapping.
11:52:24.768 [ForkJoinPool.commonPool-worker-1] INFO org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl - Not weighted
11:52:24.768 [ForkJoinPool.commonPool-worker-1] INFO org.apache.bookkeeper.client.BookKeeper - Weighted ledger placement is not enabled
11:52:24.775 [metadata-store-33-1] INFO org.apache.pulsar.metadata.bookkeeper.PulsarRegistrationClient - Update BookieInfoCache (writable bookie) ip1:port1 -> BookieServiceInfo{properties={}, endpoints=[EndpointInfo{id=bookie, port=port1, host=ip1, protocol=bookie-rpc, auth=[], extensions=[]}]}
11:52:24.775 [metadata-store-33-1] INFO org.apache.pulsar.metadata.bookkeeper.PulsarRegistrationClient - Update BookieInfoCache (writable bookie) ip1:port1 -> BookieServiceInfo{properties={}, endpoints=[EndpointInfo{id=bookie, port=port1, host=ip1, protocol=bookie-rpc, auth=[], extensions=[]}]}
11:52:24.796 [pulsar-registration-client-58-1] WARN org.apache.bookkeeper.client.TopologyAwareEnsemblePlacementPolicy - Failed to resolve network location for ip1, using default rack for it : /default-region/default-rack.
11:52:24.796 [pulsar-registration-client-58-1] INFO org.apache.bookkeeper.net.NetworkTopologyImpl - Adding a new node: /default-region/default-rack/ip1:port1
11:52:24.797 [pulsar-registration-client-58-1] WARN org.apache.bookkeeper.client.TopologyAwareEnsemblePlacementPolicy - Failed to resolve network location for ip1, using default rack for it : /default-region/default-rack.
11:52:24.797 [pulsar-registration-client-58-1] INFO org.apache.bookkeeper.net.NetworkTopologyImpl - Adding a new node: /default-region/default-rack/ip1:port2
```
Actually, the root reason is the same as https://github.com/apache/pulsar/issues/23282, two listeners in registrationClient must be executed in a sync way.
But the code implementation is async, when we restart broker, we do listener registering in PulsarRegistrationClient#watchWritableBookies. Notice that this is an async method. First listener is register in BookieRackAffinityMapping#watchAvailableBookies, using async way. Second listener is register in BookieWatcherImpl#initialBlockingBookieRead, using sync way.
That is the reason. In PulsarRegistrationClient#watchWritableBookies, two listeners is add into writableBookiesWatchers in a sync way. But when execute `getWritableBookies().thenAcceptAsync(registrationListener::onBookiesChanged, executor)`, it is async.
It is easy to confirm this issue, just add some log in each part of PulsarRegistrationClient#watchWritableBookies. Since there are two listeners, We can see getWritableBookies() would be executed twice, in a async way.
https://github.com/apache/pulsar/blob/9012422bcbaac7b38820ce545cd5a3b4f8b586d0/pulsar-metadata/src/main/java/org/apache/pulsar/metadata/bookkeeper/PulsarRegistrationClient.java#L180-L187
### Anything else?
By the way, previous version do not have this issue. Because previous version use ZKRegistrationClient as implementation of RegistrationClient, but not PulsarRegistrationClient. After PIP-45, pulsar change the implementation into PulsarRegistrationClient. Then would cause the issue.
Because ZKRegistrationClient#watchWritableBookies can ensure two listeners executed in a sync way. while PulsarRegistrationClient#watchWritableBookies can not.
### Are you willing to submit a PR?
- [X] I'm willing to submit a PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] Another issue result in broker lost bookie rack information in pulsar new version [pulsar]
Posted by "TakaHiR07 (via GitHub)" <gi...@apache.org>.
TakaHiR07 commented on issue #23330:
URL: https://github.com/apache/pulsar/issues/23330#issuecomment-2367328050
> @TakaHiR07 Thanks for reporting these issues and submitting the pull requests. Would you mind starting a dev mailing list thread to raise the awareness of these issues and so that we find a resolution?
ok
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] Another issue result in broker lost bookie rack information in pulsar new version [pulsar]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on issue #23330:
URL: https://github.com/apache/pulsar/issues/23330#issuecomment-2367277439
@TakaHiR07 Thanks for reporting these issues and submitting the pull requests. Would you mind starting a dev mailing list thread to raise the awareness of these issues and so that we find a resolution?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] Another issue result in broker lost bookie rack information in pulsar new version [pulsar]
Posted by "merlimat (via GitHub)" <gi...@apache.org>.
merlimat commented on issue #23330:
URL: https://github.com/apache/pulsar/issues/23330#issuecomment-6376814953
Thanks @TakaHiR07 for the analysis and the fix. This was fixed by #23331, which makes PulsarRegistrationClient notify every registered writable/read-only bookie listener after the initial bookie read, so the rack-awareness listener no longer misses the initial bookie list (released in 3.0.10, 3.3.5, 4.0.3 and later, and in 4.1.0). Closing this; please reopen or open a new issue if it still happens on a currently supported release.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] Another issue result in broker lost bookie rack information in pulsar new version [pulsar]
Posted by "merlimat (via GitHub)" <gi...@apache.org>.
merlimat closed issue #23330: [Bug] Another issue result in broker lost bookie rack information in pulsar new version
URL: https://github.com/apache/pulsar/issues/23330
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org