[I] [Bug] Pulsar-function BK read-retry-loop with no backoff cause CPU exhaution [pulsar]

["vroyer (via GitHub)" <gi...@apache.org>]

vroyer opened a new issue, #24677:
URL: https://github.com/apache/pulsar/issues/24677

   ### Search before reporting
   
   - [x] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar.
   
   
   ### Read release policy
   
   - [x] I understand that [unsupported versions](https://pulsar.apache.org/contribute/release-policy/#supported-versions) don't get bug fixes. I will attempt to reproduce the issue on a supported version of Pulsar client and Pulsar broker.
   
   
   ### User environment
   
   We are running an HELM deployed pulsar cluster running on an on-prem kubernetes (lunastreaming 4.1.3.18, https://github.com/datastax/pulsar/tree/ls31_4.18 ) with 3 bookies + 3 brokers.
   
   
   
   
   ### Issue Description
   
   When a bookie is unavailable because of an k8s hardware issue (1 out of 3 bookies with quorum=2), the pulsar-function try to read some metadata from bookkeeper and the unavilable bookie cause a very expensive CPU read retry loop. As the result, the pulsar-function health liveness check fails and k8s kills the pulsar-function pod. Meanwhile, some pulsar connector pods cannot start properly.
   
   ### Error messages
   
   ```text
   
   ```
   
   ### Reproducing the issue
   
   This read-retry-loop with no backoff for an unavailable bookie seems to be a bookkeeper issue (bk version 4.16.7),  because there is even no BK read backoff setting to mitigate this kind of situation. It is useless to immediately retry reading from a bookie with this error: "Cannot resolve bookieId glpdlskub016:3181, bookie does not exist or it is not running"
   
   ### Additional information
   
   _No response_
   
   ### Are you willing to submit a PR?
   
   - [ ] I'm willing to submit a PR!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [Bug] Pulsar-function BK read-retry-loop with no backoff cause CPU exhaution [pulsar]

["1991santhu (via GitHub)" <gi...@apache.org>]

1991santhu commented on issue #24677:
URL: https://github.com/apache/pulsar/issues/24677#issuecomment-5174955165

   The retry cadence in the log is worth stating precisely, because it is exactly constant rather than merely fast. Taking the `PerChannelBookieClient` lines from the paste:
   
   ```
   07:47:02.260, 02.460, 02.660, 02.862, 03.061, 03.262, 03.461, 03.663
   ```
   
   That is 200ms ± 2ms, indefinitely — no backoff, no jitter, and no ceiling. So this is not "retries are too aggressive", it is that there is no backoff policy on this path at all.
   
   There is a second amplifier worth separating from the retry itself. `DefaultBookieAddressResolver.resolve()` (bookkeeper-server) performs a live registration-client lookup on **every** call:
   
   ```java
   BookieServiceInfo info = FutureUtils.result(registrationClient.getBookieServiceInfo(bookieId)).getValue();
   ...
   } catch (BKException.BKBookieHandleNotAvailableException ex) {
       ...
       throw new BookieIdNotResolvedException(bookieId, ex);
   ```
   
   There is no caching of the negative result. So each 200ms retry does not just re-attempt a connection — it re-queries the metadata store for a bookie that was already known-unresolvable 200ms ago. With a decommissioned or unregistered bookie this is a steady-state load on the registration backend for as long as the condition persists, which likely explains why the CPU cost is severe enough to fail the liveness probe rather than merely being noisy.
   
   Two things I could not determine and that probably decide where this should be fixed:
   
   1. **Which project owns it.** @vroyer's own note suggests BookKeeper, and both the retry loop and the resolver are in `bookkeeper-server` (4.16.7 here). If that is right, the fix — a negative-resolution cache with a short TTL, and/or a backoff on repeated `BookieIdNotResolvedException` — belongs in apache/bookkeeper rather than here, and this issue is more useful as a tracking issue than as the place a PR lands.
   
   2. **Whether it reproduces on Apache Pulsar.** The environment given is `lunastreaming 4.1.3.18`, a downstream distribution. Nothing in the stack trace looks fork-specific, but confirming it on a vanilla build would remove the easiest reason to close this without triage — and it has sat 0-comment for a while.
   
   Happy to open the corresponding BookKeeper issue with the above if a maintainer agrees that is the right home for it.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org