[I] [Bug] V5 async producer applies no backpressure: the client OOMs with its memory limit at 44% of budget [pulsar]

["lhotari (via GitHub)" <gi...@apache.org>]

lhotari opened a new issue, #26470:
URL: https://github.com/apache/pulsar/issues/26470

   ### Search before reporting
   
   - [x] I searched the [issues](https://github.com/apache/pulsar/issues) and found nothing similar. Two are adjacent and are **not** duplicates of this one — see *Relationship to #26344 and #26343* below.
   
   ### Read release policy
   
   - [x] I understand that unsupported versions don't get bug fixes. I will attempt to reproduce the issue on a supported version of Pulsar client and Pulsar broker.
   
   ### User environment
   
   `pulsar-client-v5` on **master** and **branch-5.0-M1**. Not applicable to 4.x — the V5 client does not exist on branch-4.2 or earlier.
   
   ### Issue Description
   
   **A V5 async producer never pushes back on its caller.** `AsyncMessageBuilder.send()` appends a link to the target segment's `CompletableFuture` dispatch chain and returns. The client memory limit — the only backpressure the V5 API has, since it exposes no pending-message knob — is enforced by the v4 `ProducerImpl` the message reaches *later*, on a different thread. Appending to the chain is a pure allocation that always succeeds, so an application that does not await the returned futures feels nothing at all, and buffers until the heap is gone.
   
   This is **not confined to the window while a segment producer is being created**. Once the chain has fallen behind by a single link, every subsequent `send()` is again a pure append — the caller stays decoupled from all downstream backpressure for as long as the chain is behind, which under sustained load is indefinitely. The creation window is only how the chain first falls behind.
   
   **A second, independent cause makes the limit ineffective even when a message does reach the v4 producer.** The limit is charged `uncompressedSize` — the payload alone. The client actually retains roughly **2 KiB per in-flight unbatched send**: `MessageMetadata` (336 B), `MessageImpl` (152 B), `OpSendMsg` (152 B), the payload buffers, the send callbacks, and the ~6 `CompletableFuture` stages that carry the result back. For a small message that is more than an order of magnitude optimistic — a 200 MiB limit authorises ~1.6M pending 128-byte messages, several GiB of heap.
   
   Either cause alone is enough to OOM a client. Together they mean a correctly-configured memory limit provides no protection whatsoever.
   
   #### Evidence
   
   From a heap dump of the failure below (2 GiB heap, `-ml 200M`, 128-byte messages, batching off):
   
   | | |
   |---|---|
   | `MemoryLimitController.memoryLimit` | 209,715,200 |
   | `MemoryLimitController.currentUsage` | 92,335,360 — **44% of budget when the heap died** |
   | Async sends outstanding (`inFlightSends`) | 1,285,311 |
   | …that had reached the v4 producer (`pendingMessages`) | 721,369 |
   | …still queued as un-dispatched chain links | 563,942 |
   | `ProducerImpl.lastSequenceIdPublished` | **-1 — not one message acknowledged** |
   
   The application thread threw the `OutOfMemoryError` *inside* `ScalableTopicProducer.sendInternalAsync` — it had never waited and had never been refused.
   
   And the limit could not have saved it even in principle: had the whole chain drained, 1,285,311 × 128 B = 164 MiB, still under the 200 MiB limit.
   
   ### Error messages / stacktraces
   
   ```
   java.lang.OutOfMemoryError: Java heap space
   Dumping heap to /testoutput/perf-produce-oom-1788561786215.hprof ...
   Heap dump file created [2374932390 bytes in 5.450 secs]
   Terminating due to java.lang.OutOfMemoryError: Java heap space
   ```
   
   The producer log shows the failure arriving immediately after the producer connects, with no send ever completing:
   
   ```
   ... ProducerImpl - Created producer {producerId=0, topic=segment://public/default/profiletest-.../0000-ffff-0}
   java.lang.OutOfMemoryError: Java heap space
   ```
   
   ### Reproducing the issue
   
   **1. The integration test (what I used).** After #26468 the profiling harness drives a V5 producer:
   
   ```
   ./gradlew :tests:integration:profilingIntegrationTest
   ```
   
   Wait ~1 minute for the containers, then watch `tests/integration/build/pulsar-profiling/`. A `perf-produce-oom-*.hprof` appears within a second or two of the producer connecting.
   
   **2. `pulsar-perf` directly**, which is what that harness runs — a V5 async producer with `blockIfQueueFull(true)`:
   
   ```
   PULSAR_MEM="-Xmx2g -XX:+HeapDumpOnOutOfMemoryError" \
   bin/pulsar-perf produce topic://public/default/repro \
     -u pulsar://<broker>:6650 \
     -r 2147483647 -s 128 -db -m 100000000 -ml 200M
   ```
   
   `-r 2147483647` removes the rate limit, so the producer runs as fast as it can queue — which is the point: nothing downstream slows it down.
   
   **3. Not scalable-topic-specific.** The same run against a plain `persistent://` topic reproduces identically; the V5 client drives regular topics through the same `ScalableTopicProducer` over a synthetic single-segment layout.
   
   **4. The shape, in code** — an async send loop that does not await the futures, which is all it takes:
   
   ```java
   PulsarClient v5 = PulsarClient.builder()
           .serviceUrl("pulsar://localhost:6650")
           .memoryLimit(MemorySize.ofMegabytes(200))
           .build();
   
   Producer<byte[]> p = v5.newProducer(Schema.bytes())
           .topic("persistent://public/default/repro")
           .batchingPolicy(BatchingPolicy.ofDisabled())
           .create();
   
   byte[] payload = new byte[128];
   AsyncProducer<byte[]> async = p.async();
   while (true) {
       async.newMessage().value(payload).send();   // never blocks, never fails, never slows down
   }
   ```
   
   Run with a small heap (`-Xmx2g`). Note the memory limit is *configured* — the point is that it does not engage.
   
   ### Additional information
   
   #### Relationship to #26344 and #26343
   
   - **#26344** — *V5 producer can self-deadlock a Netty IO thread.* That issue is about the dispatch chain running sends on the connection's event loop, and it notes as a secondary point that "while a segment producer is being created, V5 applies no admission control at all." What it does not cover, and what this issue is about: the missing admission control is **not limited to that window**; and the memory-limit accounting shortfall, which makes the limit ineffective even after the chain catches up. The two failure modes also differ in practice — #26344 predicts a deadlock, whereas what a real workload hits first is this OOM, with the limiter still at 44% of budget.
   - **#26343** — *Producer send admission control blocks Pulsar's own internal threads, and there is no non-blocking backpressure mode.* That is the v4-level design question, and it constrains what a fix here should look like.
   
   They overlap enough that a single change may well close more than one of them; they are filed separately because the defects are distinct.
   
   #### On the fix
   
   Deliberately not proposing a design here. Worth recording as a constraint: the right direction is likely the non-blocking one from #26343 — the caller delays its own sends rather than parking a thread, and does not pull in more work than it can account for — rather than simply moving a blocking acquire earlier in the path. The hard part is bounding what is already queued without blocking. That belongs in #26343 or a PIP, not in this bug report.
   
   ### Are you willing to submit a PR?
   
   - [ ] I'm willing to submit a PR!
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [Bug] V5 async producer applies no backpressure: the client OOMs with its memory limit at 44% of budget [pulsar]

["merlimat (via GitHub)" <gi...@apache.org>]

merlimat closed issue #26470: [Bug] V5 async producer applies no backpressure: the client OOMs with its memory limit at 44% of budget
URL: https://github.com/apache/pulsar/issues/26470


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org