You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by "eugene-polozhenkov (via GitHub)" <gi...@apache.org> on 2026/08/10 15:14:10 UTC
[I] Race conditions causing thread pool starvation and connection leaks on broker/proxy restarts [pulsar-dotpulsar]
eugene-polozhenkov opened a new issue, #291:
URL: https://github.com/apache/pulsar-dotpulsar/issues/291
### Description
Three related race conditions in `ConnectionPool` and `SubProducer` that surface together when a Pulsar broker (or proxy) restarts and all producers reconnect simultaneously.
This was discovered and reproduced while using a server(s) that have about 50K producers simultaneously.
---
### Bug 1 — `ConnectionPool.GetConnection`: concurrent connection storm
**Affected file:** `src/DotPulsar/Internal/ConnectionPool.cs`
When N producers all detect a disconnect at the same moment, every one of them calls `GetConnection`. The first cache check misses for all of them (the dead connection has just been removed), so all N fall through to `EstablishNewConnection` in parallel - spawning N `Connection` objects, N TCP handshakes, and N sets of background tasks simultaneously. With 50K producers this floods the ThreadPool, causing the CLR's hill-climbing algorithm to inject hundreds of additional threads (visible as a spike to the platform thread limit), which then cascade-faults the application.
**Expected:** only one TCP connection is established per broker URL, all other producers wait and reuse it. This applies to direct broker connections and via Pulsar proxy too
**Actual:** O(producers) connections are established concurrently.
---
### Bug 2 — `ConnectionPool.DisposeConnection`: key-only removal can evict a live connection
**Affected file:** `src/DotPulsar/Internal/ConnectionPool.cs`
`DisposeConnection` is called from a `ContinueWith` that fires when a connection leaves `Connected` state. The call is queued on the thread pool. Under load (e.g. while the 50K reconnect storm from Bug 1 is in progress), the thread pool may delay the `ContinueWith` long enough for a replacement connection to be established and stored at `_connections[url]` first. The original code:
```csharp
_connections.TryRemove(serviceUrl, out var _);
```
removes whatever is currently at the key — potentially the brand-new healthy connection rather than the dead one. The evicted connection is alive and connected but unreachable through the pool, so producers create yet another connection. The orphaned one lingers until its idle timer (60 s) or ping-pong timeout disposes it.
**Expected:** only the specific dead connection is removed from the cache.
**Actual:** whichever connection is at the key at that moment is removed.
### Bug 3 — SubProducer.EstablishNewChannel: field-capture race in Task.Run lambda
**Affected file**: src/DotPulsar/Internal/SubProducer.cs
```csharp
_dispatcherCts = new CancellationTokenSource();
_dispatcherTask = Task.Run(async () => await MessageDispatcher(_channel, _dispatcherCts.Token), CancellationToken.None);
```
The lambda closes over this (field, not a local var) and reads _dispatcherCts.Token lazily when it starts executing on the thread pool. If a concurrent or rapid-fire call to EstablishNewChannel (or DisposeAsync) cancels and disposes _dispatcherCts before the queued lambda starts, reading .Token on the disposed CancellationTokenSource throws ObjectDisposedException.
DotPulsar's default exception handler treats this as a fatal fault ->ProducerState.Faulted (a terminal state) -> the producer cache immediately disposes and recreates the producer → the new producer hits the same race if the broker is still unstable → tight recreation loop → each cycle adds more Task.Run items ->
thread pool flood.
Under a broker restart with 50K producers reconnecting simultaneously the race window widens to near-deterministic because the thread pool is already saturated, increasing the scheduling delay for each lambda.
**Expected:** no ObjectDisposedException; the dispatcher always binds to the CTS and channel that were live when it was started.
**Actual:** ObjectDisposedException, fatal producer fault, recreation death spiral.
Screen below shows the case when we sequentially restart all 3 brokers 3 times. Each time threads spiked up to 2000 and proxy had up to 5K connections.
<img width="2300" height="1640" alt="Image" src="https://github.com/user-attachments/assets/d597ea10-0a3b-4da5-87c7-d7378768f661" />
### Reproduction Steps
1. Get server with many running and connected producers (50K in our case)
2. Ungracefully (kubectl delete .... grace-period=0 --force) restart brokers or proxies sequentially (graceful restarts also works but rarely allows to reproduce the issue). This step simulates a real failure or network issue
### Expected behavior
1. No Thread Pool threads spike (no thread pool starvation)
2. Proxy/Broker connections count equals N Brokers + 1 at max
### Actual behavior
1. Thread Pool threads count spike up to ax 2000 (ThreadPool starvation)
2. Proxy/Broker connections count spikes up to 5000
### Regression?
No
### Known Workarounds
N/A
Note: can be slightly mitigated by adding more brokers to spread producers connections but issue still may happen if broker with too many connections becomes unavailable
### Configuration
- DotPulsar version: 5.3.1
- .NET: 10.0
- Pulsar: 4.0.11 with proxy in front of brokers (GKE)
- OS: Linux (container)
- Approximate producer count: 50 000
### Other information
I have a fix ready and am willing to submit a PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Race conditions causing thread pool starvation and connection leaks on broker/proxy restarts [pulsar-dotpulsar]
Posted by "eugene-polozhenkov (via GitHub)" <gi...@apache.org>.
eugene-polozhenkov commented on issue #291:
URL: https://github.com/apache/pulsar-dotpulsar/issues/291#issuecomment-5242783005
> Hi [@eugene-polozhenkov](https://github.com/eugene-polozhenkov) A PR would be most welcome :-)
It's on the way!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Race conditions causing thread pool starvation and connection leaks on broker/proxy restarts [pulsar-dotpulsar]
Posted by "blankensteiner (via GitHub)" <gi...@apache.org>.
blankensteiner closed issue #291: Race conditions causing thread pool starvation and connection leaks on broker/proxy restarts
URL: https://github.com/apache/pulsar-dotpulsar/issues/291
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Race conditions causing thread pool starvation and connection leaks on broker/proxy restarts [pulsar-dotpulsar]
Posted by "blankensteiner (via GitHub)" <gi...@apache.org>.
blankensteiner commented on issue #291:
URL: https://github.com/apache/pulsar-dotpulsar/issues/291#issuecomment-5242478104
Hi @eugene-polozhenkov
A PR would be most welcome :-)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org