You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@iggy.apache.org by "TomPlanche (via GitHub)" <gi...@apache.org> on 2026/09/29 00:13:26 UTC
[I] bug(Rust SDK): TCP, QUIC and WebSocket clients block after 1000 diagnostic events [iggy]
TomPlanche opened a new issue, #4327:
URL: https://github.com/apache/iggy/issues/4327
### Bug description
The TCP, QUIC and WebSocket clients of the Rust SDK stop working after 1000 diagnostic events. The 1001st call to `publish_event` blocks forever, and the `connect()`, `disconnect()` or `shutdown()` call that publishes this event never returns. @hubcio found this problem in the review of #4288. I checked it against the code on master at commit `9aff4424a`, and a program reproduces it with the SDK at commit `5d8129e95`.
Each client creates a bounded `async_broadcast` channel with a capacity of 1000, and keeps the original receiver in its `events` field:
- [TCP](https://github.com/apache/iggy/blob/9aff4424a7908a0fdf8a16c1f39e68dd061c1729/core/sdk/src/tcp/tcp_client.rs#L574), [QUIC](https://github.com/apache/iggy/blob/9aff4424a7908a0fdf8a16c1f39e68dd061c1729/core/sdk/src/quic/quic_client.rs#L590) and [WebSocket](https://github.com/apache/iggy/blob/9aff4424a7908a0fdf8a16c1f39e68dd061c1729/core/sdk/src/websocket/websocket_client.rs#L533) call `broadcast(1000)`.
- `subscribe_events()` returns a clone of this receiver, but nothing reads the original receiver.
- [`publish_event`](https://github.com/apache/iggy/blob/9aff4424a7908a0fdf8a16c1f39e68dd061c1729/core/sdk/src/tcp/tcp_client.rs#L216-L220) calls `self.events.0.broadcast(event).await`. Overflow mode is off, thus `broadcast` waits for a free slot when the channel is full.
The channel frees the slot of a message only when every active receiver has received it. The original receiver never receives a message, thus the channel is full after 1000 events and stays full. A subscriber that reads from `subscribe_events()` does not free the slots.
The TCP client publishes `Connected` in each connect and `Disconnected` in each disconnect. For a client without auto-login, the 1001st event is the `Connected` event of the 501st connect. A long-running client that loses its connection often can reach this limit, and then its reconnect blocks forever.
- Expected: a client publishes any number of diagnostic events, and `connect()` and `disconnect()` always return.
- Actual: the 501st `connect()` of a client without auto-login never returns.
The HTTP client also creates this channel, but it does not publish events, thus it does not block.
### Possible fix
@hubcio suggests `set_overflow(true)` on the sender. In overflow mode, `broadcast` drops the oldest message when the channel is full and does not wait. A subscriber that reads too slowly then loses old events, but no client blocks.
### Affected area / component
Rust SDK
### Deployment
Compiled from source
### Versions
- Code checked: master at commit `9aff4424a7908a0fdf8a16c1f39e68dd061c1729`
- Reproduction: Rust SDK at commit `5d8129e95de1d1fdf2ed42af015340b82af1c59a`, the commit that the `repros/sdk-lifecycle` crate uses
### Sample code
The program is [`event_channel_blocks.rs`](https://github.com/TomPlanche/iggy/blob/tomplanche/repros/repros/sdk-lifecycle/src/bin/event_channel_blocks.rs) in [`repros/sdk-lifecycle`](https://github.com/TomPlanche/iggy/tree/tomplanche/repros/repros/sdk-lifecycle) on my fork. The crate uses `iggy` from `apache/iggy` at commit `5d8129e95`. The program connects and disconnects a TCP client without auto-login, up to 600 times. It sets `reestablish_after` to zero so that each connect starts at once, and it stops at the first `connect()` or `disconnect()` that takes more than 5 s.
### Logs
```text
$ cargo run --bin event_channel_blocks
connect 501 blocked
```
### Iggy server config
_No response_
### Reproduction
1. From the root of an `apache/iggy` checkout on Linux, start a server with the default root credentials:
```bash
cargo run --bin iggy-server -- --with-default-root-credentials --fresh
```
2. Clone the `tomplanche/repros` branch of my fork, then go to `repros/sdk-lifecycle`.
3. Run `cargo run --bin event_channel_blocks`. The program shows `connect 501 blocked`.
### Contribution
- [x] I'm willing to submit a pull request to fix this bug
### Good first issue
- [ ] I think this could be a good first issue for a new contributor
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] bug(Rust SDK): TCP, QUIC and WebSocket clients block after 1000 diagnostic events [iggy]
Posted by "BrandonArp (via GitHub)" <gi...@apache.org>.
BrandonArp commented on issue #4327:
URL: https://github.com/apache/iggy/issues/4327#issuecomment-6071748771
We hit this in a staging environment, and the trigger is broader than connect/disconnect cycles. A client whose server is down for a while wedges permanently, even if the application never calls `connect()` or `disconnect()` itself.
Every failed reconnect sweep ends in `fail_connect()`, which publishes `DiagnosticEvent::Disconnected`. A client that keeps trying to reconnect (from the heartbeat, or from any request on a disconnected client, which goes through `send_raw_with_response` -> `connect()`) adds one event per failed sweep. After the 1000th, the next `fail_connect()` blocks in `publish_event`.
This made it hard to diagnose. The blocked connect never releases the client's `ConnectCoordinator` (or its `routing_lock`, when a request triggered the reconnect). So every later call on that client, `get_stats()` for example, waits silently behind it. No socket is open, nothing is logged, and the client stays wedged after the server comes back. Only a new client recovers.
What happened to us: the server (0.9.0) was down for about 2.5 hours (full disk, crash loop). Two replicas of a service each held a long-lived `IggyClient` (SDK 0.11.0, `reconnection_retries=3`, `reconnection_interval=1s`). They wedged within 30s of each other, about 2.5 hours into the failed reconnects. Their health loop probes with `get_stats()` and hung there, so both stayed not-ready after the server recovered, until we restarted them. Fresh processes connected in about 3 s.
Here is a minimal repro that needs no server, because it dials a closed port (SDK 0.11.0, async-broadcast 0.7.2):
```rust
use iggy::prelude::*;
use std::time::Duration;
#[tokio::main]
async fn main() {
let cs = "iggy://iggy:iggy@127.0.0.1:1?reconnection_retries=0&reconnection_interval=1ms";
let client = IggyClientBuilder::from_connection_string(cs).unwrap().build().unwrap();
for i in 1..=1100u32 {
if tokio::time::timeout(Duration::from_secs(5), client.connect()).await.is_err() {
println!("connect #{i} hung");
let stats = tokio::time::timeout(Duration::from_secs(5), client.get_stats()).await;
println!("get_stats after that: {}", if stats.is_err() { "hung" } else { "returned" });
return;
}
}
}
```
Output: `connect #1001 hung`, then `get_stats after that: hung`. Connects #1 to #1000 each return `CannotEstablishConnection`. As the issue says, draining a `subscribe_events()` receiver in a spawned task doesn't change the result.
+1 to `set_overflow(true)`. Until a release includes it, we're running 0.11.0 with that one-line change in the TCP, QUIC and WebSocket clients. Is a fix planned for the next release?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] bug(Rust SDK): TCP, QUIC and WebSocket clients block after 1000 diagnostic events [iggy]
Posted by "hubcio (via GitHub)" <gi...@apache.org>.
hubcio closed issue #4327: bug(Rust SDK): TCP, QUIC and WebSocket clients block after 1000 diagnostic events
URL: https://github.com/apache/iggy/issues/4327
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org