You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@iggy.apache.org by "bartoszkobylinski (via GitHub)" <gi...@apache.org> on 2026/07/24 22:11:19 UTC
[I] Python SDK: omitting `poll_interval` in `consumer_group()` polls with no delay — ~30% of a core on an idle topic, and nothing in Python documents it [iggy]
bartoszkobylinski opened a new issue, #3748:
URL: https://github.com/apache/iggy/issues/3748
**TL;DR** — Leaving `poll_interval` unset on `consumer_group()` makes the consumer re-poll as fast as the broker can answer. On a 1-vCPU host with an essentially idle topic that measured **~30% of the core** (iggy-server 17.3% + consumer 12.6%) and **~34,700 context switches/sec**; passing `poll_interval=250ms` took both processes below 0.3% with no functional change. This is my omission rather than an SDK bug — I'm filing it because two of my codebases made the identical omission independently, and because the consequence isn't discoverable from Python. Please re-triage as docs/enhancement if that fits better.
### Bug description
My worker created its consumer without passing `poll_interval`. `consume_messages(callback, shutdown_event)` takes no interval either, so `consumer_group()` is the only place it can be set. With it unset the SDK leaves it `None`, and the consumer polls continuously.
The cost, on a topic receiving a handful of messages per day:
| | unset | `poll_interval=250ms` |
|---|---|---|
| iggy-server | 17.3 % | < 0.3 % |
| consumer process | 12.6 % | < 0.3 % |
| context switches/sec | ~34,700 | ~2,300–3,900 |
| PSI `cpu some avg10` | 70.2 | 1.6 |
Stopping just the broker and the consumer took the whole machine to 100% idle, so this loop was effectively everything the host was doing. The server's own CPU tracks the client's poll rate — it drops below 0.3% while a consumer is still attached and polling at 250ms.
Three things made it hard to notice:
1. In `core/sdk/src/clients/consumer_builder.rs`, `polling_retry_interval` is given a default of `IggyDuration::ONE_SECOND`, while `polling_interval` is a bare `Option<IggyDuration>` with no default — seeing the first handled for you makes the second look handled too.
2. The Python stub types it `poll_interval: Optional[datetime.timedelta] = None`, and the complete docstring for `consumer_group` is *"Creates a new consumer group consumer. Returns the consumer or a PyRuntimeError on failure."* — a Python user cannot learn what omitting it does without reading the Rust source.
3. The only `poll_interval` example in the docs is `.poll_interval(IggyDuration::from_str("1ms")?)` — 1000 polls/sec — presented with no note that it's a throughput setting for dedicated hardware.
That two separate projects of mine, written months apart, both omitted it is what makes me think it's the shape of the API rather than one careless call site.
### Suggested direction
Either a small non-zero default for `polling_interval`, or (cheaper) one sentence in the Python `consumer_group` docstring plus a non-`1ms` example in the docs.
### One question, not a complaint
Does Iggy support or plan **server-side long-polling** — the broker holding a poll open until a message arrives or a timeout expires (Kafka's `fetch.max.wait.ms`, SQS long polling)? That makes an idle consumer nearly free regardless of what the client sets, and would remove this category at the source rather than documenting around it.
### Affected area / component
Python SDK (`foreign/python`), consumer configuration; `core/sdk/src/clients/consumer_builder.rs` defaults; docs (high-level SDK page).
### Deployment
Single `iggy-server` binary under systemd, TCP transport on loopback, one stream / one topic / one partition, one consumer group.
### Versions
Server `iggy-server` 0.8.0, Python SDK `apache-iggy` 0.8.0 (PyPI).
### Hardware / environment
Proxmox LXC container, **1 vCPU**, 4 GiB RAM, no swap, ~20 other services co-resident. Local disk.
### Sample code
Before (the omission):
```python
consumer = await client.consumer_group(
group, stream, topic,
polling_strategy=PollingStrategy.Next(),
auto_commit=AutoCommit.After(AutoCommitAfter.ConsumingEachMessage()),
)
await consumer.consume_messages(on_message, shutdown)
```
After:
```python
consumer = await client.consumer_group(
group, stream, topic,
polling_strategy=PollingStrategy.Next(),
auto_commit=AutoCommit.After(AutoCommitAfter.ConsumingEachMessage()),
poll_interval=timedelta(milliseconds=250),
)
```
### Reproduction
1. Run `iggy-server` and one Python consumer built as above, on a box where you can see per-process CPU clearly (a single core makes it obvious).
2. Publish nothing — leave the topic idle.
3. Observe `iggy-server` and the consumer process in `top`, and the context-switch rate in `vmstat 2`.
4. Add `poll_interval=timedelta(milliseconds=250)` and repeat.
On Linux, `cat /proc/pressure/cpu` is the clearest single signal — it went from `some avg10=70.2` to `some avg10=1.6` across that change.
### Note on framing
I want to be straightforward that this may be working as intended: Iggy is a throughput broker, and "set the interval, that's what it's for" is a fair answer. If that's the call, I'd suggest keeping only the documentation part — the Python docstring never mentions the consequence, and the sole published example is `1ms`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Python SDK: omitting `poll_interval` in `consumer_group()` polls with no delay — ~30% of a core on an idle topic, and nothing in Python documents it [iggy]
Posted by "hubcio (via GitHub) via commits" <co...@iggy.apache.org>.
hubcio commented on issue #3748:
URL: https://github.com/apache/iggy/issues/3748#issuecomment-6661075698
this is fixed because https://github.com/apache/iggy/pull/4683 and https://github.com/apache/iggy/pull/4629 are implemented.
@bartoszkobylinski check when you'll have a moment, perferably use code from origin/master because [0.10.0 release is under vote](https://github.com/apache/iggy/discussions/4746).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Python SDK: omitting `poll_interval` in `consumer_group()` polls with no delay — ~30% of a core on an idle topic, and nothing in Python documents it [iggy]
Posted by "hubcio (via GitHub) via commits" <co...@iggy.apache.org>.
hubcio closed issue #3748: Python SDK: omitting `poll_interval` in `consumer_group()` polls with no delay — ~30% of a core on an idle topic, and nothing in Python documents it
URL: https://github.com/apache/iggy/issues/3748
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Python SDK: omitting `poll_interval` in `consumer_group()` polls with no delay — ~30% of a core on an idle topic, and nothing in Python documents it [iggy]
Posted by "hubcio (via GitHub)" <gi...@apache.org>.
hubcio commented on issue #3748:
URL: https://github.com/apache/iggy/issues/3748#issuecomment-5163365931
@bartoszkobylinski this will be fixed when https://github.com/apache/iggy/discussions/2854 is implemented. @arunsingh fyi
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Python SDK: omitting `poll_interval` in `consumer_group()` polls with no delay — ~30% of a core on an idle topic, and nothing in Python documents it [iggy]
Posted by "bartoszkobylinski (via GitHub) via commits" <co...@iggy.apache.org>.
bartoszkobylinski commented on issue #3748:
URL: https://github.com/apache/iggy/issues/3748#issuecomment-6664335099
Thanks @hubcio! Great to see deferred polling implemented, and especially the grouped fetch improvements in #4683.
I'll validate this against `origin/master` using the same idle-consumer workload that originally exposed the issue in my Python application.
I'll compare CPU utilization, context switches and idle behavior against the original 0.8.0 measurements, and also check message delivery latency.
I noticed the consumer API and commit semantics have changed as well, so I'll test those separately before migrating the production application. I've tracked the migration in my project.
I'll report back with actual measurements. Thanks for following up on this!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Python SDK: omitting `poll_interval` in `consumer_group()` polls with no delay — ~30% of a core on an idle topic, and nothing in Python documents it [iggy]
Posted by "bartoszkobylinski (via GitHub) via commits" <co...@iggy.apache.org>.
bartoszkobylinski commented on issue #3748:
URL: https://github.com/apache/iggy/issues/3748#issuecomment-6665548763
Hi @hubcio — I've completed the verification. Thanks for pointing me to #4629 and #4683!
I reproduced the original busy-polling issue and tested the new deferred polling implementation against the same workload.
**Environment**
- GitHub Actions `ubuntu-latest` (Linux)
- Server container and Python client pinned to CPU 0
- 60 seconds idle, followed by 5 individual messages sent 3 seconds apart
- Servers: `apache/iggy:0.8.0` and `apache/iggy:edge` (reporting v0.10.0)
- Python SDK: `apache-iggy` 0.8.0 and 0.9.1.dev10
**Results**
| Configuration | Server CPU | Client CPU | Context switches/s | Delivery latency |
|---|---:|---:|---:|---|
| 0.8, no `poll_interval` | 35.4% | 31.0% | ~94,600 | 1, 1, 1, 1, 1 ms |
| 0.8, `poll_interval=250ms` | 0.07% | 0.09% | ~450 | 243, 9, 9, 11, 9 ms |
| 0.10, default polling | 0.57% | 0.03% | ~800 | 2, 1, 1, 1, 1 ms |
| 0.10, `max_wait=30s`, `Auto(Processing)` | 0.78% | 0.03% | ~690 | 2, 1, 1, 1, 1 ms |
The original busy loop was reproduced across three runs (server CPU 34–41%, client CPU 27–31%).
**The fix is confirmed for this workload.** The new idle consumer adds no measurable CPU overhead above the server's idle baseline (approximately 0.3–0.6%), while delivering messages in 1–2 ms in these tests.
Two additional observations:
1. **Old client / new server compatibility:** The 0.8.0 Python client against the 0.10.0 server appeared to hang instead of reporting an actionable incompatibility error. The test eventually timed out after 10 minutes, while the server repeatedly logged `consensus client disconnected`. An explicit protocol-version error would help diagnose partial upgrades.
2. **`PollOptions` without `max_wait`:** With an explicitly supplied `PollOptions(commit=Auto)` but no `max_wait`, one of five messages took approximately 1002 ms to arrive. This did not reproduce the busy loop. It may be worth clarifying the semantics of `max_wait=None` versus omitting `poll_options` entirely.
These are observations from a small verification run, not claims of separate confirmed regressions.
Thanks again for the fixes and for following up on the original report. This is a substantial improvement for idle Python consumers!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org