You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@iggy.apache.org by "ethanlin01x (via GitHub)" <gi...@apache.org> on 2026/08/13 14:41:18 UTC

[I] Python SDK: add a consumer argument to poll_messages [iggy]

ethanlin01x opened a new issue, #3876:
URL: https://github.com/apache/iggy/issues/3876

   ### Description
   
   `poll_messages` hardcodes `Consumer::default()` (foreign/python/src/client.rs:954), making Python does not let the caller name the consumer. That id resolves to numeric 0, so every Python process polling the same topic/partition shares one server-side offset slot and two independent consumers silently split the stream instead of each receiving it. It also makes consumer groups unreachable on this path, since `ConsumerKind::ConsumerGroup` cannot be selected at all.
   
   ### Affected area / component
   
   _No response_
   
   ### Proposed solution
   
   Add a `Consumer` type to the binding, modelled as a complex enum like the other sum types there (`PollingStrategy`, `AutoCommit`), with the id accepting `str | int`:
   
   ```rust
   pub enum Consumer {
       Single { id: PyIdentifier },
       Group { id: PyIdentifier },
   }
   ```
   
   Make `consumer` a required argument on `poll_messages`, and make `partition_id` optional in the same change: under the vsr feature a consumer-group poll only resolves the member's assignment when no partition is sent, and an omitted partition reads partition 0 for a regular consumer.
   
   ```python
   await client.poll_messages(
       stream="s", topic="t", partition_id=0,
       polling_strategy=PollingStrategy.Next(), count=10, auto_commit=True,
       consumer=Consumer.Single("my-app"),
   )
   
   await client.poll_messages(
       stream="s", topic="t",
       polling_strategy=PollingStrategy.Next(), count=10, auto_commit=True,
       consumer=Consumer.Group("my-group"),
   )
   ```
   
   Note: Requiring `consumer` breaks existing callers.
   
   
   ### Alternatives considered
   
   Keep `consumer` optional and default it to `Consumer::new(Identifier::numeric(1))`, matching the serde default the HTTP API already applies. That stays backwards compatible, but it silently moves an existing `next()` user's stored offset from slot 0 to slot 1 on upgrade, and Python would still be the only SDK picking a consumer for the caller.
   
   To align with other SDKs, which all require a consumer, I lean toward the breaking option: it fails at the call site instead of quietly relocating an offset.
   
   ### Contribution
   
   - [x] I'm willing to submit a pull request to implement this feature
   
   ### Good first issue
   
   - [ ] I think this could be a good first issue for a new contributor


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Python SDK: add a consumer argument to poll_messages [iggy]

Posted by "slbotbm (via GitHub)" <gi...@apache.org>.
slbotbm commented on issue #3876:
URL: https://github.com/apache/iggy/issues/3876#issuecomment-5281919865

   could you do this for the cpp sdj as well? 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Python SDK: add a consumer argument to poll_messages [iggy]

Posted by "hubcio (via GitHub)" <gi...@apache.org>.
hubcio closed issue #3876: Python SDK: add a consumer argument to poll_messages
URL: https://github.com/apache/iggy/issues/3876


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Python SDK: add a consumer argument to poll_messages [iggy]

Posted by "slbotbm (via GitHub)" <gi...@apache.org>.
slbotbm commented on issue #3876:
URL: https://github.com/apache/iggy/issues/3876#issuecomment-5306054356

   >  Is the SDK still early enough to fix the interface, or should I keep it backwards compatible and default the consumer instead?
   
   We can break compat here because of two reasons:
   1. The previous pattern was a wrong one, and I would like not like to provide compatibility for a wrong pattern.
   2. Compat is not guaranteed until 1.0.0


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Python SDK: add a consumer argument to poll_messages [iggy]

Posted by "ethanlin01x (via GitHub)" <gi...@apache.org>.
ethanlin01x commented on issue #3876:
URL: https://github.com/apache/iggy/issues/3876#issuecomment-5282084391

   @slbotbm while you're here - any thoughts on the breaking part? Making `consumer` required updates every existing Python caller. Is the SDK still early enough to fix the interface, or should I keep it backwards compatible and default the consumer instead?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Python SDK: add a consumer argument to poll_messages [iggy]

Posted by "ethanlin01x (via GitHub)" <gi...@apache.org>.
ethanlin01x commented on issue #3876:
URL: https://github.com/apache/iggy/issues/3876#issuecomment-5282063599

   @slbotbm 
   Sure - once the Python side is done, I'm happy to spend some time on the C++ SDK too.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org