You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@iggy.apache.org by "ryerraguntla (via GitHub)" <gi...@apache.org> on 2026/06/23 00:24:29 UTC

[I] feat(gateways): Kafka OffsetCommit and OffsetFetch with persisted consumer offsets [iggy]

ryerraguntla opened a new issue, #3542:
URL: https://github.com/apache/iggy/issues/3542

   ## Summary
   
   Implement **OffsetCommit (8)** and **OffsetFetch (9)** with durable storage per #3540 design.
   
   ## Motivation
   
   Phase 2 success metrics:
   - Offsets persisted across bridge restarts
   - `kafka-console-consumer.sh --group` commits and resumes
   
   ## Depends on
   
   - #3540 — persistence design
   - #3541 — consumer group coordinator
   
   ## Scope
   
   - [ ] OffsetCommit: store `(group, topic, partition) → offset`
   - [ ] OffsetFetch: return committed offsets on JoinGroup / consumer start
   - [ ] Restart test: commit offset, restart bridge, consumer resumes
   - [ ] Stub OffsetFetch for unknown group returns empty or appropriate error
   
   ## Acceptance criteria
   
   - [ ] Consumer group resumes from last committed offset after bridge restart (Option A or B)
   - [ ] At-least-once on failure still documented
   
   ## References
   
   - Rollout plan: https://github.com/apache/iggy/discussions/3253#discussioncomment-17181030


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Kafka gateway: Kafka OffsetCommit and OffsetFetch with persisted consumer offsets [iggy]

Posted by "krishvishal (via GitHub)" <gi...@apache.org>.
krishvishal commented on issue #3542:
URL: https://github.com/apache/iggy/issues/3542#issuecomment-5988954033

   @justinmclean this PR https://github.com/apache/iggy/pull/4392 should address the above concern. Let me know if this works. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Kafka gateway: Kafka OffsetCommit and OffsetFetch with persisted consumer offsets [iggy]

Posted by "krishvishal (via GitHub)" <gi...@apache.org>.
krishvishal closed issue #3542: Kafka gateway: Kafka OffsetCommit and OffsetFetch with persisted consumer offsets
URL: https://github.com/apache/iggy/issues/3542


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Kafka gateway: Kafka OffsetCommit and OffsetFetch with persisted consumer offsets [iggy]

Posted by "krishvishal (via GitHub)" <gi...@apache.org>.
krishvishal commented on issue #3542:
URL: https://github.com/apache/iggy/issues/3542#issuecomment-5988691634

   I'm taking this up.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Kafka gateway: Kafka OffsetCommit and OffsetFetch with persisted consumer offsets [iggy]

Posted by "krishvishal (via GitHub)" <gi...@apache.org>.
krishvishal commented on issue #3542:
URL: https://github.com/apache/iggy/issues/3542#issuecomment-5908577664

   I am picking this up.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Kafka gateway: Kafka OffsetCommit and OffsetFetch with persisted consumer offsets [iggy]

Posted by "justinmclean (via GitHub)" <gi...@apache.org>.
justinmclean commented on issue #3542:
URL: https://github.com/apache/iggy/issues/3542#issuecomment-5946870699

   The offset design in OFFSET_STORAGE.md won't work against the current server. I assume this is this allready know?
   
   The plan is for the gateway to save Kafka offsets in the Iggy consumer group kafka.cg.<group> without joining that group. But the server only accepts a group offset from a client that has joined the group and owns the partition. Anyone else gets error 5009 (ConsumerGroupPartitionNotOwned). The check is fence_group_offset in core/server/src/namespace.rs.
   
   I tested this on server 0.9.0. Saving an offset without joining fails with 5009, even when the group has no members. Reading the offset without joining works.
   
   So if OffsetCommit is built as the doc describes, every Kafka commit through the gateway will fail.
   
   One fix is to skip the ownership check when the group has no members. An empty group would then act as a plain offset store. Would that be acceptable on the server side?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org