You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "Shri-Vardhan (via GitHub)" <gi...@apache.org> on 2026/05/21 15:22:20 UTC
[I] [Feature Request]: Add metrics tracking for offset commit failures and retry attempts in KafkaCommitOffset [beam]
Shri-Vardhan opened a new issue, #38587:
URL: https://github.com/apache/beam/issues/38587
### What would you like to happen?
Currently in `KafkaCommitOffset.CommitOffsetDoFn` there is no visibility into how often offset commits are failing or how many retries are being attempted. If commits are silently failing in production you have no way to know without digging through logs.
Adding Beam metrics here would make it easy to monitor commit health through existing monitoring dashboards.
Proposed approach
Add a counter for total commit failures
Add a counter for retry attempts
Add a counter for commits that exhausted all retries
Use Beam's built in `Metrics.counter()` so it works with any runner
**Why this is useful**
If you're running a high throughput pipeline and offset commits are struggling, right now you'd have no idea until you see duplicate processing. This gives you an early warning signal.
### Issue Priority
Priority: 3 (nice-to-have improvement)
### Issue Components
- [ ] Component: Python SDK
- [x] Component: Java SDK
- [ ] Component: Go SDK
- [ ] Component: Typescript SDK
- [x] Component: IO connector
- [ ] Component: Beam YAML
- [ ] Component: Beam examples
- [ ] Component: Beam playground
- [ ] Component: Beam katas
- [ ] Component: Website
- [ ] Component: Infrastructure
- [ ] Component: Spark Runner
- [ ] Component: Flink Runner
- [ ] Component: Prism Runner
- [ ] Component: Twister2 Runner
- [ ] Component: Hazelcast Jet Runner
- [ ] Component: Google Cloud Dataflow Runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature Request]: Add metrics tracking for offset commit failures and retry attempts in KafkaCommitOffset [beam]
Posted by "Shri-Vardhan (via GitHub)" <gi...@apache.org>.
Shri-Vardhan commented on issue #38587:
URL: https://github.com/apache/beam/issues/38587#issuecomment-4743817933
Closing this PR because `KafkaCommitOffset.CommitOffsetDoFn` already logs commit failures and retry attempts.
I reviewed the existing behavior in KafkaCommitOffset.CommitOffsetDoFn, and noticed that commit failures and retry attempts are already adequately captured through the current logging.
Given that, adding Beam metrics here would provide overlapping observability with limited additional value while increasing maintenance complexity.
I don't have further concerns, so I'm happy with closing this issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature Request]: Add metrics tracking for offset commit failures and retry attempts in KafkaCommitOffset [beam]
Posted by "damccorm (via GitHub)" <gi...@apache.org>.
damccorm commented on issue #38587:
URL: https://github.com/apache/beam/issues/38587#issuecomment-4520067486
cc/ @an2x @johnjcasey @scwhittle
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature Request]: Add metrics tracking for offset commit failures and retry attempts in KafkaCommitOffset [beam]
Posted by "Shri-Vardhan (via GitHub)" <gi...@apache.org>.
Shri-Vardhan commented on issue #38587:
URL: https://github.com/apache/beam/issues/38587#issuecomment-4520870505
> Hi, I would like to work on this issue. Could this be assigned to me?
Sadly , It's not in my hand's
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature Request]: Add metrics tracking for offset commit failures and retry attempts in KafkaCommitOffset [beam]
Posted by "SyedAbd110786 (via GitHub)" <gi...@apache.org>.
SyedAbd110786 commented on issue #38587:
URL: https://github.com/apache/beam/issues/38587#issuecomment-4525423976
Hi, I’m exploring this issue and trying to understand the current commit/retry flow in `KafkaCommitOffset.CommitOffsetDoFn`. I’d like to investigate whether I can contribute a fix if no one is actively working on it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature Request]: Add metrics tracking for offset commit failures and retry attempts in KafkaCommitOffset [beam]
Posted by "Kriti-dev07 (via GitHub)" <gi...@apache.org>.
Kriti-dev07 commented on issue #38587:
URL: https://github.com/apache/beam/issues/38587#issuecomment-4520767247
Hi, I would like to work on this issue. Could this be assigned to me?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature Request]: Add metrics tracking for offset commit failures and retry attempts in KafkaCommitOffset [beam]
Posted by "Shri-Vardhan (via GitHub)" <gi...@apache.org>.
Shri-Vardhan closed issue #38587: [Feature Request]: Add metrics tracking for offset commit failures and retry attempts in KafkaCommitOffset
URL: https://github.com/apache/beam/issues/38587
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org