You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@storm.apache.org by "mwkang (via GitHub)" <gi...@apache.org> on 2025/04/23 01:11:11 UTC
[I] Support for configuring group.instance.id in Kafka consumers used by Storm (storm)
mwkang opened a new issue, #8046:
URL: https://github.com/apache/storm/issues/8046
Hi Storm maintainers,
I’m currently working on integrating Kafka static membership with Storm, specifically by setting the `group.instance.id` property in Kafka consumers.
In Kubernetes environments, it's relatively straightforward to assign a stable value (e.g., using the Pod UID) to `group.instance.id`. However, in Apache Storm, I haven’t found an obvious way to assign a unique and stable identifier to each consumer instance.
It seems that Storm workers or tasks don't have any persistent or externally visible ID that can be used for this purpose, which makes it challenging to use Kafka static membership effectively.
I have a few questions:
- Is there any existing identifier within Storm (e.g., worker ID, task ID) that could be used as a stable `group.instance.id`?
- If not, are there any recommended workarounds to achieve this?
- Would it be feasible to consider adding a feature that allows users to inject a stable instance ID (perhaps via configuration or environment variable) for use with Kafka consumers?
Any suggestions or guidance would be appreciated. Thank you for your time and for maintaining this great project!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@storm.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Support for configuring group.instance.id in Kafka consumers used by Storm (storm)
Posted by "rzo1 (via GitHub)" <gi...@apache.org>.
rzo1 commented on issue #8046:
URL: https://github.com/apache/storm/issues/8046#issuecomment-2830186397
Hi,
afaik task IDs are assigned when the topology is created and remain static. Even if a task fails, restarts, or is reassigned to a different location, it will retain the same ID.
The bolt will consistently receive the same task ID as long as the overall configuration—specifically, the number of executors across the topology—doesn't change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@storm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Support for configuring group.instance.id in Kafka consumers used by Storm (storm)
Posted by "mwkang (via GitHub)" <gi...@apache.org>.
mwkang commented on issue #8046:
URL: https://github.com/apache/storm/issues/8046#issuecomment-2832147042
Thank you so much for the clarification!
Knowing that task IDs remain stable is exactly the information I needed.
Really appreciate your help!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@storm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Support for configuring group.instance.id in Kafka consumers used by Storm (storm)
Posted by "reiabreu (via GitHub)" <gi...@apache.org>.
reiabreu closed issue #8046: Support for configuring group.instance.id in Kafka consumers used by Storm
URL: https://github.com/apache/storm/issues/8046
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@storm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Support for configuring group.instance.id in Kafka consumers used by Storm (storm)
Posted by "reiabreu (via GitHub)" <gi...@apache.org>.
reiabreu commented on issue #8046:
URL: https://github.com/apache/storm/issues/8046#issuecomment-2946723210
@mwkang closing this issue. Feel free to reopen it in case you feel it's not resolved.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@storm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org