You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "ahmedabu98 (via GitHub)" <gi...@apache.org> on 2026/04/30 01:03:08 UTC
[I] [Feature Request]: [IcebergIO] Fetch a fresh spec periodically in hash distribution [beam]
ahmedabu98 opened a new issue, #38337:
URL: https://github.com/apache/beam/issues/38337
### What would you like to happen?
In `AssignDestinationsAndPartitions`, we just fetch the spec once per VM and store the generated PartitionKey.
The spec may change at any time though. To avoid working with a stale spec, we should periodically fetch it from the table
### Issue Priority
Priority: 2 (default / most feature requests should be filed as P2)
### Issue Components
- [ ] Component: Python SDK
- [ ] Component: Java SDK
- [ ] Component: Go SDK
- [ ] Component: Typescript SDK
- [ ] Component: IO connector
- [ ] Component: Beam YAML
- [ ] Component: Beam examples
- [ ] Component: Beam playground
- [ ] Component: Beam katas
- [ ] Component: Website
- [ ] Component: Infrastructure
- [ ] Component: Spark Runner
- [ ] Component: Flink Runner
- [ ] Component: Prism Runner
- [ ] Component: Twister2 Runner
- [ ] Component: Hazelcast Jet Runner
- [ ] Component: Google Cloud Dataflow Runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature Request]: [IcebergIO] Fetch a fresh spec periodically in hash distribution [beam]
Posted by "ahmedabu98 (via GitHub)" <gi...@apache.org>.
ahmedabu98 closed issue #38337: [Feature Request]: [IcebergIO] Fetch a fresh spec periodically in hash distribution
URL: https://github.com/apache/beam/issues/38337
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature Request]: [IcebergIO] Fetch a fresh spec periodically in hash distribution [beam]
Posted by "AtharvUrunkar (via GitHub)" <gi...@apache.org>.
AtharvUrunkar commented on issue #38337:
URL: https://github.com/apache/beam/issues/38337#issuecomment-4394367368
I’d like to work on this issue.
Current implementation fetches the spec only once per VM and keeps using the cached version, which can become stale if the table spec changes during runtime.
My plan is to introduce a periodic refresh mechanism with a configurable refresh interval/TTL. The worker will reload the latest spec from the table after the interval expires and update the cached spec only if changes are detected. I’ll also ensure the refresh logic is thread-safe and add unit tests covering:
* initial spec loading
* cache reuse before refresh interval
* spec refresh after interval expiry
* handling updated specs correctly
* concurrent access behavior
Please assign this issue to me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org