You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "JozoVilcek (via GitHub)" <gi...@apache.org> on 2025/02/12 12:28:05 UTC
[I] [Bug]: Race condition in FileSystems initialisation [beam]
JozoVilcek opened a new issue, #33965:
URL: https://github.com/apache/beam/issues/33965
### What happened?
When method [FileSystems.setDefaultPipelineOptions()](https://github.com/apache/beam/blob/v2.61.0/sdks/java/core/src/main/java/org/apache/beam/sdk/io/FileSystems.java#L574) is called form multiple threads followed by using the FileSystems, then some task will fail as `FILESYSTEM_REVISION` will already be set but `SCHEME_TO_FILESYSTEM` is not yet initialised.
### Issue Priority
Priority: 2 (default / most bugs should be filed as P2)
### Issue Components
- [ ] Component: Python SDK
- [x] Component: Java SDK
- [ ] Component: Go SDK
- [ ] Component: Typescript SDK
- [ ] Component: IO connector
- [ ] Component: Beam YAML
- [ ] Component: Beam examples
- [ ] Component: Beam playground
- [ ] Component: Beam katas
- [ ] Component: Website
- [ ] Component: Infrastructure
- [ ] Component: Spark Runner
- [ ] Component: Flink Runner
- [ ] Component: Samza Runner
- [ ] Component: Twister2 Runner
- [ ] Component: Hazelcast Jet Runner
- [ ] Component: Google Cloud Dataflow Runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: Race condition in FileSystems initialisation [beam]
Posted by "liferoad (via GitHub)" <gi...@apache.org>.
liferoad commented on issue #33965:
URL: https://github.com/apache/beam/issues/33965#issuecomment-2666969506
https://github.com/apache/beam/pull/34007#issuecomment-2666854778
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: Race condition in FileSystems initialisation [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #33965:
URL: https://github.com/apache/beam/issues/33965#issuecomment-3148400939
This issue has been marked as stale due to 150 days of inactivity. It will be closed in 30 days if no further activity occurs. If you think that’s incorrect or this issue still needs to be addressed, please simply write any comment. If closed, you can reopen the issue at any time. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: Race condition in FileSystems initialisation [beam]
Posted by "liferoad (via GitHub)" <gi...@apache.org>.
liferoad commented on issue #33965:
URL: https://github.com/apache/beam/issues/33965#issuecomment-2701098796
Remove this from the release blockers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: Race condition in FileSystems initialisation [beam]
Posted by "JozoVilcek (via GitHub)" <gi...@apache.org>.
JozoVilcek commented on issue #33965:
URL: https://github.com/apache/beam/issues/33965#issuecomment-2667839176
@liferoad I wonder why is this `closed` as it is still being discussed. Perhaps unwanted closed?
@Abacn , I admit my use-case is not a standard one. I am in a mixed pure Spark and Beam on Spark runner environment. I did came across this when in pure spark I am trying to reuse filesystem utility and was calling `FileSystems.setDefaultPipelineOptions()` manually. I was trying to use `registerFileSystemsOnce()` but that will not help me since `setDefaultPipelineOptions()` is called from inside `SerializablePipelineOptions`.
I understand that API is internal and subject to change. But still I believe there is race condition in the init.
@Abacn , can you share how does workers ensure this is consistently initialised under multithreaded execution. Maybe it can inspire me with some thoughts. I was not able to find it hence I believe that in Beam the race is there perhaps too just initi sequence is more loaded therefore from initial call to `setDefaultPipelineOptions()` to actual user code using the filesystem takes more time and registration finished. My code path is much much shorter and surface this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: Race condition in FileSystems initialisation [beam]
Posted by "liferoad (via GitHub)" <gi...@apache.org>.
liferoad closed issue #33965: [Bug]: Race condition in FileSystems initialisation
URL: https://github.com/apache/beam/issues/33965
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: Race condition in FileSystems initialisation [beam]
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on issue #33965:
URL: https://github.com/apache/beam/issues/33965#issuecomment-2666852198
FileSystems.setDefaultPipelineOptions() should be called once on worker startup. It is an internal method and is not supposed to be used by worker. Would you mind sharing stacktrace so we can identify excessive call in Beam code path?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: Race condition in FileSystems initialisation [beam]
Posted by "JozoVilcek (via GitHub)" <gi...@apache.org>.
JozoVilcek commented on issue #33965:
URL: https://github.com/apache/beam/issues/33965#issuecomment-2669176487
> will my PR https://github.com/apache/beam/pull/34007 fix the issue you have? I closed this since I thought https://github.com/apache/beam/pull/34007#issuecomment-2666854778 could resolve your problem. Reopen this.
@liferoad yes, something like that would fix the problem
> This class is used by Spark/Flink/Samza/Jet runners. A fix could be move to use registerFileSystemsOnce() in SerializablePipelineOptions.
@Abacn That fix works for my specific use case, but the underlying issue remains for anyone using `setDefaultPipelineOptions()`. The root cause is that the method does not update revisions and schemas simultaneously, making it susceptible to race conditions in a multi-threaded environment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: Race condition in FileSystems initialisation [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #33965: [Bug]: Race condition in FileSystems initialisation
URL: https://github.com/apache/beam/issues/33965
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: Race condition in FileSystems initialisation [beam]
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on issue #33965:
URL: https://github.com/apache/beam/issues/33965#issuecomment-2668966229
> setDefaultPipelineOptions() is called from inside SerializablePipelineOptions.
This class is used by Spark/Flink/Samza/Jet runners. A fix could be move to use `registerFileSystemsOnce()` in SerializablePipelineOptions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: Race condition in FileSystems initialisation [beam]
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on issue #33965:
URL: https://github.com/apache/beam/issues/33965#issuecomment-2668999567
This issue is the same as #33965. SerializablePipelineOptions shouldn't call FileSystems.setDefaultPipelineOption
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: Race condition in FileSystems initialisation [beam]
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on issue #33965:
URL: https://github.com/apache/beam/issues/33965#issuecomment-2669458845
> the method does not update revisions and schemas simultaneously,
setDefaultPipelineOptions() is supposed to be called once, per job, on worker, at the beginning of pipeline execution. Because it initializes FileSystems with a given PipelineOption.
A second invocation will overwrite the PipelineOption in FileSystem, if it is still used by the first pipeline, then the first pipeline is running into inconsistent state where its FileSystem interface are overriden by others
However it is used in other place that really shouldn't to, to be able to use Beam FileSystem outside pipeline execution, and SerializablePipelineOptions constructor is one of it.
So I think a proper fix is to eliminate setDefaultPipelineOptions and change to registerFileSystemsOnce
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: Race condition in FileSystems initialisation [beam]
Posted by "liferoad (via GitHub)" <gi...@apache.org>.
liferoad commented on issue #33965:
URL: https://github.com/apache/beam/issues/33965#issuecomment-2668707821
will my PR https://github.com/apache/beam/pull/34007 fix the issue you have? I closed this since I thought https://github.com/apache/beam/pull/34007#issuecomment-2666854778 could resolve your problem. Reopen this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: Race condition in FileSystems initialisation [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #33965:
URL: https://github.com/apache/beam/issues/33965#issuecomment-3249108543
This issue has been closed due to lack of activity. If you think that is incorrect, you can reopen the issue at any time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org