You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "hjtran (via GitHub)" <gi...@apache.org> on 2025/11/18 13:14:12 UTC
[I] [Bug] [python]: Possible race condition when starting many pipelines at the same time [beam]
hjtran opened a new issue, #36847:
URL: https://github.com/apache/beam/issues/36847
### What happened?
If you start many pipelines at the same time, it's possible to run into a FileExistsError race condition when two pipelines try to create the requirements cache at the same time:
```
File ".../lib/python3.11/site-packages/apache_beam/runners/runner.py", line 182, in run_pipeline
default_environment=self.default_environment(options)),
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File ".../lib/python3.11/site-packages/apache_beam/runners/runner.py", line 163, in default_environment
return environments.Environment.from_options(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File ".../lib/python3.11/site-packages/apache_beam/transforms/environments.py", line 274, in from_options
return env_class.from_options(portable_options) # type: ignore
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File ".../lib/python3.11/site-packages/apache_beam/transforms/environments.py", line 367, in from_options
artifacts=python_sdk_dependencies(options),
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File ".../lib/python3.11/site-packages/apache_beam/transforms/environments.py", line 913, in python_sdk_dependencies
return stager.Stager.create_job_resources(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File ".../lib/python3.11/site-packages/apache_beam/runners/portability/stager.py", line 221, in create_job_resources
os.makedirs(requirements_cache_path)
File "<frozen os>", line 225, in makedirs
FileExistsError: [Errno 17] File exists: '/tmp/dataflow-requirements-cache
```
I think we can probably just set `exists_ok=True` for the. makedirs call
### Issue Priority
Priority: 2 (default / most bugs should be filed as P2)
### Issue Components
- [x] Component: Python SDK
- [ ] Component: Java SDK
- [ ] Component: Go SDK
- [ ] Component: Typescript SDK
- [ ] Component: IO connector
- [ ] Component: Beam YAML
- [ ] Component: Beam examples
- [ ] Component: Beam playground
- [ ] Component: Beam katas
- [ ] Component: Website
- [ ] Component: Infrastructure
- [ ] Component: Spark Runner
- [ ] Component: Flink Runner
- [ ] Component: Samza Runner
- [ ] Component: Twister2 Runner
- [ ] Component: Hazelcast Jet Runner
- [ ] Component: Google Cloud Dataflow Runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [python]: Possible race condition when starting many pipelines at the same time [beam]
Posted by "hjtran (via GitHub)" <gi...@apache.org>.
hjtran commented on issue #36847:
URL: https://github.com/apache/beam/issues/36847#issuecomment-3547607641
@NayanMathur411 this could be a good first issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [python]: Possible race condition when starting many pipelines at the same time [beam]
Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn closed issue #36847: [Bug] [python]: Possible race condition when starting many pipelines at the same time
URL: https://github.com/apache/beam/issues/36847
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [python]: Possible race condition when starting many pipelines at the same time [beam]
Posted by "NayanMathur411 (via GitHub)" <gi...@apache.org>.
NayanMathur411 commented on issue #36847:
URL: https://github.com/apache/beam/issues/36847#issuecomment-3626515460
.take-issue
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [python]: Possible race condition when starting many pipelines at the same time [beam]
Posted by "hjtran (via GitHub)" <gi...@apache.org>.
hjtran commented on issue #36847:
URL: https://github.com/apache/beam/issues/36847#issuecomment-3547849187
This is pretty difficult to test. You might be able to reproduce by adding a `time.sleep` right before the `makedirs` (only for the purpose of reproducing).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [python]: Possible race condition when starting many pipelines at the same time [beam]
Posted by "Arunodoy18 (via GitHub)" <gi...@apache.org>.
Arunodoy18 commented on issue #36847:
URL: https://github.com/apache/beam/issues/36847#issuecomment-3580586591
@NayanMathur411,Sure go ahead with it , if any problem occurs i would love to take that too.
Thank you.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [python]: Possible race condition when starting many pipelines at the same time [beam]
Posted by "NayanMathur411 (via GitHub)" <gi...@apache.org>.
NayanMathur411 commented on issue #36847:
URL: https://github.com/apache/beam/issues/36847#issuecomment-3580556778
> HI, I think i can take this and solve this issue . .take-issue.
Hi @Arunodoy18, thanks for your interest.
I have already started working on this, and already in middle of it. Just finding time out of work to put up a review for it...
Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [python]: Possible race condition when starting many pipelines at the same time [beam]
Posted by "NayanMathur411 (via GitHub)" <gi...@apache.org>.
NayanMathur411 commented on issue #36847:
URL: https://github.com/apache/beam/issues/36847#issuecomment-3548832383
> This is pretty difficult to test. You might be able to reproduce by adding a `time.sleep` right before the `makedirs` (only for the purpose of reproducing).
Thanks for mentioning this, will start on this soon and ping if any issues faced.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [python]: Possible race condition when starting many pipelines at the same time [beam]
Posted by "Arunodoy18 (via GitHub)" <gi...@apache.org>.
Arunodoy18 commented on issue #36847:
URL: https://github.com/apache/beam/issues/36847#issuecomment-3580473429
HI, I think i can take this and solve this issue .
.take-issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org