You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "tvalentyn (via GitHub)" <gi...@apache.org> on 2023/09/21 22:43:30 UTC
[GitHub] [beam] tvalentyn opened a new issue, #28605: [Bug]: Artifact materialization can cause a race when multiple callers materialize the same artifact in the same destination.
tvalentyn opened a new issue, #28605:
URL: https://github.com/apache/beam/issues/28605
### What happened?
Beam SDK containers use the below helper to download artifacts.
https://github.com/apache/beam/blob/0146a8389b4bf94cafd244f70fc0ad13b32066cc/sdks/go/pkg/beam/artifact/materialize.go#L60
When a worker instantiates multiple SDK containers that share a destination for staged artifacts (`--semi_persist_dir`: https://github.com/apache/beam/blob/0146a8389b4bf94cafd244f70fc0ad13b32066cc/sdks/java/container/boot.go#L50), a race can occur: while one container writes a staged artifact into a staging location, another container might not be able to read it, even though the artifact was previously materialized.
This race was observed in Dataflow Python pipelines that didn't use a sibling SDK worker.
We can consider using a file lock during materialization, and possibly not re-download artifacts with the same SHA.
### Issue Priority
Priority: 2 (default / most bugs should be filed as P2)
### Issue Components
- [X] Component: Python SDK
- [X] Component: Java SDK
- [X] Component: Go SDK
- [ ] Component: Typescript SDK
- [ ] Component: IO connector
- [ ] Component: Beam examples
- [ ] Component: Beam playground
- [ ] Component: Beam katas
- [ ] Component: Website
- [ ] Component: Spark Runner
- [ ] Component: Flink Runner
- [ ] Component: Samza Runner
- [ ] Component: Twister2 Runner
- [ ] Component: Hazelcast Jet Runner
- [ ] Component: Google Cloud Dataflow Runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: Artifact materialization can cause a race when multiple callers materialize the same artifact in the same destination. [beam]
Posted by "kennknowles (via GitHub)" <gi...@apache.org>.
kennknowles commented on issue #28605:
URL: https://github.com/apache/beam/issues/28605#issuecomment-1745351426
Can you point to a change that addressed this? Is it already in 2.51.0? (apologies if this is already answer and I missed it or forgot)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] kennknowles commented on issue #28605: [Bug]: Artifact materialization can cause a race when multiple callers materialize the same artifact in the same destination.
Posted by "kennknowles (via GitHub)" <gi...@apache.org>.
kennknowles commented on issue #28605:
URL: https://github.com/apache/beam/issues/28605#issuecomment-1735454362
Which is just me saying the same thing but from the peanut gallery objecting to processes working in a way that makes assumptions about how they'll be used, such as an SDK harness assuming it has a particular relationship to docker containers and VMs. That's just pain down the line.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] tvalentyn commented on issue #28605: [Bug]: Artifact materialization can cause a race when multiple callers materialize the same artifact in the same destination.
Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on issue #28605:
URL: https://github.com/apache/beam/issues/28605#issuecomment-1739884693
sibling worker protocol has been reenabled now, and with sibling workers, this issue doesn't happen.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] tvalentyn commented on issue #28605: [Bug]: Artifact materialization can cause a race when multiple callers materialize the same artifact in the same destination.
Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on issue #28605:
URL: https://github.com/apache/beam/issues/28605#issuecomment-1739886714
At least in Dataflow runner.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: Artifact materialization can cause a race when multiple callers materialize the same artifact in the same destination. [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #28605:
URL: https://github.com/apache/beam/issues/28605#issuecomment-5293463631
This issue has been closed due to lack of activity. If you think that is incorrect, you can reopen the issue at any time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] tvalentyn commented on issue #28605: [Bug]: Artifact materialization can cause a race when multiple callers materialize the same artifact in the same destination.
Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on issue #28605:
URL: https://github.com/apache/beam/issues/28605#issuecomment-1730442820
cc: @lostluck
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: Artifact materialization can cause a race when multiple callers materialize the same artifact in the same destination. [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #28605:
URL: https://github.com/apache/beam/issues/28605#issuecomment-4969359324
This issue has been marked as stale due to 150 days of inactivity. It will be closed in 30 days if no further activity occurs. If you think that’s incorrect or this issue still needs to be addressed, please simply write any comment. If closed, you can reopen the issue at any time. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] kennknowles commented on issue #28605: [Bug]: Artifact materialization can cause a race when multiple callers materialize the same artifact in the same destination.
Posted by "kennknowles (via GitHub)" <gi...@apache.org>.
kennknowles commented on issue #28605:
URL: https://github.com/apache/beam/issues/28605#issuecomment-1735456400
I think this was on my radar due to the release. Is it common enough that it makes the sibling process feature unusable, or can we defer to 2.52.0? (I know there is already not a milestone attached - I am inviting you to attach 2.51.0 if it would have a huge negative impact on users)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] kennknowles commented on issue #28605: [Bug]: Artifact materialization can cause a race when multiple callers materialize the same artifact in the same destination.
Posted by "kennknowles (via GitHub)" <gi...@apache.org>.
kennknowles commented on issue #28605:
URL: https://github.com/apache/beam/issues/28605#issuecomment-1735451952
I'm gonna say that a process writing to a share location had better assume there might be other processes that want that location, even on accident. The two choices are: (1) generate a fresh location that no one else is going to conflict with (a la `mktemp -d`) or (2) use a location intentionally shared. I feel like in this case intentional sharing is what you want, to download once and use by all.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: Artifact materialization can cause a race when multiple callers materialize the same artifact in the same destination. [beam]
Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on issue #28605:
URL: https://github.com/apache/beam/issues/28605#issuecomment-1745564181
a change that is now fully rolled out in Dataflow runner fully enables sibling sdk worker protocol, thereby removing the failure mode in default configuration. it is a runner-controlled change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] lostluck commented on issue #28605: [Bug]: Artifact materialization can cause a race when multiple callers materialize the same artifact in the same destination.
Posted by "lostluck (via GitHub)" <gi...@apache.org>.
lostluck commented on issue #28605:
URL: https://github.com/apache/beam/issues/28605#issuecomment-1730455864
This seems like a Python specific issue, as it does multiple processes or multiple containers on a worker VM. The other SDKs (Go and Java at least) will only have a single boot cycle to to download from the artifact repo. It's hard to block race conditions across separate processes that can't meaningfully communicate.
It's also hard to know if that semi-persist directory is actually shared or not between other workers.
In more actionable commentary, that "don't download if the sha matches" is probably a good idea, since we can check for existing files, and if it exists with with a matching SHA, then it's the expected file.
If not, it might be either wrong, or in progress of being downloaded....
Whomever works on this needs to take that into consideration.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: Artifact materialization can cause a race when multiple callers materialize the same artifact in the same destination. [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #28605: [Bug]: Artifact materialization can cause a race when multiple callers materialize the same artifact in the same destination.
URL: https://github.com/apache/beam/issues/28605
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org