You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "robertwb (via GitHub)" <gi...@apache.org> on 2023/02/14 21:23:55 UTC
[GitHub] [beam] robertwb opened a new pull request, #25473: Add retry logic to Python boot script.
robertwb opened a new pull request, #25473:
URL: https://github.com/apache/beam/pull/25473
This will allow runners to act more quickly on failures rather than wait for all workers to die before the exiting the container.
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] Mention the appropriate issue in your description (for example: `addresses #123`), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment `fixes #<ISSUE NUMBER>` instead.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/get-started-contributing/#make-the-reviewers-job-easier).
To check the build health, please visit [https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md](https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md)
GitHub Actions Tests Status (on master branch)
------------------------------------------------------------------------------------------------
[](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Go+tests%22+branch%3Amaster+event%3Aschedule)
See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] commented on pull request #25473: Add retry logic to Python boot script.
Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #25473:
URL: https://github.com/apache/beam/pull/25473#issuecomment-1430413841
# [Codecov](https://codecov.io/gh/apache/beam/pull/25473?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#25473](https://codecov.io/gh/apache/beam/pull/25473?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (a6bd21a) into [master](https://codecov.io/gh/apache/beam/commit/d20d0b01c3c6bcde551420f36e13d794c930f1e2?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (d20d0b0) will **not change** coverage.
> The diff coverage is `n/a`.
```diff
@@ Coverage Diff @@
## master #25473 +/- ##
=======================================
Coverage 72.96% 72.96%
=======================================
Files 745 745
Lines 99174 99174
=======================================
Hits 72362 72362
Misses 25446 25446
Partials 1366 1366
```
:mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] antonbobkov commented on a diff in pull request #25473: Add retry logic to Python boot script.
Posted by "antonbobkov (via GitHub)" <gi...@apache.org>.
antonbobkov commented on code in PR #25473:
URL: https://github.com/apache/beam/pull/25473#discussion_r1106536433
##########
sdks/python/container/boot.go:
##########
@@ -251,20 +251,31 @@ func launchSDKProcess() error {
go func(workerId string) {
defer wg.Done()
- childPids.mu.Lock()
- if childPids.canceled {
+ errorCount := 0
+ for {
+ childPids.mu.Lock()
+ if childPids.canceled {
+ childPids.mu.Unlock()
+ return
+ }
+ log.Printf("Executing Python (worker %v): python %v", workerId, strings.Join(args, " "))
+ cmd := StartCommandEnv(map[string]string{"WORKER_ID": workerId}, "python", args...)
+ childPids.v = append(childPids.v, cmd.Process.Pid)
childPids.mu.Unlock()
- return
- }
- log.Printf("Executing Python (worker %v): python %v", workerId, strings.Join(args, " "))
- cmd := StartCommandEnv(map[string]string{"WORKER_ID": workerId}, "python", args...)
- childPids.v = append(childPids.v, cmd.Process.Pid)
- childPids.mu.Unlock()
-
- if err := cmd.Wait(); err != nil {
- log.Printf("Python (worker %v) exited: %v", workerId, err)
- } else {
- log.Printf("Python (worker %v) exited.", workerId)
+
+ if err := cmd.Wait(); err != nil {
+ // Retry on fatal errors, like OOMs and segfaults, not just
+ // DoFns throwing exceptions.
+ errorCount += 1
+ if errorCount < 4 {
+ log.Printf("Python (worker %v) exited: %v", workerId, err)
Review Comment:
perhaps add more information in the logs about retries/failures. Maybe something along the lines of "Python (worker %v) exited: %v, retrying process." and "Python (worker %v) exited: %v, out of retries: restarting SDK container."
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] robertwb merged pull request #25473: Add retry logic to Python boot script.
Posted by "robertwb (via GitHub)" <gi...@apache.org>.
robertwb merged PR #25473:
URL: https://github.com/apache/beam/pull/25473
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] robertwb commented on pull request #25473: Add retry logic to Python boot script.
Posted by "robertwb (via GitHub)" <gi...@apache.org>.
robertwb commented on PR #25473:
URL: https://github.com/apache/beam/pull/25473#issuecomment-1439326356
PTAL
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] robertwb commented on a diff in pull request #25473: Add retry logic to Python boot script.
Posted by "robertwb (via GitHub)" <gi...@apache.org>.
robertwb commented on code in PR #25473:
URL: https://github.com/apache/beam/pull/25473#discussion_r1106544214
##########
sdks/python/container/boot.go:
##########
@@ -251,20 +251,31 @@ func launchSDKProcess() error {
go func(workerId string) {
defer wg.Done()
- childPids.mu.Lock()
- if childPids.canceled {
+ errorCount := 0
+ for {
+ childPids.mu.Lock()
+ if childPids.canceled {
+ childPids.mu.Unlock()
+ return
+ }
+ log.Printf("Executing Python (worker %v): python %v", workerId, strings.Join(args, " "))
+ cmd := StartCommandEnv(map[string]string{"WORKER_ID": workerId}, "python", args...)
+ childPids.v = append(childPids.v, cmd.Process.Pid)
childPids.mu.Unlock()
- return
- }
- log.Printf("Executing Python (worker %v): python %v", workerId, strings.Join(args, " "))
- cmd := StartCommandEnv(map[string]string{"WORKER_ID": workerId}, "python", args...)
- childPids.v = append(childPids.v, cmd.Process.Pid)
- childPids.mu.Unlock()
-
- if err := cmd.Wait(); err != nil {
- log.Printf("Python (worker %v) exited: %v", workerId, err)
- } else {
- log.Printf("Python (worker %v) exited.", workerId)
+
+ if err := cmd.Wait(); err != nil {
+ // Retry on fatal errors, like OOMs and segfaults, not just
+ // DoFns throwing exceptions.
+ errorCount += 1
+ if errorCount < 4 {
+ log.Printf("Python (worker %v) exited: %v", workerId, err)
Review Comment:
Thanks. Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org