You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2022/11/25 12:50:55 UTC
[GitHub] [beam] phoerious opened a new issue, #24357: [Bug]: Python SDK: Dependency installation hangs sometimes
phoerious opened a new issue, #24357:
URL: https://github.com/apache/beam/issues/24357
### What happened?
Disclaimer: This bug reports refers to the current master branch, which contains major changes to the container boot.go, but it's very likely that it applies to previous versions as well, since the parts which most likely trigger the bug were not changed.
The Python SDK boot.go binary installs dependencies from a setup.py automatically here: https://github.com/apache/beam/blob/master/sdks/python/container/boot.go#L368
Most of the time, it works fine, but sometimes `pip` process seems to hang for no reason. The problem can be resolved by attaching to the container running the SDK and reading STDIN of the `boot` process:
```kubectl exec -ti pod/beam-flinker-cluster-taskmanager -c beam-worker-pool -- bash
ps aux | grep boot
cat /proc/<WORKER-PID>/fd/1
exit
```
I suppose this issue occurs due to a stdout buffer fillup. The boot binary uses the following utility for running `pip`, which redirects the processe's STDOUT to `os.stdout`. A solution would perhaps be to either not redirect stdout or to write it directly to `/dev/null`.
https://github.com/apache/beam/blob/master/sdks/go/pkg/beam/util/execx/exec.go#L26
### Issue Priority
Priority: 2
### Issue Component
Component: sdk-py-harness
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] tvalentyn commented on issue #24357: [Bug]: Python SDK: Dependency installation hangs sometimes
Posted by GitBox <gi...@apache.org>.
tvalentyn commented on issue #24357:
URL: https://github.com/apache/beam/issues/24357#issuecomment-1331134500
any chance this is specific to the Flink setup?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] tvalentyn commented on issue #24357: [Bug]: Python SDK: Dependency installation hangs sometimes
Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on issue #24357:
URL: https://github.com/apache/beam/issues/24357#issuecomment-1670289936
Interesting.
> These downstream lines in Go are just as problematic
I don't think we observed deadlocks in Dataflow environments but cc'ing: @lostluck our Go expert who may have suggestions and @jrmccluskey who is looking at improving logging in boot loaders.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] phoerious commented on issue #24357: [Bug]: Python SDK: Dependency installation hangs sometimes
Posted by GitBox <gi...@apache.org>.
phoerious commented on issue #24357:
URL: https://github.com/apache/beam/issues/24357#issuecomment-1331471035
No, it does not happen consistently. it is probably dependent on the host's utilisation or IO saturation. I also don't think it's Flink-related. There is nothing special in the logs. It usually hangs after pip's last installation log dump, just before the final "Successfully installed pkg1 pkg2 ..." line. Reading pip's or the worker's stdout via `cat` will let it continue like nothing happened.
Redirecting everything `/dev/null` may be a bit harsh, but redirecting pip's output to the worker's stdout is definitely useless. What is shown as the container's log output is the worker pool output, which displays only the pool's own stdout and log.Printf() messages as well as log.Printf() messages from the workers (apparently). It does not include normal stdout of child processes, including the workers. Therefore no pip output gets logged anywhere, really. In order to read or log it, you have to attach to the container and read from `/proc/PID/fd/1` directly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] lostluck commented on issue #24357: [Bug]: Python SDK: Dependency installation hangs sometimes
Posted by "lostluck (via GitHub)" <gi...@apache.org>.
lostluck commented on issue #24357:
URL: https://github.com/apache/beam/issues/24357#issuecomment-1670296380
From pkg.go.dev/os
`Stdin, Stdout, and Stderr are open Files pointing to the standard input, standard output, and standard error file descriptors.`
So it's technically whatever the OS should be doing by default.
I've suggested to Jack that we should swap those with our own buffered IO so we can parse & emit them to the logging service.
But i wouldn't suggest we try to get that change in the day before the 2.50 cut.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] tvalentyn commented on issue #24357: [Bug]: Python SDK: Dependency installation hangs sometimes
Posted by GitBox <gi...@apache.org>.
tvalentyn commented on issue #24357:
URL: https://github.com/apache/beam/issues/24357#issuecomment-1334586657
we do get pip's output in Dataflow startup logs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] phoerious commented on issue #24357: [Bug]: Python SDK: Dependency installation hangs sometimes
Posted by GitBox <gi...@apache.org>.
phoerious commented on issue #24357:
URL: https://github.com/apache/beam/issues/24357#issuecomment-1331897437
I did not change how pip is being called, so I highly doubt that it is related to my changes.
> Dataflow might be doing that, perhaps that is why we never observe such behavior.
But does it also explicitly flush the stdout of all processes? Kubelet also reads stdout of the container, but pip is not part of the container stdout.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] robertwb commented on issue #24357: [Bug]: Python SDK: Dependency installation hangs sometimes
Posted by "robertwb (via GitHub)" <gi...@apache.org>.
robertwb commented on issue #24357:
URL: https://github.com/apache/beam/issues/24357#issuecomment-1670425242
Regarding the lines at worker_pool_main.py#L16, these shouldn't be used in the dataflow service (i.e. use_subprocess, which is primarily for testing, should be false).
(I'd also be surprised if os.Stdout, etc. limited buffers.)
If we capture/redirect this somewhere, we should be very careful, as otherwise that can make debugging very difficult.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] phoerious commented on issue #24357: [Bug]: Python SDK: Dependency installation hangs sometimes
Posted by GitBox <gi...@apache.org>.
phoerious commented on issue #24357:
URL: https://github.com/apache/beam/issues/24357#issuecomment-1349360824
No. But I haven't had any such hangs either since patching the boot binary.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] phoerious commented on issue #24357: [Bug]: Python SDK: Dependency installation hangs sometimes
Posted by GitBox <gi...@apache.org>.
phoerious commented on issue #24357:
URL: https://github.com/apache/beam/issues/24357#issuecomment-1336153408
Which is the exact equivalent of what was there before, only safer.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] phoerious commented on issue #24357: [Bug]: Python SDK: Dependency installation hangs sometimes
Posted by "phoerious (via GitHub)" <gi...@apache.org>.
phoerious commented on issue #24357:
URL: https://github.com/apache/beam/issues/24357#issuecomment-1669254788
@tvalentyn I think I may have found why this is happening. The following line connects the container executable's STDOUT and then simply starts the process without interacting with it any further. This will fill up the internal I/O buffers and hang forever unless the output is consumed somehow, e.g. by calling `communicate()`. https://github.com/apache/beam/blob/master/sdks/python/apache_beam/runners/worker/worker_pool_main.py#L167
This behaviour is documented: https://docs.python.org/3/library/subprocess.html#subprocess.Popen.wait
This is a general problem with pipes, so it's not specific to Python. These downstream lines in Go are just as problematic and may already cause deadlocks if the upstream Python call doesn't: https://github.com/apache/beam/blob/master/sdks/go/pkg/beam/util/execx/exec.go#L34-L36
See also https://devblogs.microsoft.com/oldnewthing/20110707-00/?p=10223
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] jrmccluskey commented on issue #24357: [Bug]: Python SDK: Dependency installation hangs sometimes
Posted by "jrmccluskey (via GitHub)" <gi...@apache.org>.
jrmccluskey commented on issue #24357:
URL: https://github.com/apache/beam/issues/24357#issuecomment-1670324932
Based on conversations with Valentyn this week I think the buffered IO for handling these subprocess logs is the way to go, hoping to get started on design discussion end of this week or start of next week.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] phoerious commented on issue #24357: [Bug]: Python SDK: Dependency installation hangs sometimes
Posted by GitBox <gi...@apache.org>.
phoerious commented on issue #24357:
URL: https://github.com/apache/beam/issues/24357#issuecomment-1353690808
My guess is that this is just I/O wait because redirecting stdio of the process has a limited buffer size. Why this usually seems to affect only the very last line of output, I don't know, but that may be coincidence, since the pip output length is usually rather predictable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] tvalentyn commented on issue #24357: [Bug]: Python SDK: Dependency installation hangs sometimes
Posted by GitBox <gi...@apache.org>.
tvalentyn commented on issue #24357:
URL: https://github.com/apache/beam/issues/24357#issuecomment-1331133191
I frequently debug worker startup issues on Dataflow (which also uses containers), and have not seen dependency installation to just hang for no reason. I don't think redirecting to /dev/null is an option as the installation logs are useful. Are you able to repro the issue consistently? Are there any logs that demonstrate the stuckness to rule out that it is not caused by dependency issues?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] tvalentyn commented on issue #24357: [Bug]: Python SDK: Dependency installation hangs sometimes
Posted by GitBox <gi...@apache.org>.
tvalentyn commented on issue #24357:
URL: https://github.com/apache/beam/issues/24357#issuecomment-1334587891
> I did not change how pip is being called
at minimum I think you changed invocation to use: `python -m pip` instead of `pip`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] tvalentyn commented on issue #24357: [Bug]: Python SDK: Dependency installation hangs sometimes
Posted by GitBox <gi...@apache.org>.
tvalentyn commented on issue #24357:
URL: https://github.com/apache/beam/issues/24357#issuecomment-1353164219
Interesting. I don't have an explanation for the stuckness; the output from worker startup/installation commands is limited (eventually everything should be installed, no new output is generated). Is that correct? Or reusing the container means that more output may be generated with each pipeline you are submitting?
Wondering why would anything like that hang the process. And, by hanging, you mean that the output is lagging, or boot sequence stalls, and e.g. launching the SDK harness is not happening?
If you want, you could consider trying to unpatch the binary and see if issue is reproducible, so that we can think of some solution for other users. Alternatively, we can wait to see if someone else experiences this issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] tvalentyn commented on issue #24357: [Bug]: Python SDK: Dependency installation hangs sometimes
Posted by GitBox <gi...@apache.org>.
tvalentyn commented on issue #24357:
URL: https://github.com/apache/beam/issues/24357#issuecomment-1348343996
just checking if you might have any new information here. thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] tvalentyn commented on issue #24357: [Bug]: Python SDK: Dependency installation hangs sometimes
Posted by GitBox <gi...@apache.org>.
tvalentyn commented on issue #24357:
URL: https://github.com/apache/beam/issues/24357#issuecomment-1331532800
> In order to read or log it, you have to attach to the container
Dataflow might be doing that, perhaps that is why we never observe such behavior.
> I'll keep testing it
Thanks. If you can confirm that this behavior is unrelated to your change, that would be helpful. If you suspect logging is an issue, you can intentionally flood the logs a little to repro.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org