You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2022/06/04 20:47:21 UTC
[GitHub] [beam] damccorm opened a new issue, #20973: Python Beam SDK Harness hangs when installing pip packages
damccorm opened a new issue, #20973:
URL: https://github.com/apache/beam/issues/20973
When running a Beam pipeline using Flink as backend, the python sdk harness hangs when trying to install pip packages. Tested using Flink 1.10.3.
Images used:
apache/beam_python3.7_sdk:2.28.0
apache/flink:1.10.3
Beam args used are:
"\--runner=FlinkRunner",
"–flink_version=1.10", //same with 1.13
"\--flink_master=[http://flink-jobmanager.default:8081](http://flink-jobmanager.default:8081/)",
f"\--artifacts_dir=/mnt/flink",
"\--environment_type=EXTERNAL",
"\--environment_config=localhost:50000",
Specifically this was tested by running a TFX pipeline which gets submitted and registered as it should, but the SDK Harness hangs when installing:
2021/03/10 12:16:20 Initializing python harness: /opt/apache/beam/boot \--id=1-1 \--logging_endpoint=localhost:39795 \--artifact_endpoint=localhost:34095 \--provision_endpoint=localhost:42999 \--control_endpoint=localhost:38129
2021/03/10 12:16:20 Found artifact: tfx_ephemeral-0.27.0.tar.gz
2021/03/10 12:16:20 Found artifact: extra_packages.txt
2021/03/10 12:16:20 Installing setup packages ...
2021/03/10 12:16:20 Installing extra package: tfx_ephemeral-0.27.0.tar.gz
and nothing else is shown irregardless how long it is left. I can manually install the TFX package by exec into the container in < 3 min.
The Flink task-manager then waits idling and periodically logs:
2021-03-10 11:29:26,287 INFO org.apache.beam.runners.fnexecution.environment.ExternalEnvironmentFactory - Still waiting for startup of environment from localhost:50000 for worker id 1-1
Helm charts attached below.
Imported from Jira [BEAM-11959](https://issues.apache.org/jira/browse/BEAM-11959). Original Jira may contain additional context.
Reported by: ConverJens.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] KevinGG commented on issue #20973: Python Beam SDK Harness hangs when installing pip packages
Posted by GitBox <gi...@apache.org>.
KevinGG commented on issue #20973:
URL: https://github.com/apache/beam/issues/20973#issuecomment-1238480433
> @KevinGG Beam has several options to propagate dependencies, in this case we used the `--extra-package` flag. We had the same behaviour when submitting setup.py or requirements.txt files though.
I don't believe I know the answer. I'm running into other issues when sending a job with a requirements file to a remotely started flink cluster executing in DOCKER environment. I don't understand how a requirements file is staged from job submission to later execution in containers. So I'm trying to see if you've had similar issues.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] ConverJens commented on issue #20973: Python Beam SDK Harness hangs when installing pip packages
Posted by GitBox <gi...@apache.org>.
ConverJens commented on issue #20973:
URL: https://github.com/apache/beam/issues/20973#issuecomment-1239029319
@KevinGG
When you specify a requirements file when starting your beam job, beam compresses your file and sends it to the Flink job master. The job master distributes it ti the worker nodes.
But if you have an issue with this, I would encourage you to open a new issue with more information.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] KevinGG commented on issue #20973: Python Beam SDK Harness hangs when installing pip packages
Posted by GitBox <gi...@apache.org>.
KevinGG commented on issue #20973:
URL: https://github.com/apache/beam/issues/20973#issuecomment-1235065911
@damccorm Hi Danny, out of curiosity, how did you handle the extra pip dependencies in your use case? Do you have to pip freeze into a requirements file and somehow stage it to the Flink job server through artifacts_dir?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] damccorm commented on issue #20973: Python Beam SDK Harness hangs when installing pip packages
Posted by GitBox <gi...@apache.org>.
damccorm commented on issue #20973:
URL: https://github.com/apache/beam/issues/20973#issuecomment-1235373160
Hey @KevinGG I didn't actually report this issue, I just imported it from Jira. I believe @ConverJens reported it, there's also more detail in the linked Jira in the description
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] ConverJens commented on issue #20973: Python Beam SDK Harness hangs when installing pip packages
Posted by GitBox <gi...@apache.org>.
ConverJens commented on issue #20973:
URL: https://github.com/apache/beam/issues/20973#issuecomment-1235510355
@KevinGG Beam has several options to propagate dependencies, in this case we used the `--extra-package` flag.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Python Beam SDK Harness hangs when installing pip packages [beam]
Posted by "pritamdodeja (via GitHub)" <gi...@apache.org>.
pritamdodeja commented on issue #20973:
URL: https://github.com/apache/beam/issues/20973#issuecomment-4763074024
I think the reason this hangs is because the SDK harness is unable to find the artifacts staged by tensorflow transform. I don't think this is a beam issue. Flink complains that the environment is not up, and the reason the environment is not up is because kfp is looking for some of these things in /tmp/staged, whereas they exist under _wheels under default Pipeline root. If this was truly a beam issue, it would show up in a lot of other places in my opinion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org