You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by "Ahmet Altay (JIRA)" <ji...@apache.org> on 2018/03/26 23:02:00 UTC
[jira] [Comment Edited] (BEAM-3933) Beam 2.4 python wheel is not
working on dataflow runners
[ https://issues.apache.org/jira/browse/BEAM-3933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16414734#comment-16414734 ]
Ahmet Altay edited comment on BEAM-3933 at 3/26/18 11:01 PM:
-------------------------------------------------------------
The error is:
I Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-rFQTNL-build/
I You are using pip version 9.0.1, however version 9.0.3 is available.
I You should consider upgrading via the 'pip install --upgrade pip' command.
I /usr/local/bin/pip failed with exit status 1
I Dataflow base path override: https://dataflow.googleapis.com/
I Dataflow base path override: https://dataflow.googleapis.com/
I Installing setup packages ...
I Executing: /usr/local/bin/pip install /var/opt/google/dataflow/dataflow_python_sdk.tar[gcp]
I Created new fluentd log writer for: /var/log/dataflow/boot-json.log
I Processing /var/opt/google/dataflow/dataflow_python_sdk.tar
I Complete output from command python setup.py egg_info:
I Traceback (most recent call last):
I File "<string>", line 1, in <module>
I IOError: [Errno 2] No such file or directory: '/tmp/pip-FtI71Q-build/setup.py'
This is happening because DataflowRunner renames the sdk chosen by the user (by setting --sdk_location) argument to dataflow_python_sdk.tar. And renaming and installing a wheel file fails as above. Related code (https://github.com/apache/beam/blob/597e3f92bc8be692d5d8e8040b33ce0c77350fa2/sdks/python/apache_beam/runners/dataflow/internal/dependency.py#L417) needs to be fixed to name files correctly.
This only affects developing with wheel files. It should be addressed as a task for adding support to wheel files.
cc: [~charleschen] [~robertwb]
was (Author: altay):
The error is:
I Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-rFQTNL-build/
I You are using pip version 9.0.1, however version 9.0.3 is available.
I You should consider upgrading via the 'pip install --upgrade pip' command.
I /usr/local/bin/pip failed with exit status 1
I Dataflow base path override: https://dataflow.googleapis.com/
I Dataflow base path override: https://dataflow.googleapis.com/
I Installing setup packages ...
I Executing: /usr/local/bin/pip install /var/opt/google/dataflow/dataflow_python_sdk.tar[gcp]
I Created new fluentd log writer for: /var/log/dataflow/boot-json.log
I Processing /var/opt/google/dataflow/dataflow_python_sdk.tar
I Complete output from command python setup.py egg_info:
I Traceback (most recent call last):
I File "<string>", line 1, in <module>
I IOError: [Errno 2] No such file or directory: '/tmp/pip-FtI71Q-build/setup.py'
This is happening because DataflowRunner renames the sdk chosen by the user (by setting --sdk_location) argument to dataflow_python_sdk.tar. And renaming and installing a wheel file fails as above. Related code (https://github.com/apache/beam/blob/597e3f92bc8be692d5d8e8040b33ce0c77350fa2/sdks/python/apache_beam/runners/dataflow/internal/dependency.py#L417) needs to be fixed to name files correctly.
Reducing the priority, since this only affects developing with wheel files. It should be addressed as a task for adding support to wheel files.
cc: [~charleschen] [~robertwb]
> Beam 2.4 python wheel is not working on dataflow runners
> --------------------------------------------------------
>
> Key: BEAM-3933
> URL: https://issues.apache.org/jira/browse/BEAM-3933
> Project: Beam
> Issue Type: Bug
> Components: examples-python, sdk-py-harness
> Affects Versions: 2.4.0
> Reporter: yifan zou
> Assignee: Ahmet Altay
> Priority: Major
>
> Running beam examples on DataflowRunner against python wheel are failing due to:
> Error syncing pod aaa0e0f09729d923b17726c465ccfcaf ("dataflow-beamapp-jenkins-032603332-03252033-3d3b-harness-hgb3_default(aaa0e0f09729d923b17726c465ccfcaf)"), skipping: failed to "StartContainer" for "python" with CrashLoopBackOff: "Back-off 5m0s restarting failed container=python pod=dataflow-beamapp-jenkins-032603332-03252033-3d3b-harness-hgb3_default(aaa0e0f09729d923b17726c465ccfcaf)
> Failed to install packages: failed to install SDK: exit status 1
>
> Dataflow job: [https://pantheon.corp.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-03-25_20_33_34-14871401506794256600?project=apache-beam-testing]
> Jenkins job: https://builds.apache.org/job/beam_PostRelease_Python_Candidate/79/console
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)