You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@beam.apache.org by Chris Swart <ch...@healx.io> on 2019/04/22 15:05:13 UTC
Debugging Python Dataflow workflow installation failure
Hey,
I am working migrating some of our Dataflow pipelines from beam 2.4.0 and
facing issues with `Failed to install packages: failed to install workflow:
exit status 1`
My approach to debug this so far as has been building the package tar in a
Docker container and installing form there to see if there is any trouble
within my `setup.py`. I can build my package without issues in the Docker
container and running with `DirectRunner` also work but the above error
will still persist still when running with the `DataflowRunner`. I have
tried installing and running with `DirectRunner` Dataflow Docker images in
my case: `dataflow.gcr.io/v1beta3/python:2.6.0`
<http://dataflow.gcr.io/v1beta3/python:2.6.0> and this still has the same
issue.
How can I debug this issue? I find the Dataflow Runner context is quite
opaque to work with the Dataflow docs sadly don't have anything to help
with such issues either:
https://cloud.google.com/dataflow/docs/guides/troubleshooting-your-pipeline
Best wishes,
Chris
Re: Debugging Python Dataflow workflow installation failure
Posted by Ahmet Altay <al...@google.com>.
Hi Chris,
I am guess this is the same question as the StackOverflow question (
https://stackoverflow.com/questions/55729163/google-dataflow-with-python-failed-to-install-workflow-exit-status-1).
I dropped a comment there for clarification. Let's use StackOverflow for
this question.
Ahmet
On Mon, Apr 22, 2019 at 10:40 AM Niklas Hansson <
niklas.sven.hansson@gmail.com> wrote:
> Don’t think it is linked :) was a bit fast to answer before. Don’t know
> how to debug. But home some one else can help out.
>
> On Mon, 22 Apr 2019 at 17:08, Niklas Hansson <
> niklas.sven.hansson@gmail.com> wrote:
>
>> Hi,
>>
>> Could be due to PR relate to Beam [3072]. Will check later
>>
>>
>>
>> On Mon, 22 Apr 2019 at 17:05, Chris Swart <ch...@healx.io> wrote:
>>
>>> Hey,
>>>
>>> I am working migrating some of our Dataflow pipelines from beam 2.4.0
>>> and facing issues with `Failed to install packages: failed to install
>>> workflow: exit status 1`
>>>
>>> My approach to debug this so far as has been building the package tar in
>>> a Docker container and installing form there to see if there is any trouble
>>> within my `setup.py`. I can build my package without issues in the Docker
>>> container and running with `DirectRunner` also work but the above error
>>> will still persist still when running with the `DataflowRunner`. I have
>>> tried installing and running with `DirectRunner` Dataflow Docker images in
>>> my case: `dataflow.gcr.io/v1beta3/python:2.6.0`
>>> <http://dataflow.gcr.io/v1beta3/python:2.6.0> and this still has the
>>> same issue.
>>>
>>> How can I debug this issue? I find the Dataflow Runner context is quite
>>> opaque to work with the Dataflow docs sadly don't have anything to help
>>> with such issues either:
>>> https://cloud.google.com/dataflow/docs/guides/troubleshooting-your-pipeline
>>>
>>> Best wishes,
>>> Chris
>>>
>>
Re: Debugging Python Dataflow workflow installation failure
Posted by Niklas Hansson <ni...@gmail.com>.
Don’t think it is linked :) was a bit fast to answer before. Don’t know how
to debug. But home some one else can help out.
On Mon, 22 Apr 2019 at 17:08, Niklas Hansson <ni...@gmail.com>
wrote:
> Hi,
>
> Could be due to PR relate to Beam [3072]. Will check later
>
>
>
> On Mon, 22 Apr 2019 at 17:05, Chris Swart <ch...@healx.io> wrote:
>
>> Hey,
>>
>> I am working migrating some of our Dataflow pipelines from beam 2.4.0 and
>> facing issues with `Failed to install packages: failed to install workflow:
>> exit status 1`
>>
>> My approach to debug this so far as has been building the package tar in
>> a Docker container and installing form there to see if there is any trouble
>> within my `setup.py`. I can build my package without issues in the Docker
>> container and running with `DirectRunner` also work but the above error
>> will still persist still when running with the `DataflowRunner`. I have
>> tried installing and running with `DirectRunner` Dataflow Docker images in
>> my case: `dataflow.gcr.io/v1beta3/python:2.6.0`
>> <http://dataflow.gcr.io/v1beta3/python:2.6.0> and this still has the
>> same issue.
>>
>> How can I debug this issue? I find the Dataflow Runner context is quite
>> opaque to work with the Dataflow docs sadly don't have anything to help
>> with such issues either:
>> https://cloud.google.com/dataflow/docs/guides/troubleshooting-your-pipeline
>>
>> Best wishes,
>> Chris
>>
>
Re: Debugging Python Dataflow workflow installation failure
Posted by Niklas Hansson <ni...@gmail.com>.
Hi,
Could be due to PR relate to Beam [3072]. Will check later
On Mon, 22 Apr 2019 at 17:05, Chris Swart <ch...@healx.io> wrote:
> Hey,
>
> I am working migrating some of our Dataflow pipelines from beam 2.4.0 and
> facing issues with `Failed to install packages: failed to install workflow:
> exit status 1`
>
> My approach to debug this so far as has been building the package tar in a
> Docker container and installing form there to see if there is any trouble
> within my `setup.py`. I can build my package without issues in the Docker
> container and running with `DirectRunner` also work but the above error
> will still persist still when running with the `DataflowRunner`. I have
> tried installing and running with `DirectRunner` Dataflow Docker images in
> my case: `dataflow.gcr.io/v1beta3/python:2.6.0`
> <http://dataflow.gcr.io/v1beta3/python:2.6.0> and this still has the same
> issue.
>
> How can I debug this issue? I find the Dataflow Runner context is quite
> opaque to work with the Dataflow docs sadly don't have anything to help
> with such issues either:
> https://cloud.google.com/dataflow/docs/guides/troubleshooting-your-pipeline
>
> Best wishes,
> Chris
>