You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@beam.apache.org by Gareth Western <ga...@garethwestern.com> on 2017/01/23 20:21:53 UTC
Debugging a failed Dataflow job
I'm having trouble running my pipeline using the dataflow runner. The
job is submitted successfully:
Dataflow SDK version: 0.4.0
Submitted job: 2017-01-23_12_17_20-13351949104581541182
but in the Dataflow console it simply says " workflow failed". I can't
seem to find any more details regarding the cause of the failure. Where
should I be looking?
Re: Debugging a failed Dataflow job
Posted by Gareth Western <ga...@garethwestern.com>.
Thanks Chris. That's useful to know.
I cross-posted the question to SO [1] and was told that the error
message was because I had forgotten to authorise the dataflow API for
this project in the Cloud Console (PEBCAK).
1.
https://stackoverflow.com/questions/41815519/cloud-dataflow-job-failed-without-reason/
On 23. jan. 2017 23:34, Chris Fei wrote:
> I believe that documentation refers to the 1.X releases of the Google
> Dataflow SDK. BlockingDataflowPipelineRunner exists there, but it
> doesn't exist in Beam 0.4.0. Take a look at the breaking changes for
> the 2.X releases at
> https://cloud.google.com/dataflow/release-notes/release-notes-java-2,
> which outline how to migrate away from BlockingDataflowPipelineRunner.
>
> You should just be able to specify DataflowRunner in your options, and
> then add the --blockOnRun option.
>
> Chris
>
>
> On Mon, Jan 23, 2017, at 05:03 PM, Gareth Western wrote:
>>
>> Thank you. How does one use the BlockingDataflowPipelineRunner with
>> Beam?
>>
>> Specifying it in the PipelineOptions results in an Exception with a
>> message stating that it is not one of the supported pipeline runners:
>>
>> "Exception in thread "main" java.lang.IllegalArgumentException: Class
>> 'com.google.cloud.dataflow.sdk.runners.BlockingDataflowPipelineRunner'
>> does not implement PipelineRunner. Supported pipeline runners
>> [DataflowRunner, DirectRunner, FlinkRunner, TestFlinkRunner]"
>>
>>
>> On 23. jan. 2017 22:31, Lukasz Cwik wrote:
>>> Please take a look at
>>> https://cloud.google.com/dataflow/pipelines/logging#monitoring-pipeline-logs
>>>
>>> On Mon, Jan 23, 2017 at 12:21 PM, Gareth Western
>>> <gareth@garethwestern.com <ma...@garethwestern.com>> wrote:
>>>
>>> I'm having trouble running my pipeline using the dataflow
>>> runner. The job is submitted successfully:
>>>
>>> Dataflow SDK version: 0.4.0
>>> Submitted job: 2017-01-23_12_17_20-13351949104581541182
>>>
>>> but in the Dataflow console it simply says " workflow failed". I
>>> can't seem to find any more details regarding the cause of the
>>> failure. Where should I be looking?
>>>
>
Re: Debugging a failed Dataflow job
Posted by Chris Fei <cf...@gmail.com>.
I believe that documentation refers to the 1.X releases of the Google
Dataflow SDK. BlockingDataflowPipelineRunner exists there, but it
doesn't exist in Beam 0.4.0. Take a look at the breaking changes for the
2.X releases at
https://cloud.google.com/dataflow/release-notes/release-notes-java-2,
which outline how to migrate away from BlockingDataflowPipelineRunner.
You should just be able to specify DataflowRunner in your options, and
then add the --blockOnRun option.
Chris
On Mon, Jan 23, 2017, at 05:03 PM, Gareth Western wrote:
> Thank you. How does one use the BlockingDataflowPipelineRunner
> with Beam?
> Specifying it in the PipelineOptions results in an Exception with a
> message stating that it is not one of the supported pipeline runners:
> "Exception in thread "main" java.lang.IllegalArgumentException: Class
> 'com.google.cloud.dataflow.sdk.runners.BlockingDataflowPipelineRunner'
> does not implement PipelineRunner. Supported pipeline runners
> [DataflowRunner, DirectRunner, FlinkRunner, TestFlinkRunner]"
>
> On 23. jan. 2017 22:31, Lukasz Cwik wrote:
>> Please take a look at
>> https://cloud.google.com/dataflow/pipelines/logging#monitoring-pipeline-logs
>>
>> On Mon, Jan 23, 2017 at 12:21 PM, Gareth Western
>> <ga...@garethwestern.com> wrote:
>>
>>> I'm having trouble running my pipeline using the dataflow runner.
>>> The job is submitted successfully:
>>>
>>> Dataflow SDK version: 0.4.0
>>> Submitted job: 2017-01-23_12_17_20-13351949104581541182
>>>
>>> but in the Dataflow console it simply says " workflow failed". I
>>> can't seem to find any more details regarding the cause of the
>>> failure. Where should I be looking?
>>>
Re: Debugging a failed Dataflow job
Posted by Gareth Western <ga...@garethwestern.com>.
Thank you. How does one use the BlockingDataflowPipelineRunner with Beam?
Specifying it in the PipelineOptions results in an Exception with a
message stating that it is not one of the supported pipeline runners:
"Exception in thread "main" java.lang.IllegalArgumentException: Class
'com.google.cloud.dataflow.sdk.runners.BlockingDataflowPipelineRunner'
does not implement PipelineRunner. Supported pipeline runners
[DataflowRunner, DirectRunner, FlinkRunner, TestFlinkRunner]"
On 23. jan. 2017 22:31, Lukasz Cwik wrote:
> Please take a look at
> https://cloud.google.com/dataflow/pipelines/logging#monitoring-pipeline-logs
>
> On Mon, Jan 23, 2017 at 12:21 PM, Gareth Western
> <gareth@garethwestern.com <ma...@garethwestern.com>> wrote:
>
> I'm having trouble running my pipeline using the dataflow runner.
> The job is submitted successfully:
>
> Dataflow SDK version: 0.4.0
> Submitted job: 2017-01-23_12_17_20-13351949104581541182
>
> but in the Dataflow console it simply says " workflow failed". I
> can't seem to find any more details regarding the cause of the
> failure. Where should I be looking?
>
>
Re: Debugging a failed Dataflow job
Posted by Lukasz Cwik <lc...@google.com>.
Please take a look at
https://cloud.google.com/dataflow/pipelines/logging#monitoring-pipeline-logs
On Mon, Jan 23, 2017 at 12:21 PM, Gareth Western <ga...@garethwestern.com>
wrote:
> I'm having trouble running my pipeline using the dataflow runner. The job
> is submitted successfully:
>
> Dataflow SDK version: 0.4.0
> Submitted job: 2017-01-23_12_17_20-13351949104581541182
>
> but in the Dataflow console it simply says " workflow failed". I can't
> seem to find any more details regarding the cause of the failure. Where
> should I be looking?
>
>