You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@beam.apache.org by Carlos Alonso <ca...@mrcalonso.com> on 2018/01/22 21:28:19 UTC
Google Dataflow jobs stuck analysing the graph
We have submitted a couple of jobs that seem to have stuck on the graph
analysing step.
An weird A job with ID "2018-01-19_03_27_48-15138951989738918594" doesn't
exist error appears on top of the Google Dataflow jobs list page and trying
to list it using gcloud tool shows them as in Unknown state:
JOB_ID NAME TYPE CREATION_TIME STATE REGION
2018-01-19_03_27_48-15138951989738918594 myjobname2 Streaming 2018-01-19
12:27:48 Unknown europe-west1 2018-01-19_03_21_05-1065108814086334743
myjobname Streaming 2018-01-19 12:21:06 Unknown europe-west1
Trying to cancel them using gcloud tool as well doesn't work either:
~ gcloud beta dataflow jobs --project=myproject cancel
2018-01-19_03_21_05-1065108814086334743 Failed to cancel job
[2018-01-19_03_21_05-1065108814086334743]: (9027838c1500ddff): Could not
cancel workflow; user does not have sufficient permissions on project:
myproject, or the job does not exist in the project.
Any idea?
Thanks!!
Re: Google Dataflow jobs stuck analysing the graph
Posted by Lukasz Cwik <lc...@google.com>.
Try:
gcloud beta dataflow jobs --project=myproject cancel myjobname
gcloud beta dataflow jobs --project=myproject cancel myjobname2
On Mon, Jan 22, 2018 at 1:28 PM, Carlos Alonso <ca...@mrcalonso.com> wrote:
> We have submitted a couple of jobs that seem to have stuck on the graph
> analysing step.
>
> An weird A job with ID "2018-01-19_03_27_48-15138951989738918594" doesn't
> exist error appears on top of the Google Dataflow jobs list page and
> trying to list it using gcloud tool shows them as in Unknown state:
>
> JOB_ID NAME TYPE CREATION_TIME STATE REGION 2018-01-19_03_27_48-15138951989738918594
> myjobname2 Streaming 2018-01-19 12:27:48 Unknown europe-west1
> 2018-01-19_03_21_05-1065108814086334743 myjobname Streaming 2018-01-19
> 12:21:06 Unknown europe-west1
>
> Trying to cancel them using gcloud tool as well doesn't work either:
>
> ~ gcloud beta dataflow jobs --project=myproject cancel 2018-01-19_03_21_05-1065108814086334743
> Failed to cancel job [2018-01-19_03_21_05-1065108814086334743]:
> (9027838c1500ddff): Could not cancel workflow; user does not have
> sufficient permissions on project: myproject, or the job does not exist in
> the project.
>
> Any idea?
>
> Thanks!!
>
Re: Google Dataflow jobs stuck analysing the graph
Posted by Carlos Alonso <ca...@mrcalonso.com>.
Finally the jobs were errored because I ran another job that took all the
available quota for that region and the stuck jobs failed as they could not
acquire the required resources.
On Tue, Jan 23, 2018 at 10:20 AM Carlos Alonso <ca...@mrcalonso.com> wrote:
> Many thanks for your input.
>
> Unfortunately, neither of the proposals are working. When trying from the
> command line (using the job name instead of the id), same error appears
> `Failed to cancel job [myjobname]: (8b60d3b239f21d49): Could not cancel
> workflow; user does not have sufficient permissions on project: myproject,
> or the job does not exist in the project.`
>
> And I cannot cancel it on the UI as, when I click on the job details, a
> blank screen with only the message `The graph is still being analysed`
> appears.
>
> Regards
>
> On Tue, Jan 23, 2018 at 12:22 AM Alex Amato <aj...@google.com> wrote:
>
>> Hi Carlos,
>>
>> Can you try cancelling your job using the job name "*myjobname*"
>> instead of the job_id. I think that is the proper way to cancel from the
>> command line.
>>
>> Otherwise, you can try to find the job in the dataflow UI and click the
>> cancel button on the webpage to cancel your job, if this is a onetime thing.
>>
>> I believe that this should be the proper way to do this on the command
>> line:
>> gcloud beta dataflow jobs --project=myproject cancel myjobname
>> …
>>
>> On Mon, Jan 22, 2018 at 1:31 PM Lukasz Cwik <lc...@google.com> wrote:
>>
>>>
>>> ---------- Forwarded message ----------
>>> From: Carlos Alonso <ca...@mrcalonso.com>
>>> Date: Mon, Jan 22, 2018 at 1:28 PM
>>> Subject: Google Dataflow jobs stuck analysing the graph
>>> To: "user@beam.apache.org" <us...@beam.apache.org>
>>>
>>>
>>> We have submitted a couple of jobs that seem to have stuck on the graph
>>> analysing step.
>>>
>>> An weird A job with ID "2018-01-19_03_27_48-15138951989738918594"
>>> doesn't exist error appears on top of the Google Dataflow jobs list
>>> page and trying to list it using gcloud tool shows them as in Unknown
>>> state:
>>>
>>> JOB_ID NAME TYPE CREATION_TIME STATE REGION
>>> 2018-01-19_03_27_48-15138951989738918594 myjobname2 Streaming 2018-01-19
>>> 12:27:48 Unknown europe-west1 2018-01-19_03_21_05-1065108814086334743
>>> myjobname Streaming 2018-01-19 12:21:06 Unknown europe-west1
>>>
>>> Trying to cancel them using gcloud tool as well doesn't work either:
>>>
>>> ~ gcloud beta dataflow jobs --project=myproject cancel
>>> 2018-01-19_03_21_05-1065108814086334743 Failed to cancel job
>>> [2018-01-19_03_21_05-1065108814086334743]: (9027838c1500ddff): Could not
>>> cancel workflow; user does not have sufficient permissions on project:
>>> myproject, or the job does not exist in the project.
>>>
>>> Any idea?
>>>
>>> Thanks!!
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "dataflow-feedback" group.
>>> To view this discussion on the web visit
>>> https://groups.google.com/a/google.com/d/msgid/dataflow-feedback/CAF9t7_6V25%2B9U2ZPPMOGxVF0OXZ-orz%3D4Djnkz_ttA%2B8%2B_W%3D0A%40mail.gmail.com
>>> <https://groups.google.com/a/google.com/d/msgid/dataflow-feedback/CAF9t7_6V25%2B9U2ZPPMOGxVF0OXZ-orz%3D4Djnkz_ttA%2B8%2B_W%3D0A%40mail.gmail.com?utm_medium=email&utm_source=footer>
>>> .
>>>
>>
Re: Google Dataflow jobs stuck analysing the graph
Posted by Carlos Alonso <ca...@mrcalonso.com>.
Many thanks for your input.
Unfortunately, neither of the proposals are working. When trying from the
command line (using the job name instead of the id), same error appears
`Failed to cancel job [myjobname]: (8b60d3b239f21d49): Could not cancel
workflow; user does not have sufficient permissions on project: myproject,
or the job does not exist in the project.`
And I cannot cancel it on the UI as, when I click on the job details, a
blank screen with only the message `The graph is still being analysed`
appears.
Regards
On Tue, Jan 23, 2018 at 12:22 AM Alex Amato <aj...@google.com> wrote:
> Hi Carlos,
>
> Can you try cancelling your job using the job name "*myjobname*" instead
> of the job_id. I think that is the proper way to cancel from the command
> line.
>
> Otherwise, you can try to find the job in the dataflow UI and click the
> cancel button on the webpage to cancel your job, if this is a onetime thing.
>
> I believe that this should be the proper way to do this on the command
> line:
> gcloud beta dataflow jobs --project=myproject cancel myjobname
> …
>
> On Mon, Jan 22, 2018 at 1:31 PM Lukasz Cwik <lc...@google.com> wrote:
>
>>
>> ---------- Forwarded message ----------
>> From: Carlos Alonso <ca...@mrcalonso.com>
>> Date: Mon, Jan 22, 2018 at 1:28 PM
>> Subject: Google Dataflow jobs stuck analysing the graph
>> To: "user@beam.apache.org" <us...@beam.apache.org>
>>
>>
>> We have submitted a couple of jobs that seem to have stuck on the graph
>> analysing step.
>>
>> An weird A job with ID "2018-01-19_03_27_48-15138951989738918594"
>> doesn't exist error appears on top of the Google Dataflow jobs list page
>> and trying to list it using gcloud tool shows them as in Unknown state:
>>
>> JOB_ID NAME TYPE CREATION_TIME STATE REGION
>> 2018-01-19_03_27_48-15138951989738918594 myjobname2 Streaming 2018-01-19
>> 12:27:48 Unknown europe-west1 2018-01-19_03_21_05-1065108814086334743
>> myjobname Streaming 2018-01-19 12:21:06 Unknown europe-west1
>>
>> Trying to cancel them using gcloud tool as well doesn't work either:
>>
>> ~ gcloud beta dataflow jobs --project=myproject cancel
>> 2018-01-19_03_21_05-1065108814086334743 Failed to cancel job
>> [2018-01-19_03_21_05-1065108814086334743]: (9027838c1500ddff): Could not
>> cancel workflow; user does not have sufficient permissions on project:
>> myproject, or the job does not exist in the project.
>>
>> Any idea?
>>
>> Thanks!!
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "dataflow-feedback" group.
>> To view this discussion on the web visit
>> https://groups.google.com/a/google.com/d/msgid/dataflow-feedback/CAF9t7_6V25%2B9U2ZPPMOGxVF0OXZ-orz%3D4Djnkz_ttA%2B8%2B_W%3D0A%40mail.gmail.com
>> <https://groups.google.com/a/google.com/d/msgid/dataflow-feedback/CAF9t7_6V25%2B9U2ZPPMOGxVF0OXZ-orz%3D4Djnkz_ttA%2B8%2B_W%3D0A%40mail.gmail.com?utm_medium=email&utm_source=footer>
>> .
>>
>