You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@oozie.apache.org by Deepika Khera <dk...@lyris.com> on 2012/02/10 02:54:20 UTC
Killing coordinator jobs
Hi,
While testing my coordinator.xml I ended up creating a bunch of
coordinator jobs that are on oozie.
I am trying to kill them now and am unable to do so.
I get the error:
Error: E0607 : E0607: Other error in operation
[updateCoordinatorAction], An optimistic lock violation was detected
when flushing object instance "A lock could not be obtained within the
time requested {prepstmnt 1385108855 UPDATE COORD_ACTIONS SET
action_number = ?, action_xml = ?, console_url = NULL, created_conf = ?,
error_code = NULL, error_message = NULL, external_status = NULL,
missing_dependencies = ?, run_conf = ?, time_out = ?, tracker_uri =
NULL, job_type = NULL, created_time = ?, external_id = NULL, job_id = ?,
last_modified_time = ?, nominal_time = ?, sla_xml = NULL, status = ?
WHERE id IN (SELECT DISTINCT t0.id FROM COORD_ACTIONS t0 WHERE (t0.id
= ?) AND t0.bean_type = ?) [params=(int) 4, (String) <coordinator-app
xmlns="uri:oozie:coordinator:0.1" name="clickstream..., (String)
<configuration> <property> <name>oozie.coord.application.pat...,
(String)
hdfs://localhost:54310/usr/org/data/clickstream/input/2012/02/10/{...,
(String) <configuration> <property>
<name>oozie.coord.application.pat..., (int) 120, (Timestamp) 2012-02-09
17:38:59.035, (String) 0000009-120209131354394-oozie-oozi-C, (Timestamp)
2012-02-09 17:49:42.324, (Timestamp) 2012-02-09 17:35:00.0, (String)
KILLED, (String) 0000009-120209131354394-oozie-oozi-C@4, (String)
CoordinatorActionBean]} [code=30000, state=40XL1] [java.lang.String]" to
the data store. This indicates that the object was concurrently
modified in another transaction.
Seems like since all the coordinators are waiting on the same input
directory, its not letting me kill it.
Or maybe something else.
This is my local cluster and if there is a way to kill all jobs I can
even do that.
Would appreciate any pointers on how to resolve this.
Thanks,
Deepika
Re: Killing coordinator jobs
Posted by Mohammad Islam <mi...@yahoo.com>.
Wandering why don't you got the error message when you ask to kill.
Looks like it is a bug. If you think so, please create a JIRA at :
http://issues.apache.org/jira/browse/OOZIE
Regards,
Mohammad
----- Original Message -----
From: Deepika Khera <dk...@lyris.com>
To: oozie-users@incubator.apache.org
Cc:
Sent: Friday, February 10, 2012 12:56 PM
Subject: Re: Killing coordinator jobs
Thanks Mohammad. I reinstalled oozie and got rid of the coordinator
jobs. So the locking exceptions went away.
Still I was not able to kill jobs though.
Turned out the issue was that though jobs were running as user "oozie" ,
I was trying to kill them with my own user. Once I fixed that , I was
able to kill jobs.
Regards,
Deepika
On Fri, 2012-02-10 at 02:40 -0800, Mohammad Islam wrote:
> Hi Deepika,
> It is hard to debug this remotely and find the root cause.
>
> Do you see this exception when you send the Kill command?
> Or this exception is coming on and off.
> You can verify this by running "oozie kill" command and tail on that log.
>
> one easy and hacky way is to stop oozie service and restart and then kill.
> Another question: what oozie version are you using and what is the DB ?
> Regards,
> Mohammad
>
> ________________________________
> From: Deepika Khera <dk...@lyris.com>
> To: "oozie-users@incubator.apache.org" <oo...@incubator.apache.org>
> Sent: Thursday, February 9, 2012 5:54 PM
> Subject: Killing coordinator jobs
>
> Hi,
>
> While testing my coordinator.xml I ended up creating a bunch of
> coordinator jobs that are on oozie.
>
> I am trying to kill them now and am unable to do so.
>
> I get the error:
> Error: E0607 : E0607: Other error in operation
> [updateCoordinatorAction], An optimistic lock violation was detected
> when flushing object instance "A lock could not be obtained within the
> time requested {prepstmnt 1385108855 UPDATE COORD_ACTIONS SET
> action_number = ?, action_xml = ?, console_url = NULL, created_conf = ?,
> error_code = NULL, error_message = NULL, external_status = NULL,
> missing_dependencies = ?, run_conf = ?, time_out = ?, tracker_uri =
> NULL, job_type = NULL, created_time = ?, external_id = NULL, job_id = ?,
> last_modified_time = ?, nominal_time = ?, sla_xml = NULL, status = ?
> WHERE id IN (SELECT DISTINCT t0.id FROM COORD_ACTIONS t0 WHERE (t0.id
> = ?) AND t0.bean_type = ?) [params=(int) 4, (String) <coordinator-app
> xmlns="uri:oozie:coordinator:0.1" name="clickstream..., (String)
> <configuration> <property> <name>oozie.coord.application.pat...,
> (String)
> hdfs://localhost:54310/usr/org/data/clickstream/input/2012/02/10/{...,
> (String) <configuration> <property>
> <name>oozie.coord.application.pat..., (int) 120, (Timestamp) 2012-02-09
> 17:38:59.035, (String) 0000009-120209131354394-oozie-oozi-C, (Timestamp)
> 2012-02-09 17:49:42.324, (Timestamp) 2012-02-09 17:35:00.0, (String)
> KILLED, (String) 0000009-120209131354394-oozie-oozi-C@4, (String)
> CoordinatorActionBean]} [code=30000, state=40XL1] [java.lang.String]" to
> the data store. This indicates that the object was concurrently
> modified in another transaction.
>
>
>
> Seems like since all the coordinators are waiting on the same input
> directory, its not letting me kill it.
> Or maybe something else.
> This is my local cluster and if there is a way to kill all jobs I can
> even do that.
>
> Would appreciate any pointers on how to resolve this.
>
> Thanks,
> Deepika
Re: Killing coordinator jobs
Posted by Deepika Khera <dk...@lyris.com>.
Thanks Mohammad. I reinstalled oozie and got rid of the coordinator
jobs. So the locking exceptions went away.
Still I was not able to kill jobs though.
Turned out the issue was that though jobs were running as user "oozie" ,
I was trying to kill them with my own user. Once I fixed that , I was
able to kill jobs.
Regards,
Deepika
On Fri, 2012-02-10 at 02:40 -0800, Mohammad Islam wrote:
> Hi Deepika,
> It is hard to debug this remotely and find the root cause.
>
> Do you see this exception when you send the Kill command?
> Or this exception is coming on and off.
> You can verify this by running "oozie kill" command and tail on that log.
>
> one easy and hacky way is to stop oozie service and restart and then kill.
> Another question: what oozie version are you using and what is the DB ?
> Regards,
> Mohammad
>
> ________________________________
> From: Deepika Khera <dk...@lyris.com>
> To: "oozie-users@incubator.apache.org" <oo...@incubator.apache.org>
> Sent: Thursday, February 9, 2012 5:54 PM
> Subject: Killing coordinator jobs
>
> Hi,
>
> While testing my coordinator.xml I ended up creating a bunch of
> coordinator jobs that are on oozie.
>
> I am trying to kill them now and am unable to do so.
>
> I get the error:
> Error: E0607 : E0607: Other error in operation
> [updateCoordinatorAction], An optimistic lock violation was detected
> when flushing object instance "A lock could not be obtained within the
> time requested {prepstmnt 1385108855 UPDATE COORD_ACTIONS SET
> action_number = ?, action_xml = ?, console_url = NULL, created_conf = ?,
> error_code = NULL, error_message = NULL, external_status = NULL,
> missing_dependencies = ?, run_conf = ?, time_out = ?, tracker_uri =
> NULL, job_type = NULL, created_time = ?, external_id = NULL, job_id = ?,
> last_modified_time = ?, nominal_time = ?, sla_xml = NULL, status = ?
> WHERE id IN (SELECT DISTINCT t0.id FROM COORD_ACTIONS t0 WHERE (t0.id
> = ?) AND t0.bean_type = ?) [params=(int) 4, (String) <coordinator-app
> xmlns="uri:oozie:coordinator:0.1" name="clickstream..., (String)
> <configuration> <property> <name>oozie.coord.application.pat...,
> (String)
> hdfs://localhost:54310/usr/org/data/clickstream/input/2012/02/10/{...,
> (String) <configuration> <property>
> <name>oozie.coord.application.pat..., (int) 120, (Timestamp) 2012-02-09
> 17:38:59.035, (String) 0000009-120209131354394-oozie-oozi-C, (Timestamp)
> 2012-02-09 17:49:42.324, (Timestamp) 2012-02-09 17:35:00.0, (String)
> KILLED, (String) 0000009-120209131354394-oozie-oozi-C@4, (String)
> CoordinatorActionBean]} [code=30000, state=40XL1] [java.lang.String]" to
> the data store. This indicates that the object was concurrently
> modified in another transaction.
>
>
>
> Seems like since all the coordinators are waiting on the same input
> directory, its not letting me kill it.
> Or maybe something else.
> This is my local cluster and if there is a way to kill all jobs I can
> even do that.
>
> Would appreciate any pointers on how to resolve this.
>
> Thanks,
> Deepika
Re: Killing coordinator jobs
Posted by Mohammad Islam <mi...@yahoo.com>.
Hi Deepika,
It is hard to debug this remotely and find the root cause.
Do you see this exception when you send the Kill command?
Or this exception is coming on and off.
You can verify this by running "oozie kill" command and tail on that log.
one easy and hacky way is to stop oozie service and restart and then kill.
Another question: what oozie version are you using and what is the DB ?
Regards,
Mohammad
________________________________
From: Deepika Khera <dk...@lyris.com>
To: "oozie-users@incubator.apache.org" <oo...@incubator.apache.org>
Sent: Thursday, February 9, 2012 5:54 PM
Subject: Killing coordinator jobs
Hi,
While testing my coordinator.xml I ended up creating a bunch of
coordinator jobs that are on oozie.
I am trying to kill them now and am unable to do so.
I get the error:
Error: E0607 : E0607: Other error in operation
[updateCoordinatorAction], An optimistic lock violation was detected
when flushing object instance "A lock could not be obtained within the
time requested {prepstmnt 1385108855 UPDATE COORD_ACTIONS SET
action_number = ?, action_xml = ?, console_url = NULL, created_conf = ?,
error_code = NULL, error_message = NULL, external_status = NULL,
missing_dependencies = ?, run_conf = ?, time_out = ?, tracker_uri =
NULL, job_type = NULL, created_time = ?, external_id = NULL, job_id = ?,
last_modified_time = ?, nominal_time = ?, sla_xml = NULL, status = ?
WHERE id IN (SELECT DISTINCT t0.id FROM COORD_ACTIONS t0 WHERE (t0.id
= ?) AND t0.bean_type = ?) [params=(int) 4, (String) <coordinator-app
xmlns="uri:oozie:coordinator:0.1" name="clickstream..., (String)
<configuration> <property> <name>oozie.coord.application.pat...,
(String)
hdfs://localhost:54310/usr/org/data/clickstream/input/2012/02/10/{...,
(String) <configuration> <property>
<name>oozie.coord.application.pat..., (int) 120, (Timestamp) 2012-02-09
17:38:59.035, (String) 0000009-120209131354394-oozie-oozi-C, (Timestamp)
2012-02-09 17:49:42.324, (Timestamp) 2012-02-09 17:35:00.0, (String)
KILLED, (String) 0000009-120209131354394-oozie-oozi-C@4, (String)
CoordinatorActionBean]} [code=30000, state=40XL1] [java.lang.String]" to
the data store. This indicates that the object was concurrently
modified in another transaction.
Seems like since all the coordinators are waiting on the same input
directory, its not letting me kill it.
Or maybe something else.
This is my local cluster and if there is a way to kill all jobs I can
even do that.
Would appreciate any pointers on how to resolve this.
Thanks,
Deepika