You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@oozie.apache.org by "Shwetha G S (JIRA)" <ji...@apache.org> on 2013/11/22 08:13:35 UTC
[jira] [Commented] (OOZIE-1622) Multiple CoordSubmit for same
bundle
[ https://issues.apache.org/jira/browse/OOZIE-1622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13829743#comment-13829743 ]
Shwetha G S commented on OOZIE-1622:
------------------------------------
Looks like this code is the issue in RecoveryService.runBundleRecovery():
{noformat}
if (baction.getCoordId() == null) {
log.error("CoordId is null for Bundle action " + baction.getBundleActionId());
continue;
}
if (Services.get().get(JobsConcurrencyService.class).isJobIdForThisServer(baction.getCoordId())) {
if (baction.getStatus() == Job.Status.PREP) {
BundleJobBean bundleJob = null;
if (jpaService != null) {
bundleJob = BundleJobQueryExecutor.getInstance().get(
BundleJobQuery.GET_BUNDLE_JOB_ID_JOBXML_CONF, baction.getBundleId());
}
if (bundleJob != null) {
Element bAppXml = XmlUtils.parseXml(bundleJob.getJobXml());
List<Element> coordElems = bAppXml.getChildren("coordinator", bAppXml.getNamespace());
for (Element coordElem : coordElems) {
Attribute name = coordElem.getAttribute("name");
if (name.getValue().equals(baction.getCoordName())) {
Configuration coordConf = mergeConfig(coordElem, bundleJob);
coordConf.set(OozieClient.BUNDLE_ID, baction.getBundleId());
queueCallable(new CoordSubmitXCommand(coordConf,
bundleJob.getId(), name.getValue()));
}
}
}
}
{noformat}
This code creates new coord if the bundle action is in PREP and is pending and coord id is not null. Coord id not null means that coord is already created. Why create another one?
In this particular instance, bundle action was in pending state as bundle job was suspended, and coord was still in PREP state as materialisation didn't happen(I don't have debug logs, so don't know why materialisation didn't happen for 17 mins(which is another weird scenario which needs to be debugged)). Because coord action was in PREP state, I guess even bundle action was in PEP state. But this is a valid scenario to happen
> Multiple CoordSubmit for same bundle
> ------------------------------------
>
> Key: OOZIE-1622
> URL: https://issues.apache.org/jira/browse/OOZIE-1622
> Project: Oozie
> Issue Type: Bug
> Reporter: Shwetha G S
> Priority: Critical
>
> We saw a weird instance where multiple coords were created for same bundle id when the bundle was supposed to have just 1 coordinator. Here are the oozie logs:
> {noformat}
> 2013-11-19 09:09:46,473 INFO BundleStartXCommand:539 - USER[fetl] GROUP[-] TOKEN[] APP[<app name>] JOB[0484436-131016085136608-oozie-oozi-B] ACTION[-] Bundle 0484436-131016085136608-oozie-oozi-B is not in PREP status. It is in : RUNNING
> 2013-11-19 09:09:46,473 WARN BundleStartXCommand:542 - USER[fetl] GROUP[-] TOKEN[] APP[<app name>] JOB[0484436-131016085136608-oozie-oozi-B] ACTION[-] E1100: Command precondition does not hold before execution, [Bundle 0484436-131016085136608-oozie-oozi-B is not in PREP status. It is in : RUNNING], Error Code: E1100
> 2013-11-19 09:09:46,473 INFO CoordSubmitXCommand:539 - USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0484436-131016085136608-oozie-oozi-B] ACTION[-] STARTED Coordinator Submit
> 2013-11-19 09:09:46,483 INFO CoordSubmitXCommand:539 - USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0484436-131016085136608-oozie-oozi-B] ACTION[-] configDefault Doesn't exist
> 2013-11-19 09:09:46,515 INFO CoordSubmitXCommand:539 - USER[fetl] GROUP[-] TOKEN[] APP[<app name>] JOB[0484437-131016085136608-oozie-oozi-C] ACTION[-] ENDED Coordinator Submit jobId=0484437-131016085136608-oozie-oozi-C
> 2013-11-19 09:09:46,529 INFO BundleStatusUpdateXCommand:539 - USER[fetl] GROUP[-] TOKEN[] APP[<app name>] JOB[0484437-131016085136608-oozie-oozi-C] ACTION[-] Updated bundle action [0484436-131016085136608-oozie-oozi-B_<app name>] from prev status [PREP] to current coord status [PREP], and new bundle action pending [0]
> 2013-11-19 09:09:46,535 INFO CoordMaterializeTransitionXCommand:539 - USER[fetl] GROUP[-] TOKEN[] APP[<app name>] JOB[0484437-131016085136608-oozie-oozi-C] ACTION[-] materialize actions for tz=Coordinated Universal Time,
> 2013-11-19 09:09:54,590 INFO StatusTransitService$StatusTransitRunnable:539 - USER[-] GROUP[-] Set bundle job [0484436-131016085136608-oozie-oozi-B] status to 'RUNNING' from 'RUNNING'
> 2013-11-19 09:09:54,590 INFO StatusTransitService$StatusTransitRunnable:539 - USER[-] GROUP[-] Bundle job [0484436-131016085136608-oozie-oozi-B] Pending set to FALSE
> 2013-11-19 09:10:16,326 INFO CoordMaterializeTriggerService$CoordMaterializeTriggerRunnable:539 - USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] Job :0484437-131016085136608-oozie-oozi-C numWaitingActions : 0 MatThrottle : 60
> 2013-11-19 09:12:57,246 INFO StatusTransitService$StatusTransitRunnable:539 - USER[-] GROUP[-] Set bundle job [0484436-131016085136608-oozie-oozi-B] status to 'SUSPENDED' from 'SUSPENDED'
> 2013-11-19 09:12:57,246 INFO StatusTransitService$StatusTransitRunnable:539 - USER[-] GROUP[-] Bundle job [0484436-131016085136608-oozie-oozi-B] Pending set to TRUE
> 2013-11-19 09:13:16,410 INFO CoordMaterializeTriggerService$CoordMaterializeTriggerRunnable:539 - USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] Job :0484437-131016085136608-oozie-oozi-C numWaitingActions : 0 MatThrottle : 60
> 2013-11-19 09:16:16,446 INFO CoordMaterializeTriggerService$CoordMaterializeTriggerRunnable:539 - USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] Job :0484437-131016085136608-oozie-oozi-C numWaitingActions : 0 MatThrottle : 60
> 2013-11-19 09:17:00,913 INFO StatusTransitService$StatusTransitRunnable:539 - USER[-] GROUP[-] Set bundle job [0484436-131016085136608-oozie-oozi-B] status to 'RUNNING' from 'RUNNING'
> 2013-11-19 09:17:00,914 INFO StatusTransitService$StatusTransitRunnable:539 - USER[-] GROUP[-] Bundle job [0484436-131016085136608-oozie-oozi-B] Pending set to TRUE
> 2013-11-19 09:19:16,490 INFO CoordMaterializeTriggerService$CoordMaterializeTriggerRunnable:539 - USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] Job :0484437-131016085136608-oozie-oozi-C numWaitingActions : 0 MatThrottle : 60
> 2013-11-19 09:22:16,907 INFO CoordMaterializeTriggerService$CoordMaterializeTriggerRunnable:539 - USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] Job :0484437-131016085136608-oozie-oozi-C numWaitingActions : 0 MatThrottle : 60
> 2013-11-19 09:25:17,086 INFO CoordMaterializeTriggerService$CoordMaterializeTriggerRunnable:539 - USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] Job :0484437-131016085136608-oozie-oozi-C numWaitingActions : 0 MatThrottle : 60
> 2013-11-19 09:26:49,373 INFO CoordSubmitXCommand:539 - USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0484436-131016085136608-oozie-oozi-B] ACTION[-] STARTED Coordinator Submit
> 2013-11-19 09:26:49,383 INFO CoordSubmitXCommand:539 - USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0484436-131016085136608-oozie-oozi-B] ACTION[-] configDefault Doesn't exist
> 2013-11-19 09:26:49,438 INFO CoordSubmitXCommand:539 - USER[fetl] GROUP[-] TOKEN[] APP[<app name>] JOB[0484598-131016085136608-oozie-oozi-C] ACTION[-] ENDED Coordinator Submit jobId=0484598-131016085136608-oozie-oozi-C
> 2013-11-19 09:26:49,445 INFO BundleStatusUpdateXCommand:539 - USER[fetl] GROUP[-] TOKEN[] APP[<app name>] JOB[0484598-131016085136608-oozie-oozi-C] ACTION[-] Updated bundle action [0484436-131016085136608-oozie-oozi-B_<app name>] from prev status [PREP] to current coord status [PREP], and new bundle action pending [1]
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.1#6144)