You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@falcon.apache.org by "Pragya Mittal (JIRA)" <ji...@apache.org> on 2016/01/08 14:09:39 UTC
[jira] [Updated] (FALCON-1741) Rerun API behaviour different in
case of succeeded instances
[ https://issues.apache.org/jira/browse/FALCON-1741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Pragya Mittal updated FALCON-1741:
----------------------------------
Description:
In case of succeeded instances , rerun without -force API has different behaviour based on the scheduler choice.
For oozie :
The state of instances changes to RUNNING.
{noformat}
dataqa@lda01:~$ fiprerun processMerlinOozie -start 2016-01-08T12:13Z -end 2016-01-08T12:15Z
Consolidated Status: SUCCEEDED
Instances:
Instance Cluster SourceCluster Status Start End Details Log
-----------------------------------------------------------------------------------------------
2016-01-08T12:13Z ProcessMultipleClustersTest-corp-9706f068 - RUNNING 2016-01-08T13:03Z 2016-01-08T13:03Z - http://8RPCG32.corp.inmobi.com:11000/oozie?job=0001811-160104160825636-oozie-oozi-W
2016-01-08T12:13Z ProcessMultipleClustersTest-corp-0b270a1d - RUNNING 2016-01-08T13:03Z 2016-01-08T13:03Z - http://lda01:11000/oozie?job=0002247-160104115615658-oozie-oozi-W
Additional Information:
Response: ua1/RERUN
ua2/RERUN
Request Id: ua1/871377866@qtp-630572412-35 - 7190c4c8-bacb-4639-8d48-c9e639f544da
ua2/1554129706@qtp-536122141-13 - bc18127b-1bf8-4ea1-99e6-b1f10ba3a441
{noformat}
For native :
The state of instances changes to READY. Also params are present as a part of rerun API>
{noformat}
dataqa@lda01:~$ fiprerun ProcessMultipleClustersTest-agregator-coord16-8f55f59b -start 2016-01-08T12:13Z -end 2016-01-08T12:15Z
Consolidated Status: SUCCEEDED
Instances:
Instance Cluster SourceCluster Status Start End Details Log
-----------------------------------------------------------------------------------------------
2016-01-08T12:13Z ProcessMultipleClustersTest-corp-9706f068 - READY 2016-01-08T13:03Z 2016-01-08T13:03Z - http://8RPCG32.corp.inmobi.com:11000/oozie?job=0001812-160104160825636-oozie-oozi-W
Workflow params
shouldRecord=false
timeStamp=2016-01-08-12-13
mapreduce.job.user.name=dataqa
falconInPaths=NONE
feedNames=NONE
oozie.wf.rerun.failnodes=true
falcon.libpath=/tmp/falcon-regression-working/lib
ENTITY_PATH=/tmp/falcon-regression-staging/falcon/workflows/process/ProcessMultipleClustersTest-agregator-coord16-8f55f59b/3cd7594ed13b218460df53a9e827eb28_1452252282546
entityType=PROCESS
systemJMSNotificationEnabled=false
nominalTime=2016-01-08-12-13
feedInstancePaths=NONE
field2=value2
logDir=hdfs://10.14.114.202:8020/tmp/falcon-regression-staging/falcon/workflows/process/ProcessMultipleClustersTest-agregator-coord16-8f55f59b/logs
userWorkflowEngine=oozie
oozie.use.system.libpath=true
userJMSNotificationEnabled=true
queueName=default
falconDataOperation=GENERATE
falcon.system.rerun=true
fileTime=${formatTime(dateOffset(instanceTime(), 1, 'DAY'), 'yyyy-MMM-dd')}
field1=value1
workflowEngineUrl=http://10.14.114.202:11000/oozie/
datasource=NA
brokerImplClass=org.apache.activemq.ActiveMQConnectionFactory
userBrokerImplClass=org.apache.activemq.ActiveMQConnectionFactory
brokerUrl=tcp://localhost:61616
ENTITY_NAME=FALCON_PROCESS_DEFAULT_ProcessMultipleClustersTest-agregator-coord16-8f55f59b
availabilityFlag=NA
userWorkflowName=ProcessMultipleClustersTest-agregator-coord16-8f55f59b-workflow
brokerTTL=4320
entityName=ProcessMultipleClustersTest-agregator-coord16-8f55f59b
srcClusterName=NA
userBrokerUrl=tcp://10.14.114.202:61616?daemon=true
falconInputFeeds=NONE
userWorkflowVersion=1.0
oozie.libpath=/tmp/falcon-regression-staging/falcon/workflows/process/ProcessMultipleClustersTest-agregator-coord16-8f55f59b/3cd7594ed13b218460df53a9e827eb28_1452252282546/lib
cluster=ProcessMultipleClustersTest-corp-9706f068
jobPriority=NORMAL
colo.name=ua1
nameNode=hdfs://10.14.114.202:8020
user=${user()}
jobTracker=10.14.114.202:8021
oozie.wf.application.path=hdfs://10.14.114.202:8020/tmp/falcon-regression-staging/falcon/workflows/process/ProcessMultipleClustersTest-agregator-coord16-8f55f59b/3cd7594ed13b218460df53a9e827eb28_1452252282546
2016-01-08T12:13Z ProcessMultipleClustersTest-corp-0b270a1d - READY 2016-01-08T13:03Z 2016-01-08T13:03Z - http://lda01:11000/oozie?job=0002248-160104115615658-oozie-oozi-W
Workflow params
fileTime=${formatTime(dateOffset(instanceTime(), 1, 'DAY'), 'yyyy-MMM-dd')}
cluster=ProcessMultipleClustersTest-corp-0b270a1d
availabilityFlag=NA
falconDataOperation=GENERATE
falconInputFeeds=NONE
feedNames=NONE
colo.name=ua2
shouldRecord=false
falcon.libpath=/tmp/falcon-regression-working/lib
mapreduce.job.user.name=dataqa
feedInstancePaths=NONE
systemJMSNotificationEnabled=false
brokerImplClass=org.apache.activemq.ActiveMQConnectionFactory
userBrokerImplClass=org.apache.activemq.ActiveMQConnectionFactory
entityName=ProcessMultipleClustersTest-agregator-coord16-8f55f59b
falcon.system.rerun=true
jobTracker=192.168.138.236:8021
userWorkflowVersion=1.0
workflowEngineUrl=http://192.168.138.236:11000/oozie/
jobPriority=NORMAL
ENTITY_NAME=FALCON_PROCESS_DEFAULT_ProcessMultipleClustersTest-agregator-coord16-8f55f59b
nameNode=hdfs://192.168.138.236:8020
nominalTime=2016-01-08-12-13
oozie.libpath=/tmp/falcon-regression-staging/falcon/workflows/process/ProcessMultipleClustersTest-agregator-coord16-8f55f59b/e6720cfbbe1d8a50488a7c47da94b506_1452252285506/lib
oozie.use.system.libpath=true
entityType=PROCESS
srcClusterName=NA
userWorkflowEngine=oozie
oozie.wf.rerun.failnodes=true
timeStamp=2016-01-08-12-13
brokerUrl=tcp://localhost:61616
field1=value1
oozie.wf.application.path=hdfs://192.168.138.236:8020/tmp/falcon-regression-staging/falcon/workflows/process/ProcessMultipleClustersTest-agregator-coord16-8f55f59b/e6720cfbbe1d8a50488a7c47da94b506_1452252285506
queueName=default
brokerTTL=4320
userWorkflowName=ProcessMultipleClustersTest-agregator-coord16-8f55f59b-workflow
datasource=NA
ENTITY_PATH=/tmp/falcon-regression-staging/falcon/workflows/process/ProcessMultipleClustersTest-agregator-coord16-8f55f59b/e6720cfbbe1d8a50488a7c47da94b506_1452252285506
userJMSNotificationEnabled=true
userBrokerUrl=tcp://192.168.138.236:61616?daemon=true
falconInPaths=NONE
user=${user()}
field2=value2
logDir=hdfs://192.168.138.236:8020/tmp/falcon-regression-staging/falcon/workflows/process/ProcessMultipleClustersTest-agregator-coord16-8f55f59b/logs
Additional Information:
Response: ua1/RERUN
ua2/RERUN
Request Id: ua1/871377866@qtp-630572412-35 - 8d118d4d-c0ef-4335-a9af-10364498ec4f
ua2/1554129706@qtp-536122141-13 - c2a3fc50-8b05-47ce-9c85-ca432b96d923
{noformat}
> Rerun API behaviour different in case of succeeded instances
> ------------------------------------------------------------
>
> Key: FALCON-1741
> URL: https://issues.apache.org/jira/browse/FALCON-1741
> Project: Falcon
> Issue Type: Sub-task
> Components: oozie, scheduler
> Affects Versions: 0.9
> Reporter: Pragya Mittal
>
> In case of succeeded instances , rerun without -force API has different behaviour based on the scheduler choice.
> For oozie :
> The state of instances changes to RUNNING.
> {noformat}
> dataqa@lda01:~$ fiprerun processMerlinOozie -start 2016-01-08T12:13Z -end 2016-01-08T12:15Z
> Consolidated Status: SUCCEEDED
> Instances:
> Instance Cluster SourceCluster Status Start End Details Log
> -----------------------------------------------------------------------------------------------
> 2016-01-08T12:13Z ProcessMultipleClustersTest-corp-9706f068 - RUNNING 2016-01-08T13:03Z 2016-01-08T13:03Z - http://8RPCG32.corp.inmobi.com:11000/oozie?job=0001811-160104160825636-oozie-oozi-W
> 2016-01-08T12:13Z ProcessMultipleClustersTest-corp-0b270a1d - RUNNING 2016-01-08T13:03Z 2016-01-08T13:03Z - http://lda01:11000/oozie?job=0002247-160104115615658-oozie-oozi-W
> Additional Information:
> Response: ua1/RERUN
> ua2/RERUN
> Request Id: ua1/871377866@qtp-630572412-35 - 7190c4c8-bacb-4639-8d48-c9e639f544da
> ua2/1554129706@qtp-536122141-13 - bc18127b-1bf8-4ea1-99e6-b1f10ba3a441
> {noformat}
> For native :
> The state of instances changes to READY. Also params are present as a part of rerun API>
> {noformat}
> dataqa@lda01:~$ fiprerun ProcessMultipleClustersTest-agregator-coord16-8f55f59b -start 2016-01-08T12:13Z -end 2016-01-08T12:15Z
> Consolidated Status: SUCCEEDED
> Instances:
> Instance Cluster SourceCluster Status Start End Details Log
> -----------------------------------------------------------------------------------------------
> 2016-01-08T12:13Z ProcessMultipleClustersTest-corp-9706f068 - READY 2016-01-08T13:03Z 2016-01-08T13:03Z - http://8RPCG32.corp.inmobi.com:11000/oozie?job=0001812-160104160825636-oozie-oozi-W
> Workflow params
> shouldRecord=false
> timeStamp=2016-01-08-12-13
> mapreduce.job.user.name=dataqa
> falconInPaths=NONE
> feedNames=NONE
> oozie.wf.rerun.failnodes=true
> falcon.libpath=/tmp/falcon-regression-working/lib
> ENTITY_PATH=/tmp/falcon-regression-staging/falcon/workflows/process/ProcessMultipleClustersTest-agregator-coord16-8f55f59b/3cd7594ed13b218460df53a9e827eb28_1452252282546
> entityType=PROCESS
> systemJMSNotificationEnabled=false
> nominalTime=2016-01-08-12-13
> feedInstancePaths=NONE
> field2=value2
> logDir=hdfs://10.14.114.202:8020/tmp/falcon-regression-staging/falcon/workflows/process/ProcessMultipleClustersTest-agregator-coord16-8f55f59b/logs
> userWorkflowEngine=oozie
> oozie.use.system.libpath=true
> userJMSNotificationEnabled=true
> queueName=default
> falconDataOperation=GENERATE
> falcon.system.rerun=true
> fileTime=${formatTime(dateOffset(instanceTime(), 1, 'DAY'), 'yyyy-MMM-dd')}
> field1=value1
> workflowEngineUrl=http://10.14.114.202:11000/oozie/
> datasource=NA
> brokerImplClass=org.apache.activemq.ActiveMQConnectionFactory
> userBrokerImplClass=org.apache.activemq.ActiveMQConnectionFactory
> brokerUrl=tcp://localhost:61616
> ENTITY_NAME=FALCON_PROCESS_DEFAULT_ProcessMultipleClustersTest-agregator-coord16-8f55f59b
> availabilityFlag=NA
> userWorkflowName=ProcessMultipleClustersTest-agregator-coord16-8f55f59b-workflow
> brokerTTL=4320
> entityName=ProcessMultipleClustersTest-agregator-coord16-8f55f59b
> srcClusterName=NA
> userBrokerUrl=tcp://10.14.114.202:61616?daemon=true
> falconInputFeeds=NONE
> userWorkflowVersion=1.0
> oozie.libpath=/tmp/falcon-regression-staging/falcon/workflows/process/ProcessMultipleClustersTest-agregator-coord16-8f55f59b/3cd7594ed13b218460df53a9e827eb28_1452252282546/lib
> cluster=ProcessMultipleClustersTest-corp-9706f068
> jobPriority=NORMAL
> colo.name=ua1
> nameNode=hdfs://10.14.114.202:8020
> user=${user()}
> jobTracker=10.14.114.202:8021
> oozie.wf.application.path=hdfs://10.14.114.202:8020/tmp/falcon-regression-staging/falcon/workflows/process/ProcessMultipleClustersTest-agregator-coord16-8f55f59b/3cd7594ed13b218460df53a9e827eb28_1452252282546
> 2016-01-08T12:13Z ProcessMultipleClustersTest-corp-0b270a1d - READY 2016-01-08T13:03Z 2016-01-08T13:03Z - http://lda01:11000/oozie?job=0002248-160104115615658-oozie-oozi-W
> Workflow params
> fileTime=${formatTime(dateOffset(instanceTime(), 1, 'DAY'), 'yyyy-MMM-dd')}
> cluster=ProcessMultipleClustersTest-corp-0b270a1d
> availabilityFlag=NA
> falconDataOperation=GENERATE
> falconInputFeeds=NONE
> feedNames=NONE
> colo.name=ua2
> shouldRecord=false
> falcon.libpath=/tmp/falcon-regression-working/lib
> mapreduce.job.user.name=dataqa
> feedInstancePaths=NONE
> systemJMSNotificationEnabled=false
> brokerImplClass=org.apache.activemq.ActiveMQConnectionFactory
> userBrokerImplClass=org.apache.activemq.ActiveMQConnectionFactory
> entityName=ProcessMultipleClustersTest-agregator-coord16-8f55f59b
> falcon.system.rerun=true
> jobTracker=192.168.138.236:8021
> userWorkflowVersion=1.0
> workflowEngineUrl=http://192.168.138.236:11000/oozie/
> jobPriority=NORMAL
> ENTITY_NAME=FALCON_PROCESS_DEFAULT_ProcessMultipleClustersTest-agregator-coord16-8f55f59b
> nameNode=hdfs://192.168.138.236:8020
> nominalTime=2016-01-08-12-13
> oozie.libpath=/tmp/falcon-regression-staging/falcon/workflows/process/ProcessMultipleClustersTest-agregator-coord16-8f55f59b/e6720cfbbe1d8a50488a7c47da94b506_1452252285506/lib
> oozie.use.system.libpath=true
> entityType=PROCESS
> srcClusterName=NA
> userWorkflowEngine=oozie
> oozie.wf.rerun.failnodes=true
> timeStamp=2016-01-08-12-13
> brokerUrl=tcp://localhost:61616
> field1=value1
> oozie.wf.application.path=hdfs://192.168.138.236:8020/tmp/falcon-regression-staging/falcon/workflows/process/ProcessMultipleClustersTest-agregator-coord16-8f55f59b/e6720cfbbe1d8a50488a7c47da94b506_1452252285506
> queueName=default
> brokerTTL=4320
> userWorkflowName=ProcessMultipleClustersTest-agregator-coord16-8f55f59b-workflow
> datasource=NA
> ENTITY_PATH=/tmp/falcon-regression-staging/falcon/workflows/process/ProcessMultipleClustersTest-agregator-coord16-8f55f59b/e6720cfbbe1d8a50488a7c47da94b506_1452252285506
> userJMSNotificationEnabled=true
> userBrokerUrl=tcp://192.168.138.236:61616?daemon=true
> falconInPaths=NONE
> user=${user()}
> field2=value2
> logDir=hdfs://192.168.138.236:8020/tmp/falcon-regression-staging/falcon/workflows/process/ProcessMultipleClustersTest-agregator-coord16-8f55f59b/logs
> Additional Information:
> Response: ua1/RERUN
> ua2/RERUN
> Request Id: ua1/871377866@qtp-630572412-35 - 8d118d4d-c0ef-4335-a9af-10364498ec4f
> ua2/1554129706@qtp-536122141-13 - c2a3fc50-8b05-47ce-9c85-ca432b96d923
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)