You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@falcon.apache.org by "Pragya Mittal (JIRA)" <ji...@apache.org> on 2015/12/30 11:44:49 UTC
[jira] [Updated] (FALCON-1709) Inconsistent behaviour of status API
when scheduled using native scheduler
[ https://issues.apache.org/jira/browse/FALCON-1709?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Pragya Mittal updated FALCON-1709:
----------------------------------
Description:
1. status API as of now shows all the materialized instances along with the status of launched instances. Same should be reflected in case of native scheduler.
{noformat}
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<process name="ProcessWfTest-agregator-coord16-7b27e3a1" xmlns="uri:falcon:process:0.1">
<clusters>
<cluster name="ProcessWfTest-corp-e5f9ec42">
<validity start="2015-12-30T10:32Z" end="2015-12-30T10:52Z"/>
</cluster>
</clusters>
<parallel>4</parallel>
<order>FIFO</order>
<frequency>minutes(1)</frequency>
<timezone>UTC</timezone>
<properties>
<property name="queueName" value="default"/>
<property name="fileTime" value="${formatTime(dateOffset(instanceTime(), 1, 'DAY'), 'yyyy-MMM-dd')}"/>
<property name="user" value="${user()}"/>
</properties>
<workflow path="/tmp/falcon-regression/ProcessWfTest/aggregator"/>
<retry policy="periodic" delay="minutes(3)" attempts="3"/>
<ACL owner="pragya" group="dataqa" permission="*"/>
</process>
{noformat}
On hitting status from CLI :
{noformat}
dataqa@lda01:~$ fip -name ProcessWfTest-agregator-coord16-7b27e3a1 -status
Consolidated Status: SUCCEEDED
Instances:
Instance Cluster SourceCluster Status Start End Details Log
-----------------------------------------------------------------------------------------------
2015-12-30T10:32Z ProcessWfTest-corp-e5f9ec42 - RUNNING 2015-12-30T10:37Z 2015-12-30T10:37Z - http://lda01:11000/oozie?job=0000102-151230090106575-oozie-oozi-W
actions:
hdfscommands OK -
aggregator RUNNING http://lda01:8088/proxy/application_1445322981080_3831/
2015-12-30T10:33Z ProcessWfTest-corp-e5f9ec42 - RUNNING 2015-12-30T10:37Z 2015-12-30T10:37Z - http://lda01:11000/oozie?job=0000104-151230090106575-oozie-oozi-W
actions:
hdfscommands OK -
aggregator RUNNING http://lda01:8088/proxy/application_1445322981080_3832/
2015-12-30T10:34Z - - - - - - -
2015-12-30T10:35Z - - - - - - -
2015-12-30T10:36Z ProcessWfTest-corp-e5f9ec42 - RUNNING 2015-12-30T10:37Z 2015-12-30T10:37Z - http://lda01:11000/oozie?job=0000108-151230090106575-oozie-oozi-W
actions:
hdfscommands OK -
aggregator RUNNING http://lda01:8088/proxy/application_1445322981080_3834/
2015-12-30T10:37Z ProcessWfTest-corp-e5f9ec42 - RUNNING 2015-12-30T10:37Z 2015-12-30T10:37Z - http://lda01:11000/oozie?job=0000106-151230090106575-oozie-oozi-W
actions:
hdfscommands OK -
aggregator RUNNING http://lda01:8088/proxy/application_1445322981080_3833/
Additional Information:
Response: ua1/STATUS
Request Id: ua1/362835397@qtp-690339675-11 - c655267d-b177-43df-a2cf-c63f10694c9b
dataqa@lda01:~$ date
Wed Dec 30 10:37:49 GMT 2015
{noformat}
As we can see instances of time 2015-12-30T10:34Z and 2015-12-30T10:35Z didnt get materialized while instances after 2015-12-30T10:35Z got launched.
2. FIFO order is not preserved.
When i hit teh status command again from CLI following response was recorded.
{noformat}
dataqa@lda01:~$ date
Wed Dec 30 10:42:29 GMT 2015
dataqa@lda01:~$ fip -name ProcessWfTest-agregator-coord16-7b27e3a1 -status
Consolidated Status: SUCCEEDED
Instances:
Instance Cluster SourceCluster Status Start End Details Log
-----------------------------------------------------------------------------------------------
2015-12-30T10:33Z ProcessWfTest-corp-e5f9ec42 - KILLED 2015-12-30T10:41Z 2015-12-30T10:42Z - http://lda01:11000/oozie?job=0000104-151230090106575-oozie-oozi-W
actions:
hdfscommands OK -
aggregator FAILED/KILLED http://lda01:8088/proxy/application_1445322981080_3861/
2015-12-30T10:34Z ProcessWfTest-corp-e5f9ec42 - RUNNING 2015-12-30T10:42Z 2015-12-30T10:42Z - http://lda01:11000/oozie?job=0000110-151230090106575-oozie-oozi-W
actions:
hdfscommands OK -
aggregator RUNNING http://lda01:8088/proxy/application_1445322981080_3874/
2015-12-30T10:35Z ProcessWfTest-corp-e5f9ec42 - RUNNING 2015-12-30T10:42Z 2015-12-30T10:42Z - http://lda01:11000/oozie?job=0000112-151230090106575-oozie-oozi-W
actions:
hdfscommands OK -
aggregator RUNNING http://lda01:8088/proxy/application_1445322981080_3870/
2015-12-30T10:36Z ProcessWfTest-corp-e5f9ec42 - KILLED 2015-12-30T10:41Z 2015-12-30T10:42Z - http://lda01:11000/oozie?job=0000108-151230090106575-oozie-oozi-W
actions:
hdfscommands OK -
aggregator FAILED/KILLED http://lda01:8088/proxy/application_1445322981080_3863/
2015-12-30T10:37Z ProcessWfTest-corp-e5f9ec42 - KILLED 2015-12-30T10:41Z 2015-12-30T10:42Z - http://lda01:11000/oozie?job=0000106-151230090106575-oozie-oozi-W
actions:
hdfscommands OK -
aggregator FAILED/KILLED http://lda01:8088/proxy/application_1445322981080_3862/
2015-12-30T10:38Z ProcessWfTest-corp-e5f9ec42 - RUNNING 2015-12-30T10:42Z 2015-12-30T10:42Z - http://lda01:11000/oozie?job=0000114-151230090106575-oozie-oozi-W
actions:
hdfscommands OK -
aggregator RUNNING http://lda01:8088/proxy/application_1445322981080_3873/
2015-12-30T10:39Z ProcessWfTest-corp-e5f9ec42 - KILLED 2015-12-30T10:39Z 2015-12-30T10:39Z - http://lda01:11000/oozie?job=0000116-151230090106575-oozie-oozi-W
actions:
hdfscommands OK -
aggregator FAILED/KILLED http://lda01:8088/proxy/application_1445322981080_3852/
2015-12-30T10:40Z ProcessWfTest-corp-e5f9ec42 - KILLED 2015-12-30T10:40Z 2015-12-30T10:40Z - http://lda01:11000/oozie?job=0000118-151230090106575-oozie-oozi-W
actions:
hdfscommands OK -
aggregator FAILED/KILLED http://lda01:8088/proxy/application_1445322981080_3855/
2015-12-30T10:41Z ProcessWfTest-corp-e5f9ec42 - KILLED 2015-12-30T10:41Z 2015-12-30T10:41Z - http://lda01:11000/oozie?job=0000120-151230090106575-oozie-oozi-W
actions:
hdfscommands OK -
aggregator FAILED/KILLED http://lda01:8088/proxy/application_1445322981080_3858/
2015-12-30T10:42Z ProcessWfTest-corp-e5f9ec42 - RUNNING 2015-12-30T10:42Z 2015-12-30T10:42Z - http://lda01:11000/oozie?job=0000122-151230090106575-oozie-oozi-W
actions:
hdfscommands OK -
aggregator RUNNING http://lda01:8088/proxy/application_1445322981080_3869/
Additional Information:
Response: ua1/STATUS
Request Id: ua1/397575720@qtp-690339675-12 - d0942fed-ec3a-40a6-86e5-f0dfac6564c2
{noformat}
Instances of time 2015-12-30T10:34Z, 2015-12-30T10:35Z and 2015-12-30T10:38Z are running.
> Inconsistent behaviour of status API when scheduled using native scheduler
> --------------------------------------------------------------------------
>
> Key: FALCON-1709
> URL: https://issues.apache.org/jira/browse/FALCON-1709
> Project: Falcon
> Issue Type: Bug
> Components: scheduler
> Affects Versions: 0.9
> Reporter: Pragya Mittal
>
> 1. status API as of now shows all the materialized instances along with the status of launched instances. Same should be reflected in case of native scheduler.
> {noformat}
> <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
> <process name="ProcessWfTest-agregator-coord16-7b27e3a1" xmlns="uri:falcon:process:0.1">
> <clusters>
> <cluster name="ProcessWfTest-corp-e5f9ec42">
> <validity start="2015-12-30T10:32Z" end="2015-12-30T10:52Z"/>
> </cluster>
> </clusters>
> <parallel>4</parallel>
> <order>FIFO</order>
> <frequency>minutes(1)</frequency>
> <timezone>UTC</timezone>
> <properties>
> <property name="queueName" value="default"/>
> <property name="fileTime" value="${formatTime(dateOffset(instanceTime(), 1, 'DAY'), 'yyyy-MMM-dd')}"/>
> <property name="user" value="${user()}"/>
> </properties>
> <workflow path="/tmp/falcon-regression/ProcessWfTest/aggregator"/>
> <retry policy="periodic" delay="minutes(3)" attempts="3"/>
> <ACL owner="pragya" group="dataqa" permission="*"/>
> </process>
> {noformat}
> On hitting status from CLI :
> {noformat}
> dataqa@lda01:~$ fip -name ProcessWfTest-agregator-coord16-7b27e3a1 -status
> Consolidated Status: SUCCEEDED
> Instances:
> Instance Cluster SourceCluster Status Start End Details Log
> -----------------------------------------------------------------------------------------------
> 2015-12-30T10:32Z ProcessWfTest-corp-e5f9ec42 - RUNNING 2015-12-30T10:37Z 2015-12-30T10:37Z - http://lda01:11000/oozie?job=0000102-151230090106575-oozie-oozi-W
> actions:
> hdfscommands OK -
> aggregator RUNNING http://lda01:8088/proxy/application_1445322981080_3831/
> 2015-12-30T10:33Z ProcessWfTest-corp-e5f9ec42 - RUNNING 2015-12-30T10:37Z 2015-12-30T10:37Z - http://lda01:11000/oozie?job=0000104-151230090106575-oozie-oozi-W
> actions:
> hdfscommands OK -
> aggregator RUNNING http://lda01:8088/proxy/application_1445322981080_3832/
> 2015-12-30T10:34Z - - - - - - -
> 2015-12-30T10:35Z - - - - - - -
> 2015-12-30T10:36Z ProcessWfTest-corp-e5f9ec42 - RUNNING 2015-12-30T10:37Z 2015-12-30T10:37Z - http://lda01:11000/oozie?job=0000108-151230090106575-oozie-oozi-W
> actions:
> hdfscommands OK -
> aggregator RUNNING http://lda01:8088/proxy/application_1445322981080_3834/
> 2015-12-30T10:37Z ProcessWfTest-corp-e5f9ec42 - RUNNING 2015-12-30T10:37Z 2015-12-30T10:37Z - http://lda01:11000/oozie?job=0000106-151230090106575-oozie-oozi-W
> actions:
> hdfscommands OK -
> aggregator RUNNING http://lda01:8088/proxy/application_1445322981080_3833/
> Additional Information:
> Response: ua1/STATUS
> Request Id: ua1/362835397@qtp-690339675-11 - c655267d-b177-43df-a2cf-c63f10694c9b
> dataqa@lda01:~$ date
> Wed Dec 30 10:37:49 GMT 2015
> {noformat}
> As we can see instances of time 2015-12-30T10:34Z and 2015-12-30T10:35Z didnt get materialized while instances after 2015-12-30T10:35Z got launched.
> 2. FIFO order is not preserved.
> When i hit teh status command again from CLI following response was recorded.
> {noformat}
> dataqa@lda01:~$ date
> Wed Dec 30 10:42:29 GMT 2015
> dataqa@lda01:~$ fip -name ProcessWfTest-agregator-coord16-7b27e3a1 -status
> Consolidated Status: SUCCEEDED
> Instances:
> Instance Cluster SourceCluster Status Start End Details Log
> -----------------------------------------------------------------------------------------------
> 2015-12-30T10:33Z ProcessWfTest-corp-e5f9ec42 - KILLED 2015-12-30T10:41Z 2015-12-30T10:42Z - http://lda01:11000/oozie?job=0000104-151230090106575-oozie-oozi-W
> actions:
> hdfscommands OK -
> aggregator FAILED/KILLED http://lda01:8088/proxy/application_1445322981080_3861/
> 2015-12-30T10:34Z ProcessWfTest-corp-e5f9ec42 - RUNNING 2015-12-30T10:42Z 2015-12-30T10:42Z - http://lda01:11000/oozie?job=0000110-151230090106575-oozie-oozi-W
> actions:
> hdfscommands OK -
> aggregator RUNNING http://lda01:8088/proxy/application_1445322981080_3874/
> 2015-12-30T10:35Z ProcessWfTest-corp-e5f9ec42 - RUNNING 2015-12-30T10:42Z 2015-12-30T10:42Z - http://lda01:11000/oozie?job=0000112-151230090106575-oozie-oozi-W
> actions:
> hdfscommands OK -
> aggregator RUNNING http://lda01:8088/proxy/application_1445322981080_3870/
> 2015-12-30T10:36Z ProcessWfTest-corp-e5f9ec42 - KILLED 2015-12-30T10:41Z 2015-12-30T10:42Z - http://lda01:11000/oozie?job=0000108-151230090106575-oozie-oozi-W
> actions:
> hdfscommands OK -
> aggregator FAILED/KILLED http://lda01:8088/proxy/application_1445322981080_3863/
> 2015-12-30T10:37Z ProcessWfTest-corp-e5f9ec42 - KILLED 2015-12-30T10:41Z 2015-12-30T10:42Z - http://lda01:11000/oozie?job=0000106-151230090106575-oozie-oozi-W
> actions:
> hdfscommands OK -
> aggregator FAILED/KILLED http://lda01:8088/proxy/application_1445322981080_3862/
> 2015-12-30T10:38Z ProcessWfTest-corp-e5f9ec42 - RUNNING 2015-12-30T10:42Z 2015-12-30T10:42Z - http://lda01:11000/oozie?job=0000114-151230090106575-oozie-oozi-W
> actions:
> hdfscommands OK -
> aggregator RUNNING http://lda01:8088/proxy/application_1445322981080_3873/
> 2015-12-30T10:39Z ProcessWfTest-corp-e5f9ec42 - KILLED 2015-12-30T10:39Z 2015-12-30T10:39Z - http://lda01:11000/oozie?job=0000116-151230090106575-oozie-oozi-W
> actions:
> hdfscommands OK -
> aggregator FAILED/KILLED http://lda01:8088/proxy/application_1445322981080_3852/
> 2015-12-30T10:40Z ProcessWfTest-corp-e5f9ec42 - KILLED 2015-12-30T10:40Z 2015-12-30T10:40Z - http://lda01:11000/oozie?job=0000118-151230090106575-oozie-oozi-W
> actions:
> hdfscommands OK -
> aggregator FAILED/KILLED http://lda01:8088/proxy/application_1445322981080_3855/
> 2015-12-30T10:41Z ProcessWfTest-corp-e5f9ec42 - KILLED 2015-12-30T10:41Z 2015-12-30T10:41Z - http://lda01:11000/oozie?job=0000120-151230090106575-oozie-oozi-W
> actions:
> hdfscommands OK -
> aggregator FAILED/KILLED http://lda01:8088/proxy/application_1445322981080_3858/
> 2015-12-30T10:42Z ProcessWfTest-corp-e5f9ec42 - RUNNING 2015-12-30T10:42Z 2015-12-30T10:42Z - http://lda01:11000/oozie?job=0000122-151230090106575-oozie-oozi-W
> actions:
> hdfscommands OK -
> aggregator RUNNING http://lda01:8088/proxy/application_1445322981080_3869/
> Additional Information:
> Response: ua1/STATUS
> Request Id: ua1/397575720@qtp-690339675-12 - d0942fed-ec3a-40a6-86e5-f0dfac6564c2
> {noformat}
> Instances of time 2015-12-30T10:34Z, 2015-12-30T10:35Z and 2015-12-30T10:38Z are running.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)