You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@oozie.apache.org by "Hadoop QA (JIRA)" <ji...@apache.org> on 2011/09/08 06:29:09 UTC
[jira] [Created] (OOZIE-102) GH-67: input data check should have a
timeout for catch-up mode too.
GH-67: input data check should have a timeout for catch-up mode too.
---------------------------------------------------------------------
Key: OOZIE-102
URL: https://issues.apache.org/jira/browse/OOZIE-102
Project: Oozie
Issue Type: Bug
Reporter: Hadoop QA
For normal,timeout when Nominal Time + timeout > current time
For catchup,timeout when Created Time + timeout > current time
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OOZIE-102) GH-67: input data check should have
a timeout for catch-up mode too.
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OOZIE-102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13101835#comment-13101835 ]
Hadoop QA commented on OOZIE-102:
---------------------------------
anew remarked:
Although "normal" and "catchup" jobs are actually treated equally by Oozie, I think there is a true difference in use cases:
- A catchup job is started to run/redo computations whose nominal time is often far back in the past, and the input data is typically historical data and already available at the creation time of the job. But most likely the catchup also requires many more jobs to be run than for current computations.
- A current job has data dependencies on other jobs that have just finished or are expected to finish very soon. If any of these jobs are late, then this job has to wait. This is more likely to happen than in catchup mode.
Therefore, it is desirable to get expire a catchup job sooner than a current job. On the other hand, for simplicity's sake, and because Oozie currently does not have an explicit notion of "catchup", it is desirable to have the same timeout for both types of jobs.
I do not have a strong preference, but a slight tendency towards keeping it simple. So for now, I vote for a single timeout which kicks in when max(Nominal Time, Created Time) + timeout > Current Time.
> GH-67: input data check should have a timeout for catch-up mode too.
> ---------------------------------------------------------------------
>
> Key: OOZIE-102
> URL: https://issues.apache.org/jira/browse/OOZIE-102
> Project: Oozie
> Issue Type: Bug
> Reporter: Hadoop QA
>
> For normal,timeout when Nominal Time + timeout > current time
> For catchup,timeout when Created Time + timeout > current time
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OOZIE-102) GH-67: input data check should have
a timeout for catch-up mode too.
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OOZIE-102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13101836#comment-13101836 ]
Hadoop QA commented on OOZIE-102:
---------------------------------
mislam77 remarked:
We can identify the job as "catchup" by checking creationTime > nominalTime.
In that case, I prefer to use this :
Catchup Timeout = Creation Time + Concurrency * Timeout
Concurrency means how many max job could be in WAITING state or data checking state.
Overall the reason is: if there are concurrency = 5 )say), there will be at most 5 actions actively checking for their data. Now if they have chain/ripple dependency, the last one has to wait until the other 4 are finished.
To accommodate this command scenario, I want to give a relax/loose timeout.
Comments?
> GH-67: input data check should have a timeout for catch-up mode too.
> ---------------------------------------------------------------------
>
> Key: OOZIE-102
> URL: https://issues.apache.org/jira/browse/OOZIE-102
> Project: Oozie
> Issue Type: Bug
> Reporter: Hadoop QA
>
> For normal,timeout when Nominal Time + timeout > current time
> For catchup,timeout when Created Time + timeout > current time
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Closed] (OOZIE-102) GH-67: input data check should have a
timeout for catch-up mode too.
Posted by "Roman Shaposhnik (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OOZIE-102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Roman Shaposhnik closed OOZIE-102.
----------------------------------
Resolution: Fixed
> GH-67: input data check should have a timeout for catch-up mode too.
> ---------------------------------------------------------------------
>
> Key: OOZIE-102
> URL: https://issues.apache.org/jira/browse/OOZIE-102
> Project: Oozie
> Issue Type: Bug
> Reporter: Hadoop QA
>
> For normal,timeout when Nominal Time + timeout > current time
> For catchup,timeout when Created Time + timeout > current time
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OOZIE-102) GH-67: input data check should have
a timeout for catch-up mode too.
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OOZIE-102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13101837#comment-13101837 ]
Hadoop QA commented on OOZIE-102:
---------------------------------
angelokh remarked:
Closed by 821aaaa9121ca2b8218cc35f2f02e4e1df9439f2 input data check should have a timeout for catch-up mode
> GH-67: input data check should have a timeout for catch-up mode too.
> ---------------------------------------------------------------------
>
> Key: OOZIE-102
> URL: https://issues.apache.org/jira/browse/OOZIE-102
> Project: Oozie
> Issue Type: Bug
> Reporter: Hadoop QA
>
> For normal,timeout when Nominal Time + timeout > current time
> For catchup,timeout when Created Time + timeout > current time
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OOZIE-102) GH-67: input data check should have
a timeout for catch-up mode too.
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OOZIE-102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13099800#comment-13099800 ]
Hadoop QA commented on OOZIE-102:
---------------------------------
mislam77 remarked:
Since it will take sometime to find a suitable timeout for the "catch-up" job, I think we coul provide a catchup-timeout that user would optionally provide through coordinator.xml. BTW there is already a "timeout" element in current XML which determines the timeout of current/regular/normal jobs.
Comment?
> GH-67: input data check should have a timeout for catch-up mode too.
> ---------------------------------------------------------------------
>
> Key: OOZIE-102
> URL: https://issues.apache.org/jira/browse/OOZIE-102
> Project: Oozie
> Issue Type: Bug
> Reporter: Hadoop QA
>
> For normal,timeout when Nominal Time + timeout > current time
> For catchup,timeout when Created Time + timeout > current time
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira