You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@falcon.apache.org by Narayan Periwal <na...@inmobi.com> on 2016/01/11 18:17:15 UTC
Re: Review Request 42150: FALCON-1677: Support re-tries for timed-out
instances
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/42150/
-----------------------------------------------------------
(Updated Jan. 11, 2016, 5:17 p.m.)
Review request for Falcon.
Summary (updated)
-----------------
FALCON-1677: Support re-tries for timed-out instances
Bugs: FALCON-1677
https://issues.apache.org/jira/browse/FALCON-1677
Repository: falcon-git
Description
-------
Currently, Falcon retries only on failure. We should extend support in case of timed-out instances too. Earlier, since we were relying on post-processing to notify the instance status, this was not possible. Now that Falcon relies on Oozie JMS notifications, we can support retries for timed out instances too.
If a dataset is expected to get delayed for a long time, the user is currently forced to supply a large timeout value. This is an overhead in terms of Oozie having to poll for that long. If we introduce retries, the timeout can be a reasonable value with periodic/exponential back-off retries.
Diffs
-----
client/src/main/java/org/apache/falcon/recipe/RecipeToolOptions.java 0ece90b
client/src/main/java/org/apache/falcon/recipe/util/RecipeProcessBuilderUtils.java 9477bb4
client/src/main/resources/process-0.1.xsd 256a29f
common/src/main/java/org/apache/falcon/entity/EntityUtil.java f448d70
common/src/test/java/org/apache/falcon/entity/parser/ProcessEntityParserTest.java da22d56
rerun/src/main/java/org/apache/falcon/rerun/handler/RetryHandler.java 84cd93f
Diff: https://reviews.apache.org/r/42150/diff/
Testing
-------
Done.
Thanks,
Narayan Periwal
Re: Review Request 42150: FALCON-1677: Support re-tries for timed-out
instances
Posted by Narayan Periwal <na...@inmobi.com>.
> On Jan. 12, 2016, 4:30 a.m., Pallavi Rao wrote:
> > common/src/main/java/org/apache/falcon/entity/EntityUtil.java, line 673
> > <https://reviews.apache.org/r/42150/diff/1/?file=1192186#file1192186line673>
> >
> > Can you please add all feed.retry.* properties with default values to src/conf/runtime.properties?
yes...I will add them
- Narayan
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/42150/#review113927
-----------------------------------------------------------
On Jan. 12, 2016, 8:19 a.m., Narayan Periwal wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/42150/
> -----------------------------------------------------------
>
> (Updated Jan. 12, 2016, 8:19 a.m.)
>
>
> Review request for Falcon.
>
>
> Bugs: FALCON-1677
> https://issues.apache.org/jira/browse/FALCON-1677
>
>
> Repository: falcon-git
>
>
> Description
> -------
>
> Currently, Falcon retries only on failure. We should extend support in case of timed-out instances too. Earlier, since we were relying on post-processing to notify the instance status, this was not possible. Now that Falcon relies on Oozie JMS notifications, we can support retries for timed out instances too.
> If a dataset is expected to get delayed for a long time, the user is currently forced to supply a large timeout value. This is an overhead in terms of Oozie having to poll for that long. If we introduce retries, the timeout can be a reasonable value with periodic/exponential back-off retries.
>
>
> Diffs
> -----
>
> client/src/main/java/org/apache/falcon/recipe/RecipeToolOptions.java 0ece90b
> client/src/main/java/org/apache/falcon/recipe/util/RecipeProcessBuilderUtils.java 9477bb4
> client/src/main/resources/process-0.1.xsd 256a29f
> common/src/main/java/org/apache/falcon/entity/EntityUtil.java f448d70
> common/src/test/java/org/apache/falcon/entity/parser/ProcessEntityParserTest.java da22d56
> docs/src/site/twiki/EntitySpecification.twiki c4f5f76
> rerun/src/main/java/org/apache/falcon/rerun/handler/RetryHandler.java 84cd93f
> src/conf/runtime.properties f535b0f
>
> Diff: https://reviews.apache.org/r/42150/diff/
>
>
> Testing
> -------
>
> Done.
>
>
> Thanks,
>
> Narayan Periwal
>
>
Re: Review Request 42150: FALCON-1677: Support re-tries for timed-out
instances
Posted by Pallavi Rao <pa...@inmobi.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/42150/#review113927
-----------------------------------------------------------
Ship it!
Please update runtime.properties. Other than that, the patch looks good.
common/src/main/java/org/apache/falcon/entity/EntityUtil.java (line 673)
<https://reviews.apache.org/r/42150/#comment174713>
Can you please add all feed.retry.* properties with default values to src/conf/runtime.properties?
- Pallavi Rao
On Jan. 11, 2016, 5:17 p.m., Narayan Periwal wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/42150/
> -----------------------------------------------------------
>
> (Updated Jan. 11, 2016, 5:17 p.m.)
>
>
> Review request for Falcon.
>
>
> Bugs: FALCON-1677
> https://issues.apache.org/jira/browse/FALCON-1677
>
>
> Repository: falcon-git
>
>
> Description
> -------
>
> Currently, Falcon retries only on failure. We should extend support in case of timed-out instances too. Earlier, since we were relying on post-processing to notify the instance status, this was not possible. Now that Falcon relies on Oozie JMS notifications, we can support retries for timed out instances too.
> If a dataset is expected to get delayed for a long time, the user is currently forced to supply a large timeout value. This is an overhead in terms of Oozie having to poll for that long. If we introduce retries, the timeout can be a reasonable value with periodic/exponential back-off retries.
>
>
> Diffs
> -----
>
> client/src/main/java/org/apache/falcon/recipe/RecipeToolOptions.java 0ece90b
> client/src/main/java/org/apache/falcon/recipe/util/RecipeProcessBuilderUtils.java 9477bb4
> client/src/main/resources/process-0.1.xsd 256a29f
> common/src/main/java/org/apache/falcon/entity/EntityUtil.java f448d70
> common/src/test/java/org/apache/falcon/entity/parser/ProcessEntityParserTest.java da22d56
> rerun/src/main/java/org/apache/falcon/rerun/handler/RetryHandler.java 84cd93f
>
> Diff: https://reviews.apache.org/r/42150/diff/
>
>
> Testing
> -------
>
> Done.
>
>
> Thanks,
>
> Narayan Periwal
>
>
Re: Review Request 42150: FALCON-1677: Support re-tries for timed-out
instances
Posted by Narayan Periwal <na...@inmobi.com>.
> On Jan. 12, 2016, 10:26 p.m., Sowmya Ramesh wrote:
> > client/src/main/java/org/apache/falcon/recipe/util/RecipeProcessBuilderUtils.java, line 181
> > <https://reviews.apache.org/r/42150/diff/3/?file=1193927#file1193927line181>
> >
> > Minor nit: Can we also add this to recipe properties file unser "Retry policy properties" section?
Sure. Will add this property there as well.
- Narayan
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/42150/#review114081
-----------------------------------------------------------
On Jan. 12, 2016, 8:38 a.m., Narayan Periwal wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/42150/
> -----------------------------------------------------------
>
> (Updated Jan. 12, 2016, 8:38 a.m.)
>
>
> Review request for Falcon.
>
>
> Bugs: FALCON-1677
> https://issues.apache.org/jira/browse/FALCON-1677
>
>
> Repository: falcon-git
>
>
> Description
> -------
>
> Currently, Falcon retries only on failure. We should extend support in case of timed-out instances too. Earlier, since we were relying on post-processing to notify the instance status, this was not possible. Now that Falcon relies on Oozie JMS notifications, we can support retries for timed out instances too.
> If a dataset is expected to get delayed for a long time, the user is currently forced to supply a large timeout value. This is an overhead in terms of Oozie having to poll for that long. If we introduce retries, the timeout can be a reasonable value with periodic/exponential back-off retries.
>
>
> Diffs
> -----
>
> client/src/main/java/org/apache/falcon/recipe/RecipeToolOptions.java 0ece90b
> client/src/main/java/org/apache/falcon/recipe/util/RecipeProcessBuilderUtils.java 9477bb4
> client/src/main/resources/process-0.1.xsd 256a29f
> common/src/main/java/org/apache/falcon/entity/EntityUtil.java f448d70
> common/src/test/java/org/apache/falcon/entity/parser/ProcessEntityParserTest.java da22d56
> docs/src/site/twiki/EntitySpecification.twiki c4f5f76
> rerun/src/main/java/org/apache/falcon/rerun/handler/RetryHandler.java 84cd93f
> src/conf/runtime.properties f535b0f
>
> Diff: https://reviews.apache.org/r/42150/diff/
>
>
> Testing
> -------
>
> Done.
>
>
> Thanks,
>
> Narayan Periwal
>
>
Re: Review Request 42150: FALCON-1677: Support re-tries for timed-out
instances
Posted by Sowmya Ramesh <sr...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/42150/#review114081
-----------------------------------------------------------
Ship it!
Fix the minor nit and ship it!
client/src/main/java/org/apache/falcon/recipe/util/RecipeProcessBuilderUtils.java (line 181)
<https://reviews.apache.org/r/42150/#comment174871>
Minor nit: Can we also add this to recipe properties file unser "Retry policy properties" section?
- Sowmya Ramesh
On Jan. 12, 2016, 8:38 a.m., Narayan Periwal wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/42150/
> -----------------------------------------------------------
>
> (Updated Jan. 12, 2016, 8:38 a.m.)
>
>
> Review request for Falcon.
>
>
> Bugs: FALCON-1677
> https://issues.apache.org/jira/browse/FALCON-1677
>
>
> Repository: falcon-git
>
>
> Description
> -------
>
> Currently, Falcon retries only on failure. We should extend support in case of timed-out instances too. Earlier, since we were relying on post-processing to notify the instance status, this was not possible. Now that Falcon relies on Oozie JMS notifications, we can support retries for timed out instances too.
> If a dataset is expected to get delayed for a long time, the user is currently forced to supply a large timeout value. This is an overhead in terms of Oozie having to poll for that long. If we introduce retries, the timeout can be a reasonable value with periodic/exponential back-off retries.
>
>
> Diffs
> -----
>
> client/src/main/java/org/apache/falcon/recipe/RecipeToolOptions.java 0ece90b
> client/src/main/java/org/apache/falcon/recipe/util/RecipeProcessBuilderUtils.java 9477bb4
> client/src/main/resources/process-0.1.xsd 256a29f
> common/src/main/java/org/apache/falcon/entity/EntityUtil.java f448d70
> common/src/test/java/org/apache/falcon/entity/parser/ProcessEntityParserTest.java da22d56
> docs/src/site/twiki/EntitySpecification.twiki c4f5f76
> rerun/src/main/java/org/apache/falcon/rerun/handler/RetryHandler.java 84cd93f
> src/conf/runtime.properties f535b0f
>
> Diff: https://reviews.apache.org/r/42150/diff/
>
>
> Testing
> -------
>
> Done.
>
>
> Thanks,
>
> Narayan Periwal
>
>
Re: Review Request 42150: FALCON-1677: Support re-tries for timed-out
instances
Posted by Narayan Periwal <na...@inmobi.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/42150/
-----------------------------------------------------------
(Updated Jan. 13, 2016, 11:36 a.m.)
Review request for Falcon.
Bugs: FALCON-1677
https://issues.apache.org/jira/browse/FALCON-1677
Repository: falcon-git
Description
-------
Currently, Falcon retries only on failure. We should extend support in case of timed-out instances too. Earlier, since we were relying on post-processing to notify the instance status, this was not possible. Now that Falcon relies on Oozie JMS notifications, we can support retries for timed out instances too.
If a dataset is expected to get delayed for a long time, the user is currently forced to supply a large timeout value. This is an overhead in terms of Oozie having to poll for that long. If we introduce retries, the timeout can be a reasonable value with periodic/exponential back-off retries.
Diffs (updated)
-----
addons/recipes/hdfs-replication/src/main/resources/hdfs-replication.properties 7ef4095
addons/recipes/hive-disaster-recovery/src/main/resources/hive-disaster-recovery-secure.properties df4a8fd
addons/recipes/hive-disaster-recovery/src/main/resources/hive-disaster-recovery.properties c2915fb
client/src/main/java/org/apache/falcon/recipe/RecipeToolOptions.java 0ece90b
client/src/main/java/org/apache/falcon/recipe/util/RecipeProcessBuilderUtils.java 9477bb4
client/src/main/resources/process-0.1.xsd 256a29f
common/src/main/java/org/apache/falcon/entity/EntityUtil.java f448d70
common/src/test/java/org/apache/falcon/entity/parser/ProcessEntityParserTest.java da22d56
docs/src/site/twiki/EntitySpecification.twiki c4f5f76
falcon-regression/merlin/src/test/resources/HdfsRecipe/hive-disaster-recovery.properties fb2a4fc
falcon-regression/merlin/src/test/resources/HiveDrRecipe/hive-disaster-recovery.properties dd781a5
falcon-regression/merlin/src/test/resources/HiveDrSecureRecipe/hive-disaster-recovery-secure.properties da0bcd1
rerun/src/main/java/org/apache/falcon/rerun/handler/RetryHandler.java 84cd93f
src/conf/runtime.properties f535b0f
webapp/src/test/resources/hdfs-replication.properties bd77a1c
Diff: https://reviews.apache.org/r/42150/diff/
Testing
-------
Done.
Thanks,
Narayan Periwal
Re: Review Request 42150: FALCON-1677: Support re-tries for timed-out
instances
Posted by Narayan Periwal <na...@inmobi.com>.
> On Jan. 12, 2016, 5:59 p.m., Ajay Yadava wrote:
> > docs/src/site/twiki/EntitySpecification.twiki, line 901
> > <https://reviews.apache.org/r/42150/diff/3/?file=1193931#file1193931line901>
> >
> > nit: Please document how to enable / disable for feeds.
Will add it.
- Narayan
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/42150/#review114016
-----------------------------------------------------------
On Jan. 12, 2016, 8:38 a.m., Narayan Periwal wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/42150/
> -----------------------------------------------------------
>
> (Updated Jan. 12, 2016, 8:38 a.m.)
>
>
> Review request for Falcon.
>
>
> Bugs: FALCON-1677
> https://issues.apache.org/jira/browse/FALCON-1677
>
>
> Repository: falcon-git
>
>
> Description
> -------
>
> Currently, Falcon retries only on failure. We should extend support in case of timed-out instances too. Earlier, since we were relying on post-processing to notify the instance status, this was not possible. Now that Falcon relies on Oozie JMS notifications, we can support retries for timed out instances too.
> If a dataset is expected to get delayed for a long time, the user is currently forced to supply a large timeout value. This is an overhead in terms of Oozie having to poll for that long. If we introduce retries, the timeout can be a reasonable value with periodic/exponential back-off retries.
>
>
> Diffs
> -----
>
> client/src/main/java/org/apache/falcon/recipe/RecipeToolOptions.java 0ece90b
> client/src/main/java/org/apache/falcon/recipe/util/RecipeProcessBuilderUtils.java 9477bb4
> client/src/main/resources/process-0.1.xsd 256a29f
> common/src/main/java/org/apache/falcon/entity/EntityUtil.java f448d70
> common/src/test/java/org/apache/falcon/entity/parser/ProcessEntityParserTest.java da22d56
> docs/src/site/twiki/EntitySpecification.twiki c4f5f76
> rerun/src/main/java/org/apache/falcon/rerun/handler/RetryHandler.java 84cd93f
> src/conf/runtime.properties f535b0f
>
> Diff: https://reviews.apache.org/r/42150/diff/
>
>
> Testing
> -------
>
> Done.
>
>
> Thanks,
>
> Narayan Periwal
>
>
Re: Review Request 42150: FALCON-1677: Support re-tries for timed-out
instances
Posted by Ajay Yadava <aj...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/42150/#review114016
-----------------------------------------------------------
client/src/main/java/org/apache/falcon/recipe/RecipeToolOptions.java (line 41)
<https://reviews.apache.org/r/42150/#comment174801>
nit: RETRY_ON_TIMEOUT
common/src/test/java/org/apache/falcon/entity/parser/ProcessEntityParserTest.java (line 277)
<https://reviews.apache.org/r/42150/#comment174802>
nit: superfluous parentheses
docs/src/site/twiki/EntitySpecification.twiki (line 901)
<https://reviews.apache.org/r/42150/#comment174808>
nit: Please document how to enable / disable for feeds.
- Ajay Yadava
On Jan. 12, 2016, 8:38 a.m., Narayan Periwal wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/42150/
> -----------------------------------------------------------
>
> (Updated Jan. 12, 2016, 8:38 a.m.)
>
>
> Review request for Falcon.
>
>
> Bugs: FALCON-1677
> https://issues.apache.org/jira/browse/FALCON-1677
>
>
> Repository: falcon-git
>
>
> Description
> -------
>
> Currently, Falcon retries only on failure. We should extend support in case of timed-out instances too. Earlier, since we were relying on post-processing to notify the instance status, this was not possible. Now that Falcon relies on Oozie JMS notifications, we can support retries for timed out instances too.
> If a dataset is expected to get delayed for a long time, the user is currently forced to supply a large timeout value. This is an overhead in terms of Oozie having to poll for that long. If we introduce retries, the timeout can be a reasonable value with periodic/exponential back-off retries.
>
>
> Diffs
> -----
>
> client/src/main/java/org/apache/falcon/recipe/RecipeToolOptions.java 0ece90b
> client/src/main/java/org/apache/falcon/recipe/util/RecipeProcessBuilderUtils.java 9477bb4
> client/src/main/resources/process-0.1.xsd 256a29f
> common/src/main/java/org/apache/falcon/entity/EntityUtil.java f448d70
> common/src/test/java/org/apache/falcon/entity/parser/ProcessEntityParserTest.java da22d56
> docs/src/site/twiki/EntitySpecification.twiki c4f5f76
> rerun/src/main/java/org/apache/falcon/rerun/handler/RetryHandler.java 84cd93f
> src/conf/runtime.properties f535b0f
>
> Diff: https://reviews.apache.org/r/42150/diff/
>
>
> Testing
> -------
>
> Done.
>
>
> Thanks,
>
> Narayan Periwal
>
>
Re: Review Request 42150: FALCON-1677: Support re-tries for timed-out
instances
Posted by Narayan Periwal <na...@inmobi.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/42150/
-----------------------------------------------------------
(Updated Jan. 12, 2016, 8:38 a.m.)
Review request for Falcon.
Bugs: FALCON-1677
https://issues.apache.org/jira/browse/FALCON-1677
Repository: falcon-git
Description
-------
Currently, Falcon retries only on failure. We should extend support in case of timed-out instances too. Earlier, since we were relying on post-processing to notify the instance status, this was not possible. Now that Falcon relies on Oozie JMS notifications, we can support retries for timed out instances too.
If a dataset is expected to get delayed for a long time, the user is currently forced to supply a large timeout value. This is an overhead in terms of Oozie having to poll for that long. If we introduce retries, the timeout can be a reasonable value with periodic/exponential back-off retries.
Diffs (updated)
-----
client/src/main/java/org/apache/falcon/recipe/RecipeToolOptions.java 0ece90b
client/src/main/java/org/apache/falcon/recipe/util/RecipeProcessBuilderUtils.java 9477bb4
client/src/main/resources/process-0.1.xsd 256a29f
common/src/main/java/org/apache/falcon/entity/EntityUtil.java f448d70
common/src/test/java/org/apache/falcon/entity/parser/ProcessEntityParserTest.java da22d56
docs/src/site/twiki/EntitySpecification.twiki c4f5f76
rerun/src/main/java/org/apache/falcon/rerun/handler/RetryHandler.java 84cd93f
src/conf/runtime.properties f535b0f
Diff: https://reviews.apache.org/r/42150/diff/
Testing
-------
Done.
Thanks,
Narayan Periwal
Re: Review Request 42150: FALCON-1677: Support re-tries for timed-out
instances
Posted by Narayan Periwal <na...@inmobi.com>.
> On Jan. 12, 2016, 8:28 a.m., Pallavi Rao wrote:
> > docs/src/site/twiki/EntitySpecification.twiki, line 900
> > <https://reviews.apache.org/r/42150/diff/2/?file=1193923#file1193923line900>
> >
> > A line of description explaining onTimeout?
Sure. will add it.
- Narayan
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/42150/#review113953
-----------------------------------------------------------
On Jan. 12, 2016, 8:19 a.m., Narayan Periwal wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/42150/
> -----------------------------------------------------------
>
> (Updated Jan. 12, 2016, 8:19 a.m.)
>
>
> Review request for Falcon.
>
>
> Bugs: FALCON-1677
> https://issues.apache.org/jira/browse/FALCON-1677
>
>
> Repository: falcon-git
>
>
> Description
> -------
>
> Currently, Falcon retries only on failure. We should extend support in case of timed-out instances too. Earlier, since we were relying on post-processing to notify the instance status, this was not possible. Now that Falcon relies on Oozie JMS notifications, we can support retries for timed out instances too.
> If a dataset is expected to get delayed for a long time, the user is currently forced to supply a large timeout value. This is an overhead in terms of Oozie having to poll for that long. If we introduce retries, the timeout can be a reasonable value with periodic/exponential back-off retries.
>
>
> Diffs
> -----
>
> client/src/main/java/org/apache/falcon/recipe/RecipeToolOptions.java 0ece90b
> client/src/main/java/org/apache/falcon/recipe/util/RecipeProcessBuilderUtils.java 9477bb4
> client/src/main/resources/process-0.1.xsd 256a29f
> common/src/main/java/org/apache/falcon/entity/EntityUtil.java f448d70
> common/src/test/java/org/apache/falcon/entity/parser/ProcessEntityParserTest.java da22d56
> docs/src/site/twiki/EntitySpecification.twiki c4f5f76
> rerun/src/main/java/org/apache/falcon/rerun/handler/RetryHandler.java 84cd93f
> src/conf/runtime.properties f535b0f
>
> Diff: https://reviews.apache.org/r/42150/diff/
>
>
> Testing
> -------
>
> Done.
>
>
> Thanks,
>
> Narayan Periwal
>
>
Re: Review Request 42150: FALCON-1677: Support re-tries for timed-out
instances
Posted by Pallavi Rao <pa...@inmobi.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/42150/#review113953
-----------------------------------------------------------
docs/src/site/twiki/EntitySpecification.twiki (line 900)
<https://reviews.apache.org/r/42150/#comment174735>
A line of description explaining onTimeout?
- Pallavi Rao
On Jan. 12, 2016, 8:19 a.m., Narayan Periwal wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/42150/
> -----------------------------------------------------------
>
> (Updated Jan. 12, 2016, 8:19 a.m.)
>
>
> Review request for Falcon.
>
>
> Bugs: FALCON-1677
> https://issues.apache.org/jira/browse/FALCON-1677
>
>
> Repository: falcon-git
>
>
> Description
> -------
>
> Currently, Falcon retries only on failure. We should extend support in case of timed-out instances too. Earlier, since we were relying on post-processing to notify the instance status, this was not possible. Now that Falcon relies on Oozie JMS notifications, we can support retries for timed out instances too.
> If a dataset is expected to get delayed for a long time, the user is currently forced to supply a large timeout value. This is an overhead in terms of Oozie having to poll for that long. If we introduce retries, the timeout can be a reasonable value with periodic/exponential back-off retries.
>
>
> Diffs
> -----
>
> client/src/main/java/org/apache/falcon/recipe/RecipeToolOptions.java 0ece90b
> client/src/main/java/org/apache/falcon/recipe/util/RecipeProcessBuilderUtils.java 9477bb4
> client/src/main/resources/process-0.1.xsd 256a29f
> common/src/main/java/org/apache/falcon/entity/EntityUtil.java f448d70
> common/src/test/java/org/apache/falcon/entity/parser/ProcessEntityParserTest.java da22d56
> docs/src/site/twiki/EntitySpecification.twiki c4f5f76
> rerun/src/main/java/org/apache/falcon/rerun/handler/RetryHandler.java 84cd93f
> src/conf/runtime.properties f535b0f
>
> Diff: https://reviews.apache.org/r/42150/diff/
>
>
> Testing
> -------
>
> Done.
>
>
> Thanks,
>
> Narayan Periwal
>
>
Re: Review Request 42150: FALCON-1677: Support re-tries for timed-out
instances
Posted by Narayan Periwal <na...@inmobi.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/42150/
-----------------------------------------------------------
(Updated Jan. 12, 2016, 8:19 a.m.)
Review request for Falcon.
Bugs: FALCON-1677
https://issues.apache.org/jira/browse/FALCON-1677
Repository: falcon-git
Description
-------
Currently, Falcon retries only on failure. We should extend support in case of timed-out instances too. Earlier, since we were relying on post-processing to notify the instance status, this was not possible. Now that Falcon relies on Oozie JMS notifications, we can support retries for timed out instances too.
If a dataset is expected to get delayed for a long time, the user is currently forced to supply a large timeout value. This is an overhead in terms of Oozie having to poll for that long. If we introduce retries, the timeout can be a reasonable value with periodic/exponential back-off retries.
Diffs (updated)
-----
client/src/main/java/org/apache/falcon/recipe/RecipeToolOptions.java 0ece90b
client/src/main/java/org/apache/falcon/recipe/util/RecipeProcessBuilderUtils.java 9477bb4
client/src/main/resources/process-0.1.xsd 256a29f
common/src/main/java/org/apache/falcon/entity/EntityUtil.java f448d70
common/src/test/java/org/apache/falcon/entity/parser/ProcessEntityParserTest.java da22d56
docs/src/site/twiki/EntitySpecification.twiki c4f5f76
rerun/src/main/java/org/apache/falcon/rerun/handler/RetryHandler.java 84cd93f
src/conf/runtime.properties f535b0f
Diff: https://reviews.apache.org/r/42150/diff/
Testing
-------
Done.
Thanks,
Narayan Periwal