You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flume.apache.org by Steve Hoffman <st...@goofy.net> on 2013/01/17 16:08:09 UTC
Re: Review Request: FLUME-1685: ExecSource shouldn't die if the channel is
full
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/9003/
-----------------------------------------------------------
(Updated Jan. 17, 2013, 3:08 p.m.)
Review request for Flume.
Changes
-------
Just capitalized the FLUME in the title
Summary (updated)
-----------------
FLUME-1685: ExecSource shouldn't die if the channel is full
Description
-------
Rather than kill the source outright when a channel fills, the channel should throw away the event (since there is nowhere to put it). A basic back-off mechanism is also implemented so retries get spaced out the longer the channel remains full up to a configurable limit.
Patches are attached to FLUME-1685 including a backport to the 1.2 branch (which I use with Cloudera's cdh3u5 distribution)
This addresses bug FLUME-1685.
https://issues.apache.org/jira/browse/FLUME-1685
Diffs
-----
.gitignore 08c33a0
flume-ng-core/pom.xml d01baca
flume-ng-core/src/main/java/org/apache/flume/source/ExecSource.java 46f672f
flume-ng-core/src/main/java/org/apache/flume/source/ExecSourceConfigurationConstants.java 0ba0508
flume-ng-core/src/test/java/org/apache/flume/source/TestExecSource.java 8bcf320
Diff: https://reviews.apache.org/r/9003/diff/
Testing
-------
A unit test was created to test the full channel condition.
This patch has been running in our production environment for 2 months and has successfully recovered several times from full channel events.
Thanks,
Steve Hoffman
Re: Review Request: FLUME-1685: ExecSource shouldn't die if the channel is
full
Posted by Steve Hoffman <st...@goofy.net>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/9003/#review15454
-----------------------------------------------------------
.gitignore
<https://reviews.apache.org/r/9003/#comment33333>
I added these other intellij files as ignores since you already included the iml files and we wouldn't want this committed accidentally.
flume-ng-core/pom.xml
<https://reviews.apache.org/r/9003/#comment33334>
Added mocking framework for unit testing
flume-ng-core/src/main/java/org/apache/flume/source/ExecSource.java
<https://reviews.apache.org/r/9003/#comment33335>
You don't override an interface so I removed this. IntelliJ (correctly) complains.
flume-ng-core/src/main/java/org/apache/flume/source/ExecSource.java
<https://reviews.apache.org/r/9003/#comment33336>
You don't override an interface so I removed this. IntelliJ (correctly) complains.
- Steve Hoffman
On Jan. 17, 2013, 3:08 p.m., Steve Hoffman wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/9003/
> -----------------------------------------------------------
>
> (Updated Jan. 17, 2013, 3:08 p.m.)
>
>
> Review request for Flume.
>
>
> Description
> -------
>
> Rather than kill the source outright when a channel fills, the channel should throw away the event (since there is nowhere to put it). A basic back-off mechanism is also implemented so retries get spaced out the longer the channel remains full up to a configurable limit.
>
> Patches are attached to FLUME-1685 including a backport to the 1.2 branch (which I use with Cloudera's cdh3u5 distribution)
>
>
> This addresses bug FLUME-1685.
> https://issues.apache.org/jira/browse/FLUME-1685
>
>
> Diffs
> -----
>
> .gitignore 08c33a0
> flume-ng-core/pom.xml d01baca
> flume-ng-core/src/main/java/org/apache/flume/source/ExecSource.java 46f672f
> flume-ng-core/src/main/java/org/apache/flume/source/ExecSourceConfigurationConstants.java 0ba0508
> flume-ng-core/src/test/java/org/apache/flume/source/TestExecSource.java 8bcf320
>
> Diff: https://reviews.apache.org/r/9003/diff/
>
>
> Testing
> -------
>
> A unit test was created to test the full channel condition.
> This patch has been running in our production environment for 2 months and has successfully recovered several times from full channel events.
>
>
> Thanks,
>
> Steve Hoffman
>
>
Re: Review Request: FLUME-1685: ExecSource shouldn't die if the channel is
full
Posted by Mike Percy <mp...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/9003/#review15720
-----------------------------------------------------------
Hi Steve,
I'm not a fan of dropping events. How about providing a configuration option here:
1. Default behavior: Upon ChannelException, hold the current Event batch in memory, back off exponentially (up to a configured MAX_BACKOFF), and retry until it succeeds. Do not read additional data from the InputStream until the put() succeeds.
2. Configuration option to drop when the channel fills up (i.e., this patch)
There is nothing in Flume today that knowingly throws away data by default.
- Mike Percy
On Jan. 17, 2013, 3:08 p.m., Steve Hoffman wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/9003/
> -----------------------------------------------------------
>
> (Updated Jan. 17, 2013, 3:08 p.m.)
>
>
> Review request for Flume.
>
>
> Description
> -------
>
> Rather than kill the source outright when a channel fills, the channel should throw away the event (since there is nowhere to put it). A basic back-off mechanism is also implemented so retries get spaced out the longer the channel remains full up to a configurable limit.
>
> Patches are attached to FLUME-1685 including a backport to the 1.2 branch (which I use with Cloudera's cdh3u5 distribution)
>
>
> This addresses bug FLUME-1685.
> https://issues.apache.org/jira/browse/FLUME-1685
>
>
> Diffs
> -----
>
> .gitignore 08c33a0
> flume-ng-core/pom.xml d01baca
> flume-ng-core/src/main/java/org/apache/flume/source/ExecSource.java 46f672f
> flume-ng-core/src/main/java/org/apache/flume/source/ExecSourceConfigurationConstants.java 0ba0508
> flume-ng-core/src/test/java/org/apache/flume/source/TestExecSource.java 8bcf320
>
> Diff: https://reviews.apache.org/r/9003/diff/
>
>
> Testing
> -------
>
> A unit test was created to test the full channel condition.
> This patch has been running in our production environment for 2 months and has successfully recovered several times from full channel events.
>
>
> Thanks,
>
> Steve Hoffman
>
>