You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flume.apache.org by "Prasad Mujumdar (JIRA)" <ji...@apache.org> on 2011/09/08 00:57:09 UTC
[jira] [Created] (FLUME-762) DirectDriver should reopen the
source/sink if the append or next throws an exception
DirectDriver should reopen the source/sink if the append or next throws an exception
------------------------------------------------------------------------------------
Key: FLUME-762
URL: https://issues.apache.org/jira/browse/FLUME-762
Project: Flume
Issue Type: Improvement
Components: Node
Reporter: Prasad Mujumdar
Assignee: Prasad Mujumdar
Currently, if source or sink throws an exception, the driver bails out resulting the everything to shut down. Alternately, we can try to close and reopen the source or sink, and then attempt to continue. This way we can avoid blocking the flow. If the reopen fails, then it can bail out.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-762) DirectDriver should reopen the
source/sink if the append or next throws an exception
Posted by "jiraposter@reviews.apache.org (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13108994#comment-13108994 ]
jiraposter@reviews.apache.org commented on FLUME-762:
-----------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1739/
-----------------------------------------------------------
(Updated 2011-09-20 20:54:45.230862)
Review request for jmhsieh.
Changes
-------
It looks like the Thrift source is converting an Interrupt exception to IOException. Due to that, TestDiskFailoverBehavior hangs with as we restart the source after IO exception. I think we should bail out instead of retrying in such cases.
With the following change is passing
+ if (eI.getCause() instanceof InterruptedException)
+ throw eI;
Ideally, Thrift source needs to throw the Interrupted exception directly. I will go ahead and log a separate Jira for the.
Summary
-------
If the source or sink throws an exception, close, reopen and retry it. This way the flow can continue after minor/recoverable errors.
This addresses bug FLUME-762.
https://issues.apache.org/jira/browse/FLUME-762
Diffs (updated)
-----
flume-core/src/main/java/com/cloudera/flume/core/connector/DirectDriver.java 0466394
flume-core/src/test/java/com/cloudera/flume/agent/TestFlumeNode.java 9d9e1a5
flume-core/src/test/java/com/cloudera/flume/agent/diskfailover/TestDiskFailoverDeco.java a7e2d78
flume-core/src/test/java/com/cloudera/flume/core/connector/TestDirectDriverExp.java PRE-CREATION
Diff: https://reviews.apache.org/r/1739/diff
Testing
-------
Thanks,
Prasad
> DirectDriver should reopen the source/sink if the append or next throws an exception
> ------------------------------------------------------------------------------------
>
> Key: FLUME-762
> URL: https://issues.apache.org/jira/browse/FLUME-762
> Project: Flume
> Issue Type: Improvement
> Components: Node
> Reporter: Prasad Mujumdar
> Assignee: Prasad Mujumdar
> Attachments: flume-762.patch
>
>
> Currently, if source or sink throws an exception, the driver bails out resulting the everything to shut down. Alternately, we can try to close and reopen the source or sink, and then attempt to continue. This way we can avoid blocking the flow. If the reopen fails, then it can bail out.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-762) DirectDriver should reopen the
source/sink if the append or next throws an exception
Posted by "jiraposter@reviews.apache.org (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13100500#comment-13100500 ]
jiraposter@reviews.apache.org commented on FLUME-762:
-----------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1739/
-----------------------------------------------------------
(Updated 2011-09-08 17:51:55.402334)
Review request for jmhsieh.
Changes
-------
Added more comments in the test to explain each scenario
corrected the copy/paste problem in the warning message in DriverDriver.java ;)
changed the println to LOG.info
Summary
-------
If the source or sink throws an exception, close, reopen and retry it. This way the flow can continue after minor/recoverable errors.
This addresses bug FLUME-762.
https://issues.apache.org/jira/browse/FLUME-762
Diffs (updated)
-----
flume-core/src/main/java/com/cloudera/flume/core/connector/DirectDriver.java 0466394
flume-core/src/test/java/com/cloudera/flume/core/connector/TestDirectDriverExp.java PRE-CREATION
Diff: https://reviews.apache.org/r/1739/diff
Testing
-------
Thanks,
Prasad
> DirectDriver should reopen the source/sink if the append or next throws an exception
> ------------------------------------------------------------------------------------
>
> Key: FLUME-762
> URL: https://issues.apache.org/jira/browse/FLUME-762
> Project: Flume
> Issue Type: Improvement
> Components: Node
> Reporter: Prasad Mujumdar
> Assignee: Prasad Mujumdar
> Attachments: flume-762.patch
>
>
> Currently, if source or sink throws an exception, the driver bails out resulting the everything to shut down. Alternately, we can try to close and reopen the source or sink, and then attempt to continue. This way we can avoid blocking the flow. If the reopen fails, then it can bail out.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (FLUME-762) DirectDriver should reopen the
source/sink if the append or next throws an exception
Posted by "Jonathan Hsieh (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Hsieh updated FLUME-762:
---------------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
> DirectDriver should reopen the source/sink if the append or next throws an exception
> ------------------------------------------------------------------------------------
>
> Key: FLUME-762
> URL: https://issues.apache.org/jira/browse/FLUME-762
> Project: Flume
> Issue Type: Improvement
> Components: Node
> Reporter: Prasad Mujumdar
> Assignee: Prasad Mujumdar
> Attachments: flume-762.patch, flume-762.patch
>
>
> Currently, if source or sink throws an exception, the driver bails out resulting the everything to shut down. Alternately, we can try to close and reopen the source or sink, and then attempt to continue. This way we can avoid blocking the flow. If the reopen fails, then it can bail out.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-762) DirectDriver should reopen the
source/sink if the append or next throws an exception
Posted by "jiraposter@reviews.apache.org (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13101646#comment-13101646 ]
jiraposter@reviews.apache.org commented on FLUME-762:
-----------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1739/
-----------------------------------------------------------
(Updated 2011-09-09 23:27:39.389285)
Review request for jmhsieh.
Changes
-------
Cleaned up the test, updated comments.
Summary
-------
If the source or sink throws an exception, close, reopen and retry it. This way the flow can continue after minor/recoverable errors.
This addresses bug FLUME-762.
https://issues.apache.org/jira/browse/FLUME-762
Diffs (updated)
-----
flume-core/src/main/java/com/cloudera/flume/core/connector/DirectDriver.java 0466394
flume-core/src/test/java/com/cloudera/flume/core/connector/TestDirectDriverExp.java PRE-CREATION
Diff: https://reviews.apache.org/r/1739/diff
Testing
-------
Thanks,
Prasad
> DirectDriver should reopen the source/sink if the append or next throws an exception
> ------------------------------------------------------------------------------------
>
> Key: FLUME-762
> URL: https://issues.apache.org/jira/browse/FLUME-762
> Project: Flume
> Issue Type: Improvement
> Components: Node
> Reporter: Prasad Mujumdar
> Assignee: Prasad Mujumdar
> Attachments: flume-762.patch
>
>
> Currently, if source or sink throws an exception, the driver bails out resulting the everything to shut down. Alternately, we can try to close and reopen the source or sink, and then attempt to continue. This way we can avoid blocking the flow. If the reopen fails, then it can bail out.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-762) DirectDriver should reopen the
source/sink if the append or next throws an exception
Posted by "jiraposter@reviews.apache.org (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13103934#comment-13103934 ]
jiraposter@reviews.apache.org commented on FLUME-762:
-----------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1739/#review1875
-----------------------------------------------------------
Applied patch, these unit tests pass. However, to cases broke:
com.cloudera.flume.agent.TestFlumeNode.testRestartNextException
com.cloudera.flume.agent.diskfailover.TestDiskFailoverDeco.testHandleExns
Please fix.
flume-core/src/test/java/com/cloudera/flume/core/connector/TestDirectDriverExp.java
<https://reviews.apache.org/r/1739/#comment4317>
This seems a little funny, but we'll deal with this in a follow on jira.
- jmhsieh
On 2011-09-12 17:39:06, Prasad Mujumdar wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/1739/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2011-09-12 17:39:06)
bq.
bq.
bq. Review request for jmhsieh.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. If the source or sink throws an exception, close, reopen and retry it. This way the flow can continue after minor/recoverable errors.
bq.
bq.
bq. This addresses bug FLUME-762.
bq. https://issues.apache.org/jira/browse/FLUME-762
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. flume-core/src/main/java/com/cloudera/flume/core/connector/DirectDriver.java 0466394
bq. flume-core/src/test/java/com/cloudera/flume/core/connector/TestDirectDriverExp.java PRE-CREATION
bq.
bq. Diff: https://reviews.apache.org/r/1739/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq.
bq. Thanks,
bq.
bq. Prasad
bq.
bq.
> DirectDriver should reopen the source/sink if the append or next throws an exception
> ------------------------------------------------------------------------------------
>
> Key: FLUME-762
> URL: https://issues.apache.org/jira/browse/FLUME-762
> Project: Flume
> Issue Type: Improvement
> Components: Node
> Reporter: Prasad Mujumdar
> Assignee: Prasad Mujumdar
> Attachments: flume-762.patch
>
>
> Currently, if source or sink throws an exception, the driver bails out resulting the everything to shut down. Alternately, we can try to close and reopen the source or sink, and then attempt to continue. This way we can avoid blocking the flow. If the reopen fails, then it can bail out.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-762) DirectDriver should reopen the
source/sink if the append or next throws an exception
Posted by "jiraposter@reviews.apache.org (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13101654#comment-13101654 ]
jiraposter@reviews.apache.org commented on FLUME-762:
-----------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1739/#review1844
-----------------------------------------------------------
Hey Prasad, its really close. Please address the nits and we'll try to commit.
flume-core/src/test/java/com/cloudera/flume/core/connector/TestDirectDriverExp.java
<https://reviews.apache.org/r/1739/#comment4216>
Nit: Typo
flume-core/src/test/java/com/cloudera/flume/core/connector/TestDirectDriverExp.java
<https://reviews.apache.org/r/1739/#comment4225>
test name misspelled?
flume-core/src/test/java/com/cloudera/flume/core/connector/TestDirectDriverExp.java
<https://reviews.apache.org/r/1739/#comment4221>
assertNull(driver.getException())
flume-core/src/test/java/com/cloudera/flume/core/connector/TestDirectDriverExp.java
<https://reviews.apache.org/r/1739/#comment4222>
assertNull(driver.getException())
flume-core/src/test/java/com/cloudera/flume/core/connector/TestDirectDriverExp.java
<https://reviews.apache.org/r/1739/#comment4224>
Test name misspelled.
flume-core/src/test/java/com/cloudera/flume/core/connector/TestDirectDriverExp.java
<https://reviews.apache.org/r/1739/#comment4223>
assertTrue(driver.getException() instanceof InterruptedException)
- jmhsieh
On 2011-09-09 23:27:39, Prasad Mujumdar wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/1739/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2011-09-09 23:27:39)
bq.
bq.
bq. Review request for jmhsieh.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. If the source or sink throws an exception, close, reopen and retry it. This way the flow can continue after minor/recoverable errors.
bq.
bq.
bq. This addresses bug FLUME-762.
bq. https://issues.apache.org/jira/browse/FLUME-762
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. flume-core/src/main/java/com/cloudera/flume/core/connector/DirectDriver.java 0466394
bq. flume-core/src/test/java/com/cloudera/flume/core/connector/TestDirectDriverExp.java PRE-CREATION
bq.
bq. Diff: https://reviews.apache.org/r/1739/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq.
bq. Thanks,
bq.
bq. Prasad
bq.
bq.
> DirectDriver should reopen the source/sink if the append or next throws an exception
> ------------------------------------------------------------------------------------
>
> Key: FLUME-762
> URL: https://issues.apache.org/jira/browse/FLUME-762
> Project: Flume
> Issue Type: Improvement
> Components: Node
> Reporter: Prasad Mujumdar
> Assignee: Prasad Mujumdar
> Attachments: flume-762.patch
>
>
> Currently, if source or sink throws an exception, the driver bails out resulting the everything to shut down. Alternately, we can try to close and reopen the source or sink, and then attempt to continue. This way we can avoid blocking the flow. If the reopen fails, then it can bail out.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-762) DirectDriver should reopen the
source/sink if the append or next throws an exception
Posted by "jiraposter@reviews.apache.org (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13103936#comment-13103936 ]
jiraposter@reviews.apache.org commented on FLUME-762:
-----------------------------------------------------
bq. On 2011-09-13 19:55:07, jmhsieh wrote:
bq. > Applied patch, these unit tests pass. However, to cases broke:
bq. >
bq. > com.cloudera.flume.agent.TestFlumeNode.testRestartNextException
bq. > com.cloudera.flume.agent.diskfailover.TestDiskFailoverDeco.testHandleExns
bq. >
bq. > Please fix.
Rephrase: The unit test included pass, but the code broke two others (which I think are related)
- jmhsieh
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1739/#review1875
-----------------------------------------------------------
On 2011-09-12 17:39:06, Prasad Mujumdar wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/1739/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2011-09-12 17:39:06)
bq.
bq.
bq. Review request for jmhsieh.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. If the source or sink throws an exception, close, reopen and retry it. This way the flow can continue after minor/recoverable errors.
bq.
bq.
bq. This addresses bug FLUME-762.
bq. https://issues.apache.org/jira/browse/FLUME-762
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. flume-core/src/main/java/com/cloudera/flume/core/connector/DirectDriver.java 0466394
bq. flume-core/src/test/java/com/cloudera/flume/core/connector/TestDirectDriverExp.java PRE-CREATION
bq.
bq. Diff: https://reviews.apache.org/r/1739/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq.
bq. Thanks,
bq.
bq. Prasad
bq.
bq.
> DirectDriver should reopen the source/sink if the append or next throws an exception
> ------------------------------------------------------------------------------------
>
> Key: FLUME-762
> URL: https://issues.apache.org/jira/browse/FLUME-762
> Project: Flume
> Issue Type: Improvement
> Components: Node
> Reporter: Prasad Mujumdar
> Assignee: Prasad Mujumdar
> Attachments: flume-762.patch
>
>
> Currently, if source or sink throws an exception, the driver bails out resulting the everything to shut down. Alternately, we can try to close and reopen the source or sink, and then attempt to continue. This way we can avoid blocking the flow. If the reopen fails, then it can bail out.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (FLUME-762) DirectDriver should reopen the
source/sink if the append or next throws an exception
Posted by "Arvind Prabhakar (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arvind Prabhakar updated FLUME-762:
-----------------------------------
Fix Version/s: v0.9.5
> DirectDriver should reopen the source/sink if the append or next throws an exception
> ------------------------------------------------------------------------------------
>
> Key: FLUME-762
> URL: https://issues.apache.org/jira/browse/FLUME-762
> Project: Flume
> Issue Type: Improvement
> Components: Node
> Reporter: Prasad Mujumdar
> Assignee: Prasad Mujumdar
> Fix For: v0.9.5
>
> Attachments: flume-762.patch, flume-762.patch
>
>
> Currently, if source or sink throws an exception, the driver bails out resulting the everything to shut down. Alternately, we can try to close and reopen the source or sink, and then attempt to continue. This way we can avoid blocking the flow. If the reopen fails, then it can bail out.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-762) DirectDriver should reopen the
source/sink if the append or next throws an exception
Posted by "jiraposter@reviews.apache.org (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13100224#comment-13100224 ]
jiraposter@reviews.apache.org commented on FLUME-762:
-----------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1739/#review1815
-----------------------------------------------------------
Prasad, no worry about the spacing, I'm guilty of this quite a bit.
My main concern is
1) all three test essentially look the same -- just from reading the tests I don't know what the difference is supposed to be.
2) the tests don't seem to verify anything, and their descriptions make me think an exception would be expected (but they aren't).
also have some nits and a few other suggestions.
flume-core/src/test/java/com/cloudera/flume/core/connector/TestDirectDriverExp.java
<https://reviews.apache.org/r/1739/#comment4166>
LOG instead of println.
flume-core/src/main/java/com/cloudera/flume/core/connector/DirectDriver.java
<https://reviews.apache.org/r/1739/#comment4167>
Log when interrupted?
flume-core/src/main/java/com/cloudera/flume/core/connector/DirectDriver.java
<https://reviews.apache.org/r/1739/#comment4168>
Log when interrupted?
flume-core/src/main/java/com/cloudera/flume/core/connector/DirectDriver.java
<https://reviews.apache.org/r/1739/#comment4176>
nit: Are these tabs?
flume-core/src/main/java/com/cloudera/flume/core/connector/DirectDriver.java
<https://reviews.apache.org/r/1739/#comment4169>
I think this is what I mean in my previous comment:
LOG.warn("Exception in sink: " + sink.getName(), eI);
The warn/info/error method's signature is essentially:
void warn(String string, Exception ex)
flume-core/src/test/java/com/cloudera/flume/core/connector/TestDirectDriverExp.java
<https://reviews.apache.org/r/1739/#comment4172>
Nit: you could just just have a Exception as a field, and pass it in as a constructor argument. When it is time to throw it, just throw it.
I think it would be cleaner that way. Bonus: This would make the tests easier to read as well! (oh, this one throws IOException, and this one throws InterruptedException)!
flume-core/src/test/java/com/cloudera/flume/core/connector/TestDirectDriverExp.java
<https://reviews.apache.org/r/1739/#comment4170>
LOG.info
flume-core/src/test/java/com/cloudera/flume/core/connector/TestDirectDriverExp.java
<https://reviews.apache.org/r/1739/#comment4171>
LOG.info
flume-core/src/test/java/com/cloudera/flume/core/connector/TestDirectDriverExp.java
<https://reviews.apache.org/r/1739/#comment4174>
How can I tell this did what was expected?
flume-core/src/test/java/com/cloudera/flume/core/connector/TestDirectDriverExp.java
<https://reviews.apache.org/r/1739/#comment4178>
The description makes me think this should be
@Test(expected=IOException.class)
(it isn't and shouldn't be).
Please checksomething to convince the reader it works.
flume-core/src/test/java/com/cloudera/flume/core/connector/TestDirectDriverExp.java
<https://reviews.apache.org/r/1739/#comment4173>
How can I tell if this did what i expected?
flume-core/src/test/java/com/cloudera/flume/core/connector/TestDirectDriverExp.java
<https://reviews.apache.org/r/1739/#comment4177>
Description makes me think this should be:
@Test(expected=InterruptedException.class)
Needs better comment or better verfication of something in test.
flume-core/src/test/java/com/cloudera/flume/core/connector/TestDirectDriverExp.java
<https://reviews.apache.org/r/1739/#comment4175>
How can I tell this did what was expected?
- jmhsieh
On 2011-09-08 07:25:11, Prasad Mujumdar wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/1739/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2011-09-08 07:25:11)
bq.
bq.
bq. Review request for jmhsieh.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. If the source or sink throws an exception, close, reopen and retry it. This way the flow can continue after minor/recoverable errors.
bq.
bq.
bq. This addresses bug FLUME-762.
bq. https://issues.apache.org/jira/browse/FLUME-762
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. flume-core/src/main/java/com/cloudera/flume/core/connector/DirectDriver.java 0466394
bq. flume-core/src/test/java/com/cloudera/flume/core/connector/TestDirectDriverExp.java PRE-CREATION
bq.
bq. Diff: https://reviews.apache.org/r/1739/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq.
bq. Thanks,
bq.
bq. Prasad
bq.
bq.
> DirectDriver should reopen the source/sink if the append or next throws an exception
> ------------------------------------------------------------------------------------
>
> Key: FLUME-762
> URL: https://issues.apache.org/jira/browse/FLUME-762
> Project: Flume
> Issue Type: Improvement
> Components: Node
> Reporter: Prasad Mujumdar
> Assignee: Prasad Mujumdar
> Attachments: flume-762.patch
>
>
> Currently, if source or sink throws an exception, the driver bails out resulting the everything to shut down. Alternately, we can try to close and reopen the source or sink, and then attempt to continue. This way we can avoid blocking the flow. If the reopen fails, then it can bail out.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-762) DirectDriver should reopen the
source/sink if the append or next throws an exception
Posted by "jiraposter@reviews.apache.org (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13099699#comment-13099699 ]
jiraposter@reviews.apache.org commented on FLUME-762:
-----------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1739/#review1806
-----------------------------------------------------------
I think if we get an interrupted exception that percolates up, we need to bail out. Can you add a test for this? (also, since it may be the third time source is implemented, maybe refactor?).
Also, to avoid having 2 second long tests, you can have next return null after some count. This means that source is out of data and should exit the driver "cleanly".
Nits: are you using tabs? please replace them with spaces.
flume-core/src/main/java/com/cloudera/flume/core/connector/DirectDriver.java
<https://reviews.apache.org/r/1739/#comment4148>
I'm pretty convinced we need to bail out completely like before if we get InterruptedExceptions here.
IO and any other runtime should retry once.
In either case, I think we should check the stopped variable before retrying.
flume-core/src/main/java/com/cloudera/flume/core/connector/DirectDriver.java
<https://reviews.apache.org/r/1739/#comment4149>
nit: formatting looks funny.
- LOG not indented properly.
- extra trailing spaces
flume-core/src/main/java/com/cloudera/flume/core/connector/DirectDriver.java
<https://reviews.apache.org/r/1739/#comment4153>
Maybe say "Unexpected exception in source" ...
Also, use LOG.warn("string", eI). Doing eI.printStackTrace might not make the log file!
flume-core/src/main/java/com/cloudera/flume/core/connector/DirectDriver.java
<https://reviews.apache.org/r/1739/#comment4151>
Add something about "attempting to recover" in the log message?
flume-core/src/main/java/com/cloudera/flume/core/connector/DirectDriver.java
<https://reviews.apache.org/r/1739/#comment4150>
nit: is this a tab?
flume-core/src/main/java/com/cloudera/flume/core/connector/DirectDriver.java
<https://reviews.apache.org/r/1739/#comment4155>
Same comments about exception logging and Interrupted Exception.
flume-core/src/main/java/com/cloudera/flume/core/connector/DirectDriver.java
<https://reviews.apache.org/r/1739/#comment4154>
Might want to update here saying that we attempted to recover but could not.
flume-core/src/test/java/com/cloudera/flume/core/connector/TestDirectDriverExp.java
<https://reviews.apache.org/r/1739/#comment4156>
funny indents.
flume-core/src/test/java/com/cloudera/flume/core/connector/TestDirectDriverExp.java
<https://reviews.apache.org/r/1739/#comment4157>
Need to check something to make this a valid test? (or add comment/log make it clear that success is if no exception happens).
flume-core/src/test/java/com/cloudera/flume/core/connector/TestDirectDriverExp.java
<https://reviews.apache.org/r/1739/#comment4158>
formatting funny again.
- jmhsieh
On 2011-09-07 23:18:39, Prasad Mujumdar wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/1739/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2011-09-07 23:18:39)
bq.
bq.
bq. Review request for jmhsieh.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. If the source or sink throws an exception, close, reopen and retry it. This way the flow can continue after minor/recoverable errors.
bq.
bq.
bq. This addresses bug FLUME-762.
bq. https://issues.apache.org/jira/browse/FLUME-762
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. flume-core/src/main/java/com/cloudera/flume/core/connector/DirectDriver.java 0466394
bq. flume-core/src/test/java/com/cloudera/flume/core/connector/TestDirectDriverExp.java PRE-CREATION
bq.
bq. Diff: https://reviews.apache.org/r/1739/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq.
bq. Thanks,
bq.
bq. Prasad
bq.
bq.
> DirectDriver should reopen the source/sink if the append or next throws an exception
> ------------------------------------------------------------------------------------
>
> Key: FLUME-762
> URL: https://issues.apache.org/jira/browse/FLUME-762
> Project: Flume
> Issue Type: Improvement
> Components: Node
> Reporter: Prasad Mujumdar
> Assignee: Prasad Mujumdar
> Attachments: flume-762.patch
>
>
> Currently, if source or sink throws an exception, the driver bails out resulting the everything to shut down. Alternately, we can try to close and reopen the source or sink, and then attempt to continue. This way we can avoid blocking the flow. If the reopen fails, then it can bail out.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-762) DirectDriver should reopen the
source/sink if the append or next throws an exception
Posted by "jiraposter@reviews.apache.org (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13105169#comment-13105169 ]
jiraposter@reviews.apache.org commented on FLUME-762:
-----------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1739/
-----------------------------------------------------------
(Updated 2011-09-15 07:12:10.592511)
Review request for jmhsieh.
Changes
-------
updated the tests that failed due to this changes -
com.cloudera.flume.agent.TestFlumeNode.testRestartNextException
com.cloudera.flume.agent.diskfailover.TestDiskFailoverDeco.testHandleExns
Summary
-------
If the source or sink throws an exception, close, reopen and retry it. This way the flow can continue after minor/recoverable errors.
This addresses bug FLUME-762.
https://issues.apache.org/jira/browse/FLUME-762
Diffs (updated)
-----
flume-core/src/test/java/com/cloudera/flume/agent/TestFlumeNode.java 9d9e1a5
flume-core/src/test/java/com/cloudera/flume/agent/diskfailover/TestDiskFailoverDeco.java a7e2d78
flume-core/src/test/java/com/cloudera/flume/core/connector/TestDirectDriverExp.java PRE-CREATION
flume-core/src/main/java/com/cloudera/flume/core/connector/DirectDriver.java 0466394
Diff: https://reviews.apache.org/r/1739/diff
Testing
-------
Thanks,
Prasad
> DirectDriver should reopen the source/sink if the append or next throws an exception
> ------------------------------------------------------------------------------------
>
> Key: FLUME-762
> URL: https://issues.apache.org/jira/browse/FLUME-762
> Project: Flume
> Issue Type: Improvement
> Components: Node
> Reporter: Prasad Mujumdar
> Assignee: Prasad Mujumdar
> Attachments: flume-762.patch
>
>
> Currently, if source or sink throws an exception, the driver bails out resulting the everything to shut down. Alternately, we can try to close and reopen the source or sink, and then attempt to continue. This way we can avoid blocking the flow. If the reopen fails, then it can bail out.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-762) DirectDriver should reopen the
source/sink if the append or next throws an exception
Posted by "jiraposter@reviews.apache.org (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13099652#comment-13099652 ]
jiraposter@reviews.apache.org commented on FLUME-762:
-----------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1739/
-----------------------------------------------------------
Review request for jmhsieh.
Summary
-------
If the source or sink throws an exception, close, reopen and retry it. This way the flow can continue after minor/recoverable errors.
This addresses bug FLUME-762.
https://issues.apache.org/jira/browse/FLUME-762
Diffs
-----
flume-core/src/main/java/com/cloudera/flume/core/connector/DirectDriver.java 0466394
flume-core/src/test/java/com/cloudera/flume/core/connector/TestDirectDriverExp.java PRE-CREATION
Diff: https://reviews.apache.org/r/1739/diff
Testing
-------
Thanks,
Prasad
> DirectDriver should reopen the source/sink if the append or next throws an exception
> ------------------------------------------------------------------------------------
>
> Key: FLUME-762
> URL: https://issues.apache.org/jira/browse/FLUME-762
> Project: Flume
> Issue Type: Improvement
> Components: Node
> Reporter: Prasad Mujumdar
> Assignee: Prasad Mujumdar
> Attachments: flume-762.patch
>
>
> Currently, if source or sink throws an exception, the driver bails out resulting the everything to shut down. Alternately, we can try to close and reopen the source or sink, and then attempt to continue. This way we can avoid blocking the flow. If the reopen fails, then it can bail out.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-762) DirectDriver should reopen the
source/sink if the append or next throws an exception
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13117917#comment-13117917 ]
jiraposter@reviews.apache.org commented on FLUME-762:
-----------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1739/#review2213
-----------------------------------------------------------
Ship it!
I had to upgrade to a dedicated machine to get tests to pass cleanly. I'll take care of the spacing nits before committing.
Thanks Prasad!
- jmhsieh
On 2011-09-20 20:54:45, Prasad Mujumdar wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/1739/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2011-09-20 20:54:45)
bq.
bq.
bq. Review request for jmhsieh.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. If the source or sink throws an exception, close, reopen and retry it. This way the flow can continue after minor/recoverable errors.
bq.
bq.
bq. This addresses bug FLUME-762.
bq. https://issues.apache.org/jira/browse/FLUME-762
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. flume-core/src/main/java/com/cloudera/flume/core/connector/DirectDriver.java 0466394
bq. flume-core/src/test/java/com/cloudera/flume/agent/TestFlumeNode.java 9d9e1a5
bq. flume-core/src/test/java/com/cloudera/flume/agent/diskfailover/TestDiskFailoverDeco.java a7e2d78
bq. flume-core/src/test/java/com/cloudera/flume/core/connector/TestDirectDriverExp.java PRE-CREATION
bq.
bq. Diff: https://reviews.apache.org/r/1739/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq.
bq. Thanks,
bq.
bq. Prasad
bq.
bq.
> DirectDriver should reopen the source/sink if the append or next throws an exception
> ------------------------------------------------------------------------------------
>
> Key: FLUME-762
> URL: https://issues.apache.org/jira/browse/FLUME-762
> Project: Flume
> Issue Type: Improvement
> Components: Node
> Reporter: Prasad Mujumdar
> Assignee: Prasad Mujumdar
> Attachments: flume-762.patch
>
>
> Currently, if source or sink throws an exception, the driver bails out resulting the everything to shut down. Alternately, we can try to close and reopen the source or sink, and then attempt to continue. This way we can avoid blocking the flow. If the reopen fails, then it can bail out.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-762) DirectDriver should reopen the
source/sink if the append or next throws an exception
Posted by "jiraposter@reviews.apache.org (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13105730#comment-13105730 ]
jiraposter@reviews.apache.org commented on FLUME-762:
-----------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1739/
-----------------------------------------------------------
(Updated 2011-09-15 22:18:43.052108)
Review request for jmhsieh.
Changes
-------
Addresss test failures, plus corrected comments
Summary
-------
If the source or sink throws an exception, close, reopen and retry it. This way the flow can continue after minor/recoverable errors.
This addresses bug FLUME-762.
https://issues.apache.org/jira/browse/FLUME-762
Diffs (updated)
-----
flume-core/src/main/java/com/cloudera/flume/core/connector/DirectDriver.java 0466394
flume-core/src/test/java/com/cloudera/flume/agent/TestFlumeNode.java 9d9e1a5
flume-core/src/test/java/com/cloudera/flume/agent/diskfailover/TestDiskFailoverDeco.java a7e2d78
flume-core/src/test/java/com/cloudera/flume/core/connector/TestDirectDriverExp.java PRE-CREATION
Diff: https://reviews.apache.org/r/1739/diff
Testing
-------
Thanks,
Prasad
> DirectDriver should reopen the source/sink if the append or next throws an exception
> ------------------------------------------------------------------------------------
>
> Key: FLUME-762
> URL: https://issues.apache.org/jira/browse/FLUME-762
> Project: Flume
> Issue Type: Improvement
> Components: Node
> Reporter: Prasad Mujumdar
> Assignee: Prasad Mujumdar
> Attachments: flume-762.patch
>
>
> Currently, if source or sink throws an exception, the driver bails out resulting the everything to shut down. Alternately, we can try to close and reopen the source or sink, and then attempt to continue. This way we can avoid blocking the flow. If the reopen fails, then it can bail out.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-762) DirectDriver should reopen the
source/sink if the append or next throws an exception
Posted by "jiraposter@reviews.apache.org (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13102837#comment-13102837 ]
jiraposter@reviews.apache.org commented on FLUME-762:
-----------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1739/
-----------------------------------------------------------
(Updated 2011-09-12 17:39:06.621797)
Review request for jmhsieh.
Changes
-------
fixed typos in comments, test names etc
added assert checks at the end of tests.
Summary
-------
If the source or sink throws an exception, close, reopen and retry it. This way the flow can continue after minor/recoverable errors.
This addresses bug FLUME-762.
https://issues.apache.org/jira/browse/FLUME-762
Diffs (updated)
-----
flume-core/src/main/java/com/cloudera/flume/core/connector/DirectDriver.java 0466394
flume-core/src/test/java/com/cloudera/flume/core/connector/TestDirectDriverExp.java PRE-CREATION
Diff: https://reviews.apache.org/r/1739/diff
Testing
-------
Thanks,
Prasad
> DirectDriver should reopen the source/sink if the append or next throws an exception
> ------------------------------------------------------------------------------------
>
> Key: FLUME-762
> URL: https://issues.apache.org/jira/browse/FLUME-762
> Project: Flume
> Issue Type: Improvement
> Components: Node
> Reporter: Prasad Mujumdar
> Assignee: Prasad Mujumdar
> Attachments: flume-762.patch
>
>
> Currently, if source or sink throws an exception, the driver bails out resulting the everything to shut down. Alternately, we can try to close and reopen the source or sink, and then attempt to continue. This way we can avoid blocking the flow. If the reopen fails, then it can bail out.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (FLUME-762) DirectDriver should reopen the
source/sink if the append or next throws an exception
Posted by "Jonathan Hsieh (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Hsieh updated FLUME-762:
---------------------------------
Attachment: flume-762.patch
Attached the patch I committed.
Thanks Prasad!
> DirectDriver should reopen the source/sink if the append or next throws an exception
> ------------------------------------------------------------------------------------
>
> Key: FLUME-762
> URL: https://issues.apache.org/jira/browse/FLUME-762
> Project: Flume
> Issue Type: Improvement
> Components: Node
> Reporter: Prasad Mujumdar
> Assignee: Prasad Mujumdar
> Attachments: flume-762.patch, flume-762.patch
>
>
> Currently, if source or sink throws an exception, the driver bails out resulting the everything to shut down. Alternately, we can try to close and reopen the source or sink, and then attempt to continue. This way we can avoid blocking the flow. If the reopen fails, then it can bail out.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (FLUME-762) DirectDriver should reopen the
source/sink if the append or next throws an exception
Posted by "Prasad Mujumdar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Prasad Mujumdar updated FLUME-762:
----------------------------------
Attachment: flume-762.patch
> DirectDriver should reopen the source/sink if the append or next throws an exception
> ------------------------------------------------------------------------------------
>
> Key: FLUME-762
> URL: https://issues.apache.org/jira/browse/FLUME-762
> Project: Flume
> Issue Type: Improvement
> Components: Node
> Reporter: Prasad Mujumdar
> Assignee: Prasad Mujumdar
> Attachments: flume-762.patch
>
>
> Currently, if source or sink throws an exception, the driver bails out resulting the everything to shut down. Alternately, we can try to close and reopen the source or sink, and then attempt to continue. This way we can avoid blocking the flow. If the reopen fails, then it can bail out.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (FLUME-762) DirectDriver should reopen the
source/sink if the append or next throws an exception
Posted by "Prasad Mujumdar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Prasad Mujumdar updated FLUME-762:
----------------------------------
Status: Patch Available (was: Open)
> DirectDriver should reopen the source/sink if the append or next throws an exception
> ------------------------------------------------------------------------------------
>
> Key: FLUME-762
> URL: https://issues.apache.org/jira/browse/FLUME-762
> Project: Flume
> Issue Type: Improvement
> Components: Node
> Reporter: Prasad Mujumdar
> Assignee: Prasad Mujumdar
> Attachments: flume-762.patch
>
>
> Currently, if source or sink throws an exception, the driver bails out resulting the everything to shut down. Alternately, we can try to close and reopen the source or sink, and then attempt to continue. This way we can avoid blocking the flow. If the reopen fails, then it can bail out.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-762) DirectDriver should reopen the
source/sink if the append or next throws an exception
Posted by "jiraposter@reviews.apache.org (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13100127#comment-13100127 ]
jiraposter@reviews.apache.org commented on FLUME-762:
-----------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1739/
-----------------------------------------------------------
(Updated 2011-09-08 07:25:11.363988)
Review request for jmhsieh.
Changes
-------
Handled the InterruptedException which results into driver abort. Updated the test to add that scenario and refactor the test classes.
Looks like my profiles got messed up which caused the tab/space formatting problems. sorry about that.
thanx
Prasad
Summary
-------
If the source or sink throws an exception, close, reopen and retry it. This way the flow can continue after minor/recoverable errors.
This addresses bug FLUME-762.
https://issues.apache.org/jira/browse/FLUME-762
Diffs (updated)
-----
flume-core/src/main/java/com/cloudera/flume/core/connector/DirectDriver.java 0466394
flume-core/src/test/java/com/cloudera/flume/core/connector/TestDirectDriverExp.java PRE-CREATION
Diff: https://reviews.apache.org/r/1739/diff
Testing
-------
Thanks,
Prasad
> DirectDriver should reopen the source/sink if the append or next throws an exception
> ------------------------------------------------------------------------------------
>
> Key: FLUME-762
> URL: https://issues.apache.org/jira/browse/FLUME-762
> Project: Flume
> Issue Type: Improvement
> Components: Node
> Reporter: Prasad Mujumdar
> Assignee: Prasad Mujumdar
> Attachments: flume-762.patch
>
>
> Currently, if source or sink throws an exception, the driver bails out resulting the everything to shut down. Alternately, we can try to close and reopen the source or sink, and then attempt to continue. This way we can avoid blocking the flow. If the reopen fails, then it can bail out.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira