You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flume.apache.org by "Brock Noland (Created) (JIRA)" <ji...@apache.org> on 2012/03/29 00:59:27 UTC
[jira] [Created] (FLUME-1075) HDFSEventSink begin is called when
transaction opened due to other error
HDFSEventSink begin is called when transaction opened due to other error
------------------------------------------------------------------------
Key: FLUME-1075
URL: https://issues.apache.org/jira/browse/FLUME-1075
Project: Flume
Issue Type: Bug
Components: Channel
Affects Versions: v1.1.0
Reporter: Brock Noland
If you HDFSEvenSink is unable to close a transaction, then the next time process is called, begin is called on that same transaction and another error is thrown.
{noformat}
2012-03-28 17:47:36,011 (SinkRunner-PollingRunner-DefaultSinkProcessor) [ERROR - org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:165)] Unhandled exception, logging and sleeping for 5000ms
java.lang.IllegalStateException: close() called when transaction is OPEN - you must either commit or rollback first
at com.google.common.base.Preconditions.checkState(Preconditions.java:172)
at org.apache.flume.channel.BasicTransactionSemantics.close(BasicTransactionSemantics.java:179)
at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:381)
at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:65)
at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:148)
at java.lang.Thread.run(Thread.java:662)
2012-03-28 17:47:41,013 (SinkRunner-PollingRunner-DefaultSinkProcessor) [ERROR - org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:374)] process failed
java.lang.IllegalStateException: begin() called when transaction is OPEN!
at com.google.common.base.Preconditions.checkState(Preconditions.java:145)
at org.apache.flume.channel.BasicTransactionSemantics.begin(BasicTransactionSemantics.java:131)
at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:323)
at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:65)
at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:148)
at java.lang.Thread.run(Thread.java:662)
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (FLUME-1075) HDFSEventSink begin is called when
transaction opened due to other error
Posted by "Prasad Mujumdar (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Prasad Mujumdar updated FLUME-1075:
-----------------------------------
Attachment: FLUME-1075.patch.2
> HDFSEventSink begin is called when transaction opened due to other error
> ------------------------------------------------------------------------
>
> Key: FLUME-1075
> URL: https://issues.apache.org/jira/browse/FLUME-1075
> Project: Flume
> Issue Type: Bug
> Components: Channel
> Affects Versions: v1.1.0
> Reporter: Brock Noland
> Assignee: Prasad Mujumdar
> Attachments: FLUME-1075.patch.2
>
>
> If you HDFSEvenSink is unable to close a transaction, then the next time process is called, begin is called on that same transaction and another error is thrown.
> {noformat}
> 2012-03-28 17:47:36,011 (SinkRunner-PollingRunner-DefaultSinkProcessor) [ERROR - org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:165)] Unhandled exception, logging and sleeping for 5000ms
> java.lang.IllegalStateException: close() called when transaction is OPEN - you must either commit or rollback first
> at com.google.common.base.Preconditions.checkState(Preconditions.java:172)
> at org.apache.flume.channel.BasicTransactionSemantics.close(BasicTransactionSemantics.java:179)
> at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:381)
> at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:65)
> at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:148)
> at java.lang.Thread.run(Thread.java:662)
> 2012-03-28 17:47:41,013 (SinkRunner-PollingRunner-DefaultSinkProcessor) [ERROR - org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:374)] process failed
> java.lang.IllegalStateException: begin() called when transaction is OPEN!
> at com.google.common.base.Preconditions.checkState(Preconditions.java:145)
> at org.apache.flume.channel.BasicTransactionSemantics.begin(BasicTransactionSemantics.java:131)
> at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:323)
> at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:65)
> at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:148)
> at java.lang.Thread.run(Thread.java:662)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-1075) HDFSEventSink begin is called when
transaction opened due to other error
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13242593#comment-13242593 ]
jiraposter@reviews.apache.org commented on FLUME-1075:
------------------------------------------------------
bq. On 2012-03-30 08:33:50, Mike Percy wrote:
bq. > Since we are catching Throwable here, we may want to do it for FLUME-1074 too. Shouldn't we do this in all places where Transaction.commit() is called, if for nothing else than consistency?
bq.
bq. Brock Noland wrote:
bq. +1
Perhaps we should have a cleanup method in transaction interface that rolls back the transaction if its still open and then close it. This way we can consolidate the transaction error handling in a single finally block instead of multiple catch blocks.
- Prasad
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4569/#review6570
-----------------------------------------------------------
On 2012-03-30 05:38:44, Prasad Mujumdar wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/4569/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2012-03-30 05:38:44)
bq.
bq.
bq. Review request for Flume, Arvind Prabhakar and Hari Shreedharan.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. Unhandled thorwable in HDFS sink can leave transaction open.
bq.
bq.
bq. This addresses bug FLUME-1075.
bq. https://issues.apache.org/jira/browse/FLUME-1075
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java eee9221
bq.
bq. Diff: https://reviews.apache.org/r/4569/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq. regression tests
bq.
bq.
bq. Thanks,
bq.
bq. Prasad
bq.
bq.
> HDFSEventSink begin is called when transaction opened due to other error
> ------------------------------------------------------------------------
>
> Key: FLUME-1075
> URL: https://issues.apache.org/jira/browse/FLUME-1075
> Project: Flume
> Issue Type: Bug
> Components: Channel
> Affects Versions: v1.1.0
> Reporter: Brock Noland
> Assignee: Prasad Mujumdar
>
> If you HDFSEvenSink is unable to close a transaction, then the next time process is called, begin is called on that same transaction and another error is thrown.
> {noformat}
> 2012-03-28 17:47:36,011 (SinkRunner-PollingRunner-DefaultSinkProcessor) [ERROR - org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:165)] Unhandled exception, logging and sleeping for 5000ms
> java.lang.IllegalStateException: close() called when transaction is OPEN - you must either commit or rollback first
> at com.google.common.base.Preconditions.checkState(Preconditions.java:172)
> at org.apache.flume.channel.BasicTransactionSemantics.close(BasicTransactionSemantics.java:179)
> at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:381)
> at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:65)
> at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:148)
> at java.lang.Thread.run(Thread.java:662)
> 2012-03-28 17:47:41,013 (SinkRunner-PollingRunner-DefaultSinkProcessor) [ERROR - org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:374)] process failed
> java.lang.IllegalStateException: begin() called when transaction is OPEN!
> at com.google.common.base.Preconditions.checkState(Preconditions.java:145)
> at org.apache.flume.channel.BasicTransactionSemantics.begin(BasicTransactionSemantics.java:131)
> at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:323)
> at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:65)
> at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:148)
> at java.lang.Thread.run(Thread.java:662)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-1075) HDFSEventSink begin is called when
transaction opened due to other error
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13242324#comment-13242324 ]
jiraposter@reviews.apache.org commented on FLUME-1075:
------------------------------------------------------
bq. On 2012-03-30 08:33:50, Mike Percy wrote:
bq. > Since we are catching Throwable here, we may want to do it for FLUME-1074 too. Shouldn't we do this in all places where Transaction.commit() is called, if for nothing else than consistency?
+1
- Brock
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4569/#review6570
-----------------------------------------------------------
On 2012-03-30 05:38:44, Prasad Mujumdar wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/4569/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2012-03-30 05:38:44)
bq.
bq.
bq. Review request for Flume, Arvind Prabhakar and Hari Shreedharan.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. Unhandled thorwable in HDFS sink can leave transaction open.
bq.
bq.
bq. This addresses bug FLUME-1075.
bq. https://issues.apache.org/jira/browse/FLUME-1075
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java eee9221
bq.
bq. Diff: https://reviews.apache.org/r/4569/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq. regression tests
bq.
bq.
bq. Thanks,
bq.
bq. Prasad
bq.
bq.
> HDFSEventSink begin is called when transaction opened due to other error
> ------------------------------------------------------------------------
>
> Key: FLUME-1075
> URL: https://issues.apache.org/jira/browse/FLUME-1075
> Project: Flume
> Issue Type: Bug
> Components: Channel
> Affects Versions: v1.1.0
> Reporter: Brock Noland
> Assignee: Prasad Mujumdar
>
> If you HDFSEvenSink is unable to close a transaction, then the next time process is called, begin is called on that same transaction and another error is thrown.
> {noformat}
> 2012-03-28 17:47:36,011 (SinkRunner-PollingRunner-DefaultSinkProcessor) [ERROR - org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:165)] Unhandled exception, logging and sleeping for 5000ms
> java.lang.IllegalStateException: close() called when transaction is OPEN - you must either commit or rollback first
> at com.google.common.base.Preconditions.checkState(Preconditions.java:172)
> at org.apache.flume.channel.BasicTransactionSemantics.close(BasicTransactionSemantics.java:179)
> at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:381)
> at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:65)
> at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:148)
> at java.lang.Thread.run(Thread.java:662)
> 2012-03-28 17:47:41,013 (SinkRunner-PollingRunner-DefaultSinkProcessor) [ERROR - org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:374)] process failed
> java.lang.IllegalStateException: begin() called when transaction is OPEN!
> at com.google.common.base.Preconditions.checkState(Preconditions.java:145)
> at org.apache.flume.channel.BasicTransactionSemantics.begin(BasicTransactionSemantics.java:131)
> at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:323)
> at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:65)
> at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:148)
> at java.lang.Thread.run(Thread.java:662)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-1075) HDFSEventSink begin is called when
transaction opened due to other error
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13242163#comment-13242163 ]
jiraposter@reviews.apache.org commented on FLUME-1075:
------------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4569/#review6570
-----------------------------------------------------------
Since we are catching Throwable here, we may want to do it for FLUME-1074 too. Shouldn't we do this in all places where Transaction.commit() is called, if for nothing else than consistency?
- Mike
On 2012-03-30 05:38:44, Prasad Mujumdar wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/4569/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2012-03-30 05:38:44)
bq.
bq.
bq. Review request for Flume, Arvind Prabhakar and Hari Shreedharan.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. Unhandled thorwable in HDFS sink can leave transaction open.
bq.
bq.
bq. This addresses bug FLUME-1075.
bq. https://issues.apache.org/jira/browse/FLUME-1075
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java eee9221
bq.
bq. Diff: https://reviews.apache.org/r/4569/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq. regression tests
bq.
bq.
bq. Thanks,
bq.
bq. Prasad
bq.
bq.
> HDFSEventSink begin is called when transaction opened due to other error
> ------------------------------------------------------------------------
>
> Key: FLUME-1075
> URL: https://issues.apache.org/jira/browse/FLUME-1075
> Project: Flume
> Issue Type: Bug
> Components: Channel
> Affects Versions: v1.1.0
> Reporter: Brock Noland
> Assignee: Prasad Mujumdar
>
> If you HDFSEvenSink is unable to close a transaction, then the next time process is called, begin is called on that same transaction and another error is thrown.
> {noformat}
> 2012-03-28 17:47:36,011 (SinkRunner-PollingRunner-DefaultSinkProcessor) [ERROR - org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:165)] Unhandled exception, logging and sleeping for 5000ms
> java.lang.IllegalStateException: close() called when transaction is OPEN - you must either commit or rollback first
> at com.google.common.base.Preconditions.checkState(Preconditions.java:172)
> at org.apache.flume.channel.BasicTransactionSemantics.close(BasicTransactionSemantics.java:179)
> at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:381)
> at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:65)
> at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:148)
> at java.lang.Thread.run(Thread.java:662)
> 2012-03-28 17:47:41,013 (SinkRunner-PollingRunner-DefaultSinkProcessor) [ERROR - org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:374)] process failed
> java.lang.IllegalStateException: begin() called when transaction is OPEN!
> at com.google.common.base.Preconditions.checkState(Preconditions.java:145)
> at org.apache.flume.channel.BasicTransactionSemantics.begin(BasicTransactionSemantics.java:131)
> at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:323)
> at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:65)
> at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:148)
> at java.lang.Thread.run(Thread.java:662)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-1075) HDFSEventSink begin is called when
transaction opened due to other error
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13242644#comment-13242644 ]
jiraposter@reviews.apache.org commented on FLUME-1075:
------------------------------------------------------
bq. On 2012-03-30 08:33:50, Mike Percy wrote:
bq. > Since we are catching Throwable here, we may want to do it for FLUME-1074 too. Shouldn't we do this in all places where Transaction.commit() is called, if for nothing else than consistency?
bq.
bq. Brock Noland wrote:
bq. +1
bq.
bq. Prasad Mujumdar wrote:
bq. Perhaps we should have a cleanup method in transaction interface that rolls back the transaction if its still open and then close it. This way we can consolidate the transaction error handling in a single finally block instead of multiple catch blocks.
bq.
bq. Brock Noland wrote:
bq. I mailed flume-dev on a similar issue "Channel/Transaction States"
I would like to note that I think catching Throwable in 1074 and or creating this idea of a cleanup shouldn't hold up either 1074 or 1075. These are both good incremental changes.
- Brock
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4569/#review6570
-----------------------------------------------------------
On 2012-03-30 05:38:44, Prasad Mujumdar wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/4569/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2012-03-30 05:38:44)
bq.
bq.
bq. Review request for Flume, Arvind Prabhakar and Hari Shreedharan.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. Unhandled thorwable in HDFS sink can leave transaction open.
bq.
bq.
bq. This addresses bug FLUME-1075.
bq. https://issues.apache.org/jira/browse/FLUME-1075
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java eee9221
bq.
bq. Diff: https://reviews.apache.org/r/4569/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq. regression tests
bq.
bq.
bq. Thanks,
bq.
bq. Prasad
bq.
bq.
> HDFSEventSink begin is called when transaction opened due to other error
> ------------------------------------------------------------------------
>
> Key: FLUME-1075
> URL: https://issues.apache.org/jira/browse/FLUME-1075
> Project: Flume
> Issue Type: Bug
> Components: Channel
> Affects Versions: v1.1.0
> Reporter: Brock Noland
> Assignee: Prasad Mujumdar
>
> If you HDFSEvenSink is unable to close a transaction, then the next time process is called, begin is called on that same transaction and another error is thrown.
> {noformat}
> 2012-03-28 17:47:36,011 (SinkRunner-PollingRunner-DefaultSinkProcessor) [ERROR - org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:165)] Unhandled exception, logging and sleeping for 5000ms
> java.lang.IllegalStateException: close() called when transaction is OPEN - you must either commit or rollback first
> at com.google.common.base.Preconditions.checkState(Preconditions.java:172)
> at org.apache.flume.channel.BasicTransactionSemantics.close(BasicTransactionSemantics.java:179)
> at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:381)
> at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:65)
> at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:148)
> at java.lang.Thread.run(Thread.java:662)
> 2012-03-28 17:47:41,013 (SinkRunner-PollingRunner-DefaultSinkProcessor) [ERROR - org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:374)] process failed
> java.lang.IllegalStateException: begin() called when transaction is OPEN!
> at com.google.common.base.Preconditions.checkState(Preconditions.java:145)
> at org.apache.flume.channel.BasicTransactionSemantics.begin(BasicTransactionSemantics.java:131)
> at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:323)
> at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:65)
> at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:148)
> at java.lang.Thread.run(Thread.java:662)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (FLUME-1075) HDFSEventSink begin is called when
transaction opened due to other error
Posted by "Arvind Prabhakar (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arvind Prabhakar updated FLUME-1075:
------------------------------------
Resolution: Fixed
Fix Version/s: v1.2.0
Status: Resolved (was: Patch Available)
Patch committed. Thanks Prasad!
> HDFSEventSink begin is called when transaction opened due to other error
> ------------------------------------------------------------------------
>
> Key: FLUME-1075
> URL: https://issues.apache.org/jira/browse/FLUME-1075
> Project: Flume
> Issue Type: Bug
> Components: Channel
> Affects Versions: v1.1.0
> Reporter: Brock Noland
> Assignee: Prasad Mujumdar
> Fix For: v1.2.0
>
> Attachments: FLUME-1075.patch.2
>
>
> If you HDFSEvenSink is unable to close a transaction, then the next time process is called, begin is called on that same transaction and another error is thrown.
> {noformat}
> 2012-03-28 17:47:36,011 (SinkRunner-PollingRunner-DefaultSinkProcessor) [ERROR - org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:165)] Unhandled exception, logging and sleeping for 5000ms
> java.lang.IllegalStateException: close() called when transaction is OPEN - you must either commit or rollback first
> at com.google.common.base.Preconditions.checkState(Preconditions.java:172)
> at org.apache.flume.channel.BasicTransactionSemantics.close(BasicTransactionSemantics.java:179)
> at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:381)
> at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:65)
> at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:148)
> at java.lang.Thread.run(Thread.java:662)
> 2012-03-28 17:47:41,013 (SinkRunner-PollingRunner-DefaultSinkProcessor) [ERROR - org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:374)] process failed
> java.lang.IllegalStateException: begin() called when transaction is OPEN!
> at com.google.common.base.Preconditions.checkState(Preconditions.java:145)
> at org.apache.flume.channel.BasicTransactionSemantics.begin(BasicTransactionSemantics.java:131)
> at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:323)
> at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:65)
> at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:148)
> at java.lang.Thread.run(Thread.java:662)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-1075) HDFSEventSink begin is called when
transaction opened due to other error
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13242691#comment-13242691 ]
jiraposter@reviews.apache.org commented on FLUME-1075:
------------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4569/#review6587
-----------------------------------------------------------
+1 from me, I agree that we should get this in right away. Thanks!
- Mike
On 2012-03-30 19:18:19, Prasad Mujumdar wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/4569/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2012-03-30 19:18:19)
bq.
bq.
bq. Review request for Flume, Arvind Prabhakar and Hari Shreedharan.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. Unhandled thorwable in HDFS sink can leave transaction open.
bq.
bq.
bq. This addresses bug FLUME-1075.
bq. https://issues.apache.org/jira/browse/FLUME-1075
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java eee9221
bq.
bq. Diff: https://reviews.apache.org/r/4569/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq. regression tests
bq.
bq.
bq. Thanks,
bq.
bq. Prasad
bq.
bq.
> HDFSEventSink begin is called when transaction opened due to other error
> ------------------------------------------------------------------------
>
> Key: FLUME-1075
> URL: https://issues.apache.org/jira/browse/FLUME-1075
> Project: Flume
> Issue Type: Bug
> Components: Channel
> Affects Versions: v1.1.0
> Reporter: Brock Noland
> Assignee: Prasad Mujumdar
>
> If you HDFSEvenSink is unable to close a transaction, then the next time process is called, begin is called on that same transaction and another error is thrown.
> {noformat}
> 2012-03-28 17:47:36,011 (SinkRunner-PollingRunner-DefaultSinkProcessor) [ERROR - org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:165)] Unhandled exception, logging and sleeping for 5000ms
> java.lang.IllegalStateException: close() called when transaction is OPEN - you must either commit or rollback first
> at com.google.common.base.Preconditions.checkState(Preconditions.java:172)
> at org.apache.flume.channel.BasicTransactionSemantics.close(BasicTransactionSemantics.java:179)
> at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:381)
> at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:65)
> at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:148)
> at java.lang.Thread.run(Thread.java:662)
> 2012-03-28 17:47:41,013 (SinkRunner-PollingRunner-DefaultSinkProcessor) [ERROR - org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:374)] process failed
> java.lang.IllegalStateException: begin() called when transaction is OPEN!
> at com.google.common.base.Preconditions.checkState(Preconditions.java:145)
> at org.apache.flume.channel.BasicTransactionSemantics.begin(BasicTransactionSemantics.java:131)
> at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:323)
> at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:65)
> at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:148)
> at java.lang.Thread.run(Thread.java:662)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-1075) HDFSEventSink begin is called when
transaction opened due to other error
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13242083#comment-13242083 ]
jiraposter@reviews.apache.org commented on FLUME-1075:
------------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4569/
-----------------------------------------------------------
Review request for Flume, Arvind Prabhakar and Hari Shreedharan.
Summary
-------
Unhandled thorwable in HDFS sink can leave transaction open.
This addresses bug FLUME-1075.
https://issues.apache.org/jira/browse/FLUME-1075
Diffs
-----
flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java eee9221
Diff: https://reviews.apache.org/r/4569/diff
Testing
-------
regression tests
Thanks,
Prasad
> HDFSEventSink begin is called when transaction opened due to other error
> ------------------------------------------------------------------------
>
> Key: FLUME-1075
> URL: https://issues.apache.org/jira/browse/FLUME-1075
> Project: Flume
> Issue Type: Bug
> Components: Channel
> Affects Versions: v1.1.0
> Reporter: Brock Noland
> Assignee: Prasad Mujumdar
>
> If you HDFSEvenSink is unable to close a transaction, then the next time process is called, begin is called on that same transaction and another error is thrown.
> {noformat}
> 2012-03-28 17:47:36,011 (SinkRunner-PollingRunner-DefaultSinkProcessor) [ERROR - org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:165)] Unhandled exception, logging and sleeping for 5000ms
> java.lang.IllegalStateException: close() called when transaction is OPEN - you must either commit or rollback first
> at com.google.common.base.Preconditions.checkState(Preconditions.java:172)
> at org.apache.flume.channel.BasicTransactionSemantics.close(BasicTransactionSemantics.java:179)
> at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:381)
> at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:65)
> at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:148)
> at java.lang.Thread.run(Thread.java:662)
> 2012-03-28 17:47:41,013 (SinkRunner-PollingRunner-DefaultSinkProcessor) [ERROR - org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:374)] process failed
> java.lang.IllegalStateException: begin() called when transaction is OPEN!
> at com.google.common.base.Preconditions.checkState(Preconditions.java:145)
> at org.apache.flume.channel.BasicTransactionSemantics.begin(BasicTransactionSemantics.java:131)
> at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:323)
> at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:65)
> at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:148)
> at java.lang.Thread.run(Thread.java:662)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-1075) HDFSEventSink begin is called when
transaction opened due to other error
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13242656#comment-13242656 ]
jiraposter@reviews.apache.org commented on FLUME-1075:
------------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4569/
-----------------------------------------------------------
(Updated 2012-03-30 19:18:19.682350)
Review request for Flume, Arvind Prabhakar and Hari Shreedharan.
Summary
-------
Unhandled thorwable in HDFS sink can leave transaction open.
This addresses bug FLUME-1075.
https://issues.apache.org/jira/browse/FLUME-1075
Diffs (updated)
-----
flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java eee9221
Diff: https://reviews.apache.org/r/4569/diff
Testing
-------
regression tests
Thanks,
Prasad
> HDFSEventSink begin is called when transaction opened due to other error
> ------------------------------------------------------------------------
>
> Key: FLUME-1075
> URL: https://issues.apache.org/jira/browse/FLUME-1075
> Project: Flume
> Issue Type: Bug
> Components: Channel
> Affects Versions: v1.1.0
> Reporter: Brock Noland
> Assignee: Prasad Mujumdar
>
> If you HDFSEvenSink is unable to close a transaction, then the next time process is called, begin is called on that same transaction and another error is thrown.
> {noformat}
> 2012-03-28 17:47:36,011 (SinkRunner-PollingRunner-DefaultSinkProcessor) [ERROR - org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:165)] Unhandled exception, logging and sleeping for 5000ms
> java.lang.IllegalStateException: close() called when transaction is OPEN - you must either commit or rollback first
> at com.google.common.base.Preconditions.checkState(Preconditions.java:172)
> at org.apache.flume.channel.BasicTransactionSemantics.close(BasicTransactionSemantics.java:179)
> at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:381)
> at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:65)
> at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:148)
> at java.lang.Thread.run(Thread.java:662)
> 2012-03-28 17:47:41,013 (SinkRunner-PollingRunner-DefaultSinkProcessor) [ERROR - org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:374)] process failed
> java.lang.IllegalStateException: begin() called when transaction is OPEN!
> at com.google.common.base.Preconditions.checkState(Preconditions.java:145)
> at org.apache.flume.channel.BasicTransactionSemantics.begin(BasicTransactionSemantics.java:131)
> at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:323)
> at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:65)
> at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:148)
> at java.lang.Thread.run(Thread.java:662)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-1075) HDFSEventSink begin is called when
transaction opened due to other error
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13242655#comment-13242655 ]
jiraposter@reviews.apache.org commented on FLUME-1075:
------------------------------------------------------
bq. On 2012-03-30 08:33:50, Mike Percy wrote:
bq. > Since we are catching Throwable here, we may want to do it for FLUME-1074 too. Shouldn't we do this in all places where Transaction.commit() is called, if for nothing else than consistency?
bq.
bq. Brock Noland wrote:
bq. +1
bq.
bq. Prasad Mujumdar wrote:
bq. Perhaps we should have a cleanup method in transaction interface that rolls back the transaction if its still open and then close it. This way we can consolidate the transaction error handling in a single finally block instead of multiple catch blocks.
bq.
bq. Brock Noland wrote:
bq. I mailed flume-dev on a similar issue "Channel/Transaction States"
bq.
bq. Brock Noland wrote:
bq. I would like to note that I think catching Throwable in 1074 and or creating this idea of a cleanup shouldn't hold up either 1074 or 1075. These are both good incremental changes.
@Brock, sorry I didn't see the thread earlier. I will log a new ticket once we have a consensus on the dev list thread.
@Mike, if we are going to change the close() logic, then we probably won't need to change all the exception handling code.
- Prasad
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4569/#review6570
-----------------------------------------------------------
On 2012-03-30 05:38:44, Prasad Mujumdar wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/4569/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2012-03-30 05:38:44)
bq.
bq.
bq. Review request for Flume, Arvind Prabhakar and Hari Shreedharan.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. Unhandled thorwable in HDFS sink can leave transaction open.
bq.
bq.
bq. This addresses bug FLUME-1075.
bq. https://issues.apache.org/jira/browse/FLUME-1075
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java eee9221
bq.
bq. Diff: https://reviews.apache.org/r/4569/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq. regression tests
bq.
bq.
bq. Thanks,
bq.
bq. Prasad
bq.
bq.
> HDFSEventSink begin is called when transaction opened due to other error
> ------------------------------------------------------------------------
>
> Key: FLUME-1075
> URL: https://issues.apache.org/jira/browse/FLUME-1075
> Project: Flume
> Issue Type: Bug
> Components: Channel
> Affects Versions: v1.1.0
> Reporter: Brock Noland
> Assignee: Prasad Mujumdar
>
> If you HDFSEvenSink is unable to close a transaction, then the next time process is called, begin is called on that same transaction and another error is thrown.
> {noformat}
> 2012-03-28 17:47:36,011 (SinkRunner-PollingRunner-DefaultSinkProcessor) [ERROR - org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:165)] Unhandled exception, logging and sleeping for 5000ms
> java.lang.IllegalStateException: close() called when transaction is OPEN - you must either commit or rollback first
> at com.google.common.base.Preconditions.checkState(Preconditions.java:172)
> at org.apache.flume.channel.BasicTransactionSemantics.close(BasicTransactionSemantics.java:179)
> at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:381)
> at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:65)
> at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:148)
> at java.lang.Thread.run(Thread.java:662)
> 2012-03-28 17:47:41,013 (SinkRunner-PollingRunner-DefaultSinkProcessor) [ERROR - org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:374)] process failed
> java.lang.IllegalStateException: begin() called when transaction is OPEN!
> at com.google.common.base.Preconditions.checkState(Preconditions.java:145)
> at org.apache.flume.channel.BasicTransactionSemantics.begin(BasicTransactionSemantics.java:131)
> at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:323)
> at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:65)
> at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:148)
> at java.lang.Thread.run(Thread.java:662)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (FLUME-1075) HDFSEventSink begin is called when
transaction opened due to other error
Posted by "Prasad Mujumdar (Assigned) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Prasad Mujumdar reassigned FLUME-1075:
--------------------------------------
Assignee: Prasad Mujumdar
> HDFSEventSink begin is called when transaction opened due to other error
> ------------------------------------------------------------------------
>
> Key: FLUME-1075
> URL: https://issues.apache.org/jira/browse/FLUME-1075
> Project: Flume
> Issue Type: Bug
> Components: Channel
> Affects Versions: v1.1.0
> Reporter: Brock Noland
> Assignee: Prasad Mujumdar
>
> If you HDFSEvenSink is unable to close a transaction, then the next time process is called, begin is called on that same transaction and another error is thrown.
> {noformat}
> 2012-03-28 17:47:36,011 (SinkRunner-PollingRunner-DefaultSinkProcessor) [ERROR - org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:165)] Unhandled exception, logging and sleeping for 5000ms
> java.lang.IllegalStateException: close() called when transaction is OPEN - you must either commit or rollback first
> at com.google.common.base.Preconditions.checkState(Preconditions.java:172)
> at org.apache.flume.channel.BasicTransactionSemantics.close(BasicTransactionSemantics.java:179)
> at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:381)
> at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:65)
> at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:148)
> at java.lang.Thread.run(Thread.java:662)
> 2012-03-28 17:47:41,013 (SinkRunner-PollingRunner-DefaultSinkProcessor) [ERROR - org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:374)] process failed
> java.lang.IllegalStateException: begin() called when transaction is OPEN!
> at com.google.common.base.Preconditions.checkState(Preconditions.java:145)
> at org.apache.flume.channel.BasicTransactionSemantics.begin(BasicTransactionSemantics.java:131)
> at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:323)
> at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:65)
> at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:148)
> at java.lang.Thread.run(Thread.java:662)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-1075) HDFSEventSink begin is called when
transaction opened due to other error
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13242733#comment-13242733 ]
jiraposter@reviews.apache.org commented on FLUME-1075:
------------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4569/#review6592
-----------------------------------------------------------
Ship it!
+1.
- Hari
On 2012-03-30 19:18:19, Prasad Mujumdar wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/4569/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2012-03-30 19:18:19)
bq.
bq.
bq. Review request for Flume, Arvind Prabhakar and Hari Shreedharan.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. Unhandled thorwable in HDFS sink can leave transaction open.
bq.
bq.
bq. This addresses bug FLUME-1075.
bq. https://issues.apache.org/jira/browse/FLUME-1075
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java eee9221
bq.
bq. Diff: https://reviews.apache.org/r/4569/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq. regression tests
bq.
bq.
bq. Thanks,
bq.
bq. Prasad
bq.
bq.
> HDFSEventSink begin is called when transaction opened due to other error
> ------------------------------------------------------------------------
>
> Key: FLUME-1075
> URL: https://issues.apache.org/jira/browse/FLUME-1075
> Project: Flume
> Issue Type: Bug
> Components: Channel
> Affects Versions: v1.1.0
> Reporter: Brock Noland
> Assignee: Prasad Mujumdar
> Attachments: FLUME-1075.patch.2
>
>
> If you HDFSEvenSink is unable to close a transaction, then the next time process is called, begin is called on that same transaction and another error is thrown.
> {noformat}
> 2012-03-28 17:47:36,011 (SinkRunner-PollingRunner-DefaultSinkProcessor) [ERROR - org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:165)] Unhandled exception, logging and sleeping for 5000ms
> java.lang.IllegalStateException: close() called when transaction is OPEN - you must either commit or rollback first
> at com.google.common.base.Preconditions.checkState(Preconditions.java:172)
> at org.apache.flume.channel.BasicTransactionSemantics.close(BasicTransactionSemantics.java:179)
> at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:381)
> at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:65)
> at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:148)
> at java.lang.Thread.run(Thread.java:662)
> 2012-03-28 17:47:41,013 (SinkRunner-PollingRunner-DefaultSinkProcessor) [ERROR - org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:374)] process failed
> java.lang.IllegalStateException: begin() called when transaction is OPEN!
> at com.google.common.base.Preconditions.checkState(Preconditions.java:145)
> at org.apache.flume.channel.BasicTransactionSemantics.begin(BasicTransactionSemantics.java:131)
> at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:323)
> at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:65)
> at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:148)
> at java.lang.Thread.run(Thread.java:662)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-1075) HDFSEventSink begin is called when
transaction opened due to other error
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13242325#comment-13242325 ]
jiraposter@reviews.apache.org commented on FLUME-1075:
------------------------------------------------------
bq. On 2012-03-30 05:50:30, Arvind Prabhakar wrote:
bq. > flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java, line 398
bq. > <https://reviews.apache.org/r/4569/diff/1/?file=97555#file97555line398>
bq. >
bq. > Since the code is now catching Throwable, please modify the throw logic to test and rethrow any Error types. For example:
bq. >
bq. > } catch (Throwable th) {
bq. > ...
bq. > if (th instance of Error) {
bq. > throw (Error) th;
bq. > } else {
bq. > throw new EventDeliveryException(th);
bq. > }
bq. > }
+1
- Brock
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4569/#review6567
-----------------------------------------------------------
On 2012-03-30 05:38:44, Prasad Mujumdar wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/4569/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2012-03-30 05:38:44)
bq.
bq.
bq. Review request for Flume, Arvind Prabhakar and Hari Shreedharan.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. Unhandled thorwable in HDFS sink can leave transaction open.
bq.
bq.
bq. This addresses bug FLUME-1075.
bq. https://issues.apache.org/jira/browse/FLUME-1075
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java eee9221
bq.
bq. Diff: https://reviews.apache.org/r/4569/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq. regression tests
bq.
bq.
bq. Thanks,
bq.
bq. Prasad
bq.
bq.
> HDFSEventSink begin is called when transaction opened due to other error
> ------------------------------------------------------------------------
>
> Key: FLUME-1075
> URL: https://issues.apache.org/jira/browse/FLUME-1075
> Project: Flume
> Issue Type: Bug
> Components: Channel
> Affects Versions: v1.1.0
> Reporter: Brock Noland
> Assignee: Prasad Mujumdar
>
> If you HDFSEvenSink is unable to close a transaction, then the next time process is called, begin is called on that same transaction and another error is thrown.
> {noformat}
> 2012-03-28 17:47:36,011 (SinkRunner-PollingRunner-DefaultSinkProcessor) [ERROR - org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:165)] Unhandled exception, logging and sleeping for 5000ms
> java.lang.IllegalStateException: close() called when transaction is OPEN - you must either commit or rollback first
> at com.google.common.base.Preconditions.checkState(Preconditions.java:172)
> at org.apache.flume.channel.BasicTransactionSemantics.close(BasicTransactionSemantics.java:179)
> at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:381)
> at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:65)
> at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:148)
> at java.lang.Thread.run(Thread.java:662)
> 2012-03-28 17:47:41,013 (SinkRunner-PollingRunner-DefaultSinkProcessor) [ERROR - org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:374)] process failed
> java.lang.IllegalStateException: begin() called when transaction is OPEN!
> at com.google.common.base.Preconditions.checkState(Preconditions.java:145)
> at org.apache.flume.channel.BasicTransactionSemantics.begin(BasicTransactionSemantics.java:131)
> at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:323)
> at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:65)
> at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:148)
> at java.lang.Thread.run(Thread.java:662)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-1075) HDFSEventSink begin is called when
transaction opened due to other error
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13242088#comment-13242088 ]
jiraposter@reviews.apache.org commented on FLUME-1075:
------------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4569/#review6567
-----------------------------------------------------------
Thanks for the patch Prasad. I have one suggestion noted below for your consideration.
flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java
<https://reviews.apache.org/r/4569/#comment14248>
Since the code is now catching Throwable, please modify the throw logic to test and rethrow any Error types. For example:
} catch (Throwable th) {
...
if (th instance of Error) {
throw (Error) th;
} else {
throw new EventDeliveryException(th);
}
}
- Arvind
On 2012-03-30 05:38:44, Prasad Mujumdar wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/4569/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2012-03-30 05:38:44)
bq.
bq.
bq. Review request for Flume, Arvind Prabhakar and Hari Shreedharan.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. Unhandled thorwable in HDFS sink can leave transaction open.
bq.
bq.
bq. This addresses bug FLUME-1075.
bq. https://issues.apache.org/jira/browse/FLUME-1075
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java eee9221
bq.
bq. Diff: https://reviews.apache.org/r/4569/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq. regression tests
bq.
bq.
bq. Thanks,
bq.
bq. Prasad
bq.
bq.
> HDFSEventSink begin is called when transaction opened due to other error
> ------------------------------------------------------------------------
>
> Key: FLUME-1075
> URL: https://issues.apache.org/jira/browse/FLUME-1075
> Project: Flume
> Issue Type: Bug
> Components: Channel
> Affects Versions: v1.1.0
> Reporter: Brock Noland
> Assignee: Prasad Mujumdar
>
> If you HDFSEvenSink is unable to close a transaction, then the next time process is called, begin is called on that same transaction and another error is thrown.
> {noformat}
> 2012-03-28 17:47:36,011 (SinkRunner-PollingRunner-DefaultSinkProcessor) [ERROR - org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:165)] Unhandled exception, logging and sleeping for 5000ms
> java.lang.IllegalStateException: close() called when transaction is OPEN - you must either commit or rollback first
> at com.google.common.base.Preconditions.checkState(Preconditions.java:172)
> at org.apache.flume.channel.BasicTransactionSemantics.close(BasicTransactionSemantics.java:179)
> at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:381)
> at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:65)
> at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:148)
> at java.lang.Thread.run(Thread.java:662)
> 2012-03-28 17:47:41,013 (SinkRunner-PollingRunner-DefaultSinkProcessor) [ERROR - org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:374)] process failed
> java.lang.IllegalStateException: begin() called when transaction is OPEN!
> at com.google.common.base.Preconditions.checkState(Preconditions.java:145)
> at org.apache.flume.channel.BasicTransactionSemantics.begin(BasicTransactionSemantics.java:131)
> at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:323)
> at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:65)
> at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:148)
> at java.lang.Thread.run(Thread.java:662)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-1075) HDFSEventSink begin is called when
transaction opened due to other error
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13243914#comment-13243914 ]
Hudson commented on FLUME-1075:
-------------------------------
Integrated in flume-trunk #154 (See [https://builds.apache.org/job/flume-trunk/154/])
FLUME-1075. HDFSEventSink begin is called when transaction opened due to other error.
(Prasad Mujumdar via Arvind Prabhakar) (Revision 1308171)
Result = SUCCESS
arvind : http://svn.apache.org/viewvc/?view=rev&rev=1308171
Files :
* /incubator/flume/trunk/flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java
> HDFSEventSink begin is called when transaction opened due to other error
> ------------------------------------------------------------------------
>
> Key: FLUME-1075
> URL: https://issues.apache.org/jira/browse/FLUME-1075
> Project: Flume
> Issue Type: Bug
> Components: Channel
> Affects Versions: v1.1.0
> Reporter: Brock Noland
> Assignee: Prasad Mujumdar
> Fix For: v1.2.0
>
> Attachments: FLUME-1075.patch.2
>
>
> If you HDFSEvenSink is unable to close a transaction, then the next time process is called, begin is called on that same transaction and another error is thrown.
> {noformat}
> 2012-03-28 17:47:36,011 (SinkRunner-PollingRunner-DefaultSinkProcessor) [ERROR - org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:165)] Unhandled exception, logging and sleeping for 5000ms
> java.lang.IllegalStateException: close() called when transaction is OPEN - you must either commit or rollback first
> at com.google.common.base.Preconditions.checkState(Preconditions.java:172)
> at org.apache.flume.channel.BasicTransactionSemantics.close(BasicTransactionSemantics.java:179)
> at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:381)
> at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:65)
> at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:148)
> at java.lang.Thread.run(Thread.java:662)
> 2012-03-28 17:47:41,013 (SinkRunner-PollingRunner-DefaultSinkProcessor) [ERROR - org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:374)] process failed
> java.lang.IllegalStateException: begin() called when transaction is OPEN!
> at com.google.common.base.Preconditions.checkState(Preconditions.java:145)
> at org.apache.flume.channel.BasicTransactionSemantics.begin(BasicTransactionSemantics.java:131)
> at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:323)
> at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:65)
> at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:148)
> at java.lang.Thread.run(Thread.java:662)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-1075) HDFSEventSink begin is called when
transaction opened due to other error
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13242595#comment-13242595 ]
jiraposter@reviews.apache.org commented on FLUME-1075:
------------------------------------------------------
bq. On 2012-03-30 08:33:50, Mike Percy wrote:
bq. > Since we are catching Throwable here, we may want to do it for FLUME-1074 too. Shouldn't we do this in all places where Transaction.commit() is called, if for nothing else than consistency?
bq.
bq. Brock Noland wrote:
bq. +1
bq.
bq. Prasad Mujumdar wrote:
bq. Perhaps we should have a cleanup method in transaction interface that rolls back the transaction if its still open and then close it. This way we can consolidate the transaction error handling in a single finally block instead of multiple catch blocks.
I mailed flume-dev on a similar issue "Channel/Transaction States"
- Brock
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4569/#review6570
-----------------------------------------------------------
On 2012-03-30 05:38:44, Prasad Mujumdar wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/4569/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2012-03-30 05:38:44)
bq.
bq.
bq. Review request for Flume, Arvind Prabhakar and Hari Shreedharan.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. Unhandled thorwable in HDFS sink can leave transaction open.
bq.
bq.
bq. This addresses bug FLUME-1075.
bq. https://issues.apache.org/jira/browse/FLUME-1075
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java eee9221
bq.
bq. Diff: https://reviews.apache.org/r/4569/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq. regression tests
bq.
bq.
bq. Thanks,
bq.
bq. Prasad
bq.
bq.
> HDFSEventSink begin is called when transaction opened due to other error
> ------------------------------------------------------------------------
>
> Key: FLUME-1075
> URL: https://issues.apache.org/jira/browse/FLUME-1075
> Project: Flume
> Issue Type: Bug
> Components: Channel
> Affects Versions: v1.1.0
> Reporter: Brock Noland
> Assignee: Prasad Mujumdar
>
> If you HDFSEvenSink is unable to close a transaction, then the next time process is called, begin is called on that same transaction and another error is thrown.
> {noformat}
> 2012-03-28 17:47:36,011 (SinkRunner-PollingRunner-DefaultSinkProcessor) [ERROR - org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:165)] Unhandled exception, logging and sleeping for 5000ms
> java.lang.IllegalStateException: close() called when transaction is OPEN - you must either commit or rollback first
> at com.google.common.base.Preconditions.checkState(Preconditions.java:172)
> at org.apache.flume.channel.BasicTransactionSemantics.close(BasicTransactionSemantics.java:179)
> at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:381)
> at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:65)
> at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:148)
> at java.lang.Thread.run(Thread.java:662)
> 2012-03-28 17:47:41,013 (SinkRunner-PollingRunner-DefaultSinkProcessor) [ERROR - org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:374)] process failed
> java.lang.IllegalStateException: begin() called when transaction is OPEN!
> at com.google.common.base.Preconditions.checkState(Preconditions.java:145)
> at org.apache.flume.channel.BasicTransactionSemantics.begin(BasicTransactionSemantics.java:131)
> at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:323)
> at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:65)
> at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:148)
> at java.lang.Thread.run(Thread.java:662)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (FLUME-1075) HDFSEventSink begin is called when
transaction opened due to other error
Posted by "Prasad Mujumdar (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Prasad Mujumdar updated FLUME-1075:
-----------------------------------
Status: Patch Available (was: Open)
> HDFSEventSink begin is called when transaction opened due to other error
> ------------------------------------------------------------------------
>
> Key: FLUME-1075
> URL: https://issues.apache.org/jira/browse/FLUME-1075
> Project: Flume
> Issue Type: Bug
> Components: Channel
> Affects Versions: v1.1.0
> Reporter: Brock Noland
> Assignee: Prasad Mujumdar
> Attachments: FLUME-1075.patch.2
>
>
> If you HDFSEvenSink is unable to close a transaction, then the next time process is called, begin is called on that same transaction and another error is thrown.
> {noformat}
> 2012-03-28 17:47:36,011 (SinkRunner-PollingRunner-DefaultSinkProcessor) [ERROR - org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:165)] Unhandled exception, logging and sleeping for 5000ms
> java.lang.IllegalStateException: close() called when transaction is OPEN - you must either commit or rollback first
> at com.google.common.base.Preconditions.checkState(Preconditions.java:172)
> at org.apache.flume.channel.BasicTransactionSemantics.close(BasicTransactionSemantics.java:179)
> at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:381)
> at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:65)
> at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:148)
> at java.lang.Thread.run(Thread.java:662)
> 2012-03-28 17:47:41,013 (SinkRunner-PollingRunner-DefaultSinkProcessor) [ERROR - org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:374)] process failed
> java.lang.IllegalStateException: begin() called when transaction is OPEN!
> at com.google.common.base.Preconditions.checkState(Preconditions.java:145)
> at org.apache.flume.channel.BasicTransactionSemantics.begin(BasicTransactionSemantics.java:131)
> at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:323)
> at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:65)
> at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:148)
> at java.lang.Thread.run(Thread.java:662)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira