You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flume.apache.org by "Hari Shreedharan (JIRA)" <ji...@apache.org> on 2012/06/04 11:08:23 UTC
[jira] [Created] (FLUME-1255) HbaseSink should catch exceptions
thrown by channel.take and rollback or commit.
Hari Shreedharan created FLUME-1255:
---------------------------------------
Summary: HbaseSink should catch exceptions thrown by channel.take and rollback or commit.
Key: FLUME-1255
URL: https://issues.apache.org/jira/browse/FLUME-1255
Project: Flume
Issue Type: Bug
Reporter: Hari Shreedharan
Channels like Memory channel throw exception when reaching txn capacity. If sink's batchsize is more than txn capacity the take function throws exception.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-1255) Channels should not throw
exceptions in channel.take if it has reached txn capacity
Posted by "Juhani Connolly (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13289086#comment-13289086 ]
Juhani Connolly commented on FLUME-1255:
----------------------------------------
Fixing it to be more user friendly would be good, but what would you suggest? Returning null would give the false belief that there are no more events available, likely affecting the behavior of some sinks, and also going against the spirit of the interface. The current way MemoryChannel is set up is to preserve some strict invariants so that rollbacks should always work.
It seems more like a configuration error than a bug to me, though it's certainly not easy for a user to understand.
> Channels should not throw exceptions in channel.take if it has reached txn capacity
> -----------------------------------------------------------------------------------
>
> Key: FLUME-1255
> URL: https://issues.apache.org/jira/browse/FLUME-1255
> Project: Flume
> Issue Type: Bug
> Reporter: Hari Shreedharan
>
> Channels like Memory channel throw exception when reaching txn capacity. If sink's batchsize is more than txn capacity the take function throws exception.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-1255) HbaseSink should catch exceptions
thrown by channel.take and rollback or commit.
Posted by "Hari Shreedharan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13288685#comment-13288685 ]
Hari Shreedharan commented on FLUME-1255:
-----------------------------------------
Seems like this is likely to affect HDFS sink too.
> HbaseSink should catch exceptions thrown by channel.take and rollback or commit.
> --------------------------------------------------------------------------------
>
> Key: FLUME-1255
> URL: https://issues.apache.org/jira/browse/FLUME-1255
> Project: Flume
> Issue Type: Bug
> Reporter: Hari Shreedharan
>
> Channels like Memory channel throw exception when reaching txn capacity. If sink's batchsize is more than txn capacity the take function throws exception.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (FLUME-1255) Channels should not throw
exceptions in channel.take if it has reached txn capacity
Posted by "Hari Shreedharan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13289157#comment-13289157 ]
Hari Shreedharan edited comment on FLUME-1255 at 6/5/12 5:18 AM:
-----------------------------------------------------------------
Hi Juhani,
Yes, it is more of a config error. The issue here is that because take throws exceptions, the HbaseSink ends up throwing an exception and break out of process(), *without* committing or rolling back the transaction(and thus closing it). Due to this, the next time txn.begin() is called, an exception is thrown because the txn was never closed. Even for sinks like HDFSEventSink, which catches the exception, it is problematic because it rolls back the txn(This is probably the right thing to do, since a channel throwing an exception indicates an error condition, and the sink does not need to care what error it was - and reaching txn capacity is certainly not an error - so we should not throw an exception in that case) - which means the sink will never actually commit a transaction. This is primarily a correctness issue, not a user-friendliness one.
There is no reason we should fail with an exception if the transaction capacity of the channel is smaller than the batch size of the sink - since such a configuration is absolutely valid.
was (Author: hshreedharan):
Hi Juhani,
Yes, it is more of a config error. The issue here is that because take throws exceptions, our sinks end up throwing an exception and break out of process(), *without* committing or rolling back the transaction(and thus closing it). Due to this, the next time txn.begin() is called, an exception is thrown because the txn was never closed. Even for sinks like HDFSEventSink, which catches the exception, it is problematic because it rolls back the txn(This is probably the right thing to do, since a channel throwing an exception indicates an error condition, and the sink does not need to care what error it was - and reaching txn capacity is certainly not an error - so we should not throw an exception in that case) - which means the sink will never actually commit a transaction. This is primarily a correctness issue, not a user-friendliness one.
There is no reason we should fail with an exception if the transaction capacity of the channel is smaller than the batch size of the sink - since such a configuration is absolutely valid.
> Channels should not throw exceptions in channel.take if it has reached txn capacity
> -----------------------------------------------------------------------------------
>
> Key: FLUME-1255
> URL: https://issues.apache.org/jira/browse/FLUME-1255
> Project: Flume
> Issue Type: Bug
> Reporter: Hari Shreedharan
>
> Channels like Memory channel throw exception when reaching txn capacity. If sink's batchsize is more than txn capacity the take function throws exception.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (FLUME-1255) HbaseSink should catch
exceptions thrown by channel.take and rollback or commit.
Posted by "Hari Shreedharan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13288685#comment-13288685 ]
Hari Shreedharan edited comment on FLUME-1255 at 6/4/12 5:09 PM:
-----------------------------------------------------------------
Seems like this is likely to affect HDFS sink too. Maybe we should fix the channel's behavior?
was (Author: hshreedharan):
Seems like this is likely to affect HDFS sink too.
> HbaseSink should catch exceptions thrown by channel.take and rollback or commit.
> --------------------------------------------------------------------------------
>
> Key: FLUME-1255
> URL: https://issues.apache.org/jira/browse/FLUME-1255
> Project: Flume
> Issue Type: Bug
> Reporter: Hari Shreedharan
>
> Channels like Memory channel throw exception when reaching txn capacity. If sink's batchsize is more than txn capacity the take function throws exception.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (FLUME-1255) Channels should not throw exceptions
in channel.take if it has reached txn capacity
Posted by "Hari Shreedharan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hari Shreedharan updated FLUME-1255:
------------------------------------
Summary: Channels should not throw exceptions in channel.take if it has reached txn capacity (was: HbaseSink should catch exceptions thrown by channel.take and rollback or commit.)
> Channels should not throw exceptions in channel.take if it has reached txn capacity
> -----------------------------------------------------------------------------------
>
> Key: FLUME-1255
> URL: https://issues.apache.org/jira/browse/FLUME-1255
> Project: Flume
> Issue Type: Bug
> Reporter: Hari Shreedharan
>
> Channels like Memory channel throw exception when reaching txn capacity. If sink's batchsize is more than txn capacity the take function throws exception.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-1255) Channels should not throw
exceptions in channel.take if it has reached txn capacity
Posted by "Hari Shreedharan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13289157#comment-13289157 ]
Hari Shreedharan commented on FLUME-1255:
-----------------------------------------
Hi Juhani,
Yes, it is more of a config error. The issue here is that because take throws exceptions, our sinks end up throwing an exception and break out of process(), *without* committing or rolling back the transaction(and thus closing it). Due to this, the next time txn.begin() is called, an exception is thrown because the txn was never closed. Even for sinks like HDFSEventSink, which catches the exception, it is problematic because it rolls back the txn(This is probably the right thing to do, since a channel throwing an exception indicates an error condition, and the sink does not need to care what error it was - and reaching txn capacity is certainly not an error - so we should not throw an exception in that case) - which means the sink will never actually commit a transaction. This is primarily a correctness issue, not a user-friendliness one.
There is no reason we should fail with an exception if the transaction capacity of the channel is smaller than the batch size of the sink - since such a configuration is absolutely valid.
> Channels should not throw exceptions in channel.take if it has reached txn capacity
> -----------------------------------------------------------------------------------
>
> Key: FLUME-1255
> URL: https://issues.apache.org/jira/browse/FLUME-1255
> Project: Flume
> Issue Type: Bug
> Reporter: Hari Shreedharan
>
> Channels like Memory channel throw exception when reaching txn capacity. If sink's batchsize is more than txn capacity the take function throws exception.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-1255) Channels should not throw
exceptions in channel.take if it has reached txn capacity
Posted by "Juhani Connolly (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13289180#comment-13289180 ]
Juhani Connolly commented on FLUME-1255:
----------------------------------------
I agree that the current failing with an exception isn't ideal, but I think this is a limitation of the Channel interface, not necessarily one of the individual component implementations. We can only communicate one of three things:
- "Here's some data we got for you"
- "Sorry no more data"
- "Something's wrong"
The last "something's wrong" is probably too vague. When the transaction is full, saying "no more data" is plain wrong.
The only "correct" solution I can see is adding a mechanism to allow more specific communication between the components, in this case TransactionLimitException or something that tells a source/sink that everything is ok, but they can't handle more until the current transaction is committed. Do you(or anyone else) have any other ideas how to get around this?
> Channels should not throw exceptions in channel.take if it has reached txn capacity
> -----------------------------------------------------------------------------------
>
> Key: FLUME-1255
> URL: https://issues.apache.org/jira/browse/FLUME-1255
> Project: Flume
> Issue Type: Bug
> Reporter: Hari Shreedharan
>
> Channels like Memory channel throw exception when reaching txn capacity. If sink's batchsize is more than txn capacity the take function throws exception.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira