You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flume.apache.org by "Brock Noland (Created) (JIRA)" <ji...@apache.org> on 2012/02/26 07:47:49 UTC
[jira] [Created] (FLUME-998) JDBC Channel take does not block
JDBC Channel take does not block
--------------------------------
Key: FLUME-998
URL: https://issues.apache.org/jira/browse/FLUME-998
Project: Flume
Issue Type: Improvement
Components: Channel
Affects Versions: v1.0.0
Reporter: Brock Noland
Priority: Critical
Fix For: v1.1.0
Memory Channel takes block for a few seconds so that if there are no events at present, the thread consuming events does not spin in a tight loop. JDBC Channel does not block and returns null immediately, as such the thread gettings events spins hard.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-998) JDBC Channel take does not block
Posted by "Brock Noland (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13216987#comment-13216987 ]
Brock Noland commented on FLUME-998:
------------------------------------
@Juhani and Peter, thanks guys, I started a discussion on standard behavior on flume-dev "Source/Channel/Sink Behavior Standardization". This is good information for that discussion.
@Peter, I would love to see the changes you have made.
Brock
> JDBC Channel take does not block
> --------------------------------
>
> Key: FLUME-998
> URL: https://issues.apache.org/jira/browse/FLUME-998
> Project: Flume
> Issue Type: Improvement
> Components: Channel
> Affects Versions: v1.0.0
> Reporter: Brock Noland
> Priority: Critical
> Fix For: v1.1.0
>
>
> Memory Channel takes block for a few seconds if there are no events to take. This so the thread consuming events does not spin in a tight loop. JDBC Channel does not block and returns null immediately, as such the thread gettings events spins hard.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-998) JDBC Channel take does not block
Posted by "Juhani Connolly (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13216972#comment-13216972 ]
Juhani Connolly commented on FLUME-998:
---------------------------------------
Okay... Having a look at the HDFS sink,it appears that that will just return ready regardless of there being data or not.
AvroSink on the other hand will return BACKOFF if there is no new data.
Perhaps my interpretation of expected behavior is mistaken? Unfortunately right now, the Sink interface is undocumented so the intentions for ready and backoff are not entirely clear.
I personally feel that having sinks return backoff when there is no data makes sense. However having all channels block for a while on take would be a reasonable backup to have in case of poorly behaving sinks.
> JDBC Channel take does not block
> --------------------------------
>
> Key: FLUME-998
> URL: https://issues.apache.org/jira/browse/FLUME-998
> Project: Flume
> Issue Type: Improvement
> Components: Channel
> Affects Versions: v1.0.0
> Reporter: Brock Noland
> Priority: Critical
> Fix For: v1.1.0
>
>
> Memory Channel takes block for a few seconds if there are no events to take. This so the thread consuming events does not spin in a tight loop. JDBC Channel does not block and returns null immediately, as such the thread gettings events spins hard.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-998) JDBC Channel take does not block
Posted by "Juhani Connolly (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13216970#comment-13216970 ]
Juhani Connolly commented on FLUME-998:
---------------------------------------
If there is nothing there to take, the sink should be sending backoff requests so as not to send non-stop requests. If the thread fetching events actually "spins hard" there is something wrong at that level which needs to be fixed.
> JDBC Channel take does not block
> --------------------------------
>
> Key: FLUME-998
> URL: https://issues.apache.org/jira/browse/FLUME-998
> Project: Flume
> Issue Type: Improvement
> Components: Channel
> Affects Versions: v1.0.0
> Reporter: Brock Noland
> Priority: Critical
> Fix For: v1.1.0
>
>
> Memory Channel takes block for a few seconds if there are no events to take. This so the thread consuming events does not spin in a tight loop. JDBC Channel does not block and returns null immediately, as such the thread gettings events spins hard.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-998) JDBC Channel take does not block
Posted by "Peter Newcomb (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13216986#comment-13216986 ]
Peter Newcomb commented on FLUME-998:
-------------------------------------
I agree, and feel that the bug is with HDFSEventSink and any other sink that fails to return BACKOFF if process()'s first call to take() returns null. I patched our version of HDFSEventSink to make it do so, but have so far failed to open a JIRA or post a patch for review simply due to a lack of time, and that I'd need to factor apart the many other patches I've made to HDFSEventSink as well.
I'll be doing all of that as soon as I find a round tuit. :-) If anyone's interested in looking at the full set of changes I've made before I get a chance to do so, I'd be happy to share the omnibus patch. I think that all of the changes I've made are just for correctness, performance, and reliability, nothing really specific for our application or that actually changes its behavior. I did recently incorporate Brock's changes from FLUME-985 (thanks!), so it's at least up-to-date with respect to that patch.
-peter
> JDBC Channel take does not block
> --------------------------------
>
> Key: FLUME-998
> URL: https://issues.apache.org/jira/browse/FLUME-998
> Project: Flume
> Issue Type: Improvement
> Components: Channel
> Affects Versions: v1.0.0
> Reporter: Brock Noland
> Priority: Critical
> Fix For: v1.1.0
>
>
> Memory Channel takes block for a few seconds if there are no events to take. This so the thread consuming events does not spin in a tight loop. JDBC Channel does not block and returns null immediately, as such the thread gettings events spins hard.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (FLUME-998) JDBC Channel take does not block
Posted by "Brock Noland (Resolved) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-998?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Brock Noland resolved FLUME-998.
--------------------------------
Resolution: Won't Fix
Fix Version/s: (was: v1.1.0)
It has been decided the channels should not block but that the HDFSEventSink should return backoff when the channel has no events. Another JIRA will be submitted to fix this.
> JDBC Channel take does not block
> --------------------------------
>
> Key: FLUME-998
> URL: https://issues.apache.org/jira/browse/FLUME-998
> Project: Flume
> Issue Type: Improvement
> Components: Channel
> Affects Versions: v1.0.0
> Reporter: Brock Noland
> Priority: Critical
>
> Memory Channel takes block for a few seconds if there are no events to take. This so the thread consuming events does not spin in a tight loop. JDBC Channel does not block and returns null immediately, as such the thread gettings events spins hard.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (FLUME-998) JDBC Channel take does not block
Posted by "Brock Noland (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-998?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Brock Noland updated FLUME-998:
-------------------------------
Description: Memory Channel takes block for a few seconds if there are no events to take. This so the thread consuming events does not spin in a tight loop. JDBC Channel does not block and returns null immediately, as such the thread gettings events spins hard. (was: Memory Channel takes block for a few seconds so that if there are no events at present, the thread consuming events does not spin in a tight loop. JDBC Channel does not block and returns null immediately, as such the thread gettings events spins hard.)
> JDBC Channel take does not block
> --------------------------------
>
> Key: FLUME-998
> URL: https://issues.apache.org/jira/browse/FLUME-998
> Project: Flume
> Issue Type: Improvement
> Components: Channel
> Affects Versions: v1.0.0
> Reporter: Brock Noland
> Priority: Critical
> Fix For: v1.1.0
>
>
> Memory Channel takes block for a few seconds if there are no events to take. This so the thread consuming events does not spin in a tight loop. JDBC Channel does not block and returns null immediately, as such the thread gettings events spins hard.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-998) JDBC Channel take does not block
Posted by "Arvind Prabhakar (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13217822#comment-13217822 ]
Arvind Prabhakar commented on FLUME-998:
----------------------------------------
As I mentioned in the other thread, it is by design that the channels do not block. This should be marked as not a bug.
> JDBC Channel take does not block
> --------------------------------
>
> Key: FLUME-998
> URL: https://issues.apache.org/jira/browse/FLUME-998
> Project: Flume
> Issue Type: Improvement
> Components: Channel
> Affects Versions: v1.0.0
> Reporter: Brock Noland
> Priority: Critical
> Fix For: v1.1.0
>
>
> Memory Channel takes block for a few seconds if there are no events to take. This so the thread consuming events does not spin in a tight loop. JDBC Channel does not block and returns null immediately, as such the thread gettings events spins hard.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira