You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flume.apache.org by "Seetharam Venkatesh (JIRA)" <ji...@apache.org> on 2012/08/07 20:06:09 UTC
[jira] [Commented] (FLUME-1045) Proposal to support disk based
spooling
[ https://issues.apache.org/jira/browse/FLUME-1045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13430489#comment-13430489 ]
Seetharam Venkatesh commented on FLUME-1045:
--------------------------------------------
IMHO, what Sharad outlines is a very reasonable approach and should not be hard to tie things together.
bq. I'm just worried here that current channel interface do not expose number of stored entries. So questions like "is memory channel full" or "do file channel contains less than memory channel capacity might not work
Jarcec, why do you consider this a problem? Do you consider this leaky? Metrics on top of the channel is necessary IMO and is not leaky.
> Proposal to support disk based spooling
> ---------------------------------------
>
> Key: FLUME-1045
> URL: https://issues.apache.org/jira/browse/FLUME-1045
> Project: Flume
> Issue Type: New Feature
> Affects Versions: v1.0.0
> Reporter: Inder SIngh
> Priority: Minor
> Labels: patch
> Attachments: FLUME-1045-1.patch, FLUME-1045-2.patch
>
>
> 1. Problem Description
> A sink being unavailable at any stage in the pipeline causes it to back-off and retry after a while. Channel's associated with such sinks start buffering data with the caveat that if you are using a memory channel it can result in a domino effect on the entire pipeline. There could be legitimate down times eg: HDFS sink being down for name node maintenance, hadoop upgrades.
> 2. Why not use a durable channel (JDBC, FileChannel)?
> Want high throughput and support sink down times as a first class use-case.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira