You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flume.apache.org by "Praveen Ramachandra (Created) (JIRA)" <ji...@apache.org> on 2011/11/17 07:01:55 UTC
[jira] [Created] (FLUME-851) Scribe style of handling events
especially when there is a large event backlog built-up
Scribe style of handling events especially when there is a large event backlog built-up
---------------------------------------------------------------------------------------
Key: FLUME-851
URL: https://issues.apache.org/jira/browse/FLUME-851
Project: Flume
Issue Type: Improvement
Components: Channel, Node
Affects Versions: NG alpha 1
Environment: All
Reporter: Praveen Ramachandra
Newer events are of much higher value to be delivered than old events that have been accumulated.
Scribe does this very well and would like to incorporate it in flume-ng. Currently we use a dirty hack to realize this important/critical feature. If this is incorporated in the platform, users of flume-ng will benefit from this immensly
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-851) Scribe style of handling events
especially when there is a large event backlog built-up
Posted by "E. Sammer (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-851?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13154742#comment-13154742 ]
E. Sammer commented on FLUME-851:
---------------------------------
Praveen:
I agree this is a great feature. It should be possible to write a Channel implementation that has a priority comparator (or a similar implementation) that uses the timestamp in the event headers. Currently, we don't guarantee that all events have a timestamp header, but it's early enough in NG's life that we could make it true (or simply add System.currentTimeInMillis() on Channel#put(Event) if the header doesn't exist). We could also add the notion of an Event TTL (i.e. discard this even if it can't delivered within X millis) as another way of achieving something similar. Can you describe the ideal behavior vs. what you've already implemented, if anything, so we can plan out what this might look like?
Thanks!
> Scribe style of handling events especially when there is a large event backlog built-up
> ---------------------------------------------------------------------------------------
>
> Key: FLUME-851
> URL: https://issues.apache.org/jira/browse/FLUME-851
> Project: Flume
> Issue Type: Improvement
> Components: Channel, Node
> Affects Versions: NG alpha 1
> Environment: All
> Reporter: Praveen Ramachandra
>
> Newer events are of much higher value to be delivered than old events that have been accumulated.
> Scribe does this very well and would like to incorporate it in flume-ng. Currently we use a dirty hack to realize this important/critical feature. If this is incorporated in the platform, users of flume-ng will benefit from this immensly
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-851) Scribe style of handling events
especially when there is a large event backlog built-up
Posted by "Praveen Ramachandra (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-851?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13154883#comment-13154883 ]
Praveen Ramachandra commented on FLUME-851:
-------------------------------------------
Hi Sammer,
I think this can work even without any custom time fields. We can look at the scribe's mirror its implementation. All of scribe code is 6k C code, hopefully it is simple to understand and mimic or better yet improve on it. At best we can make it configurable at a channel level, with system wide default.
Ideal Behavior
When there is event backlog and new events come to the system, newer events are delivered ahead of "backlogged" events. Once the backlog is cleared or when there is no backlog in the system there is no deliberate/explicit change to the order in which events are delivered i.e., system default behavior kicks in.
Implemented Behavior
Create 2 channels, one called backlog-bucket and the other sunnyday-bucket. While the source is delivering the events to the bucket, it first checks if there is more than "x" events in sunnyday-bucket, if true it delivers events to backlog-bucket. Any throttling applied while draining the events from buckets is
applied only to backlog-bucket. The idea is that the platform would have allocated enough resources to deal with sunny-day scenario with some added buffer.
Hope this helps
> Scribe style of handling events especially when there is a large event backlog built-up
> ---------------------------------------------------------------------------------------
>
> Key: FLUME-851
> URL: https://issues.apache.org/jira/browse/FLUME-851
> Project: Flume
> Issue Type: Improvement
> Components: Channel, Node
> Affects Versions: NG alpha 1
> Environment: All
> Reporter: Praveen Ramachandra
>
> Newer events are of much higher value to be delivered than old events that have been accumulated.
> Scribe does this very well and would like to incorporate it in flume-ng. Currently we use a dirty hack to realize this important/critical feature. If this is incorporated in the platform, users of flume-ng will benefit from this immensly
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira