[jira] [Commented] (FLUME-1318) Graceful shutdown and drain for memory channel

["Inder SIngh (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLUME-1318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13410037#comment-13410037 ] 

Inder SIngh commented on FLUME-1318:
------------------------------------

My two cents ->

Follow a similar approach for all channels i.e. should be a part of lifecycle.

1. sinks should stop after their corresponding channel, sources can close in any order.
2. stopping a channel should isolate it from all attached sources after the current trans is committed.
3. channels ends it's life once it's drained completely if they aren't persistent.
4. sinks end their life if their corresponding channels aren't there or they have drained completely.

Potential issues ->

1. if a sink is stuck eg: (HDFSSink when HDFS is down) might not drain causing SHUTDOWN to be blocked.
so if a sink becomes unresponsive(timeout?) then no way to save data from channels
                
> Graceful shutdown and drain for memory channel
> ----------------------------------------------
>
>                 Key: FLUME-1318
>                 URL: https://issues.apache.org/jira/browse/FLUME-1318
>             Project: Flume
>          Issue Type: Improvement
>          Components: Channel, Sinks+Sources
>            Reporter: Mubarak Seyed
>              Labels: newbie
>
> It appears from our test that if i go and shutdown the agent which uses memory channel, there is no clean way of throttling incoming requests to source, draining events from memory channel and do clean graceful shutdown of an agent. We don't have a visibility of how much data(events) in memory channel are getting dropped when shutdown happens.
> It would be good if we add a method to throttle incoming requests to source, drain events from memory channel and then shutdown sink/channel/source. (We can certainly add counters/metrics as part of channel monitoring). I think file-channel covers by adding it to WAL.
> HBase RegionServer shutdown handler does the some drain process (it syncs the edits in Write-Ahead-Log and notifies the master for regions move/assignment)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira