[jira] [Commented] (FLUME-2306) onIdleCallback is not canceled when stop hdfs sink

["chenshangan (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLUME-2306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13888389#comment-13888389 ] 

chenshangan commented on FLUME-2306:
------------------------------------

Agreed, thanks a lot.

> onIdleCallback is not canceled when stop hdfs sink
> --------------------------------------------------
>
>                 Key: FLUME-2306
>                 URL: https://issues.apache.org/jira/browse/FLUME-2306
>             Project: Flume
>          Issue Type: Bug
>          Components: Sinks+Sources
>    Affects Versions: v1.4.0
>            Reporter: chenshangan
>
> the hdfs sink cached 5000 open files by default and it cost quite a lot of memory in total when using lzo CompressedStream. We should open the idleTimeout feature to resolve it. But there seems to be a bug with this feature. When stopping flume, HDFSWriter does not cancel the idle scheduler, which might cause flume not to stop. So I extend the current close() method in HDFSWriter as follows, and use it in HDFSEventSink when stop the sink component :
> /**
>    * when stop flume, all schedulers should be canceled
>    * @param cancelIdleCallback
>    * @throws IOException
>    * @throws InterruptedException
>    */
>   public void close(boolean cancelIdleCallback) throws IOException, InterruptedException{
>       close();
>       if(cancelIdleCallback){
>           if (idleFuture != null && !idleFuture.isDone()) {
>               idleFuture.cancel(false); // do not cancel myself if running!
>               idleFuture = null;
>           }
>       }
>   }



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)