[jira] [Commented] (FLUME-1665) Data from FileChannel will be duplicated when restarting configuration

["Denny Ye (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLUME-1665?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13488415#comment-13488415 ] 

Denny Ye commented on FLUME-1665:
---------------------------------

hi Hari, after taking with you, I checked and improved my test cases to track missing and duplicate. I modified code as I said to write checkpoint even if restart, part of tests also have failed. Thus, that might be not a problem in your interpretation. In logically, I will raise an example to describe process what we have met.

=====================================
Transaction batch size is 1000 in Sink, and SinkRunner was running.
When it was consuming at number#500 in this batch, Sink thread received 'interrupt' notification from modification of configuration file.
Then error happened and rollback to original of this batch, it was looks like that this batch hasn't been consumed yet.
Actually, that 500 events has already been consumed into HDFS file before interruption. 

Next loop, when Flume restarted all components, it also would sending all events at that batch to HDFS. Duplicate has happening.

=====================================
Following the example above, I removed the 'interrupt' from modification of configuration file to ensure the consumption perfectly (100th lines at SinkRunner.java). Because we have already set the stop single, the Sink thread will exit at end of this batch. We don't need to interrupt that thread immediately. In this case, all the test cases has passed normally.

What's the opinion of yours for my interpretation?
                
> Data from FileChannel will be duplicated when restarting configuration
> ----------------------------------------------------------------------
>
>                 Key: FLUME-1665
>                 URL: https://issues.apache.org/jira/browse/FLUME-1665
>             Project: Flume
>          Issue Type: Bug
>          Components: Channel
>    Affects Versions: v1.2.0, v1.3.0
>            Reporter: Denny Ye
>              Labels: FileChannel
>
> When Flume process was running, I changed configuration property and Flume rebooted without process restarting. Events will be duplicated in next loop, also has been consumed before all components have stopped. 
> I found the root cause. When FileChannel was stopping, it should save the 'inflightPuts' and 'inflightTakes' into disk for preparing in next loop.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira