[jira] [Created] (KAFKA-515) Log cleanup can close a file channel opnened by Log.read before the transfer completes

["Swapnil Ghike (JIRA)" <ji...@apache.org>]

Swapnil Ghike created KAFKA-515:
-----------------------------------

             Summary: Log cleanup can close a file channel opnened by Log.read before the transfer completes
                 Key: KAFKA-515
                 URL: https://issues.apache.org/jira/browse/KAFKA-515
             Project: Kafka
          Issue Type: Bug
    Affects Versions: 0.8
            Reporter: Swapnil Ghike
             Fix For: 0.8


If consumers are lagging behind, then log cleanup activities can close a file channel opened by Log.read 
1. before the transfer the starts (broker will probably throw an exception in this case) OR
2. during the transfer (possibility of half baked corrupted data being sent to consumer?)

We probably haven't hit this race condition in practice because the consumers consume data well before the logs are cleaned up.

To avoid this issue, we could avoid cleaning up the file until the transfer is complete. Reference counting?

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (KAFKA-515) Log cleanup can close a file channel opnened by Log.read before the transfer completes

["Jay Kreps (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13456210#comment-13456210 ] 

Jay Kreps commented on KAFKA-515:
---------------------------------

A simpler approach then reference counting would be to delete the segment in two phases. First remove it from the segment list. This will prevent future requests from using that segment file. Then wait a bit. Then delete the file. This is essentially a race condition since we can't really guarantee that no requests still reference the file, however if we wait (say) 60 seconds it should be extremely unlikely. This could be done in the LogManager.cleanupLogs() method. In the first iteration the logs to be deleted would be added to a list pending deletion. On a subsequent iteration, after sufficient time had passed, the log would be deleted.
                
> Log cleanup can close a file channel opnened by Log.read before the transfer completes
> --------------------------------------------------------------------------------------
>
>                 Key: KAFKA-515
>                 URL: https://issues.apache.org/jira/browse/KAFKA-515
>             Project: Kafka
>          Issue Type: Bug
>    Affects Versions: 0.8
>            Reporter: Swapnil Ghike
>              Labels: bugs
>             Fix For: 0.8
>
>
> If consumers are lagging behind, then log cleanup activities can close a file channel opened by Log.read 
> 1. before the transfer the starts (broker will probably throw an exception in this case) OR
> 2. during the transfer (possibility of half baked corrupted data being sent to consumer?)
> We probably haven't hit this race condition in practice because the consumers consume data well before the logs are cleaned up.
> To avoid this issue, we could avoid cleaning up the file until the transfer is complete. Reference counting?

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira