[jira] [Commented] (KAFKA-636) Make log segment delete asynchronous

["Jun Rao (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13529142#comment-13529142 ] 

Jun Rao commented on KAFKA-636:
-------------------------------

Thanks for the patch. +1. Just one minor comment:

1. Log.deleteSegment(): In asyncDeleteFiles(), is it better to log the deleting of a log segment in info instead of debug?

                
> Make log segment delete asynchronous
> ------------------------------------
>
>                 Key: KAFKA-636
>                 URL: https://issues.apache.org/jira/browse/KAFKA-636
>             Project: Kafka
>          Issue Type: Bug
>            Reporter: Jay Kreps
>            Assignee: Jay Kreps
>         Attachments: KAFKA-636-v1.patch
>
>
> We have a few corner-case bugs around delete of segment files:
> 1. It is possible for delete and truncate to kind of cross streams and end up with a case where you have no segments.
> 2. Reads on the log have no locking (which is good) but as a result deleting a segment that is being read will result in some kind of I/O exception.
> 3. We can't easily fix the synchronization problems without deleting files inside the log's write lock. This can be a problem as deleting a 2GB segment can take a couple of seconds even on an unloaded system.
> The proposed fix for these problems is to make file removal asynchronous using the following scheme as the new delete scheme:
> 1. Immediately remove the file from segment map and rename the file from X to X.deleted (e.g. 0000000.log to 000000.log.deleted. We think renaming a file will not impact reads since the file is already open and hence the name is irrelevant. This will always be O(1) and can be done inside the write lock.
> 2. Schedule a future operation to delete the file. The time to wait would be configurable but we would just default it to 60 seconds and probably no one would ever change it.
> 3. On startup we would delete any files with the .deleted suffix as they would have been pending deletes that didn't take place.
> I plan to do this soon working against the refactored log (KAFKA-521). We can opt to back port the patch for 0.8 if we are feeling daring.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira