[jira] [Commented] (BOOKKEEPER-180) bookie server doesn't quit when running out of disk space

["Vinay (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/BOOKKEEPER-180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13292704#comment-13292704 ] 

Vinay commented on BOOKKEEPER-180:
----------------------------------

hi,
Here One more scenario needs to be handled. 
Adding new ledger and flushing is failed in SyncThread due to disk full. But Server did not shutdown here.

{noformat}2012-06-11 140014,696 - ERROR [SyncThreadInterleavedLedgerStorage@156] - Exception flushing Ledger
java.io.IOException No space left on device
	at sun.nio.ch.FileDispatcher.write0(Native Method)
	at sun.nio.ch.FileDispatcher.write(FileDispatcher.java39)
	at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java69)
	at sun.nio.ch.IOUtil.write(IOUtil.java26)
	at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java198)
	at org.apache.bookkeeper.bookie.BufferedChannel.flush(BufferedChannel.java109)
	at org.apache.bookkeeper.bookie.EntryLogger.flush(EntryLogger.java280)
	at org.apache.bookkeeper.bookie.InterleavedLedgerStorage.flush(InterleavedLedgerStorage.java154)
	at org.apache.bookkeeper.bookie.Bookie$SyncThread.run(Bookie.java200){noformat}
                
> bookie server doesn't quit when running out of disk space
> ---------------------------------------------------------
>
>                 Key: BOOKKEEPER-180
>                 URL: https://issues.apache.org/jira/browse/BOOKKEEPER-180
>             Project: Bookkeeper
>          Issue Type: Bug
>          Components: bookkeeper-server
>            Reporter: Sijie Guo
>            Assignee: Ivan Kelly
>             Fix For: 4.1.0
>
>         Attachments: BK-180.diff, BK-180.diff_v2, BK-180.diff_v3, BK-180.diff_v4, conn3.png
>
>
> we found that the publish throughput drops down when one bookie server ran out of disk space (due to we don't do log rotation   which exhausts disk space). 
> did some investigation, we found that bookie server doesn't quit when encountering no disk space issue. so hub server treat this bookie server as available. The adding requests would be sent to this bookie server, some adding requests are put in journal queue to flush, but the journal flush thread has quit due to no disk space. so these adding requests didn't respond to bookie client until it read timeout and chose other bookie servers.
> we did an experiment to shut down the ran-out-of-disk-space bookie, the publish throughput went up again quickly.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira