[GitHub] [bookkeeper] RaulGracia opened a new issue #3150: How to recover Bookies that completely run out of storage space?

[GitBox <gi...@apache.org>]

RaulGracia opened a new issue #3150:
URL: https://github.com/apache/bookkeeper/issues/3150


   **QUESTION**
   We have done recent tests to observe the behavior of Bookies when exhausting its storage space. In one test, we got the 3 Bookies (10GB journal, 10GB ledger) in read-only mode (expected). But after some time (perhaps after a GC cycle, which may require extra storage), we observed that the 3 Bookies were in `CrashLoopBackoff` (Kubernetes deployment):
   ```
   # k get po
   NAME                                                READY   STATUS             RESTARTS   AGE
   bookkeeper-bookie-0                                 0/1     Running            247        20h
   bookkeeper-bookie-1                                 0/1     CrashLoopBackOff   253        20h
   bookkeeper-bookie-2                                 0/1     CrashLoopBackOff   239        20h
   ```
   
   The reason for Bookies not being able to start is that there is no space left on device to start some of the internal Bookie processes:
   ```
   2022-03-21 17:51:11,309 - ERROR - [BookieJournal-3181:Journal@1146] - I/O exception in Journal thread!
   2022-03-21 17:51:11,309 - ERROR - [BookieJournal-3181:Journal@1146] - I/O exception in Journal thread!java.io.IOException: No space left on device 
   at java.base/java.io.UnixFileSystem.createFileExclusively(Native Method) at java.base/java.io.File.createNewFile(File.java:1035) 
   at org.apache.bookkeeper.bookie.JournalChannel.<init>(JournalChannel.java:159) 
   at org.apache.bookkeeper.bookie.JournalChannel.<init>(JournalChannel.java:117) 
   at org.apache.bookkeeper.bookie.Journal.run(Journal.java:963)
   2022-03-21 17:51:11,310 - INFO  - [BookieJournal-3181:Journal@1158] - Journal exited loop!
   ```
   
   _The question is: is there a suggested recovery procedure if we find Bookies in this situation?_ 
   One constraint to any potential solution to this problem: we need the data of the impacted Bookies to be available, as the system that uses Bookkeeper requires accessing it at least once. 
   
   We have considered resizing the Bookie volumes. My understanding is that, if we achieve this, that would solve the problem and Bookies would be able to boot. In case of a volume not being resizable, we have also considered adding new journal/ledger directories to the Bookies that are backed up on new volumes, but I don't know if this would work (we may need to play around with the Cookie and metadata of the Bookies).
   
   Having a procedure to deal with this problem (if it does not exist) in the documentation would be great as well.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@bookkeeper.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org