You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@bookkeeper.apache.org by GitBox <gi...@apache.org> on 2022/10/12 10:14:48 UTC

[GitHub] [bookkeeper] keyboardbobo opened a new issue, #3524: In high-traffic scenarios, disk space reclamation is slow

keyboardbobo opened a new issue, #3524:
URL: https://github.com/apache/bookkeeper/issues/3524

   We have 4 pulsar brokers and 4 bookie, the broker traffic is about 400M, the bookie traffic is about 1G, and the traffic is relatively large. During the day of the stress test, there will be a period of time when disk space will continue to rise and the recovery will be slow. The time interval and current limit values associated with recycling have been adjusted and do not solve the problem. But once I restart a bookie, the disk space can be reclaimed 50% in 10 minutes, I don't think what may be the cause of the problem?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@bookkeeper.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] In high-traffic scenarios, disk space reclamation is slow [bookkeeper]

Posted by "shoothzj (via GitHub)" <gi...@apache.org>.
shoothzj closed issue #3524: In high-traffic scenarios, disk space reclamation is slow
URL: https://github.com/apache/bookkeeper/issues/3524


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@bookkeeper.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [bookkeeper] david-streamlio commented on issue #3524: In high-traffic scenarios, disk space reclamation is slow

Posted by GitBox <gi...@apache.org>.
david-streamlio commented on issue #3524:
URL: https://github.com/apache/bookkeeper/issues/3524#issuecomment-1301031262

   @keyboardbobo Any feedback on how the system performed after making these recommended configuration changes? 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@bookkeeper.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [bookkeeper] dlg99 commented on issue #3524: In high-traffic scenarios, disk space reclamation is slow

Posted by GitBox <gi...@apache.org>.
dlg99 commented on issue #3524:
URL: https://github.com/apache/bookkeeper/issues/3524#issuecomment-1280270192

   It is hard to guess what's going on exactly.
   It can be anything from too aggressively configured major GC that takes too long to XFS's speculative preallocation (there are xfs tools IIRC that let you explore that).
   https://www.pulsar-neighborhood.io/articles/apache-bookkeeper-disk-usage-with-apache-pulsar/ has some general ideas (bookie-related part), for the rest you'd have to check logs and metrics


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@bookkeeper.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] In high-traffic scenarios, disk space reclamation is slow [bookkeeper]

Posted by "shoothzj (via GitHub)" <gi...@apache.org>.
shoothzj commented on issue #3524:
URL: https://github.com/apache/bookkeeper/issues/3524#issuecomment-2092555820

   closed for no updates.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@bookkeeper.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [bookkeeper] dlg99 commented on issue #3524: In high-traffic scenarios, disk space reclamation is slow

Posted by GitBox <gi...@apache.org>.
dlg99 commented on issue #3524:
URL: https://github.com/apache/bookkeeper/issues/3524#issuecomment-1281485553

   I can't say anything about data loss, I assume you'd notice that. TBH I doubt that.
   
   As I said earlier, i'd look into two possibilities: 
   
   1. * xfs prealloc (can be reconfigured if it is a problem) or similar file system magic:  shut down the bookie gracefully, don't bring it up. If the disk space released at this point I'd start poking into prealloc configuration.
   2. * GC configuration. Major GC is too aggressive, tries to compact entry logs that don't have a lot of space to save (especially without this change https://github.com/apache/bookkeeper/pull/3390) with verifyMetadataOnGC=false - it rewrites entry log with ledgers that deleted already but were live when GC started. When you restart the bookie it re-reads ledgers data and lots of entry logs can be deleted without having to go through the compaction, this releases a lot of space quickly.  
   
   in the second case I'd try verifyMetadataOnGC=true (a bit more requests to ZK during compaction), set minorCompactionMaxTimeMillis to 30-60min, majorCompactionMaxTimeMillis to an e.g. 2x of that, cherry-pick PR I mentioned above into the build you are running (if you can). 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@bookkeeper.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [bookkeeper] keyboardbobo commented on issue #3524: In high-traffic scenarios, disk space reclamation is slow

Posted by GitBox <gi...@apache.org>.
keyboardbobo commented on issue #3524:
URL: https://github.com/apache/bookkeeper/issues/3524#issuecomment-1282126901

   > I can't say anything about data loss, I assume you'd notice that. TBH I doubt that.
   > 
   > As I said earlier, i'd look into two possibilities:
   > 
   > 1. xfs prealloc (can be reconfigured if it is a problem) or similar file system magic:  shut down the bookie gracefully, don't bring it up. If the disk space released at this point I'd start poking into prealloc configuration.
   > 2. GC configuration. Major GC is too aggressive, tries to compact entry logs that don't have a lot of space to save (especially without this change [[Issue 3389] Prioritize compaction of entry logs with the lowest amount of remaining usable data #3390](https://github.com/apache/bookkeeper/pull/3390)) with verifyMetadataOnGC=false - it rewrites entry log with ledgers that deleted already but were live when GC started. When you restart the bookie it re-reads ledgers data and lots of entry logs can be deleted without having to go through the compaction, this releases a lot of space quickly.
   > 
   > in the second case I'd try verifyMetadataOnGC=true (a bit more requests to ZK during compaction), set minorCompactionMaxTimeMillis to 30-60min, majorCompactionMaxTimeMillis to an e.g. 2x of that, cherry-pick PR I mentioned above into the build you are running (if you can).
   
   Thanks for your suggestion, I'll give it a try @dlg99 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@bookkeeper.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [bookkeeper] keyboardbobo commented on issue #3524: In high-traffic scenarios, disk space reclamation is slow

Posted by GitBox <gi...@apache.org>.
keyboardbobo commented on issue #3524:
URL: https://github.com/apache/bookkeeper/issues/3524#issuecomment-1280376557

   > It is hard to guess what's going on exactly. It can be anything from too aggressively configured major GC that takes too long to XFS's speculative preallocation (there are xfs tools IIRC that let you explore that). https://www.pulsar-neighborhood.io/articles/apache-bookkeeper-disk-usage-with-apache-pulsar/ has some general ideas (bookie-related part), for the rest you'd have to check logs and metrics
   
   @dlg99 Thank you for your reply
   thank you for your reply. We have closed journal, and some files in the `entryLogMetaMap `of my debug `GarbageCollectorThread `have a usage rate of more than 99% before restarting (the disk has been confirmed to be refreshed before restarting), but the usage rate has become 0% after restarting. Is it possible to ledgersDb data Lost, resulting in accidental deletion?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@bookkeeper.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org