You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@bookkeeper.apache.org by GitBox <gi...@apache.org> on 2021/09/17 03:11:44 UTC
[GitHub] [bookkeeper] hrsakai opened a new pull request #2797: Make maxConcurrentZkRequests for gc configurable
hrsakai opened a new pull request #2797:
URL: https://github.com/apache/bookkeeper/pull/2797
### Motivation
In one day, zookeepers became high cpu usage and disk full.
The cause of this is bookie's gc of overreplicated ledgers.
Gc created/deleted zk nodes under `/ledgers/underreplication/locks` very frequently and some bookies ran gc at same time.
As a result, zookeepers created a lot of snapshots and became disk full.
I want to configure max zk concurrent requests lower than 1000(default) to avoid heavy traffic at a specific time.
### Changes
* Make max zk concurrent requests for garbage collection of overreplicated ledgers configurable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@bookkeeper.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bookkeeper] hrsakai commented on pull request #2797: Make maxConcurrentZkRequests for gc configurable
Posted by GitBox <gi...@apache.org>.
hrsakai commented on pull request #2797:
URL: https://github.com/apache/bookkeeper/pull/2797#issuecomment-923492582
rerun failure checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@bookkeeper.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bookkeeper] eolivelli merged pull request #2797: Make maxConcurrentZkRequests for gc configurable
Posted by GitBox <gi...@apache.org>.
eolivelli merged pull request #2797:
URL: https://github.com/apache/bookkeeper/pull/2797
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@bookkeeper.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bookkeeper] hrsakai commented on a change in pull request #2797: Make maxConcurrentZkRequests for gc configurable
Posted by GitBox <gi...@apache.org>.
hrsakai commented on a change in pull request #2797:
URL: https://github.com/apache/bookkeeper/pull/2797#discussion_r717182931
##########
File path: bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/ScanAndCompareGarbageCollector.java
##########
@@ -101,14 +101,16 @@ public ScanAndCompareGarbageCollector(LedgerManager ledgerManager, CompactableLe
if (gcOverReplicatedLedgerIntervalMillis > 0) {
this.enableGcOverReplicatedLedger = true;
}
+ this.maxConcurrentRequests = conf.getGcOverreplicatedLedgerMaxConcurrentRequests();
this.zkServers = ZKMetadataDriverBase.resolveZkServers(conf);
this.zkLedgersRootPath = ZKMetadataDriverBase.resolveZkLedgersRootPath(conf);
- LOG.info("Over Replicated Ledger Deletion : enabled=" + enableGcOverReplicatedLedger + ", interval="
- + gcOverReplicatedLedgerIntervalMillis);
+ LOG.info("Over Replicated Ledger Deletion : enabled={}, interval={}, maxConcurrentRequest={}",
Review comment:
@eolivelli
can you please review again and merge this PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@bookkeeper.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bookkeeper] hrsakai commented on a change in pull request #2797: Make maxConcurrentZkRequests for gc configurable
Posted by GitBox <gi...@apache.org>.
hrsakai commented on a change in pull request #2797:
URL: https://github.com/apache/bookkeeper/pull/2797#discussion_r711589774
##########
File path: bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/ScanAndCompareGarbageCollector.java
##########
@@ -101,14 +101,16 @@ public ScanAndCompareGarbageCollector(LedgerManager ledgerManager, CompactableLe
if (gcOverReplicatedLedgerIntervalMillis > 0) {
this.enableGcOverReplicatedLedger = true;
}
+ this.maxConcurrentRequests = conf.getGcOverreplicatedLedgerMaxConcurrentRequests();
this.zkServers = ZKMetadataDriverBase.resolveZkServers(conf);
this.zkLedgersRootPath = ZKMetadataDriverBase.resolveZkLedgersRootPath(conf);
- LOG.info("Over Replicated Ledger Deletion : enabled=" + enableGcOverReplicatedLedger + ", interval="
- + gcOverReplicatedLedgerIntervalMillis);
+ LOG.info("Over Replicated Ledger Deletion : enabled={}, interval={}, maxConcurrentRequest={}",
Review comment:
Fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@bookkeeper.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bookkeeper] hrsakai commented on a change in pull request #2797: Make maxConcurrentZkRequests for gc configurable
Posted by GitBox <gi...@apache.org>.
hrsakai commented on a change in pull request #2797:
URL: https://github.com/apache/bookkeeper/pull/2797#discussion_r717182931
##########
File path: bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/ScanAndCompareGarbageCollector.java
##########
@@ -101,14 +101,16 @@ public ScanAndCompareGarbageCollector(LedgerManager ledgerManager, CompactableLe
if (gcOverReplicatedLedgerIntervalMillis > 0) {
this.enableGcOverReplicatedLedger = true;
}
+ this.maxConcurrentRequests = conf.getGcOverreplicatedLedgerMaxConcurrentRequests();
this.zkServers = ZKMetadataDriverBase.resolveZkServers(conf);
this.zkLedgersRootPath = ZKMetadataDriverBase.resolveZkLedgersRootPath(conf);
- LOG.info("Over Replicated Ledger Deletion : enabled=" + enableGcOverReplicatedLedger + ", interval="
- + gcOverReplicatedLedgerIntervalMillis);
+ LOG.info("Over Replicated Ledger Deletion : enabled={}, interval={}, maxConcurrentRequest={}",
Review comment:
@eolivelli
can you please review again and merge this PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@bookkeeper.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bookkeeper] hrsakai commented on pull request #2797: Make maxConcurrentZkRequests for gc configurable
Posted by GitBox <gi...@apache.org>.
hrsakai commented on pull request #2797:
URL: https://github.com/apache/bookkeeper/pull/2797#issuecomment-923436944
rerun failure checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@bookkeeper.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bookkeeper] nicoloboschi commented on a change in pull request #2797: Make maxConcurrentZkRequests for gc configurable
Posted by GitBox <gi...@apache.org>.
nicoloboschi commented on a change in pull request #2797:
URL: https://github.com/apache/bookkeeper/pull/2797#discussion_r710822934
##########
File path: bookkeeper-server/src/main/java/org/apache/bookkeeper/conf/ServerConfiguration.java
##########
@@ -111,6 +111,8 @@
protected static final String GC_WAIT_TIME = "gcWaitTime";
protected static final String IS_FORCE_GC_ALLOW_WHEN_NO_SPACE = "isForceGCAllowWhenNoSpace";
protected static final String GC_OVERREPLICATED_LEDGER_WAIT_TIME = "gcOverreplicatedLedgerWaitTime";
+ protected static final String GC_OVERREPLICATED_LEDGER_MAX_CONCURRENT_ZK_REQUESTS =
Review comment:
since this property is not tightly related to ZK but it's more a logical option, IMHO we should not mention ZK in the name of the property
GC_OVERREPLICATED_LEDGER_MAX_CONCURRENT_REQUESTS ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@bookkeeper.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bookkeeper] hrsakai commented on pull request #2797: Make maxConcurrentZkRequests for gc configurable
Posted by GitBox <gi...@apache.org>.
hrsakai commented on pull request #2797:
URL: https://github.com/apache/bookkeeper/pull/2797#issuecomment-965973241
rerun failure checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@bookkeeper.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bookkeeper] eolivelli commented on a change in pull request #2797: Make maxConcurrentZkRequests for gc configurable
Posted by GitBox <gi...@apache.org>.
eolivelli commented on a change in pull request #2797:
URL: https://github.com/apache/bookkeeper/pull/2797#discussion_r711518760
##########
File path: bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/ScanAndCompareGarbageCollector.java
##########
@@ -101,14 +101,16 @@ public ScanAndCompareGarbageCollector(LedgerManager ledgerManager, CompactableLe
if (gcOverReplicatedLedgerIntervalMillis > 0) {
this.enableGcOverReplicatedLedger = true;
}
+ this.maxConcurrentRequests = conf.getGcOverreplicatedLedgerMaxConcurrentRequests();
this.zkServers = ZKMetadataDriverBase.resolveZkServers(conf);
this.zkLedgersRootPath = ZKMetadataDriverBase.resolveZkLedgersRootPath(conf);
- LOG.info("Over Replicated Ledger Deletion : enabled=" + enableGcOverReplicatedLedger + ", interval="
- + gcOverReplicatedLedgerIntervalMillis);
+ LOG.info("Over Replicated Ledger Deletion : enabled={}, interval={}, maxConcurrentRequest={}",
Review comment:
Typo: maxConcurrentRequest > maxConcurrentRequests
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@bookkeeper.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bookkeeper] hrsakai commented on pull request #2797: Make maxConcurrentZkRequests for gc configurable
Posted by GitBox <gi...@apache.org>.
hrsakai commented on pull request #2797:
URL: https://github.com/apache/bookkeeper/pull/2797#issuecomment-921597974
@nicoloboschi
Thank you for your review.
I renamed to `GC_OVERREPLICATED_LEDGER_MAX_CONCURRENT_REQUESTS`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@bookkeeper.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org