You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@bookkeeper.apache.org by "ethqunzhong (via GitHub)" <gi...@apache.org> on 2024/05/23 04:21:12 UTC
[PR] improve: Enrich GC metrics to better analyze GC behavior and the time consumption of each phase. [bookkeeper]
ethqunzhong opened a new pull request, #4384:
URL: https://github.com/apache/bookkeeper/pull/4384
### Motivation
Enrich GC metrics to better analyze GC behavior and the time consumption of each phase.
In our online environment, we found that some clusters spend a lot of time scanning metadata during gc, so the newly added metrics can more conveniently help us analyze the cluster gc behavior
this PR add 4 new metrics:
bookie_GC_LEDGER_RUNTIME
bookie_EXTRACT_META_RUNTIME
bookie_COMPACT_RUNTIME
bookie_ENTRY_LOG_COMPACT_RATIO
### Changes
- bookie_GC_LEDGER_RUNTIME
operation stats of doing gc ledgers based on metaStore, Time consumption for comparing ledger meta between local and metadata store (zk).
- bookie_EXTRACT_META_RUNTIME
Time consumption for extracting Meta from entryLogs.
- bookie_COMPACT_RUNTIME
Time consumption of entry log compaction.
- bookie_ENTRY_LOG_COMPACT_RATIO
Current proportion of compacted entry log files that have been executed (Provide reference for users when setting CompactionThreshold values, and configure more reasonable values)
<img width="904" alt="image" src="https://github.com/apache/bookkeeper/assets/16517186/79870ed5-5191-429c-b6e3-e8a08054bcd1">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@bookkeeper.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] improve: Enrich GC metrics to better analyze GC behavior and the time consumption of each phase. [bookkeeper]
Posted by "ethqunzhong (via GitHub)" <gi...@apache.org>.
ethqunzhong commented on PR #4384:
URL: https://github.com/apache/bookkeeper/pull/4384#issuecomment-2126203787
@hangc0276 @zymap PTAL. thx
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@bookkeeper.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] improve: Enrich GC metrics to better analyze GC behavior and the time consumption of each phase. [bookkeeper]
Posted by "shoothzj (via GitHub)" <gi...@apache.org>.
shoothzj merged PR #4384:
URL: https://github.com/apache/bookkeeper/pull/4384
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@bookkeeper.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] improve: Enrich GC metrics to better analyze GC behavior and the time consumption of each phase. [bookkeeper]
Posted by "hangc0276 (via GitHub)" <gi...@apache.org>.
hangc0276 commented on code in PR #4384:
URL: https://github.com/apache/bookkeeper/pull/4384#discussion_r1614985907
##########
bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/GarbageCollectorThread.java:
##########
@@ -649,9 +666,11 @@ void doCompactEntryLogs(double threshold, long maxTimeMillis) throws EntryLogMet
LOG.debug("Compaction ran for {}ms but was limited by {}ms", timeDiff, maxTimeMillis);
}
}
- LOG.info(
- "Compaction: entry log usage buckets[10% 20% 30% 40% 50% 60% 70% 80% 90% 100%] = {}, compacted {}",
- entryLogUsageBuckets, compactedBuckets);
+ int totalEntryLogNum = Arrays.stream(entryLogUsageBuckets).sum();
+ int compactedEntryLogNum = Arrays.stream(compactedBuckets).sum();
+ this.entryLogCompactRatio = totalEntryLogNum == 0 ? 0 : (double) compactedEntryLogNum / totalEntryLogNum;
Review Comment:
I suggest exposing the whole bucket statistics to metrics instead of a ratio. It will be more helpful for debugging the bookie ledger disk usage not cleaning up the issue. You can refer to Pulsar `pulsar_ml_EntrySizeBuckets` metrics
https://pulsar.apache.org/docs/3.2.x/reference-metrics/#managedledger-metrics
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@bookkeeper.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] improve: Enrich GC metrics to better analyze GC behavior and the time consumption of each phase. [bookkeeper]
Posted by "shoothzj (via GitHub)" <gi...@apache.org>.
shoothzj commented on code in PR #4384:
URL: https://github.com/apache/bookkeeper/pull/4384#discussion_r1615517506
##########
bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/GarbageCollectorThread.java:
##########
@@ -649,9 +666,11 @@ void doCompactEntryLogs(double threshold, long maxTimeMillis) throws EntryLogMet
LOG.debug("Compaction ran for {}ms but was limited by {}ms", timeDiff, maxTimeMillis);
}
}
- LOG.info(
- "Compaction: entry log usage buckets[10% 20% 30% 40% 50% 60% 70% 80% 90% 100%] = {}, compacted {}",
- entryLogUsageBuckets, compactedBuckets);
+ int totalEntryLogNum = Arrays.stream(entryLogUsageBuckets).sum();
+ int compactedEntryLogNum = Arrays.stream(compactedBuckets).sum();
+ this.entryLogCompactRatio = totalEntryLogNum == 0 ? 0 : (double) compactedEntryLogNum / totalEntryLogNum;
Review Comment:
+1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@bookkeeper.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] improve: Enrich GC metrics to better analyze GC behavior and the time consumption of each phase. [bookkeeper]
Posted by "ethqunzhong (via GitHub)" <gi...@apache.org>.
ethqunzhong commented on code in PR #4384:
URL: https://github.com/apache/bookkeeper/pull/4384#discussion_r1636324625
##########
bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/GarbageCollectorThread.java:
##########
@@ -649,9 +666,11 @@ void doCompactEntryLogs(double threshold, long maxTimeMillis) throws EntryLogMet
LOG.debug("Compaction ran for {}ms but was limited by {}ms", timeDiff, maxTimeMillis);
}
}
- LOG.info(
- "Compaction: entry log usage buckets[10% 20% 30% 40% 50% 60% 70% 80% 90% 100%] = {}, compacted {}",
- entryLogUsageBuckets, compactedBuckets);
+ int totalEntryLogNum = Arrays.stream(entryLogUsageBuckets).sum();
+ int compactedEntryLogNum = Arrays.stream(compactedBuckets).sum();
+ this.entryLogCompactRatio = totalEntryLogNum == 0 ? 0 : (double) compactedEntryLogNum / totalEntryLogNum;
Review Comment:
make sence. udpate new metrics `entry_log_usage_buckets_le_*` to expose the whole bucket statistics.
The metrics are as follows
<img width="890" alt="image" src="https://github.com/apache/bookkeeper/assets/16517186/74e7451c-855a-4d78-86b8-f8976fc0180c">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@bookkeeper.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] improve: Enrich GC metrics to better analyze GC behavior and the time consumption of each phase. [bookkeeper]
Posted by "shoothzj (via GitHub)" <gi...@apache.org>.
shoothzj commented on PR #4384:
URL: https://github.com/apache/bookkeeper/pull/4384#issuecomment-2238053127
@dlg99 @hangc0276 could you please help take a look again? @eolivelli PTAL, thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@bookkeeper.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] improve: Enrich GC metrics to better analyze GC behavior and the time consumption of each phase. [bookkeeper]
Posted by "ethqunzhong (via GitHub)" <gi...@apache.org>.
ethqunzhong commented on code in PR #4384:
URL: https://github.com/apache/bookkeeper/pull/4384#discussion_r1636320811
##########
bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/GarbageCollectorThread.java:
##########
@@ -411,14 +415,20 @@ public void runWithFlags(boolean force, boolean suspendMajor, boolean suspendMin
try {
// gc inactive/deleted ledgers
// this is used in extractMetaFromEntryLogs to calculate the usage of entry log
+ long gcLedgersStart = MathUtils.nowInNano();
doGcLedgers();
+ gcStats.getGcLedgerRuntime()
+ .registerSuccessfulEvent(MathUtils.nowInNano() - gcLedgersStart, TimeUnit.NANOSECONDS);
Review Comment:
updated.
##########
bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/GarbageCollectorThread.java:
##########
@@ -411,14 +415,20 @@ public void runWithFlags(boolean force, boolean suspendMajor, boolean suspendMin
try {
// gc inactive/deleted ledgers
// this is used in extractMetaFromEntryLogs to calculate the usage of entry log
+ long gcLedgersStart = MathUtils.nowInNano();
doGcLedgers();
Review Comment:
updated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@bookkeeper.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] improve: Enrich GC metrics to better analyze GC behavior and the time consumption of each phase. [bookkeeper]
Posted by "hangc0276 (via GitHub)" <gi...@apache.org>.
hangc0276 commented on code in PR #4384:
URL: https://github.com/apache/bookkeeper/pull/4384#discussion_r1614985215
##########
bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/GarbageCollectorThread.java:
##########
@@ -411,14 +415,20 @@ public void runWithFlags(boolean force, boolean suspendMajor, boolean suspendMin
try {
// gc inactive/deleted ledgers
// this is used in extractMetaFromEntryLogs to calculate the usage of entry log
+ long gcLedgersStart = MathUtils.nowInNano();
doGcLedgers();
Review Comment:
The same as extractMetaFromEntryLogs and doGcEntryLogs
##########
bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/GarbageCollectorThread.java:
##########
@@ -411,14 +415,20 @@ public void runWithFlags(boolean force, boolean suspendMajor, boolean suspendMin
try {
// gc inactive/deleted ledgers
// this is used in extractMetaFromEntryLogs to calculate the usage of entry log
+ long gcLedgersStart = MathUtils.nowInNano();
doGcLedgers();
Review Comment:
doGcLedgers may fail, can we record the metric in doGcLedgers method so that we can record the failed metrics?
##########
bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/GarbageCollectorThread.java:
##########
@@ -411,14 +415,20 @@ public void runWithFlags(boolean force, boolean suspendMajor, boolean suspendMin
try {
// gc inactive/deleted ledgers
// this is used in extractMetaFromEntryLogs to calculate the usage of entry log
+ long gcLedgersStart = MathUtils.nowInNano();
doGcLedgers();
+ gcStats.getGcLedgerRuntime()
+ .registerSuccessfulEvent(MathUtils.nowInNano() - gcLedgersStart, TimeUnit.NANOSECONDS);
Review Comment:
We can use `MathUtils.elapsedNanos(gcLedgersStart)` instead
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@bookkeeper.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] improve: Enrich GC metrics to better analyze GC behavior and the time consumption of each phase. [bookkeeper]
Posted by "ethqunzhong (via GitHub)" <gi...@apache.org>.
ethqunzhong commented on PR #4384:
URL: https://github.com/apache/bookkeeper/pull/4384#issuecomment-2172104127
@hangc0276 @shoothzj PTAL. thx
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@bookkeeper.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org