You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@iotdb.apache.org by "MrQuansy (via GitHub)" <gi...@apache.org> on 2024/12/15 15:20:08 UTC
[PR] Add eviction metrics in binary allocator [iotdb]
MrQuansy opened a new pull request, #14432:
URL: https://github.com/apache/iotdb/pull/14432
## Description
This PR introduces new metrics to track memory eviction statistics in the Binary Allocator component. The allocator implements two distinct eviction strategies:
Sample-based eviction: Periodically samples memory regions to identify eviction candidates
GC-based eviction: Leverages garbage collection cycles to reclaim unused memory
The primary enhancement is the addition of metrics that monitor the cumulative size of evicted memory through both strategies. These metrics provide visibility into the allocator's memory reclamation behavior and efficiency.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add eviction metrics in binary allocator [iotdb]
Posted by "OneSizeFitsQuorum (via GitHub)" <gi...@apache.org>.
OneSizeFitsQuorum merged PR #14432:
URL: https://github.com/apache/iotdb/pull/14432
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add eviction metrics in binary allocator [iotdb]
Posted by "OneSizeFitsQuorum (via GitHub)" <gi...@apache.org>.
OneSizeFitsQuorum commented on code in PR #14432:
URL: https://github.com/apache/iotdb/pull/14432#discussion_r1887802090
##########
iotdb-core/node-commons/src/main/java/org/apache/iotdb/commons/binaryallocator/arena/Arena.java:
##########
@@ -182,22 +190,25 @@ private void updateSample() {
average.sample(getActiveSize());
}
- private void resize() {
+ private long resize() {
average.update();
int needRemain = (int) Math.ceil(average.average()) - getActiveSize();
- evict(getQueueSize() - needRemain);
+ return evict(getQueueSize() - needRemain);
}
- private void evict(double ratio) {
- evict((int) (getQueueSize() * ratio));
+ private long evict(double ratio) {
+ return evict((int) (getQueueSize() * ratio));
}
- private void evict(int num) {
+ private long evict(int num) {
+ int evicted = 0;
Review Comment:
long
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add eviction metrics in binary allocator [iotdb]
Posted by "OneSizeFitsQuorum (via GitHub)" <gi...@apache.org>.
OneSizeFitsQuorum commented on code in PR #14432:
URL: https://github.com/apache/iotdb/pull/14432#discussion_r1886355862
##########
iotdb-core/node-commons/src/main/java/org/apache/iotdb/commons/binaryallocator/arena/Arena.java:
##########
@@ -182,22 +188,25 @@ private void updateSample() {
average.sample(getActiveSize());
}
- private void resize() {
+ private int resize() {
average.update();
int needRemain = (int) Math.ceil(average.average()) - getActiveSize();
- evict(getQueueSize() - needRemain);
+ return evict(getQueueSize() - needRemain);
}
- private void evict(double ratio) {
- evict((int) (getQueueSize() * ratio));
+ private int evict(double ratio) {
+ return evict((int) (getQueueSize() * ratio));
}
- private void evict(int num) {
+ private int evict(int num) {
+ int evicted = 0;
while (num > 0 && !queue.isEmpty()) {
queue.poll();
evictions.incrementAndGet();
num--;
+ evicted += byteArraySize;
Review Comment:
use long？or 2GB will be the max?
##########
iotdb-core/node-commons/src/main/java/org/apache/iotdb/commons/binaryallocator/metric/BinaryAllocatorMetrics.java:
##########
@@ -95,10 +111,28 @@ public void unbindFrom(AbstractMetricService metricService) {
Metric.BINARY_ALLOCATOR.toString(),
Tag.NAME.toString(),
ALLOCATE_FROM_JVM);
+ metricService.remove(
+ MetricType.COUNTER,
+ Metric.BINARY_ALLOCATOR.toString(),
+ Tag.NAME.toString(),
+ EVICTED_BY_SAMPLE_EVICTION);
+ metricService.remove(
+ MetricType.COUNTER,
+ Metric.BINARY_ALLOCATOR.toString(),
+ Tag.NAME.toString(),
+ EVICTED_BY_GC_EVICTION);
}
- public void updateCounter(int allocateFromSlabDelta, int allocateFromJVMDelta) {
+ public void updateAllocationCounter(int allocateFromSlabDelta, int allocateFromJVMDelta) {
allocateFromSlab.inc(allocateFromSlabDelta);
allocateFromJVM.inc(allocateFromJVMDelta);
}
+
+ public void updateGcEvictionCounter(int evictedByGcEvictionDelta) {
Review Comment:
long
##########
iotdb-core/node-commons/src/main/java/org/apache/iotdb/commons/binaryallocator/metric/BinaryAllocatorMetrics.java:
##########
@@ -95,10 +111,28 @@ public void unbindFrom(AbstractMetricService metricService) {
Metric.BINARY_ALLOCATOR.toString(),
Tag.NAME.toString(),
ALLOCATE_FROM_JVM);
+ metricService.remove(
+ MetricType.COUNTER,
+ Metric.BINARY_ALLOCATOR.toString(),
+ Tag.NAME.toString(),
+ EVICTED_BY_SAMPLE_EVICTION);
+ metricService.remove(
+ MetricType.COUNTER,
+ Metric.BINARY_ALLOCATOR.toString(),
+ Tag.NAME.toString(),
+ EVICTED_BY_GC_EVICTION);
}
- public void updateCounter(int allocateFromSlabDelta, int allocateFromJVMDelta) {
+ public void updateAllocationCounter(int allocateFromSlabDelta, int allocateFromJVMDelta) {
allocateFromSlab.inc(allocateFromSlabDelta);
allocateFromJVM.inc(allocateFromJVMDelta);
}
+
+ public void updateGcEvictionCounter(int evictedByGcEvictionDelta) {
+ evictedByGcEviction.inc(evictedByGcEvictionDelta);
+ }
+
+ public void updateSampleEvictionCounter(int evictedBySampleEvictionDelta) {
Review Comment:
long
##########
iotdb-core/node-commons/src/main/java/org/apache/iotdb/commons/binaryallocator/BinaryAllocator.java:
##########
@@ -231,17 +233,19 @@ public void runGcEviction(long curGcTimePercent) {
return;
}
+ int evictedSize = 0;
Review Comment:
long
##########
iotdb-core/node-commons/src/main/java/org/apache/iotdb/commons/binaryallocator/BinaryAllocator.java:
##########
@@ -252,9 +256,11 @@ public SampleEvictor(String name, Duration evictorShutdownTimeoutDuration) {
@Override
public void run() {
+ int evictedSize = 0;
Review Comment:
long
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org