You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@accumulo.apache.org by "dlmarion (via GitHub)" <gi...@apache.org> on 2026/06/29 13:24:45 UTC
[PR] Attempt at reducing GC memory usage when gathering candidates [accumulo]
dlmarion opened a new pull request, #6449:
URL: https://github.com/apache/accumulo/pull/6449
This change pushes the Value filtering to the server-side using an iterator instead of using a filter in a Stream, and sets an initial size on the candidate ArrayList to try and reduce resize operations.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Attempt at reducing GC memory usage when gathering candidates [accumulo]
Posted by "dlmarion (via GitHub)" <gi...@apache.org>.
dlmarion commented on code in PR #6449:
URL: https://github.com/apache/accumulo/pull/6449#discussion_r3493209355
##########
server/gc/src/main/java/org/apache/accumulo/gc/GCRun.java:
##########
@@ -150,7 +150,7 @@ public List<GcCandidate> readCandidatesThatFitInMemory(Iterator<GcCandidate> can
// Converting the bytes to approximate number of characters for batch size.
long candidateBatchSize = getCandidateBatchSize() / 2;
- List<GcCandidate> candidatesBatch = new ArrayList<>();
+ List<GcCandidate> candidatesBatch = new ArrayList<>(256);
Review Comment:
Yes. The value of 256 is something arbitrary, I was just trying to pick a sensible value.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Attempt at reducing GC memory usage when gathering candidates [accumulo]
Posted by "dlmarion (via GitHub)" <gi...@apache.org>.
dlmarion merged PR #6449:
URL: https://github.com/apache/accumulo/pull/6449
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Attempt at reducing GC memory usage when gathering candidates [accumulo]
Posted by "ddanielr (via GitHub)" <gi...@apache.org>.
ddanielr commented on code in PR #6449:
URL: https://github.com/apache/accumulo/pull/6449#discussion_r3493140669
##########
server/base/src/main/java/org/apache/accumulo/server/metadata/ServerAmpleImpl.java:
##########
@@ -270,7 +270,8 @@ public Iterator<GcCandidate> getGcCandidates(DataLevel level) {
throw new RuntimeException(e);
}
scanner.setRange(range);
- return scanner.stream().filter(entry -> entry.getValue().equals(SkewedKeyValue.NAME))
+ scanner.addScanIterator(new IteratorSetting(25, "gcCandidate", GcCandidateFilter.class));
Review Comment:
Are we keeping track of the system iterator priorities being used?
I see the default value of the versioning iterator being set at 20, but that's just a hardcoded value.
https://github.com/apache/accumulo/blob/9d1377c75641f135259294255f9411b1f481f99c/core/src/main/java/org/apache/accumulo/core/iteratorsImpl/IteratorConfigUtil.java#L110-L111
Wondering if that initial value of `20` should be refactored into a system constant of `DEFAULT_ITERATOR_PRIORITY`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Attempt at reducing GC memory usage when gathering candidates [accumulo]
Posted by "dlmarion (via GitHub)" <gi...@apache.org>.
dlmarion commented on code in PR #6449:
URL: https://github.com/apache/accumulo/pull/6449#discussion_r3493705980
##########
server/base/src/main/java/org/apache/accumulo/server/metadata/ServerAmpleImpl.java:
##########
@@ -270,7 +270,8 @@ public Iterator<GcCandidate> getGcCandidates(DataLevel level) {
throw new RuntimeException(e);
}
scanner.setRange(range);
- return scanner.stream().filter(entry -> entry.getValue().equals(SkewedKeyValue.NAME))
+ scanner.addScanIterator(new IteratorSetting(25, "gcCandidate", GcCandidateFilter.class));
Review Comment:
I'm not seeing a constant. I see a literal `20` being used in main in `IteratorConfigUtil.getInitialTableIteratorSettings`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Attempt at reducing GC memory usage when gathering candidates [accumulo]
Posted by "dlmarion (via GitHub)" <gi...@apache.org>.
dlmarion commented on code in PR #6449:
URL: https://github.com/apache/accumulo/pull/6449#discussion_r3493231776
##########
server/base/src/main/java/org/apache/accumulo/server/metadata/ServerAmpleImpl.java:
##########
@@ -270,7 +270,8 @@ public Iterator<GcCandidate> getGcCandidates(DataLevel level) {
throw new RuntimeException(e);
}
scanner.setRange(range);
- return scanner.stream().filter(entry -> entry.getValue().equals(SkewedKeyValue.NAME))
+ scanner.addScanIterator(new IteratorSetting(25, "gcCandidate", GcCandidateFilter.class));
Review Comment:
> Are we keeping track of the system iterator priorities being used?
I don't think there is a constant in 2.1, there might be a constant in 4.0. I can look for that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Attempt at reducing GC memory usage when gathering candidates [accumulo]
Posted by "ddanielr (via GitHub)" <gi...@apache.org>.
ddanielr commented on code in PR #6449:
URL: https://github.com/apache/accumulo/pull/6449#discussion_r3493038735
##########
server/gc/src/main/java/org/apache/accumulo/gc/GCRun.java:
##########
@@ -150,7 +150,7 @@ public List<GcCandidate> readCandidatesThatFitInMemory(Iterator<GcCandidate> can
// Converting the bytes to approximate number of characters for batch size.
long candidateBatchSize = getCandidateBatchSize() / 2;
- List<GcCandidate> candidatesBatch = new ArrayList<>();
+ List<GcCandidate> candidatesBatch = new ArrayList<>(256);
Review Comment:
Is this change just to avoid duplicative ArrayList object creation when increasing the list size?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org