You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@lucene.apache.org by jp...@apache.org on 2024/10/15 15:03:27 UTC
(lucene) branch branch_10x updated (5727b8ce591 -> 7a14a77d907)
This is an automated email from the ASF dual-hosted git repository.
jpountz pushed a change to branch branch_10x
in repository https://gitbox.apache.org/repos/asf/lucene.git
from 5727b8ce591 Optimize convert byte2int. (#13889)
new 92c4f512243 Better handle dynamic pruning when the leading clause has a single impact block. (#13904)
new 7a14a77d907 Skip madvise calls on tiny inner files of compound files. (#13917)
The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
lucene/CHANGES.txt | 4 ++++
.../lucene/search/BlockMaxConjunctionBulkScorer.java | 17 ++++++++++++++---
.../apache/lucene/store/MemorySegmentIndexInput.java | 20 ++++++++++++++------
3 files changed, 32 insertions(+), 9 deletions(-)
(lucene) 02/02: Skip madvise calls on tiny inner files of compound files. (#13917)
Posted by jp...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
jpountz pushed a commit to branch branch_10x
in repository https://gitbox.apache.org/repos/asf/lucene.git
commit 7a14a77d90743743193066a1138a9bfb69b74a56
Author: Adrien Grand <jp...@gmail.com>
AuthorDate: Tue Oct 15 16:17:51 2024 +0200
Skip madvise calls on tiny inner files of compound files. (#13917)
This commit skips `madvise` calls on inner files of compound files that are
smaller than the page size.
---
.../apache/lucene/store/MemorySegmentIndexInput.java | 20 ++++++++++++++------
1 file changed, 14 insertions(+), 6 deletions(-)
diff --git a/lucene/core/src/java21/org/apache/lucene/store/MemorySegmentIndexInput.java b/lucene/core/src/java21/org/apache/lucene/store/MemorySegmentIndexInput.java
index db6d19b0a7f..70fc1eb9488 100644
--- a/lucene/core/src/java21/org/apache/lucene/store/MemorySegmentIndexInput.java
+++ b/lucene/core/src/java21/org/apache/lucene/store/MemorySegmentIndexInput.java
@@ -570,12 +570,20 @@ abstract class MemorySegmentIndexInput extends IndexInput
if (NATIVE_ACCESS.isPresent() && advice != ReadAdvice.NORMAL) {
// No need to madvise with a normal advice, since it's the OS' default.
final NativeAccess nativeAccess = NATIVE_ACCESS.get();
- slice.advise(
- 0,
- slice.length,
- segment -> {
- nativeAccess.madvise(segment, advice);
- });
+ if (length >= nativeAccess.getPageSize()) {
+ // Only set the read advice if the inner file is large enough. Otherwise the cons are likely
+ // outweighing the pros as we're:
+ // - potentially overriding the advice of other files that share the same pages,
+ // - paying the cost of a madvise system call for little value.
+ // We could align inner files with the page size to avoid the first issue, but again the
+ // pros don't clearly overweigh the cons.
+ slice.advise(
+ 0,
+ slice.length,
+ segment -> {
+ nativeAccess.madvise(segment, advice);
+ });
+ }
}
return slice;
}
(lucene) 01/02: Better handle dynamic pruning when the leading clause has a single impact block. (#13904)
Posted by jp...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
jpountz pushed a commit to branch branch_10x
in repository https://gitbox.apache.org/repos/asf/lucene.git
commit 92c4f5122431674e9a46124ece97295fee812c57
Author: Adrien Grand <jp...@gmail.com>
AuthorDate: Tue Oct 15 16:17:20 2024 +0200
Better handle dynamic pruning when the leading clause has a single impact block. (#13904)
`BlockMaxConjunctionBulkScorer` only checks if it can early exit based on
impacts once per window, and windows are computed using impact blocks of the
leading clause. So this logic is defeated if the leading clause produces a
single block (e.g. `ConstantScoreQuery`). This commit addresses this problem by
artificially lowering the window size to contain ~128 docs of the leading
clause.
---
lucene/CHANGES.txt | 4 ++++
.../lucene/search/BlockMaxConjunctionBulkScorer.java | 17 ++++++++++++++---
2 files changed, 18 insertions(+), 3 deletions(-)
diff --git a/lucene/CHANGES.txt b/lucene/CHANGES.txt
index 6acd0414e07..13e2990a086 100644
--- a/lucene/CHANGES.txt
+++ b/lucene/CHANGES.txt
@@ -27,6 +27,10 @@ Optimizations
* GITHUB#13800: MaxScoreBulkScorer now recomputes scorer partitions when the
minimum competitive allows for a more favorable partitioning. (Adrien Grand)
+* GITHUB#13904: BlockMaxConjunctionBulkScorer can now early exit when the
+ leading clause has a single impact block (e.g. ConstantScoreQuery).
+ (Adrien Grand)
+
Bug Fixes
---------------------
* GITHUB#13832: Fixed an issue where the DefaultPassageFormatter.format method did not format passages as intended
diff --git a/lucene/core/src/java/org/apache/lucene/search/BlockMaxConjunctionBulkScorer.java b/lucene/core/src/java/org/apache/lucene/search/BlockMaxConjunctionBulkScorer.java
index 9f4531540a5..d7e939f4c13 100644
--- a/lucene/core/src/java/org/apache/lucene/search/BlockMaxConjunctionBulkScorer.java
+++ b/lucene/core/src/java/org/apache/lucene/search/BlockMaxConjunctionBulkScorer.java
@@ -85,9 +85,20 @@ final class BlockMaxConjunctionBulkScorer extends BulkScorer {
int windowMin = Math.max(lead1.docID(), min);
while (windowMin < max) {
- // Use impacts of the least costly scorer to compute windows
- // NOTE: windowMax is inclusive
- int windowMax = Math.min(scorers[0].advanceShallow(windowMin), max - 1);
+ // Use impacts of the least costly scorer to compute windows to keep the per-block overhead
+ // under control.
+ // NOTE: windowMax is inclusive.
+ int windowMax = scorer1.advanceShallow(windowMin);
+ if (windowMax == DocIdSetIterator.NO_MORE_DOCS) {
+ // If the query doesn't have impacts anymore, or has a single block for the whole doc ID
+ // space (e.g. ConstantScoreQuery), then we try to create a block that has ~128 docs of the
+ // leading clause. This gives us higher chances to exit early based on the maximum scores of
+ // other clauses.
+ long windowSize = 128L * maxDoc / Math.max(1, lead1.cost());
+ windowSize = Math.max(windowSize, 128L);
+ windowMax = (int) Math.min(Integer.MAX_VALUE, windowMin + windowSize);
+ }
+ windowMax = Math.min(windowMax, max - 1);
float maxWindowScore = Float.POSITIVE_INFINITY;
if (0 < scorable.minCompetitiveScore) {