You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@lucene.apache.org by "gf2121 (via GitHub)" <gi...@apache.org> on 2023/09/29 16:37:50 UTC
[GitHub] [lucene] gf2121 opened a new pull request, #12610: Improve fallback sorter for BKD
gf2121 opened a new pull request, #12610:
URL: https://github.com/apache/lucene/pull/12610
### Description
This PR proposes to use a more efficient way to compare bytes when RadixSorter fallback to MergeSorter.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Improve fallback sorter for BKD [lucene]
Posted by "jpountz (via GitHub)" <gi...@apache.org>.
jpountz commented on code in PR #12610:
URL: https://github.com/apache/lucene/pull/12610#discussion_r1342470366
##########
lucene/core/src/java/org/apache/lucene/util/bkd/MutablePointTreeReaderUtils.java:
##########
@@ -81,6 +86,40 @@ protected int byteAt(int i, int k) {
return (reader.getDocID(i) >>> Math.max(0, shift)) & 0xff;
}
}
+
+ @Override
+ protected Sorter getFallbackSorter(int k) {
+ return new InPlaceMergeSorter() {
+ @Override
+ protected int compare(int i, int j) {
+ if (k < config.packedBytesLength) {
+ reader.getValue(i, scratch1);
+ reader.getValue(j, scratch2);
+ int v =
+ Arrays.compareUnsigned(
+ scratch1.bytes,
+ scratch1.offset + k,
+ scratch1.offset + scratch1.length,
+ scratch2.bytes,
+ scratch2.offset + k,
+ scratch2.offset + scratch2.length);
+ if (v != 0) {
+ return v;
+ }
+ }
+ if (bitsPerDocId == 0) {
Review Comment:
can this ever be 0?
##########
lucene/core/src/java/org/apache/lucene/util/bkd/MutablePointTreeReaderUtils.java:
##########
@@ -81,6 +86,40 @@ protected int byteAt(int i, int k) {
return (reader.getDocID(i) >>> Math.max(0, shift)) & 0xff;
}
}
+
+ @Override
+ protected Sorter getFallbackSorter(int k) {
Review Comment:
I wonder if it would help to take advantage of `ArrayUtil#getUnsignedComparator`, e.g. something like below:
```java
int cmpStartOffset;
ByteArrayComparator packedBytesComparator;
if (k >= config.packedBytesLength) {
cmpStartOffset = config.packedBytesLength;
packedBytesComparator = (a, ai, b, bi) -> 0;
} else if (config.packedBytesLength >= 4 && config.packedBytesLength - k <= 4) {
cmpStartOffset = config.packedBytesLength - 4;
packedBytesComparator = ArrayUtil.getUnsignedComparator(4);
} else if (config.packedBytesLength >= 8 && config.packedBytesLength - k <= 8) {
cmpStartOffset = config.packedBytesLength - 8;
packedBytesComparator = ArrayUtil.getUnsignedComparator(8);
} else {
cmpStartOffset = k;
packedBytesComparator = ArrayUtil.getUnsignedComparator(config.packedBytesLength - k);
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Improve fallback sorter for BKD [lucene]
Posted by "gf2121 (via GitHub)" <gi...@apache.org>.
gf2121 commented on code in PR #12610:
URL: https://github.com/apache/lucene/pull/12610#discussion_r1346202698
##########
lucene/core/src/java/org/apache/lucene/util/bkd/MutablePointTreeReaderUtils.java:
##########
@@ -81,6 +86,40 @@ protected int byteAt(int i, int k) {
return (reader.getDocID(i) >>> Math.max(0, shift)) & 0xff;
}
}
+
+ @Override
+ protected Sorter getFallbackSorter(int k) {
+ return new InPlaceMergeSorter() {
+ @Override
+ protected int compare(int i, int j) {
+ if (k < config.packedBytesLength) {
+ reader.getValue(i, scratch1);
+ reader.getValue(j, scratch2);
+ int v =
+ Arrays.compareUnsigned(
+ scratch1.bytes,
+ scratch1.offset + k,
+ scratch1.offset + scratch1.length,
+ scratch2.bytes,
+ scratch2.offset + k,
+ scratch2.offset + scratch2.length);
+ if (v != 0) {
+ return v;
+ }
+ }
+ if (bitsPerDocId == 0) {
Review Comment:
Yes, see this [line](https://github.com/apache/lucene/blob/eb14a1a0a69757dcbda4829c4a160ecd1a5530ad/lucene/core/src/java/org/apache/lucene/util/bkd/MutablePointTreeReaderUtils.java#L59). We are avoiding doc id comparing with a stable sort if index sorting in not enabled :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Improve fallback sorter for BKD [lucene]
Posted by "gf2121 (via GitHub)" <gi...@apache.org>.
gf2121 commented on code in PR #12610:
URL: https://github.com/apache/lucene/pull/12610#discussion_r1346210779
##########
lucene/core/src/java/org/apache/lucene/util/bkd/MutablePointTreeReaderUtils.java:
##########
@@ -81,6 +86,40 @@ protected int byteAt(int i, int k) {
return (reader.getDocID(i) >>> Math.max(0, shift)) & 0xff;
}
}
+
+ @Override
+ protected Sorter getFallbackSorter(int k) {
Review Comment:
Thanks for the review and the great advice! As `Arrays#compareUnsigned` need to compare bytes one by one when length < 8, This way should speed things up.
However, the newest patch (as well as origin patch that using `Arrays#compareUnsigned`) shows a slower speed than `main` in my benchmark. I'm still digging to understand what happened. I'll change this PR to draft until figuring it out.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org