You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@lucene.apache.org by GitBox <gi...@apache.org> on 2021/10/06 09:37:18 UTC
[GitHub] [lucene] jpountz opened a new pull request #357: LUCENE-10153: Speed up BKDWriter using VarHandles.
jpountz opened a new pull request #357:
URL: https://github.com/apache/lucene/pull/357
See https://issues.apache.org/jira/browse/LUCENE-10153.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] jpountz merged pull request #357: LUCENE-10153: Speed up BKDWriter using VarHandles.
Posted by GitBox <gi...@apache.org>.
jpountz merged pull request #357:
URL: https://github.com/apache/lucene/pull/357
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] jpountz commented on pull request #357: LUCENE-10153: Speed up BKDWriter using VarHandles.
Posted by GitBox <gi...@apache.org>.
jpountz commented on pull request #357:
URL: https://github.com/apache/lucene/pull/357#issuecomment-935899999
I agree that improved readability is a nice side-effect. I don't know if this actually triggers object creations since we are returning references to existing methods, do you know @dweiss ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] rmuir commented on a change in pull request #357: LUCENE-10153: Speed up BKDWriter using VarHandles.
Posted by GitBox <gi...@apache.org>.
rmuir commented on a change in pull request #357:
URL: https://github.com/apache/lucene/pull/357#discussion_r723259603
##########
File path: lucene/core/src/java/org/apache/lucene/util/ArrayUtil.java
##########
@@ -694,4 +694,81 @@ public static int compareUnsigned4(byte[] a, int aOffset, byte[] b, int bOffset)
return Integer.compareUnsigned(
(int) BitUtil.VH_BE_INT.get(a, aOffset), (int) BitUtil.VH_BE_INT.get(b, bOffset));
}
+
+ /**
+ * Return a comparator that computes the common prefix length across the next {@code numBytes} of
+ * the provided arrays.
+ */
+ public static ByteArrayComparator getPrefixLengthComparator(int numBytes) {
Review comment:
Thanks! If we later need them for something outside of this package (e.g. a points query), then we can look at putting them in a more generic place.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] rmuir commented on a change in pull request #357: LUCENE-10153: Speed up BKDWriter using VarHandles.
Posted by GitBox <gi...@apache.org>.
rmuir commented on a change in pull request #357:
URL: https://github.com/apache/lucene/pull/357#discussion_r723176939
##########
File path: lucene/core/src/java/org/apache/lucene/util/ArrayUtil.java
##########
@@ -694,4 +694,81 @@ public static int compareUnsigned4(byte[] a, int aOffset, byte[] b, int bOffset)
return Integer.compareUnsigned(
(int) BitUtil.VH_BE_INT.get(a, aOffset), (int) BitUtil.VH_BE_INT.get(b, bOffset));
}
+
+ /**
+ * Return a comparator that computes the common prefix length across the next {@code numBytes} of
+ * the provided arrays.
+ */
+ public static ByteArrayComparator getPrefixLengthComparator(int numBytes) {
Review comment:
also, wouldnt need to be public this way. the current patch explodes the public API and exposes a LOT of surface area on arrayutil, but it is all unnecessary.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] jpountz commented on pull request #357: LUCENE-10153: Speed up BKDWriter using VarHandles.
Posted by GitBox <gi...@apache.org>.
jpountz commented on pull request #357:
URL: https://github.com/apache/lucene/pull/357#issuecomment-936016565
I like that BKDConfig today only records data (bytesPerDim, etc.) rather than logic (how to compare byte[] arrays), so I have a bias towards keeping things as they are today.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] jpountz commented on a change in pull request #357: LUCENE-10153: Speed up BKDWriter using VarHandles.
Posted by GitBox <gi...@apache.org>.
jpountz commented on a change in pull request #357:
URL: https://github.com/apache/lucene/pull/357#discussion_r723290327
##########
File path: lucene/core/src/java/org/apache/lucene/util/ArrayUtil.java
##########
@@ -694,4 +694,81 @@ public static int compareUnsigned4(byte[] a, int aOffset, byte[] b, int bOffset)
return Integer.compareUnsigned(
(int) BitUtil.VH_BE_INT.get(a, aOffset), (int) BitUtil.VH_BE_INT.get(b, bOffset));
}
+
+ /**
+ * Return a comparator that computes the common prefix length across the next {@code numBytes} of
+ * the provided arrays.
+ */
+ public static ByteArrayComparator getPrefixLengthComparator(int numBytes) {
Review comment:
+1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] rmuir commented on a change in pull request #357: LUCENE-10153: Speed up BKDWriter using VarHandles.
Posted by GitBox <gi...@apache.org>.
rmuir commented on a change in pull request #357:
URL: https://github.com/apache/lucene/pull/357#discussion_r723158701
##########
File path: lucene/core/src/java/org/apache/lucene/util/ArrayUtil.java
##########
@@ -694,4 +694,81 @@ public static int compareUnsigned4(byte[] a, int aOffset, byte[] b, int bOffset)
return Integer.compareUnsigned(
(int) BitUtil.VH_BE_INT.get(a, aOffset), (int) BitUtil.VH_BE_INT.get(b, bOffset));
}
+
+ /**
+ * Return a comparator that computes the common prefix length across the next {@code numBytes} of
+ * the provided arrays.
+ */
+ public static ByteArrayComparator getPrefixLengthComparator(int numBytes) {
Review comment:
should all this fixed-length stuff really be here, or simply in BKD? Seems very specific to BKD.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] jpountz commented on a change in pull request #357: LUCENE-10153: Speed up BKDWriter using VarHandles.
Posted by GitBox <gi...@apache.org>.
jpountz commented on a change in pull request #357:
URL: https://github.com/apache/lucene/pull/357#discussion_r723240957
##########
File path: lucene/core/src/java/org/apache/lucene/util/ArrayUtil.java
##########
@@ -694,4 +694,81 @@ public static int compareUnsigned4(byte[] a, int aOffset, byte[] b, int bOffset)
return Integer.compareUnsigned(
(int) BitUtil.VH_BE_INT.get(a, aOffset), (int) BitUtil.VH_BE_INT.get(b, bOffset));
}
+
+ /**
+ * Return a comparator that computes the common prefix length across the next {@code numBytes} of
+ * the provided arrays.
+ */
+ public static ByteArrayComparator getPrefixLengthComparator(int numBytes) {
Review comment:
This is a good point, I moved the new helpers to a pkg-private class.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] iverase commented on pull request #357: LUCENE-10153: Speed up BKDWriter using VarHandles.
Posted by GitBox <gi...@apache.org>.
iverase commented on pull request #357:
URL: https://github.com/apache/lucene/pull/357#issuecomment-935961662
I am not worry about object creation but because we always call it with the same parameter (config.bytesPerDim), it seems to belong to config.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org