You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@lucene.apache.org by GitBox <gi...@apache.org> on 2021/09/27 16:32:32 UTC
[GitHub] [lucene] jpountz opened a new pull request #322: LUCENE-10125: Speed up computation of exceptions.
jpountz opened a new pull request #322:
URL: https://github.com/apache/lucene/pull/322
Even though it was not the driver for the slowdown, in LUCENE-10125 we
identified that the move to PFOR had slowed down indexing significantly
for fields indexed with indexOptions=DOCS. This patch gets some of the
peformance back by using the `LongHeap` that we introduced for vectors
instead of sorting the same array over and over again.
On the NYC Taxis benchmark, I observed ~8% faster merges of postings
with this change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] jpountz merged pull request #322: LUCENE-10125: Speed up computation of exceptions.
Posted by GitBox <gi...@apache.org>.
jpountz merged pull request #322:
URL: https://github.com/apache/lucene/pull/322
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] uschindler commented on pull request #322: LUCENE-10125: Speed up computation of exceptions.
Posted by GitBox <gi...@apache.org>.
uschindler commented on pull request #322:
URL: https://github.com/apache/lucene/pull/322#issuecomment-928072988
Ah thats a different problem, sorry!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] msokolov commented on a change in pull request #322: LUCENE-10125: Speed up computation of exceptions.
Posted by GitBox <gi...@apache.org>.
msokolov commented on a change in pull request #322:
URL: https://github.com/apache/lucene/pull/322#discussion_r717756811
##########
File path: lucene/core/src/java/org/apache/lucene/codecs/lucene90/PForUtil.java
##########
@@ -59,24 +60,30 @@ static boolean allEqual(long[] l) {
/** Encode 128 integers from {@code longs} into {@code out}. */
void encode(long[] longs, DataOutput out) throws IOException {
// Determine the top MAX_EXCEPTIONS + 1 values
- final long[] top = new long[MAX_EXCEPTIONS + 1];
- Arrays.fill(top, -1L);
- for (int i = 0; i < ForUtil.BLOCK_SIZE; ++i) {
- if (longs[i] > top[0]) {
- top[0] = longs[i];
- Arrays.sort(
- top); // For only a small number of entries we just sort on every iteration instead of
- // maintaining a PQ
+ final LongHeap top = LongHeap.create(LongHeap.Order.MIN, MAX_EXCEPTIONS + 1);
Review comment:
exciting that this is finding new uses :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] jpountz merged pull request #322: LUCENE-10125: Speed up computation of exceptions.
Posted by GitBox <gi...@apache.org>.
jpountz merged pull request #322:
URL: https://github.com/apache/lucene/pull/322
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] jpountz commented on pull request #322: LUCENE-10125: Speed up computation of exceptions.
Posted by GitBox <gi...@apache.org>.
jpountz commented on pull request #322:
URL: https://github.com/apache/lucene/pull/322#issuecomment-929337085
@gsmiller I merged this change in order to be able to iterate quickly on the NYC Taxis benchmark, but if you have feedback about the change, I'll be happy to address it in a different PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] jpountz commented on a change in pull request #322: LUCENE-10125: Speed up computation of exceptions.
Posted by GitBox <gi...@apache.org>.
jpountz commented on a change in pull request #322:
URL: https://github.com/apache/lucene/pull/322#discussion_r717760225
##########
File path: lucene/core/src/java/org/apache/lucene/codecs/lucene90/PForUtil.java
##########
@@ -59,24 +60,30 @@ static boolean allEqual(long[] l) {
/** Encode 128 integers from {@code longs} into {@code out}. */
void encode(long[] longs, DataOutput out) throws IOException {
// Determine the top MAX_EXCEPTIONS + 1 values
- final long[] top = new long[MAX_EXCEPTIONS + 1];
- Arrays.fill(top, -1L);
- for (int i = 0; i < ForUtil.BLOCK_SIZE; ++i) {
- if (longs[i] > top[0]) {
- top[0] = longs[i];
- Arrays.sort(
- top); // For only a small number of entries we just sort on every iteration instead of
- // maintaining a PQ
+ final LongHeap top = LongHeap.create(LongHeap.Order.MIN, MAX_EXCEPTIONS + 1);
Review comment:
Haha, who doesn't need a heap?!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] jpountz commented on a change in pull request #322: LUCENE-10125: Speed up computation of exceptions.
Posted by GitBox <gi...@apache.org>.
jpountz commented on a change in pull request #322:
URL: https://github.com/apache/lucene/pull/322#discussion_r717760225
##########
File path: lucene/core/src/java/org/apache/lucene/codecs/lucene90/PForUtil.java
##########
@@ -59,24 +60,30 @@ static boolean allEqual(long[] l) {
/** Encode 128 integers from {@code longs} into {@code out}. */
void encode(long[] longs, DataOutput out) throws IOException {
// Determine the top MAX_EXCEPTIONS + 1 values
- final long[] top = new long[MAX_EXCEPTIONS + 1];
- Arrays.fill(top, -1L);
- for (int i = 0; i < ForUtil.BLOCK_SIZE; ++i) {
- if (longs[i] > top[0]) {
- top[0] = longs[i];
- Arrays.sort(
- top); // For only a small number of entries we just sort on every iteration instead of
- // maintaining a PQ
+ final LongHeap top = LongHeap.create(LongHeap.Order.MIN, MAX_EXCEPTIONS + 1);
Review comment:
Haha, who doesn't need a heap?!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] msokolov commented on a change in pull request #322: LUCENE-10125: Speed up computation of exceptions.
Posted by GitBox <gi...@apache.org>.
msokolov commented on a change in pull request #322:
URL: https://github.com/apache/lucene/pull/322#discussion_r717756811
##########
File path: lucene/core/src/java/org/apache/lucene/codecs/lucene90/PForUtil.java
##########
@@ -59,24 +60,30 @@ static boolean allEqual(long[] l) {
/** Encode 128 integers from {@code longs} into {@code out}. */
void encode(long[] longs, DataOutput out) throws IOException {
// Determine the top MAX_EXCEPTIONS + 1 values
- final long[] top = new long[MAX_EXCEPTIONS + 1];
- Arrays.fill(top, -1L);
- for (int i = 0; i < ForUtil.BLOCK_SIZE; ++i) {
- if (longs[i] > top[0]) {
- top[0] = longs[i];
- Arrays.sort(
- top); // For only a small number of entries we just sort on every iteration instead of
- // maintaining a PQ
+ final LongHeap top = LongHeap.create(LongHeap.Order.MIN, MAX_EXCEPTIONS + 1);
Review comment:
exciting that this is finding new uses :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] uschindler commented on pull request #322: LUCENE-10125: Speed up computation of exceptions.
Posted by GitBox <gi...@apache.org>.
uschindler commented on pull request #322:
URL: https://github.com/apache/lucene/pull/322#issuecomment-928072110
Should we close this one?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] jpountz commented on pull request #322: LUCENE-10125: Speed up computation of exceptions.
Posted by GitBox <gi...@apache.org>.
jpountz commented on pull request #322:
URL: https://github.com/apache/lucene/pull/322#issuecomment-929337085
@gsmiller I merged this change in order to be able to iterate quickly on the NYC Taxis benchmark, but if you have feedback about the change, I'll be happy to address it in a different PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org