You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@lucene.apache.org by GitBox <gi...@apache.org> on 2020/05/11 05:52:20 UTC
[GitHub] [lucene-solr] iverase opened a new pull request #1503: LUCENE-9368: SIMD-based decoding of BKD docIds
iverase opened a new pull request #1503:
URL: https://github.com/apache/lucene-solr/pull/1503
This commit reuse the decoding technique used for posting to encode BKD docIds. One if the biggest differences is that in this case we want to expand our integers directly into an integer arrays, or feed the decoding values directly to the `IntersectVisitor`.
In order to do that we reorder the incoming docIds arrays so we can decode them later on in order.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene-solr] iverase commented on pull request #1503: LUCENE-9368: SIMD-based decoding of BKD docIds
Posted by GitBox <gi...@apache.org>.
iverase commented on pull request #1503:
URL: https://github.com/apache/lucene-solr/pull/1503#issuecomment-626557468
I have run benchmarks for this change.
For points:
|Approach|Index time Dev (sec)|Index time Base (sec)|diff|Force merge time Dev (sec)|Force merge time (sec) Base|diff|Index size Dev (GB)|Index size Base (GB)|diff|Reader heap Dev (MB)|Reader heap (MB) Base|diff|
|--------|:------|:------|:------|:------|:------|:------|:------|:------|:------|:------|:------|:------|
|points|109.6s|113.2s|-3%|0.0s|0.0s| 0%|0.49|0.50|-3%|0.04|0.04| 0%|
|geo3d|139.4s|142.6s|-2%|0.0s|0.0s| 0%|0.69|0.71|-2%|0.04|0.04| 0%|
|shapes|177.9s|180.1s|-1%|0.0s|0.0s| 0%|1.25|1.26|-1%|0.04|0.04| 0%|
|Approach|Shape|M hits/sec Dev |M hits/sec Base|diff|QPS Dev |QPS Base|diff |Hit count Dev |Hit count Base |diff|
|--------|:------|:------|:------|:------|:------|:------|:------|:------|:------|:------|
|points|polyRussia|17.78|17.39| 2%|5.07|4.96| 2%|3508846|3508846| 0%|
|points|polyMedium|9.65|9.67|-0%|118.27|118.47|-0%|2693559|2693559| 0%|
|points|poly 10|86.77|84.09| 3%|54.87|53.18| 3%|355809475|355809475| 0%|
|points|box|83.58|80.52| 4%|85.05|81.93| 4%|221118844|221118844| 0%|
|points|distance|82.08|79.08| 4%|48.22|46.46| 4%|382961957|382961957| 0%|
|points|nearest 10|0.03|0.03|-1%|2842.06|2867.86|-1%|60844404|60844404| 0%|
|points|sort|36.51|37.33|-2%|37.15|37.98|-2%|221118844|221118844| 0%|
|geo3d|polyRussia|0.66|0.66| 0%|0.19|0.19| 0%|3508671|3508671| 0%|
|geo3d|polyMedium|0.72|0.72| 0%|8.85|8.85| 0%|2693541|2693541| 0%|
|geo3d|poly 10|54.41|52.58| 3%|34.40|33.24| 3%|355855227|355855227| 0%|
|geo3d|box|59.93|58.04| 3%|60.98|59.06| 3%|221118844|221118844| 0%|
|geo3d|distance|76.13|72.90| 4%|44.68|42.79| 4%|383371904|383371904| 0%|
|shapes|polyRussia|9.84|9.97|-1%|2.80|2.84|-1%|3508846|3508846| 0%|
|shapes|polyMedium|3.40|3.34| 2%|41.68|40.93| 2%|2693559|2693559| 0%|
|shapes|poly 10|43.06|42.42| 2%|27.23|26.82| 2%|355809475|355809475| 0%|
|shapes|box|42.94|42.22| 2%|43.70|42.96| 2%|221118844|221118844| 0%|
|shapes|distance|45.22|43.85| 3%|26.57|25.77| 3%|382961957|382961957| 0%|
For shapes:
|Approach|Index time Dev (sec)|Index time Base (sec)|diff|Force merge time Dev (sec)|Force merge time (sec) Base|diff|Index size Dev (GB)|Index size Base (GB)|diff|Reader heap Dev (MB)|Reader heap (MB) Base|diff|
|--------|:------|:------|:------|:------|:------|:------|:------|:------|:------|:------|:------|:------|
|shapes|456.5s|459.6s|-1%|0.0s|0.0s| 0%|2.17|2.24|-3%|0.01|0.01| 0%|
|Shape|operation|M hits/sec Dev |M hits/sec Base|diff|QPS Dev |QPS Base|diff |Hit count Dev |Hit count Base |diff|
|--------|:------|:------|:------|:------|:------|:------|:------|:------|:------|:------|
|point|intersects|0.00|0.00| 1%|376.77|373.16| 1%|2644|2644| 0%|
|box|intersects|7.39|6.80| 9%|50.25|46.23| 9%|33081264|33081264| 0%|
|distance|intersects|6.40|5.85| 9%|22.46|20.55| 9%|64062400|64062400| 0%|
|poly 10|intersects|5.17|4.93| 5%|19.71|18.79| 5%|59064569|59064569| 0%|
|polyMedium|intersects|0.37|0.35| 5%|23.24|22.13| 5%|528812|528812| 0%|
|polyRussia|intersects|1.07|1.08|-1%|4.37|4.42|-1%|244848|244848| 0%|
|point|contains|0.00|0.00| 5%|364.09|347.95| 5%|2644|2644| 0%|
|box|contains|0.00|0.00| 9%|46.46|42.74| 9%|484|484| 0%|
|distance|contains|0.00|0.00| 9%|22.72|20.89| 9%|406|406| 0%|Merge
|poly 10|contains|0.00|0.00| 4%|18.93|18.12| 4%|402|402| 0%|
|polyMedium|contains|0.00|0.00| 1%|17.26|17.07| 1%|147|147| 0%|
|point|within|0.00|0.00| 0%|416.16|389.28| 7%|0|0| 0%|
|box|within|0.54|0.50| 9%|3.71|3.41| 9%|32911251|32911251| 0%|
|distance|within|0.98|0.92| 6%|3.44|3.25| 6%|63868270|63868270| 0%|
|poly 10|within|0.87|0.84| 4%|3.34|3.20| 4%|58873224|58873224| 0%|
|polyMedium|within|0.05|0.05| 2%|3.37|3.30| 2%|522739|522739| 0%|
|polyRussia|within|0.67|0.67| 0%|2.75|2.74| 0%|244661|244661| 0%|
|point|disjoint|268.62|266.87| 1%|20.40|20.27| 1%|2962178156|2962178156| 0%|
|box|disjoint|200.03|193.95| 3%|15.37|14.90| 3%|2929099536|2929099536| 0%|
|distance|disjoint|143.01|139.55| 2%|11.10|10.83| 2%|2898118400|2898118400| 0%|
|poly 10|disjoint|134.23|131.60| 2%|10.40|10.20| 2%|2903116231|2903116231| 0%|
|polyMedium|disjoint|144.73|145.44|-0%|11.01|11.06|-0%|433924372|433924372| 0%|
|polyRussia|disjoint|50.95|51.76|-2%|3.94|4.01|-2%|12920400|12920400| 0%|
All in all we get slightly smaller index and slightly faster queries.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene-solr] iverase closed pull request #1503: LUCENE-9368: SIMD-based decoding of BKD docIds
Posted by GitBox <gi...@apache.org>.
iverase closed pull request #1503:
URL: https://github.com/apache/lucene-solr/pull/1503
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene-solr] iverase commented on pull request #1503: LUCENE-9368: SIMD-based decoding of BKD docIds
Posted by GitBox <gi...@apache.org>.
iverase commented on pull request #1503:
URL: https://github.com/apache/lucene-solr/pull/1503#issuecomment-634558069
I am going to close this in favour of https://github.com/apache/lucene-solr/pull/1538 as it less intrusive and performance test shows even better behaviour.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org