You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@lucene.apache.org by GitBox <gi...@apache.org> on 2022/01/05 17:56:56 UTC
[GitHub] [lucene] gsmiller opened a new pull request #585: LUCENE-10356: Further optimize facet counting for single-valued TaxonomyFacetCounts
gsmiller opened a new pull request #585:
URL: https://github.com/apache/lucene/pull/585
# Description
Facet implementations have seen performance improvements by unwrapping singleton doc values for situations where the underlying field is actually single-valued. This change adds the optimization for counting in taxonomy faceting (bringing consistency with the countAll implementation along with SSDV faceting, etc.).
# Solution
Try unwrapping the `SortedNumericDocValues` as a `NumericDocValues` and use the single-valued field directly if possible.
# Tests
Existing tests cover this faceting implementation. Ran benchmarks as well. Saw very marginal improvements and no regressions.
# Checklist
Please review the following and check all that apply:
- [x] I have reviewed the guidelines for [How to Contribute](https://wiki.apache.org/lucene/HowToContribute) and my code conforms to the standards described there to the best of my ability.
- [x] I have created a Jira issue and added the issue ID to my pull request title.
- [x] I have given Lucene maintainers [access](https://help.github.com/en/articles/allowing-changes-to-a-pull-request-branch-created-from-a-fork) to contribute to my PR branch. (optional but recommended)
- [x] I have developed this patch against the `main` branch.
- [x] I have run `./gradlew check`.
- [ ] I have added tests for my changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] gsmiller commented on pull request #585: LUCENE-10356: Further optimize facet counting for single-valued TaxonomyFacetCounts
Posted by GitBox <gi...@apache.org>.
gsmiller commented on pull request #585:
URL: https://github.com/apache/lucene/pull/585#issuecomment-1005949811
Maybe a very small improvement with this change, but nothing particularly impactful. I wonder how often we actually trigger this case in our benchmarks? Certainly not as often as with the "count all" cases. I think it's worth making this change though for consistency (and I think there's a small improvement there anyway).
```
TaskQPS baseline StdDevQPS candidate StdDev Pct diff p-value
BrowseDayOfYearTaxoFacets 14.53 (10.8%) 14.08 (11.4%) -3.1% ( -22% - 21%) 0.378
BrowseDateTaxoFacets 14.48 (10.7%) 14.07 (11.2%) -2.8% ( -22% - 21%) 0.416
HighIntervalsOrdered 3.00 (5.2%) 2.94 (7.0%) -1.8% ( -13% - 10%) 0.359
OrNotHighHigh 1044.76 (4.5%) 1028.26 (3.0%) -1.6% ( -8% - 6%) 0.188
MedIntervalsOrdered 35.82 (4.8%) 35.28 (6.1%) -1.5% ( -11% - 9%) 0.384
OrNotHighMed 969.48 (2.7%) 957.92 (2.3%) -1.2% ( -5% - 3%) 0.129
OrHighLow 1228.88 (2.8%) 1215.30 (2.5%) -1.1% ( -6% - 4%) 0.188
BrowseMonthTaxoFacets 15.15 (10.3%) 14.99 (10.0%) -1.1% ( -19% - 21%) 0.740
OrNotHighLow 864.18 (2.4%) 856.27 (2.0%) -0.9% ( -5% - 3%) 0.187
LowIntervalsOrdered 14.43 (2.5%) 14.32 (3.3%) -0.7% ( -6% - 5%) 0.422
BrowseRandomLabelTaxoFacets 12.34 (9.2%) 12.26 (9.9%) -0.6% ( -18% - 20%) 0.834
OrHighNotHigh 877.19 (3.9%) 872.54 (3.8%) -0.5% ( -7% - 7%) 0.662
HighPhrase 106.42 (1.7%) 106.06 (1.4%) -0.3% ( -3% - 2%) 0.498
MedTerm 1930.85 (4.6%) 1924.69 (3.5%) -0.3% ( -8% - 8%) 0.806
HighTerm 1383.88 (4.6%) 1380.07 (4.1%) -0.3% ( -8% - 8%) 0.843
MedPhrase 744.32 (2.1%) 742.71 (2.4%) -0.2% ( -4% - 4%) 0.761
LowTerm 3065.99 (3.3%) 3060.87 (3.8%) -0.2% ( -7% - 7%) 0.882
LowPhrase 82.62 (1.6%) 82.51 (1.4%) -0.1% ( -3% - 2%) 0.768
OrHighNotLow 1094.83 (4.7%) 1094.10 (3.1%) -0.1% ( -7% - 8%) 0.958
MedSloppyPhrase 104.44 (2.5%) 104.41 (3.3%) -0.0% ( -5% - 5%) 0.980
OrHighMed 193.98 (4.4%) 193.96 (4.6%) -0.0% ( -8% - 9%) 0.994
Respell 65.50 (1.1%) 65.52 (1.2%) 0.0% ( -2% - 2%) 0.913
Fuzzy2 84.12 (1.1%) 84.16 (1.0%) 0.0% ( -2% - 2%) 0.901
AndHighLow 1163.35 (3.5%) 1163.95 (3.1%) 0.1% ( -6% - 6%) 0.961
BrowseRandomLabelSSDVFacets 9.34 (1.7%) 9.35 (2.9%) 0.1% ( -4% - 4%) 0.940
BrowseMonthSSDVFacets 12.91 (13.1%) 12.92 (13.1%) 0.1% ( -23% - 30%) 0.978
OrHighNotMed 1094.06 (3.9%) 1095.46 (3.1%) 0.1% ( -6% - 7%) 0.908
PKLookup 171.75 (3.0%) 172.12 (3.3%) 0.2% ( -5% - 6%) 0.829
Prefix3 365.59 (9.9%) 366.48 (9.1%) 0.2% ( -17% - 21%) 0.936
Fuzzy1 113.46 (1.2%) 113.82 (1.2%) 0.3% ( -2% - 2%) 0.390
Wildcard 38.65 (8.2%) 38.77 (7.8%) 0.3% ( -14% - 17%) 0.898
HighSloppyPhrase 11.81 (3.3%) 11.85 (3.9%) 0.3% ( -6% - 7%) 0.773
OrHighHigh 12.04 (4.0%) 12.10 (4.0%) 0.5% ( -7% - 8%) 0.713
LowSloppyPhrase 7.75 (3.6%) 7.79 (4.3%) 0.5% ( -7% - 8%) 0.703
LowSpanNear 29.78 (2.5%) 29.96 (2.4%) 0.6% ( -4% - 5%) 0.420
HighTermMonthSort 126.96 (16.0%) 127.81 (14.3%) 0.7% ( -25% - 36%) 0.889
HighSpanNear 12.43 (3.3%) 12.52 (3.2%) 0.7% ( -5% - 7%) 0.488
MedSpanNear 12.64 (2.9%) 12.73 (2.9%) 0.8% ( -4% - 6%) 0.402
AndHighMed 55.03 (5.2%) 55.48 (5.6%) 0.8% ( -9% - 12%) 0.631
TermDTSort 100.53 (15.0%) 101.68 (14.3%) 1.1% ( -24% - 35%) 0.805
AndHighHigh 38.00 (5.1%) 38.49 (6.3%) 1.3% ( -9% - 13%) 0.481
AndHighHighDayTaxoFacets 32.44 (2.4%) 32.89 (1.8%) 1.4% ( -2% - 5%) 0.039
BrowseDayOfYearSSDVFacets 12.17 (11.0%) 12.36 (13.2%) 1.5% ( -20% - 28%) 0.696
HighTermTitleBDVSort 115.13 (16.1%) 117.00 (15.2%) 1.6% ( -25% - 39%) 0.744
AndHighMedDayTaxoFacets 29.45 (2.5%) 29.96 (2.2%) 1.7% ( -2% - 6%) 0.024
MedTermDayTaxoFacets 39.09 (4.8%) 39.85 (6.7%) 1.9% ( -9% - 14%) 0.294
OrHighMedDayTaxoFacets 13.61 (5.6%) 13.91 (5.3%) 2.2% ( -8% - 13%) 0.199
HighTermDayOfYearSort 145.34 (15.1%) 149.53 (14.4%) 2.9% ( -23% - 38%) 0.536
IntNRQ 112.30 (12.5%) 116.40 (1.7%) 3.7% ( -9% - 20%) 0.197
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] gf2121 commented on a change in pull request #585: LUCENE-10356: Further optimize facet counting for single-valued TaxonomyFacetCounts
Posted by GitBox <gi...@apache.org>.
gf2121 commented on a change in pull request #585:
URL: https://github.com/apache/lucene/pull/585#discussion_r779279138
##########
File path: lucene/facet/src/java/org/apache/lucene/facet/taxonomy/FastTaxonomyFacetCounts.java
##########
@@ -71,17 +71,27 @@ public FastTaxonomyFacetCounts(
private final void count(List<MatchingDocs> matchingDocs) throws IOException {
for (MatchingDocs hits : matchingDocs) {
- SortedNumericDocValues dv = hits.context.reader().getSortedNumericDocValues(indexFieldName);
- if (dv == null) {
+ SortedNumericDocValues multiValued =
+ hits.context.reader().getSortedNumericDocValues(indexFieldName);
+ if (multiValued == null) {
continue;
}
+ NumericDocValues singleValued = DocValues.unwrapSingleton(multiValued);
+
+ DocIdSetIterator valuesIt = singleValued != null ? singleValued : multiValued;
DocIdSetIterator it =
- ConjunctionUtils.intersectIterators(Arrays.asList(hits.bits.iterator(), dv));
+ ConjunctionUtils.intersectIterators(Arrays.asList(hits.bits.iterator(), valuesIt));
- for (int doc = it.nextDoc(); doc != DocIdSetIterator.NO_MORE_DOCS; doc = it.nextDoc()) {
- for (int i = 0; i < dv.docValueCount(); i++) {
- increment((int) dv.nextValue());
+ if (singleValued != null) {
+ for (int doc = it.nextDoc(); doc != DocIdSetIterator.NO_MORE_DOCS; doc = it.nextDoc()) {
Review comment:
Maybe simplify this a bit with `while(it.nextDoc() != DocIdSetIterator.NO_MORE_DOCS)` as `doc` is not used in the loop body?
##########
File path: lucene/facet/src/java/org/apache/lucene/facet/taxonomy/FastTaxonomyFacetCounts.java
##########
@@ -91,31 +101,36 @@ private final void count(List<MatchingDocs> matchingDocs) throws IOException {
private final void countAll(IndexReader reader) throws IOException {
for (LeafReaderContext context : reader.leaves()) {
- SortedNumericDocValues dv = context.reader().getSortedNumericDocValues(indexFieldName);
- if (dv == null) {
+ SortedNumericDocValues multiValued =
Review comment:
+1
##########
File path: lucene/facet/src/java/org/apache/lucene/facet/taxonomy/FastTaxonomyFacetCounts.java
##########
@@ -71,17 +71,27 @@ public FastTaxonomyFacetCounts(
private final void count(List<MatchingDocs> matchingDocs) throws IOException {
for (MatchingDocs hits : matchingDocs) {
- SortedNumericDocValues dv = hits.context.reader().getSortedNumericDocValues(indexFieldName);
- if (dv == null) {
+ SortedNumericDocValues multiValued =
+ hits.context.reader().getSortedNumericDocValues(indexFieldName);
+ if (multiValued == null) {
continue;
}
+ NumericDocValues singleValued = DocValues.unwrapSingleton(multiValued);
+
+ DocIdSetIterator valuesIt = singleValued != null ? singleValued : multiValued;
DocIdSetIterator it =
- ConjunctionUtils.intersectIterators(Arrays.asList(hits.bits.iterator(), dv));
+ ConjunctionUtils.intersectIterators(Arrays.asList(hits.bits.iterator(), valuesIt));
- for (int doc = it.nextDoc(); doc != DocIdSetIterator.NO_MORE_DOCS; doc = it.nextDoc()) {
- for (int i = 0; i < dv.docValueCount(); i++) {
- increment((int) dv.nextValue());
+ if (singleValued != null) {
+ for (int doc = it.nextDoc(); doc != DocIdSetIterator.NO_MORE_DOCS; doc = it.nextDoc()) {
+ increment((int) singleValued.longValue());
+ }
+ } else {
+ for (int doc = it.nextDoc(); doc != DocIdSetIterator.NO_MORE_DOCS; doc = it.nextDoc()) {
Review comment:
We can use `while(it.nextDoc() != DocIdSetIterator.NO_MORE_DOCS)` here too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] gsmiller merged pull request #585: LUCENE-10356: Further optimize facet counting for single-valued TaxonomyFacetCounts
Posted by GitBox <gi...@apache.org>.
gsmiller merged pull request #585:
URL: https://github.com/apache/lucene/pull/585
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] gsmiller commented on pull request #585: LUCENE-10356: Further optimize facet counting for single-valued TaxonomyFacetCounts
Posted by GitBox <gi...@apache.org>.
gsmiller commented on pull request #585:
URL: https://github.com/apache/lucene/pull/585#issuecomment-1006842156
Thanks for the feedback @gf2121! I appreciate you taking a look since it appears you've dug into this faceting code pretty deeply.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] gsmiller commented on a change in pull request #585: LUCENE-10356: Further optimize facet counting for single-valued TaxonomyFacetCounts
Posted by GitBox <gi...@apache.org>.
gsmiller commented on a change in pull request #585:
URL: https://github.com/apache/lucene/pull/585#discussion_r779026781
##########
File path: lucene/facet/src/java/org/apache/lucene/facet/taxonomy/FastTaxonomyFacetCounts.java
##########
@@ -91,31 +101,36 @@ private final void count(List<MatchingDocs> matchingDocs) throws IOException {
private final void countAll(IndexReader reader) throws IOException {
for (LeafReaderContext context : reader.leaves()) {
- SortedNumericDocValues dv = context.reader().getSortedNumericDocValues(indexFieldName);
- if (dv == null) {
+ SortedNumericDocValues multiValued =
Review comment:
I took the liberty to suggest renaming variables here for consistency with other faceting implementations that do this and for slightly improved readability (IMO anyway).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org