You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@lucene.apache.org by GitBox <gi...@apache.org> on 2020/06/24 15:38:09 UTC
[GitHub] [lucene-solr] mayya-sharipova opened a new pull request #1610: LUCENE-9384: Backport for field sort optimization
mayya-sharipova opened a new pull request #1610:
URL: https://github.com/apache/lucene-solr/pull/1610
Backport for: LUCENE-9280: Collectors to skip noncompetitive documents (#1351)
Similar how scorers can update their iterators to skip non-competitive
documents, collectors and comparators should also provide and update
iterators that allow them to skip non-competive documents.
To enable sort optimization for numeric sort fields,
the following needs to be done:
1) the field should be indexed with both doc_values and points, that
must have the same field name and same data
2) SortField#setCanSkipNonCompetitiveDocs must be set
3) totalHitsThreshold should not be set to max value.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene-solr] mayya-sharipova commented on pull request #1610: LUCENE-9384: Backport for field sort optimization
Posted by GitBox <gi...@apache.org>.
mayya-sharipova commented on pull request #1610:
URL: https://github.com/apache/lucene-solr/pull/1610#issuecomment-663000392
@jimczi @jpountz Thanks for your feedback, I have addressed it:
- renamed `SortField#setSkipNonCompetitiveDocs` to `SortField#setCanUsePoints `
- remove `canUsePoints` field from `SortField#equals` and `hashCode`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene-solr] jimczi commented on a change in pull request #1610: LUCENE-9384: Backport for field sort optimization
Posted by GitBox <gi...@apache.org>.
jimczi commented on a change in pull request #1610:
URL: https://github.com/apache/lucene-solr/pull/1610#discussion_r445460474
##########
File path: lucene/core/src/java/org/apache/lucene/search/SortField.java
##########
@@ -394,6 +399,21 @@ public String toString() {
return buffer.toString();
}
+
+ /**
+ * For numeric sort fields, setting this field, indicates that
+ * the same numeric data has been indexed with two fields: doc values and points and
+ * that these fields have the same name.
+ * This allows to use sort optimization and skip non-competitive documents.
+ */
+ public void setSkipNonCompetitiveDocs() {
Review comment:
I wonder if we should make the option less ambiguous by naming it something like `setCanUsePoints` or something along those lines ? I am afraid that this could be misleading if we set this option on a sort field that use the _score or a keyword field for instance. We should maybe at least validate that the field is not `null` ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene-solr] mayya-sharipova merged pull request #1610: LUCENE-9384: Backport for field sort optimization
Posted by GitBox <gi...@apache.org>.
mayya-sharipova merged pull request #1610:
URL: https://github.com/apache/lucene-solr/pull/1610
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene-solr] mayya-sharipova commented on a change in pull request #1610: LUCENE-9384: Backport for field sort optimization
Posted by GitBox <gi...@apache.org>.
mayya-sharipova commented on a change in pull request #1610:
URL: https://github.com/apache/lucene-solr/pull/1610#discussion_r450234884
##########
File path: lucene/core/src/java/org/apache/lucene/search/SortField.java
##########
@@ -394,6 +399,21 @@ public String toString() {
return buffer.toString();
}
+
+ /**
+ * For numeric sort fields, setting this field, indicates that
+ * the same numeric data has been indexed with two fields: doc values and points and
+ * that these fields have the same name.
+ * This allows to use sort optimization and skip non-competitive documents.
+ */
+ public void setSkipNonCompetitiveDocs() {
Review comment:
@jpountz Do you have suggestions how we should expose sort optimization in 8.x? Are you ok with a proposal to have a method for `SortField::setCanUsePointsForSort` that when set `true` indicates that the same data is used for indexing points and docValues?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene-solr] jpountz commented on a change in pull request #1610: LUCENE-9384: Backport for field sort optimization
Posted by GitBox <gi...@apache.org>.
jpountz commented on a change in pull request #1610:
URL: https://github.com/apache/lucene-solr/pull/1610#discussion_r463156089
##########
File path: lucene/core/src/java/org/apache/lucene/search/SortField.java
##########
@@ -428,7 +428,6 @@ public boolean equals(Object o) {
&& other.reverse == this.reverse
&& Objects.equals(this.comparatorSource, other.comparatorSource)
&& Objects.equals(this.missingValue, other.missingValue)
- && other.skipNonCompetitiveDocs == this.skipNonCompetitiveDocs
Review comment:
Can you add a comment to say that we're not taking it into account on purpose? I foresee that someone will file a PR to add it after running a code analysis tool that would complain that not all fields are taken into account for equals otherwise.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene-solr] jpountz commented on a change in pull request #1610: LUCENE-9384: Backport for field sort optimization
Posted by GitBox <gi...@apache.org>.
jpountz commented on a change in pull request #1610:
URL: https://github.com/apache/lucene-solr/pull/1610#discussion_r454999807
##########
File path: lucene/core/src/java/org/apache/lucene/search/SortField.java
##########
@@ -394,6 +399,21 @@ public String toString() {
return buffer.toString();
}
+
+ /**
+ * For numeric sort fields, setting this field, indicates that
+ * the same numeric data has been indexed with two fields: doc values and points and
+ * that these fields have the same name.
+ * This allows to use sort optimization and skip non-competitive documents.
+ */
+ public void setSkipNonCompetitiveDocs() {
Review comment:
IMO `setCanUsePoints` is good enough since this is a "SortField" so it should be obvious that this is for sorting?
##########
File path: lucene/core/src/java/org/apache/lucene/search/SortField.java
##########
@@ -408,6 +428,7 @@ public boolean equals(Object o) {
&& other.reverse == this.reverse
&& Objects.equals(this.comparatorSource, other.comparatorSource)
&& Objects.equals(this.missingValue, other.missingValue)
+ && other.skipNonCompetitiveDocs == this.skipNonCompetitiveDocs
Review comment:
I wonder if we should actually take it into account for equals/hashcode as this could disable index sorting optimizations, which check for equality of the index-time and search-time SortField instances.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene-solr] mayya-sharipova commented on a change in pull request #1610: LUCENE-9384: Backport for field sort optimization
Posted by GitBox <gi...@apache.org>.
mayya-sharipova commented on a change in pull request #1610:
URL: https://github.com/apache/lucene-solr/pull/1610#discussion_r445737021
##########
File path: lucene/core/src/java/org/apache/lucene/search/SortField.java
##########
@@ -394,6 +399,21 @@ public String toString() {
return buffer.toString();
}
+
+ /**
+ * For numeric sort fields, setting this field, indicates that
+ * the same numeric data has been indexed with two fields: doc values and points and
+ * that these fields have the same name.
+ * This allows to use sort optimization and skip non-competitive documents.
+ */
+ public void setSkipNonCompetitiveDocs() {
Review comment:
How about `setCanUsePointsForSort`? I am also fine with `setCanUsePoints`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene-solr] mayya-sharipova commented on a change in pull request #1610: LUCENE-9384: Backport for field sort optimization
Posted by GitBox <gi...@apache.org>.
mayya-sharipova commented on a change in pull request #1610:
URL: https://github.com/apache/lucene-solr/pull/1610#discussion_r455321149
##########
File path: lucene/core/src/java/org/apache/lucene/search/SortField.java
##########
@@ -408,6 +428,7 @@ public boolean equals(Object o) {
&& other.reverse == this.reverse
&& Objects.equals(this.comparatorSource, other.comparatorSource)
&& Objects.equals(this.missingValue, other.missingValue)
+ && other.skipNonCompetitiveDocs == this.skipNonCompetitiveDocs
Review comment:
@jpountz Thank you for your feedback. Alternatively we can add `setCanUsePoints` parameter to `Sort` object instead of `SortField`? What do you think?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org