You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@lucene.apache.org by "ebradshaw (via GitHub)" <gi...@apache.org> on 2025/05/04 23:13:57 UTC
[PR] Overrides rewrite in PointRangeQuery to optimize AllDocs/NoDocs cases [lucene]
ebradshaw opened a new pull request, #14609:
URL: https://github.com/apache/lucene/pull/14609
Overrides rewrite in PointRangeQuery range to handle cases where the query either fully contains or fully excludes all documents within the shard.
Often, particularly when using time based partitioning, range queries may overlap several indexes. Many of these indexes have timestamp values that are fully contained by the query, in which case the query can be rewritten to a MatchAllDocsQuery or a FieldExistsQuery. On the other hand, many indexes can be fully excluded if they're outside the requested time range, in which case the query can be rewritten to a MatchNoDocsQuery.
While a similar optimization exists at the leaf level in the createWeight function, rewriting at the shard level enables other optimizations downstream.
Please let me know if this has been ruled out in the past for other reasons or if the implementation misses anything. Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Overrides rewrite in PointRangeQuery to optimize AllDocs/NoDocs cases [lucene]
Posted by "jpountz (via GitHub)" <gi...@apache.org>.
jpountz commented on code in PR #14609:
URL: https://github.com/apache/lucene/pull/14609#discussion_r2081563312
##########
lucene/core/src/java/org/apache/lucene/search/PointRangeQuery.java:
##########
@@ -580,4 +534,114 @@ public final String toString(String field) {
* @return human readable value for debugging
*/
protected abstract String toString(int dimension, byte[] value);
+
+ @Override
+ public Query rewrite(IndexSearcher searcher) throws IOException {
+ IndexReader reader = searcher.getIndexReader();
+
+ for (LeafReaderContext leaf : reader.leaves()) {
+ checkValidPointValues(leaf.reader().getPointValues(field));
+ }
+
+ // fetch the global min/max packed values across all segments
+ byte[] globalMinPacked = PointValues.getMinPackedValue(reader, getField());
+ byte[] globalMaxPacked = PointValues.getMaxPackedValue(reader, getField());
+
+ if (globalMinPacked == null || globalMaxPacked == null) {
+ return super.rewrite(searcher);
Review Comment:
A null min or max packed value indicates an empty field, so you could rewrite to a `MatchNoDocsQuery` here as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Overrides rewrite in PointRangeQuery to optimize AllDocs/NoDocs cases [lucene]
Posted by "jpountz (via GitHub)" <gi...@apache.org>.
jpountz commented on PR #14609:
URL: https://github.com/apache/lucene/pull/14609#issuecomment-2852297311
+1 to do this
Feel free to ask me for a review on a PR if you'd live to give it a try.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Overrides rewrite in PointRangeQuery to optimize AllDocs/NoDocs cases [lucene]
Posted by "jpountz (via GitHub)" <gi...@apache.org>.
jpountz commented on code in PR #14609:
URL: https://github.com/apache/lucene/pull/14609#discussion_r2074166256
##########
lucene/core/src/java/org/apache/lucene/search/PointRangeQuery.java:
##########
@@ -45,8 +49,8 @@
* <p>For a single-dimensional field this query is a simple range query; in a multi-dimensional
* field it's a box shape.
*
- * @see PointValues
* @lucene.experimental
+ * @see PointValues
Review Comment:
I don't think we have a rule around it, but the lucene.experimental / lucene.internal tag is usually the last one, so let's not swap these two lines?
##########
lucene/core/src/java/org/apache/lucene/search/PointRangeQuery.java:
##########
@@ -580,4 +557,111 @@ public final String toString(String field) {
* @return human readable value for debugging
*/
protected abstract String toString(int dimension, byte[] value);
+
+ @Override
+ public Query rewrite(IndexSearcher searcher) throws IOException {
+ IndexReader reader = searcher.getIndexReader();
+
+ for (LeafReaderContext leaf : reader.leaves()) {
+ checkValidPointValues(leaf.reader().getPointValues(field));
+ }
+
+ // fetch the global min/max packed values across all segments
+ byte[] globalMinPacked = PointValues.getMinPackedValue(reader, getField());
+ byte[] globalMaxPacked = PointValues.getMaxPackedValue(reader, getField());
+ if (globalMinPacked == null || globalMaxPacked == null) {
+ return super.rewrite(searcher);
+ }
+
+ int dims = getNumDims();
+ int bytesPerDim = getBytesPerDim();
+ byte[] queryLow = getLowerPoint();
+ byte[] queryHigh = getUpperPoint();
+
+ int fullyContainedCount = 0;
+ int fullyExcludedCount = 0;
+
+ // Check each dimension individually
+ for (int dim = 0; dim < dims; dim++) {
+ int offset = dim * bytesPerDim;
+ BytesRef qLow = new BytesRef(queryLow, offset, bytesPerDim);
+ BytesRef qHigh = new BytesRef(queryHigh, offset, bytesPerDim);
+ BytesRef gMin = new BytesRef(globalMinPacked, offset, bytesPerDim);
+ BytesRef gMax = new BytesRef(globalMaxPacked, offset, bytesPerDim);
+
+ if (qLow.compareTo(gMin) <= 0 && qHigh.compareTo(gMax) >= 0) {
+ fullyContainedCount++;
+ } else if (qLow.compareTo(gMax) > 0 || qHigh.compareTo(gMin) < 0) {
+ fullyExcludedCount++;
+ }
+ }
+
+ if (fullyContainedCount == dims) {
+ if (canRewriteToMatchAllQuery(reader)) {
+ return new MatchAllDocsQuery();
+ } else if (canRewriteToFieldExistsQuery(reader)) {
+ return new FieldExistsQuery(field);
+ }
+ } else if (fullyExcludedCount == dims) {
+ return new MatchNoDocsQuery();
+ }
Review Comment:
I wonder if we can somehow reuse the existing `relate(byte[], byte[])` method?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Overrides rewrite in PointRangeQuery to optimize AllDocs/NoDocs cases [lucene]
Posted by "jainankitk (via GitHub)" <gi...@apache.org>.
jainankitk merged PR #14609:
URL: https://github.com/apache/lucene/pull/14609
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Overrides rewrite in PointRangeQuery to optimize AllDocs/NoDocs cases [lucene]
Posted by "ebradshaw (via GitHub)" <gi...@apache.org>.
ebradshaw commented on code in PR #14609:
URL: https://github.com/apache/lucene/pull/14609#discussion_r2080658986
##########
lucene/core/src/java/org/apache/lucene/search/PointRangeQuery.java:
##########
@@ -580,4 +557,111 @@ public final String toString(String field) {
* @return human readable value for debugging
*/
protected abstract String toString(int dimension, byte[] value);
+
+ @Override
+ public Query rewrite(IndexSearcher searcher) throws IOException {
+ IndexReader reader = searcher.getIndexReader();
+
+ for (LeafReaderContext leaf : reader.leaves()) {
+ checkValidPointValues(leaf.reader().getPointValues(field));
+ }
+
+ // fetch the global min/max packed values across all segments
+ byte[] globalMinPacked = PointValues.getMinPackedValue(reader, getField());
+ byte[] globalMaxPacked = PointValues.getMaxPackedValue(reader, getField());
+ if (globalMinPacked == null || globalMaxPacked == null) {
+ return super.rewrite(searcher);
+ }
+
+ int dims = getNumDims();
+ int bytesPerDim = getBytesPerDim();
+ byte[] queryLow = getLowerPoint();
+ byte[] queryHigh = getUpperPoint();
+
+ int fullyContainedCount = 0;
+ int fullyExcludedCount = 0;
+
+ // Check each dimension individually
+ for (int dim = 0; dim < dims; dim++) {
+ int offset = dim * bytesPerDim;
+ BytesRef qLow = new BytesRef(queryLow, offset, bytesPerDim);
+ BytesRef qHigh = new BytesRef(queryHigh, offset, bytesPerDim);
+ BytesRef gMin = new BytesRef(globalMinPacked, offset, bytesPerDim);
+ BytesRef gMax = new BytesRef(globalMaxPacked, offset, bytesPerDim);
+
+ if (qLow.compareTo(gMin) <= 0 && qHigh.compareTo(gMax) >= 0) {
+ fullyContainedCount++;
+ } else if (qLow.compareTo(gMax) > 0 || qHigh.compareTo(gMin) < 0) {
+ fullyExcludedCount++;
+ }
Review Comment:
Since we're using the relate method, we'll get the short circuit for excluded, but it'll still run through every contained check to determine inside vs crosses. Should be okay since it's just a rewrite call?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Overrides rewrite in PointRangeQuery to optimize AllDocs/NoDocs cases [lucene]
Posted by "ebradshaw (via GitHub)" <gi...@apache.org>.
ebradshaw commented on PR #14609:
URL: https://github.com/apache/lucene/pull/14609#issuecomment-2864750920
I think we're good now. Let me know if you guys see anything else, and thanks for the quick review!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Overrides rewrite in PointRangeQuery to optimize AllDocs/NoDocs cases [lucene]
Posted by "jainankitk (via GitHub)" <gi...@apache.org>.
jainankitk commented on code in PR #14609:
URL: https://github.com/apache/lucene/pull/14609#discussion_r2078631302
##########
lucene/core/src/java/org/apache/lucene/search/PointRangeQuery.java:
##########
@@ -580,4 +557,111 @@ public final String toString(String field) {
* @return human readable value for debugging
*/
protected abstract String toString(int dimension, byte[] value);
+
+ @Override
+ public Query rewrite(IndexSearcher searcher) throws IOException {
+ IndexReader reader = searcher.getIndexReader();
+
+ for (LeafReaderContext leaf : reader.leaves()) {
+ checkValidPointValues(leaf.reader().getPointValues(field));
+ }
+
+ // fetch the global min/max packed values across all segments
+ byte[] globalMinPacked = PointValues.getMinPackedValue(reader, getField());
+ byte[] globalMaxPacked = PointValues.getMaxPackedValue(reader, getField());
+ if (globalMinPacked == null || globalMaxPacked == null) {
+ return super.rewrite(searcher);
+ }
+
+ int dims = getNumDims();
+ int bytesPerDim = getBytesPerDim();
+ byte[] queryLow = getLowerPoint();
+ byte[] queryHigh = getUpperPoint();
+
+ int fullyContainedCount = 0;
+ int fullyExcludedCount = 0;
+
+ // Check each dimension individually
+ for (int dim = 0; dim < dims; dim++) {
+ int offset = dim * bytesPerDim;
+ BytesRef qLow = new BytesRef(queryLow, offset, bytesPerDim);
+ BytesRef qHigh = new BytesRef(queryHigh, offset, bytesPerDim);
+ BytesRef gMin = new BytesRef(globalMinPacked, offset, bytesPerDim);
+ BytesRef gMax = new BytesRef(globalMaxPacked, offset, bytesPerDim);
+
+ if (qLow.compareTo(gMin) <= 0 && qHigh.compareTo(gMax) >= 0) {
+ fullyContainedCount++;
+ } else if (qLow.compareTo(gMax) > 0 || qHigh.compareTo(gMin) < 0) {
+ fullyExcludedCount++;
+ }
Review Comment:
We can `return super.rewrite(searcher)` if any dimension is not fullyContained / Excluded?
##########
lucene/core/src/java/org/apache/lucene/search/PointRangeQuery.java:
##########
@@ -580,4 +557,111 @@ public final String toString(String field) {
* @return human readable value for debugging
*/
protected abstract String toString(int dimension, byte[] value);
+
+ @Override
+ public Query rewrite(IndexSearcher searcher) throws IOException {
+ IndexReader reader = searcher.getIndexReader();
+
+ for (LeafReaderContext leaf : reader.leaves()) {
+ checkValidPointValues(leaf.reader().getPointValues(field));
+ }
+
+ // fetch the global min/max packed values across all segments
+ byte[] globalMinPacked = PointValues.getMinPackedValue(reader, getField());
+ byte[] globalMaxPacked = PointValues.getMaxPackedValue(reader, getField());
+ if (globalMinPacked == null || globalMaxPacked == null) {
+ return super.rewrite(searcher);
+ }
+
+ int dims = getNumDims();
+ int bytesPerDim = getBytesPerDim();
+ byte[] queryLow = getLowerPoint();
+ byte[] queryHigh = getUpperPoint();
+
+ int fullyContainedCount = 0;
+ int fullyExcludedCount = 0;
+
+ // Check each dimension individually
+ for (int dim = 0; dim < dims; dim++) {
+ int offset = dim * bytesPerDim;
+ BytesRef qLow = new BytesRef(queryLow, offset, bytesPerDim);
+ BytesRef qHigh = new BytesRef(queryHigh, offset, bytesPerDim);
+ BytesRef gMin = new BytesRef(globalMinPacked, offset, bytesPerDim);
+ BytesRef gMax = new BytesRef(globalMaxPacked, offset, bytesPerDim);
+
+ if (qLow.compareTo(gMin) <= 0 && qHigh.compareTo(gMax) >= 0) {
+ fullyContainedCount++;
+ } else if (qLow.compareTo(gMax) > 0 || qHigh.compareTo(gMin) < 0) {
+ fullyExcludedCount++;
+ }
+ }
+
+ if (fullyContainedCount == dims) {
+ if (canRewriteToMatchAllQuery(reader)) {
+ return new MatchAllDocsQuery();
+ } else if (canRewriteToFieldExistsQuery(reader)) {
+ return new FieldExistsQuery(field);
+ }
+ } else if (fullyExcludedCount == dims) {
+ return new MatchNoDocsQuery();
+ }
Review Comment:
+1 to using `relate(byte[], byte[])`. Actually looking at `relate(byte[], byte[])` method, I realized that we don't need `fullyExcludedCount == dims` for rewriting into `MatchNoDocsQuery`. Any dimension being completely outside is sufficient condition for the query to be rewritten as `MatchNoDocsQuery`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Overrides rewrite in PointRangeQuery to optimize AllDocs/NoDocs cases [lucene]
Posted by "ebradshaw (via GitHub)" <gi...@apache.org>.
ebradshaw commented on code in PR #14609:
URL: https://github.com/apache/lucene/pull/14609#discussion_r2080657472
##########
lucene/core/src/java/org/apache/lucene/search/PointRangeQuery.java:
##########
@@ -580,4 +557,111 @@ public final String toString(String field) {
* @return human readable value for debugging
*/
protected abstract String toString(int dimension, byte[] value);
+
+ @Override
+ public Query rewrite(IndexSearcher searcher) throws IOException {
+ IndexReader reader = searcher.getIndexReader();
+
+ for (LeafReaderContext leaf : reader.leaves()) {
+ checkValidPointValues(leaf.reader().getPointValues(field));
+ }
+
+ // fetch the global min/max packed values across all segments
+ byte[] globalMinPacked = PointValues.getMinPackedValue(reader, getField());
+ byte[] globalMaxPacked = PointValues.getMaxPackedValue(reader, getField());
+ if (globalMinPacked == null || globalMaxPacked == null) {
+ return super.rewrite(searcher);
+ }
+
+ int dims = getNumDims();
+ int bytesPerDim = getBytesPerDim();
+ byte[] queryLow = getLowerPoint();
+ byte[] queryHigh = getUpperPoint();
+
+ int fullyContainedCount = 0;
+ int fullyExcludedCount = 0;
+
+ // Check each dimension individually
+ for (int dim = 0; dim < dims; dim++) {
+ int offset = dim * bytesPerDim;
+ BytesRef qLow = new BytesRef(queryLow, offset, bytesPerDim);
+ BytesRef qHigh = new BytesRef(queryHigh, offset, bytesPerDim);
+ BytesRef gMin = new BytesRef(globalMinPacked, offset, bytesPerDim);
+ BytesRef gMax = new BytesRef(globalMaxPacked, offset, bytesPerDim);
+
+ if (qLow.compareTo(gMin) <= 0 && qHigh.compareTo(gMax) >= 0) {
+ fullyContainedCount++;
+ } else if (qLow.compareTo(gMax) > 0 || qHigh.compareTo(gMin) < 0) {
+ fullyExcludedCount++;
+ }
+ }
+
+ if (fullyContainedCount == dims) {
+ if (canRewriteToMatchAllQuery(reader)) {
+ return new MatchAllDocsQuery();
+ } else if (canRewriteToFieldExistsQuery(reader)) {
+ return new FieldExistsQuery(field);
+ }
+ } else if (fullyExcludedCount == dims) {
+ return new MatchNoDocsQuery();
+ }
Review Comment:
Agreed. Missed the relate method on the first run but that logic is nice and clean.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Overrides rewrite in PointRangeQuery to optimize AllDocs/NoDocs cases [lucene]
Posted by "jainankitk (via GitHub)" <gi...@apache.org>.
jainankitk commented on code in PR #14609:
URL: https://github.com/apache/lucene/pull/14609#discussion_r2082377177
##########
lucene/core/src/java/org/apache/lucene/search/PointRangeQuery.java:
##########
@@ -580,4 +557,111 @@ public final String toString(String field) {
* @return human readable value for debugging
*/
protected abstract String toString(int dimension, byte[] value);
+
+ @Override
+ public Query rewrite(IndexSearcher searcher) throws IOException {
+ IndexReader reader = searcher.getIndexReader();
+
+ for (LeafReaderContext leaf : reader.leaves()) {
+ checkValidPointValues(leaf.reader().getPointValues(field));
+ }
+
+ // fetch the global min/max packed values across all segments
+ byte[] globalMinPacked = PointValues.getMinPackedValue(reader, getField());
+ byte[] globalMaxPacked = PointValues.getMaxPackedValue(reader, getField());
+ if (globalMinPacked == null || globalMaxPacked == null) {
+ return super.rewrite(searcher);
+ }
+
+ int dims = getNumDims();
+ int bytesPerDim = getBytesPerDim();
+ byte[] queryLow = getLowerPoint();
+ byte[] queryHigh = getUpperPoint();
+
+ int fullyContainedCount = 0;
+ int fullyExcludedCount = 0;
+
+ // Check each dimension individually
+ for (int dim = 0; dim < dims; dim++) {
+ int offset = dim * bytesPerDim;
+ BytesRef qLow = new BytesRef(queryLow, offset, bytesPerDim);
+ BytesRef qHigh = new BytesRef(queryHigh, offset, bytesPerDim);
+ BytesRef gMin = new BytesRef(globalMinPacked, offset, bytesPerDim);
+ BytesRef gMax = new BytesRef(globalMaxPacked, offset, bytesPerDim);
+
+ if (qLow.compareTo(gMin) <= 0 && qHigh.compareTo(gMax) >= 0) {
+ fullyContainedCount++;
+ } else if (qLow.compareTo(gMax) > 0 || qHigh.compareTo(gMin) < 0) {
+ fullyExcludedCount++;
+ }
Review Comment:
Yeah, as you said this is only for rewrite and logic is much cleaner by reusing the relate method
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Overrides rewrite in PointRangeQuery to optimize AllDocs/NoDocs cases [lucene]
Posted by "jainankitk (via GitHub)" <gi...@apache.org>.
jainankitk commented on PR #14609:
URL: https://github.com/apache/lucene/pull/14609#issuecomment-2868309175
Thanks @ebradshaw for this change, and congratulations on your first contribution in lucene! :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org