You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by "gianm (via GitHub)" <gi...@apache.org> on 2025/08/27 05:07:36 UTC
[PR] MSQ: Push down limits in ScanQueryFrameProcessor. (druid)
gianm opened a new pull request, #18441:
URL: https://github.com/apache/druid/pull/18441
When the scan query sorts by something that matches the underlying cursor, stop reading early.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] MSQ: Push down limits in ScanQueryFrameProcessor. (druid)
Posted by "gianm (via GitHub)" <gi...@apache.org>.
gianm merged PR #18441:
URL: https://github.com/apache/druid/pull/18441
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] MSQ: Push down limits in ScanQueryFrameProcessor. (druid)
Posted by "clintropolis (via GitHub)" <gi...@apache.org>.
clintropolis commented on code in PR #18441:
URL: https://github.com/apache/druid/pull/18441#discussion_r2326157160
##########
multi-stage-query/src/main/java/org/apache/druid/msq/querykit/scan/ScanQueryFrameProcessor.java:
##########
@@ -479,6 +522,16 @@ private long setNextCursor(
this.cursorCloser = cursorCloser;
this.segment = segment;
this.cursorOffset.reset();
+ this.cursorRowsRead = 0;
+
+ if (query.isLimited()
+ && ordering.size() >= query.getOrderBys().size()
+ && query.getOrderBys().equals(ordering.subList(0, query.getOrderBys().size()))) {
Review Comment:
oh right, i forgot how this worked :+1:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] MSQ: Push down limits in ScanQueryFrameProcessor. (druid)
Posted by "clintropolis (via GitHub)" <gi...@apache.org>.
clintropolis commented on code in PR #18441:
URL: https://github.com/apache/druid/pull/18441#discussion_r2326159228
##########
multi-stage-query/src/main/java/org/apache/druid/msq/querykit/scan/ScanQueryFrameProcessor.java:
##########
@@ -479,6 +522,16 @@ private long setNextCursor(
this.cursorCloser = cursorCloser;
this.segment = segment;
this.cursorOffset.reset();
+ this.cursorRowsRead = 0;
+
+ if (query.isLimited()
+ && ordering.size() >= query.getOrderBys().size()
+ && query.getOrderBys().equals(ordering.subList(0, query.getOrderBys().size()))) {
Review Comment:
well, i suppose its not very sophisticated right now in terms of how it reports ordering when i look at the code for queryable index holder,
```
final List<OrderBy> indexOrdering = index.getOrdering();
if (Cursors.preferDescendingTimeOrdering(cursorBuildSpec)
&& Cursors.getTimeOrdering(indexOrdering) == Order.ASCENDING) {
this.ordering = Cursors.descendingTimeOrder();
} else {
this.ordering = indexOrdering;
}
```
but probably good enough for now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] MSQ: Push down limits in ScanQueryFrameProcessor. (druid)
Posted by "gianm (via GitHub)" <gi...@apache.org>.
gianm commented on code in PR #18441:
URL: https://github.com/apache/druid/pull/18441#discussion_r2326145297
##########
multi-stage-query/src/main/java/org/apache/druid/msq/querykit/scan/ScanQueryFrameProcessor.java:
##########
@@ -479,6 +522,16 @@ private long setNextCursor(
this.cursorCloser = cursorCloser;
this.segment = segment;
this.cursorOffset.reset();
+ this.cursorRowsRead = 0;
+
+ if (query.isLimited()
+ && ordering.size() >= query.getOrderBys().size()
+ && query.getOrderBys().equals(ordering.subList(0, query.getOrderBys().size()))) {
Review Comment:
Wouldn't the holder for a reverse cursor report reverse order in its `getOrdering()`? In that case this way of checking would be OK, I think.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] MSQ: Push down limits in ScanQueryFrameProcessor. (druid)
Posted by "gianm (via GitHub)" <gi...@apache.org>.
gianm commented on code in PR #18441:
URL: https://github.com/apache/druid/pull/18441#discussion_r2326149274
##########
multi-stage-query/src/main/java/org/apache/druid/msq/querykit/scan/ScanQueryFrameProcessor.java:
##########
@@ -497,4 +550,19 @@ private ColumnSelectorFactory wrapColumnSelectorFactoryIfNeeded(final ColumnSele
}
return baseColumnSelectorFactory;
}
+
+ /**
+ * Returns the {@link ScanQuery#getScanRowsOffset()} plus {@link ScanQuery#getScanRowsLimit()}, or
+ * {@link Long#MAX_VALUE} if that addition would overflow.
+ */
+ private long getQueryOffsetPlusLimit()
+ {
+ final long sum = query.getScanRowsOffset() + query.getScanRowsLimit();
+ if (sum < 0) {
+ // Overflow
+ return Long.MAX_VALUE;
Review Comment:
I was thinking no, because `query.getScanRowsLimit()` is set to `Long.MAX_VALUE` to indicate "unlimited", and if there is an offset then the addition will overflow. In this case setting the result to `Long.MAX_VALUE` is the right thing to do because it still means "unlimited".
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] MSQ: Push down limits in ScanQueryFrameProcessor. (druid)
Posted by "clintropolis (via GitHub)" <gi...@apache.org>.
clintropolis commented on code in PR #18441:
URL: https://github.com/apache/druid/pull/18441#discussion_r2323311108
##########
multi-stage-query/src/main/java/org/apache/druid/msq/querykit/scan/ScanQueryFrameProcessor.java:
##########
@@ -497,4 +550,19 @@ private ColumnSelectorFactory wrapColumnSelectorFactoryIfNeeded(final ColumnSele
}
return baseColumnSelectorFactory;
}
+
+ /**
+ * Returns the {@link ScanQuery#getScanRowsOffset()} plus {@link ScanQuery#getScanRowsLimit()}, or
+ * {@link Long#MAX_VALUE} if that addition would overflow.
+ */
+ private long getQueryOffsetPlusLimit()
+ {
+ final long sum = query.getScanRowsOffset() + query.getScanRowsLimit();
+ if (sum < 0) {
+ // Overflow
+ return Long.MAX_VALUE;
Review Comment:
should this be a failure?
##########
multi-stage-query/src/main/java/org/apache/druid/msq/querykit/scan/ScanQueryFrameProcessor.java:
##########
@@ -479,6 +522,16 @@ private long setNextCursor(
this.cursorCloser = cursorCloser;
this.segment = segment;
this.cursorOffset.reset();
+ this.cursorRowsRead = 0;
+
+ if (query.isLimited()
+ && ordering.size() >= query.getOrderBys().size()
+ && query.getOrderBys().equals(ordering.subList(0, query.getOrderBys().size()))) {
Review Comment:
in the future it might be nice to have a 'same but reversed direction' check so that we could make reverse cursors match too
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org