You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by "kgyrtkirk (via GitHub)" <gi...@apache.org> on 2023/10/24 12:36:46 UTC
[PR] Process pure ordering changes with windowing operators (druid)
kgyrtkirk opened a new pull request, #15241:
URL: https://github.com/apache/druid/pull/15241
* adds a new query build path: `DruidQuery#toScanAndSortQuery` which:
* builds a `ScanQuery` without considering the current `ordering`
* builds an operator to execute the sort
* fixes a `null` string to `"null"` literal string conversion in the frame serializer codes
* fixes some `DrillWindowQueryTest` cases
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Process pure ordering changes with windowing operators (druid)
Posted by "kgyrtkirk (via GitHub)" <gi...@apache.org>.
kgyrtkirk commented on code in PR #15241:
URL: https://github.com/apache/druid/pull/15241#discussion_r1374529038
##########
processing/src/main/java/org/apache/druid/query/operator/NaiveSortOperator.java:
##########
@@ -33,11 +34,11 @@
public class NaiveSortOperator implements Operator
{
private final Operator child;
- private final ArrayList<ColumnWithDirection> sortColumns;
+ private final List<ColumnWithDirection> sortColumns;
Review Comment:
* I think `ArrayList` have spread to a lot of places when these classes were created
* its harder to test with `ArrayList` types on the interfaces...I didn't wanted to go down further; I think wrapping stuff into the `ArrayList` when the sorter is created should be rare enough
* intellij checks also encourages to use `Collections.singletonList` and `emptyList` which are not `ArrayLists` either....
would you like to take an alternate approach?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Process pure ordering changes with windowing operators (druid)
Posted by "kgyrtkirk (via GitHub)" <gi...@apache.org>.
kgyrtkirk commented on code in PR #15241:
URL: https://github.com/apache/druid/pull/15241#discussion_r1375189099
##########
sql/src/main/java/org/apache/druid/sql/calcite/rel/DruidOuterQueryRel.java:
##########
@@ -47,7 +47,7 @@
*/
public class DruidOuterQueryRel extends DruidRel<DruidOuterQueryRel>
{
- private static final TableDataSource DUMMY_DATA_SOURCE = new TableDataSource("__subquery__");
+ public static final TableDataSource DUMMY_DATA_SOURCE = new TableDataSource("__subquery__");
Review Comment:
yes - totally agree that is not the best like this...
I think its rather odd to pass this constant [here](https://github.com/apache/druid/blob/737947754dfed23e649d980e2fc71b33ecb6e479/sql/src/main/java/org/apache/druid/sql/calcite/rel/DruidOuterQueryRel.java#L123) as its being used to show the internal parts that this outer query has a "TableDataSource"...
I think we should probably
* remove that `?:`:
* change the `DUMMY_QUERY_DATA_SOURCE` to not look like a `ScanQuery` - instead some opaque `query`; so that it can't interfere with things
* possibly reconsider the `instanceof` at [DruidQuery#computeQuery](https://github.com/apache/druid/blob/737947754dfed23e649d980e2fc71b33ecb6e479/sql/src/main/java/org/apache/druid/sql/calcite/rel/DruidQuery.java#L981) as when there is no `window` because of that `?:` it essentially never becomes true (during early stages of planning) because it passes a non-query datasource ....
...but I can try the `isConcreate()` impl approach for now - but if that breaks stuff; I would probably go back to the current reference checking approach - and submit the above changes in a followup PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Process pure ordering changes with windowing operators (druid)
Posted by "abhishekagarwal87 (via GitHub)" <gi...@apache.org>.
abhishekagarwal87 merged PR #15241:
URL: https://github.com/apache/druid/pull/15241
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Process pure ordering changes with windowing operators (druid)
Posted by "kgyrtkirk (via GitHub)" <gi...@apache.org>.
kgyrtkirk commented on code in PR #15241:
URL: https://github.com/apache/druid/pull/15241#discussion_r1374077304
##########
processing/src/main/java/org/apache/druid/query/operator/OffsetLimit.java:
##########
@@ -0,0 +1,139 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.query.operator;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.google.common.base.Preconditions;
+import org.apache.druid.error.DruidException;
+import java.util.Objects;
+
+public class OffsetLimit
Review Comment:
yes; we should consolidate the 2 classes - I was not able to use it and move it right away; it would have been a refactor step which I wanted to avoid as it would touch some places of the planner as well - so I would rather put it into a separate PR; in which only that's happening.
Since we have an `Operator` interface; I think it might be a bit confusing to add that keyword to this class's name - I think after we will only have one it might be already better.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Process pure ordering changes with windowing operators (druid)
Posted by "kgyrtkirk (via GitHub)" <gi...@apache.org>.
kgyrtkirk commented on code in PR #15241:
URL: https://github.com/apache/druid/pull/15241#discussion_r1375186285
##########
processing/src/main/java/org/apache/druid/query/scan/ScanQueryQueryToolChest.java:
##########
@@ -207,6 +208,19 @@ public RowSignature resultArraySignature(final ScanQuery query)
}
}
+ private ColumnType getDataSoruceColumnType(DataSource dataSource, String columnName)
Review Comment:
jeez; I make so many typos lately that I should
probably there are some plugins to warn about issues like this :D
I see [this](https://github.com/check-spelling/check-spelling) ; but one which could be integrated into the build would be probably even better...
fixed it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Process pure ordering changes with windowing operators (druid)
Posted by "abhishekagarwal87 (via GitHub)" <gi...@apache.org>.
abhishekagarwal87 commented on code in PR #15241:
URL: https://github.com/apache/druid/pull/15241#discussion_r1375415167
##########
sql/src/main/java/org/apache/druid/sql/calcite/rel/DruidOuterQueryRel.java:
##########
@@ -47,7 +47,14 @@
*/
public class DruidOuterQueryRel extends DruidRel<DruidOuterQueryRel>
{
- private static final TableDataSource DUMMY_DATA_SOURCE = new TableDataSource("__subquery__");
+ private static final TableDataSource DUMMY_DATA_SOURCE = new TableDataSource("__subquery__")
+ {
+ @Override
+ public boolean isConcrete()
Review Comment:
should be helpful to have some comments associated with this override.
##########
processing/src/main/java/org/apache/druid/query/operator/NaiveSortOperator.java:
##########
@@ -33,11 +34,11 @@
public class NaiveSortOperator implements Operator
{
private final Operator child;
- private final ArrayList<ColumnWithDirection> sortColumns;
+ private final List<ColumnWithDirection> sortColumns;
Review Comment:
No. makes sense though that makes me wonder why not just do it for NaiveSorter and its implementations too. So you don't have to create an ArrayList at line 61
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Process pure ordering changes with windowing operators (druid)
Posted by "abhishekagarwal87 (via GitHub)" <gi...@apache.org>.
abhishekagarwal87 commented on code in PR #15241:
URL: https://github.com/apache/druid/pull/15241#discussion_r1374283108
##########
processing/src/main/java/org/apache/druid/query/operator/NaiveSortOperator.java:
##########
@@ -33,11 +34,11 @@
public class NaiveSortOperator implements Operator
{
private final Operator child;
- private final ArrayList<ColumnWithDirection> sortColumns;
+ private final List<ColumnWithDirection> sortColumns;
Review Comment:
whats this change for?
##########
processing/src/main/java/org/apache/druid/query/rowsandcols/LazilyDecoratedRowsAndColumns.java:
##########
@@ -238,7 +239,8 @@ private Pair<byte[], RowSignature> materializeStorageAdapter(StorageAdapter as)
throw new ISE("accumulated[%s] non-null, why did we get multiple cursors?", accumulated);
}
- int theLimit = limit == -1 ? Integer.MAX_VALUE : limit;
+ long offsetRemaining = limit.getOffset();
Review Comment:
nit - variable name is confusing. offset is fixed :)
##########
sql/src/main/java/org/apache/druid/sql/calcite/rel/DruidQuery.java:
##########
@@ -1473,13 +1481,78 @@ private WindowOperatorQuery toWindowQuery()
);
}
+ /**
+ * Create an OperatorQuery which runs an order on top of a scan.
+ */
+ @Nullable
+ private WindowOperatorQuery toScanAndSortQuery()
+ {
+ if (sorting == null
+ || sorting.getOrderBys().isEmpty()
+ || sorting.getProjection() != null) {
+ return null;
+ }
+
+ ScanQuery scan = toScanQuery(false);
+ if (scan == null) {
+ return null;
+ }
+
+ // Reject cases which would sort the datasource directly
+ if (dataSource != DruidOuterQueryRel.DUMMY_DATA_SOURCE && dataSource.isConcrete()) {
+ List<String> orderByColumnNames = sorting.getOrderBys()
+ .stream().map(OrderByColumnSpec::getDimension)
+ .collect(Collectors.toList());
+ plannerContext.setPlanningError(
+ "SQL query requires order by non-time column [%s], which is not supported.",
Review Comment:
```suggestion
"SQL query requires ordering a table by non-time column [%s], which is not supported.",
```
##########
processing/src/main/java/org/apache/druid/query/operator/OffsetLimit.java:
##########
@@ -0,0 +1,139 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.query.operator;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.google.common.base.Preconditions;
+import org.apache.druid.error.DruidException;
+import java.util.Objects;
+
+public class OffsetLimit
+{
+ protected final long offset;
+ protected final long limit;
+
+ public static final OffsetLimit NONE = new OffsetLimit(0, -1);
+
+ @JsonCreator
+ public OffsetLimit(
+ @JsonProperty("offset") long offset,
+ @JsonProperty("limit") long limit)
+ {
+ Preconditions.checkArgument(offset >= 0, "offset >= 0");
+ this.offset = offset;
+ this.limit = limit < 0 ? -1 : limit;
+ }
+
+ @JsonProperty("offset")
+ public long getOffset()
+ {
+ return offset;
+ }
+
+ @JsonProperty("limit")
+ public long getLimit()
+ {
+ return limit;
+ }
+
+ public boolean isPresent()
+ {
+ return hasOffset() || hasLimit();
+ }
+
+ public boolean hasOffset()
+ {
+ return offset > 0;
+ }
+
+ public boolean hasLimit()
+ {
+ return limit >= 0;
+ }
+
+ public static OffsetLimit limit(int limit2)
+ {
+ return new OffsetLimit(0, limit2);
+ }
+
+ public long getLimitOrMax()
+ {
+ if (limit < 0) {
+ return Long.MAX_VALUE;
+ } else {
+ return limit;
+ }
+ }
+
+ @Override
+ public boolean equals(Object o)
+ {
+ if (this == o) {
+ return true;
+ }
+ if (!(o instanceof OffsetLimit)) {
+ return false;
+ }
+ OffsetLimit that = (OffsetLimit) o;
+ return limit == that.limit && offset == that.offset;
+ }
+
+ @Override
+ public int hashCode()
+ {
+ return Objects.hash(limit, offset);
+ }
+
+ @Override
+ public String toString()
+ {
+ return "OffsetLimit{" +
+ "offset=" + offset +
+ ", limit=" + limit +
+ '}';
+ }
+
+ public long getFetchFromIndex(long maxIndex)
+ {
+ if (maxIndex <= offset) {
+ return 0;
+ }
+ return offset;
+ }
+
+ public long getFetchToIndex(long maxIndex)
+ {
+ if (maxIndex <= offset) {
+ return 0;
+ }
+ if (hasLimit()) {
+ long toIndex = limit + offset;
+ if (limit > Long.MAX_VALUE - offset) {
+ throw DruidException.defensive(
+ "Cannot compute toIndex due to overflow [%s]",
+ this);
Review Comment:
Shouldn't this be checked in the constructor itself? Pretty wild if we hit this one. 😄
##########
processing/src/main/java/org/apache/druid/query/operator/OffsetLimit.java:
##########
@@ -0,0 +1,139 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.query.operator;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.google.common.base.Preconditions;
+import org.apache.druid.error.DruidException;
+import java.util.Objects;
+
+public class OffsetLimit
+{
+ protected final long offset;
+ protected final long limit;
+
+ public static final OffsetLimit NONE = new OffsetLimit(0, -1);
+
+ @JsonCreator
+ public OffsetLimit(
+ @JsonProperty("offset") long offset,
+ @JsonProperty("limit") long limit)
+ {
+ Preconditions.checkArgument(offset >= 0, "offset >= 0");
+ this.offset = offset;
+ this.limit = limit < 0 ? -1 : limit;
+ }
+
+ @JsonProperty("offset")
+ public long getOffset()
+ {
+ return offset;
+ }
+
+ @JsonProperty("limit")
+ public long getLimit()
+ {
+ return limit;
+ }
+
+ public boolean isPresent()
+ {
+ return hasOffset() || hasLimit();
+ }
+
+ public boolean hasOffset()
+ {
+ return offset > 0;
+ }
+
+ public boolean hasLimit()
+ {
+ return limit >= 0;
+ }
+
+ public static OffsetLimit limit(int limit2)
+ {
+ return new OffsetLimit(0, limit2);
+ }
+
+ public long getLimitOrMax()
+ {
+ if (limit < 0) {
+ return Long.MAX_VALUE;
+ } else {
+ return limit;
+ }
+ }
+
+ @Override
+ public boolean equals(Object o)
+ {
+ if (this == o) {
+ return true;
+ }
+ if (!(o instanceof OffsetLimit)) {
+ return false;
+ }
+ OffsetLimit that = (OffsetLimit) o;
+ return limit == that.limit && offset == that.offset;
+ }
+
+ @Override
+ public int hashCode()
+ {
+ return Objects.hash(limit, offset);
+ }
+
+ @Override
+ public String toString()
+ {
+ return "OffsetLimit{" +
+ "offset=" + offset +
+ ", limit=" + limit +
+ '}';
+ }
+
+ public long getFetchFromIndex(long maxIndex)
+ {
+ if (maxIndex <= offset) {
+ return 0;
+ }
+ return offset;
+ }
+
+ public long getFetchToIndex(long maxIndex)
+ {
+ if (maxIndex <= offset) {
+ return 0;
+ }
+ if (hasLimit()) {
+ long toIndex = limit + offset;
+ if (limit > Long.MAX_VALUE - offset) {
+ throw DruidException.defensive(
+ "Cannot compute toIndex due to overflow [%s]",
+ this);
Review Comment:
Shouldn't this be checked in the constructor itself? Pretty wild if we hit this one. 😄
##########
processing/src/main/java/org/apache/druid/query/operator/OffsetLimit.java:
##########
@@ -0,0 +1,139 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.query.operator;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.google.common.base.Preconditions;
+import org.apache.druid.error.DruidException;
+import java.util.Objects;
+
+public class OffsetLimit
+{
+ protected final long offset;
+ protected final long limit;
+
+ public static final OffsetLimit NONE = new OffsetLimit(0, -1);
+
+ @JsonCreator
+ public OffsetLimit(
+ @JsonProperty("offset") long offset,
+ @JsonProperty("limit") long limit)
+ {
+ Preconditions.checkArgument(offset >= 0, "offset >= 0");
+ this.offset = offset;
+ this.limit = limit < 0 ? -1 : limit;
+ }
+
+ @JsonProperty("offset")
+ public long getOffset()
+ {
+ return offset;
+ }
+
+ @JsonProperty("limit")
+ public long getLimit()
+ {
+ return limit;
+ }
+
+ public boolean isPresent()
+ {
+ return hasOffset() || hasLimit();
+ }
+
+ public boolean hasOffset()
+ {
+ return offset > 0;
+ }
+
+ public boolean hasLimit()
+ {
+ return limit >= 0;
+ }
+
+ public static OffsetLimit limit(int limit2)
+ {
+ return new OffsetLimit(0, limit2);
+ }
+
+ public long getLimitOrMax()
+ {
+ if (limit < 0) {
+ return Long.MAX_VALUE;
+ } else {
+ return limit;
+ }
+ }
+
+ @Override
+ public boolean equals(Object o)
+ {
+ if (this == o) {
+ return true;
+ }
+ if (!(o instanceof OffsetLimit)) {
+ return false;
+ }
+ OffsetLimit that = (OffsetLimit) o;
+ return limit == that.limit && offset == that.offset;
+ }
+
+ @Override
+ public int hashCode()
+ {
+ return Objects.hash(limit, offset);
+ }
+
+ @Override
+ public String toString()
+ {
+ return "OffsetLimit{" +
+ "offset=" + offset +
+ ", limit=" + limit +
+ '}';
+ }
+
+ public long getFetchFromIndex(long maxIndex)
Review Comment:
can you please add some javadocs?
##########
sql/src/main/java/org/apache/druid/sql/calcite/rel/DruidQuery.java:
##########
@@ -1473,13 +1481,78 @@ private WindowOperatorQuery toWindowQuery()
);
}
+ /**
+ * Create an OperatorQuery which runs an order on top of a scan.
+ */
+ @Nullable
+ private WindowOperatorQuery toScanAndSortQuery()
+ {
+ if (sorting == null
+ || sorting.getOrderBys().isEmpty()
+ || sorting.getProjection() != null) {
+ return null;
+ }
+
+ ScanQuery scan = toScanQuery(false);
+ if (scan == null) {
+ return null;
+ }
+
+ // Reject cases which would sort the datasource directly
+ if (dataSource != DruidOuterQueryRel.DUMMY_DATA_SOURCE && dataSource.isConcrete()) {
+ List<String> orderByColumnNames = sorting.getOrderBys()
+ .stream().map(OrderByColumnSpec::getDimension)
+ .collect(Collectors.toList());
+ plannerContext.setPlanningError(
+ "SQL query requires order by non-time column [%s], which is not supported.",
+ orderByColumnNames);
+ return null;
+ }
+
+ QueryDataSource newDataSource = new QueryDataSource(scan);
+ ArrayList<ColumnWithDirection> sortColumns = getColumnWithDriectionsFromOrderBys(sorting.getOrderBys());
+ RowSignature signature = getOutputRowSignature();
+ List<OperatorFactory> operators = new ArrayList<>();
+
+ operators.add(new NaiveSortOperatorFactory(sortColumns));
+ if (!sorting.getOffsetLimit().isNone()) {
+ operators.add(new ScanOperatorFactory(
+ null,
+ null,
+ sorting.getOffsetLimit().toOperatorOffsetLimit(),
+ null,
+ null,
+ null));
+ }
+
+ return new WindowOperatorQuery(
+ newDataSource,
+ new LegacySegmentSpec(Intervals.ETERNITY),
+ plannerContext.queryContextMap(),
+ signature,
+ operators,
+ null);
+ }
+
+ private ArrayList<ColumnWithDirection> getColumnWithDriectionsFromOrderBys(List<OrderByColumnSpec> orderBys)
+ {
+ ArrayList<ColumnWithDirection> ordering = new ArrayList<>();
+ for (OrderByColumnSpec orderBySpec : orderBys) {
+ Direction direction = orderBySpec.getDirection() == OrderByColumnSpec.Direction.ASCENDING
+ ? ColumnWithDirection.Direction.ASC
+ : ColumnWithDirection.Direction.DESC;
+ ordering.add(new ColumnWithDirection(orderBySpec.getDimension(), direction));
+ }
+ return ordering;
+ }
+
/**
* Return this query as a Scan query, or null if this query is not compatible with Scan.
Review Comment:
could you add a few more details here as to how considerSorting parameter is used?
##########
sql/src/main/java/org/apache/druid/sql/calcite/rel/DruidQuery.java:
##########
@@ -1473,13 +1481,78 @@ private WindowOperatorQuery toWindowQuery()
);
}
+ /**
+ * Create an OperatorQuery which runs an order on top of a scan.
+ */
+ @Nullable
+ private WindowOperatorQuery toScanAndSortQuery()
+ {
+ if (sorting == null
+ || sorting.getOrderBys().isEmpty()
+ || sorting.getProjection() != null) {
+ return null;
+ }
+
+ ScanQuery scan = toScanQuery(false);
+ if (scan == null) {
+ return null;
+ }
+
+ // Reject cases which would sort the datasource directly
+ if (dataSource != DruidOuterQueryRel.DUMMY_DATA_SOURCE && dataSource.isConcrete()) {
+ List<String> orderByColumnNames = sorting.getOrderBys()
+ .stream().map(OrderByColumnSpec::getDimension)
+ .collect(Collectors.toList());
+ plannerContext.setPlanningError(
+ "SQL query requires order by non-time column [%s], which is not supported.",
+ orderByColumnNames);
+ return null;
+ }
+
+ QueryDataSource newDataSource = new QueryDataSource(scan);
+ ArrayList<ColumnWithDirection> sortColumns = getColumnWithDriectionsFromOrderBys(sorting.getOrderBys());
Review Comment:
```suggestion
ArrayList<ColumnWithDirection> sortColumns = getColumnWithDirectionsFromOrderBys(sorting.getOrderBys());
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Process pure ordering changes with windowing operators (druid)
Posted by "somu-imply (via GitHub)" <gi...@apache.org>.
somu-imply commented on code in PR #15241:
URL: https://github.com/apache/druid/pull/15241#discussion_r1374843865
##########
processing/src/main/java/org/apache/druid/query/operator/OffsetLimit.java:
##########
@@ -0,0 +1,139 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.query.operator;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.google.common.base.Preconditions;
+import org.apache.druid.error.DruidException;
+import java.util.Objects;
+
+public class OffsetLimit
Review Comment:
Since this is a release blocker, let's not try to refactor everything here and fix as a followup
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Process pure ordering changes with windowing operators (druid)
Posted by "clintropolis (via GitHub)" <gi...@apache.org>.
clintropolis commented on code in PR #15241:
URL: https://github.com/apache/druid/pull/15241#discussion_r1375106261
##########
sql/src/main/java/org/apache/druid/sql/calcite/rel/DruidOuterQueryRel.java:
##########
@@ -47,7 +47,7 @@
*/
public class DruidOuterQueryRel extends DruidRel<DruidOuterQueryRel>
{
- private static final TableDataSource DUMMY_DATA_SOURCE = new TableDataSource("__subquery__");
+ public static final TableDataSource DUMMY_DATA_SOURCE = new TableDataSource("__subquery__");
Review Comment:
i wonder if we should just override `isConcrete` to return false instead of making this public and checking specifically for this (or maybe make a special dummy datasource to use in planner that also implements isConcrete as false), because we could simplify the scan query conversion to just check for `!isConcrete` instead of checking for this instance or concrete
##########
sql/src/main/java/org/apache/druid/sql/calcite/rel/DruidOuterQueryRel.java:
##########
@@ -47,7 +47,7 @@
*/
public class DruidOuterQueryRel extends DruidRel<DruidOuterQueryRel>
{
- private static final TableDataSource DUMMY_DATA_SOURCE = new TableDataSource("__subquery__");
+ public static final TableDataSource DUMMY_DATA_SOURCE = new TableDataSource("__subquery__");
Review Comment:
i wonder if we should just override `isConcrete` to return false instead of making this public and checking specifically for this (or maybe make a special dummy datasource to use in planner that also implements `isConcrete` as false), because we could simplify the scan query conversion to just check for `!isConcrete` instead of checking for this instance or concrete
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Process pure ordering changes with windowing operators (druid)
Posted by "kgyrtkirk (via GitHub)" <gi...@apache.org>.
kgyrtkirk commented on code in PR #15241:
URL: https://github.com/apache/druid/pull/15241#discussion_r1374534132
##########
processing/src/main/java/org/apache/druid/query/operator/OffsetLimit.java:
##########
@@ -0,0 +1,139 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.query.operator;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.google.common.base.Preconditions;
+import org.apache.druid.error.DruidException;
+import java.util.Objects;
+
+public class OffsetLimit
+{
+ protected final long offset;
+ protected final long limit;
+
+ public static final OffsetLimit NONE = new OffsetLimit(0, -1);
+
+ @JsonCreator
+ public OffsetLimit(
+ @JsonProperty("offset") long offset,
+ @JsonProperty("limit") long limit)
+ {
+ Preconditions.checkArgument(offset >= 0, "offset >= 0");
+ this.offset = offset;
+ this.limit = limit < 0 ? -1 : limit;
+ }
+
+ @JsonProperty("offset")
+ public long getOffset()
+ {
+ return offset;
+ }
+
+ @JsonProperty("limit")
+ public long getLimit()
+ {
+ return limit;
+ }
+
+ public boolean isPresent()
+ {
+ return hasOffset() || hasLimit();
+ }
+
+ public boolean hasOffset()
+ {
+ return offset > 0;
+ }
+
+ public boolean hasLimit()
+ {
+ return limit >= 0;
+ }
+
+ public static OffsetLimit limit(int limit2)
+ {
+ return new OffsetLimit(0, limit2);
+ }
+
+ public long getLimitOrMax()
+ {
+ if (limit < 0) {
+ return Long.MAX_VALUE;
+ } else {
+ return limit;
+ }
+ }
+
+ @Override
+ public boolean equals(Object o)
+ {
+ if (this == o) {
+ return true;
+ }
+ if (!(o instanceof OffsetLimit)) {
+ return false;
+ }
+ OffsetLimit that = (OffsetLimit) o;
+ return limit == that.limit && offset == that.offset;
+ }
+
+ @Override
+ public int hashCode()
+ {
+ return Objects.hash(limit, offset);
+ }
+
+ @Override
+ public String toString()
+ {
+ return "OffsetLimit{" +
+ "offset=" + offset +
+ ", limit=" + limit +
+ '}';
+ }
+
+ public long getFetchFromIndex(long maxIndex)
+ {
+ if (maxIndex <= offset) {
+ return 0;
+ }
+ return offset;
+ }
+
+ public long getFetchToIndex(long maxIndex)
+ {
+ if (maxIndex <= offset) {
+ return 0;
+ }
+ if (hasLimit()) {
+ long toIndex = limit + offset;
+ if (limit > Long.MAX_VALUE - offset) {
+ throw DruidException.defensive(
+ "Cannot compute toIndex due to overflow [%s]",
+ this);
Review Comment:
we have some places where `Long.MAX_VALUE` is substituted - I'm now feel like its even safer to not add this throw... or at least not now - I'll get back to it in a later refactor
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Process pure ordering changes with windowing operators (druid)
Posted by "kgyrtkirk (via GitHub)" <gi...@apache.org>.
kgyrtkirk commented on PR #15241:
URL: https://github.com/apache/druid/pull/15241#issuecomment-1782711935
> But now throws a NPE. Although this is under window functions and need not be investigated as a part of this PR. I'll open an issue to track it.
it was a an issue caused by:
* ScanQuery [not returning types](https://github.com/apache/druid/blob/60c2ad597ab44610298e94987582443368863269/processing/src/main/java/org/apache/druid/query/scan/ScanQueryQueryToolChest.java#L200) for columns not being mapped by a VC
* so later on some COMPLEX column type was created as a [last restort](https://github.com/apache/druid/blob/60c2ad597ab44610298e94987582443368863269/processing/src/main/java/org/apache/druid/query/rowsandcols/ArrayListRowsAndColumns.java#L150) with a `null` inner type...
* which was dereferenced during a lookup in [by a ConcurrentHashMap](https://github.com/apache/druid/blob/60c2ad597ab44610298e94987582443368863269/processing/src/main/java/org/apache/druid/segment/column/TypeStrategies.java#L56)
I've fixed it in this PR as the underlying datasource did know the correct types; it was just not forwarded correctly - but we can remove it if that's preffered
I've added your query to `CalciteQueryTest` :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Process pure ordering changes with windowing operators (druid)
Posted by "kgyrtkirk (via GitHub)" <gi...@apache.org>.
kgyrtkirk commented on code in PR #15241:
URL: https://github.com/apache/druid/pull/15241#discussion_r1375184722
##########
processing/src/test/java/org/apache/druid/query/operator/NaivePartitioningOperatorFactoryTest.java:
##########
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.query.operator;
+
+import com.google.common.testing.EqualsTester;
+import org.junit.Test;
+
+import java.util.Arrays;
+
+public class NaivePartitioningOperatorFactoryTest
+{
+ @Test
+ public void testEquals()
+ {
+ new EqualsTester()
Review Comment:
didn't seen that sorry - its not easier at all to use it... but its nice to `declare` equalityGroup-s :D
would you like me to remove them?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Process pure ordering changes with windowing operators (druid)
Posted by "kgyrtkirk (via GitHub)" <gi...@apache.org>.
kgyrtkirk commented on code in PR #15241:
URL: https://github.com/apache/druid/pull/15241#discussion_r1374630205
##########
processing/src/main/java/org/apache/druid/query/rowsandcols/LazilyDecoratedRowsAndColumns.java:
##########
@@ -238,7 +239,8 @@ private Pair<byte[], RowSignature> materializeStorageAdapter(StorageAdapter as)
throw new ISE("accumulated[%s] non-null, why did we get multiple cursors?", accumulated);
}
- int theLimit = limit == -1 ? Integer.MAX_VALUE : limit;
+ long offsetRemaining = limit.getOffset();
Review Comment:
totally agree :)
renamed to `remainingRowsToSkip` and `remainingRowsToFetch`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Process pure ordering changes with windowing operators (druid)
Posted by "abhishekagarwal87 (via GitHub)" <gi...@apache.org>.
abhishekagarwal87 commented on PR #15241:
URL: https://github.com/apache/druid/pull/15241#issuecomment-1784070073
@kgyrtkirk - You will also need to change the docs here https://druid.apache.org/docs/latest/querying/scan-query.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Process pure ordering changes with windowing operators (druid)
Posted by "kgyrtkirk (via GitHub)" <gi...@apache.org>.
kgyrtkirk commented on code in PR #15241:
URL: https://github.com/apache/druid/pull/15241#discussion_r1375184340
##########
sql/src/main/java/org/apache/druid/sql/calcite/rel/DruidQuery.java:
##########
@@ -1474,12 +1487,93 @@ private WindowOperatorQuery toWindowQuery()
}
/**
- * Return this query as a Scan query, or null if this query is not compatible with Scan.
+ * Create an OperatorQuery which runs an order on top of a scan.
+ */
+ @Nullable
+ private WindowOperatorQuery toScanAndSortQuery()
+ {
+ if (sorting == null
+ || sorting.getOrderBys().isEmpty()
+ || sorting.getProjection() != null) {
+ return null;
+ }
+
+ ScanQuery scan = toScanQuery(false);
+ if (scan == null) {
+ return null;
+ }
+
+ if (queryRunsOnHistorical()) {
+ // Currently only non-time orderings of subqueries are allowed.
+ List<String> orderByColumnNames = sorting.getOrderBys()
+ .stream().map(OrderByColumnSpec::getDimension)
+ .collect(Collectors.toList());
+ plannerContext.setPlanningError(
+ "SQL query requires ordering a table by non-time column [%s], which is not supported.",
+ orderByColumnNames);
Review Comment:
I noticed it as well - however didn't digged into it before; I think the eclipse formatter of the project is outdated - I had to fix a few issues already (because checkstyle followed different rules).
regarding this one: it was off as well; but I don't yet see a way to enable formatting to (1) but was able to set only (2)
```
format_1(123,
23123,
3213,
2321321
);
format_2(
123,
23123,
3213,
2321321
);
```
as a matter of fact I don't know if `format_1` would be the preffered or not.
I've set `format_2`; as putting the closing `)` on the same line have caused quite a few annoying mistakes for me in the last couple days...
more-or-less related: we could enforce these formatting things (see [this PR](https://github.com/apache/druid/pull/14937) ); but I didn't wanted to do it for the whole project before we agree on the rules we should follow
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Process pure ordering changes with windowing operators (druid)
Posted by "kgyrtkirk (via GitHub)" <gi...@apache.org>.
kgyrtkirk commented on code in PR #15241:
URL: https://github.com/apache/druid/pull/15241#discussion_r1374297370
##########
processing/src/test/java/org/apache/druid/query/operator/NaiveSortOperatorTest.java:
##########
@@ -0,0 +1,66 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.query.operator;
+
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableMap;
+import org.apache.druid.query.operator.Operator.Signal;
+import org.apache.druid.query.operator.window.RowsAndColumnsHelper;
+import org.apache.druid.query.rowsandcols.MapOfColumnsRowsAndColumns;
+import org.apache.druid.query.rowsandcols.RowsAndColumns;
+import org.apache.druid.query.rowsandcols.column.IntArrayColumn;
+import org.junit.Test;
+
+public class NaiveSortOperatorTest
+{
+ @Test
+ public void testNoInputisHandledCorrectly()
+ {
+ NaiveSortOperator op = new NaiveSortOperator(
+ InlineScanOperator.make(),
+ ImmutableList.of(ColumnWithDirection.ascending("someColumn")));
+
+ new OperatorTestHelper()
+ .withPushFn(() -> (someRac) -> Signal.GO)
+ .runToCompletion(op);
+ }
+
+ @Test
+ public void testSort()
Review Comment:
yeah; was thinking about adding it - but thinked that it might be covered elsewhere :D
doesn't hurt to add a test here as well...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Process pure ordering changes with windowing operators (druid)
Posted by "clintropolis (via GitHub)" <gi...@apache.org>.
clintropolis commented on code in PR #15241:
URL: https://github.com/apache/druid/pull/15241#discussion_r1375106261
##########
sql/src/main/java/org/apache/druid/sql/calcite/rel/DruidOuterQueryRel.java:
##########
@@ -47,7 +47,7 @@
*/
public class DruidOuterQueryRel extends DruidRel<DruidOuterQueryRel>
{
- private static final TableDataSource DUMMY_DATA_SOURCE = new TableDataSource("__subquery__");
+ public static final TableDataSource DUMMY_DATA_SOURCE = new TableDataSource("__subquery__");
Review Comment:
i wonder if we should just override `isConcrete` to return false instead of making this public and checking specifically for this (or maybe make a special dummy datasource class to use in planner that also implements `isConcrete` as false), because we could simplify the scan query conversion to just check for `!isConcrete` instead of checking for this instance or concrete
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Process pure ordering changes with windowing operators (druid)
Posted by "github-advanced-security[bot] (via GitHub)" <gi...@apache.org>.
github-advanced-security[bot] commented on code in PR #15241:
URL: https://github.com/apache/druid/pull/15241#discussion_r1375212595
##########
sql/src/main/java/org/apache/druid/sql/calcite/rel/DruidOuterQueryRel.java:
##########
@@ -47,7 +47,13 @@
*/
public class DruidOuterQueryRel extends DruidRel<DruidOuterQueryRel>
{
- private static final TableDataSource DUMMY_DATA_SOURCE = new TableDataSource("__subquery__");
+ private static final TableDataSource DUMMY_DATA_SOURCE = new TableDataSource("__subquery__")
+ {
+ public boolean isConcrete()
Review Comment:
## Missing Override annotation
This method overrides [TableDataSource.isConcrete](1); it is advisable to add an Override annotation.
[Show more details](https://github.com/apache/druid/security/code-scanning/5940)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Process pure ordering changes with windowing operators (druid)
Posted by "somu-imply (via GitHub)" <gi...@apache.org>.
somu-imply commented on code in PR #15241:
URL: https://github.com/apache/druid/pull/15241#discussion_r1373642265
##########
sql/src/main/java/org/apache/druid/sql/calcite/rel/DruidQuery.java:
##########
@@ -1473,13 +1481,78 @@ private WindowOperatorQuery toWindowQuery()
);
}
+ /**
+ * Create an OperatorQuery which runs an order on top of a scan.
+ */
+ @Nullable
+ private WindowOperatorQuery toScanAndSortQuery()
+ {
+ if (sorting == null
+ || sorting.getOrderBys().isEmpty()
+ || sorting.getProjection() != null) {
+ return null;
+ }
+
+ ScanQuery scan = toScanQuery(false);
+ if (scan == null) {
+ return null;
+ }
+
+ // Reject cases which would sort the datasource directly
+ if (dataSource != DruidOuterQueryRel.DUMMY_DATA_SOURCE && dataSource.isConcrete()) {
Review Comment:
Maybe use Object.equals here
##########
sql/src/main/java/org/apache/druid/sql/calcite/planner/CalciteRulesManager.java:
##########
@@ -237,7 +233,12 @@ public List<Program> programs(final PlannerContext plannerContext)
boolean isDebug = plannerContext.queryContext().isDebug();
return ImmutableList.of(
- Programs.sequence(preProgram, Programs.ofRules(druidConventionRuleSet(plannerContext))),
+ Programs.sequence(
Review Comment:
This is great to tell us how the plans look before the hep planner starts. Probably put a comment here saying that these are printed only when the Debug flag is set and developers should set that and run queries to get more info on the plans. a
##########
sql/src/main/java/org/apache/druid/sql/calcite/rel/DruidQuery.java:
##########
@@ -1473,13 +1481,78 @@ private WindowOperatorQuery toWindowQuery()
);
}
+ /**
+ * Create an OperatorQuery which runs an order on top of a scan.
+ */
+ @Nullable
+ private WindowOperatorQuery toScanAndSortQuery()
+ {
+ if (sorting == null
+ || sorting.getOrderBys().isEmpty()
+ || sorting.getProjection() != null) {
+ return null;
+ }
+
+ ScanQuery scan = toScanQuery(false);
+ if (scan == null) {
+ return null;
+ }
+
+ // Reject cases which would sort the datasource directly
Review Comment:
We can be more direct and say that we are currently allowing non-time ordering only on subqueries.
##########
processing/src/test/java/org/apache/druid/query/operator/NaiveSortOperatorTest.java:
##########
@@ -0,0 +1,66 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.query.operator;
+
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableMap;
+import org.apache.druid.query.operator.Operator.Signal;
+import org.apache.druid.query.operator.window.RowsAndColumnsHelper;
+import org.apache.druid.query.rowsandcols.MapOfColumnsRowsAndColumns;
+import org.apache.druid.query.rowsandcols.RowsAndColumns;
+import org.apache.druid.query.rowsandcols.column.IntArrayColumn;
+import org.junit.Test;
+
+public class NaiveSortOperatorTest
+{
+ @Test
+ public void testNoInputisHandledCorrectly()
+ {
+ NaiveSortOperator op = new NaiveSortOperator(
+ InlineScanOperator.make(),
+ ImmutableList.of(ColumnWithDirection.ascending("someColumn")));
+
+ new OperatorTestHelper()
+ .withPushFn(() -> (someRac) -> Signal.GO)
+ .runToCompletion(op);
+ }
+
+ @Test
+ public void testSort()
Review Comment:
nit. rename to testSortAscending() and also add a testSortDescending()
##########
processing/src/main/java/org/apache/druid/query/operator/OffsetLimit.java:
##########
@@ -0,0 +1,139 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.query.operator;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.google.common.base.Preconditions;
+import org.apache.druid.error.DruidException;
+import java.util.Objects;
+
+public class OffsetLimit
Review Comment:
Also can we call this a OffsetLimitOperator ?
##########
processing/src/main/java/org/apache/druid/query/operator/OffsetLimit.java:
##########
@@ -0,0 +1,139 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.query.operator;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.google.common.base.Preconditions;
+import org.apache.druid.error.DruidException;
+import java.util.Objects;
+
+public class OffsetLimit
+{
+ protected final long offset;
+ protected final long limit;
+
+ public static final OffsetLimit NONE = new OffsetLimit(0, -1);
+
+ @JsonCreator
+ public OffsetLimit(
+ @JsonProperty("offset") long offset,
+ @JsonProperty("limit") long limit)
+ {
+ Preconditions.checkArgument(offset >= 0, "offset >= 0");
+ this.offset = offset;
+ this.limit = limit < 0 ? -1 : limit;
+ }
+
+ @JsonProperty("offset")
+ public long getOffset()
+ {
+ return offset;
+ }
+
+ @JsonProperty("limit")
+ public long getLimit()
+ {
+ return limit;
+ }
+
+ public boolean isPresent()
+ {
+ return hasOffset() || hasLimit();
+ }
+
+ public boolean hasOffset()
+ {
+ return offset > 0;
+ }
+
+ public boolean hasLimit()
+ {
+ return limit >= 0;
+ }
+
+ public static OffsetLimit limit(int limit2)
+ {
+ return new OffsetLimit(0, limit2);
+ }
+
+ public long getLimitOrMax()
+ {
+ if (limit < 0) {
+ return Long.MAX_VALUE;
+ } else {
+ return limit;
+ }
+ }
+
+ @Override
+ public boolean equals(Object o)
+ {
+ if (this == o) {
+ return true;
+ }
+ if (!(o instanceof OffsetLimit)) {
+ return false;
+ }
+ OffsetLimit that = (OffsetLimit) o;
+ return limit == that.limit && offset == that.offset;
+ }
+
+ @Override
+ public int hashCode()
+ {
+ return Objects.hash(limit, offset);
+ }
+
+ @Override
+ public String toString()
+ {
+ return "OffsetLimit{" +
+ "offset=" + offset +
+ ", limit=" + limit +
+ '}';
+ }
+
+ public long getFetchFromIndex(long maxIndex)
Review Comment:
nit. getFromIndex or getPositionFromIndex might be a better name. same for getFetchToIndex
##########
processing/src/main/java/org/apache/druid/query/operator/OffsetLimit.java:
##########
@@ -0,0 +1,139 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.query.operator;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.google.common.base.Preconditions;
+import org.apache.druid.error.DruidException;
+import java.util.Objects;
+
+public class OffsetLimit
Review Comment:
Question: The planner also has an offset limit object which does similar stuff. Maybe we could move this to an utility? Am not sure though so asked the Q
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Process pure ordering changes with windowing operators (druid)
Posted by "github-advanced-security[bot] (via GitHub)" <gi...@apache.org>.
github-advanced-security[bot] commented on code in PR #15241:
URL: https://github.com/apache/druid/pull/15241#discussion_r1370150359
##########
sql/src/main/java/org/apache/druid/sql/calcite/rel/DruidQuery.java:
##########
@@ -1473,14 +1480,47 @@
);
}
+ /**
+ * Create an OperatorQuery which runs an order on top of a scan.
+ */
+ // FIXME: could have a better name...
+ @Nullable
+ private WindowOperatorQuery toScanAndSortQuery()
+ {
+ if (sorting == null || !sorting.isPureOrder()) {
+ return null;
+ }
+
+ ScanQuery scan = toScanQuery(false);
+ if (scan == null) {
+ return null;
+ }
+
+ QueryDataSource newDataSource = new QueryDataSource(scan);
+
+ ArrayList<ColumnWithDirection> sortColumns = ColumnWithDirection.fromOrderBysColumnSpecs(sorting.getOrderBys());
+ NaiveSortOperatorFactory sortOperator = new NaiveSortOperatorFactory(sortColumns);
+
+ RowSignature signature = getOutputRowSignature();
+ return new WindowOperatorQuery(
+ newDataSource,
+ new LegacySegmentSpec(Intervals.ETERNITY),
+ plannerContext.queryContextMap(),
+ signature,
+ Collections.singletonList(sortOperator),
+ null);
+ }
+
/**
* Return this query as a Scan query, or null if this query is not compatible with Scan.
*
* @return query or null
*/
@Nullable
- private ScanQuery toScanQuery()
+ private ScanQuery toScanQuery(boolean considerSorting)
{
+ final Sorting sorting = considerSorting ? this.sorting : null;
Review Comment:
## Possible confusion of local and field
Potentially confusing name: method [toScanQuery](1) also refers to field [sorting](2) (as this.sorting).
[Show more details](https://github.com/apache/druid/security/code-scanning/5928)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Process pure ordering changes with windowing operators (druid)
Posted by "github-advanced-security[bot] (via GitHub)" <gi...@apache.org>.
github-advanced-security[bot] commented on code in PR #15241:
URL: https://github.com/apache/druid/pull/15241#discussion_r1372262214
##########
processing/src/test/java/org/apache/druid/query/operator/OffsetLimitTest.java:
##########
@@ -0,0 +1,131 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.query.operator;
+
+import org.junit.Test;
+
+import java.util.HashSet;
+import java.util.Set;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertTrue;
+
+public class OffsetLimitTest
+{
+ @Test
+ public void testNone()
+ {
+ assertFalse(OffsetLimit.NONE.isPresent());
+ assertFalse(OffsetLimit.NONE.hasOffset());
+ assertFalse(OffsetLimit.NONE.hasLimit());
+ }
+
+ @Test
+ public void testOffset()
+ {
+ int offset = 3;
+ OffsetLimit ol = new OffsetLimit(offset, -1);
+ assertTrue(ol.hasOffset());
+ assertFalse(ol.hasLimit());
+ assertEquals(offset, ol.getOffset());
+ assertEquals(-1, ol.getLimit());
+ assertEquals(Long.MAX_VALUE, ol.getLimitOrMax());
+ assertEquals(offset, ol.getFetchFromIndex(Long.MAX_VALUE));
+ assertEquals(Long.MAX_VALUE, ol.getFetchToIndex(Long.MAX_VALUE));
+ assertEquals(0, ol.getFetchFromIndex(1));
+ assertEquals(0, ol.getFetchFromIndex(offset));
+ assertEquals(0, ol.getFetchToIndex(offset));
+ }
+
+ @Test
+ public void testLimit()
+ {
+ OffsetLimit ol = new OffsetLimit(0, 4);
+ assertFalse(ol.hasOffset());
+ assertTrue(ol.hasLimit());
+ assertEquals(0, ol.getOffset());
+ assertEquals(4, ol.getLimit());
+ assertEquals(4, ol.getLimitOrMax());
+ assertEquals(0, ol.getFetchFromIndex(Long.MAX_VALUE));
+ assertEquals(4, ol.getFetchToIndex(Long.MAX_VALUE));
+ assertEquals(0, ol.getFetchFromIndex(2));
+ assertEquals(2, ol.getFetchToIndex(2));
+ }
+
+ @Test
+ public void testOffsetLimit()
+ {
+ int offset = 3;
+ int limit = 10;
+ OffsetLimit ol = new OffsetLimit(offset, limit);
+ assertTrue(ol.hasOffset());
+ assertTrue(ol.hasLimit());
+ assertEquals(offset, ol.getOffset());
+ assertEquals(limit, ol.getLimit());
+ assertEquals(limit, ol.getLimitOrMax());
+ assertEquals(offset, ol.getFetchFromIndex(Long.MAX_VALUE));
+ assertEquals(offset + limit, ol.getFetchToIndex(Long.MAX_VALUE));
+ assertEquals(0, ol.getFetchFromIndex(offset));
+ assertEquals(0, ol.getFetchToIndex(offset));
+ assertEquals(offset, ol.getFetchFromIndex(offset + 1));
+ assertEquals(offset + 1, ol.getFetchToIndex(offset + 1));
+ }
+
+ @Test(expected = IllegalArgumentException.class)
+ public void testInvalidOffset()
+ {
+ new OffsetLimit(-1, -1);
+ }
+
+ @Test
+ public void testNegativeLimitsAreNotDifferent()
+ {
+ OffsetLimit ol1 = new OffsetLimit(1, -1);
+ OffsetLimit ol2 = new OffsetLimit(1, -2);
+ assertEquals(ol1, ol2);
+ }
+
+ @Test
+ public void testEquals()
+ {
+ Set<Object> set = new HashSet<>();
+
+ long[] offsets = new long[] {0, 1, 100};
+ long[] limits = new long[] {-1, 5, 10, 1000};
+ for (long offset : offsets) {
+ for (long limit : limits) {
+ OffsetLimit a = new OffsetLimit(offset, limit);
+ OffsetLimit b = new OffsetLimit(offset, limit);
+ assertEquals(a.hashCode(), b.hashCode());
+
+ set.add(new OffsetLimit(offset, limit)
Review Comment:
## Inconsistent equals and hashCode
Class overrides [hashCode](1) but not equals.
[Show more details](https://github.com/apache/druid/security/code-scanning/5932)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Process pure ordering changes with windowing operators (druid)
Posted by "clintropolis (via GitHub)" <gi...@apache.org>.
clintropolis commented on code in PR #15241:
URL: https://github.com/apache/druid/pull/15241#discussion_r1375104355
##########
processing/src/test/java/org/apache/druid/query/operator/NaivePartitioningOperatorFactoryTest.java:
##########
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.query.operator;
+
+import com.google.common.testing.EqualsTester;
+import org.junit.Test;
+
+import java.util.Arrays;
+
+public class NaivePartitioningOperatorFactoryTest
+{
+ @Test
+ public void testEquals()
+ {
+ new EqualsTester()
Review Comment:
nit: we've been using `EqualsVerifier` for many equals/hashcode tests, is it easier to use this one for these classes for some reason? (I don't see anything using this anywhere else in codebase)
##########
sql/src/main/java/org/apache/druid/sql/calcite/rel/DruidQuery.java:
##########
@@ -1474,12 +1487,93 @@ private WindowOperatorQuery toWindowQuery()
}
/**
- * Return this query as a Scan query, or null if this query is not compatible with Scan.
+ * Create an OperatorQuery which runs an order on top of a scan.
+ */
+ @Nullable
+ private WindowOperatorQuery toScanAndSortQuery()
+ {
+ if (sorting == null
+ || sorting.getOrderBys().isEmpty()
+ || sorting.getProjection() != null) {
+ return null;
+ }
+
+ ScanQuery scan = toScanQuery(false);
+ if (scan == null) {
+ return null;
+ }
+
+ if (queryRunsOnHistorical()) {
+ // Currently only non-time orderings of subqueries are allowed.
+ List<String> orderByColumnNames = sorting.getOrderBys()
+ .stream().map(OrderByColumnSpec::getDimension)
+ .collect(Collectors.toList());
+ plannerContext.setPlanningError(
+ "SQL query requires ordering a table by non-time column [%s], which is not supported.",
+ orderByColumnNames);
Review Comment:
super nitpick, but much of codebase puts trailing `)` on newlines so stuff is symmetrical ⚖️ (noticed this in lots of other places too)
```suggestion
plannerContext.setPlanningError(
"SQL query requires ordering a table by non-time column [%s], which is not supported.",
orderByColumnNames
);
```
not a blocker by any means, just throwing this out there since i find it more aesthetically pleasing 😅
##########
sql/src/main/java/org/apache/druid/sql/calcite/rel/DruidOuterQueryRel.java:
##########
@@ -47,7 +47,7 @@
*/
public class DruidOuterQueryRel extends DruidRel<DruidOuterQueryRel>
{
- private static final TableDataSource DUMMY_DATA_SOURCE = new TableDataSource("__subquery__");
+ public static final TableDataSource DUMMY_DATA_SOURCE = new TableDataSource("__subquery__");
Review Comment:
i wonder if we should just override `isConcrete` to return false instead of making this public and checking specifically for this, because we could simplify the scan query conversion to just check for `!isConcrete` instead of checking for this instance or concrete
##########
sql/src/main/java/org/apache/druid/sql/calcite/rel/DruidQuery.java:
##########
@@ -1474,12 +1487,93 @@ private WindowOperatorQuery toWindowQuery()
}
/**
- * Return this query as a Scan query, or null if this query is not compatible with Scan.
+ * Create an OperatorQuery which runs an order on top of a scan.
+ */
+ @Nullable
+ private WindowOperatorQuery toScanAndSortQuery()
+ {
+ if (sorting == null
+ || sorting.getOrderBys().isEmpty()
+ || sorting.getProjection() != null) {
+ return null;
+ }
+
+ ScanQuery scan = toScanQuery(false);
+ if (scan == null) {
+ return null;
+ }
+
+ if (queryRunsOnHistorical()) {
+ // Currently only non-time orderings of subqueries are allowed.
+ List<String> orderByColumnNames = sorting.getOrderBys()
+ .stream().map(OrderByColumnSpec::getDimension)
+ .collect(Collectors.toList());
+ plannerContext.setPlanningError(
+ "SQL query requires ordering a table by non-time column [%s], which is not supported.",
+ orderByColumnNames);
Review Comment:
super nitpick, but much of codebase puts trailing `)` on newlines so stuff is symmetrical ⚖️ (noticed this in lots of other places too)
```suggestion
plannerContext.setPlanningError(
"SQL query requires ordering a table by non-time column [%s], which is not supported.",
orderByColumnNames
);
```
not a blocker by any means, just throwing this out there since i find it more aesthetically pleasing 😅
##########
processing/src/main/java/org/apache/druid/query/scan/ScanQueryQueryToolChest.java:
##########
@@ -207,6 +208,19 @@ public RowSignature resultArraySignature(final ScanQuery query)
}
}
+ private ColumnType getDataSoruceColumnType(DataSource dataSource, String columnName)
Review Comment:
typo, should be `getDataSourceColumnType`, also mark `@Nullable`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Process pure ordering changes with windowing operators (druid)
Posted by "kgyrtkirk (via GitHub)" <gi...@apache.org>.
kgyrtkirk commented on PR #15241:
URL: https://github.com/apache/druid/pull/15241#issuecomment-1783885226
@somu-imply, @abhishekagarwal87, @clintropolis could you please take another look?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Process pure ordering changes with windowing operators (druid)
Posted by "kgyrtkirk (via GitHub)" <gi...@apache.org>.
kgyrtkirk commented on code in PR #15241:
URL: https://github.com/apache/druid/pull/15241#discussion_r1374290163
##########
sql/src/main/java/org/apache/druid/sql/calcite/rel/DruidQuery.java:
##########
@@ -1473,13 +1481,78 @@ private WindowOperatorQuery toWindowQuery()
);
}
+ /**
+ * Create an OperatorQuery which runs an order on top of a scan.
+ */
+ @Nullable
+ private WindowOperatorQuery toScanAndSortQuery()
+ {
+ if (sorting == null
+ || sorting.getOrderBys().isEmpty()
+ || sorting.getProjection() != null) {
+ return null;
+ }
+
+ ScanQuery scan = toScanQuery(false);
+ if (scan == null) {
+ return null;
+ }
+
+ // Reject cases which would sort the datasource directly
+ if (dataSource != DruidOuterQueryRel.DUMMY_DATA_SOURCE && dataSource.isConcrete()) {
Review Comment:
I intentionally wanted to check for the reference of that object - so that it identifies that it was called from [this point in DruidOuterQuery](https://github.com/apache/druid/blob/60c2ad597ab44610298e94987582443368863269/sql/src/main/java/org/apache/druid/sql/calcite/rel/DruidOuterQueryRel.java#L123)
later this should probably revised - as it's already a bit odd that `OuterRel` passes a `DataSource` which is `concreate` - meanwhile its known to be a subquery
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Process pure ordering changes with windowing operators (druid)
Posted by "somu-imply (via GitHub)" <gi...@apache.org>.
somu-imply commented on code in PR #15241:
URL: https://github.com/apache/druid/pull/15241#discussion_r1373679396
##########
sql/src/main/java/org/apache/druid/sql/calcite/planner/CalciteRulesManager.java:
##########
@@ -237,7 +233,12 @@ public List<Program> programs(final PlannerContext plannerContext)
boolean isDebug = plannerContext.queryContext().isDebug();
return ImmutableList.of(
- Programs.sequence(preProgram, Programs.ofRules(druidConventionRuleSet(plannerContext))),
+ Programs.sequence(
Review Comment:
This is great to tell us how the plans look before the hep planner starts. Probably put a comment here saying that these are printed only when the Debug flag is set and developers should set that and run queries to get more info on the plans.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Process pure ordering changes with windowing operators (druid)
Posted by "github-advanced-security[bot] (via GitHub)" <gi...@apache.org>.
github-advanced-security[bot] commented on code in PR #15241:
URL: https://github.com/apache/druid/pull/15241#discussion_r1374440220
##########
processing/src/test/java/org/apache/druid/query/operator/NaiveSortOperatorTest.java:
##########
@@ -0,0 +1,93 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.query.operator;
+
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableMap;
+import org.apache.druid.query.operator.Operator.Signal;
+import org.apache.druid.query.operator.window.RowsAndColumnsHelper;
+import org.apache.druid.query.rowsandcols.MapOfColumnsRowsAndColumns;
+import org.apache.druid.query.rowsandcols.RowsAndColumns;
+import org.apache.druid.query.rowsandcols.column.Column;
+import org.apache.druid.query.rowsandcols.column.IntArrayColumn;
+import org.junit.Test;
+
+public class NaiveSortOperatorTest
+{
+ @Test
+ public void testNoInputisHandledCorrectly()
+ {
+ NaiveSortOperator op = new NaiveSortOperator(
+ InlineScanOperator.make(),
+ ImmutableList.of(ColumnWithDirection.ascending("someColumn")));
+
+ new OperatorTestHelper()
+ .withPushFn(() -> (someRac) -> Signal.GO)
+ .runToCompletion(op);
+ }
+
+ @Test
+ public void testSortAscending()
+ {
+ RowsAndColumns rac1 = racForColumn("c", new int[] {5, 3, 1});
+ RowsAndColumns rac2 = racForColumn("c", new int[] {2, 6, 4});
+
+ NaiveSortOperator op = new NaiveSortOperator(
+ InlineScanOperator.make(rac1, rac2),
+ ImmutableList.of(ColumnWithDirection.ascending("c")));
+
+ new OperatorTestHelper()
+ .expectAndStopAfter(
+ new RowsAndColumnsHelper()
+ .expectColumn("c", new int[] {1, 2, 3, 4, 5, 6}))
+ .runToCompletion(op);
+ }
+
+ @Test
+ public void testSortDescending()
+ {
+ RowsAndColumns rac1 = racForColumn("c", new int[] {5, 3, 1});
+ RowsAndColumns rac2 = racForColumn("c", new int[] {2, 6, 4});
+
+ NaiveSortOperator op = new NaiveSortOperator(
+ InlineScanOperator.make(rac1, rac2),
+ ImmutableList.of(ColumnWithDirection.descending("c")));
+
+ new OperatorTestHelper()
+ .expectAndStopAfter(
+ new RowsAndColumnsHelper()
+ .expectColumn("c", new int[] {6, 5, 4, 3, 2, 1}))
+ .runToCompletion(op);
+ }
+
+ private MapOfColumnsRowsAndColumns racForColumn(String k1, Object arr)
Review Comment:
## Confusing overloading of methods
Method NaiveSortOperatorTest.racForColumn(..) could be confused with overloaded method [racForColumn](1), since dispatch depends on static types.
[Show more details](https://github.com/apache/druid/security/code-scanning/5936)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org