You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@drill.apache.org by GitBox <gi...@apache.org> on 2020/02/08 07:20:29 UTC
[GitHub] [drill] paul-rogers opened a new pull request #1974: DRILL-7574:
Generalize the projection parser
paul-rogers opened a new pull request #1974: DRILL-7574: Generalize the projection parser
URL: https://github.com/apache/drill/pull/1974
Adds support for multi-dimensional arrays, and columns
projected as both an array and a map.
## Description
EVF contains a bit of code called the"projection parser": it takes a projection list and converts it into a form useful for scan projection. The prior version handled single-level arrays, such as needed for the `columns` column in the text reader. For JSON, we must handle arbitrary column structures such as:
```
a, a.b, a[1], a[1][2], a.[1][2].b
```
The revised version is more forgiving; it just describes what appears in the projection list without trying to enforce semantic rules like the previous version did.
Modified consumers of the projection parser to reflect the changes.
## Documentation
This feature is not user-visible.
## Testing
Added new unit tests, revised existing unit tests. Reran all unit tests.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [drill] KazydubB commented on a change in pull request #1974:
DRILL-7574: Generalize the projection parser
Posted by GitBox <gi...@apache.org>.
KazydubB commented on a change in pull request #1974: DRILL-7574: Generalize the projection parser
URL: https://github.com/apache/drill/pull/1974#discussion_r378763948
##########
File path: exec/java-exec/src/main/java/org/apache/drill/exec/physical/resultSet/project/RequestedColumn.java
##########
@@ -0,0 +1,137 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.physical.resultSet.project;
+
+/**
+ * Plan-time properties of a requested column.Represents
+ * a consolidated view of the set of references to a column.
+ * For example, the project list might contain:<br>
+ * <tt>SELECT columns[4], columns[8]</tt><br>
+ * <tt>SELECT a.b, a.c</tt><br>
+ * <tt>SELECT columns, columns[1]</tt><br>
+ * <tt>SELECT a, a.b</tt><br>
+ * In each case, the same column is referenced in different
+ * forms which are consolidated into this abstraction.
+ * <p>
+ * The resulting information is a "pattern": a form of reference
+ * which which a concrete type can be compatible or not. The project
Review comment:
```suggestion
* with which a concrete type can be compatible or not. The project
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [drill] KazydubB commented on a change in pull request #1974:
DRILL-7574: Generalize the projection parser
Posted by GitBox <gi...@apache.org>.
KazydubB commented on a change in pull request #1974: DRILL-7574: Generalize the projection parser
URL: https://github.com/apache/drill/pull/1974#discussion_r377027268
##########
File path: exec/java-exec/src/main/java/org/apache/drill/exec/physical/resultSet/project/RequestedColumn.java
##########
@@ -0,0 +1,137 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.physical.resultSet.project;
+
+/**
+ * Plan-time properties of a requested column.Represents
+ * a consolidated view of the set of references to a column.
+ * For example, the project list might contain:<br>
+ * <tt>SELECT columns[4], columns[8]</tt><br>
+ * <tt>SELECT a.b, a.c</tt><br>
+ * <tt>SELECT columns, columns[1]</tt><br>
+ * <tt>SELECT a, a.b</tt><br>
+ * In each case, the same column is referenced in different
+ * forms which are consolidated into this abstraction.
+ * <p>
+ * The resulting information is a "pattern": a form of reference
+ * which which a concrete type can be compatible or not. The project
+ * list does not contain sufficient information to definitively pick
+ * a type; it only excludes certain types.
+ * <p>
+ * Depending on the syntax, we can infer if a column must
+ * be an array or map. This is definitive: though we know that
+ * columns of the form above must be an array or a map,
+ * we cannot know if a simple column reference might refer
+ * to an array or map.
+ *
+ * <h4>Compatibility Rules<h4>
+ *
+ * The pattern given by projection is consistent with certain concrete types
+ * as follows. + means any number of additional qualifiers.
+ * <p>
+ * <table>
+ * <tr><th>Type</th><th>Consistent with</th></tr>
+ * <tr><td>Non-repeated MAP</td>
+ * <td>{@code a+} {@code a.b+}</td></tr>
+ * <tr><td>Repeated MAP</td>
+ * <td>{@code a+} {@code a.b+} {@code a[n].b+}</td>></tr>
+ * <tr><td>Non-repeated Scalar</td>
+ * <td>{@code a}</td></tr>
+ * <tr><td>Repeated Scalar</td>
+ * <td>{@code a} {@code a[n]}</td></tr>
+ * <tr><td>Non-repeated DICT</td>
+ * <td>{@code a} {@code a['key']}</td></tr>
+ * <tr><td>Repeated DICT</td>
+ * <td>{@code a} {@code a[n]} {@code a['key']} {@code a[n]['key']}</td></tr>
+ * <tr><td>Non-repeated LIST</td>
+ * <td>{@code a} {@code a[n]}</td></tr>
+ * <tr><td>Repeated LIST</td>
+ * <td>{@code a} {@code a[n]} {@code a[n][n]}</td></tr>
+ * </table>
+ *
+ * MAP, DICT, UNION and LIST are structured types: projection can reach
+ * into the structure to any number of levels. In such a case, when sufficient
+ * schema information is available, the above rules can be applied recursively
+ * to each level of structure. The recursion can be done in the class for a
+ * DICT (since there is only one child type), but must be external for other
+ * complex types. For MAP, the column can report which specific members
+ * are projected.
+ * <p>
+ * The Text reader allows the {@code columns} column, which allows the
+ * user to specify indexes. This class reports which indexes were actually
+ * selected. Index information is available only at the top level, but
+ * not for 2+ dimensions.
+ */
+public interface RequestedColumn {
+ String name();
+ String fullName();
+ boolean nameEquals(String target);
+
+ boolean isWildcard();
+
+ /**
+ * @return true if this column has no qualifiers. Example:
+ * {@code a}.
+ */
+ boolean isSimple();
+
+ /**
+ * Report whether the projection implies a tuple. Example:
+ * {@code a.b}. Not that this method, and others can only tell
+ * if the projection implies a tuple; the actual column may
+ * be a tuple (MAP), but be projected simply. The map
+ * format also describes a DICT with a VARCHAR key.
+ *
+ * @return true if the column has a map-like projection.
+ */
+ boolean isTuple();
+ RequestedTuple tuple();
+
+ /**
+ * Report whether the first qualifier is an array.
+ * Example: {@code a[1]}. The array format also describes
+ * a DICT with an integer key.
+ * @return true if the column must be an array.
+ */
+ boolean isArray();
+
+ int arrayDims();
+
+ /**
+ * @return true if the column has enumerated indexes.
+ * Example: {@code a[2], a[5]}
+ */
+ boolean hasIndexes();
+
+ /**
+ * Return the maximum index value, if indexes where given.
Review comment:
```suggestion
* Return the maximum index value, if indexes were given.
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [drill] KazydubB commented on a change in pull request #1974:
DRILL-7574: Generalize the projection parser
Posted by GitBox <gi...@apache.org>.
KazydubB commented on a change in pull request #1974: DRILL-7574: Generalize the projection parser
URL: https://github.com/apache/drill/pull/1974#discussion_r378763444
##########
File path: exec/java-exec/src/main/java/org/apache/drill/exec/physical/resultSet/project/RequestedColumn.java
##########
@@ -0,0 +1,137 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.physical.resultSet.project;
+
+/**
+ * Plan-time properties of a requested column.Represents
Review comment:
```suggestion
* Plan-time properties of a requested column. Represents
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [drill] KazydubB commented on a change in pull request #1974:
DRILL-7574: Generalize the projection parser
Posted by GitBox <gi...@apache.org>.
KazydubB commented on a change in pull request #1974: DRILL-7574: Generalize the projection parser
URL: https://github.com/apache/drill/pull/1974#discussion_r380154237
##########
File path: exec/java-exec/src/main/java/org/apache/drill/exec/physical/resultSet/project/RequestedColumn.java
##########
@@ -0,0 +1,137 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.physical.resultSet.project;
+
+/**
+ * Plan-time properties of a requested column.Represents
+ * a consolidated view of the set of references to a column.
+ * For example, the project list might contain:<br>
+ * <tt>SELECT columns[4], columns[8]</tt><br>
+ * <tt>SELECT a.b, a.c</tt><br>
+ * <tt>SELECT columns, columns[1]</tt><br>
+ * <tt>SELECT a, a.b</tt><br>
+ * In each case, the same column is referenced in different
+ * forms which are consolidated into this abstraction.
+ * <p>
+ * The resulting information is a "pattern": a form of reference
+ * which which a concrete type can be compatible or not. The project
+ * list does not contain sufficient information to definitively pick
+ * a type; it only excludes certain types.
+ * <p>
+ * Depending on the syntax, we can infer if a column must
+ * be an array or map. This is definitive: though we know that
+ * columns of the form above must be an array or a map,
+ * we cannot know if a simple column reference might refer
+ * to an array or map.
+ *
+ * <h4>Compatibility Rules<h4>
+ *
+ * The pattern given by projection is consistent with certain concrete types
+ * as follows. + means any number of additional qualifiers.
+ * <p>
+ * <table>
+ * <tr><th>Type</th><th>Consistent with</th></tr>
+ * <tr><td>Non-repeated MAP</td>
+ * <td>{@code a+} {@code a.b+}</td></tr>
+ * <tr><td>Repeated MAP</td>
+ * <td>{@code a+} {@code a.b+} {@code a[n].b+}</td>></tr>
+ * <tr><td>Non-repeated Scalar</td>
+ * <td>{@code a}</td></tr>
+ * <tr><td>Repeated Scalar</td>
+ * <td>{@code a} {@code a[n]}</td></tr>
+ * <tr><td>Non-repeated DICT</td>
+ * <td>{@code a} {@code a['key']}</td></tr>
+ * <tr><td>Repeated DICT</td>
+ * <td>{@code a} {@code a[n]} {@code a['key']} {@code a[n]['key']}</td></tr>
Review comment:
Checked whether `m.a` is supported for `MAP` arrays: it looks like this is not supported in Drill.
For a json file `file.json`
```
{"sa": [{"a": 1}, {"a": 2}, {"a": 3}]}
{"sa": [{"a": 1}]}
```
following query ``select t.sa.a kv from dfs.`file.json` t`` produces two rows with `null` value each. (Should have returned an error instead?)
In case when types are known during planning, e.g. in case when querying Hive table, there is following validation: `VALIDATION ERROR: From line 1, column 27 to line 1, column 28: Cannot apply 'ITEM' to arguments of type 'ITEM(<RECORDTYPE(INTEGER SN, RECORDTYPE(INTEGER TN, VARCHAR(65535) TS) ARRAY SA) ARRAY>, <CHAR(2)>)'. Supported form(s): <ARRAY>[<INTEGER>]
<MAP>[<VALUE>]`
(used following test in `TestHiveStructs.java`:
```
@Test
public void strWithArr2ByIdxP0111() throws Exception {
HiveTestUtilities.assertNativeScanUsed(queryBuilder(), "struct_tbl_p");
testBuilder()
.sqlQuery("SELECT rid, t.str_wa_2.fa.sn p0 FROM hive.struct_tbl_p t")
.unOrdered()
.baselineColumns("rid", "p0")
.expectsEmptyResultSet()
.go();
}
```
)
However, such behavior is present in Hive, but for repeated (Drill's) `MAP` only (but not for repeated `DICT`), IIRC.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [drill] KazydubB commented on a change in pull request #1974:
DRILL-7574: Generalize the projection parser
Posted by GitBox <gi...@apache.org>.
KazydubB commented on a change in pull request #1974: DRILL-7574: Generalize the projection parser
URL: https://github.com/apache/drill/pull/1974#discussion_r377021700
##########
File path: exec/java-exec/src/main/java/org/apache/drill/exec/physical/resultSet/project/Qualifier.java
##########
@@ -0,0 +1,162 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.physical.resultSet.project;
+
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+import org.apache.drill.exec.physical.resultSet.project.RequestedTuple.TupleProjectionType;
+
+/**
+ * Represents one level of qualifier for a column. Analogous to
+ * a {@code SchemaPath}, but represents the result of coalescing
+ * multiple occurrences of the same column.
+ */
+public class Qualifier implements QualifierContainer {
+ /**
+ * Marker to indicate that that a) the item is an
+ * array, and b) that all indexes are to be projected.
+ * Used when seeing both a and a[x].
+ */
+ private static final Set<Integer> ALL_INDEXES = new HashSet<>();
+
+ private Set<Integer> indexes;
+ private RequestedTuple members;
+ private Qualifier child;
+
+ @Override
+ public Qualifier qualifier() { return child; }
+
+ @Override
+ public Qualifier requireQualifier() {
+ if (child == null) {
+ child = new Qualifier();
+ }
+ return child;
+ }
+
+ public boolean isArray() {
+ return indexes != null;
+ }
+
+ public boolean hasIndexes() {
+ return isArray() && indexes != ALL_INDEXES;
+ }
+
+ public boolean hasIndex(int index) {
+ return hasIndexes() && indexes.contains(index);
+ }
+
+ public int maxIndex() {
+ if (! hasIndexes()) {
+ return 0;
Review comment:
Is it safe to have `0` instead of `-1`, for example? Does it imply that there's always `a[0]` selected? (I'm very likely getting this wrong.)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [drill] KazydubB commented on a change in pull request #1974:
DRILL-7574: Generalize the projection parser
Posted by GitBox <gi...@apache.org>.
KazydubB commented on a change in pull request #1974: DRILL-7574: Generalize the projection parser
URL: https://github.com/apache/drill/pull/1974#discussion_r377007494
##########
File path: exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/scan/project/projSet/ProjectionChecker.java
##########
@@ -0,0 +1,87 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.physical.impl.scan.project.projSet;
+
+import org.apache.drill.common.exceptions.CustomErrorContext;
+import org.apache.drill.common.exceptions.UserException;
+import org.apache.drill.common.types.TypeProtos.MinorType;
+import org.apache.drill.common.types.Types;
+import org.apache.drill.exec.physical.resultSet.project.RequestedColumn;
+import org.apache.drill.exec.physical.resultSet.project.RequestedTuple;
+import org.apache.drill.exec.record.metadata.ColumnMetadata;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class ProjectionChecker {
+ private static final Logger logger = LoggerFactory.getLogger(ProjectionChecker.class);
+
+ private ProjectionChecker() { }
+
+ public static boolean isConsistent(RequestedTuple tuple, ColumnMetadata readCol) {
+ if (tuple == null || !tuple.isProjected(readCol.name())) {
+ return true;
+ }
+ // If the column is projected, it may be projected implicitly.
+ // Only check explicit projection.
+ RequestedColumn col = tuple.get(readCol.name());
+ if (col == null) {
+ return true;
+ } else {
+ return isConsistent(col, readCol);
+ }
+ }
+
+ /**
+ * Does a superficial check of projection type against the given column.
+ * Does not handle subtleties such as DICT key types, actual types
+ * in a UNION, etc.
+ */
Review comment:
Can you add `@return` here, please?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [drill] paul-rogers commented on a change in pull request #1974:
DRILL-7574: Generalize the projection parser
Posted by GitBox <gi...@apache.org>.
paul-rogers commented on a change in pull request #1974: DRILL-7574: Generalize the projection parser
URL: https://github.com/apache/drill/pull/1974#discussion_r379127271
##########
File path: exec/java-exec/src/main/java/org/apache/drill/exec/physical/resultSet/project/RequestedColumn.java
##########
@@ -0,0 +1,137 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.physical.resultSet.project;
+
+/**
+ * Plan-time properties of a requested column.Represents
+ * a consolidated view of the set of references to a column.
+ * For example, the project list might contain:<br>
+ * <tt>SELECT columns[4], columns[8]</tt><br>
+ * <tt>SELECT a.b, a.c</tt><br>
+ * <tt>SELECT columns, columns[1]</tt><br>
+ * <tt>SELECT a, a.b</tt><br>
+ * In each case, the same column is referenced in different
+ * forms which are consolidated into this abstraction.
+ * <p>
+ * The resulting information is a "pattern": a form of reference
+ * which which a concrete type can be compatible or not. The project
+ * list does not contain sufficient information to definitively pick
+ * a type; it only excludes certain types.
+ * <p>
+ * Depending on the syntax, we can infer if a column must
+ * be an array or map. This is definitive: though we know that
+ * columns of the form above must be an array or a map,
+ * we cannot know if a simple column reference might refer
+ * to an array or map.
+ *
+ * <h4>Compatibility Rules<h4>
+ *
+ * The pattern given by projection is consistent with certain concrete types
+ * as follows. + means any number of additional qualifiers.
+ * <p>
+ * <table>
+ * <tr><th>Type</th><th>Consistent with</th></tr>
+ * <tr><td>Non-repeated MAP</td>
+ * <td>{@code a+} {@code a.b+}</td></tr>
+ * <tr><td>Repeated MAP</td>
+ * <td>{@code a+} {@code a.b+} {@code a[n].b+}</td>></tr>
+ * <tr><td>Non-repeated Scalar</td>
+ * <td>{@code a}</td></tr>
+ * <tr><td>Repeated Scalar</td>
+ * <td>{@code a} {@code a[n]}</td></tr>
+ * <tr><td>Non-repeated DICT</td>
+ * <td>{@code a} {@code a['key']}</td></tr>
+ * <tr><td>Repeated DICT</td>
+ * <td>{@code a} {@code a[n]} {@code a['key']} {@code a[n]['key']}</td></tr>
Review comment:
Agreed. Changed the wording to suggest the mappings are conceptual. I *think* we allow `m.a` for `MAP` arrays, the Project operator(?) will pull out that one column as a top-level array. Conceptually, we could do the same for `DICT`. Implementation-wise, your suggestion might be more practical.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [drill] KazydubB commented on a change in pull request #1974:
DRILL-7574: Generalize the projection parser
Posted by GitBox <gi...@apache.org>.
KazydubB commented on a change in pull request #1974: DRILL-7574: Generalize the projection parser
URL: https://github.com/apache/drill/pull/1974#discussion_r378772986
##########
File path: exec/java-exec/src/main/java/org/apache/drill/exec/physical/resultSet/project/RequestedColumn.java
##########
@@ -0,0 +1,137 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.physical.resultSet.project;
+
+/**
+ * Plan-time properties of a requested column.Represents
+ * a consolidated view of the set of references to a column.
+ * For example, the project list might contain:<br>
+ * <tt>SELECT columns[4], columns[8]</tt><br>
+ * <tt>SELECT a.b, a.c</tt><br>
+ * <tt>SELECT columns, columns[1]</tt><br>
+ * <tt>SELECT a, a.b</tt><br>
+ * In each case, the same column is referenced in different
+ * forms which are consolidated into this abstraction.
+ * <p>
+ * The resulting information is a "pattern": a form of reference
+ * which which a concrete type can be compatible or not. The project
+ * list does not contain sufficient information to definitively pick
+ * a type; it only excludes certain types.
+ * <p>
+ * Depending on the syntax, we can infer if a column must
+ * be an array or map. This is definitive: though we know that
+ * columns of the form above must be an array or a map,
+ * we cannot know if a simple column reference might refer
+ * to an array or map.
+ *
+ * <h4>Compatibility Rules<h4>
+ *
+ * The pattern given by projection is consistent with certain concrete types
+ * as follows. + means any number of additional qualifiers.
+ * <p>
+ * <table>
+ * <tr><th>Type</th><th>Consistent with</th></tr>
+ * <tr><td>Non-repeated MAP</td>
+ * <td>{@code a+} {@code a.b+}</td></tr>
+ * <tr><td>Repeated MAP</td>
+ * <td>{@code a+} {@code a.b+} {@code a[n].b+}</td>></tr>
+ * <tr><td>Non-repeated Scalar</td>
+ * <td>{@code a}</td></tr>
+ * <tr><td>Repeated Scalar</td>
+ * <td>{@code a} {@code a[n]}</td></tr>
+ * <tr><td>Non-repeated DICT</td>
+ * <td>{@code a} {@code a['key']}</td></tr>
+ * <tr><td>Repeated DICT</td>
+ * <td>{@code a} {@code a[n]} {@code a['key']} {@code a[n]['key']}</td></tr>
Review comment:
I am not sure whether `a['key']` pattern should be supported for `Repeated DICT` columns - maybe an UDF, e.g. `ARRAY<V> dict_values(DICT<K,V>)`, suits better for this purpose? As it provides clearer intentions and avoids mistakes.
What do you think?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [drill] KazydubB commented on a change in pull request #1974:
DRILL-7574: Generalize the projection parser
Posted by GitBox <gi...@apache.org>.
KazydubB commented on a change in pull request #1974: DRILL-7574: Generalize the projection parser
URL: https://github.com/apache/drill/pull/1974#discussion_r377035722
##########
File path: exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/scan/TestFileMetadataColumnParser.java
##########
@@ -236,14 +199,12 @@ public void testLegacyWildcardAndFileMetadata() {
* As above, but include implicit columns before and after the
* wildcard.
*/
-
@Test
public void testLegacyWildcardAndFileMetadataMixed() {
Path filePath = new Path("hdfs:///w/x/y/z.csv");
FileMetadataOptions options = standardOptions(filePath);
options.useLegacyWildcardExpansion(true);
- options.useLegacyExpansionLocation(false);
- FileMetadataManager metadataManager = new FileMetadataManager(
+ FileMetadataManager metadataManager = new FileMetadataManager(
Review comment:
```suggestion
FileMetadataManager metadataManager = new FileMetadataManager(
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [drill] KazydubB commented on a change in pull request #1974:
DRILL-7574: Generalize the projection parser
Posted by GitBox <gi...@apache.org>.
KazydubB commented on a change in pull request #1974: DRILL-7574: Generalize the projection parser
URL: https://github.com/apache/drill/pull/1974#discussion_r377038532
##########
File path: exec/java-exec/src/test/java/org/apache/drill/exec/physical/resultSet/impl/RowSetTestUtils.java
##########
@@ -58,6 +58,10 @@ private RowSetTestUtils() { }
new SchemaPath[] {SchemaPath.STAR_COLUMN});
}
+ public static List<SchemaPath> projectNone() {
+ return new ArrayList<>();
Review comment:
Is it safe to return unmodifiable instance, such as `Collections.emptyList()` instead?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [drill] paul-rogers commented on a change in pull request #1974:
DRILL-7574: Generalize the projection parser
Posted by GitBox <gi...@apache.org>.
paul-rogers commented on a change in pull request #1974: DRILL-7574: Generalize the projection parser
URL: https://github.com/apache/drill/pull/1974#discussion_r379117659
##########
File path: exec/java-exec/src/main/java/org/apache/drill/exec/physical/resultSet/project/Qualifier.java
##########
@@ -0,0 +1,162 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.physical.resultSet.project;
+
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+import org.apache.drill.exec.physical.resultSet.project.RequestedTuple.TupleProjectionType;
+
+/**
+ * Represents one level of qualifier for a column. Analogous to
+ * a {@code SchemaPath}, but represents the result of coalescing
+ * multiple occurrences of the same column.
+ */
+public class Qualifier implements QualifierContainer {
+ /**
+ * Marker to indicate that that a) the item is an
+ * array, and b) that all indexes are to be projected.
+ * Used when seeing both a and a[x].
+ */
+ private static final Set<Integer> ALL_INDEXES = new HashSet<>();
+
+ private Set<Integer> indexes;
+ private RequestedTuple members;
+ private Qualifier child;
+
+ @Override
+ public Qualifier qualifier() { return child; }
+
+ @Override
+ public Qualifier requireQualifier() {
+ if (child == null) {
+ child = new Qualifier();
+ }
+ return child;
+ }
+
+ public boolean isArray() {
+ return indexes != null;
+ }
+
+ public boolean hasIndexes() {
+ return isArray() && indexes != ALL_INDEXES;
+ }
+
+ public boolean hasIndex(int index) {
+ return hasIndexes() && indexes.contains(index);
+ }
+
+ public int maxIndex() {
+ if (! hasIndexes()) {
+ return 0;
Review comment:
Good question. The assumption is that we know the column to be projected as an array, and we now want to know how many indexes appeared in the projection list. Added more documentation to `RequestedColumn` to explain the details.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [drill] paul-rogers commented on a change in pull request #1974:
DRILL-7574: Generalize the projection parser
Posted by GitBox <gi...@apache.org>.
paul-rogers commented on a change in pull request #1974: DRILL-7574: Generalize the projection parser
URL: https://github.com/apache/drill/pull/1974#discussion_r380891905
##########
File path: exec/java-exec/src/main/java/org/apache/drill/exec/physical/resultSet/project/RequestedColumn.java
##########
@@ -0,0 +1,137 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.physical.resultSet.project;
+
+/**
+ * Plan-time properties of a requested column.Represents
+ * a consolidated view of the set of references to a column.
+ * For example, the project list might contain:<br>
+ * <tt>SELECT columns[4], columns[8]</tt><br>
+ * <tt>SELECT a.b, a.c</tt><br>
+ * <tt>SELECT columns, columns[1]</tt><br>
+ * <tt>SELECT a, a.b</tt><br>
+ * In each case, the same column is referenced in different
+ * forms which are consolidated into this abstraction.
+ * <p>
+ * The resulting information is a "pattern": a form of reference
+ * which which a concrete type can be compatible or not. The project
+ * list does not contain sufficient information to definitively pick
+ * a type; it only excludes certain types.
+ * <p>
+ * Depending on the syntax, we can infer if a column must
+ * be an array or map. This is definitive: though we know that
+ * columns of the form above must be an array or a map,
+ * we cannot know if a simple column reference might refer
+ * to an array or map.
+ *
+ * <h4>Compatibility Rules<h4>
+ *
+ * The pattern given by projection is consistent with certain concrete types
+ * as follows. + means any number of additional qualifiers.
+ * <p>
+ * <table>
+ * <tr><th>Type</th><th>Consistent with</th></tr>
+ * <tr><td>Non-repeated MAP</td>
+ * <td>{@code a+} {@code a.b+}</td></tr>
+ * <tr><td>Repeated MAP</td>
+ * <td>{@code a+} {@code a.b+} {@code a[n].b+}</td>></tr>
+ * <tr><td>Non-repeated Scalar</td>
+ * <td>{@code a}</td></tr>
+ * <tr><td>Repeated Scalar</td>
+ * <td>{@code a} {@code a[n]}</td></tr>
+ * <tr><td>Non-repeated DICT</td>
+ * <td>{@code a} {@code a['key']}</td></tr>
+ * <tr><td>Repeated DICT</td>
+ * <td>{@code a} {@code a[n]} {@code a['key']} {@code a[n]['key']}</td></tr>
Review comment:
@KazydubB , thanks for checking. It seems that the project operator is making up a null column since it does not know how to do the actual projection. I'll see how I can incorporate this additional restriction into the code here.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [drill] paul-rogers commented on issue #1974: DRILL-7574:
Generalize the projection parser
Posted by GitBox <gi...@apache.org>.
paul-rogers commented on issue #1974: DRILL-7574: Generalize the projection parser
URL: https://github.com/apache/drill/pull/1974#issuecomment-586634913
Failing due to the known UDF issue, other tests seem OK:
```
[ERROR] Errors:
[ERROR] TestDynamicUDFSupport.testDuplicatedJarInLocalRegistry ยป UserRemote VALIDATION...
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [drill] KazydubB commented on a change in pull request #1974:
DRILL-7574: Generalize the projection parser
Posted by GitBox <gi...@apache.org>.
KazydubB commented on a change in pull request #1974: DRILL-7574: Generalize the projection parser
URL: https://github.com/apache/drill/pull/1974#discussion_r377031167
##########
File path: exec/java-exec/src/main/java/org/apache/drill/exec/physical/resultSet/project/RequestedTuple.java
##########
@@ -19,13 +19,10 @@
import java.util.List;
-import org.apache.drill.common.expression.PathSegment;
-
/**
* Represents the set of columns projected for a tuple (row or map.)
- * The projected columns might themselves be columns, so returns a
- * projection set for such columns. Represents the set of requested
- * columns and tuples as expressed in the physical plan.
+ * Each column may have structure: a set of referenced names or
+ * array indicies.
Review comment:
```suggestion
* array indices.
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [drill] asfgit closed pull request #1974: DRILL-7574: Generalize
the projection parser
Posted by GitBox <gi...@apache.org>.
asfgit closed pull request #1974: DRILL-7574: Generalize the projection parser
URL: https://github.com/apache/drill/pull/1974
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services