You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@drill.apache.org by arina-ielchiieva <gi...@git.apache.org> on 2018/01/31 14:10:56 UTC

[GitHub] drill pull request #1104: DRILL-6118: Handle item star columns during projec...

GitHub user arina-ielchiieva opened a pull request:

    https://github.com/apache/drill/pull/1104

    DRILL-6118: Handle item star columns during project / filter push dow…

    …n and directory pruning
    
    1. Added DrillFilterItemStarReWriterRule to re-write item star fields to regular field references.
    2. Refactored DrillPushProjectIntoScanRule to handle item star fields, factored out helper classes and methods from PreUitl.class.
    3. Fixed issue with dynamic star usage (after Calcite upgrade old usage of star was still present, replaced WILDCARD -> DYNAMIC_STAR  for clarity).
    4. Added unit tests to check project / filter push down and directory pruning with item star.
    
    Details in [DRILL-6118](https://issues.apache.org/jira/browse/DRILL-6118).

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/arina-ielchiieva/drill DRILL-6118

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/drill/pull/1104.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #1104
    
----
commit 4673bfb593ca6422d58fa9e0e6eb281a69f1ed69
Author: Arina Ielchiieva <ar...@...>
Date:   2017-12-21T17:31:00Z

    DRILL-6118: Handle item star columns during project / filter push down and directory pruning
    
    1. Added DrillFilterItemStarReWriterRule to re-write item star fields to regular field references.
    2. Refactored DrillPushProjectIntoScanRule to handle item star fields, factored out helper classes and methods from PreUitl.class.
    3. Fixed issue with dynamic star usage (after Calcite upgrade old usage of star was still present, replaced WILDCARD -> DYNAMIC_STAR  for clarity).
    4. Added unit tests to check project / filter push down and directory pruning with item star.

----


---

[GitHub] drill issue #1104: DRILL-6118: Handle item star columns during project / fil...

Posted by arina-ielchiieva <gi...@git.apache.org>.
Github user arina-ielchiieva commented on the issue:

    https://github.com/apache/drill/pull/1104
  
    @amansinha100 rebased and ran pre-commit tests. All good expept of 16 expected failures.


---

[GitHub] drill issue #1104: DRILL-6118: Handle item star columns during project / fil...

Posted by vvysotskyi <gi...@git.apache.org>.
Github user vvysotskyi commented on the issue:

    https://github.com/apache/drill/pull/1104
  
    @paul-rogers, as you know, there were some differences between the commits in the older Calcite version (1.4.0-r23) which was used by Drill and commits merged into Apache master. One of such commits was `[CALCITE-1150] Add dynamic record type and dynamic star for schema-on-read table`. In our older Calcite fork `DYNAMIC_STAR` was `*`, but after rebase onto Calcite 1.15, it became `**`.
    
    As I understand, `DYNAMIC_STAR` itself means "default column" which will be added to table row type for the case when we have `select *` query and the schema of the underlying table is not known. 
    
    As for the name of constant, I think current `DYNAMIC_STAR` more suitable, since Calcite also uses a similar name for the same string: `DynamicRecordType.DYNAMIC_STAR_PREFIX`. So current name at least helps to avoid divergence in naming for Drill and Calcite.


---

[GitHub] drill pull request #1104: DRILL-6118: Handle item star columns during projec...

Posted by chunhui-shi <gi...@git.apache.org>.
Github user chunhui-shi commented on a diff in the pull request:

    https://github.com/apache/drill/pull/1104#discussion_r166088073
  
    --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/DrillFilterItemStarReWriterRule.java ---
    @@ -0,0 +1,232 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one
    + * or more contributor license agreements.  See the NOTICE file
    + * distributed with this work for additional information
    + * regarding copyright ownership.  The ASF licenses this file
    + * to you under the Apache License, Version 2.0 (the
    + * "License"); you may not use this file except in compliance
    + * with the License.  You may obtain a copy of the License at
    + *
    + * http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +package org.apache.drill.exec.planner.logical;
    +
    +import com.google.common.collect.ImmutableList;
    +import com.google.common.collect.ImmutableSet;
    +import org.apache.calcite.adapter.enumerable.EnumerableTableScan;
    +import org.apache.calcite.plan.RelOptRule;
    +import org.apache.calcite.plan.RelOptRuleCall;
    +import org.apache.calcite.plan.RelOptRuleOperand;
    +import org.apache.calcite.plan.RelOptTable;
    +import org.apache.calcite.prepare.RelOptTableImpl;
    +import org.apache.calcite.rel.RelNode;
    +import org.apache.calcite.rel.core.CorrelationId;
    +import org.apache.calcite.rel.core.Filter;
    +import org.apache.calcite.rel.core.Project;
    +import org.apache.calcite.rel.core.TableScan;
    +import org.apache.calcite.rel.logical.LogicalFilter;
    +import org.apache.calcite.rel.logical.LogicalProject;
    +import org.apache.calcite.rel.type.RelDataType;
    +import org.apache.calcite.rel.type.RelDataTypeFactory;
    +import org.apache.calcite.rel.type.RelDataTypeField;
    +import org.apache.calcite.rex.RexCall;
    +import org.apache.calcite.rex.RexInputRef;
    +import org.apache.calcite.rex.RexNode;
    +import org.apache.calcite.rex.RexVisitorImpl;
    +import org.apache.calcite.schema.Table;
    +import org.apache.drill.exec.planner.types.RelDataTypeDrillImpl;
    +import org.apache.drill.exec.planner.types.RelDataTypeHolder;
    +import org.apache.drill.exec.util.Utilities;
    +
    +import java.util.ArrayList;
    +import java.util.Collection;
    +import java.util.HashMap;
    +import java.util.List;
    +import java.util.Map;
    +
    +import static org.apache.drill.exec.planner.logical.FieldsReWriterUtil.DesiredField;
    +import static org.apache.drill.exec.planner.logical.FieldsReWriterUtil.FieldsReWriter;
    +
    +/**
    + * Rule will transform filter -> project -> scan call with item star fields in filter
    + * into project -> filter -> project -> scan where item star fields are pushed into scan
    + * and replaced with actual field references.
    + *
    + * This will help partition pruning and push down rules to detect fields that can be pruned or push downed.
    + * Item star operator appears when sub-select or cte with star are used as source.
    + */
    +public class DrillFilterItemStarReWriterRule extends RelOptRule {
    +
    +  public static final DrillFilterItemStarReWriterRule INSTANCE = new DrillFilterItemStarReWriterRule(
    +      RelOptHelper.some(Filter.class, RelOptHelper.some(Project.class, RelOptHelper.any( TableScan.class))),
    +      "DrillFilterItemStarReWriterRule");
    +
    +  private DrillFilterItemStarReWriterRule(RelOptRuleOperand operand, String id) {
    +    super(operand, id);
    +  }
    +
    +  @Override
    +  public void onMatch(RelOptRuleCall call) {
    +    Filter filterRel = call.rel(0);
    +    Project projectRel = call.rel(1);
    +    TableScan scanRel = call.rel(2);
    +
    +    ItemStarFieldsVisitor itemStarFieldsVisitor = new ItemStarFieldsVisitor(filterRel.getRowType().getFieldNames());
    --- End diff --
    
    To make sure this works for complex expressions, could you add tests similar to testPushProjectIntoScan to test some expressions that include more than two fields in expressions in project(e.g. "select a+b") and in filter(e.g. "where a+b < 5")?  


---

[GitHub] drill issue #1104: DRILL-6118: Handle item star columns during project / fil...

Posted by amansinha100 <gi...@git.apache.org>.
Github user amansinha100 commented on the issue:

    https://github.com/apache/drill/pull/1104
  
    +1


---

[GitHub] drill issue #1104: DRILL-6118: Handle item star columns during project / fil...

Posted by paul-rogers <gi...@git.apache.org>.
Github user paul-rogers commented on the issue:

    https://github.com/apache/drill/pull/1104
  
    I noticed this change when merging with the latest master. What does `DYNAMIC_STAR` mean? The original `WILDCARD` name was more theoretical. How is `DYNAMIC_STAR` different from `STAR` (other than being `**`)?
    
    Can we call it `WILDCARD` and `isWildcard()` that the string is `**`, presumably for internal reasons, seems less worth calling out in the name.


---

[GitHub] drill pull request #1104: DRILL-6118: Handle item star columns during projec...

Posted by arina-ielchiieva <gi...@git.apache.org>.
Github user arina-ielchiieva closed the pull request at:

    https://github.com/apache/drill/pull/1104


---

[GitHub] drill pull request #1104: DRILL-6118: Handle item star columns during projec...

Posted by chunhui-shi <gi...@git.apache.org>.
Github user chunhui-shi commented on a diff in the pull request:

    https://github.com/apache/drill/pull/1104#discussion_r166066830
  
    --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/project/ProjectRecordBatch.java ---
    @@ -596,10 +596,10 @@ private void classifyExpr(final NamedExpression ex, final RecordBatch incoming,
         final NameSegment ref = ex.getRef().getRootSegment();
         final boolean exprHasPrefix = expr.getPath().contains(StarColumnHelper.PREFIX_DELIMITER);
         final boolean refHasPrefix = ref.getPath().contains(StarColumnHelper.PREFIX_DELIMITER);
    -    final boolean exprIsStar = expr.getPath().equals(SchemaPath.WILDCARD);
    -    final boolean refContainsStar = ref.getPath().contains(SchemaPath.WILDCARD);
    -    final boolean exprContainsStar = expr.getPath().contains(SchemaPath.WILDCARD);
    -    final boolean refEndsWithStar = ref.getPath().endsWith(SchemaPath.WILDCARD);
    +    final boolean exprIsStar = expr.getPath().equals(SchemaPath.DYNAMIC_STAR);
    --- End diff --
    
    Why don't we need to handle WILDCARD case anymore?


---

[GitHub] drill pull request #1104: DRILL-6118: Handle item star columns during projec...

Posted by arina-ielchiieva <gi...@git.apache.org>.
Github user arina-ielchiieva commented on a diff in the pull request:

    https://github.com/apache/drill/pull/1104#discussion_r168801629
  
    --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/DrillFilterItemStarReWriterRule.java ---
    @@ -0,0 +1,232 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one
    + * or more contributor license agreements.  See the NOTICE file
    + * distributed with this work for additional information
    + * regarding copyright ownership.  The ASF licenses this file
    + * to you under the Apache License, Version 2.0 (the
    + * "License"); you may not use this file except in compliance
    + * with the License.  You may obtain a copy of the License at
    + *
    + * http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +package org.apache.drill.exec.planner.logical;
    +
    +import com.google.common.collect.ImmutableList;
    +import com.google.common.collect.ImmutableSet;
    +import org.apache.calcite.adapter.enumerable.EnumerableTableScan;
    +import org.apache.calcite.plan.RelOptRule;
    +import org.apache.calcite.plan.RelOptRuleCall;
    +import org.apache.calcite.plan.RelOptRuleOperand;
    +import org.apache.calcite.plan.RelOptTable;
    +import org.apache.calcite.prepare.RelOptTableImpl;
    +import org.apache.calcite.rel.RelNode;
    +import org.apache.calcite.rel.core.CorrelationId;
    +import org.apache.calcite.rel.core.Filter;
    +import org.apache.calcite.rel.core.Project;
    +import org.apache.calcite.rel.core.TableScan;
    +import org.apache.calcite.rel.logical.LogicalFilter;
    +import org.apache.calcite.rel.logical.LogicalProject;
    +import org.apache.calcite.rel.type.RelDataType;
    +import org.apache.calcite.rel.type.RelDataTypeFactory;
    +import org.apache.calcite.rel.type.RelDataTypeField;
    +import org.apache.calcite.rex.RexCall;
    +import org.apache.calcite.rex.RexInputRef;
    +import org.apache.calcite.rex.RexNode;
    +import org.apache.calcite.rex.RexVisitorImpl;
    +import org.apache.calcite.schema.Table;
    +import org.apache.drill.exec.planner.types.RelDataTypeDrillImpl;
    +import org.apache.drill.exec.planner.types.RelDataTypeHolder;
    +import org.apache.drill.exec.util.Utilities;
    +
    +import java.util.ArrayList;
    +import java.util.Collection;
    +import java.util.HashMap;
    +import java.util.List;
    +import java.util.Map;
    +
    +import static org.apache.drill.exec.planner.logical.FieldsReWriterUtil.DesiredField;
    +import static org.apache.drill.exec.planner.logical.FieldsReWriterUtil.FieldsReWriter;
    +
    +/**
    + * Rule will transform filter -> project -> scan call with item star fields in filter
    + * into project -> filter -> project -> scan where item star fields are pushed into scan
    + * and replaced with actual field references.
    + *
    + * This will help partition pruning and push down rules to detect fields that can be pruned or push downed.
    + * Item star operator appears when sub-select or cte with star are used as source.
    + */
    +public class DrillFilterItemStarReWriterRule extends RelOptRule {
    +
    +  public static final DrillFilterItemStarReWriterRule INSTANCE = new DrillFilterItemStarReWriterRule(
    +      RelOptHelper.some(Filter.class, RelOptHelper.some(Project.class, RelOptHelper.any( TableScan.class))),
    +      "DrillFilterItemStarReWriterRule");
    +
    +  private DrillFilterItemStarReWriterRule(RelOptRuleOperand operand, String id) {
    +    super(operand, id);
    +  }
    +
    +  @Override
    +  public void onMatch(RelOptRuleCall call) {
    +    Filter filterRel = call.rel(0);
    +    Project projectRel = call.rel(1);
    +    TableScan scanRel = call.rel(2);
    +
    +    ItemStarFieldsVisitor itemStarFieldsVisitor = new ItemStarFieldsVisitor(filterRel.getRowType().getFieldNames());
    --- End diff --
    
    @chunhui-shi added more unit tests. Please review.


---

[GitHub] drill issue #1104: DRILL-6118: Handle item star columns during project / fil...

Posted by amansinha100 <gi...@git.apache.org>.
Github user amansinha100 commented on the issue:

    https://github.com/apache/drill/pull/1104
  
    @arina-ielchiieva pls rebase on latest master. 


---

[GitHub] drill issue #1104: DRILL-6118: Handle item star columns during project / fil...

Posted by arina-ielchiieva <gi...@git.apache.org>.
Github user arina-ielchiieva commented on the issue:

    https://github.com/apache/drill/pull/1104
  
    @chunhui-shi please review.


---

[GitHub] drill pull request #1104: DRILL-6118: Handle item star columns during projec...

Posted by chunhui-shi <gi...@git.apache.org>.
Github user chunhui-shi commented on a diff in the pull request:

    https://github.com/apache/drill/pull/1104#discussion_r166094020
  
    --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/DrillFilterItemStarReWriterRule.java ---
    @@ -0,0 +1,232 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one
    + * or more contributor license agreements.  See the NOTICE file
    + * distributed with this work for additional information
    + * regarding copyright ownership.  The ASF licenses this file
    + * to you under the Apache License, Version 2.0 (the
    + * "License"); you may not use this file except in compliance
    + * with the License.  You may obtain a copy of the License at
    + *
    + * http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +package org.apache.drill.exec.planner.logical;
    +
    +import com.google.common.collect.ImmutableList;
    +import com.google.common.collect.ImmutableSet;
    +import org.apache.calcite.adapter.enumerable.EnumerableTableScan;
    +import org.apache.calcite.plan.RelOptRule;
    +import org.apache.calcite.plan.RelOptRuleCall;
    +import org.apache.calcite.plan.RelOptRuleOperand;
    +import org.apache.calcite.plan.RelOptTable;
    +import org.apache.calcite.prepare.RelOptTableImpl;
    +import org.apache.calcite.rel.RelNode;
    +import org.apache.calcite.rel.core.CorrelationId;
    +import org.apache.calcite.rel.core.Filter;
    +import org.apache.calcite.rel.core.Project;
    +import org.apache.calcite.rel.core.TableScan;
    +import org.apache.calcite.rel.logical.LogicalFilter;
    +import org.apache.calcite.rel.logical.LogicalProject;
    +import org.apache.calcite.rel.type.RelDataType;
    +import org.apache.calcite.rel.type.RelDataTypeFactory;
    +import org.apache.calcite.rel.type.RelDataTypeField;
    +import org.apache.calcite.rex.RexCall;
    +import org.apache.calcite.rex.RexInputRef;
    +import org.apache.calcite.rex.RexNode;
    +import org.apache.calcite.rex.RexVisitorImpl;
    +import org.apache.calcite.schema.Table;
    +import org.apache.drill.exec.planner.types.RelDataTypeDrillImpl;
    +import org.apache.drill.exec.planner.types.RelDataTypeHolder;
    +import org.apache.drill.exec.util.Utilities;
    +
    +import java.util.ArrayList;
    +import java.util.Collection;
    +import java.util.HashMap;
    +import java.util.List;
    +import java.util.Map;
    +
    +import static org.apache.drill.exec.planner.logical.FieldsReWriterUtil.DesiredField;
    +import static org.apache.drill.exec.planner.logical.FieldsReWriterUtil.FieldsReWriter;
    +
    +/**
    + * Rule will transform filter -> project -> scan call with item star fields in filter
    + * into project -> filter -> project -> scan where item star fields are pushed into scan
    + * and replaced with actual field references.
    + *
    + * This will help partition pruning and push down rules to detect fields that can be pruned or push downed.
    + * Item star operator appears when sub-select or cte with star are used as source.
    + */
    +public class DrillFilterItemStarReWriterRule extends RelOptRule {
    +
    +  public static final DrillFilterItemStarReWriterRule INSTANCE = new DrillFilterItemStarReWriterRule(
    +      RelOptHelper.some(Filter.class, RelOptHelper.some(Project.class, RelOptHelper.any( TableScan.class))),
    +      "DrillFilterItemStarReWriterRule");
    +
    +  private DrillFilterItemStarReWriterRule(RelOptRuleOperand operand, String id) {
    +    super(operand, id);
    +  }
    +
    +  @Override
    +  public void onMatch(RelOptRuleCall call) {
    +    Filter filterRel = call.rel(0);
    +    Project projectRel = call.rel(1);
    +    TableScan scanRel = call.rel(2);
    +
    +    ItemStarFieldsVisitor itemStarFieldsVisitor = new ItemStarFieldsVisitor(filterRel.getRowType().getFieldNames());
    --- End diff --
    
    Other test cases should be covered are: 
    nested field names, 
    refer to two different fields under the same parent, eg. a.b and a.c.
    and array type referred in filters and projects.


---

[GitHub] drill pull request #1104: DRILL-6118: Handle item star columns during projec...

Posted by vvysotskyi <gi...@git.apache.org>.
Github user vvysotskyi commented on a diff in the pull request:

    https://github.com/apache/drill/pull/1104#discussion_r166310161
  
    --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/project/ProjectRecordBatch.java ---
    @@ -596,10 +596,10 @@ private void classifyExpr(final NamedExpression ex, final RecordBatch incoming,
         final NameSegment ref = ex.getRef().getRootSegment();
         final boolean exprHasPrefix = expr.getPath().contains(StarColumnHelper.PREFIX_DELIMITER);
         final boolean refHasPrefix = ref.getPath().contains(StarColumnHelper.PREFIX_DELIMITER);
    -    final boolean exprIsStar = expr.getPath().equals(SchemaPath.WILDCARD);
    -    final boolean refContainsStar = ref.getPath().contains(SchemaPath.WILDCARD);
    -    final boolean exprContainsStar = expr.getPath().contains(SchemaPath.WILDCARD);
    -    final boolean refEndsWithStar = ref.getPath().endsWith(SchemaPath.WILDCARD);
    +    final boolean exprIsStar = expr.getPath().equals(SchemaPath.DYNAMIC_STAR);
    --- End diff --
    
    This change became required after Calcite update. With the changes in CALCITE-1150, `*` is replaced by `**` after a query is parsed and `**` is added to the RowType. Therefore WILDCARD can't come from the plan and its usage should be replaced by `**`.


---

[GitHub] drill issue #1104: DRILL-6118: Handle item star columns during project / fil...

Posted by arina-ielchiieva <gi...@git.apache.org>.
Github user arina-ielchiieva commented on the issue:

    https://github.com/apache/drill/pull/1104
  
    Merged into Apache master with commit id 9073aed67d89e8b2188870d6c812706085c9c41b


---