You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@drill.apache.org by jacques-n <gi...@git.apache.org> on 2016/03/04 22:28:48 UTC

[GitHub] drill pull request: DRILL-4474: Ensure that ConvertCountToDirectSc...

GitHub user jacques-n opened a pull request:

    https://github.com/apache/drill/pull/406

    DRILL-4474: Ensure that ConvertCountToDirectScan only pushes through project when project is trivial.

    There are probably additional optimizations here but as the code stood, we completely ignored whatever was in the project above the scan below the count(x).

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/jacques-n/drill DRILL-4474

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/drill/pull/406.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #406
    
----
commit 5ab13c68357008c07a86386aa14f6f92d8584e9b
Author: Jacques Nadeau <ja...@apache.org>
Date:   2016-03-04T21:27:26Z

    DRILL-4474: Ensure that ConvertCountToDirectScan only pushes through project when project is trivial.

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request: DRILL-4474: Ensure that ConvertCountToDirectSc...

Posted by jinfengni <gi...@git.apache.org>.
Github user jinfengni commented on a diff in the pull request:

    https://github.com/apache/drill/pull/406#discussion_r55444463
  
    --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/ConvertCountToDirectScan.java ---
    @@ -103,6 +104,10 @@ public void onMatch(RelOptRuleCall call) {
           return;
         }
     
    +    if (proj != null && !ProjectRemoveRule.isTrivial(proj)) {
    --- End diff --
    
    I have a patch, which works fine for Jacque's new unit test. It continues to use directScan for simple count query. The patch is pending pre-commit & unit test run. Will update results shortly. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request: DRILL-4474: Ensure that ConvertCountToDirectSc...

Posted by jinfengni <gi...@git.apache.org>.
Github user jinfengni commented on a diff in the pull request:

    https://github.com/apache/drill/pull/406#discussion_r55428219
  
    --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/ConvertCountToDirectScan.java ---
    @@ -103,6 +104,10 @@ public void onMatch(RelOptRuleCall call) {
           return;
         }
     
    +    if (proj != null && !ProjectRemoveRule.isTrivial(proj)) {
    --- End diff --
    
    With the patch, the following query will not use directScan. 
    
    {code}
    select count(*) from cp.`tpch/nation.parquet`;
    {code}
    
    {code}
    00-00    Screen : rowType = RecordType(BIGINT EXPR$0): rowcount = 1.0, cumulative cost = {75.1 rows, 425.1 cpu, 0.0 io, 0.0 network, 0.0 memory}, id = 169
    00-01      Project(EXPR$0=[$0]) : rowType = RecordType(BIGINT EXPR$0): rowcount = 1.0, cumulative cost = {75.0 rows, 425.0 cpu, 0.0 io, 0.0 network, 0.0 memory}, id = 168
    00-02        StreamAgg(group=[{}], EXPR$0=[COUNT()]) : rowType = RecordType(BIGINT EXPR$0): rowcount = 1.0, cumulative cost = {75.0 rows, 425.0 cpu, 0.0 io, 0.0 network, 0.0 memory}, id = 167
    00-03          Project($f0=[0]) : rowType = RecordType(INTEGER $f0): rowcount = 25.0, cumulative cost = {50.0 rows, 125.0 cpu, 0.0 io, 0.0 network, 0.0 memory}, id = 166
    00-04            Scan(groupscan=[ParquetGroupScan [entries=[ReadEntryWithPath [path=classpath:/tpch/nation.parquet]], selectionRoot=classpath:/tpch/nation.parquet, numFiles=1, usedMetadataFile=false, columns=[]]]) : rowType = RecordType(): rowcount = 25.0, cumulative cost = {25.0 rows, 25.0 cpu, 0.0 io, 0.0 network, 0.0 memory}, id = 165
    {code}
    
    I debug a bit. Seems Line 115 is fine. But something is worng in the code Line 117 - 123. 



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request: DRILL-4474: Ensure that ConvertCountToDirectSc...

Posted by amansinha100 <gi...@git.apache.org>.
Github user amansinha100 commented on the pull request:

    https://github.com/apache/drill/pull/406#issuecomment-193971973
  
    Agree with @jinfengni that the current fix can cause performance regression for simpler count queries.  I will change my review to -1 and let's see how to get the proper nullability check. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request: DRILL-4474: Ensure that ConvertCountToDirectSc...

Posted by jinfengni <gi...@git.apache.org>.
Github user jinfengni commented on a diff in the pull request:

    https://github.com/apache/drill/pull/406#discussion_r55421789
  
    --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/ConvertCountToDirectScan.java ---
    @@ -103,6 +104,10 @@ public void onMatch(RelOptRuleCall call) {
           return;
         }
     
    +    if (proj != null && !ProjectRemoveRule.isTrivial(proj)) {
    --- End diff --
    
    I feel that this check might over-kill some optimization opportunity.  For example,
    
    select count(100) 
    from `parquetTable`;
    
    In this case, count(100) is equal to rowcount in parquet table. However, the project is not a trial project, meaning the new code will disable the optimization.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request: DRILL-4474: Ensure that ConvertCountToDirectSc...

Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:

    https://github.com/apache/drill/pull/406


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request: DRILL-4474: Ensure that ConvertCountToDirectSc...

Posted by jinfengni <gi...@git.apache.org>.
Github user jinfengni commented on a diff in the pull request:

    https://github.com/apache/drill/pull/406#discussion_r55422075
  
    --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/ConvertCountToDirectScan.java ---
    @@ -103,6 +104,10 @@ public void onMatch(RelOptRuleCall call) {
           return;
         }
     
    +    if (proj != null && !ProjectRemoveRule.isTrivial(proj)) {
    --- End diff --
    
    We have a check whether the input to count() is nullable in Line 115. In theory, if the input is non-nullable, then count(non-nullalbe expression) = rowcount.
    
    My guess is that the query (case expression) with incorrect result is caused by the wrong type resolution for the case expression. 
    
     


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] drill pull request: DRILL-4474: Ensure that ConvertCountToDirectSc...

Posted by amansinha100 <gi...@git.apache.org>.
Github user amansinha100 commented on the pull request:

    https://github.com/apache/drill/pull/406#issuecomment-192531157
  
    +1.   Yes, seems like the project was not being checked at all and the nullability check did not work for expressions inside the count aggregate. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---