You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by GitBox <gi...@apache.org> on 2022/09/29 21:30:03 UTC

[GitHub] [pinot] agavra opened a new pull request, #9497: add debug string to query plan [MINOR]

agavra opened a new pull request, #9497:
URL: https://github.com/apache/pinot/pull/9497

   Adds functionality to be able to call `QueryPlan#debugString()` to see an `EXPLAIN`-like string. Here are some examples:
   
   `SELECT * FROM b ORDER BY col1, col2 DESC LIMIT 3`:
   ```
   MAIL_RECEIVE
       MAIL_SEND
           SORT (LIMIT 3)
               MAIL_RECEIVE
                   MAIL_SEND
                       TABLE SCAN (b)
   ```
   
   `SELECT * FROM a JOIN b ON a.col1 = b.col1 JOIN c ON a.col3 = c.col3`:
   ```
   MAIL_RECEIVE
       MAIL_SEND
           JOIN
               MAIL_RECEIVE
                   MAIL_SEND
                       JOIN
                           MAIL_RECEIVE
                               MAIL_SEND
                                   TABLE SCAN (a)
                           MAIL_RECEIVE
                               MAIL_SEND
                                   TABLE SCAN (b)
               MAIL_RECEIVE
                   MAIL_SEND
                       TABLE SCAN (c)
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] siddharthteotia commented on pull request #9497: [multi stage] add debug string to query plan

Posted by GitBox <gi...@apache.org>.
siddharthteotia commented on PR #9497:
URL: https://github.com/apache/pinot/pull/9497#issuecomment-1263957519

   We should consider doing a follow-up to wire this into EXPLAIN - https://github.com/apache/pinot/pull/9280


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] agavra commented on a diff in pull request #9497: add debug string to query plan [MINOR]

Posted by GitBox <gi...@apache.org>.
agavra commented on code in PR #9497:
URL: https://github.com/apache/pinot/pull/9497#discussion_r984779048


##########
pinot-query-planner/src/main/java/org/apache/pinot/query/planner/stage/AggregateNode.java:
##########
@@ -54,4 +54,9 @@ public List<RexExpression> getAggCalls() {
   public List<RexExpression> getGroupSet() {
     return _groupSet;
   }
+
+  @Override
+  public String debugString() {

Review Comment:
   @siddharthteotia you read my mind 😉 that's _just_ what I was working on (I'll update PR description with examples once I've worked out the bugs).



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] codecov-commenter commented on pull request #9497: add debug string to query plan [MINOR]

Posted by GitBox <gi...@apache.org>.
codecov-commenter commented on PR #9497:
URL: https://github.com/apache/pinot/pull/9497#issuecomment-1262927951

   # [Codecov](https://codecov.io/gh/apache/pinot/pull/9497?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#9497](https://codecov.io/gh/apache/pinot/pull/9497?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (01e3af3) into [master](https://codecov.io/gh/apache/pinot/commit/3057712fdc468951d72be9d676c17b5a8f0af850?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (3057712) will **increase** coverage by `1.18%`.
   > The diff coverage is `0.00%`.
   
   ```diff
   @@             Coverage Diff              @@
   ##             master    #9497      +/-   ##
   ============================================
   + Coverage     68.66%   69.85%   +1.18%     
   + Complexity     5136     4869     -267     
   ============================================
     Files          1916     1917       +1     
     Lines        102069   102233     +164     
     Branches      15481    15511      +30     
   ============================================
   + Hits          70083    71412    +1329     
   + Misses        26980    25781    -1199     
   - Partials       5006     5040      +34     
   ```
   
   | Flag | Coverage Δ | |
   |---|---|---|
   | integration1 | `25.96% <0.00%> (-0.17%)` | :arrow_down: |
   | integration2 | `24.73% <0.00%> (?)` | |
   | unittests1 | `67.14% <0.00%> (+0.01%)` | :arrow_up: |
   | unittests2 | `15.50% <0.00%> (-0.02%)` | :arrow_down: |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   | [Impacted Files](https://codecov.io/gh/apache/pinot/pull/9497?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
   |---|---|---|
   | [...java/org/apache/pinot/query/planner/QueryPlan.java](https://codecov.io/gh/apache/pinot/pull/9497/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtcXVlcnktcGxhbm5lci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvcXVlcnkvcGxhbm5lci9RdWVyeVBsYW4uamF2YQ==) | `28.57% <0.00%> (-71.43%)` | :arrow_down: |
   | [...pache/pinot/query/planner/stage/AggregateNode.java](https://codecov.io/gh/apache/pinot/pull/9497/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtcXVlcnktcGxhbm5lci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvcXVlcnkvcGxhbm5lci9zdGFnZS9BZ2dyZWdhdGVOb2RlLmphdmE=) | `91.66% <0.00%> (-8.34%)` | :arrow_down: |
   | [...g/apache/pinot/query/planner/stage/FilterNode.java](https://codecov.io/gh/apache/pinot/pull/9497/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtcXVlcnktcGxhbm5lci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvcXVlcnkvcGxhbm5lci9zdGFnZS9GaWx0ZXJOb2RlLmphdmE=) | `85.71% <0.00%> (-14.29%)` | :arrow_down: |
   | [...org/apache/pinot/query/planner/stage/JoinNode.java](https://codecov.io/gh/apache/pinot/pull/9497/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtcXVlcnktcGxhbm5lci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvcXVlcnkvcGxhbm5lci9zdGFnZS9Kb2luTm9kZS5qYXZh) | `94.73% <0.00%> (-5.27%)` | :arrow_down: |
   | [.../pinot/query/planner/stage/MailboxReceiveNode.java](https://codecov.io/gh/apache/pinot/pull/9497/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtcXVlcnktcGxhbm5lci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvcXVlcnkvcGxhbm5lci9zdGFnZS9NYWlsYm94UmVjZWl2ZU5vZGUuamF2YQ==) | `90.90% <0.00%> (-9.10%)` | :arrow_down: |
   | [...che/pinot/query/planner/stage/MailboxSendNode.java](https://codecov.io/gh/apache/pinot/pull/9497/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtcXVlcnktcGxhbm5lci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvcXVlcnkvcGxhbm5lci9zdGFnZS9NYWlsYm94U2VuZE5vZGUuamF2YQ==) | `90.90% <0.00%> (-9.10%)` | :arrow_down: |
   | [.../apache/pinot/query/planner/stage/ProjectNode.java](https://codecov.io/gh/apache/pinot/pull/9497/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtcXVlcnktcGxhbm5lci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvcXVlcnkvcGxhbm5lci9zdGFnZS9Qcm9qZWN0Tm9kZS5qYXZh) | `85.71% <0.00%> (-14.29%)` | :arrow_down: |
   | [...org/apache/pinot/query/planner/stage/SortNode.java](https://codecov.io/gh/apache/pinot/pull/9497/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtcXVlcnktcGxhbm5lci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvcXVlcnkvcGxhbm5lci9zdGFnZS9Tb3J0Tm9kZS5qYXZh) | `94.11% <0.00%> (-5.89%)` | :arrow_down: |
   | [...pache/pinot/query/planner/stage/TableScanNode.java](https://codecov.io/gh/apache/pinot/pull/9497/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtcXVlcnktcGxhbm5lci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvcXVlcnkvcGxhbm5lci9zdGFnZS9UYWJsZVNjYW5Ob2RlLmphdmE=) | `88.88% <0.00%> (-11.12%)` | :arrow_down: |
   | [...rg/apache/pinot/query/planner/stage/ValueNode.java](https://codecov.io/gh/apache/pinot/pull/9497/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtcXVlcnktcGxhbm5lci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvcXVlcnkvcGxhbm5lci9zdGFnZS9WYWx1ZU5vZGUuamF2YQ==) | `28.57% <0.00%> (-2.20%)` | :arrow_down: |
   | ... and [163 more](https://codecov.io/gh/apache/pinot/pull/9497/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
   
   :mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] siddharthteotia commented on a diff in pull request #9497: add debug string to query plan [MINOR]

Posted by GitBox <gi...@apache.org>.
siddharthteotia commented on code in PR #9497:
URL: https://github.com/apache/pinot/pull/9497#discussion_r984094036


##########
pinot-query-planner/src/main/java/org/apache/pinot/query/planner/stage/AggregateNode.java:
##########
@@ -54,4 +54,9 @@ public List<RexExpression> getAggCalls() {
   public List<RexExpression> getGroupSet() {
     return _groupSet;
   }
+
+  @Override
+  public String debugString() {

Review Comment:
   In the existing engine, we implement `toExplainString` at the physical operator level and each operator also has a EXPLAIN_NAME. 
   
   Even in this case, I think the API could be called `explainString` or something similar even though we are doing at the logical node level and not at the operator level. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] siddharthteotia merged pull request #9497: add debug string to query plan [MINOR]

Posted by GitBox <gi...@apache.org>.
siddharthteotia merged PR #9497:
URL: https://github.com/apache/pinot/pull/9497


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] siddharthteotia commented on a diff in pull request #9497: add debug string to query plan [MINOR]

Posted by GitBox <gi...@apache.org>.
siddharthteotia commented on code in PR #9497:
URL: https://github.com/apache/pinot/pull/9497#discussion_r984095507


##########
pinot-query-planner/src/main/java/org/apache/pinot/query/planner/stage/AggregateNode.java:
##########
@@ -54,4 +54,9 @@ public List<RexExpression> getAggCalls() {
   public List<RexExpression> getGroupSet() {
     return _groupSet;
   }
+
+  @Override
+  public String debugString() {

Review Comment:
   Ohh I just noticed this is not wired into the EXPLAIN PLAN. We do support EXPLAIN for multi-stage btw. We can probably also do this by wiring this up with EXPLAIN PLAN for multi-stage (can do as a follow up as well).
   
    The output in the description looks reasonably good



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] siddharthteotia commented on a diff in pull request #9497: add debug string to query plan [MINOR]

Posted by GitBox <gi...@apache.org>.
siddharthteotia commented on code in PR #9497:
URL: https://github.com/apache/pinot/pull/9497#discussion_r984774728


##########
pinot-query-planner/src/main/java/org/apache/pinot/query/planner/stage/AggregateNode.java:
##########
@@ -54,4 +54,9 @@ public List<RexExpression> getAggCalls() {
   public List<RexExpression> getGroupSet() {
     return _groupSet;
   }
+
+  @Override
+  public String debugString() {

Review Comment:
   can we also add instances / worker assignment into the output for the overall plan ?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] siddharthteotia commented on a diff in pull request #9497: add debug string to query plan [MINOR]

Posted by GitBox <gi...@apache.org>.
siddharthteotia commented on code in PR #9497:
URL: https://github.com/apache/pinot/pull/9497#discussion_r984095507


##########
pinot-query-planner/src/main/java/org/apache/pinot/query/planner/stage/AggregateNode.java:
##########
@@ -54,4 +54,9 @@ public List<RexExpression> getAggCalls() {
   public List<RexExpression> getGroupSet() {
     return _groupSet;
   }
+
+  @Override
+  public String debugString() {

Review Comment:
   Ohh I just noticed this is not wired into the EXPLAIN PLAN. We do support EXPLAIN (just the logical at this point) for multi-stage btw. We can probably also do this by wiring this up with EXPLAIN PLAN for multi-stage (can do as a follow up as well).
   
    The output in the description looks reasonably good



##########
pinot-query-planner/src/main/java/org/apache/pinot/query/planner/stage/AggregateNode.java:
##########
@@ -54,4 +54,9 @@ public List<RexExpression> getAggCalls() {
   public List<RexExpression> getGroupSet() {
     return _groupSet;
   }
+
+  @Override
+  public String debugString() {

Review Comment:
   Ohh I just noticed this is not wired into the EXPLAIN PLAN. We do support EXPLAIN (just the logical at this point iirc) for multi-stage btw. We can probably also do this by wiring this up with EXPLAIN PLAN for multi-stage (can do as a follow up as well).
   
    The output in the description looks reasonably good



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] siddharthteotia commented on a diff in pull request #9497: add debug string to query plan [MINOR]

Posted by GitBox <gi...@apache.org>.
siddharthteotia commented on code in PR #9497:
URL: https://github.com/apache/pinot/pull/9497#discussion_r984774728


##########
pinot-query-planner/src/main/java/org/apache/pinot/query/planner/stage/AggregateNode.java:
##########
@@ -54,4 +54,9 @@ public List<RexExpression> getAggCalls() {
   public List<RexExpression> getGroupSet() {
     return _groupSet;
   }
+
+  @Override
+  public String debugString() {

Review Comment:
   can we also add instances / worker assignment into the output ?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org