You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Bikramjeet Vig (Code Review)" <ge...@cloudera.org> on 2018/10/16 20:07:12 UTC

[Impala-ASF-CR] IMPALA-7351: Add estimates to Exchange node

Bikramjeet Vig has uploaded this change for review. ( http://gerrit.cloudera.org:8080/11692


Change subject: IMPALA-7351: Add estimates to Exchange node
......................................................................

IMPALA-7351: Add estimates to Exchange node

Added rough estimates for exchange node and a justification of the
method in the in-line comments.

Testing:
Updated Planner tests.

Change-Id: I5b577f9511abc48b992e814d50bba4959f23f7fd
---
M be/src/util/backend-gflag-util.cc
M common/thrift/BackendGflags.thrift
M fe/src/main/java/org/apache/impala/planner/ExchangeNode.java
M fe/src/main/java/org/apache/impala/planner/PlanNode.java
M testdata/workloads/functional-planner/queries/PlannerTest/kudu-selectivity.test
M testdata/workloads/functional-planner/queries/PlannerTest/max-row-size.test
M testdata/workloads/functional-planner/queries/PlannerTest/resource-requirements.test
M testdata/workloads/functional-planner/queries/PlannerTest/spillable-buffer-sizing.test
M testdata/workloads/functional-planner/queries/PlannerTest/tpcds-all.test
M testdata/workloads/functional-planner/queries/PlannerTest/tpch-all.test
M testdata/workloads/functional-planner/queries/PlannerTest/tpch-nested.test
11 files changed, 856 insertions(+), 797 deletions(-)



  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/92/11692/2
-- 
To view, visit http://gerrit.cloudera.org:8080/11692
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I5b577f9511abc48b992e814d50bba4959f23f7fd
Gerrit-Change-Number: 11692
Gerrit-PatchSet: 2
Gerrit-Owner: Bikramjeet Vig <bi...@cloudera.com>

[Impala-ASF-CR] IMPALA-7351: Add estimates to Exchange node

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11692 )

Change subject: IMPALA-7351: Add estimates to Exchange node
......................................................................


Patch Set 5:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/3349/ DRY_RUN=false


-- 
To view, visit http://gerrit.cloudera.org:8080/11692
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I5b577f9511abc48b992e814d50bba4959f23f7fd
Gerrit-Change-Number: 11692
Gerrit-PatchSet: 5
Gerrit-Owner: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Ho <kw...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vuk Ercegovac <ve...@cloudera.com>
Gerrit-Comment-Date: Wed, 24 Oct 2018 23:07:17 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7351: Add estimates to Exchange node

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11692 )

Change subject: IMPALA-7351: Add estimates to Exchange node
......................................................................


Patch Set 3:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/1137/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/11692
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I5b577f9511abc48b992e814d50bba4959f23f7fd
Gerrit-Change-Number: 11692
Gerrit-PatchSet: 3
Gerrit-Owner: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Ho <kw...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vuk Ercegovac <ve...@cloudera.com>
Gerrit-Comment-Date: Tue, 23 Oct 2018 21:59:41 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7351: Add estimates to Exchange node

Posted by "Bikramjeet Vig (Code Review)" <ge...@cloudera.org>.
Hello Michael Ho, Tim Armstrong, Impala Public Jenkins, Vuk Ercegovac, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/11692

to look at the new patch set (#6).

Change subject: IMPALA-7351: Add estimates to Exchange node
......................................................................

IMPALA-7351: Add estimates to Exchange node

Added rough estimates for exchange node and a justification of the
method in the in-line comments.

Testing:
Updated Planner tests.

Change-Id: I5b577f9511abc48b992e814d50bba4959f23f7fd
---
M be/src/util/backend-gflag-util.cc
M common/thrift/BackendGflags.thrift
M fe/src/main/java/org/apache/impala/planner/ExchangeNode.java
M fe/src/main/java/org/apache/impala/planner/PlanNode.java
M testdata/workloads/functional-planner/queries/PlannerTest/kudu-selectivity.test
M testdata/workloads/functional-planner/queries/PlannerTest/max-row-size.test
M testdata/workloads/functional-planner/queries/PlannerTest/resource-requirements.test
M testdata/workloads/functional-planner/queries/PlannerTest/spillable-buffer-sizing.test
M testdata/workloads/functional-planner/queries/PlannerTest/tpcds-all.test
M testdata/workloads/functional-planner/queries/PlannerTest/tpch-all.test
M testdata/workloads/functional-planner/queries/PlannerTest/tpch-nested.test
M testdata/workloads/functional-query/queries/QueryTest/explain-level2.test
M testdata/workloads/functional-query/queries/QueryTest/explain-level3.test
13 files changed, 920 insertions(+), 831 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/92/11692/6
-- 
To view, visit http://gerrit.cloudera.org:8080/11692
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I5b577f9511abc48b992e814d50bba4959f23f7fd
Gerrit-Change-Number: 11692
Gerrit-PatchSet: 6
Gerrit-Owner: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Ho <kw...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vuk Ercegovac <ve...@cloudera.com>

[Impala-ASF-CR] IMPALA-7351: Add estimates to Exchange node

Posted by "Michael Ho (Code Review)" <ge...@cloudera.org>.
Michael Ho has removed Michael Ho from this change.  ( http://gerrit.cloudera.org:8080/11692 )

Change subject: IMPALA-7351: Add estimates to Exchange node
......................................................................


Removed reviewer Michael Ho.
-- 
To view, visit http://gerrit.cloudera.org:8080/11692
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: deleteReviewer
Gerrit-Change-Id: I5b577f9511abc48b992e814d50bba4959f23f7fd
Gerrit-Change-Number: 11692
Gerrit-PatchSet: 2
Gerrit-Owner: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Ho <kw...@cloudera.com>

[Impala-ASF-CR] IMPALA-7351: Add estimates to Exchange node

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11692 )

Change subject: IMPALA-7351: Add estimates to Exchange node
......................................................................


Patch Set 5: Verified-1

Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/3349/


-- 
To view, visit http://gerrit.cloudera.org:8080/11692
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I5b577f9511abc48b992e814d50bba4959f23f7fd
Gerrit-Change-Number: 11692
Gerrit-PatchSet: 5
Gerrit-Owner: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Ho <kw...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vuk Ercegovac <ve...@cloudera.com>
Gerrit-Comment-Date: Thu, 25 Oct 2018 02:59:07 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7351: Add estimates to Exchange node

Posted by "Tim Armstrong (Code Review)" <ge...@cloudera.org>.
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/11692 )

Change subject: IMPALA-7351: Add estimates to Exchange node
......................................................................


Patch Set 3:

(3 comments)

http://gerrit.cloudera.org:8080/#/c/11692/3//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/11692/3//COMMIT_MSG@10
PS3, Line 10: method in the in-line comments.
Do we know what the likely upper bound is for the memory estimates on larger clusters and data sets?

It seems like the worst case is probably a merging exchange on a large cluster right, since you have a large number of queues?

My understanding is that in the other cases the queue size provides a reasonable cap because there is only one queue.


http://gerrit.cloudera.org:8080/#/c/11692/3/fe/src/main/java/org/apache/impala/planner/ExchangeNode.java
File fe/src/main/java/org/apache/impala/planner/ExchangeNode.java:

http://gerrit.cloudera.org:8080/#/c/11692/3/fe/src/main/java/org/apache/impala/planner/ExchangeNode.java@77
PS3, Line 77:     if (!streamSink.getOutputPartition().isPartitioned() && fragment_.isPartitioned()) {
nit: can just return this expression


http://gerrit.cloudera.org:8080/#/c/11692/3/fe/src/main/java/org/apache/impala/planner/ExchangeNode.java@241
PS3, Line 241:     long deferredBatchQueueSize = avgRowBatchByteSize * numSenders;
nit: unnecessary intermediate variable



-- 
To view, visit http://gerrit.cloudera.org:8080/11692
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I5b577f9511abc48b992e814d50bba4959f23f7fd
Gerrit-Change-Number: 11692
Gerrit-PatchSet: 3
Gerrit-Owner: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Ho <kw...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vuk Ercegovac <ve...@cloudera.com>
Gerrit-Comment-Date: Wed, 24 Oct 2018 18:15:50 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-7351: Add estimates to Exchange node

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11692 )

Change subject: IMPALA-7351: Add estimates to Exchange node
......................................................................


Patch Set 5: Code-Review+2


-- 
To view, visit http://gerrit.cloudera.org:8080/11692
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I5b577f9511abc48b992e814d50bba4959f23f7fd
Gerrit-Change-Number: 11692
Gerrit-PatchSet: 5
Gerrit-Owner: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Ho <kw...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vuk Ercegovac <ve...@cloudera.com>
Gerrit-Comment-Date: Wed, 24 Oct 2018 23:07:16 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7351: Add estimates to Exchange node

Posted by "Tim Armstrong (Code Review)" <ge...@cloudera.org>.
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/11692 )

Change subject: IMPALA-7351: Add estimates to Exchange node
......................................................................


Patch Set 4: Code-Review+2


-- 
To view, visit http://gerrit.cloudera.org:8080/11692
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I5b577f9511abc48b992e814d50bba4959f23f7fd
Gerrit-Change-Number: 11692
Gerrit-PatchSet: 4
Gerrit-Owner: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Ho <kw...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vuk Ercegovac <ve...@cloudera.com>
Gerrit-Comment-Date: Wed, 24 Oct 2018 23:06:30 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7351: Add estimates to Exchange node

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11692 )

Change subject: IMPALA-7351: Add estimates to Exchange node
......................................................................


Patch Set 7: Verified+1


-- 
To view, visit http://gerrit.cloudera.org:8080/11692
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I5b577f9511abc48b992e814d50bba4959f23f7fd
Gerrit-Change-Number: 11692
Gerrit-PatchSet: 7
Gerrit-Owner: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Ho <kw...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vuk Ercegovac <ve...@cloudera.com>
Gerrit-Comment-Date: Thu, 25 Oct 2018 21:17:10 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7351: Add estimates to Exchange node

Posted by "Bikramjeet Vig (Code Review)" <ge...@cloudera.org>.
Bikramjeet Vig has posted comments on this change. ( http://gerrit.cloudera.org:8080/11692 )

Change subject: IMPALA-7351: Add estimates to Exchange node
......................................................................


Patch Set 6: Code-Review+2

> Looks like test_explain_level* needs to be updated. Maybe we can
 > just regex out the estimate from the expected output

Done


-- 
To view, visit http://gerrit.cloudera.org:8080/11692
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I5b577f9511abc48b992e814d50bba4959f23f7fd
Gerrit-Change-Number: 11692
Gerrit-PatchSet: 6
Gerrit-Owner: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Ho <kw...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vuk Ercegovac <ve...@cloudera.com>
Gerrit-Comment-Date: Thu, 25 Oct 2018 17:19:55 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7351: Add estimates to Exchange node

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11692 )

Change subject: IMPALA-7351: Add estimates to Exchange node
......................................................................


Patch Set 4:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/1147/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/11692
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I5b577f9511abc48b992e814d50bba4959f23f7fd
Gerrit-Change-Number: 11692
Gerrit-PatchSet: 4
Gerrit-Owner: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Ho <kw...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vuk Ercegovac <ve...@cloudera.com>
Gerrit-Comment-Date: Wed, 24 Oct 2018 23:47:07 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7351: Add estimates to Exchange node

Posted by "Tim Armstrong (Code Review)" <ge...@cloudera.org>.
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/11692 )

Change subject: IMPALA-7351: Add estimates to Exchange node
......................................................................


Patch Set 5:

Looks like test_explain_level* needs to be updated. Maybe we can just regex out the estimate from the expected output


-- 
To view, visit http://gerrit.cloudera.org:8080/11692
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I5b577f9511abc48b992e814d50bba4959f23f7fd
Gerrit-Change-Number: 11692
Gerrit-PatchSet: 5
Gerrit-Owner: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Ho <kw...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vuk Ercegovac <ve...@cloudera.com>
Gerrit-Comment-Date: Thu, 25 Oct 2018 16:35:19 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7351: Add estimates to Exchange node

Posted by "Bikramjeet Vig (Code Review)" <ge...@cloudera.org>.
Hello Michael Ho, Tim Armstrong, Impala Public Jenkins, Vuk Ercegovac, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/11692

to look at the new patch set (#3).

Change subject: IMPALA-7351: Add estimates to Exchange node
......................................................................

IMPALA-7351: Add estimates to Exchange node

Added rough estimates for exchange node and a justification of the
method in the in-line comments.

Testing:
Updated Planner tests.

Change-Id: I5b577f9511abc48b992e814d50bba4959f23f7fd
---
M be/src/util/backend-gflag-util.cc
M common/thrift/BackendGflags.thrift
M fe/src/main/java/org/apache/impala/planner/ExchangeNode.java
M fe/src/main/java/org/apache/impala/planner/PlanNode.java
M testdata/workloads/functional-planner/queries/PlannerTest/kudu-selectivity.test
M testdata/workloads/functional-planner/queries/PlannerTest/max-row-size.test
M testdata/workloads/functional-planner/queries/PlannerTest/resource-requirements.test
M testdata/workloads/functional-planner/queries/PlannerTest/spillable-buffer-sizing.test
M testdata/workloads/functional-planner/queries/PlannerTest/tpcds-all.test
M testdata/workloads/functional-planner/queries/PlannerTest/tpch-all.test
M testdata/workloads/functional-planner/queries/PlannerTest/tpch-nested.test
11 files changed, 902 insertions(+), 811 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/92/11692/3
-- 
To view, visit http://gerrit.cloudera.org:8080/11692
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I5b577f9511abc48b992e814d50bba4959f23f7fd
Gerrit-Change-Number: 11692
Gerrit-PatchSet: 3
Gerrit-Owner: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Ho <kw...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vuk Ercegovac <ve...@cloudera.com>

[Impala-ASF-CR] IMPALA-7351: Add estimates to Exchange node

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11692 )

Change subject: IMPALA-7351: Add estimates to Exchange node
......................................................................


Patch Set 7:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/3354/ DRY_RUN=false


-- 
To view, visit http://gerrit.cloudera.org:8080/11692
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I5b577f9511abc48b992e814d50bba4959f23f7fd
Gerrit-Change-Number: 11692
Gerrit-PatchSet: 7
Gerrit-Owner: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Ho <kw...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vuk Ercegovac <ve...@cloudera.com>
Gerrit-Comment-Date: Thu, 25 Oct 2018 17:21:27 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7351: Add estimates to Exchange node

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11692 )

Change subject: IMPALA-7351: Add estimates to Exchange node
......................................................................


Patch Set 7: Code-Review+2


-- 
To view, visit http://gerrit.cloudera.org:8080/11692
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I5b577f9511abc48b992e814d50bba4959f23f7fd
Gerrit-Change-Number: 11692
Gerrit-PatchSet: 7
Gerrit-Owner: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Ho <kw...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vuk Ercegovac <ve...@cloudera.com>
Gerrit-Comment-Date: Thu, 25 Oct 2018 17:21:26 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7351: Add estimates to Exchange node

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11692 )

Change subject: IMPALA-7351: Add estimates to Exchange node
......................................................................


Patch Set 6:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/1159/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/11692
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I5b577f9511abc48b992e814d50bba4959f23f7fd
Gerrit-Change-Number: 11692
Gerrit-PatchSet: 6
Gerrit-Owner: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Ho <kw...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vuk Ercegovac <ve...@cloudera.com>
Gerrit-Comment-Date: Thu, 25 Oct 2018 17:52:34 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7351: Add estimates to Exchange node

Posted by "Michael Ho (Code Review)" <ge...@cloudera.org>.
Michael Ho has posted comments on this change. ( http://gerrit.cloudera.org:8080/11692 )

Change subject: IMPALA-7351: Add estimates to Exchange node
......................................................................


Patch Set 2:

(4 comments)

http://gerrit.cloudera.org:8080/#/c/11692/2/fe/src/main/java/org/apache/impala/planner/ExchangeNode.java
File fe/src/main/java/org/apache/impala/planner/ExchangeNode.java:

http://gerrit.cloudera.org:8080/#/c/11692/2/fe/src/main/java/org/apache/impala/planner/ExchangeNode.java@192
PS2, Line 192: There is also a deferred rpc queue which queues at max
             :     // one rpc payload (containing the rowbatch) per sender in-case the row
             :     // batch queue hits the previously mentioned soft limit
In the long run, we should consider either of the followings:
- enable spilling in exchange node
- throttle the sender until space opens up on exchange node. May waste a bit of network bandwidth in some cases due to resend.

I suppose either of the above may eventually get the memory consumption of an exchange node under control.


http://gerrit.cloudera.org:8080/#/c/11692/2/fe/src/main/java/org/apache/impala/planner/ExchangeNode.java@214
PS2, Line 214: // TODO: will this be different for a broadcast exchange?
Yes, for broadcasting, each exchange node will get the aggregate of all senders' produced row batches. May be worth having that distinction here.

Also, does the following line make assumption that there is no skew in distribution during data shuffling ? May be worth documenting that.


http://gerrit.cloudera.org:8080/#/c/11692/2/fe/src/main/java/org/apache/impala/planner/ExchangeNode.java@223
PS2, Line 223: cardinality_
How is this different from getCardinality() ?


http://gerrit.cloudera.org:8080/#/c/11692/2/fe/src/main/java/org/apache/impala/planner/PlanNode.java
File fe/src/main/java/org/apache/impala/planner/PlanNode.java:

http://gerrit.cloudera.org:8080/#/c/11692/2/fe/src/main/java/org/apache/impala/planner/PlanNode.java@74
PS2, Line 74: 
nit: extra blank line



-- 
To view, visit http://gerrit.cloudera.org:8080/11692
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I5b577f9511abc48b992e814d50bba4959f23f7fd
Gerrit-Change-Number: 11692
Gerrit-PatchSet: 2
Gerrit-Owner: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Ho <kw...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vuk Ercegovac <ve...@cloudera.com>
Gerrit-Comment-Date: Tue, 23 Oct 2018 00:56:52 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-7351: Add estimates to Exchange node

Posted by "Bikramjeet Vig (Code Review)" <ge...@cloudera.org>.
Bikramjeet Vig has posted comments on this change. ( http://gerrit.cloudera.org:8080/11692 )

Change subject: IMPALA-7351: Add estimates to Exchange node
......................................................................


Patch Set 2:

(8 comments)

http://gerrit.cloudera.org:8080/#/c/11692/2/fe/src/main/java/org/apache/impala/planner/ExchangeNode.java
File fe/src/main/java/org/apache/impala/planner/ExchangeNode.java:

http://gerrit.cloudera.org:8080/#/c/11692/2/fe/src/main/java/org/apache/impala/planner/ExchangeNode.java@192
PS2, Line 192: There is also a deferred rpc queue which queues at max
             :     // one rpc payload (containing the rowbatch) per sender in-case the row
             :     // batch queue hits the previously mentioned soft limit
> In the long run, we should consider either of the followings:
yup that can be handled as a part of IMPALA-5485


http://gerrit.cloudera.org:8080/#/c/11692/2/fe/src/main/java/org/apache/impala/planner/ExchangeNode.java@201
PS2, Line 201:     int numSenders = 0;
> I think I'd prefer if some of these sub-computations were broken out into t
Done


http://gerrit.cloudera.org:8080/#/c/11692/2/fe/src/main/java/org/apache/impala/planner/ExchangeNode.java@213
PS2, Line 213: Reciev
> Receive
Done


http://gerrit.cloudera.org:8080/#/c/11692/2/fe/src/main/java/org/apache/impala/planner/ExchangeNode.java@214
PS2, Line 214: // TODO: will this be different for a broadcast exchange?
> Yes, for broadcasting, each exchange node will get the aggregate of all sen
Done


http://gerrit.cloudera.org:8080/#/c/11692/2/fe/src/main/java/org/apache/impala/planner/ExchangeNode.java@214
PS2, Line 214: // TODO: will this be different for a broadcast exchange?
> Yeah agreed on both points.
Done


http://gerrit.cloudera.org:8080/#/c/11692/2/fe/src/main/java/org/apache/impala/planner/ExchangeNode.java@223
PS2, Line 223: cardinality_
> How is this different from getCardinality() ?
Done


http://gerrit.cloudera.org:8080/#/c/11692/2/fe/src/main/java/org/apache/impala/planner/ExchangeNode.java@230
PS2, Line 230:     // TODO: set a min cap = MemPool::INITIAL_CHUNK_SIZE * someFactor? I have noticed a
> This seems reasonable. If you're empirically seeing a minimum of 16KB that 
Done


http://gerrit.cloudera.org:8080/#/c/11692/2/fe/src/main/java/org/apache/impala/planner/PlanNode.java
File fe/src/main/java/org/apache/impala/planner/PlanNode.java:

http://gerrit.cloudera.org:8080/#/c/11692/2/fe/src/main/java/org/apache/impala/planner/PlanNode.java@74
PS2, Line 74: 
> nit: extra blank line
Done



-- 
To view, visit http://gerrit.cloudera.org:8080/11692
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I5b577f9511abc48b992e814d50bba4959f23f7fd
Gerrit-Change-Number: 11692
Gerrit-PatchSet: 2
Gerrit-Owner: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Ho <kw...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vuk Ercegovac <ve...@cloudera.com>
Gerrit-Comment-Date: Tue, 23 Oct 2018 21:16:14 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-7351: Add estimates to Exchange node

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/11692 )

Change subject: IMPALA-7351: Add estimates to Exchange node
......................................................................

IMPALA-7351: Add estimates to Exchange node

Added rough estimates for exchange node and a justification of the
method in the in-line comments.

Testing:
Updated Planner tests.

Change-Id: I5b577f9511abc48b992e814d50bba4959f23f7fd
Reviewed-on: http://gerrit.cloudera.org:8080/11692
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
M be/src/util/backend-gflag-util.cc
M common/thrift/BackendGflags.thrift
M fe/src/main/java/org/apache/impala/planner/ExchangeNode.java
M fe/src/main/java/org/apache/impala/planner/PlanNode.java
M testdata/workloads/functional-planner/queries/PlannerTest/kudu-selectivity.test
M testdata/workloads/functional-planner/queries/PlannerTest/max-row-size.test
M testdata/workloads/functional-planner/queries/PlannerTest/resource-requirements.test
M testdata/workloads/functional-planner/queries/PlannerTest/spillable-buffer-sizing.test
M testdata/workloads/functional-planner/queries/PlannerTest/tpcds-all.test
M testdata/workloads/functional-planner/queries/PlannerTest/tpch-all.test
M testdata/workloads/functional-planner/queries/PlannerTest/tpch-nested.test
M testdata/workloads/functional-query/queries/QueryTest/explain-level2.test
M testdata/workloads/functional-query/queries/QueryTest/explain-level3.test
13 files changed, 920 insertions(+), 831 deletions(-)

Approvals:
  Impala Public Jenkins: Looks good to me, approved; Verified

-- 
To view, visit http://gerrit.cloudera.org:8080/11692
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I5b577f9511abc48b992e814d50bba4959f23f7fd
Gerrit-Change-Number: 11692
Gerrit-PatchSet: 8
Gerrit-Owner: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Ho <kw...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vuk Ercegovac <ve...@cloudera.com>

[Impala-ASF-CR] IMPALA-7351: Add estimates to Exchange node

Posted by "Bikramjeet Vig (Code Review)" <ge...@cloudera.org>.
Hello Michael Ho, Tim Armstrong, Impala Public Jenkins, Vuk Ercegovac, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/11692

to look at the new patch set (#4).

Change subject: IMPALA-7351: Add estimates to Exchange node
......................................................................

IMPALA-7351: Add estimates to Exchange node

Added rough estimates for exchange node and a justification of the
method in the in-line comments.

Testing:
Updated Planner tests.

Change-Id: I5b577f9511abc48b992e814d50bba4959f23f7fd
---
M be/src/util/backend-gflag-util.cc
M common/thrift/BackendGflags.thrift
M fe/src/main/java/org/apache/impala/planner/ExchangeNode.java
M fe/src/main/java/org/apache/impala/planner/PlanNode.java
M testdata/workloads/functional-planner/queries/PlannerTest/kudu-selectivity.test
M testdata/workloads/functional-planner/queries/PlannerTest/max-row-size.test
M testdata/workloads/functional-planner/queries/PlannerTest/resource-requirements.test
M testdata/workloads/functional-planner/queries/PlannerTest/spillable-buffer-sizing.test
M testdata/workloads/functional-planner/queries/PlannerTest/tpcds-all.test
M testdata/workloads/functional-planner/queries/PlannerTest/tpch-all.test
M testdata/workloads/functional-planner/queries/PlannerTest/tpch-nested.test
11 files changed, 900 insertions(+), 811 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/92/11692/4
-- 
To view, visit http://gerrit.cloudera.org:8080/11692
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I5b577f9511abc48b992e814d50bba4959f23f7fd
Gerrit-Change-Number: 11692
Gerrit-PatchSet: 4
Gerrit-Owner: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Ho <kw...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vuk Ercegovac <ve...@cloudera.com>

[Impala-ASF-CR] IMPALA-7351: Add estimates to Exchange node

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11692 )

Change subject: IMPALA-7351: Add estimates to Exchange node
......................................................................


Patch Set 2:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/1062/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/11692
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I5b577f9511abc48b992e814d50bba4959f23f7fd
Gerrit-Change-Number: 11692
Gerrit-PatchSet: 2
Gerrit-Owner: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Ho
Gerrit-Comment-Date: Tue, 16 Oct 2018 20:42:07 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7351: Add estimates to Exchange node

Posted by "Tim Armstrong (Code Review)" <ge...@cloudera.org>.
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/11692 )

Change subject: IMPALA-7351: Add estimates to Exchange node
......................................................................


Patch Set 2:

(5 comments)

http://gerrit.cloudera.org:8080/#/c/11692/2/fe/src/main/java/org/apache/impala/planner/ExchangeNode.java
File fe/src/main/java/org/apache/impala/planner/ExchangeNode.java:

http://gerrit.cloudera.org:8080/#/c/11692/2/fe/src/main/java/org/apache/impala/planner/ExchangeNode.java@201
PS2, Line 201:     int numSenders = 0;
I think I'd prefer if some of these sub-computations were broken out into their own functions. E.g. computeNumSenders(), estimateTotalQueueByteSize(), estimateRowBatchByteSize(). As-is it's hard to see the structure of the overall computation.


http://gerrit.cloudera.org:8080/#/c/11692/2/fe/src/main/java/org/apache/impala/planner/ExchangeNode.java@213
PS2, Line 213: Reciev
Receive


http://gerrit.cloudera.org:8080/#/c/11692/2/fe/src/main/java/org/apache/impala/planner/ExchangeNode.java@214
PS2, Line 214: // TODO: will this be different for a broadcast exchange?
> Yes, for broadcasting, each exchange node will get the aggregate of all sen
Yeah agreed on both points.


http://gerrit.cloudera.org:8080/#/c/11692/2/fe/src/main/java/org/apache/impala/planner/ExchangeNode.java@230
PS2, Line 230:     // TODO: set a min cap = MemPool::INITIAL_CHUNK_SIZE * someFactor? I have noticed a
This seems reasonable. If you're empirically seeing a minimum of 16KB that seems like a good choice.


http://gerrit.cloudera.org:8080/#/c/11692/2/fe/src/main/java/org/apache/impala/planner/PlanNode.java
File fe/src/main/java/org/apache/impala/planner/PlanNode.java:

http://gerrit.cloudera.org:8080/#/c/11692/2/fe/src/main/java/org/apache/impala/planner/PlanNode.java@75
PS2, Line 75: 
Extra blank line



-- 
To view, visit http://gerrit.cloudera.org:8080/11692
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I5b577f9511abc48b992e814d50bba4959f23f7fd
Gerrit-Change-Number: 11692
Gerrit-PatchSet: 2
Gerrit-Owner: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Ho <kw...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vuk Ercegovac <ve...@cloudera.com>
Gerrit-Comment-Date: Tue, 23 Oct 2018 15:59:10 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-7351: Add estimates to Exchange node

Posted by "Bikramjeet Vig (Code Review)" <ge...@cloudera.org>.
Bikramjeet Vig has posted comments on this change. ( http://gerrit.cloudera.org:8080/11692 )

Change subject: IMPALA-7351: Add estimates to Exchange node
......................................................................


Patch Set 3:

(3 comments)

http://gerrit.cloudera.org:8080/#/c/11692/3//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/11692/3//COMMIT_MSG@10
PS3, Line 10: method in the in-line comments.
> Do we know what the likely upper bound is for the memory estimates on large
Yes, the worst case is with a merging exchange, large sized rows, large number of rows and a huge cluster such that the number of queues is alot too. In that case the mem estimate would be => (18MB * num_senders)
{18MB comes from default values of exchg_node_buffer_size_bytes + ROWBATCH_MAX_MEM_USAGE = 10 + 8)
In this case it does however seem likely that the overestimation might not be by a huge factor, since large rows and this many number of senders can overwhelm an exchange node


http://gerrit.cloudera.org:8080/#/c/11692/3/fe/src/main/java/org/apache/impala/planner/ExchangeNode.java
File fe/src/main/java/org/apache/impala/planner/ExchangeNode.java:

http://gerrit.cloudera.org:8080/#/c/11692/3/fe/src/main/java/org/apache/impala/planner/ExchangeNode.java@77
PS3, Line 77:     if (!streamSink.getOutputPartition().isPartitioned() && fragment_.isPartitioned()) {
> nit: can just return this expression
Done


http://gerrit.cloudera.org:8080/#/c/11692/3/fe/src/main/java/org/apache/impala/planner/ExchangeNode.java@241
PS3, Line 241:     long deferredBatchQueueSize = avgRowBatchByteSize * numSenders;
> nit: unnecessary intermediate variable
Done



-- 
To view, visit http://gerrit.cloudera.org:8080/11692
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I5b577f9511abc48b992e814d50bba4959f23f7fd
Gerrit-Change-Number: 11692
Gerrit-PatchSet: 3
Gerrit-Owner: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Ho <kw...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vuk Ercegovac <ve...@cloudera.com>
Gerrit-Comment-Date: Wed, 24 Oct 2018 21:02:19 +0000
Gerrit-HasComments: Yes