You are viewing a plain text version of this content. The canonical link for it is here.

Posted to reviews@impala.apache.org by "Alex Behm (Code Review)" <ge...@cloudera.org> on 2017/06/02 17:52:07 UTC

[Impala-ASF-CR] IMPALA-5381: Adds DEFAULT JOIN STRATEGY query option.

Alex Behm has uploaded a new change for review.

  http://gerrit.cloudera.org:8080/7059

Change subject: IMPALA-5381: Adds DEFAULT_JOIN_STRATEGY query option.
......................................................................

IMPALA-5381: Adds DEFAULT_JOIN_STRATEGY query option.

Adds a new query option DEFAULT_JOIN_STRATEGY to control
which join strategy is chosen when the join inputs have
an unknown cardinality (e.g., missing stats) or when
the expected costs of the different strategies are equal.

Values for DEFAULT_JOIN_STRATEGY: [BROADCAST, SHUFFLE]
Default: BROADCAST

Note that this change effectively undoes IMPALA-5120.

Testing:
- Added new planner tests
- Core/hdfs run passed

Change-Id: Ibd34442f422129d53bef5493fc9cbe7375a0765c
---
M be/src/service/query-options.cc
M be/src/service/query-options.h
M common/thrift/ImpalaInternalService.thrift
M common/thrift/ImpalaService.thrift
M fe/src/main/java/org/apache/impala/planner/DistributedPlanner.java
M fe/src/test/java/org/apache/impala/planner/PlannerTest.java
A testdata/workloads/functional-planner/queries/PlannerTest/default-join-strategy-broadcast.test
A testdata/workloads/functional-planner/queries/PlannerTest/default-join-strategy-shuffle.test
M testdata/workloads/functional-planner/queries/PlannerTest/joins.test
9 files changed, 237 insertions(+), 88 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/59/7059/1
-- 
To view, visit http://gerrit.cloudera.org:8080/7059
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: Ibd34442f422129d53bef5493fc9cbe7375a0765c
Gerrit-PatchSet: 1
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Alex Behm <al...@cloudera.com>

[Impala-ASF-CR] IMPALA-5381: Adds DEFAULT JOIN DISTRIBUTION MODE query option.

Posted by "Alex Behm (Code Review)" <ge...@cloudera.org>.

Hello Marcel Kornacker,

I'd like you to reexamine a change.  Please visit

    http://gerrit.cloudera.org:8080/7059

to look at the new patch set (#2).

Change subject: IMPALA-5381: Adds DEFAULT_JOIN_DISTRIBUTION_MODE query option.
......................................................................

IMPALA-5381: Adds DEFAULT_JOIN_DISTRIBUTION_MODE query option.

Adds a new query option DEFAULT_JOIN_DISTRIBUTION_MODE to
control which join distribution mode is chosen when the join
inputs have an unknown cardinality (e.g., missing stats) or when
the expected costs of the different strategies are equal.

Values for DEFAULT_JOIN_DISTRIBUTION_MODE: [BROADCAST, SHUFFLE]
Default: BROADCAST

Note that this change effectively undoes IMPALA-5120.

Testing:
- Added new planner tests
- Core/hdfs run passed

Change-Id: Ibd34442f422129d53bef5493fc9cbe7375a0765c
---
M be/src/service/query-options.cc
M be/src/service/query-options.h
M common/thrift/ImpalaInternalService.thrift
M common/thrift/ImpalaService.thrift
M fe/src/main/java/org/apache/impala/planner/DistributedPlanner.java
M fe/src/main/java/org/apache/impala/planner/JoinNode.java
M fe/src/test/java/org/apache/impala/planner/PlannerTest.java
A testdata/workloads/functional-planner/queries/PlannerTest/default-join-distr-mode-broadcast.test
A testdata/workloads/functional-planner/queries/PlannerTest/default-join-distr-mode.test
M testdata/workloads/functional-planner/queries/PlannerTest/joins.test
10 files changed, 244 insertions(+), 88 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/59/7059/2
-- 
To view, visit http://gerrit.cloudera.org:8080/7059
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ibd34442f422129d53bef5493fc9cbe7375a0765c
Gerrit-PatchSet: 2
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Marcel Kornacker <ma...@cloudera.com>

[Impala-ASF-CR] IMPALA-5381: Adds DEFAULT JOIN DISTRIBUTION MODE query option.

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has submitted this change and it was merged.

Change subject: IMPALA-5381: Adds DEFAULT_JOIN_DISTRIBUTION_MODE query option.
......................................................................


IMPALA-5381: Adds DEFAULT_JOIN_DISTRIBUTION_MODE query option.

Adds a new query option DEFAULT_JOIN_DISTRIBUTION_MODE to
control which join distribution mode is chosen when the join
inputs have an unknown cardinality (e.g., missing stats) or when
the expected costs of the different strategies are equal.

Values for DEFAULT_JOIN_DISTRIBUTION_MODE: [BROADCAST, SHUFFLE]
Default: BROADCAST

Note that this change effectively undoes IMPALA-5120.

Testing:
- Added new planner tests
- Core/hdfs run passed

Change-Id: Ibd34442f422129d53bef5493fc9cbe7375a0765c
Reviewed-on: http://gerrit.cloudera.org:8080/7059
Reviewed-by: Alex Behm <al...@cloudera.com>
Tested-by: Impala Public Jenkins
---
M be/src/service/query-options.cc
M be/src/service/query-options.h
M common/thrift/ImpalaInternalService.thrift
M common/thrift/ImpalaService.thrift
M fe/src/main/java/org/apache/impala/planner/DistributedPlanner.java
M fe/src/main/java/org/apache/impala/planner/JoinNode.java
M fe/src/test/java/org/apache/impala/planner/PlannerTest.java
A testdata/workloads/functional-planner/queries/PlannerTest/default-join-distr-mode-broadcast.test
A testdata/workloads/functional-planner/queries/PlannerTest/default-join-distr-mode-shuffle.test
M testdata/workloads/functional-planner/queries/PlannerTest/joins.test
10 files changed, 242 insertions(+), 88 deletions(-)

Approvals:
  Impala Public Jenkins: Verified
  Alex Behm: Looks good to me, approved



-- 
To view, visit http://gerrit.cloudera.org:8080/7059
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: Ibd34442f422129d53bef5493fc9cbe7375a0765c
Gerrit-PatchSet: 5
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Marcel Kornacker <ma...@cloudera.com>

[Impala-ASF-CR] IMPALA-5381: Adds DEFAULT JOIN DISTRIBUTION MODE query option.

Posted by "Alex Behm (Code Review)" <ge...@cloudera.org>.

Hello Marcel Kornacker,

I'd like you to reexamine a change.  Please visit

    http://gerrit.cloudera.org:8080/7059

to look at the new patch set (#3).

Change subject: IMPALA-5381: Adds DEFAULT_JOIN_DISTRIBUTION_MODE query option.
......................................................................

IMPALA-5381: Adds DEFAULT_JOIN_DISTRIBUTION_MODE query option.

Adds a new query option DEFAULT_JOIN_DISTRIBUTION_MODE to
control which join distribution mode is chosen when the join
inputs have an unknown cardinality (e.g., missing stats) or when
the expected costs of the different strategies are equal.

Values for DEFAULT_JOIN_DISTRIBUTION_MODE: [BROADCAST, SHUFFLE]
Default: BROADCAST

Note that this change effectively undoes IMPALA-5120.

Testing:
- Added new planner tests
- Core/hdfs run passed

Change-Id: Ibd34442f422129d53bef5493fc9cbe7375a0765c
---
M be/src/service/query-options.cc
M be/src/service/query-options.h
M common/thrift/ImpalaInternalService.thrift
M common/thrift/ImpalaService.thrift
M fe/src/main/java/org/apache/impala/planner/DistributedPlanner.java
M fe/src/main/java/org/apache/impala/planner/JoinNode.java
M fe/src/test/java/org/apache/impala/planner/PlannerTest.java
A testdata/workloads/functional-planner/queries/PlannerTest/default-join-distr-mode-broadcast.test
A testdata/workloads/functional-planner/queries/PlannerTest/default-join-distr-mode-shuffle.test
M testdata/workloads/functional-planner/queries/PlannerTest/joins.test
10 files changed, 244 insertions(+), 88 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/59/7059/3
-- 
To view, visit http://gerrit.cloudera.org:8080/7059
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ibd34442f422129d53bef5493fc9cbe7375a0765c
Gerrit-PatchSet: 3
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Marcel Kornacker <ma...@cloudera.com>

[Impala-ASF-CR] IMPALA-5381: Adds DEFAULT JOIN DISTRIBUTION MODE query option.

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change.

Change subject: IMPALA-5381: Adds DEFAULT_JOIN_DISTRIBUTION_MODE query option.
......................................................................


Patch Set 4: Verified+1

-- 
To view, visit http://gerrit.cloudera.org:8080/7059
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: Ibd34442f422129d53bef5493fc9cbe7375a0765c
Gerrit-PatchSet: 4
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Marcel Kornacker <ma...@cloudera.com>
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-5381: Adds DEFAULT JOIN DISTRIBUTION MODE query option.

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change.

Change subject: IMPALA-5381: Adds DEFAULT_JOIN_DISTRIBUTION_MODE query option.
......................................................................


Patch Set 4:

Build started: http://jenkins.impala.io:8080/job/gerrit-verify-dryrun/673/

-- 
To view, visit http://gerrit.cloudera.org:8080/7059
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: Ibd34442f422129d53bef5493fc9cbe7375a0765c
Gerrit-PatchSet: 4
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Marcel Kornacker <ma...@cloudera.com>
Gerrit-HasComments: No

Re: [Impala-ASF-CR] IMPALA-5381: Adds DEFAULT JOIN DISTRIBUTION MODE query option.

Posted by Alexander Behm <al...@cloudera.com>.

Failed with a known issue IMPALA-5427. Retrying.

On Sat, Jun 3, 2017 at 8:50 PM, Impala Public Jenkins (Code Review) <
gerrit@cloudera.org> wrote:

> Impala Public Jenkins has posted comments on this change.
>
> Change subject: IMPALA-5381: Adds DEFAULT_JOIN_DISTRIBUTION_MODE query
> option.
> ......................................................................
>
>
> Patch Set 4:
>
> Build started: http://jenkins.impala.io:8080/job/gerrit-verify-dryrun/674/
>
> --
> To view, visit http://gerrit.cloudera.org:8080/7059
> To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
>
> Gerrit-MessageType: comment
> Gerrit-Change-Id: Ibd34442f422129d53bef5493fc9cbe7375a0765c
> Gerrit-PatchSet: 4
> Gerrit-Project: Impala-ASF
> Gerrit-Branch: master
> Gerrit-Owner: Alex Behm <al...@cloudera.com>
> Gerrit-Reviewer: Alex Behm <al...@cloudera.com>
> Gerrit-Reviewer: Impala Public Jenkins
> Gerrit-Reviewer: Marcel Kornacker <ma...@cloudera.com>
> Gerrit-HasComments: No
>

[Impala-ASF-CR] IMPALA-5381: Adds DEFAULT JOIN DISTRIBUTION MODE query option.

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change.

Change subject: IMPALA-5381: Adds DEFAULT_JOIN_DISTRIBUTION_MODE query option.
......................................................................


Patch Set 4:

Build started: http://jenkins.impala.io:8080/job/gerrit-verify-dryrun/674/

-- 
To view, visit http://gerrit.cloudera.org:8080/7059
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: Ibd34442f422129d53bef5493fc9cbe7375a0765c
Gerrit-PatchSet: 4
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Marcel Kornacker <ma...@cloudera.com>
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-5381: Adds DEFAULT JOIN DISTRIBUTION MODE query option.

Posted by "Alex Behm (Code Review)" <ge...@cloudera.org>.

Alex Behm has posted comments on this change.

Change subject: IMPALA-5381: Adds DEFAULT_JOIN_DISTRIBUTION_MODE query option.
......................................................................


Patch Set 4: Code-Review+2

Passed an exhaustive run.

-- 
To view, visit http://gerrit.cloudera.org:8080/7059
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: Ibd34442f422129d53bef5493fc9cbe7375a0765c
Gerrit-PatchSet: 4
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Marcel Kornacker <ma...@cloudera.com>
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-5381: Adds DEFAULT JOIN DISTRIBUTION MODE query option.

Posted by "Alex Behm (Code Review)" <ge...@cloudera.org>.

Alex Behm has posted comments on this change.

Change subject: IMPALA-5381: Adds DEFAULT_JOIN_DISTRIBUTION_MODE query option.
......................................................................


Patch Set 3: Code-Review+2

Going to run this through exhaustive before committing to make sure the subtle changes did not break anything.

-- 
To view, visit http://gerrit.cloudera.org:8080/7059
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: Ibd34442f422129d53bef5493fc9cbe7375a0765c
Gerrit-PatchSet: 3
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Marcel Kornacker <ma...@cloudera.com>
Gerrit-HasComments: No