You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Qifan Chen (Code Review)" <ge...@cloudera.org> on 2021/07/19 16:32:45 UTC

[Impala-ASF-CR] IMPALA-10754 test overlap min max filters on sorted columns failed during GVO 

Qifan Chen has uploaded this change for review. ( http://gerrit.cloudera.org:8080/17696


Change subject: IMPALA-10754 test_overlap_min_max_filters_on_sorted_columns failed during GVO
......................................................................

IMPALA-10754 test_overlap_min_max_filters_on_sorted_columns failed during GVO

This patch addresses a failure in ubuntu-16.04 dockerised test in the
following test in overlap_min_max_filters_on_sorted_columns.test by
reducing the threshold value from 58 to 50.

  set minmax_filter_fast_code_path=on;
  set MINMAX_FILTER_THRESHOLD=0.0;
  SET RUNTIME_FILTER_WAIT_TIME_MS=$RUNTIME_FILTER_WAIT_TIME_MS;
  select straight_join count(a.timestamp_col) from
  alltypes_timestamp_col_only a join [SHUFFLE] alltypes_limited b
  where a.timestamp_col = b.timestamp_col and b.tinyint_col = 4;
  ---- RUNTIME_PROFILE
  aggregation(SUM, NumRuntimeFilteredPages)> 49

Testing:
   Ran the unit test successfully.

Change-Id: Icb4cc7d533139c4a2b46a872234a47d46cb8a17c
---
M testdata/workloads/functional-query/queries/QueryTest/overlap_min_max_filters_on_sorted_columns.test
1 file changed, 2 insertions(+), 2 deletions(-)



  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/96/17696/2
-- 
To view, visit http://gerrit.cloudera.org:8080/17696
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: Icb4cc7d533139c4a2b46a872234a47d46cb8a17c
Gerrit-Change-Number: 17696
Gerrit-PatchSet: 2
Gerrit-Owner: Qifan Chen <qc...@cloudera.com>

[Impala-ASF-CR] IMPALA-10754: test overlap min max filters on sorted columns failed during GVO

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17696 )

Change subject: IMPALA-10754: test_overlap_min_max_filters_on_sorted_columns failed during GVO
......................................................................


Patch Set 5:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/9116/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/17696
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icb4cc7d533139c4a2b46a872234a47d46cb8a17c
Gerrit-Change-Number: 17696
Gerrit-PatchSet: 5
Gerrit-Owner: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Mon, 19 Jul 2021 17:41:49 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10754: test overlap min max filters on sorted columns failed during GVO

Posted by "Wenzhe Zhou (Code Review)" <ge...@cloudera.org>.
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/17696 )

Change subject: IMPALA-10754: test_overlap_min_max_filters_on_sorted_columns failed during GVO
......................................................................


Patch Set 5: Code-Review+1


-- 
To view, visit http://gerrit.cloudera.org:8080/17696
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icb4cc7d533139c4a2b46a872234a47d46cb8a17c
Gerrit-Change-Number: 17696
Gerrit-PatchSet: 5
Gerrit-Owner: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Comment-Date: Tue, 20 Jul 2021 00:24:42 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10754: test overlap min max filters on sorted columns failed during GVO

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17696 )

Change subject: IMPALA-10754: test_overlap_min_max_filters_on_sorted_columns failed during GVO
......................................................................


Patch Set 2:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/9114/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/17696
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icb4cc7d533139c4a2b46a872234a47d46cb8a17c
Gerrit-Change-Number: 17696
Gerrit-PatchSet: 2
Gerrit-Owner: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Mon, 19 Jul 2021 17:02:17 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10754: test overlap min max filters on sorted columns failed during GVO

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/17696 )

Change subject: IMPALA-10754: test_overlap_min_max_filters_on_sorted_columns failed during GVO
......................................................................

IMPALA-10754: test_overlap_min_max_filters_on_sorted_columns failed during GVO

This patch addresses a failure in ubuntu-16.04 dockerised test. The test
involved is found in overlap_min_max_filters_on_sorted_columns.test as
follows.

  set minmax_filter_fast_code_path=on;
  set MINMAX_FILTER_THRESHOLD=0.0;
  SET RUNTIME_FILTER_WAIT_TIME_MS=$RUNTIME_FILTER_WAIT_TIME_MS;
  select straight_join count(a.timestamp_col) from
  alltypes_timestamp_col_only a join [SHUFFLE] alltypes_limited b
  where a.timestamp_col = b.timestamp_col and b.tinyint_col = 4;
  ---- RUNTIME_PROFILE
  aggregation(SUM, NumRuntimeFilteredPages)> 57

The patch reduces the threshold from 58 to 50.

Testing:
   Ran the unit test successfully.

Change-Id: Icb4cc7d533139c4a2b46a872234a47d46cb8a17c
Reviewed-on: http://gerrit.cloudera.org:8080/17696
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
M testdata/workloads/functional-query/queries/QueryTest/overlap_min_max_filters_on_sorted_columns.test
1 file changed, 2 insertions(+), 2 deletions(-)

Approvals:
  Impala Public Jenkins: Looks good to me, approved; Verified

-- 
To view, visit http://gerrit.cloudera.org:8080/17696
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: Icb4cc7d533139c4a2b46a872234a47d46cb8a17c
Gerrit-Change-Number: 17696
Gerrit-PatchSet: 7
Gerrit-Owner: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>

[Impala-ASF-CR] IMPALA-10754: test overlap min max filters on sorted columns failed during GVO

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17696 )

Change subject: IMPALA-10754: test_overlap_min_max_filters_on_sorted_columns failed during GVO
......................................................................


Patch Set 3:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/9115/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/17696
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icb4cc7d533139c4a2b46a872234a47d46cb8a17c
Gerrit-Change-Number: 17696
Gerrit-PatchSet: 3
Gerrit-Owner: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Mon, 19 Jul 2021 17:36:12 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10754: test overlap min max filters on sorted columns failed during GVO

Posted by "Wenzhe Zhou (Code Review)" <ge...@cloudera.org>.
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/17696 )

Change subject: IMPALA-10754: test_overlap_min_max_filters_on_sorted_columns failed during GVO
......................................................................


Patch Set 5:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/17696/5//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/17696/5//COMMIT_MSG@20
PS5, Line 20: NumRuntimeFilteredPages
What's cause for NumRuntimeFilteredPages to change in different environment? Is it possible for the test to fail in other environment after changing it 50?



-- 
To view, visit http://gerrit.cloudera.org:8080/17696
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icb4cc7d533139c4a2b46a872234a47d46cb8a17c
Gerrit-Change-Number: 17696
Gerrit-PatchSet: 5
Gerrit-Owner: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Comment-Date: Mon, 19 Jul 2021 23:12:33 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-10754: test overlap min max filters on sorted columns failed during GVO

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17696 )

Change subject: IMPALA-10754: test_overlap_min_max_filters_on_sorted_columns failed during GVO
......................................................................


Patch Set 6: Code-Review+2


-- 
To view, visit http://gerrit.cloudera.org:8080/17696
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icb4cc7d533139c4a2b46a872234a47d46cb8a17c
Gerrit-Change-Number: 17696
Gerrit-PatchSet: 6
Gerrit-Owner: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 20 Jul 2021 16:41:32 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10754: test overlap min max filters on sorted columns failed during GVO

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17696 )

Change subject: IMPALA-10754: test_overlap_min_max_filters_on_sorted_columns failed during GVO
......................................................................


Patch Set 6: Verified+1


-- 
To view, visit http://gerrit.cloudera.org:8080/17696
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icb4cc7d533139c4a2b46a872234a47d46cb8a17c
Gerrit-Change-Number: 17696
Gerrit-PatchSet: 6
Gerrit-Owner: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 20 Jul 2021 22:56:47 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10754: test overlap min max filters on sorted columns failed during GVO

Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/17696 )

Change subject: IMPALA-10754: test_overlap_min_max_filters_on_sorted_columns failed during GVO
......................................................................


Patch Set 5: Code-Review+2


-- 
To view, visit http://gerrit.cloudera.org:8080/17696
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icb4cc7d533139c4a2b46a872234a47d46cb8a17c
Gerrit-Change-Number: 17696
Gerrit-PatchSet: 5
Gerrit-Owner: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 20 Jul 2021 13:36:18 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10754: test overlap min max filters on sorted columns failed during GVO

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17696 )

Change subject: IMPALA-10754: test_overlap_min_max_filters_on_sorted_columns failed during GVO
......................................................................


Patch Set 6:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/7325/ DRY_RUN=false


-- 
To view, visit http://gerrit.cloudera.org:8080/17696
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icb4cc7d533139c4a2b46a872234a47d46cb8a17c
Gerrit-Change-Number: 17696
Gerrit-PatchSet: 6
Gerrit-Owner: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 20 Jul 2021 16:41:33 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10754: test overlap min max filters on sorted columns failed during GVO

Posted by "Qifan Chen (Code Review)" <ge...@cloudera.org>.
Qifan Chen has uploaded a new patch set (#5). ( http://gerrit.cloudera.org:8080/17696 )

Change subject: IMPALA-10754: test_overlap_min_max_filters_on_sorted_columns failed during GVO
......................................................................

IMPALA-10754: test_overlap_min_max_filters_on_sorted_columns failed during GVO

This patch addresses a failure in ubuntu-16.04 dockerised test. The test
involved is found in overlap_min_max_filters_on_sorted_columns.test as
follows.

  set minmax_filter_fast_code_path=on;
  set MINMAX_FILTER_THRESHOLD=0.0;
  SET RUNTIME_FILTER_WAIT_TIME_MS=$RUNTIME_FILTER_WAIT_TIME_MS;
  select straight_join count(a.timestamp_col) from
  alltypes_timestamp_col_only a join [SHUFFLE] alltypes_limited b
  where a.timestamp_col = b.timestamp_col and b.tinyint_col = 4;
  ---- RUNTIME_PROFILE
  aggregation(SUM, NumRuntimeFilteredPages)> 57

The patch reduces the threshold from 58 to 50.

Testing:
   Ran the unit test successfully.

Change-Id: Icb4cc7d533139c4a2b46a872234a47d46cb8a17c
---
M testdata/workloads/functional-query/queries/QueryTest/overlap_min_max_filters_on_sorted_columns.test
1 file changed, 2 insertions(+), 2 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/96/17696/5
-- 
To view, visit http://gerrit.cloudera.org:8080/17696
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Icb4cc7d533139c4a2b46a872234a47d46cb8a17c
Gerrit-Change-Number: 17696
Gerrit-PatchSet: 5
Gerrit-Owner: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>

[Impala-ASF-CR] IMPALA-10754: test overlap min max filters on sorted columns failed during GVO

Posted by "Qifan Chen (Code Review)" <ge...@cloudera.org>.
Qifan Chen has uploaded a new patch set (#3). ( http://gerrit.cloudera.org:8080/17696 )

Change subject: IMPALA-10754: test_overlap_min_max_filters_on_sorted_columns failed during GVO
......................................................................

IMPALA-10754: test_overlap_min_max_filters_on_sorted_columns failed during GVO

This patch addresses a failure in ubuntu-16.04 dockerised test in the
following test in overlap_min_max_filters_on_sorted_columns.test by
reducing the threshold value from 58 to 50.

  set minmax_filter_fast_code_path=on;
  set MINMAX_FILTER_THRESHOLD=0.0;
  SET RUNTIME_FILTER_WAIT_TIME_MS=$RUNTIME_FILTER_WAIT_TIME_MS;
  select straight_join count(a.timestamp_col) from
  alltypes_timestamp_col_only a join [SHUFFLE] alltypes_limited b
  where a.timestamp_col = b.timestamp_col and b.tinyint_col = 4;
  ---- RUNTIME_PROFILE
  aggregation(SUM, NumRuntimeFilteredPages)> 57

Testing:
   Ran the unit test successfully.

Change-Id: Icb4cc7d533139c4a2b46a872234a47d46cb8a17c
---
M testdata/workloads/functional-query/queries/QueryTest/overlap_min_max_filters_on_sorted_columns.test
1 file changed, 2 insertions(+), 2 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/96/17696/3
-- 
To view, visit http://gerrit.cloudera.org:8080/17696
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Icb4cc7d533139c4a2b46a872234a47d46cb8a17c
Gerrit-Change-Number: 17696
Gerrit-PatchSet: 3
Gerrit-Owner: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>

[Impala-ASF-CR] IMPALA-10754: test overlap min max filters on sorted columns failed during GVO

Posted by "Qifan Chen (Code Review)" <ge...@cloudera.org>.
Qifan Chen has posted comments on this change. ( http://gerrit.cloudera.org:8080/17696 )

Change subject: IMPALA-10754: test_overlap_min_max_filters_on_sorted_columns failed during GVO
......................................................................


Patch Set 5:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/17696/5//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/17696/5//COMMIT_MSG@20
PS5, Line 20: NumRuntimeFilteredPages
> What's cause for NumRuntimeFilteredPages to change in different environment
The likely cause is that the dockerized machine behaves differently than our R&D box, possibly in the way that the filters reach the scan nodes slowly. In the failed test, RUNTIME_FILTER_WAIT_TIME_MS is set to 60000. If the filter does not reach the scan node within that duration, the scan node can start scanning right away. 

Since the actual value is 57 vs. the expected value of 58, we miss one page. 

Setting the expected to 50 should provide reasonable rooms.



-- 
To view, visit http://gerrit.cloudera.org:8080/17696
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icb4cc7d533139c4a2b46a872234a47d46cb8a17c
Gerrit-Change-Number: 17696
Gerrit-PatchSet: 5
Gerrit-Owner: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Comment-Date: Mon, 19 Jul 2021 23:55:19 +0000
Gerrit-HasComments: Yes