You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Quanlong Huang (Code Review)" <ge...@cloudera.org> on 2022/05/17 12:30:56 UTC
[Impala-ASF-CR] IMPALA-11295: deflake TestParquet.test multiple blocks mt dop
Quanlong Huang has uploaded this change for review. ( http://gerrit.cloudera.org:8080/18533
Change subject: IMPALA-11295: deflake TestParquet.test_multiple_blocks_mt_dop
......................................................................
IMPALA-11295: deflake TestParquet.test_multiple_blocks_mt_dop
TestParquet.test_multiple_blocks_mt_dop runs a query on 6 scan ranges
using mt_dop=2. It then verifies the sum of ranges read on a backend is
2 (6/3). The test assumes that counters of the 2 instances on the same
host are printed consecutively. However, this is not always true. They
could be interleaving.
This patch makes the test more robust by grouping the counters based on
the host.
Test
- I can't reproduce the issue locally. But I'm able to run the new test
100 times without any error.
Change-Id: I16c576c41a212f83dda82a83931ab336a78a41e4
---
M tests/query_test/test_scanners.py
1 file changed, 11 insertions(+), 4 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/33/18533/1
--
To view, visit http://gerrit.cloudera.org:8080/18533
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I16c576c41a212f83dda82a83931ab336a78a41e4
Gerrit-Change-Number: 18533
Gerrit-PatchSet: 1
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
[Impala-ASF-CR] IMPALA-11295: Deflake TestParquet.test multiple blocks mt dop
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Hello Michael Smith, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/18533
to look at the new patch set (#3).
Change subject: IMPALA-11295: Deflake TestParquet.test_multiple_blocks_mt_dop
......................................................................
IMPALA-11295: Deflake TestParquet.test_multiple_blocks_mt_dop
TestParquet.test_multiple_blocks_mt_dop runs a query on 6 scan ranges
using mt_dop=2. It then verifies the sum of ranges read on a backend is
2 (6/3). The test assumes that counters of the 2 instances on the same
host are printed consecutively. However, this is not always true. They
could be interleaving.
This patch makes the test more robust by grouping the counters based on
the host.
Test
- I can't reproduce the issue locally. But I'm able to run the new test
100 times without any error.
Change-Id: I16c576c41a212f83dda82a83931ab336a78a41e4
---
M tests/query_test/test_scanners.py
1 file changed, 14 insertions(+), 4 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/33/18533/3
--
To view, visit http://gerrit.cloudera.org:8080/18533
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I16c576c41a212f83dda82a83931ab336a78a41e4
Gerrit-Change-Number: 18533
Gerrit-PatchSet: 3
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
[Impala-ASF-CR] IMPALA-11295: deflake TestParquet.test multiple blocks mt dop
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18533 )
Change subject: IMPALA-11295: deflake TestParquet.test_multiple_blocks_mt_dop
......................................................................
Patch Set 2:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/10587/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/18533
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I16c576c41a212f83dda82a83931ab336a78a41e4
Gerrit-Change-Number: 18533
Gerrit-PatchSet: 2
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Wed, 18 May 2022 02:32:16 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11295: Deflake TestParquet.test multiple blocks mt dop
Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/18533 )
Change subject: IMPALA-11295: Deflake TestParquet.test_multiple_blocks_mt_dop
......................................................................
Patch Set 3: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/18533
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I16c576c41a212f83dda82a83931ab336a78a41e4
Gerrit-Change-Number: 18533
Gerrit-PatchSet: 3
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Mon, 23 May 2022 07:22:42 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11295: Deflake TestParquet.test multiple blocks mt dop
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18533 )
Change subject: IMPALA-11295: Deflake TestParquet.test_multiple_blocks_mt_dop
......................................................................
Patch Set 4: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/18533
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I16c576c41a212f83dda82a83931ab336a78a41e4
Gerrit-Change-Number: 18533
Gerrit-PatchSet: 4
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Mon, 23 May 2022 22:29:06 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11295: deflake TestParquet.test multiple blocks mt dop
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18533 )
Change subject: IMPALA-11295: deflake TestParquet.test_multiple_blocks_mt_dop
......................................................................
Patch Set 1:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/10583/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/18533
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I16c576c41a212f83dda82a83931ab336a78a41e4
Gerrit-Change-Number: 18533
Gerrit-PatchSet: 1
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Tue, 17 May 2022 12:50:58 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11295: deflake TestParquet.test multiple blocks mt dop
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Hello Michael Smith, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/18533
to look at the new patch set (#2).
Change subject: IMPALA-11295: deflake TestParquet.test_multiple_blocks_mt_dop
......................................................................
IMPALA-11295: deflake TestParquet.test_multiple_blocks_mt_dop
TestParquet.test_multiple_blocks_mt_dop runs a query on 6 scan ranges
using mt_dop=2. It then verifies the sum of ranges read on a backend is
2 (6/3). The test assumes that counters of the 2 instances on the same
host are printed consecutively. However, this is not always true. They
could be interleaving.
This patch makes the test more robust by grouping the counters based on
the host.
Test
- I can't reproduce the issue locally. But I'm able to run the new test
100 times without any error.
Change-Id: I16c576c41a212f83dda82a83931ab336a78a41e4
---
M tests/query_test/test_scanners.py
1 file changed, 14 insertions(+), 4 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/33/18533/2
--
To view, visit http://gerrit.cloudera.org:8080/18533
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I16c576c41a212f83dda82a83931ab336a78a41e4
Gerrit-Change-Number: 18533
Gerrit-PatchSet: 2
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
[Impala-ASF-CR] IMPALA-11295: Deflake TestParquet.test multiple blocks mt dop
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/18533 )
Change subject: IMPALA-11295: Deflake TestParquet.test_multiple_blocks_mt_dop
......................................................................
IMPALA-11295: Deflake TestParquet.test_multiple_blocks_mt_dop
TestParquet.test_multiple_blocks_mt_dop runs a query on 6 scan ranges
using mt_dop=2. It then verifies the sum of ranges read on a backend is
2 (6/3). The test assumes that counters of the 2 instances on the same
host are printed consecutively. However, this is not always true. They
could be interleaving.
This patch makes the test more robust by grouping the counters based on
the host.
Test
- I can't reproduce the issue locally. But I'm able to run the new test
100 times without any error.
Change-Id: I16c576c41a212f83dda82a83931ab336a78a41e4
Reviewed-on: http://gerrit.cloudera.org:8080/18533
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
M tests/query_test/test_scanners.py
1 file changed, 14 insertions(+), 4 deletions(-)
Approvals:
Impala Public Jenkins: Looks good to me, approved; Verified
--
To view, visit http://gerrit.cloudera.org:8080/18533
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I16c576c41a212f83dda82a83931ab336a78a41e4
Gerrit-Change-Number: 18533
Gerrit-PatchSet: 5
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
[Impala-ASF-CR] IMPALA-11295: deflake TestParquet.test multiple blocks mt dop
Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/18533 )
Change subject: IMPALA-11295: deflake TestParquet.test_multiple_blocks_mt_dop
......................................................................
Patch Set 1:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/18533/1/tests/query_test/test_scanners.py
File tests/query_test/test_scanners.py:
http://gerrit.cloudera.org:8080/#/c/18533/1/tests/query_test/test_scanners.py@a698
PS1, Line 698:
Was this an invalid assumption, or did it change at some point?
http://gerrit.cloudera.org:8080/#/c/18533/1/tests/query_test/test_scanners.py@699
PS1, Line 699: # Again skip the Averaged Fragment; it comes first in the runtime profile.
If ranges_complete_list starts with an Averaged Fragment, it's not clear to me why `len(host_list) == len(ranges_complete_list)`. But your changes seem consistent with what was already here.
--
To view, visit http://gerrit.cloudera.org:8080/18533
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I16c576c41a212f83dda82a83931ab336a78a41e4
Gerrit-Change-Number: 18533
Gerrit-PatchSet: 1
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Tue, 17 May 2022 19:23:05 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11295: deflake TestParquet.test multiple blocks mt dop
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/18533 )
Change subject: IMPALA-11295: deflake TestParquet.test_multiple_blocks_mt_dop
......................................................................
Patch Set 1:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/18533/1/tests/query_test/test_scanners.py
File tests/query_test/test_scanners.py:
http://gerrit.cloudera.org:8080/#/c/18533/1/tests/query_test/test_scanners.py@a698
PS1, Line 698:
> Was this an invalid assumption, or did it change at some point?
Good question. I think it's an invalid assumption but I'm not pretty sure about this. I'll look into more details about this.
http://gerrit.cloudera.org:8080/#/c/18533/1/tests/query_test/test_scanners.py@699
PS1, Line 699: # Again skip the Averaged Fragment; it comes first in the runtime profile.
> If ranges_complete_list starts with an Averaged Fragment, it's not clear to
Sorry that I should have added more comments. The host list is extracted for each fragment instance. There are 7 instances. The first one is the coordinator fragment instance which doesn't have the ScanRangesComplete counter, so we can skip it. The other 6 instances are all the scan fragment. The profile is attached at https://issues.apache.org/jira/secure/attachment/13043781/test_multiple_blocks_mt_dop.profile.txt
Added more comments in the next patch set.
--
To view, visit http://gerrit.cloudera.org:8080/18533
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I16c576c41a212f83dda82a83931ab336a78a41e4
Gerrit-Change-Number: 18533
Gerrit-PatchSet: 1
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Wed, 18 May 2022 02:12:23 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11295: Deflake TestParquet.test multiple blocks mt dop
Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/18533 )
Change subject: IMPALA-11295: Deflake TestParquet.test_multiple_blocks_mt_dop
......................................................................
Patch Set 3: Code-Review+1
--
To view, visit http://gerrit.cloudera.org:8080/18533
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I16c576c41a212f83dda82a83931ab336a78a41e4
Gerrit-Change-Number: 18533
Gerrit-PatchSet: 3
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Wed, 18 May 2022 17:37:22 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11295: Deflake TestParquet.test multiple blocks mt dop
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18533 )
Change subject: IMPALA-11295: Deflake TestParquet.test_multiple_blocks_mt_dop
......................................................................
Patch Set 4:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/8148/ DRY_RUN=false
--
To view, visit http://gerrit.cloudera.org:8080/18533
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I16c576c41a212f83dda82a83931ab336a78a41e4
Gerrit-Change-Number: 18533
Gerrit-PatchSet: 4
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Mon, 23 May 2022 22:29:07 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11295: Deflake TestParquet.test multiple blocks mt dop
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18533 )
Change subject: IMPALA-11295: Deflake TestParquet.test_multiple_blocks_mt_dop
......................................................................
Patch Set 4: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/18533
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I16c576c41a212f83dda82a83931ab336a78a41e4
Gerrit-Change-Number: 18533
Gerrit-PatchSet: 4
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Tue, 24 May 2022 02:53:32 +0000
Gerrit-HasComments: No