You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Riza Suminto (Code Review)" <ge...@cloudera.org> on 2023/10/26 05:11:59 UTC
[Impala-ASF-CR] IMPALA-12518: Combine all exec option dimension in test vector.py
Riza Suminto has uploaded this change for review. ( http://gerrit.cloudera.org:8080/20625
Change subject: IMPALA-12518: Combine all exec_option dimension in test_vector.py
......................................................................
IMPALA-12518: Combine all exec_option dimension in test_vector.py
Before this patch, when writing pytest that exercise custom query option
values, we need to declare it by making new test dimension, followed by
deepcopying the original vector, and inserting the selected dimension
value into 'exec_option' dictionary in generated vector.
This patch simplify this steps by accounting dimensions that is intended
to be part of 'exec_option' and automatically combining them during
vector generation in test_vector.py. Such dimension should be registered
via the new ImpalaTestMatrix.add_exec_option_dimension() function.
function add_exec_option_dimension() in test_dimensions.py is renamed to
add_mandatory_exec_option() to make it consistent with the same
functionality in ImpalaTestMatrix and avoid confusion with the new
ImpalaTestMatrix.add_exec_option_dimension() function. Function name
add_exec_option_dimension() in test_dimensions.py is then repurposed as
a shorthand for ImpalaTestMatrix.add_exec_option_dimension().
The remaining changes for other pytest files will be done gradually.
Testing:
- Fix bug in TestIcebergV2Table and confirm that both True and False
value for 'disable_optimized_iceberg_v2_read' options are exercised.
- Run and pass all modified tests in this patch.
Change-Id: I3adba260990fccf4d2f2e7c8c4e4fadc6fd43fe1
---
M tests/common/test_dimensions.py
M tests/common/test_vector.py
M tests/custom_cluster/test_kudu.py
M tests/query_test/test_async_codegen.py
M tests/query_test/test_iceberg.py
M tests/query_test/test_kudu.py
M tests/query_test/test_runtime_filters.py
7 files changed, 86 insertions(+), 31 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/25/20625/1
--
To view, visit http://gerrit.cloudera.org:8080/20625
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I3adba260990fccf4d2f2e7c8c4e4fadc6fd43fe1
Gerrit-Change-Number: 20625
Gerrit-PatchSet: 1
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
[Impala-ASF-CR] IMPALA-12518: Combine all exec option dimension in test vector.py
Posted by "Riza Suminto (Code Review)" <ge...@cloudera.org>.
Hello Gabor Kaszab, Zoltan Borok-Nagy, Joe McDonnell, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/20625
to look at the new patch set (#2).
Change subject: IMPALA-12518: Combine all exec_option dimension in test_vector.py
......................................................................
IMPALA-12518: Combine all exec_option dimension in test_vector.py
Before this patch, when writing pytest that exercise custom query option
values, we need to declare it by making new test dimension, followed by
deepcopying the original vector, and inserting the selected dimension
value into 'exec_option' dictionary in generated vector.
This patch simplify this steps by accounting dimensions that is intended
to be part of 'exec_option' and automatically combining them during
vector generation in test_vector.py. Such dimension should be registered
via the new ImpalaTestMatrix.add_exec_option_dimension() function.
function add_exec_option_dimension() in test_dimensions.py is renamed to
add_mandatory_exec_option() to make it consistent with the same
functionality in ImpalaTestMatrix and avoid confusion with the new
ImpalaTestMatrix.add_exec_option_dimension() function. Function name
add_exec_option_dimension() in test_dimensions.py is then repurposed as
a shorthand for ImpalaTestMatrix.add_exec_option_dimension().
The remaining changes for other pytest files will be done gradually.
Testing:
- Fix bug in TestIcebergV2Table and confirm that both True and False
value for 'disable_optimized_iceberg_v2_read' options are exercised.
- Run and pass all modified tests in this patch.
Change-Id: I3adba260990fccf4d2f2e7c8c4e4fadc6fd43fe1
---
M tests/common/test_dimensions.py
M tests/common/test_vector.py
M tests/custom_cluster/test_kudu.py
M tests/query_test/test_async_codegen.py
M tests/query_test/test_iceberg.py
M tests/query_test/test_kudu.py
M tests/query_test/test_runtime_filters.py
7 files changed, 86 insertions(+), 31 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/25/20625/2
--
To view, visit http://gerrit.cloudera.org:8080/20625
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I3adba260990fccf4d2f2e7c8c4e4fadc6fd43fe1
Gerrit-Change-Number: 20625
Gerrit-PatchSet: 2
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
[Impala-ASF-CR] IMPALA-12518: Combine all exec option dimension in test vector.py
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20625 )
Change subject: IMPALA-12518: Combine all exec_option dimension in test_vector.py
......................................................................
Patch Set 2: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/20625
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I3adba260990fccf4d2f2e7c8c4e4fadc6fd43fe1
Gerrit-Change-Number: 20625
Gerrit-PatchSet: 2
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Fri, 27 Oct 2023 01:03:25 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12518: Combine all exec option dimension in test vector.py
Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/20625 )
Change subject: IMPALA-12518: Combine all exec_option dimension in test_vector.py
......................................................................
Patch Set 2:
(1 comment)
Test output looks consistent with prior runs.
http://gerrit.cloudera.org:8080/#/c/20625/2/tests/common/test_vector.py
File tests/common/test_vector.py:
http://gerrit.cloudera.org:8080/#/c/20625/2/tests/common/test_vector.py@63
PS2, Line 63: EXEC_OPTION_KEY = 'exec_option'
We should use this more widely. Can be addressed in another ticket though.
--
To view, visit http://gerrit.cloudera.org:8080/20625
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I3adba260990fccf4d2f2e7c8c4e4fadc6fd43fe1
Gerrit-Change-Number: 20625
Gerrit-PatchSet: 2
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Fri, 27 Oct 2023 02:04:34 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12518: Combine all exec option dimension in test vector.py
Posted by "Riza Suminto (Code Review)" <ge...@cloudera.org>.
Riza Suminto has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/20625 )
Change subject: IMPALA-12518: Combine all exec_option dimension in test_vector.py
......................................................................
IMPALA-12518: Combine all exec_option dimension in test_vector.py
Before this patch, when writing pytest that exercise custom query option
values, we need to declare it by making new test dimension, followed by
deepcopying the original vector, and inserting the selected dimension
value into 'exec_option' dictionary in generated vector.
This patch simplify this steps by accounting dimensions that is intended
to be part of 'exec_option' and automatically combining them during
vector generation in test_vector.py. Such dimension should be registered
via the new ImpalaTestMatrix.add_exec_option_dimension() function.
function add_exec_option_dimension() in test_dimensions.py is renamed to
add_mandatory_exec_option() to make it consistent with the same
functionality in ImpalaTestMatrix and avoid confusion with the new
ImpalaTestMatrix.add_exec_option_dimension() function. Function name
add_exec_option_dimension() in test_dimensions.py is then repurposed as
a shorthand for ImpalaTestMatrix.add_exec_option_dimension().
The remaining changes for other pytest files will be done gradually.
Testing:
- Fix bug in TestIcebergV2Table and confirm that both True and False
value for 'disable_optimized_iceberg_v2_read' options are exercised.
- Run and pass all modified tests in this patch.
Change-Id: I3adba260990fccf4d2f2e7c8c4e4fadc6fd43fe1
Reviewed-on: http://gerrit.cloudera.org:8080/20625
Reviewed-by: Zoltan Borok-Nagy <bo...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
Reviewed-by: Michael Smith <mi...@cloudera.com>
---
M tests/common/test_dimensions.py
M tests/common/test_vector.py
M tests/custom_cluster/test_kudu.py
M tests/query_test/test_async_codegen.py
M tests/query_test/test_iceberg.py
M tests/query_test/test_kudu.py
M tests/query_test/test_runtime_filters.py
7 files changed, 86 insertions(+), 31 deletions(-)
Approvals:
Zoltan Borok-Nagy: Looks good to me, but someone else must approve
Impala Public Jenkins: Verified
Michael Smith: Looks good to me, approved
--
To view, visit http://gerrit.cloudera.org:8080/20625
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I3adba260990fccf4d2f2e7c8c4e4fadc6fd43fe1
Gerrit-Change-Number: 20625
Gerrit-PatchSet: 3
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
[Impala-ASF-CR] IMPALA-12518: Combine all exec option dimension in test vector.py
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20625 )
Change subject: IMPALA-12518: Combine all exec_option dimension in test_vector.py
......................................................................
Patch Set 2:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/9851/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/20625
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I3adba260990fccf4d2f2e7c8c4e4fadc6fd43fe1
Gerrit-Change-Number: 20625
Gerrit-PatchSet: 2
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Thu, 26 Oct 2023 20:38:08 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12518: Combine all exec option dimension in test vector.py
Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/20625 )
Change subject: IMPALA-12518: Combine all exec_option dimension in test_vector.py
......................................................................
Patch Set 2: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/20625
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I3adba260990fccf4d2f2e7c8c4e4fadc6fd43fe1
Gerrit-Change-Number: 20625
Gerrit-PatchSet: 2
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Fri, 27 Oct 2023 02:04:38 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12518: Combine all exec option dimension in test vector.py
Posted by "Riza Suminto (Code Review)" <ge...@cloudera.org>.
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/20625 )
Change subject: IMPALA-12518: Combine all exec_option dimension in test_vector.py
......................................................................
Patch Set 2:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/20625/1/tests/common/test_vector.py
File tests/common/test_vector.py:
http://gerrit.cloudera.org:8080/#/c/20625/1/tests/common/test_vector.py@111
PS1, Line 111: self.independent_exec_option_names = set()
Ran core tests overnight and found python compatibility issues around list(). Changed this to set() in patch set 2.
That should pass all core tests.
--
To view, visit http://gerrit.cloudera.org:8080/20625
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I3adba260990fccf4d2f2e7c8c4e4fadc6fd43fe1
Gerrit-Change-Number: 20625
Gerrit-PatchSet: 2
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Thu, 26 Oct 2023 16:19:49 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12518: Combine all exec option dimension in test vector.py
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20625 )
Change subject: IMPALA-12518: Combine all exec_option dimension in test_vector.py
......................................................................
Patch Set 1:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/14258/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/20625
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I3adba260990fccf4d2f2e7c8c4e4fadc6fd43fe1
Gerrit-Change-Number: 20625
Gerrit-PatchSet: 1
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Thu, 26 Oct 2023 05:40:26 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12518: Combine all exec option dimension in test vector.py
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20625 )
Change subject: IMPALA-12518: Combine all exec_option dimension in test_vector.py
......................................................................
Patch Set 2:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/14265/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/20625
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I3adba260990fccf4d2f2e7c8c4e4fadc6fd43fe1
Gerrit-Change-Number: 20625
Gerrit-PatchSet: 2
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Thu, 26 Oct 2023 16:44:55 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12518: Combine all exec option dimension in test vector.py
Posted by "Riza Suminto (Code Review)" <ge...@cloudera.org>.
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/20625 )
Change subject: IMPALA-12518: Combine all exec_option dimension in test_vector.py
......................................................................
Patch Set 3:
Thank you Zoltan and Michael!
--
To view, visit http://gerrit.cloudera.org:8080/20625
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I3adba260990fccf4d2f2e7c8c4e4fadc6fd43fe1
Gerrit-Change-Number: 20625
Gerrit-PatchSet: 3
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Fri, 27 Oct 2023 03:23:07 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12518: Combine all exec option dimension in test vector.py
Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/20625 )
Change subject: IMPALA-12518: Combine all exec_option dimension in test_vector.py
......................................................................
Patch Set 2: Code-Review+1
This refactoring makes sense to me. And thanks for fixing the Iceberg test bug.
--
To view, visit http://gerrit.cloudera.org:8080/20625
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I3adba260990fccf4d2f2e7c8c4e4fadc6fd43fe1
Gerrit-Change-Number: 20625
Gerrit-PatchSet: 2
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Thu, 26 Oct 2023 20:03:51 +0000
Gerrit-HasComments: No