You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Riza Suminto (Code Review)" <ge...@cloudera.org> on 2025/08/20 20:28:07 UTC
[Impala-ASF-CR] IMPALA-14333: [part 1] Make all pytest compatible with Python3
Riza Suminto has uploaded this change for review. ( http://gerrit.cloudera.org:8080/23319
Change subject: IMPALA-14333: [part 1] Make all pytest compatible with Python3
......................................................................
IMPALA-14333: [part 1] Make all pytest compatible with Python3
Running exhaustive tests with env var IMPALA_USE_PYTHON3_TESTS=true
reveals some tests that require adjustment. This patch made such
adjustment, which mostly revolves around encoding difference and string
vs bytes type in Python3. Followings are the details:
Moved one test case from delimited-latin-text.test to
test_delimited_text.py for easier binary comparison.
Recreate structs.parq (content inside
functional_parquet.complextypes_structs) from
functional_orc_def.complextypes_structs so that querying both of them
will return the same result simply by setting option
convert_legacy_hive_parquet_utc_timestamps=true. Setting
timezone='Europe/Budapest' is not required anymore.
Skip testing HBase at test_except and test_partitioned_top_n because
HBase does not return return rows where alltypesagg.tinyint_col is null.
Add _run_query_with_client() in test_ranger.py to allow reusing single
Impala client for running several queries. Also make sure to close the
clients when test is done.
Added ImpalaTestSuite._convert_bytes() as utility function to decode
bytes in Python3. This often needed when inspecting return value of
subprocess.check_output() as string.
Implement DataTypeMetaclass.__lt__ to substitute
DataTypeMetaclass.__cmp__ that is ignored in Python3 (see
https://peps.python.org/pep-0207/).
Fix WEB_CERT_ERR difference in test_ipv6.py.
Fix trivial integer parsing in test_restart_services.py.
Fix various encoding issues in test_saml2_sso.py,
test_shell_commandline.py, and test_shell_interactive.py.
Switch parallel loops in test_stress_infra.py to simple loop that
fail-proof in Python3.
Reduce test dimension in test_date_queries.py and
test_decimal_queries.py for json file format because table date_tbl and
decimal_tiny is contrained only for json/none/none (functional_json).
Switch to binary comparson in test_iceberg.py where needed.
Removed unused vector argument and fix some trivial flake8 issues.
Simplify create_impala_shell_executable_dimension to skip testing
python2 impala-shell when IMPALA_USE_PYTHON3_TESTS=true.
Switching IMPALA_USE_PYTHON3_TESTS to true by default will be addressed
in next patch.
Testing:
Pass exhaustive tests with IMPALA_USE_PYTHON3_TESTS=true.
Change-Id: I401a93b6cc7bcd17f41d24e7a310e0c882a550d4
---
M shell/impala_shell/impala_shell.py
M testdata/ComplexTypesTbl/structs.parq
M testdata/workloads/functional-query/queries/QueryTest/delimited-latin-text.test
M testdata/workloads/functional-query/queries/QueryTest/inline-view.test
M testdata/workloads/functional-query/queries/QueryTest/sort-complex.test
M testdata/workloads/functional-query/queries/QueryTest/top-n-complex.test
M tests/authorization/test_ranger.py
M tests/common/impala_test_suite.py
M tests/comparison/cluster.py
M tests/comparison/db_types.py
M tests/custom_cluster/test_ipv6.py
M tests/custom_cluster/test_restart_services.py
M tests/custom_cluster/test_saml2_sso.py
M tests/custom_cluster/test_scratch_disk.py
M tests/infra/test_stress_infra.py
M tests/query_test/test_date_queries.py
M tests/query_test/test_decimal_queries.py
M tests/query_test/test_delimited_text.py
M tests/query_test/test_iceberg.py
M tests/query_test/test_nested_types.py
M tests/query_test/test_queries.py
M tests/shell/test_shell_commandline.py
M tests/shell/test_shell_interactive.py
M tests/shell/util.py
M tests/statestore/test_statestore.py
M tests/stress/test_update_stress.py
M tests/util/shell_util.py
27 files changed, 333 insertions(+), 233 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/19/23319/1
--
To view, visit http://gerrit.cloudera.org:8080/23319
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I401a93b6cc7bcd17f41d24e7a310e0c882a550d4
Gerrit-Change-Number: 23319
Gerrit-PatchSet: 1
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
[Impala-ASF-CR] IMPALA-14333: [part 1] Make all pytest compatible with Python3
Posted by "Riza Suminto (Code Review)" <ge...@cloudera.org>.
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/23319 )
Change subject: IMPALA-14333: [part 1] Make all pytest compatible with Python3
......................................................................
Patch Set 3:
(5 comments)
http://gerrit.cloudera.org:8080/#/c/23319/1/tests/authorization/test_ranger.py
File tests/authorization/test_ranger.py:
http://gerrit.cloudera.org:8080/#/c/23319/1/tests/authorization/test_ranger.py@816
PS1, Line 816:
> To me, this function does not clarify or simplify the code. Instead, the c
I make it as function so that it will minimize the change when it replaces _run_query_as_user() calls.
I hope we don't need to backport this patch. But if we do, it should be easier this way.
http://gerrit.cloudera.org:8080/#/c/23319/1/tests/authorization/test_ranger.py@892
PS1, Line 892: admin_client = self.create_impala_client(user=ADMIN)
> Can both of these lines use the with self.create_impala_client() as client_
We can use with-as statement, but that will shift the spaces and cause larger changes.
Closing manually is more practical in this case.
http://gerrit.cloudera.org:8080/#/c/23319/1/tests/custom_cluster/test_saml2_sso.py
File tests/custom_cluster/test_saml2_sso.py:
http://gerrit.cloudera.org:8080/#/c/23319/1/tests/custom_cluster/test_saml2_sso.py@111
PS1, Line 111: SSO_ARGS_WITH_GROUP_FILTER = (
> Since this line is being modified, why not switch to using .format?
Done
http://gerrit.cloudera.org:8080/#/c/23319/1/tests/custom_cluster/test_saml2_sso.py@240
PS1, Line 240:
> I'm curious why this additional return variable is needed?
Leftover from my debugging. Removed.
http://gerrit.cloudera.org:8080/#/c/23319/2/tests/shell/test_shell_interactive.py
File tests/shell/test_shell_interactive.py:
http://gerrit.cloudera.org:8080/#/c/23319/2/tests/shell/test_shell_interactive.py@192
PS2, Line 192: cls.ImpalaTestMatrix.add_dimension(create_impala_shell_executable_dimension())
> Duplicate of https://gerrit.cloudera.org/c/23319/2/tests/common/impala_test
Removed.
--
To view, visit http://gerrit.cloudera.org:8080/23319
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I401a93b6cc7bcd17f41d24e7a310e0c882a550d4
Gerrit-Change-Number: 23319
Gerrit-PatchSet: 3
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Fri, 22 Aug 2025 00:08:28 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-14333: Run impala-py.test using Python3
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23319 )
Change subject: IMPALA-14333: Run impala-py.test using Python3
......................................................................
Patch Set 4:
Build Failed
https://jenkins.impala.io/job/gerrit-code-review-checks/20236/ : Initial code review checks failed. See linked job for details on the failure.
--
To view, visit http://gerrit.cloudera.org:8080/23319
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I401a93b6cc7bcd17f41d24e7a310e0c882a550d4
Gerrit-Change-Number: 23319
Gerrit-PatchSet: 4
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Fri, 22 Aug 2025 16:32:47 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14333: Run impala-py.test using Python3
Posted by "Jason Fehr (Code Review)" <ge...@cloudera.org>.
Jason Fehr has posted comments on this change. ( http://gerrit.cloudera.org:8080/23319 )
Change subject: IMPALA-14333: Run impala-py.test using Python3
......................................................................
Patch Set 17: Code-Review+2
(3 comments)
http://gerrit.cloudera.org:8080/#/c/23319/5//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/23319/5//COMMIT_MSG@42
PS5, Line 42: test_shell_commandline.py, and test_shell_interactive.py.
> Done
Done
http://gerrit.cloudera.org:8080/#/c/23319/5//COMMIT_MSG@45
PS5, Line 45:
> Personally, I rather move away from python2 totally. There are only few tes
Ack. I also misunderstood this. Python 2 test dimension will still get generated when system Python 2 is available which will run Python 2 tests against Python 2 impala-shell. The only dimension not generated is Python 3 tests against Python 2 impala-shell which makes sense.
https://gerrit.cloudera.org/c/23319/17/tests/shell/util.py#390
http://gerrit.cloudera.org:8080/#/c/23319/5//COMMIT_MSG@59
PS5, Line 59: Several test logic require modification due to intermittent issue in
> I suspect some custom cluster test unknowingly start HiveServer2 manually (
Done
--
To view, visit http://gerrit.cloudera.org:8080/23319
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I401a93b6cc7bcd17f41d24e7a310e0c882a550d4
Gerrit-Change-Number: 23319
Gerrit-PatchSet: 17
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Tue, 02 Sep 2025 19:07:40 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-14333: Run impala-py.test using Python3
Posted by "Riza Suminto (Code Review)" <ge...@cloudera.org>.
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/23319 )
Change subject: IMPALA-14333: Run impala-py.test using Python3
......................................................................
Patch Set 17:
> Patch Set 17:
>
> Thank you Michael and Jason for your review!
> I'll rebase this patch later today once IMPALA-13731 and other flaky tests fixes merged.
>
> I'll also need to remind everyone that once this patch merged, Impala will stop testing impala-shell + Python2 combo. If later on we realize it still needed somehow, we can revert the default value of IMPALA_USE_PYTHON3_TESTS back to false in impala-config.sh.
Forgot to mention, thanks to Csaba and Laszlo as well for your feedback!
--
To view, visit http://gerrit.cloudera.org:8080/23319
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I401a93b6cc7bcd17f41d24e7a310e0c882a550d4
Gerrit-Change-Number: 23319
Gerrit-PatchSet: 17
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Tue, 02 Sep 2025 19:16:15 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14333: Run impala-py.test using Python3
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23319 )
Change subject: IMPALA-14333: Run impala-py.test using Python3
......................................................................
Patch Set 16:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/12397/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/23319
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I401a93b6cc7bcd17f41d24e7a310e0c882a550d4
Gerrit-Change-Number: 23319
Gerrit-PatchSet: 16
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Fri, 29 Aug 2025 04:24:35 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14333: Run impala-py.test using Python3
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23319 )
Change subject: IMPALA-14333: Run impala-py.test using Python3
......................................................................
Patch Set 17: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/23319
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I401a93b6cc7bcd17f41d24e7a310e0c882a550d4
Gerrit-Change-Number: 23319
Gerrit-PatchSet: 17
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Sat, 30 Aug 2025 01:10:17 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14333: Run impala-py.test using Python3
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23319 )
Change subject: IMPALA-14333: Run impala-py.test using Python3
......................................................................
Patch Set 7: Verified-1
Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/12368/
--
To view, visit http://gerrit.cloudera.org:8080/23319
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I401a93b6cc7bcd17f41d24e7a310e0c882a550d4
Gerrit-Change-Number: 23319
Gerrit-PatchSet: 7
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Sun, 24 Aug 2025 09:04:26 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14333: Run impala-py.test using Python3
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23319 )
Change subject: IMPALA-14333: Run impala-py.test using Python3
......................................................................
Patch Set 5:
Build Failed
https://jenkins.impala.io/job/gerrit-code-review-checks/20237/ : Initial code review checks failed. See linked job for details on the failure.
--
To view, visit http://gerrit.cloudera.org:8080/23319
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I401a93b6cc7bcd17f41d24e7a310e0c882a550d4
Gerrit-Change-Number: 23319
Gerrit-PatchSet: 5
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Fri, 22 Aug 2025 16:54:08 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14333: [part 1] Make all pytest compatible with Python3
Posted by "Riza Suminto (Code Review)" <ge...@cloudera.org>.
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/23319 )
Change subject: IMPALA-14333: [part 1] Make all pytest compatible with Python3
......................................................................
Patch Set 3:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/23319/3//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/23319/3//COMMIT_MSG@63
PS3, Line 63: python2 impala-shell
> When I start this, Python3 pytest + Python2 shell test combo shows breakage
These are the failing one:
shell/test_shell_commandline.py::TestImpalaShell::()::test_international_characters
shell/test_shell_commandline.py::TestImpalaShell::()::test_output_file_utf8
shell/test_shell_commandline.py::TestImpalaShell::()::test_binary_as_string
shell/test_shell_commandline.py::TestImpalaShell::()::test_binary_display
shell/test_shell_commandline.py::TestImpalaShell::()::test_utf8_decoding_error_handling
shell/test_shell_commandline.py::TestImpalaShell::()::test_international_characters_profile
shell/test_shell_commandline.py::TestImpalaShell::()::test_international_characters_prettyprint_tabs
shell/test_shell_commandline.py::TestImpalaShell::()::test_international_characters
Stacktrace is like this in some of those:
E AssertionError: Cmd ['-B', '-q', "select '?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?,?, ?, ?, ?, ?, ?,?, ?, ?, ?, ?, ?, ?, ?, ?'"] was expected to succeed: /home/rsuminto/workspace/imp
ala/shell/build/python2_venv/local/lib/python2.7/site-packages/impala_shell/option_parser.py:423: UnicodeWarning: Unicode equal comparison failed to convert both arguments to Unicode - interpreting
them as being unequal
E if '--live_progress' in sys.argv and '--disable_live_progress' in sys.argv:
E /home/rsuminto/workspace/impala/shell/build/python2_venv/local/lib/python2.7/site-packages/impala_shell/option_parser.py:427: UnicodeWarning: Unicode equal comparison failed to convert both arguments to Unicode - interpreting them as being unequal
E if '--strict_hs2_protocol' in sys.argv:
E /home/rsuminto/workspace/impala/shell/build/python2_venv/local/lib/python2.7/site-packages/impala_shell/option_parser.py:433: UnicodeWarning: Unicode equal comparison failed to convert both argu
ments to Unicode - interpreting them as being unequal
E if '--verbose' in sys.argv and '--quiet' in sys.argv: E Starting Impala Shell with no authentication using Python 2.7.18
E Warning: live_progress only applies to interactive shell sessions, and is being skipped for now.
E Opened TCP connection to localhost:21050
E Connected to localhost:21050 E Server version: impalad version 5.0.0-SNAPSHOT DEBUG (build 644e8df61e6c5324ff5e1406982f0a9bfa952c9c) E Traceback (most recent call last): E File "/home/rsuminto/workspace/impala/shell/build/python2_venv/bin/impala-shell", line 8, in <module> E sys.exit(impala_shell_main())
E File "/home/rsuminto/workspace/impala/shell/build/python2_venv/local/lib/python2.7/site-packages/impala_shell/impala_shell.py", line 2480, in impala_shell_main
E if execute_queries_non_interactive_mode(options, query_options):
E File "/home/rsuminto/workspace/impala/shell/build/python2_venv/local/lib/python2.7/site-packages/impala_shell/impala_shell.py", line 2130, in execute_queries_non_interactive_mode
E and shell.execute_query_list(queries))
E File "/home/rsuminto/workspace/impala/shell/build/python2_venv/local/lib/python2.7/site-packages/impala_shell/impala_shell.py", line 1908, in execute_query_list
E if self.onecmd(q) is CmdStatus.ERROR:
E File "/home/rsuminto/workspace/impala/shell/build/python2_venv/local/lib/python2.7/site-packages/impala_shell/impala_shell.py", line 829, in onecmd
E return func(arg)
E File "/home/rsuminto/workspace/impala/shell/build/python2_venv/local/lib/python2.7/site-packages/impala_shell/impala_shell.py", line 1311, in do_select
E return self._execute_stmt(query_str, print_web_link=True)
E File "/home/rsuminto/workspace/impala/shell/build/python2_venv/local/lib/python2.7/site-packages/impala_shell/impala_shell.py", line 1466, in _execute_stmt
E self._print_if_verbose("Query: %s" % query_str)
E File "/home/rsuminto/workspace/impala/shell/build/python2_venv/local/lib/python2.7/site-packages/impala_shell/impala_shell.py", line 1218, in _print_if_verbose
E print(message, file=file_descriptor)
E UnicodeEncodeError: 'ascii' codec can't encode character u'\u0410' in position 15: ordinal not in range(128)
--
To view, visit http://gerrit.cloudera.org:8080/23319
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I401a93b6cc7bcd17f41d24e7a310e0c882a550d4
Gerrit-Change-Number: 23319
Gerrit-PatchSet: 3
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Fri, 22 Aug 2025 14:26:48 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-14333: Run impala-py.test using Python3
Posted by "Riza Suminto (Code Review)" <ge...@cloudera.org>.
Hello Laszlo Gaal, Jason Fehr, Csaba Ringhofer, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/23319
to look at the new patch set (#10).
Change subject: IMPALA-14333: Run impala-py.test using Python3
......................................................................
IMPALA-14333: Run impala-py.test using Python3
Running exhaustive tests with env var IMPALA_USE_PYTHON3_TESTS=true
reveals some tests that require adjustment. This patch made such
adjustment, which mostly revolves around encoding differences and string
vs bytes type in Python3. This patch also switch the default to run
pytest with Python3 by setting IMPALA_USE_PYTHON3_TESTS=true. The
following are the details:
Change hash() function in conftest.py to crc32() to produce
deterministic hash. Hash randomization is enabled by default since
Python 3.3 (see
https://docs.python.org/3/reference/datamodel.html#object.__hash__).
This cause test sharding (like --shard_tests=1/2) produce inconsistent
set of tests per shard. Always restart minicluster during custom cluster
tests if --shard_tests argument is set, because test order may change
and affect test correctness, depending on whether running on fresh
minicluster or not.
Moved one test case from delimited-latin-text.test to
test_delimited_text.py for easier binary comparison.
Add bytes_to_str() as a utility function to decode bytes in Python3.
This is often needed when inspecting the return value of
subprocess.check_output() as a string.
Implement DataTypeMetaclass.__lt__ to substitute
DataTypeMetaclass.__cmp__ that is ignored in Python3 (see
https://peps.python.org/pep-0207/).
Fix WEB_CERT_ERR difference in test_ipv6.py.
Fix trivial integer parsing in test_restart_services.py.
Fix various encoding issues in test_saml2_sso.py,
test_shell_commandline.py, and test_shell_interactive.py.
Change timeout in Impala.for_each_impalad() from sys.maxsize to 2^31-1.
Switch to binary comparison in test_iceberg.py where needed.
Specify text mode when calling tempfile.NamedTemporaryFile().
Simplify create_impala_shell_executable_dimension to skip testing
python2 impala-shell when IMPALA_USE_PYTHON3_TESTS=true. The reason is
that several UTF-8 related tests in test_shell_commandline.py break in
Python3 pytest + Python2 impala-shell combo.
Removed unused vector argument and fixed some trivial flake8 issues.
Several test logic require modification due to vector permutation
difference (see IMPALA-13125) and other intermittent issue in Python3
pytest. These include:
Add _run_query_with_client() in test_ranger.py to allow reusing a single
Impala client for running several queries. Ensure clients are closed
when the test is done. Mark several tests in test_ranger.py with
SkipIfFS.hive because they run queries through beeline + HiveServer2,
but Ozone and S3 build environment does not start HiveServer2 by
default.
Set timezone='Europe/Budapest' in test_sort and test_top_n to get
consistent result for reading functional_parquet.complextypes_structs
and functional_orc_def.complextypes_structs.
Skip testing HBase at test_except and test_partitioned_top_n because
HBase does not return rows where alltypesagg.tinyint_col is null.
Reduce test dimension in test_date_queries.py and
test_decimal_queries.py for json file format because table date_tbl and
decimal_tiny are contrained only for json/none/none (functional_json).
Increase the sleep period from 0.1 to 0.5 seconds per iteration in
test_statestore.py and mark TestStatestore to execute serially. This is
because TServer appears to shut down more slowly when run concurrently
with other tests. Handle the deprecation of Thread.setDaemon() as well.
Always force_restart=True each test method in TestLoggingCore,
TestShellInteractiveReconnect, and TestQueryRetries to prevent them from
reusing minicluster from previous test method. Some of these tests
destruct minicluster (kill impalad) and will produce minidump if metrics
verifier for next tests fail to detect healthy minicluster state.
Testing:
Pass exhaustive tests with IMPALA_USE_PYTHON3_TESTS=true.
Change-Id: I401a93b6cc7bcd17f41d24e7a310e0c882a550d4
---
M bin/impala-config.sh
M shell/impala_shell/impala_shell.py
M testdata/workloads/functional-query/queries/QueryTest/delimited-latin-text.test
M testdata/workloads/functional-query/queries/QueryTest/inline-view.test
M testdata/workloads/functional-query/queries/QueryTest/sort-complex.test
M testdata/workloads/functional-query/queries/QueryTest/top-n-complex.test
M tests/authorization/test_ranger.py
M tests/common/custom_cluster_test_suite.py
M tests/common/impala_test_suite.py
M tests/common/skip.py
M tests/comparison/cluster.py
M tests/comparison/db_types.py
M tests/conftest.py
M tests/custom_cluster/test_ipv6.py
M tests/custom_cluster/test_logging.py
M tests/custom_cluster/test_query_retries.py
M tests/custom_cluster/test_restart_services.py
M tests/custom_cluster/test_s3a_access.py
M tests/custom_cluster/test_saml2_sso.py
M tests/custom_cluster/test_scratch_disk.py
M tests/custom_cluster/test_shell_interactive_reconnect.py
M tests/query_test/test_date_queries.py
M tests/query_test/test_decimal_queries.py
M tests/query_test/test_delimited_text.py
M tests/query_test/test_iceberg.py
M tests/query_test/test_queries.py
M tests/shell/test_shell_commandline.py
M tests/shell/test_shell_interactive.py
M tests/shell/util.py
M tests/statestore/test_statestore.py
M tests/stress/test_update_stress.py
M tests/util/hdfs_util.py
M tests/util/parse_util.py
M tests/util/shell_util.py
34 files changed, 428 insertions(+), 270 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/19/23319/10
--
To view, visit http://gerrit.cloudera.org:8080/23319
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I401a93b6cc7bcd17f41d24e7a310e0c882a550d4
Gerrit-Change-Number: 23319
Gerrit-PatchSet: 10
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
[Impala-ASF-CR] IMPALA-14333: Run impala-py.test using Python3
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23319 )
Change subject: IMPALA-14333: Run impala-py.test using Python3
......................................................................
Patch Set 14: Verified-1
Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/12394/
--
To view, visit http://gerrit.cloudera.org:8080/23319
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I401a93b6cc7bcd17f41d24e7a310e0c882a550d4
Gerrit-Change-Number: 23319
Gerrit-PatchSet: 14
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Thu, 28 Aug 2025 21:53:28 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14333: Run impala-py.test using Python3
Posted by "Riza Suminto (Code Review)" <ge...@cloudera.org>.
Hello Laszlo Gaal, Jason Fehr, Csaba Ringhofer, Michael Smith, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/23319
to look at the new patch set (#13).
Change subject: IMPALA-14333: Run impala-py.test using Python3
......................................................................
IMPALA-14333: Run impala-py.test using Python3
Running exhaustive tests with env var IMPALA_USE_PYTHON3_TESTS=true
reveals some tests that require adjustment. This patch made such
adjustment, which mostly revolves around encoding differences and string
vs bytes type in Python3. This patch also switch the default to run
pytest with Python3 by setting IMPALA_USE_PYTHON3_TESTS=true. The
following are the details:
Change hash() function in conftest.py to crc32() to produce
deterministic hash. Hash randomization is enabled by default since
Python 3.3 (see
https://docs.python.org/3/reference/datamodel.html#object.__hash__).
This cause test sharding (like --shard_tests=1/2) produce inconsistent
set of tests per shard. Always restart minicluster during custom cluster
tests if --shard_tests argument is set, because test order may change
and affect test correctness, depending on whether running on fresh
minicluster or not.
Moved one test case from delimited-latin-text.test to
test_delimited_text.py for easier binary comparison.
Add bytes_to_str() as a utility function to decode bytes in Python3.
This is often needed when inspecting the return value of
subprocess.check_output() as a string.
Implement DataTypeMetaclass.__lt__ to substitute
DataTypeMetaclass.__cmp__ that is ignored in Python3 (see
https://peps.python.org/pep-0207/).
Fix WEB_CERT_ERR difference in test_ipv6.py.
Fix trivial integer parsing in test_restart_services.py.
Fix various encoding issues in test_saml2_sso.py,
test_shell_commandline.py, and test_shell_interactive.py.
Change timeout in Impala.for_each_impalad() from sys.maxsize to 2^31-1.
Switch to binary comparison in test_iceberg.py where needed.
Specify text mode when calling tempfile.NamedTemporaryFile().
Simplify create_impala_shell_executable_dimension to skip testing
python2 impala-shell when IMPALA_USE_PYTHON3_TESTS=true. The reason is
that several UTF-8 related tests in test_shell_commandline.py break in
Python3 pytest + Python2 impala-shell combo.
Removed unused vector argument and fixed some trivial flake8 issues.
Several test logic require modification due to vector permutation
difference (see IMPALA-13125) and other intermittent issue in Python3
pytest. These include:
Add _run_query_with_client() in test_ranger.py to allow reusing a single
Impala client for running several queries. Ensure clients are closed
when the test is done. Mark several tests in test_ranger.py with
SkipIfFS.hive because they run queries through beeline + HiveServer2,
but Ozone and S3 build environment does not start HiveServer2 by
default.
Skip testing HBase at test_except and test_partitioned_top_n because
HBase does not return rows where alltypesagg.tinyint_col is null.
Increase the sleep period from 0.1 to 0.5 seconds per iteration in
test_statestore.py and mark TestStatestore to execute serially. This is
because TServer appears to shut down more slowly when run concurrently
with other tests. Handle the deprecation of Thread.setDaemon() as well.
Always force_restart=True each test method in TestLoggingCore,
TestShellInteractiveReconnect, and TestQueryRetries to prevent them from
reusing minicluster from previous test method. Some of these tests
destruct minicluster (kill impalad) and will produce minidump if metrics
verifier for next tests fail to detect healthy minicluster state.
Testing:
Pass exhaustive tests with IMPALA_USE_PYTHON3_TESTS=true.
Change-Id: I401a93b6cc7bcd17f41d24e7a310e0c882a550d4
---
M bin/impala-config.sh
M shell/impala_shell/impala_shell.py
M testdata/workloads/functional-query/queries/QueryTest/delimited-latin-text.test
M tests/authorization/test_ranger.py
M tests/common/custom_cluster_test_suite.py
M tests/common/impala_test_suite.py
M tests/common/skip.py
M tests/comparison/cluster.py
M tests/comparison/db_types.py
M tests/conftest.py
M tests/custom_cluster/test_ipv6.py
M tests/custom_cluster/test_logging.py
M tests/custom_cluster/test_query_retries.py
M tests/custom_cluster/test_restart_services.py
M tests/custom_cluster/test_s3a_access.py
M tests/custom_cluster/test_saml2_sso.py
M tests/custom_cluster/test_scratch_disk.py
M tests/custom_cluster/test_shell_interactive_reconnect.py
M tests/query_test/test_delimited_text.py
M tests/query_test/test_iceberg.py
M tests/query_test/test_queries.py
M tests/shell/test_shell_commandline.py
M tests/shell/test_shell_interactive.py
M tests/shell/util.py
M tests/statestore/test_statestore.py
M tests/stress/test_update_stress.py
M tests/util/hdfs_util.py
M tests/util/parse_util.py
M tests/util/shell_util.py
29 files changed, 401 insertions(+), 262 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/19/23319/13
--
To view, visit http://gerrit.cloudera.org:8080/23319
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I401a93b6cc7bcd17f41d24e7a310e0c882a550d4
Gerrit-Change-Number: 23319
Gerrit-PatchSet: 13
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
[Impala-ASF-CR] IMPALA-14333: Run impala-py.test using Python3
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23319 )
Change subject: IMPALA-14333: Run impala-py.test using Python3
......................................................................
Patch Set 11:
Build Failed
https://jenkins.impala.io/job/gerrit-code-review-checks/20262/ : Initial code review checks failed. See linked job for details on the failure.
--
To view, visit http://gerrit.cloudera.org:8080/23319
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I401a93b6cc7bcd17f41d24e7a310e0c882a550d4
Gerrit-Change-Number: 23319
Gerrit-PatchSet: 11
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Tue, 26 Aug 2025 00:06:09 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14333: [part 1] Make all pytest compatible with Python3
Posted by "Riza Suminto (Code Review)" <ge...@cloudera.org>.
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/23319 )
Change subject: IMPALA-14333: [part 1] Make all pytest compatible with Python3
......................................................................
Patch Set 2:
> Patch Set 2:
>
> Build Failed
>
> https://jenkins.impala.io/job/gerrit-code-review-checks/20217/ : Initial code review checks failed. See linked job for details on the failure.
flake8 errors out here:
14:51:16 Traceback (most recent call last):
14:51:16 File "/var/lib/jenkins/workspace/gerrit-auto-critic-test/Impala/bin/jenkins/critique-gerrit-review.py", line 490, in <module>
14:51:16 comments = get_flake8_comments(base_revision, revision)
14:51:16 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
14:51:16 File "/var/lib/jenkins/workspace/gerrit-auto-critic-test/Impala/bin/jenkins/critique-gerrit-review.py", line 168, in get_flake8_comments
14:51:16 raise Exception("Did not expect flake8-diff to write to stderr:\n{0}".format(stderr))
14:51:16 Exception: Did not expect flake8-diff to write to stderr:
14:51:16 2025-08-20 21:51:14,780 flake8diff.utils ERROR b"<unknown>:890: SyntaxWarning: invalid escape sequence '\\$'\n<unknown>:898: SyntaxWarning: invalid escape sequence '\\$'\n<unknown>:1091: SyntaxWarning: invalid escape sequence '\\;'\n"
Not sure what happen, but local run in my machine does not produce any flake8 warnings.
--
To view, visit http://gerrit.cloudera.org:8080/23319
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I401a93b6cc7bcd17f41d24e7a310e0c882a550d4
Gerrit-Change-Number: 23319
Gerrit-PatchSet: 2
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Wed, 20 Aug 2025 23:26:29 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14333: [part 1] Make all pytest compatible with Python3
Posted by "Riza Suminto (Code Review)" <ge...@cloudera.org>.
Hello Jason Fehr, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/23319
to look at the new patch set (#3).
Change subject: IMPALA-14333: [part 1] Make all pytest compatible with Python3
......................................................................
IMPALA-14333: [part 1] Make all pytest compatible with Python3
Running exhaustive tests with env var IMPALA_USE_PYTHON3_TESTS=true
reveals some tests that require adjustment. This patch made such
adjustment, which mostly revolves around encoding difference and string
vs bytes type in Python3. Followings are the details:
Moved one test case from delimited-latin-text.test to
test_delimited_text.py for easier binary comparison.
Set timezone='Europe/Budapest' in test_sort and test_top_n to get
consistent result for reading functional_parquet.complextypes_structs
and functional_orc_def.complextypes_structs.
Skip testing HBase at test_except and test_partitioned_top_n because
HBase does not return return rows where alltypesagg.tinyint_col is null.
Add _run_query_with_client() in test_ranger.py to allow reusing single
Impala client for running several queries. Make sure to close the
clients when test is done. Mark several tests in test_ranger.py with
SkipIfFS.hive because they run query through beeline + HiveServer2, but
Ozone and S3 build environment does not start HiveServer2 by default.
Add bytes_to_str() as utility function to decode bytes in Python3. This
often needed when inspecting return value of subprocess.check_output()
as string.
Implement DataTypeMetaclass.__lt__ to substitute
DataTypeMetaclass.__cmp__ that is ignored in Python3 (see
https://peps.python.org/pep-0207/).
Fix WEB_CERT_ERR difference in test_ipv6.py.
Fix trivial integer parsing in test_restart_services.py.
Fix various encoding issues in test_saml2_sso.py,
test_shell_commandline.py, and test_shell_interactive.py.
Switch parallel loops in test_stress_infra.py to simple loop that
fail-proof in Python3.
Reduce test dimension in test_date_queries.py and
test_decimal_queries.py for json file format because table date_tbl and
decimal_tiny is contrained only for json/none/none (functional_json).
Switch to binary comparison in test_iceberg.py where needed.
Specify text mode when calling tempfile.NamedTemporaryFile().
Increase sleep period from 0.1 to 0.5 second per iteration in
test_statestore.py because TServer seem to shutdown slower. Remove
deprecated setDaemon() calls.
Removed unused vector argument and fix some trivial flake8 issues.
Simplify create_impala_shell_executable_dimension to skip testing
python2 impala-shell when IMPALA_USE_PYTHON3_TESTS=true.
Switching IMPALA_USE_PYTHON3_TESTS to true by default will be addressed
in next patch.
Testing:
Pass exhaustive tests with IMPALA_USE_PYTHON3_TESTS=true.
Change-Id: I401a93b6cc7bcd17f41d24e7a310e0c882a550d4
---
M shell/impala_shell/impala_shell.py
M testdata/workloads/functional-query/queries/QueryTest/delimited-latin-text.test
M testdata/workloads/functional-query/queries/QueryTest/inline-view.test
M testdata/workloads/functional-query/queries/QueryTest/sort-complex.test
M testdata/workloads/functional-query/queries/QueryTest/top-n-complex.test
M tests/authorization/test_ranger.py
M tests/common/impala_test_suite.py
M tests/common/skip.py
M tests/comparison/cluster.py
M tests/comparison/db_types.py
M tests/custom_cluster/test_ipv6.py
M tests/custom_cluster/test_restart_services.py
M tests/custom_cluster/test_s3a_access.py
M tests/custom_cluster/test_saml2_sso.py
M tests/custom_cluster/test_scratch_disk.py
M tests/infra/test_stress_infra.py
M tests/query_test/test_date_queries.py
M tests/query_test/test_decimal_queries.py
M tests/query_test/test_delimited_text.py
M tests/query_test/test_iceberg.py
M tests/query_test/test_queries.py
M tests/shell/test_shell_commandline.py
M tests/shell/test_shell_interactive.py
M tests/shell/util.py
M tests/statestore/test_statestore.py
M tests/stress/test_update_stress.py
M tests/util/hdfs_util.py
M tests/util/parse_util.py
M tests/util/shell_util.py
29 files changed, 364 insertions(+), 243 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/19/23319/3
--
To view, visit http://gerrit.cloudera.org:8080/23319
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I401a93b6cc7bcd17f41d24e7a310e0c882a550d4
Gerrit-Change-Number: 23319
Gerrit-PatchSet: 3
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
[Impala-ASF-CR] IMPALA-14333: Run impala-py.test using Python3
Posted by "Riza Suminto (Code Review)" <ge...@cloudera.org>.
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/23319 )
Change subject: IMPALA-14333: Run impala-py.test using Python3
......................................................................
Patch Set 10:
> Patch Set 10:
>
> Build Failed
>
> https://jenkins.impala.io/job/gerrit-code-review-checks/20254/ : Initial code review checks failed. See linked job for details on the failure.
This failed due to intermittent issue in two subtask
https://jenkins.impala.io/job/ubuntu-20.04-dockerised-tests/3907/
https://jenkins.impala.io/job/all-build-options-ub2004/3074/
Retried them and both pass
https://jenkins.impala.io/job/ubuntu-20.04-dockerised-tests/3909/
https://jenkins.impala.io/job/all-build-options-ub2004/3076/
This patch set is ready for review.
--
To view, visit http://gerrit.cloudera.org:8080/23319
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I401a93b6cc7bcd17f41d24e7a310e0c882a550d4
Gerrit-Change-Number: 23319
Gerrit-PatchSet: 10
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Mon, 25 Aug 2025 21:48:45 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14333: Run impala-py.test using Python3
Posted by "Riza Suminto (Code Review)" <ge...@cloudera.org>.
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/23319 )
Change subject: IMPALA-14333: Run impala-py.test using Python3
......................................................................
Patch Set 17:
Thank you Michael and Jason for your review!
I'll rebase this patch later today once IMPALA-13731 and other flaky tests fixes merged.
I'll also need to remind everyone that once this patch merged, Impala will stop testing impala-shell + Python2 combo. If later on we realize it still needed somehow, we can revert the default value of IMPALA_USE_PYTHON3_TESTS back to false in impala-config.sh.
--
To view, visit http://gerrit.cloudera.org:8080/23319
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I401a93b6cc7bcd17f41d24e7a310e0c882a550d4
Gerrit-Change-Number: 23319
Gerrit-PatchSet: 17
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Tue, 02 Sep 2025 19:13:24 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14333: Run impala-py.test using Python3
Posted by "Riza Suminto (Code Review)" <ge...@cloudera.org>.
Hello Laszlo Gaal, Jason Fehr, Csaba Ringhofer, Michael Smith, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/23319
to look at the new patch set (#14).
Change subject: IMPALA-14333: Run impala-py.test using Python3
......................................................................
IMPALA-14333: Run impala-py.test using Python3
Running exhaustive tests with env var IMPALA_USE_PYTHON3_TESTS=true
reveals some tests that require adjustment. This patch made such
adjustment, which mostly revolves around encoding differences and string
vs bytes type in Python3. This patch also switch the default to run
pytest with Python3 by setting IMPALA_USE_PYTHON3_TESTS=true. The
following are the details:
Change hash() function in conftest.py to crc32() to produce
deterministic hash. Hash randomization is enabled by default since
Python 3.3 (see
https://docs.python.org/3/reference/datamodel.html#object.__hash__).
This cause test sharding (like --shard_tests=1/2) produce inconsistent
set of tests per shard. Always restart minicluster during custom cluster
tests if --shard_tests argument is set, because test order may change
and affect test correctness, depending on whether running on fresh
minicluster or not.
Moved one test case from delimited-latin-text.test to
test_delimited_text.py for easier binary comparison.
Add bytes_to_str() as a utility function to decode bytes in Python3.
This is often needed when inspecting the return value of
subprocess.check_output() as a string.
Implement DataTypeMetaclass.__lt__ to substitute
DataTypeMetaclass.__cmp__ that is ignored in Python3 (see
https://peps.python.org/pep-0207/).
Fix WEB_CERT_ERR difference in test_ipv6.py.
Fix trivial integer parsing in test_restart_services.py.
Fix various encoding issues in test_saml2_sso.py,
test_shell_commandline.py, and test_shell_interactive.py.
Change timeout in Impala.for_each_impalad() from sys.maxsize to 2^31-1.
Switch to binary comparison in test_iceberg.py where needed.
Specify text mode when calling tempfile.NamedTemporaryFile().
Simplify create_impala_shell_executable_dimension to skip testing
python2 impala-shell when IMPALA_USE_PYTHON3_TESTS=true. The reason is
that several UTF-8 related tests in test_shell_commandline.py break in
Python3 pytest + Python2 impala-shell combo.
Removed unused vector argument and fixed some trivial flake8 issues.
Several test logic require modification due to intermittent issue in
Python3 pytest. These include:
Add _run_query_with_client() in test_ranger.py to allow reusing a single
Impala client for running several queries. Ensure clients are closed
when the test is done. Mark several tests in test_ranger.py with
SkipIfFS.hive because they run queries through beeline + HiveServer2,
but Ozone and S3 build environment does not start HiveServer2 by
default.
Increase the sleep period from 0.1 to 0.5 seconds per iteration in
test_statestore.py and mark TestStatestore to execute serially. This is
because TServer appears to shut down more slowly when run concurrently
with other tests. Handle the deprecation of Thread.setDaemon() as well.
Always force_restart=True each test method in TestLoggingCore,
TestShellInteractiveReconnect, and TestQueryRetries to prevent them from
reusing minicluster from previous test method. Some of these tests
destruct minicluster (kill impalad) and will produce minidump if metrics
verifier for next tests fail to detect healthy minicluster state.
Testing:
Pass exhaustive tests with IMPALA_USE_PYTHON3_TESTS=true.
Change-Id: I401a93b6cc7bcd17f41d24e7a310e0c882a550d4
---
M bin/impala-config.sh
M shell/impala_shell/impala_shell.py
M testdata/workloads/functional-query/queries/QueryTest/delimited-latin-text.test
M tests/authorization/test_ranger.py
M tests/common/custom_cluster_test_suite.py
M tests/common/impala_test_suite.py
M tests/common/skip.py
M tests/comparison/cluster.py
M tests/comparison/db_types.py
M tests/conftest.py
M tests/custom_cluster/test_ipv6.py
M tests/custom_cluster/test_logging.py
M tests/custom_cluster/test_query_retries.py
M tests/custom_cluster/test_restart_services.py
M tests/custom_cluster/test_s3a_access.py
M tests/custom_cluster/test_saml2_sso.py
M tests/custom_cluster/test_scratch_disk.py
M tests/custom_cluster/test_shell_interactive_reconnect.py
M tests/query_test/test_delimited_text.py
M tests/query_test/test_iceberg.py
M tests/query_test/test_queries.py
M tests/shell/test_shell_commandline.py
M tests/shell/test_shell_interactive.py
M tests/shell/util.py
M tests/statestore/test_statestore.py
M tests/stress/test_update_stress.py
M tests/util/hdfs_util.py
M tests/util/parse_util.py
M tests/util/shell_util.py
29 files changed, 401 insertions(+), 262 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/19/23319/14
--
To view, visit http://gerrit.cloudera.org:8080/23319
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I401a93b6cc7bcd17f41d24e7a310e0c882a550d4
Gerrit-Change-Number: 23319
Gerrit-PatchSet: 14
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
[Impala-ASF-CR] IMPALA-14333: Run impala-py.test using Python3
Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/23319 )
Change subject: IMPALA-14333: Run impala-py.test using Python3
......................................................................
Patch Set 15: Code-Review+1
--
To view, visit http://gerrit.cloudera.org:8080/23319
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I401a93b6cc7bcd17f41d24e7a310e0c882a550d4
Gerrit-Change-Number: 23319
Gerrit-PatchSet: 15
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Thu, 28 Aug 2025 22:54:51 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14333: Run impala-py.test using Python3
Posted by "Riza Suminto (Code Review)" <ge...@cloudera.org>.
Hello Jason Fehr, Csaba Ringhofer, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/23319
to look at the new patch set (#6).
Change subject: IMPALA-14333: Run impala-py.test using Python3
......................................................................
IMPALA-14333: Run impala-py.test using Python3
Running exhaustive tests with env var IMPALA_USE_PYTHON3_TESTS=true
reveals some tests that require adjustment. This patch made such
adjustment, which mostly revolves around encoding differences and string
vs bytes type in Python3. This patch also switch the default to run
pytest with Python3 by setting IMPALA_USE_PYTHON3_TESTS=true. The
following are the details:
Moved one test case from delimited-latin-text.test to
test_delimited_text.py for easier binary comparison.
Add bytes_to_str() as a utility function to decode bytes in Python3.
This is often needed when inspecting the return value of
subprocess.check_output() as a string.
Implement DataTypeMetaclass.__lt__ to substitute
DataTypeMetaclass.__cmp__ that is ignored in Python3 (see
https://peps.python.org/pep-0207/).
Fix WEB_CERT_ERR difference in test_ipv6.py.
Fix trivial integer parsing in test_restart_services.py.
Fix various encoding issues in test_saml2_sso.py,
test_shell_commandline.py, and test_shell_interactive.py.
Switch parallel loops in test_stress_infra.py to a simple loop that
fail-proof in Python3.
Switch to binary comparison in test_iceberg.py where needed.
Specify text mode when calling tempfile.NamedTemporaryFile().
Simplify create_impala_shell_executable_dimension to skip testing
python2 impala-shell when IMPALA_USE_PYTHON3_TESTS=true. The reason is
that several UTF-8 related tests in test_shell_commandline.py break in
Python3 pytest + Python2 impala-shell combo.
Removed unused vector argument and fixed some trivial flake8 issues.
Several test logic require modification due to vector permutation
difference (see IMPALA-13125) and other intermittent issue in Python3
pytest. These include:
Add _run_query_with_client() in test_ranger.py to allow reusing a single
Impala client for running several queries. Ensure clients are closed
when the test is done. Mark several tests in test_ranger.py with
SkipIfFS.hive because they run queries through beeline + HiveServer2,
but Ozone and S3 build environment does not start HiveServer2 by
default.
Set timezone='Europe/Budapest' in test_sort and test_top_n to get
consistent result for reading functional_parquet.complextypes_structs
and functional_orc_def.complextypes_structs.
Skip testing HBase at test_except and test_partitioned_top_n because
HBase does not return rows where alltypesagg.tinyint_col is null.
Reduce test dimension in test_date_queries.py and
test_decimal_queries.py for json file format because table date_tbl and
decimal_tiny are contrained only for json/none/none (functional_json).
Increase the sleep period from 0.1 to 0.5 seconds per iteration in
test_statestore.py and mark TestStatestore to execute serially. This is
because TServer appears to shut down more slowly when run concurrently
with other tests. Handle the deprecation of Thread.setDaemon() as well.
Testing:
Pass exhaustive tests with IMPALA_USE_PYTHON3_TESTS=true.
Change-Id: I401a93b6cc7bcd17f41d24e7a310e0c882a550d4
---
M bin/impala-config.sh
M shell/impala_shell/impala_shell.py
M testdata/workloads/functional-query/queries/QueryTest/delimited-latin-text.test
M testdata/workloads/functional-query/queries/QueryTest/inline-view.test
M testdata/workloads/functional-query/queries/QueryTest/sort-complex.test
M testdata/workloads/functional-query/queries/QueryTest/top-n-complex.test
M tests/authorization/test_ranger.py
M tests/common/impala_test_suite.py
M tests/common/skip.py
M tests/comparison/cluster.py
M tests/comparison/db_types.py
M tests/custom_cluster/test_ipv6.py
M tests/custom_cluster/test_restart_services.py
M tests/custom_cluster/test_s3a_access.py
M tests/custom_cluster/test_saml2_sso.py
M tests/custom_cluster/test_scratch_disk.py
M tests/infra/test_stress_infra.py
M tests/query_test/test_date_queries.py
M tests/query_test/test_decimal_queries.py
M tests/query_test/test_delimited_text.py
M tests/query_test/test_iceberg.py
M tests/query_test/test_queries.py
M tests/shell/test_shell_commandline.py
M tests/shell/test_shell_interactive.py
M tests/shell/util.py
M tests/statestore/test_statestore.py
M tests/stress/test_update_stress.py
M tests/util/hdfs_util.py
M tests/util/parse_util.py
M tests/util/shell_util.py
30 files changed, 381 insertions(+), 243 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/19/23319/6
--
To view, visit http://gerrit.cloudera.org:8080/23319
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I401a93b6cc7bcd17f41d24e7a310e0c882a550d4
Gerrit-Change-Number: 23319
Gerrit-PatchSet: 6
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
[Impala-ASF-CR] IMPALA-14333: Run impala-py.test using Python3
Posted by "Riza Suminto (Code Review)" <ge...@cloudera.org>.
Hello Jason Fehr, Csaba Ringhofer, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/23319
to look at the new patch set (#5).
Change subject: IMPALA-14333: Run impala-py.test using Python3
......................................................................
IMPALA-14333: Run impala-py.test using Python3
Running exhaustive tests with env var IMPALA_USE_PYTHON3_TESTS=true
reveals some tests that require adjustment. This patch made such
adjustment, which mostly revolves around encoding differences and string
vs bytes type in Python3. This patch also switch the default to run
pytest with Python3 by setting IMPALA_USE_PYTHON3_TESTS=true. The
following are the details:
Moved one test case from delimited-latin-text.test to
test_delimited_text.py for easier binary comparison.
Add bytes_to_str() as a utility function to decode bytes in Python3.
This is often needed when inspecting the return value of
subprocess.check_output() as a string.
Implement DataTypeMetaclass.__lt__ to substitute
DataTypeMetaclass.__cmp__ that is ignored in Python3 (see
https://peps.python.org/pep-0207/).
Fix WEB_CERT_ERR difference in test_ipv6.py.
Fix trivial integer parsing in test_restart_services.py.
Fix various encoding issues in test_saml2_sso.py,
test_shell_commandline.py, and test_shell_interactive.py.
Switch parallel loops in test_stress_infra.py to a simple loop that
fail-proof in Python3.
Switch to binary comparison in test_iceberg.py where needed.
Specify text mode when calling tempfile.NamedTemporaryFile().
Simplify create_impala_shell_executable_dimension to skip testing
python2 impala-shell when IMPALA_USE_PYTHON3_TESTS=true. The reason is
that several UTF-8 related tests in test_shell_commandline.py break in
Python3 pytest + Python2 impala-shell combo.
Removed unused vector argument and fixed some trivial flake8 issues.
Several test logic require modification due to vector permutation
difference (see IMPALA-13125) and other intermittent issue in Python3
pytest. These include:
Add _run_query_with_client() in test_ranger.py to allow reusing a single
Impala client for running several queries. Ensure clients are closed
when the test is done. Mark several tests in test_ranger.py with
SkipIfFS.hive because they run queries through beeline + HiveServer2,
but Ozone and S3 build environment does not start HiveServer2 by
default.
Set timezone='Europe/Budapest' in test_sort and test_top_n to get
consistent result for reading functional_parquet.complextypes_structs
and functional_orc_def.complextypes_structs.
Skip testing HBase at test_except and test_partitioned_top_n because
HBase does not return rows where alltypesagg.tinyint_col is null.
Reduce test dimension in test_date_queries.py and
test_decimal_queries.py for json file format because table date_tbl and
decimal_tiny are contrained only for json/none/none (functional_json).
Increase the sleep period from 0.1 to 0.5 seconds per iteration in
test_statestore.py and mark TestStatestore to execute serially. This is
because TServer appears to shut down more slowly when run concurrently
with other tests. Handle the deprecation of Thread.setDaemon() as well.
Testing:
Pass exhaustive tests with IMPALA_USE_PYTHON3_TESTS=true.
Change-Id: I401a93b6cc7bcd17f41d24e7a310e0c882a550d4
---
M bin/impala-config.sh
M shell/impala_shell/impala_shell.py
M testdata/workloads/functional-query/queries/QueryTest/delimited-latin-text.test
M testdata/workloads/functional-query/queries/QueryTest/inline-view.test
M testdata/workloads/functional-query/queries/QueryTest/sort-complex.test
M testdata/workloads/functional-query/queries/QueryTest/top-n-complex.test
M tests/authorization/test_ranger.py
M tests/common/impala_test_suite.py
M tests/common/skip.py
M tests/comparison/cluster.py
M tests/comparison/db_types.py
M tests/custom_cluster/test_ipv6.py
M tests/custom_cluster/test_restart_services.py
M tests/custom_cluster/test_s3a_access.py
M tests/custom_cluster/test_saml2_sso.py
M tests/custom_cluster/test_scratch_disk.py
M tests/infra/test_stress_infra.py
M tests/query_test/test_date_queries.py
M tests/query_test/test_decimal_queries.py
M tests/query_test/test_delimited_text.py
M tests/query_test/test_iceberg.py
M tests/query_test/test_queries.py
M tests/shell/test_shell_commandline.py
M tests/shell/test_shell_interactive.py
M tests/shell/util.py
M tests/statestore/test_statestore.py
M tests/stress/test_update_stress.py
M tests/util/hdfs_util.py
M tests/util/parse_util.py
M tests/util/shell_util.py
30 files changed, 381 insertions(+), 243 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/19/23319/5
--
To view, visit http://gerrit.cloudera.org:8080/23319
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I401a93b6cc7bcd17f41d24e7a310e0c882a550d4
Gerrit-Change-Number: 23319
Gerrit-PatchSet: 5
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
[Impala-ASF-CR] IMPALA-14333: Run impala-py.test using Python3
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23319 )
Change subject: IMPALA-14333: Run impala-py.test using Python3
......................................................................
Patch Set 10:
Build Failed
https://jenkins.impala.io/job/gerrit-code-review-checks/20254/ : Initial code review checks failed. See linked job for details on the failure.
--
To view, visit http://gerrit.cloudera.org:8080/23319
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I401a93b6cc7bcd17f41d24e7a310e0c882a550d4
Gerrit-Change-Number: 23319
Gerrit-PatchSet: 10
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Mon, 25 Aug 2025 14:18:47 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14333: Run impala-py.test using Python3
Posted by "Riza Suminto (Code Review)" <ge...@cloudera.org>.
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/23319 )
Change subject: IMPALA-14333: Run impala-py.test using Python3
......................................................................
Patch Set 12:
> Patch Set 12:
>
> I think that the test fixes for the issues revealed by the different test vector generations should be merged in a separate commit than the Py2->3 change, as it seems just an accident the Python version change led to more test vectors.
Ack. Will wait for https://gerrit.cloudera.org/#/c/23342/.
--
To view, visit http://gerrit.cloudera.org:8080/23319
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I401a93b6cc7bcd17f41d24e7a310e0c882a550d4
Gerrit-Change-Number: 23319
Gerrit-PatchSet: 12
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Tue, 26 Aug 2025 17:29:40 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14333: Run impala-py.test using Python3
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23319 )
Change subject: IMPALA-14333: Run impala-py.test using Python3
......................................................................
Patch Set 6: Verified-1
Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/12362/
--
To view, visit http://gerrit.cloudera.org:8080/23319
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I401a93b6cc7bcd17f41d24e7a310e0c882a550d4
Gerrit-Change-Number: 23319
Gerrit-PatchSet: 6
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Sat, 23 Aug 2025 04:07:53 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14333: Run impala-py.test using Python3
Posted by "Riza Suminto (Code Review)" <ge...@cloudera.org>.
Hello Jason Fehr, Csaba Ringhofer, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/23319
to look at the new patch set (#8).
Change subject: IMPALA-14333: Run impala-py.test using Python3
......................................................................
IMPALA-14333: Run impala-py.test using Python3
Running exhaustive tests with env var IMPALA_USE_PYTHON3_TESTS=true
reveals some tests that require adjustment. This patch made such
adjustment, which mostly revolves around encoding differences and string
vs bytes type in Python3. This patch also switch the default to run
pytest with Python3 by setting IMPALA_USE_PYTHON3_TESTS=true. The
following are the details:
Moved one test case from delimited-latin-text.test to
test_delimited_text.py for easier binary comparison.
Add bytes_to_str() as a utility function to decode bytes in Python3.
This is often needed when inspecting the return value of
subprocess.check_output() as a string.
Implement DataTypeMetaclass.__lt__ to substitute
DataTypeMetaclass.__cmp__ that is ignored in Python3 (see
https://peps.python.org/pep-0207/).
Fix WEB_CERT_ERR difference in test_ipv6.py.
Fix trivial integer parsing in test_restart_services.py.
Fix various encoding issues in test_saml2_sso.py,
test_shell_commandline.py, and test_shell_interactive.py.
Change timeout in Impala.for_each_impalad() from sys.maxsize to 2^31-1.
Switch to binary comparison in test_iceberg.py where needed.
Specify text mode when calling tempfile.NamedTemporaryFile().
Simplify create_impala_shell_executable_dimension to skip testing
python2 impala-shell when IMPALA_USE_PYTHON3_TESTS=true. The reason is
that several UTF-8 related tests in test_shell_commandline.py break in
Python3 pytest + Python2 impala-shell combo.
Removed unused vector argument and fixed some trivial flake8 issues.
Several test logic require modification due to vector permutation
difference (see IMPALA-13125) and other intermittent issue in Python3
pytest. These include:
Add _run_query_with_client() in test_ranger.py to allow reusing a single
Impala client for running several queries. Ensure clients are closed
when the test is done. Mark several tests in test_ranger.py with
SkipIfFS.hive because they run queries through beeline + HiveServer2,
but Ozone and S3 build environment does not start HiveServer2 by
default.
Set timezone='Europe/Budapest' in test_sort and test_top_n to get
consistent result for reading functional_parquet.complextypes_structs
and functional_orc_def.complextypes_structs.
Skip testing HBase at test_except and test_partitioned_top_n because
HBase does not return rows where alltypesagg.tinyint_col is null.
Reduce test dimension in test_date_queries.py and
test_decimal_queries.py for json file format because table date_tbl and
decimal_tiny are contrained only for json/none/none (functional_json).
Increase the sleep period from 0.1 to 0.5 seconds per iteration in
test_statestore.py and mark TestStatestore to execute serially. This is
because TServer appears to shut down more slowly when run concurrently
with other tests. Handle the deprecation of Thread.setDaemon() as well.
Always force_restart=True each test method in
TestShellInteractiveReconnect and TestQueryRetries to prevent them from
reusing minicluster from previous test method. Some of these tests
destruct minicluster (kill impalad) and will produce minidump if metrics
verifier for next tests fail to detect healty minicluster state.
Testing:
Pass exhaustive tests with IMPALA_USE_PYTHON3_TESTS=true.
Change-Id: I401a93b6cc7bcd17f41d24e7a310e0c882a550d4
---
M bin/impala-config.sh
M shell/impala_shell/impala_shell.py
M testdata/workloads/functional-query/queries/QueryTest/delimited-latin-text.test
M testdata/workloads/functional-query/queries/QueryTest/inline-view.test
M testdata/workloads/functional-query/queries/QueryTest/sort-complex.test
M testdata/workloads/functional-query/queries/QueryTest/top-n-complex.test
M tests/authorization/test_ranger.py
M tests/common/custom_cluster_test_suite.py
M tests/common/impala_test_suite.py
M tests/common/skip.py
M tests/comparison/cluster.py
M tests/comparison/db_types.py
M tests/custom_cluster/test_ipv6.py
M tests/custom_cluster/test_query_retries.py
M tests/custom_cluster/test_restart_services.py
M tests/custom_cluster/test_s3a_access.py
M tests/custom_cluster/test_saml2_sso.py
M tests/custom_cluster/test_scratch_disk.py
M tests/custom_cluster/test_shell_interactive_reconnect.py
M tests/query_test/test_date_queries.py
M tests/query_test/test_decimal_queries.py
M tests/query_test/test_delimited_text.py
M tests/query_test/test_iceberg.py
M tests/query_test/test_queries.py
M tests/shell/test_shell_commandline.py
M tests/shell/test_shell_interactive.py
M tests/shell/util.py
M tests/statestore/test_statestore.py
M tests/stress/test_update_stress.py
M tests/util/hdfs_util.py
M tests/util/parse_util.py
M tests/util/shell_util.py
32 files changed, 416 insertions(+), 263 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/19/23319/8
--
To view, visit http://gerrit.cloudera.org:8080/23319
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I401a93b6cc7bcd17f41d24e7a310e0c882a550d4
Gerrit-Change-Number: 23319
Gerrit-PatchSet: 8
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
[Impala-ASF-CR] IMPALA-14333: Run impala-py.test using Python3
Posted by "Riza Suminto (Code Review)" <ge...@cloudera.org>.
Hello Laszlo Gaal, Jason Fehr, Csaba Ringhofer, Michael Smith, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/23319
to look at the new patch set (#12).
Change subject: IMPALA-14333: Run impala-py.test using Python3
......................................................................
IMPALA-14333: Run impala-py.test using Python3
Running exhaustive tests with env var IMPALA_USE_PYTHON3_TESTS=true
reveals some tests that require adjustment. This patch made such
adjustment, which mostly revolves around encoding differences and string
vs bytes type in Python3. This patch also switch the default to run
pytest with Python3 by setting IMPALA_USE_PYTHON3_TESTS=true. The
following are the details:
Change hash() function in conftest.py to crc32() to produce
deterministic hash. Hash randomization is enabled by default since
Python 3.3 (see
https://docs.python.org/3/reference/datamodel.html#object.__hash__).
This cause test sharding (like --shard_tests=1/2) produce inconsistent
set of tests per shard. Always restart minicluster during custom cluster
tests if --shard_tests argument is set, because test order may change
and affect test correctness, depending on whether running on fresh
minicluster or not.
Moved one test case from delimited-latin-text.test to
test_delimited_text.py for easier binary comparison.
Add bytes_to_str() as a utility function to decode bytes in Python3.
This is often needed when inspecting the return value of
subprocess.check_output() as a string.
Implement DataTypeMetaclass.__lt__ to substitute
DataTypeMetaclass.__cmp__ that is ignored in Python3 (see
https://peps.python.org/pep-0207/).
Fix WEB_CERT_ERR difference in test_ipv6.py.
Fix trivial integer parsing in test_restart_services.py.
Fix various encoding issues in test_saml2_sso.py,
test_shell_commandline.py, and test_shell_interactive.py.
Change timeout in Impala.for_each_impalad() from sys.maxsize to 2^31-1.
Switch to binary comparison in test_iceberg.py where needed.
Specify text mode when calling tempfile.NamedTemporaryFile().
Simplify create_impala_shell_executable_dimension to skip testing
python2 impala-shell when IMPALA_USE_PYTHON3_TESTS=true. The reason is
that several UTF-8 related tests in test_shell_commandline.py break in
Python3 pytest + Python2 impala-shell combo.
Removed unused vector argument and fixed some trivial flake8 issues.
Several test logic require modification due to vector permutation
difference (see IMPALA-13125) and other intermittent issue in Python3
pytest. These include:
Add _run_query_with_client() in test_ranger.py to allow reusing a single
Impala client for running several queries. Ensure clients are closed
when the test is done. Mark several tests in test_ranger.py with
SkipIfFS.hive because they run queries through beeline + HiveServer2,
but Ozone and S3 build environment does not start HiveServer2 by
default.
Set timezone='Europe/Budapest' in test_sort and test_top_n to get
consistent result for reading functional_parquet.complextypes_structs
and functional_orc_def.complextypes_structs.
Skip testing HBase at test_except and test_partitioned_top_n because
HBase does not return rows where alltypesagg.tinyint_col is null.
Reduce test dimension in test_date_queries.py and
test_decimal_queries.py for json file format because table date_tbl and
decimal_tiny are contrained only for json/none/none (functional_json).
Increase the sleep period from 0.1 to 0.5 seconds per iteration in
test_statestore.py and mark TestStatestore to execute serially. This is
because TServer appears to shut down more slowly when run concurrently
with other tests. Handle the deprecation of Thread.setDaemon() as well.
Always force_restart=True each test method in TestLoggingCore,
TestShellInteractiveReconnect, and TestQueryRetries to prevent them from
reusing minicluster from previous test method. Some of these tests
destruct minicluster (kill impalad) and will produce minidump if metrics
verifier for next tests fail to detect healthy minicluster state.
Testing:
Pass exhaustive tests with IMPALA_USE_PYTHON3_TESTS=true.
Change-Id: I401a93b6cc7bcd17f41d24e7a310e0c882a550d4
---
M bin/impala-config.sh
M shell/impala_shell/impala_shell.py
M testdata/workloads/functional-query/queries/QueryTest/delimited-latin-text.test
M testdata/workloads/functional-query/queries/QueryTest/inline-view.test
M testdata/workloads/functional-query/queries/QueryTest/sort-complex.test
M testdata/workloads/functional-query/queries/QueryTest/top-n-complex.test
M tests/authorization/test_ranger.py
M tests/common/custom_cluster_test_suite.py
M tests/common/impala_test_suite.py
M tests/common/skip.py
M tests/comparison/cluster.py
M tests/comparison/db_types.py
M tests/conftest.py
M tests/custom_cluster/test_ipv6.py
M tests/custom_cluster/test_logging.py
M tests/custom_cluster/test_query_retries.py
M tests/custom_cluster/test_restart_services.py
M tests/custom_cluster/test_s3a_access.py
M tests/custom_cluster/test_saml2_sso.py
M tests/custom_cluster/test_scratch_disk.py
M tests/custom_cluster/test_shell_interactive_reconnect.py
M tests/query_test/test_date_queries.py
M tests/query_test/test_decimal_queries.py
M tests/query_test/test_delimited_text.py
M tests/query_test/test_iceberg.py
M tests/query_test/test_queries.py
M tests/shell/test_shell_commandline.py
M tests/shell/test_shell_interactive.py
M tests/shell/util.py
M tests/statestore/test_statestore.py
M tests/stress/test_update_stress.py
M tests/util/hdfs_util.py
M tests/util/parse_util.py
M tests/util/shell_util.py
34 files changed, 431 insertions(+), 273 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/19/23319/12
--
To view, visit http://gerrit.cloudera.org:8080/23319
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I401a93b6cc7bcd17f41d24e7a310e0c882a550d4
Gerrit-Change-Number: 23319
Gerrit-PatchSet: 12
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
[Impala-ASF-CR] IMPALA-14333: Run impala-py.test using Python3
Posted by "Riza Suminto (Code Review)" <ge...@cloudera.org>.
Hello Laszlo Gaal, Jason Fehr, Csaba Ringhofer, Michael Smith, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/23319
to look at the new patch set (#15).
Change subject: IMPALA-14333: Run impala-py.test using Python3
......................................................................
IMPALA-14333: Run impala-py.test using Python3
Running exhaustive tests with env var IMPALA_USE_PYTHON3_TESTS=true
reveals some tests that require adjustment. This patch made such
adjustment, which mostly revolves around encoding differences and string
vs bytes type in Python3. This patch also switch the default to run
pytest with Python3 by setting IMPALA_USE_PYTHON3_TESTS=true. The
following are the details:
Change hash() function in conftest.py to crc32() to produce
deterministic hash. Hash randomization is enabled by default since
Python 3.3 (see
https://docs.python.org/3/reference/datamodel.html#object.__hash__).
This cause test sharding (like --shard_tests=1/2) produce inconsistent
set of tests per shard. Always restart minicluster during custom cluster
tests if --shard_tests argument is set, because test order may change
and affect test correctness, depending on whether running on fresh
minicluster or not.
Moved one test case from delimited-latin-text.test to
test_delimited_text.py for easier binary comparison.
Add bytes_to_str() as a utility function to decode bytes in Python3.
This is often needed when inspecting the return value of
subprocess.check_output() as a string.
Implement DataTypeMetaclass.__lt__ to substitute
DataTypeMetaclass.__cmp__ that is ignored in Python3 (see
https://peps.python.org/pep-0207/).
Fix WEB_CERT_ERR difference in test_ipv6.py.
Fix trivial integer parsing in test_restart_services.py.
Fix various encoding issues in test_saml2_sso.py,
test_shell_commandline.py, and test_shell_interactive.py.
Change timeout in Impala.for_each_impalad() from sys.maxsize to 2^31-1.
Switch to binary comparison in test_iceberg.py where needed.
Specify text mode when calling tempfile.NamedTemporaryFile().
Simplify create_impala_shell_executable_dimension to skip testing
python2 impala-shell when IMPALA_USE_PYTHON3_TESTS=true. The reason is
that several UTF-8 related tests in test_shell_commandline.py break in
Python3 pytest + Python2 impala-shell combo.
Removed unused vector argument and fixed some trivial flake8 issues.
Several test logic require modification due to intermittent issue in
Python3 pytest. These include:
Add _run_query_with_client() in test_ranger.py to allow reusing a single
Impala client for running several queries. Ensure clients are closed
when the test is done. Mark several tests in test_ranger.py with
SkipIfFS.hive because they run queries through beeline + HiveServer2,
but Ozone and S3 build environment does not start HiveServer2 by
default.
Increase the sleep period from 0.1 to 0.5 seconds per iteration in
test_statestore.py and mark TestStatestore to execute serially. This is
because TServer appears to shut down more slowly when run concurrently
with other tests. Handle the deprecation of Thread.setDaemon() as well.
Always force_restart=True each test method in TestLoggingCore,
TestShellInteractiveReconnect, and TestQueryRetries to prevent them from
reusing minicluster from previous test method. Some of these tests
destruct minicluster (kill impalad) and will produce minidump if metrics
verifier for next tests fail to detect healthy minicluster state.
Testing:
Pass exhaustive tests with IMPALA_USE_PYTHON3_TESTS=true.
Change-Id: I401a93b6cc7bcd17f41d24e7a310e0c882a550d4
---
M bin/impala-config.sh
M shell/impala_shell/impala_shell.py
M testdata/workloads/functional-query/queries/QueryTest/delimited-latin-text.test
M tests/authorization/test_ranger.py
M tests/common/custom_cluster_test_suite.py
M tests/common/impala_test_suite.py
M tests/common/skip.py
M tests/comparison/cluster.py
M tests/comparison/db_types.py
M tests/conftest.py
M tests/custom_cluster/test_ipv6.py
M tests/custom_cluster/test_logging.py
M tests/custom_cluster/test_query_retries.py
M tests/custom_cluster/test_restart_services.py
M tests/custom_cluster/test_s3a_access.py
M tests/custom_cluster/test_saml2_sso.py
M tests/custom_cluster/test_scratch_disk.py
M tests/custom_cluster/test_shell_interactive_reconnect.py
M tests/query_test/test_delimited_text.py
M tests/query_test/test_iceberg.py
M tests/query_test/test_queries.py
M tests/shell/test_shell_commandline.py
M tests/shell/test_shell_interactive.py
M tests/shell/util.py
M tests/statestore/test_statestore.py
M tests/stress/test_update_stress.py
M tests/util/hdfs_util.py
M tests/util/parse_util.py
M tests/util/shell_util.py
29 files changed, 404 insertions(+), 262 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/19/23319/15
--
To view, visit http://gerrit.cloudera.org:8080/23319
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I401a93b6cc7bcd17f41d24e7a310e0c882a550d4
Gerrit-Change-Number: 23319
Gerrit-PatchSet: 15
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
[Impala-ASF-CR] IMPALA-14333: Run impala-py.test using Python3
Posted by "Jason Fehr (Code Review)" <ge...@cloudera.org>.
Jason Fehr has posted comments on this change. ( http://gerrit.cloudera.org:8080/23319 )
Change subject: IMPALA-14333: Run impala-py.test using Python3
......................................................................
Patch Set 17:
(1 comment)
> Patch Set 17:
>
> Thank you Michael and Jason for your review!
> I'll rebase this patch later today once IMPALA-13731 and other flaky tests fixes merged.
>
> I'll also need to remind everyone that once this patch merged, Impala will stop testing impala-shell + Python2 combo. If later on we realize it still needed somehow, we can revert the default value of IMPALA_USE_PYTHON3_TESTS back to false in impala-config.sh.
Since Impala shell still supports Python 2, will the Impala Shell release process still test on Python 2?
http://gerrit.cloudera.org:8080/#/c/23319/5//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/23319/5//COMMIT_MSG@45
PS5, Line 45:
> Ack. I also misunderstood this. Python 2 test dimension will still get ge
Clarification: Python 2 test dimension will not be created if IMPALA_USE_PYTHON3_TESTS=true.
--
To view, visit http://gerrit.cloudera.org:8080/23319
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I401a93b6cc7bcd17f41d24e7a310e0c882a550d4
Gerrit-Change-Number: 23319
Gerrit-PatchSet: 17
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Tue, 02 Sep 2025 19:28:03 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-14333: Run impala-py.test using Python3
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23319 )
Change subject: IMPALA-14333: Run impala-py.test using Python3
......................................................................
Patch Set 18:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/12412/ DRY_RUN=false
--
To view, visit http://gerrit.cloudera.org:8080/23319
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I401a93b6cc7bcd17f41d24e7a310e0c882a550d4
Gerrit-Change-Number: 23319
Gerrit-PatchSet: 18
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Wed, 03 Sep 2025 04:39:32 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14333: [part 1] Make all pytest compatible with Python3
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23319 )
Change subject: IMPALA-14333: [part 1] Make all pytest compatible with Python3
......................................................................
Patch Set 2:
Build Failed
https://jenkins.impala.io/job/gerrit-code-review-checks/20217/ : Initial code review checks failed. See linked job for details on the failure.
--
To view, visit http://gerrit.cloudera.org:8080/23319
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I401a93b6cc7bcd17f41d24e7a310e0c882a550d4
Gerrit-Change-Number: 23319
Gerrit-PatchSet: 2
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Wed, 20 Aug 2025 22:16:25 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14333: Run impala-py.test using Python3
Posted by "Riza Suminto (Code Review)" <ge...@cloudera.org>.
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/23319 )
Change subject: IMPALA-14333: Run impala-py.test using Python3
......................................................................
Patch Set 12:
Patch set 12 is a rebase and attempt to pass gerrit-auto-critic-test.
--
To view, visit http://gerrit.cloudera.org:8080/23319
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I401a93b6cc7bcd17f41d24e7a310e0c882a550d4
Gerrit-Change-Number: 23319
Gerrit-PatchSet: 12
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Tue, 26 Aug 2025 01:45:57 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14333: Run impala-py.test using Python3
Posted by "Riza Suminto (Code Review)" <ge...@cloudera.org>.
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/23319 )
Change subject: IMPALA-14333: Run impala-py.test using Python3
......................................................................
Patch Set 9:
https://docs.python.org/3/reference/datamodel.html#object.__hash__
Changed in version 3.3: Hash randomization is enabled by default.
This cause test sharding (like --shard_tests=1/2) produce inconsistent set of tests per shard.
Need to change hash() function in conftest.py to something deterministic. crc32() is a possible candidate. To validate the consistency/inconsistency, run following command from tests/ dir
CLUSTER_TEST_FILES="" ./run-custom-cluster-tests.sh --shard_tests=1/2 --collect-only | less
--
To view, visit http://gerrit.cloudera.org:8080/23319
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I401a93b6cc7bcd17f41d24e7a310e0c882a550d4
Gerrit-Change-Number: 23319
Gerrit-PatchSet: 9
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Mon, 25 Aug 2025 03:29:35 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14333: Run impala-py.test using Python3
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23319 )
Change subject: IMPALA-14333: Run impala-py.test using Python3
......................................................................
Patch Set 12:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/12375/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/23319
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I401a93b6cc7bcd17f41d24e7a310e0c882a550d4
Gerrit-Change-Number: 23319
Gerrit-PatchSet: 12
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Tue, 26 Aug 2025 03:21:19 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14333: Run impala-py.test using Python3
Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/23319 )
Change subject: IMPALA-14333: Run impala-py.test using Python3
......................................................................
Patch Set 13:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/23319/13//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/23319/13//COMMIT_MSG@57
PS13, Line 57: Several test logic require modification due to vector permutation
: difference (see IMPALA-13125) and other intermittent issue in Python3
: pytest. These include:
AFAIK this should not be relevant after https://gerrit.cloudera.org/#/c/23342/, as it changes the dimensions to an ordered dict, so Python 2 and 3 should create the same vectors.
http://gerrit.cloudera.org:8080/#/c/23319/13//COMMIT_MSG@69
PS13, Line 69: Skip testing HBase at test_except and test_partitioned_top_n because
: HBase does not return rows where alltypesagg.tinyint_col is null.
Also included in https://gerrit.cloudera.org/#/c/23342/
--
To view, visit http://gerrit.cloudera.org:8080/23319
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I401a93b6cc7bcd17f41d24e7a310e0c882a550d4
Gerrit-Change-Number: 23319
Gerrit-PatchSet: 13
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Thu, 28 Aug 2025 16:06:50 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-14333: Run impala-py.test using Python3
Posted by "Riza Suminto (Code Review)" <ge...@cloudera.org>.
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/23319 )
Change subject: IMPALA-14333: Run impala-py.test using Python3
......................................................................
Patch Set 16:
> Patch Set 16: Verified-1
>
> Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/12397/
This pass everything except flaky test_async_load described in IMPALA-13731.
--
To view, visit http://gerrit.cloudera.org:8080/23319
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I401a93b6cc7bcd17f41d24e7a310e0c882a550d4
Gerrit-Change-Number: 23319
Gerrit-PatchSet: 16
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Fri, 29 Aug 2025 14:34:52 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14333: [part 1] Make all pytest compatible with Python3
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23319 )
Change subject: IMPALA-14333: [part 1] Make all pytest compatible with Python3
......................................................................
Patch Set 3:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/12354/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/23319
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I401a93b6cc7bcd17f41d24e7a310e0c882a550d4
Gerrit-Change-Number: 23319
Gerrit-PatchSet: 3
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Fri, 22 Aug 2025 03:33:53 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14333: [part 1] Make all pytest compatible with Python3
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23319 )
Change subject: IMPALA-14333: [part 1] Make all pytest compatible with Python3
......................................................................
Patch Set 3: Verified-1
Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/12354/
--
To view, visit http://gerrit.cloudera.org:8080/23319
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I401a93b6cc7bcd17f41d24e7a310e0c882a550d4
Gerrit-Change-Number: 23319
Gerrit-PatchSet: 3
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Fri, 22 Aug 2025 08:44:59 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14333: Run impala-py.test using Python3
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23319 )
Change subject: IMPALA-14333: Run impala-py.test using Python3
......................................................................
Patch Set 9:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/12371/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/23319
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I401a93b6cc7bcd17f41d24e7a310e0c882a550d4
Gerrit-Change-Number: 23319
Gerrit-PatchSet: 9
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Sun, 24 Aug 2025 23:02:17 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14333: Run impala-py.test using Python3
Posted by "Riza Suminto (Code Review)" <ge...@cloudera.org>.
Hello Laszlo Gaal, Jason Fehr, Csaba Ringhofer, Michael Smith, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/23319
to look at the new patch set (#16).
Change subject: IMPALA-14333: Run impala-py.test using Python3
......................................................................
IMPALA-14333: Run impala-py.test using Python3
Running exhaustive tests with env var IMPALA_USE_PYTHON3_TESTS=true
reveals some tests that require adjustment. This patch made such
adjustment, which mostly revolves around encoding differences and string
vs bytes type in Python3. This patch also switch the default to run
pytest with Python3 by setting IMPALA_USE_PYTHON3_TESTS=true. The
following are the details:
Change hash() function in conftest.py to crc32() to produce
deterministic hash. Hash randomization is enabled by default since
Python 3.3 (see
https://docs.python.org/3/reference/datamodel.html#object.__hash__).
This cause test sharding (like --shard_tests=1/2) produce inconsistent
set of tests per shard. Always restart minicluster during custom cluster
tests if --shard_tests argument is set, because test order may change
and affect test correctness, depending on whether running on fresh
minicluster or not.
Moved one test case from delimited-latin-text.test to
test_delimited_text.py for easier binary comparison.
Add bytes_to_str() as a utility function to decode bytes in Python3.
This is often needed when inspecting the return value of
subprocess.check_output() as a string.
Implement DataTypeMetaclass.__lt__ to substitute
DataTypeMetaclass.__cmp__ that is ignored in Python3 (see
https://peps.python.org/pep-0207/).
Fix WEB_CERT_ERR difference in test_ipv6.py.
Fix trivial integer parsing in test_restart_services.py.
Fix various encoding issues in test_saml2_sso.py,
test_shell_commandline.py, and test_shell_interactive.py.
Change timeout in Impala.for_each_impalad() from sys.maxsize to 2^31-1.
Switch to binary comparison in test_iceberg.py where needed.
Specify text mode when calling tempfile.NamedTemporaryFile().
Simplify create_impala_shell_executable_dimension to skip testing dev
and python2 impala-shell when IMPALA_USE_PYTHON3_TESTS=true. The reason
is that several UTF-8 related tests in test_shell_commandline.py break
in Python3 pytest + Python2 impala-shell combo. This skipping already
happen automatically in build OS without system Python2 available like
RHEL9 (IMPALA_SYSTEM_PYTHON2 env var is empty).
Removed unused vector argument and fixed some trivial flake8 issues.
Several test logic require modification due to intermittent issue in
Python3 pytest. These include:
Add _run_query_with_client() in test_ranger.py to allow reusing a single
Impala client for running several queries. Ensure clients are closed
when the test is done. Mark several tests in test_ranger.py with
SkipIfFS.hive because they run queries through beeline + HiveServer2,
but Ozone and S3 build environment does not start HiveServer2 by
default.
Increase the sleep period from 0.1 to 0.5 seconds per iteration in
test_statestore.py and mark TestStatestore to execute serially. This is
because TServer appears to shut down more slowly when run concurrently
with other tests. Handle the deprecation of Thread.setDaemon() as well.
Always force_restart=True each test method in TestLoggingCore,
TestShellInteractiveReconnect, and TestQueryRetries to prevent them from
reusing minicluster from previous test method. Some of these tests
destruct minicluster (kill impalad) and will produce minidump if metrics
verifier for next tests fail to detect healthy minicluster state.
Testing:
Pass exhaustive tests with IMPALA_USE_PYTHON3_TESTS=true.
Change-Id: I401a93b6cc7bcd17f41d24e7a310e0c882a550d4
---
M bin/impala-config.sh
M shell/impala_shell/impala_shell.py
M testdata/workloads/functional-query/queries/QueryTest/delimited-latin-text.test
M tests/authorization/test_ranger.py
M tests/common/custom_cluster_test_suite.py
M tests/common/impala_test_suite.py
M tests/common/skip.py
M tests/comparison/cluster.py
M tests/comparison/db_types.py
M tests/conftest.py
M tests/custom_cluster/test_ipv6.py
M tests/custom_cluster/test_logging.py
M tests/custom_cluster/test_query_retries.py
M tests/custom_cluster/test_restart_services.py
M tests/custom_cluster/test_s3a_access.py
M tests/custom_cluster/test_saml2_sso.py
M tests/custom_cluster/test_scratch_disk.py
M tests/custom_cluster/test_shell_interactive_reconnect.py
M tests/query_test/test_delimited_text.py
M tests/query_test/test_iceberg.py
M tests/query_test/test_queries.py
M tests/shell/test_shell_commandline.py
M tests/shell/test_shell_interactive.py
M tests/shell/util.py
M tests/statestore/test_statestore.py
M tests/stress/test_update_stress.py
M tests/util/hdfs_util.py
M tests/util/parse_util.py
M tests/util/shell_util.py
29 files changed, 402 insertions(+), 263 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/19/23319/16
--
To view, visit http://gerrit.cloudera.org:8080/23319
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I401a93b6cc7bcd17f41d24e7a310e0c882a550d4
Gerrit-Change-Number: 23319
Gerrit-PatchSet: 16
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
[Impala-ASF-CR] IMPALA-14333: [part 1] Make all pytest compatible with Python3
Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/23319 )
Change subject: IMPALA-14333: [part 1] Make all pytest compatible with Python3
......................................................................
Patch Set 3:
(5 comments)
Great work, it would be great to leave Python 2 behind in our infra!
http://gerrit.cloudera.org:8080/#/c/23319/1//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/23319/1//COMMIT_MSG@7
PS1, Line 7: part 1]
This patch tries to keep tests working with both Python2 and Python3 - once the default will be Python3, it will be very easy to break Python2 unless we run precommit tests with Python 2.
What is you plan for removing Python 2 test support completely? There could be also a follow up Jira to clean up Python 2 related code, e.g search for sys.version_info.major
It also doesn't seem an extreme step to me to remove Python 2 test support in this patch - not having to be Python 2 compatible would simplify some code and if exhaustive tests pass, then I don't see a good reason to use Python 2 tests again.
http://gerrit.cloudera.org:8080/#/c/23319/1//COMMIT_MSG@17
PS1, Line 17: Set timezone='Europe/Budapest' in test_sort and test_top_n to get
: consistent result for reading functional_parquet.complextypes_structs
: and functional_orc_def.complextypes_structs.
:
: Skip testing HBase at test_except and test_partitioned_top_n because
: HBase does not return return rows where alltypesagg.tinyint_col is null.
:
: Add _run_query_with_client() in test_ranger.py to allow reusing single
: Impala client for running several queries. Make sure to close the
: clients when test is done. Mark several tests in test_ranger.py with
: SkipIfFS.hive because they run query through beeline + HiveServer2, but
: Ozone and S3 build environment does not start HiveServer2 by default.
:
I don't get these changes - these seem to change test logic, and I don't know how these are related to Python 2 vs 3
http://gerrit.cloudera.org:8080/#/c/23319/1//COMMIT_MSG@49
PS1, Line 49: test_decimal_queries.py for json file format because table date_tbl and
: decimal_tiny is contrained only for json/none/none (functional_json).
:
Same as for line 17 - how is this related to Python 2 vs 3?
http://gerrit.cloudera.org:8080/#/c/23319/3//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/23319/3//COMMIT_MSG@63
PS3, Line 63: python2 impala-shell
Would it be hard to keep python2 impala-shell tests alive? While the Python used for testing only matter for Impala developer, python2 support for impala-shell may have some value for someone. I wouldn't put much effort into it though.
http://gerrit.cloudera.org:8080/#/c/23319/3/tests/custom_cluster/test_saml2_sso.py
File tests/custom_cluster/test_saml2_sso.py:
http://gerrit.cloudera.org:8080/#/c/23319/3/tests/custom_cluster/test_saml2_sso.py@221
PS3, Line 221: body = "SAMLResponse=%s&RelayState=%s" % (authn_resp, relay_state)
Where did the base64 encoding go?
Btw fyi there is a review in parallel that heavily modifies this file: https://gerrit.cloudera.org/#/c/23237/
--
To view, visit http://gerrit.cloudera.org:8080/23319
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I401a93b6cc7bcd17f41d24e7a310e0c882a550d4
Gerrit-Change-Number: 23319
Gerrit-PatchSet: 3
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Fri, 22 Aug 2025 13:04:45 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-14333: Run impala-py.test using Python3
Posted by "Riza Suminto (Code Review)" <ge...@cloudera.org>.
Hello Jason Fehr, Csaba Ringhofer, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/23319
to look at the new patch set (#4).
Change subject: IMPALA-14333: Run impala-py.test using Python3
......................................................................
IMPALA-14333: Run impala-py.test using Python3
Running exhaustive tests with env var IMPALA_USE_PYTHON3_TESTS=true
reveals some tests that require adjustment. This patch made such
adjustment, which mostly revolves around encoding differences and string
vs bytes type in Python3. This patch also switch the default to run
pytest with Python3 by setting IMPALA_USE_PYTHON3_TESTS=true. The
following are the details:
Moved one test case from delimited-latin-text.test to
test_delimited_text.py for easier binary comparison.
Add bytes_to_str() as a utility function to decode bytes in Python3.
This is often needed when inspecting the return value of
subprocess.check_output() as a string.
Implement DataTypeMetaclass.__lt__ to substitute
DataTypeMetaclass.__cmp__ that is ignored in Python3 (see
https://peps.python.org/pep-0207/).
Fix WEB_CERT_ERR difference in test_ipv6.py.
Fix trivial integer parsing in test_restart_services.py.
Fix various encoding issues in test_saml2_sso.py,
test_shell_commandline.py, and test_shell_interactive.py.
Switch parallel loops in test_stress_infra.py to a simple loop that
fail-proof in Python3.
Switch to binary comparison in test_iceberg.py where needed.
Specify text mode when calling tempfile.NamedTemporaryFile().
Simplify create_impala_shell_executable_dimension to skip testing
python2 impala-shell when IMPALA_USE_PYTHON3_TESTS=true. The reason is
that several UTF-8 related tests in test_shell_commandline.py break in
Python3 pytest + Python2 impala-shell combo.
Removed unused vector argument and fixed some trivial flake8 issues.
Several test logic require modification due to vector permutation
difference (see IMPALA-13125) and other intermittent issue in Python3
pytest. These include:
Add _run_query_with_client() in test_ranger.py to allow reusing a single
Impala client for running several queries. Ensure clients are closed
when the test is done. Mark several tests in test_ranger.py with
SkipIfFS.hive because they run queries through beeline + HiveServer2,
but Ozone and S3 build environment does not start HiveServer2 by
default.
Set timezone='Europe/Budapest' in test_sort and test_top_n to get
consistent result for reading functional_parquet.complextypes_structs
and functional_orc_def.complextypes_structs.
Skip testing HBase at test_except and test_partitioned_top_n because
HBase does not return rows where alltypesagg.tinyint_col is null.
Reduce test dimension in test_date_queries.py and
test_decimal_queries.py for json file format because table date_tbl and
decimal_tiny are contrained only for json/none/none (functional_json).
Increase the sleep period from 0.1 to 0.5 seconds per iteration in
test_statestore.py and mark TestStatestore to execute serially. This is
because TServer appears to shut down more slowly when run concurrently
with other tests. Handle the deprecation of Thread.setDaemon() as well.
Testing:
Pass exhaustive tests with IMPALA_USE_PYTHON3_TESTS=true.
Change-Id: I401a93b6cc7bcd17f41d24e7a310e0c882a550d4
---
M bin/impala-config.sh
M shell/impala_shell/impala_shell.py
M testdata/workloads/functional-query/queries/QueryTest/delimited-latin-text.test
M testdata/workloads/functional-query/queries/QueryTest/inline-view.test
M testdata/workloads/functional-query/queries/QueryTest/sort-complex.test
M testdata/workloads/functional-query/queries/QueryTest/top-n-complex.test
M tests/authorization/test_ranger.py
M tests/common/impala_test_suite.py
M tests/common/skip.py
M tests/comparison/cluster.py
M tests/comparison/db_types.py
M tests/custom_cluster/test_ipv6.py
M tests/custom_cluster/test_restart_services.py
M tests/custom_cluster/test_s3a_access.py
M tests/custom_cluster/test_saml2_sso.py
M tests/custom_cluster/test_scratch_disk.py
M tests/infra/test_stress_infra.py
M tests/query_test/test_date_queries.py
M tests/query_test/test_decimal_queries.py
M tests/query_test/test_delimited_text.py
M tests/query_test/test_iceberg.py
M tests/query_test/test_queries.py
M tests/shell/test_shell_commandline.py
M tests/shell/test_shell_interactive.py
M tests/shell/util.py
M tests/statestore/test_statestore.py
M tests/stress/test_update_stress.py
M tests/util/hdfs_util.py
M tests/util/parse_util.py
M tests/util/shell_util.py
30 files changed, 379 insertions(+), 243 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/19/23319/4
--
To view, visit http://gerrit.cloudera.org:8080/23319
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I401a93b6cc7bcd17f41d24e7a310e0c882a550d4
Gerrit-Change-Number: 23319
Gerrit-PatchSet: 4
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
[Impala-ASF-CR] IMPALA-14333: Run impala-py.test using Python3
Posted by "Riza Suminto (Code Review)" <ge...@cloudera.org>.
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/23319 )
Change subject: IMPALA-14333: Run impala-py.test using Python3
......................................................................
Patch Set 4:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/23319/4/tests/custom_cluster/test_saml2_sso.py
File tests/custom_cluster/test_saml2_sso.py:
http://gerrit.cloudera.org:8080/#/c/23319/4/tests/custom_cluster/test_saml2_sso.py@220
PS4, Line 220: authn_resp = self._generate_authn_response(request_id, attributes_xml)
: body = "SAMLResponse=%s&RelayState=%s" % (authn_resp, relay_state)
: opener = build_opener(NoRedirection)
: req = Request(TestClientSaml.SP_CALLBACK_URL, encode_if_needed(body))
This way works in both python2 and python3:
authn_resp = self._generate_authn_response(request_id, attributes_xml)
encoded_authn_resp = base64.urlsafe_b64encode(authn_resp.encode('utf-8'))
body = b"SAMLResponse=" + encoded_authn_resp + b"&RelayState=" + relay_state.encode('utf-8')
opener = build_opener(NoRedirection)
req = Request(TestClientSaml.SP_CALLBACK_URL, encode_if_needed(body))
--
To view, visit http://gerrit.cloudera.org:8080/23319
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I401a93b6cc7bcd17f41d24e7a310e0c882a550d4
Gerrit-Change-Number: 23319
Gerrit-PatchSet: 4
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Fri, 22 Aug 2025 16:18:13 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-14333: Run impala-py.test using Python3
Posted by "Riza Suminto (Code Review)" <ge...@cloudera.org>.
Hello Jason Fehr, Csaba Ringhofer, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/23319
to look at the new patch set (#7).
Change subject: IMPALA-14333: Run impala-py.test using Python3
......................................................................
IMPALA-14333: Run impala-py.test using Python3
Running exhaustive tests with env var IMPALA_USE_PYTHON3_TESTS=true
reveals some tests that require adjustment. This patch made such
adjustment, which mostly revolves around encoding differences and string
vs bytes type in Python3. This patch also switch the default to run
pytest with Python3 by setting IMPALA_USE_PYTHON3_TESTS=true. The
following are the details:
Moved one test case from delimited-latin-text.test to
test_delimited_text.py for easier binary comparison.
Add bytes_to_str() as a utility function to decode bytes in Python3.
This is often needed when inspecting the return value of
subprocess.check_output() as a string.
Implement DataTypeMetaclass.__lt__ to substitute
DataTypeMetaclass.__cmp__ that is ignored in Python3 (see
https://peps.python.org/pep-0207/).
Fix WEB_CERT_ERR difference in test_ipv6.py.
Fix trivial integer parsing in test_restart_services.py.
Fix various encoding issues in test_saml2_sso.py,
test_shell_commandline.py, and test_shell_interactive.py.
Change timeout in Impala.for_each_impalad() from sys.maxsize to 2^31-1.
Switch to binary comparison in test_iceberg.py where needed.
Specify text mode when calling tempfile.NamedTemporaryFile().
Simplify create_impala_shell_executable_dimension to skip testing
python2 impala-shell when IMPALA_USE_PYTHON3_TESTS=true. The reason is
that several UTF-8 related tests in test_shell_commandline.py break in
Python3 pytest + Python2 impala-shell combo.
Removed unused vector argument and fixed some trivial flake8 issues.
Several test logic require modification due to vector permutation
difference (see IMPALA-13125) and other intermittent issue in Python3
pytest. These include:
Add _run_query_with_client() in test_ranger.py to allow reusing a single
Impala client for running several queries. Ensure clients are closed
when the test is done. Mark several tests in test_ranger.py with
SkipIfFS.hive because they run queries through beeline + HiveServer2,
but Ozone and S3 build environment does not start HiveServer2 by
default.
Set timezone='Europe/Budapest' in test_sort and test_top_n to get
consistent result for reading functional_parquet.complextypes_structs
and functional_orc_def.complextypes_structs.
Skip testing HBase at test_except and test_partitioned_top_n because
HBase does not return rows where alltypesagg.tinyint_col is null.
Reduce test dimension in test_date_queries.py and
test_decimal_queries.py for json file format because table date_tbl and
decimal_tiny are contrained only for json/none/none (functional_json).
Increase the sleep period from 0.1 to 0.5 seconds per iteration in
test_statestore.py and mark TestStatestore to execute serially. This is
because TServer appears to shut down more slowly when run concurrently
with other tests. Handle the deprecation of Thread.setDaemon() as well.
Testing:
Pass exhaustive tests with IMPALA_USE_PYTHON3_TESTS=true.
Change-Id: I401a93b6cc7bcd17f41d24e7a310e0c882a550d4
---
M bin/impala-config.sh
M shell/impala_shell/impala_shell.py
M testdata/workloads/functional-query/queries/QueryTest/delimited-latin-text.test
M testdata/workloads/functional-query/queries/QueryTest/inline-view.test
M testdata/workloads/functional-query/queries/QueryTest/sort-complex.test
M testdata/workloads/functional-query/queries/QueryTest/top-n-complex.test
M tests/authorization/test_ranger.py
M tests/common/custom_cluster_test_suite.py
M tests/common/impala_test_suite.py
M tests/common/skip.py
M tests/comparison/cluster.py
M tests/comparison/db_types.py
M tests/custom_cluster/test_ipv6.py
M tests/custom_cluster/test_restart_services.py
M tests/custom_cluster/test_s3a_access.py
M tests/custom_cluster/test_saml2_sso.py
M tests/custom_cluster/test_scratch_disk.py
M tests/query_test/test_date_queries.py
M tests/query_test/test_decimal_queries.py
M tests/query_test/test_delimited_text.py
M tests/query_test/test_iceberg.py
M tests/query_test/test_queries.py
M tests/shell/test_shell_commandline.py
M tests/shell/test_shell_interactive.py
M tests/shell/util.py
M tests/statestore/test_statestore.py
M tests/stress/test_update_stress.py
M tests/util/hdfs_util.py
M tests/util/parse_util.py
M tests/util/shell_util.py
30 files changed, 381 insertions(+), 245 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/19/23319/7
--
To view, visit http://gerrit.cloudera.org:8080/23319
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I401a93b6cc7bcd17f41d24e7a310e0c882a550d4
Gerrit-Change-Number: 23319
Gerrit-PatchSet: 7
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
[Impala-ASF-CR] IMPALA-14333: Run impala-py.test using Python3
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23319 )
Change subject: IMPALA-14333: Run impala-py.test using Python3
......................................................................
Patch Set 7:
Build Failed
https://jenkins.impala.io/job/gerrit-code-review-checks/20246/ : Initial code review checks failed. See linked job for details on the failure.
--
To view, visit http://gerrit.cloudera.org:8080/23319
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I401a93b6cc7bcd17f41d24e7a310e0c882a550d4
Gerrit-Change-Number: 23319
Gerrit-PatchSet: 7
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Sun, 24 Aug 2025 03:30:06 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14333: Run impala-py.test using Python3
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23319 )
Change subject: IMPALA-14333: Run impala-py.test using Python3
......................................................................
Patch Set 7:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/12368/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/23319
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I401a93b6cc7bcd17f41d24e7a310e0c882a550d4
Gerrit-Change-Number: 23319
Gerrit-PatchSet: 7
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Sun, 24 Aug 2025 03:32:25 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14333: [part 1] Make all pytest compatible with Python3
Posted by "Riza Suminto (Code Review)" <ge...@cloudera.org>.
Hello Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/23319
to look at the new patch set (#2).
Change subject: IMPALA-14333: [part 1] Make all pytest compatible with Python3
......................................................................
IMPALA-14333: [part 1] Make all pytest compatible with Python3
Running exhaustive tests with env var IMPALA_USE_PYTHON3_TESTS=true
reveals some tests that require adjustment. This patch made such
adjustment, which mostly revolves around encoding difference and string
vs bytes type in Python3. Followings are the details:
Moved one test case from delimited-latin-text.test to
test_delimited_text.py for easier binary comparison.
Recreate structs.parq (content inside
functional_parquet.complextypes_structs) from
functional_orc_def.complextypes_structs so that querying both of them
will return the same result simply by setting option
convert_legacy_hive_parquet_utc_timestamps=true. Setting
timezone='Europe/Budapest' is not required anymore.
Skip testing HBase at test_except and test_partitioned_top_n because
HBase does not return return rows where alltypesagg.tinyint_col is null.
Add _run_query_with_client() in test_ranger.py to allow reusing single
Impala client for running several queries. Also make sure to close the
clients when test is done.
Added ImpalaTestSuite._convert_bytes() as utility function to decode
bytes in Python3. This often needed when inspecting return value of
subprocess.check_output() as string.
Implement DataTypeMetaclass.__lt__ to substitute
DataTypeMetaclass.__cmp__ that is ignored in Python3 (see
https://peps.python.org/pep-0207/).
Fix WEB_CERT_ERR difference in test_ipv6.py.
Fix trivial integer parsing in test_restart_services.py.
Fix various encoding issues in test_saml2_sso.py,
test_shell_commandline.py, and test_shell_interactive.py.
Switch parallel loops in test_stress_infra.py to simple loop that
fail-proof in Python3.
Reduce test dimension in test_date_queries.py and
test_decimal_queries.py for json file format because table date_tbl and
decimal_tiny is contrained only for json/none/none (functional_json).
Switch to binary comparson in test_iceberg.py where needed.
Removed unused vector argument and fix some trivial flake8 issues.
Simplify create_impala_shell_executable_dimension to skip testing
python2 impala-shell when IMPALA_USE_PYTHON3_TESTS=true.
Switching IMPALA_USE_PYTHON3_TESTS to true by default will be addressed
in next patch.
Testing:
Pass exhaustive tests with IMPALA_USE_PYTHON3_TESTS=true.
Change-Id: I401a93b6cc7bcd17f41d24e7a310e0c882a550d4
---
M shell/impala_shell/impala_shell.py
M testdata/ComplexTypesTbl/structs.parq
M testdata/workloads/functional-query/queries/QueryTest/delimited-latin-text.test
M testdata/workloads/functional-query/queries/QueryTest/inline-view.test
M testdata/workloads/functional-query/queries/QueryTest/sort-complex.test
M testdata/workloads/functional-query/queries/QueryTest/top-n-complex.test
M tests/authorization/test_ranger.py
M tests/common/impala_test_suite.py
M tests/comparison/cluster.py
M tests/comparison/db_types.py
M tests/custom_cluster/test_ipv6.py
M tests/custom_cluster/test_restart_services.py
M tests/custom_cluster/test_saml2_sso.py
M tests/custom_cluster/test_scratch_disk.py
M tests/infra/test_stress_infra.py
M tests/query_test/test_date_queries.py
M tests/query_test/test_decimal_queries.py
M tests/query_test/test_delimited_text.py
M tests/query_test/test_iceberg.py
M tests/query_test/test_nested_types.py
M tests/query_test/test_queries.py
M tests/shell/test_shell_commandline.py
M tests/shell/test_shell_interactive.py
M tests/shell/util.py
M tests/statestore/test_statestore.py
M tests/stress/test_update_stress.py
M tests/util/shell_util.py
27 files changed, 333 insertions(+), 233 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/19/23319/2
--
To view, visit http://gerrit.cloudera.org:8080/23319
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I401a93b6cc7bcd17f41d24e7a310e0c882a550d4
Gerrit-Change-Number: 23319
Gerrit-PatchSet: 2
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
[Impala-ASF-CR] IMPALA-14333: Run impala-py.test using Python3
Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/23319 )
Change subject: IMPALA-14333: Run impala-py.test using Python3
......................................................................
Patch Set 10: Code-Review+1
(1 comment)
This is a lot to put in one commit. The commit message has too many different changes it needs to cover, and it's tricky to connect them to specific changes in the commit. I'd prefer it were split up a bit. However the individual changes here look ok.
http://gerrit.cloudera.org:8080/#/c/23319/10/tests/shell/test_shell_commandline.py
File tests/shell/test_shell_commandline.py:
http://gerrit.cloudera.org:8080/#/c/23319/10/tests/shell/test_shell_commandline.py@1108
PS10, Line 1108: f.write("select \n")
nit: should be 2-space indentation
--
To view, visit http://gerrit.cloudera.org:8080/23319
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I401a93b6cc7bcd17f41d24e7a310e0c882a550d4
Gerrit-Change-Number: 23319
Gerrit-PatchSet: 10
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Mon, 25 Aug 2025 23:19:28 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-14333: Run impala-py.test using Python3
Posted by "Riza Suminto (Code Review)" <ge...@cloudera.org>.
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/23319 )
Change subject: IMPALA-14333: Run impala-py.test using Python3
......................................................................
Patch Set 11:
(1 comment)
Apologies for the long commit messages.
Splitting the patch would be best. However, if we want to flip-the-switch right away, I think it is best to land all the changes in single commit. Thus, there will be no in-between patches slip that require additional adjustment later on.
If unrelated patch merged ahead of this and broke Python3 pytest, it will be easier for me to simply rebase and fix them in this single patch.
http://gerrit.cloudera.org:8080/#/c/23319/10/tests/shell/test_shell_commandline.py
File tests/shell/test_shell_commandline.py:
http://gerrit.cloudera.org:8080/#/c/23319/10/tests/shell/test_shell_commandline.py@1108
PS10, Line 1108: f.write("select \n")
> nit: should be 2-space indentation
Done
--
To view, visit http://gerrit.cloudera.org:8080/23319
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I401a93b6cc7bcd17f41d24e7a310e0c882a550d4
Gerrit-Change-Number: 23319
Gerrit-PatchSet: 11
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Mon, 25 Aug 2025 23:45:01 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-14333: [part 1] Make all pytest compatible with Python3
Posted by "Riza Suminto (Code Review)" <ge...@cloudera.org>.
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/23319 )
Change subject: IMPALA-14333: [part 1] Make all pytest compatible with Python3
......................................................................
Patch Set 3:
(5 comments)
http://gerrit.cloudera.org:8080/#/c/23319/1//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/23319/1//COMMIT_MSG@7
PS1, Line 7: part 1]
> This patch tries to keep tests working with both Python2 and Python3 - once
My intention with splitting the adjustment vs the default switch is to make sure everything is in working order before the default switch, and make it easy to revert just the default switch part if things goes wrong that we have not seen.
But I agree with your thought, and I can squash the second part to this first part.
I'll file a new JIRA under https://issues.apache.org/jira/browse/IMPALA-8508 to cleanup Python2 remnants. I do wish to consult with Joe first before doing that.
http://gerrit.cloudera.org:8080/#/c/23319/1//COMMIT_MSG@17
PS1, Line 17: Set timezone='Europe/Budapest' in test_sort and test_top_n to get
: consistent result for reading functional_parquet.complextypes_structs
: and functional_orc_def.complextypes_structs.
:
: Skip testing HBase at test_except and test_partitioned_top_n because
: HBase does not return return rows where alltypesagg.tinyint_col is null.
:
: Add _run_query_with_client() in test_ranger.py to allow reusing single
: Impala client for running several queries. Make sure to close the
: clients when test is done. Mark several tests in test_ranger.py with
: SkipIfFS.hive because they run query through beeline + HiveServer2, but
: Ozone and S3 build environment does not start HiveServer2 by default.
:
> I don't get these changes - these seem to change test logic, and I don't kn
Ranger adjustment is done because I found it failing with a bunch of Impyla stacktrace of "TSocket read 0 bytes". I thought it might be related to the nature of this test where it always open new client to execute impala testcase queries using _run_query_as_user(). Changing it to reuse client resolved the issue in Python3, but admittedly, I have not dig deeper into it.
For others, there is a different permutation behavior between Python2 vs Python3 that cause our test run silently not exercise some test vector (https://issues.apache.org/jira/browse/IMPALA-13125). Python3 reveal test breakage in some of these unexercised vector. Hence these adjustment.
http://gerrit.cloudera.org:8080/#/c/23319/1//COMMIT_MSG@49
PS1, Line 49: test_decimal_queries.py for json file format because table date_tbl and
: decimal_tiny is contrained only for json/none/none (functional_json).
:
> Same as for line 17 - how is this related to Python 2 vs 3?
Explained in https://gerrit.cloudera.org/c/23319/1//COMMIT_MSG#29
http://gerrit.cloudera.org:8080/#/c/23319/3//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/23319/3//COMMIT_MSG@63
PS3, Line 63: python2 impala-shell
> Would it be hard to keep python2 impala-shell tests alive? While the Python
When I start this, Python3 pytest + Python2 shell test combo shows breakage in test_shell_commandline.py and test_shell_interactive.py. I'm just being pragmatic here in reducing test combination and moving the transition forward. I hope we can revisit this combo later after switching to Python3 by default.
http://gerrit.cloudera.org:8080/#/c/23319/3/tests/custom_cluster/test_saml2_sso.py
File tests/custom_cluster/test_saml2_sso.py:
http://gerrit.cloudera.org:8080/#/c/23319/3/tests/custom_cluster/test_saml2_sso.py@221
PS3, Line 221: body = "SAMLResponse=%s&RelayState=%s" % (authn_resp, relay_state)
> Where did the base64 encoding go?
I'm not sure what happen here, but removing the base64 encoding make the test work in Python2 and Python3.
Someone with more experience around SAML2 should chime in on this.
--
To view, visit http://gerrit.cloudera.org:8080/23319
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I401a93b6cc7bcd17f41d24e7a310e0c882a550d4
Gerrit-Change-Number: 23319
Gerrit-PatchSet: 3
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Fri, 22 Aug 2025 14:15:22 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-14333: Run impala-py.test using Python3
Posted by "Riza Suminto (Code Review)" <ge...@cloudera.org>.
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/23319 )
Change subject: IMPALA-14333: Run impala-py.test using Python3
......................................................................
Patch Set 6:
(3 comments)
http://gerrit.cloudera.org:8080/#/c/23319/5//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/23319/5//COMMIT_MSG@42
PS5, Line 42: python2 impala-shell when IMPALA_USE_PYTHON3_TESTS=true. The reason is
> Nit: extra blank line
Done
http://gerrit.cloudera.org:8080/#/c/23319/5//COMMIT_MSG@45
PS5, Line 45:
> Are we going to need separate builds to test impala-shell with the tests ru
Personally, I rather move away from python2 totally. There are only few test that still cover Python2 shell today (calls create_impala_shell_executable_dimension()):
tests/custom_cluster/test_client_ssl.py
tests/custom_cluster/test_ipv6.py
tests/query_test/test_date_queries.py
tests/shell/test_shell_commandline.py
tests/shell/test_shell_commandline_kerberos_auth.py
tests/shell/test_shell_interactive.py
But I suspect only tests/shell/test_shell_commandline.py has breakage due to UTF-8 tests.
http://gerrit.cloudera.org:8080/#/c/23319/5//COMMIT_MSG@59
PS5, Line 59:
> Why did these tests work when run on Python 2 but break when run on Python
I suspect some custom cluster test unknowingly start HiveServer2 manually (see CustomClusterTestSuite._start_hive_service()). Not sure why things change after Pyton3 switch.
On the other hand, I see in run-all.sh that only TARGET_FILESYSTEM=hdfs should have HiveServer2 by default.
https://github.com/apache/impala/blob/23a14a2/testdata/bin/run-all.sh#L72
--
To view, visit http://gerrit.cloudera.org:8080/23319
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I401a93b6cc7bcd17f41d24e7a310e0c882a550d4
Gerrit-Change-Number: 23319
Gerrit-PatchSet: 6
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Fri, 22 Aug 2025 21:39:18 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-14333: Run impala-py.test using Python3
Posted by "Riza Suminto (Code Review)" <ge...@cloudera.org>.
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/23319 )
Change subject: IMPALA-14333: Run impala-py.test using Python3
......................................................................
Patch Set 14:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/23319/13//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/23319/13//COMMIT_MSG@57
PS13, Line 57: Several test logic require modification due to intermittent issue in
: Python3 pytest. These include:
:
> AFAIK this should not be relevant after https://gerrit.cloudera.org/#/c/233
Done
http://gerrit.cloudera.org:8080/#/c/23319/13//COMMIT_MSG@69
PS13, Line 69: test_statestore.py and mark TestStatestore to execute serially. This is
: because TServer appears to shut down more slowly when run concurr
> Also included in https://gerrit.cloudera.org/#/c/23342/
Done
--
To view, visit http://gerrit.cloudera.org:8080/23319
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I401a93b6cc7bcd17f41d24e7a310e0c882a550d4
Gerrit-Change-Number: 23319
Gerrit-PatchSet: 14
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Thu, 28 Aug 2025 16:12:48 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-14333: Run impala-py.test using Python3
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23319 )
Change subject: IMPALA-14333: Run impala-py.test using Python3
......................................................................
Patch Set 16: Verified-1
Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/12397/
--
To view, visit http://gerrit.cloudera.org:8080/23319
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I401a93b6cc7bcd17f41d24e7a310e0c882a550d4
Gerrit-Change-Number: 23319
Gerrit-PatchSet: 16
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Fri, 29 Aug 2025 09:35:39 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14333: Run impala-py.test using Python3
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23319 )
Change subject: IMPALA-14333: Run impala-py.test using Python3
......................................................................
Patch Set 6:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/12362/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/23319
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I401a93b6cc7bcd17f41d24e7a310e0c882a550d4
Gerrit-Change-Number: 23319
Gerrit-PatchSet: 6
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Fri, 22 Aug 2025 22:53:46 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14333: Run impala-py.test using Python3
Posted by "Riza Suminto (Code Review)" <ge...@cloudera.org>.
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/23319 )
Change subject: IMPALA-14333: Run impala-py.test using Python3
......................................................................
Patch Set 5:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/23319/4/tests/custom_cluster/test_saml2_sso.py
File tests/custom_cluster/test_saml2_sso.py:
http://gerrit.cloudera.org:8080/#/c/23319/4/tests/custom_cluster/test_saml2_sso.py@220
PS4, Line 220: authn_resp = self._generate_authn_response(request_id, attributes_xml)
: encoded_authn_resp = base64.urlsafe_b64encode(authn_resp.encode('utf-8'))
: body = (b"SAMLResponse=" + encoded_authn_resp + b"&RelayState="
: + encode_if_needed(relay_state))
> This way works in both python2 and python3:
Done
--
To view, visit http://gerrit.cloudera.org:8080/23319
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I401a93b6cc7bcd17f41d24e7a310e0c882a550d4
Gerrit-Change-Number: 23319
Gerrit-PatchSet: 5
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Fri, 22 Aug 2025 16:50:55 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-14333: Run impala-py.test using Python3
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23319 )
Change subject: IMPALA-14333: Run impala-py.test using Python3
......................................................................
Patch Set 14:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/12394/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/23319
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I401a93b6cc7bcd17f41d24e7a310e0c882a550d4
Gerrit-Change-Number: 23319
Gerrit-PatchSet: 14
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Thu, 28 Aug 2025 16:28:26 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14333: Run impala-py.test using Python3
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23319 )
Change subject: IMPALA-14333: Run impala-py.test using Python3
......................................................................
Patch Set 17:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/12399/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/23319
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I401a93b6cc7bcd17f41d24e7a310e0c882a550d4
Gerrit-Change-Number: 23319
Gerrit-PatchSet: 17
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Fri, 29 Aug 2025 20:01:47 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14333: Run impala-py.test using Python3
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23319 )
Change subject: IMPALA-14333: Run impala-py.test using Python3
......................................................................
Patch Set 15:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/20306/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/23319
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I401a93b6cc7bcd17f41d24e7a310e0c882a550d4
Gerrit-Change-Number: 23319
Gerrit-PatchSet: 15
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Thu, 28 Aug 2025 22:58:26 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14333: [part 1] Make all pytest compatible with Python3
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23319 )
Change subject: IMPALA-14333: [part 1] Make all pytest compatible with Python3
......................................................................
Patch Set 1:
Build Failed
https://jenkins.impala.io/job/gerrit-code-review-checks/20216/ : Initial code review checks failed. See linked job for details on the failure.
--
To view, visit http://gerrit.cloudera.org:8080/23319
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I401a93b6cc7bcd17f41d24e7a310e0c882a550d4
Gerrit-Change-Number: 23319
Gerrit-PatchSet: 1
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Wed, 20 Aug 2025 21:42:13 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14333: Run impala-py.test using Python3
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23319 )
Change subject: IMPALA-14333: Run impala-py.test using Python3
......................................................................
Patch Set 9: Verified-1
Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/12371/
--
To view, visit http://gerrit.cloudera.org:8080/23319
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I401a93b6cc7bcd17f41d24e7a310e0c882a550d4
Gerrit-Change-Number: 23319
Gerrit-PatchSet: 9
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Mon, 25 Aug 2025 04:21:57 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14333: Run impala-py.test using Python3
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23319 )
Change subject: IMPALA-14333: Run impala-py.test using Python3
......................................................................
Patch Set 8: Verified-1
Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/12369/
--
To view, visit http://gerrit.cloudera.org:8080/23319
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I401a93b6cc7bcd17f41d24e7a310e0c882a550d4
Gerrit-Change-Number: 23319
Gerrit-PatchSet: 8
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Sun, 24 Aug 2025 20:11:21 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14333: Run impala-py.test using Python3
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23319 )
Change subject: IMPALA-14333: Run impala-py.test using Python3
......................................................................
Patch Set 10: Verified-1
Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/12373/
--
To view, visit http://gerrit.cloudera.org:8080/23319
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I401a93b6cc7bcd17f41d24e7a310e0c882a550d4
Gerrit-Change-Number: 23319
Gerrit-PatchSet: 10
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Mon, 25 Aug 2025 19:23:33 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14333: Run impala-py.test using Python3
Posted by "Riza Suminto (Code Review)" <ge...@cloudera.org>.
Hello Laszlo Gaal, Jason Fehr, Csaba Ringhofer, Michael Smith, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/23319
to look at the new patch set (#11).
Change subject: IMPALA-14333: Run impala-py.test using Python3
......................................................................
IMPALA-14333: Run impala-py.test using Python3
Running exhaustive tests with env var IMPALA_USE_PYTHON3_TESTS=true
reveals some tests that require adjustment. This patch made such
adjustment, which mostly revolves around encoding differences and string
vs bytes type in Python3. This patch also switch the default to run
pytest with Python3 by setting IMPALA_USE_PYTHON3_TESTS=true. The
following are the details:
Change hash() function in conftest.py to crc32() to produce
deterministic hash. Hash randomization is enabled by default since
Python 3.3 (see
https://docs.python.org/3/reference/datamodel.html#object.__hash__).
This cause test sharding (like --shard_tests=1/2) produce inconsistent
set of tests per shard. Always restart minicluster during custom cluster
tests if --shard_tests argument is set, because test order may change
and affect test correctness, depending on whether running on fresh
minicluster or not.
Moved one test case from delimited-latin-text.test to
test_delimited_text.py for easier binary comparison.
Add bytes_to_str() as a utility function to decode bytes in Python3.
This is often needed when inspecting the return value of
subprocess.check_output() as a string.
Implement DataTypeMetaclass.__lt__ to substitute
DataTypeMetaclass.__cmp__ that is ignored in Python3 (see
https://peps.python.org/pep-0207/).
Fix WEB_CERT_ERR difference in test_ipv6.py.
Fix trivial integer parsing in test_restart_services.py.
Fix various encoding issues in test_saml2_sso.py,
test_shell_commandline.py, and test_shell_interactive.py.
Change timeout in Impala.for_each_impalad() from sys.maxsize to 2^31-1.
Switch to binary comparison in test_iceberg.py where needed.
Specify text mode when calling tempfile.NamedTemporaryFile().
Simplify create_impala_shell_executable_dimension to skip testing
python2 impala-shell when IMPALA_USE_PYTHON3_TESTS=true. The reason is
that several UTF-8 related tests in test_shell_commandline.py break in
Python3 pytest + Python2 impala-shell combo.
Removed unused vector argument and fixed some trivial flake8 issues.
Several test logic require modification due to vector permutation
difference (see IMPALA-13125) and other intermittent issue in Python3
pytest. These include:
Add _run_query_with_client() in test_ranger.py to allow reusing a single
Impala client for running several queries. Ensure clients are closed
when the test is done. Mark several tests in test_ranger.py with
SkipIfFS.hive because they run queries through beeline + HiveServer2,
but Ozone and S3 build environment does not start HiveServer2 by
default.
Set timezone='Europe/Budapest' in test_sort and test_top_n to get
consistent result for reading functional_parquet.complextypes_structs
and functional_orc_def.complextypes_structs.
Skip testing HBase at test_except and test_partitioned_top_n because
HBase does not return rows where alltypesagg.tinyint_col is null.
Reduce test dimension in test_date_queries.py and
test_decimal_queries.py for json file format because table date_tbl and
decimal_tiny are contrained only for json/none/none (functional_json).
Increase the sleep period from 0.1 to 0.5 seconds per iteration in
test_statestore.py and mark TestStatestore to execute serially. This is
because TServer appears to shut down more slowly when run concurrently
with other tests. Handle the deprecation of Thread.setDaemon() as well.
Always force_restart=True each test method in TestLoggingCore,
TestShellInteractiveReconnect, and TestQueryRetries to prevent them from
reusing minicluster from previous test method. Some of these tests
destruct minicluster (kill impalad) and will produce minidump if metrics
verifier for next tests fail to detect healthy minicluster state.
Testing:
Pass exhaustive tests with IMPALA_USE_PYTHON3_TESTS=true.
Change-Id: I401a93b6cc7bcd17f41d24e7a310e0c882a550d4
---
M bin/impala-config.sh
M shell/impala_shell/impala_shell.py
M testdata/workloads/functional-query/queries/QueryTest/delimited-latin-text.test
M testdata/workloads/functional-query/queries/QueryTest/inline-view.test
M testdata/workloads/functional-query/queries/QueryTest/sort-complex.test
M testdata/workloads/functional-query/queries/QueryTest/top-n-complex.test
M tests/authorization/test_ranger.py
M tests/common/custom_cluster_test_suite.py
M tests/common/impala_test_suite.py
M tests/common/skip.py
M tests/comparison/cluster.py
M tests/comparison/db_types.py
M tests/conftest.py
M tests/custom_cluster/test_ipv6.py
M tests/custom_cluster/test_logging.py
M tests/custom_cluster/test_query_retries.py
M tests/custom_cluster/test_restart_services.py
M tests/custom_cluster/test_s3a_access.py
M tests/custom_cluster/test_saml2_sso.py
M tests/custom_cluster/test_scratch_disk.py
M tests/custom_cluster/test_shell_interactive_reconnect.py
M tests/query_test/test_date_queries.py
M tests/query_test/test_decimal_queries.py
M tests/query_test/test_delimited_text.py
M tests/query_test/test_iceberg.py
M tests/query_test/test_queries.py
M tests/shell/test_shell_commandline.py
M tests/shell/test_shell_interactive.py
M tests/shell/util.py
M tests/statestore/test_statestore.py
M tests/stress/test_update_stress.py
M tests/util/hdfs_util.py
M tests/util/parse_util.py
M tests/util/shell_util.py
34 files changed, 428 insertions(+), 270 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/19/23319/11
--
To view, visit http://gerrit.cloudera.org:8080/23319
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I401a93b6cc7bcd17f41d24e7a310e0c882a550d4
Gerrit-Change-Number: 23319
Gerrit-PatchSet: 11
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
[Impala-ASF-CR] IMPALA-14333: Run impala-py.test using Python3
Posted by "Jason Fehr (Code Review)" <ge...@cloudera.org>.
Jason Fehr has posted comments on this change. ( http://gerrit.cloudera.org:8080/23319 )
Change subject: IMPALA-14333: Run impala-py.test using Python3
......................................................................
Patch Set 5:
(8 comments)
http://gerrit.cloudera.org:8080/#/c/23319/5//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/23319/5//COMMIT_MSG@42
PS5, Line 42:
Nit: extra blank line
http://gerrit.cloudera.org:8080/#/c/23319/5//COMMIT_MSG@45
PS5, Line 45: Python3 pytest + Python2 impala-shell combo.
Are we going to need separate builds to test impala-shell with the tests running on Python 2? Otherwise, it seems to me testing of impala-shell on Python 2 will have use cases going untested.
http://gerrit.cloudera.org:8080/#/c/23319/5//COMMIT_MSG@59
PS5, Line 59: default.
Why did these tests work when run on Python 2 but break when run on Python 3?
http://gerrit.cloudera.org:8080/#/c/23319/1/tests/authorization/test_ranger.py
File tests/authorization/test_ranger.py:
http://gerrit.cloudera.org:8080/#/c/23319/1/tests/authorization/test_ranger.py@816
PS1, Line 816:
> I make it as function so that it will minimize the change when it replaces
Sounds good to me.
http://gerrit.cloudera.org:8080/#/c/23319/1/tests/authorization/test_ranger.py@892
PS1, Line 892: admin_client = self.create_impala_client(user=ADMIN)
> We can use with-as statement, but that will shift the spaces and cause larg
Sounds good to me.
http://gerrit.cloudera.org:8080/#/c/23319/1/tests/custom_cluster/test_saml2_sso.py
File tests/custom_cluster/test_saml2_sso.py:
http://gerrit.cloudera.org:8080/#/c/23319/1/tests/custom_cluster/test_saml2_sso.py@111
PS1, Line 111: SSO_ARGS_WITH_GROUP_FILTER = (
> Done
Done
http://gerrit.cloudera.org:8080/#/c/23319/1/tests/custom_cluster/test_saml2_sso.py@240
PS1, Line 240: msg_id=str(uuid.uuid4()),
> Leftover from my debugging. Removed.
Done
http://gerrit.cloudera.org:8080/#/c/23319/2/tests/shell/test_shell_interactive.py
File tests/shell/test_shell_interactive.py:
http://gerrit.cloudera.org:8080/#/c/23319/2/tests/shell/test_shell_interactive.py@192
PS2, Line 192:
> Removed.
Done
--
To view, visit http://gerrit.cloudera.org:8080/23319
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I401a93b6cc7bcd17f41d24e7a310e0c882a550d4
Gerrit-Change-Number: 23319
Gerrit-PatchSet: 5
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Fri, 22 Aug 2025 19:45:04 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-14333: Run impala-py.test using Python3
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23319 )
Change subject: IMPALA-14333: Run impala-py.test using Python3
......................................................................
Patch Set 8:
Build Failed
https://jenkins.impala.io/job/gerrit-code-review-checks/20247/ : Initial code review checks failed. See linked job for details on the failure.
--
To view, visit http://gerrit.cloudera.org:8080/23319
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I401a93b6cc7bcd17f41d24e7a310e0c882a550d4
Gerrit-Change-Number: 23319
Gerrit-PatchSet: 8
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Sun, 24 Aug 2025 14:54:22 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14333: Run impala-py.test using Python3
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23319 )
Change subject: IMPALA-14333: Run impala-py.test using Python3
......................................................................
Patch Set 8:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/12369/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/23319
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I401a93b6cc7bcd17f41d24e7a310e0c882a550d4
Gerrit-Change-Number: 23319
Gerrit-PatchSet: 8
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Sun, 24 Aug 2025 14:28:32 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14333: Run impala-py.test using Python3
Posted by "Riza Suminto (Code Review)" <ge...@cloudera.org>.
Hello Jason Fehr, Csaba Ringhofer, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/23319
to look at the new patch set (#9).
Change subject: IMPALA-14333: Run impala-py.test using Python3
......................................................................
IMPALA-14333: Run impala-py.test using Python3
Running exhaustive tests with env var IMPALA_USE_PYTHON3_TESTS=true
reveals some tests that require adjustment. This patch made such
adjustment, which mostly revolves around encoding differences and string
vs bytes type in Python3. This patch also switch the default to run
pytest with Python3 by setting IMPALA_USE_PYTHON3_TESTS=true. The
following are the details:
Moved one test case from delimited-latin-text.test to
test_delimited_text.py for easier binary comparison.
Add bytes_to_str() as a utility function to decode bytes in Python3.
This is often needed when inspecting the return value of
subprocess.check_output() as a string.
Implement DataTypeMetaclass.__lt__ to substitute
DataTypeMetaclass.__cmp__ that is ignored in Python3 (see
https://peps.python.org/pep-0207/).
Fix WEB_CERT_ERR difference in test_ipv6.py.
Fix trivial integer parsing in test_restart_services.py.
Fix various encoding issues in test_saml2_sso.py,
test_shell_commandline.py, and test_shell_interactive.py.
Change timeout in Impala.for_each_impalad() from sys.maxsize to 2^31-1.
Switch to binary comparison in test_iceberg.py where needed.
Specify text mode when calling tempfile.NamedTemporaryFile().
Simplify create_impala_shell_executable_dimension to skip testing
python2 impala-shell when IMPALA_USE_PYTHON3_TESTS=true. The reason is
that several UTF-8 related tests in test_shell_commandline.py break in
Python3 pytest + Python2 impala-shell combo.
Removed unused vector argument and fixed some trivial flake8 issues.
Several test logic require modification due to vector permutation
difference (see IMPALA-13125) and other intermittent issue in Python3
pytest. These include:
Add _run_query_with_client() in test_ranger.py to allow reusing a single
Impala client for running several queries. Ensure clients are closed
when the test is done. Mark several tests in test_ranger.py with
SkipIfFS.hive because they run queries through beeline + HiveServer2,
but Ozone and S3 build environment does not start HiveServer2 by
default.
Set timezone='Europe/Budapest' in test_sort and test_top_n to get
consistent result for reading functional_parquet.complextypes_structs
and functional_orc_def.complextypes_structs.
Skip testing HBase at test_except and test_partitioned_top_n because
HBase does not return rows where alltypesagg.tinyint_col is null.
Reduce test dimension in test_date_queries.py and
test_decimal_queries.py for json file format because table date_tbl and
decimal_tiny are contrained only for json/none/none (functional_json).
Increase the sleep period from 0.1 to 0.5 seconds per iteration in
test_statestore.py and mark TestStatestore to execute serially. This is
because TServer appears to shut down more slowly when run concurrently
with other tests. Handle the deprecation of Thread.setDaemon() as well.
Always force_restart=True each test method in TestLoggingCore,
TestShellInteractiveReconnect, and TestQueryRetries to prevent them from
reusing minicluster from previous test method. Some of these tests
destruct minicluster (kill impalad) and will produce minidump if metrics
verifier for next tests fail to detect healthy minicluster state.
Testing:
Pass exhaustive tests with IMPALA_USE_PYTHON3_TESTS=true.
Change-Id: I401a93b6cc7bcd17f41d24e7a310e0c882a550d4
---
M bin/impala-config.sh
M shell/impala_shell/impala_shell.py
M testdata/workloads/functional-query/queries/QueryTest/delimited-latin-text.test
M testdata/workloads/functional-query/queries/QueryTest/inline-view.test
M testdata/workloads/functional-query/queries/QueryTest/sort-complex.test
M testdata/workloads/functional-query/queries/QueryTest/top-n-complex.test
M tests/authorization/test_ranger.py
M tests/common/custom_cluster_test_suite.py
M tests/common/impala_test_suite.py
M tests/common/skip.py
M tests/comparison/cluster.py
M tests/comparison/db_types.py
M tests/custom_cluster/test_ipv6.py
M tests/custom_cluster/test_logging.py
M tests/custom_cluster/test_query_retries.py
M tests/custom_cluster/test_restart_services.py
M tests/custom_cluster/test_s3a_access.py
M tests/custom_cluster/test_saml2_sso.py
M tests/custom_cluster/test_scratch_disk.py
M tests/custom_cluster/test_shell_interactive_reconnect.py
M tests/query_test/test_date_queries.py
M tests/query_test/test_decimal_queries.py
M tests/query_test/test_delimited_text.py
M tests/query_test/test_iceberg.py
M tests/query_test/test_queries.py
M tests/shell/test_shell_commandline.py
M tests/shell/test_shell_interactive.py
M tests/shell/util.py
M tests/statestore/test_statestore.py
M tests/stress/test_update_stress.py
M tests/util/hdfs_util.py
M tests/util/parse_util.py
M tests/util/shell_util.py
33 files changed, 422 insertions(+), 266 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/19/23319/9
--
To view, visit http://gerrit.cloudera.org:8080/23319
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I401a93b6cc7bcd17f41d24e7a310e0c882a550d4
Gerrit-Change-Number: 23319
Gerrit-PatchSet: 9
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
[Impala-ASF-CR] IMPALA-14333: Run impala-py.test using Python3
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/23319 )
Change subject: IMPALA-14333: Run impala-py.test using Python3
......................................................................
IMPALA-14333: Run impala-py.test using Python3
Running exhaustive tests with env var IMPALA_USE_PYTHON3_TESTS=true
reveals some tests that require adjustment. This patch made such
adjustment, which mostly revolves around encoding differences and string
vs bytes type in Python3. This patch also switch the default to run
pytest with Python3 by setting IMPALA_USE_PYTHON3_TESTS=true. The
following are the details:
Change hash() function in conftest.py to crc32() to produce
deterministic hash. Hash randomization is enabled by default since
Python 3.3 (see
https://docs.python.org/3/reference/datamodel.html#object.__hash__).
This cause test sharding (like --shard_tests=1/2) produce inconsistent
set of tests per shard. Always restart minicluster during custom cluster
tests if --shard_tests argument is set, because test order may change
and affect test correctness, depending on whether running on fresh
minicluster or not.
Moved one test case from delimited-latin-text.test to
test_delimited_text.py for easier binary comparison.
Add bytes_to_str() as a utility function to decode bytes in Python3.
This is often needed when inspecting the return value of
subprocess.check_output() as a string.
Implement DataTypeMetaclass.__lt__ to substitute
DataTypeMetaclass.__cmp__ that is ignored in Python3 (see
https://peps.python.org/pep-0207/).
Fix WEB_CERT_ERR difference in test_ipv6.py.
Fix trivial integer parsing in test_restart_services.py.
Fix various encoding issues in test_saml2_sso.py,
test_shell_commandline.py, and test_shell_interactive.py.
Change timeout in Impala.for_each_impalad() from sys.maxsize to 2^31-1.
Switch to binary comparison in test_iceberg.py where needed.
Specify text mode when calling tempfile.NamedTemporaryFile().
Simplify create_impala_shell_executable_dimension to skip testing dev
and python2 impala-shell when IMPALA_USE_PYTHON3_TESTS=true. The reason
is that several UTF-8 related tests in test_shell_commandline.py break
in Python3 pytest + Python2 impala-shell combo. This skipping already
happen automatically in build OS without system Python2 available like
RHEL9 (IMPALA_SYSTEM_PYTHON2 env var is empty).
Removed unused vector argument and fixed some trivial flake8 issues.
Several test logic require modification due to intermittent issue in
Python3 pytest. These include:
Add _run_query_with_client() in test_ranger.py to allow reusing a single
Impala client for running several queries. Ensure clients are closed
when the test is done. Mark several tests in test_ranger.py with
SkipIfFS.hive because they run queries through beeline + HiveServer2,
but Ozone and S3 build environment does not start HiveServer2 by
default.
Increase the sleep period from 0.1 to 0.5 seconds per iteration in
test_statestore.py and mark TestStatestore to execute serially. This is
because TServer appears to shut down more slowly when run concurrently
with other tests. Handle the deprecation of Thread.setDaemon() as well.
Always force_restart=True each test method in TestLoggingCore,
TestShellInteractiveReconnect, and TestQueryRetries to prevent them from
reusing minicluster from previous test method. Some of these tests
destruct minicluster (kill impalad) and will produce minidump if metrics
verifier for next tests fail to detect healthy minicluster state.
Testing:
Pass exhaustive tests with IMPALA_USE_PYTHON3_TESTS=true.
Change-Id: I401a93b6cc7bcd17f41d24e7a310e0c882a550d4
Reviewed-on: http://gerrit.cloudera.org:8080/23319
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
M bin/impala-config.sh
M shell/impala_shell/impala_shell.py
M testdata/workloads/functional-query/queries/QueryTest/delimited-latin-text.test
M tests/authorization/test_ranger.py
M tests/common/custom_cluster_test_suite.py
M tests/common/impala_test_suite.py
M tests/common/skip.py
M tests/comparison/cluster.py
M tests/comparison/db_types.py
M tests/conftest.py
M tests/custom_cluster/test_ipv6.py
M tests/custom_cluster/test_logging.py
M tests/custom_cluster/test_query_retries.py
M tests/custom_cluster/test_restart_services.py
M tests/custom_cluster/test_s3a_access.py
M tests/custom_cluster/test_saml2_sso.py
M tests/custom_cluster/test_scratch_disk.py
M tests/custom_cluster/test_shell_interactive_reconnect.py
M tests/query_test/test_delimited_text.py
M tests/query_test/test_iceberg.py
M tests/query_test/test_queries.py
M tests/shell/test_shell_commandline.py
M tests/shell/test_shell_interactive.py
M tests/shell/util.py
M tests/statestore/test_statestore.py
M tests/stress/test_update_stress.py
M tests/util/hdfs_util.py
M tests/util/parse_util.py
M tests/util/shell_util.py
29 files changed, 402 insertions(+), 263 deletions(-)
Approvals:
Impala Public Jenkins: Looks good to me, approved; Verified
--
To view, visit http://gerrit.cloudera.org:8080/23319
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I401a93b6cc7bcd17f41d24e7a310e0c882a550d4
Gerrit-Change-Number: 23319
Gerrit-PatchSet: 19
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
[Impala-ASF-CR] IMPALA-14333: Run impala-py.test using Python3
Posted by "Riza Suminto (Code Review)" <ge...@cloudera.org>.
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/23319 )
Change subject: IMPALA-14333: Run impala-py.test using Python3
......................................................................
Patch Set 17:
> Patch Set 17:
>
> Since Impala shell still supports Python 2, will the Impala Shell release process still test on Python 2?
I think our build script will still build impala-shell for python 2, but only test against one built for python 3. We already have this behavior when building Impala using newer OS that does not support python 2 anymore. I mentioned in commit message:
> This skipping already happen automatically in build OS without system Python2 available like RHEL9 (IMPALA_SYSTEM_PYTHON2 env var is empty).
--
To view, visit http://gerrit.cloudera.org:8080/23319
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I401a93b6cc7bcd17f41d24e7a310e0c882a550d4
Gerrit-Change-Number: 23319
Gerrit-PatchSet: 17
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Tue, 02 Sep 2025 19:35:59 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14333: Run impala-py.test using Python3
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23319 )
Change subject: IMPALA-14333: Run impala-py.test using Python3
......................................................................
Patch Set 10:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/12373/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/23319
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I401a93b6cc7bcd17f41d24e7a310e0c882a550d4
Gerrit-Change-Number: 23319
Gerrit-PatchSet: 10
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Mon, 25 Aug 2025 13:52:54 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14333: Run impala-py.test using Python3
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23319 )
Change subject: IMPALA-14333: Run impala-py.test using Python3
......................................................................
Patch Set 12:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/20265/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/23319
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I401a93b6cc7bcd17f41d24e7a310e0c882a550d4
Gerrit-Change-Number: 23319
Gerrit-PatchSet: 12
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Tue, 26 Aug 2025 02:12:13 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14333: Run impala-py.test using Python3
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23319 )
Change subject: IMPALA-14333: Run impala-py.test using Python3
......................................................................
Patch Set 15:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/12396/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/23319
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I401a93b6cc7bcd17f41d24e7a310e0c882a550d4
Gerrit-Change-Number: 23319
Gerrit-PatchSet: 15
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Thu, 28 Aug 2025 22:32:37 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14333: Run impala-py.test using Python3
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23319 )
Change subject: IMPALA-14333: Run impala-py.test using Python3
......................................................................
Patch Set 13:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/20298/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/23319
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I401a93b6cc7bcd17f41d24e7a310e0c882a550d4
Gerrit-Change-Number: 23319
Gerrit-PatchSet: 13
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Thu, 28 Aug 2025 16:26:19 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14333: Run impala-py.test using Python3
Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/23319 )
Change subject: IMPALA-14333: Run impala-py.test using Python3
......................................................................
Patch Set 17: Code-Review+1
--
To view, visit http://gerrit.cloudera.org:8080/23319
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I401a93b6cc7bcd17f41d24e7a310e0c882a550d4
Gerrit-Change-Number: 23319
Gerrit-PatchSet: 17
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Tue, 02 Sep 2025 15:43:36 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14333: Run impala-py.test using Python3
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23319 )
Change subject: IMPALA-14333: Run impala-py.test using Python3
......................................................................
Patch Set 18: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/23319
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I401a93b6cc7bcd17f41d24e7a310e0c882a550d4
Gerrit-Change-Number: 23319
Gerrit-PatchSet: 18
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Wed, 03 Sep 2025 04:39:31 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14333: Run impala-py.test using Python3
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23319 )
Change subject: IMPALA-14333: Run impala-py.test using Python3
......................................................................
Patch Set 18: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/23319
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I401a93b6cc7bcd17f41d24e7a310e0c882a550d4
Gerrit-Change-Number: 23319
Gerrit-PatchSet: 18
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Wed, 03 Sep 2025 10:01:28 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14333: Run impala-py.test using Python3
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23319 )
Change subject: IMPALA-14333: Run impala-py.test using Python3
......................................................................
Patch Set 16:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/20314/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/23319
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I401a93b6cc7bcd17f41d24e7a310e0c882a550d4
Gerrit-Change-Number: 23319
Gerrit-PatchSet: 16
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Fri, 29 Aug 2025 04:49:33 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14333: Run impala-py.test using Python3
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23319 )
Change subject: IMPALA-14333: Run impala-py.test using Python3
......................................................................
Patch Set 15: Verified-1
Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/12396/
--
To view, visit http://gerrit.cloudera.org:8080/23319
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I401a93b6cc7bcd17f41d24e7a310e0c882a550d4
Gerrit-Change-Number: 23319
Gerrit-PatchSet: 15
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Fri, 29 Aug 2025 03:52:18 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14333: [part 1] Make all pytest compatible with Python3
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23319 )
Change subject: IMPALA-14333: [part 1] Make all pytest compatible with Python3
......................................................................
Patch Set 1:
Build Failed
https://jenkins.impala.io/job/gerrit-code-review-checks/20213/ : Initial code review checks failed. See linked job for details on the failure.
--
To view, visit http://gerrit.cloudera.org:8080/23319
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I401a93b6cc7bcd17f41d24e7a310e0c882a550d4
Gerrit-Change-Number: 23319
Gerrit-PatchSet: 1
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Wed, 20 Aug 2025 20:54:31 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14333: [part 1] Make all pytest compatible with Python3
Posted by "Jason Fehr (Code Review)" <ge...@cloudera.org>.
Jason Fehr has posted comments on this change. ( http://gerrit.cloudera.org:8080/23319 )
Change subject: IMPALA-14333: [part 1] Make all pytest compatible with Python3
......................................................................
Patch Set 2:
(5 comments)
http://gerrit.cloudera.org:8080/#/c/23319/1/tests/authorization/test_ranger.py
File tests/authorization/test_ranger.py:
http://gerrit.cloudera.org:8080/#/c/23319/1/tests/authorization/test_ranger.py@816
PS1, Line 816: def _run_query_with_client(self, query, impala_client, expect_success):
To me, this function does not clarify or simplify the code. Instead, the code would be clearer if either self.execute_query_expect_success or self.execute_query_expect_failure was called directly in the tests below.
http://gerrit.cloudera.org:8080/#/c/23319/1/tests/authorization/test_ranger.py@892
PS1, Line 892: user_client = self.create_impala_client(user=OWNER_USER)
Can both of these lines use the with self.create_impala_client() as client_name syntax instead of manually closing?
Same question in the below tests that have the same pattern.
http://gerrit.cloudera.org:8080/#/c/23319/1/tests/custom_cluster/test_saml2_sso.py
File tests/custom_cluster/test_saml2_sso.py:
http://gerrit.cloudera.org:8080/#/c/23319/1/tests/custom_cluster/test_saml2_sso.py@111
PS1, Line 111: SSO_ARGS_WITH_GROUP_FILTER = (SSO_ARGS + " "
Since this line is being modified, why not switch to using .format?
SSO_ARGS_WITH_GROUP_FILTER = "{} --saml2_group_filter=group1,group2 --saml2_group_attribute_name=eduPersonAffiliation".format(SSO_ARGS)
http://gerrit.cloudera.org:8080/#/c/23319/1/tests/custom_cluster/test_saml2_sso.py@240
PS1, Line 240: return resp
I'm curious why this additional return variable is needed?
http://gerrit.cloudera.org:8080/#/c/23319/2/tests/shell/test_shell_interactive.py
File tests/shell/test_shell_interactive.py:
http://gerrit.cloudera.org:8080/#/c/23319/2/tests/shell/test_shell_interactive.py@192
PS2, Line 192: def _convert_bytes(self, bytes):
Duplicate of https://gerrit.cloudera.org/c/23319/2/tests/common/impala_test_suite.py#1825
--
To view, visit http://gerrit.cloudera.org:8080/23319
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I401a93b6cc7bcd17f41d24e7a310e0c882a550d4
Gerrit-Change-Number: 23319
Gerrit-PatchSet: 2
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Thu, 21 Aug 2025 17:29:20 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-14333: [part 1] Make all pytest compatible with Python3
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23319 )
Change subject: IMPALA-14333: [part 1] Make all pytest compatible with Python3
......................................................................
Patch Set 3:
Build Failed
https://jenkins.impala.io/job/gerrit-code-review-checks/20230/ : Initial code review checks failed. See linked job for details on the failure.
--
To view, visit http://gerrit.cloudera.org:8080/23319
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I401a93b6cc7bcd17f41d24e7a310e0c882a550d4
Gerrit-Change-Number: 23319
Gerrit-PatchSet: 3
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Fri, 22 Aug 2025 00:35:00 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14333: Run impala-py.test using Python3
Posted by "Riza Suminto (Code Review)" <ge...@cloudera.org>.
Riza Suminto has removed a vote on this change.
Change subject: IMPALA-14333: Run impala-py.test using Python3
......................................................................
Removed Verified-1 by Impala Public Jenkins <im...@cloudera.com>
--
To view, visit http://gerrit.cloudera.org:8080/23319
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: deleteVote
Gerrit-Change-Id: I401a93b6cc7bcd17f41d24e7a310e0c882a550d4
Gerrit-Change-Number: 23319
Gerrit-PatchSet: 10
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
[Impala-ASF-CR] IMPALA-14333: Run impala-py.test using Python3
Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/23319 )
Change subject: IMPALA-14333: Run impala-py.test using Python3
......................................................................
Patch Set 11: Code-Review+1
(1 comment)
http://gerrit.cloudera.org:8080/#/c/23319/10/tests/shell/test_shell_commandline.py
File tests/shell/test_shell_commandline.py:
http://gerrit.cloudera.org:8080/#/c/23319/10/tests/shell/test_shell_commandline.py@1108
PS10, Line 1108: f.write("select \n")
> nit: should be 2-space indentation
Done
--
To view, visit http://gerrit.cloudera.org:8080/23319
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I401a93b6cc7bcd17f41d24e7a310e0c882a550d4
Gerrit-Change-Number: 23319
Gerrit-PatchSet: 11
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Mon, 25 Aug 2025 23:39:50 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-14333: Run impala-py.test using Python3
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23319 )
Change subject: IMPALA-14333: Run impala-py.test using Python3
......................................................................
Patch Set 9:
Build Failed
https://jenkins.impala.io/job/gerrit-code-review-checks/20250/ : Initial code review checks failed. See linked job for details on the failure.
--
To view, visit http://gerrit.cloudera.org:8080/23319
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I401a93b6cc7bcd17f41d24e7a310e0c882a550d4
Gerrit-Change-Number: 23319
Gerrit-PatchSet: 9
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Sun, 24 Aug 2025 23:29:06 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14333: Run impala-py.test using Python3
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23319 )
Change subject: IMPALA-14333: Run impala-py.test using Python3
......................................................................
Patch Set 12: Verified-1
Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/12375/
--
To view, visit http://gerrit.cloudera.org:8080/23319
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I401a93b6cc7bcd17f41d24e7a310e0c882a550d4
Gerrit-Change-Number: 23319
Gerrit-PatchSet: 12
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Tue, 26 Aug 2025 08:46:48 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14333: Run impala-py.test using Python3
Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/23319 )
Change subject: IMPALA-14333: Run impala-py.test using Python3
......................................................................
Patch Set 12:
I have a patch that replaces the pairwise test vector generation with something that actually covers all pairs:
https://gerrit.cloudera.org/#/c/23342/
The first version bumped the number of customer cluster/EE tests from 11K to 17K in exhaustive runs and revealed several issues found by your patch. I updated it to fix some failures and reduce the number of tests a bit (it is still running)
I copied some of the fixes from here to that patch, while I did a few things differently (removing some file format + compression pairs)
I think that the test fixes for the issues revealed by the different test vector generations should be merged in a separate commit than the Py2->3 change, as it seems just an accident the Python version change led to more test vectors.
--
To view, visit http://gerrit.cloudera.org:8080/23319
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I401a93b6cc7bcd17f41d24e7a310e0c882a550d4
Gerrit-Change-Number: 23319
Gerrit-PatchSet: 12
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Tue, 26 Aug 2025 11:57:20 +0000
Gerrit-HasComments: No