You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "ASF subversion and git services (Jira)" <ji...@apache.org> on 2023/12/17 22:40:00 UTC
[jira] [Commented] (IMPALA-12629) TestRanger fails in test_column_masking and test_row_filtering
[ https://issues.apache.org/jira/browse/IMPALA-12629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17797972#comment-17797972 ]
ASF subversion and git services commented on IMPALA-12629:
----------------------------------------------------------
Commit 1141a6a80c45bef2d1a354301e20e6ac63aa9e97 in impala's branch refs/heads/master from Zoltan Borok-Nagy
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=1141a6a80 ]
IMPALA-12629: Fix Iceberg V2 tests in non-HDFS environment
The UPDATE part 2 patch (https://gerrit.cloudera.org/#/c/20677/)
added new ranger tests that verifies that row filtering and column
masking are working on Iceberg V2 tables with delete files.
The Iceberg V2 tables we load during data loading are already written
and the position delete files have hard-coded file URIs that refer
to the data files. Unfortunately these URIs start with 'hdfs://...'.
Therefore these tables cannot be used in non-HDFS environment.
To quickly fix this, this patch introduces the 'IS_HDFS_ONLY' test
section. Tests that are annotated with it are only executed when
the underlying storage system is HDFS. This new test section can be
also used in the future to make it easier to write Impala/Hive interop
tests, because currently we can only run Hive in an HDFS environment.
So we could just annotate the tests that have HIVE_QUERY with
IS_HDFS_ONLY, and we could also add a HIVE_RESULTS section to check
the results Hive produces.
Testing:
* Ran the tests with '-s' option in HDFS environment and verified
that the IS_HDFS_ONLY tests are executed
* Ran the ranger tests in Ozone environment and they passed
Change-Id: Ifa72041e281e3d5083adaee2f8a41bb1c2c8d19f
Reviewed-on: http://gerrit.cloudera.org:8080/20803
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
> TestRanger fails in test_column_masking and test_row_filtering
> --------------------------------------------------------------
>
> Key: IMPALA-12629
> URL: https://issues.apache.org/jira/browse/IMPALA-12629
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
> Affects Versions: Impala 1.4.2
> Reporter: Andrew Sherman
> Assignee: Zoltán Borók-Nagy
> Priority: Critical
>
> Fang-Yu asking you take a look because you reviewed IMPALA-12398 which is the suspicious recent change.
> test_column_masking:
> {code}
> authorization/test_ranger.py:1510: in test_column_masking
> test_file_vars={'$UNIQUE_DB': unique_database})
> common/impala_test_suite.py:751: in run_test_case
> self.__verify_results_and_errors(vector, test_section, result, use_db)
> common/impala_test_suite.py:587: in __verify_results_and_errors
> replace_filenames_with_placeholder)
> common/test_result_verifier.py:487: in verify_raw_results
> VERIFIER_MAP[verifier](expected, actual)
> common/test_result_verifier.py:296: in verify_query_result_is_equal
> assert expected_results == actual_results
> E assert Comparing QueryTestResults (expected vs actual):
> E 1,'NULL' == 1,'NULL'
> E 3,'NULL' != 2,'NULL'
> E Number of rows returned (expected vs actual): 2 != 3
> {code}
> test_row_filtering:
> {code}
> authorization/test_ranger.py:1685: in test_row_filtering
> test_file_vars={'$UNIQUE_DB': unique_database})
> common/impala_test_suite.py:751: in run_test_case
> self.__verify_results_and_errors(vector, test_section, result, use_db)
> common/impala_test_suite.py:587: in __verify_results_and_errors
> replace_filenames_with_placeholder)
> common/test_result_verifier.py:487: in verify_raw_results
> VERIFIER_MAP[verifier](expected, actual)
> common/test_result_verifier.py:296: in verify_query_result_is_equal
> assert expected_results == actual_results
> E assert Comparing QueryTestResults (expected vs actual):
> E 1,'a' == 1,'a'
> E 3,'c' == 3,'c'
> E 5,'X' == 5,'X'
> E Number of rows returned (expected vs actual): 3 != 6
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org