You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "ASF subversion and git services (Jira)" <ji...@apache.org> on 2021/04/02 07:56:00 UTC
[jira] [Commented] (IMPALA-10624)
TestIcebergTable::test_alter_iceberg_tables failed by stale file format
[ https://issues.apache.org/jira/browse/IMPALA-10624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17313687#comment-17313687 ]
ASF subversion and git services commented on IMPALA-10624:
----------------------------------------------------------
Commit b40870d55b68b1307569380b135475824aeac2e4 in impala's branch refs/heads/master from Zoltan Borok-Nagy
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=b40870d ]
IMPALA-10624: TestIcebergTable::test_alter_iceberg_tables failed by stale file format
During table creation we pushed all the table properties to the
Iceberg table, e.g. 'iceberg.file_format'. This works well with
current Iceberg version, but causes bugs with newer versions.
Newer versions of Iceberg refresh the HMS properties from the
table properties for every table update. That means if we
intially set a table property, we cannot modify it later because
Iceberg will rewrite it with the old value on every table update.
These table properties are not needed at the Iceberg table level,
so we can just skip passing them during table creation.
Testing:
* ran existing tests
* ran tests with newer version Iceberg
* checked manually that we don't set these properties at the Iceberg
table level
Change-Id: Iff8d5d1d90444aba11d47cfce522aaa45a4a74cc
Reviewed-on: http://gerrit.cloudera.org:8080/17248
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
> TestIcebergTable::test_alter_iceberg_tables failed by stale file format
> -----------------------------------------------------------------------
>
> Key: IMPALA-10624
> URL: https://issues.apache.org/jira/browse/IMPALA-10624
> Project: IMPALA
> Issue Type: Bug
> Reporter: Quanlong Huang
> Assignee: Zoltán Borók-Nagy
> Priority: Critical
>
> TestIcebergTable::test_alter_iceberg_tables fails in a downstream core build:
> {code}
> query_test/test_iceberg.py:50: in test_alter_iceberg_tables
> self.run_test_case('QueryTest/iceberg-alter', vector, use_db=unique_database)
> common/impala_test_suite.py:665: in run_test_case
> result = exec_fn(query, user=test_section.get('USER', '').strip() or None)
> common/impala_test_suite.py:603: in __exec_in_impala
> result = self.__execute_query(target_impalad_client, query, user=user)
> common/impala_test_suite.py:923: in __execute_query
> return impalad_client.execute(query, user=user)
> common/impala_connection.py:205: in execute
> return self.__beeswax_client.execute(sql_stmt, user=user)
> beeswax/impala_beeswax.py:187: in execute
> handle = self.__execute_query(query_string.strip(), user=user)
> beeswax/impala_beeswax.py:365: in __execute_query
> self.wait_for_finished(handle)
> beeswax/impala_beeswax.py:386: in wait_for_finished
> raise ImpalaBeeswaxException("Query aborted:" + error_log, None)
> E ImpalaBeeswaxException: ImpalaBeeswaxException:
> E Query aborted:Encountered parse error in tail of ORC file hdfs://localhost:20500/test-warehouse/test_alter_iceberg_tables_f665cbcc.db/iceberg_changing_fileformats/data/d94f2a969c360675-cd07d0bb00000000_1360147332_data.0.parq: Not an ORC file
> {code}
> The last several queries are
> {code}
> ALTER TABLE iceberg_changing_fileformats set TBLPROPERTIES('iceberg.file_format'='parquet');
> -- 2021-03-29 19:23:30,588 INFO MainThread: Started query 344ec76ac00e022e:1685ff5200000000
> -- executing against localhost:21000
> DESCRIBE FORMATTED iceberg_changing_fileformats;
> -- 2021-03-29 19:23:30,595 INFO MainThread: Started query d2455af1b84c7f10:1edbc8f600000000
> -- executing against localhost:21000
> INSERT INTO iceberg_changing_fileformats values (123);
> -- 2021-03-29 19:23:30,605 INFO MainThread: Started query d94f2a969c360675:cd07d0bb00000000
> -- executing against localhost:21000
> SELECT * FROM iceberg_changing_fileformats;
> -- 2021-03-29 19:23:30,824 INFO MainThread: Started query c74f132c2eda13fc:0b27b87000000000
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org