You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "Tamas Mate (Jira)" <ji...@apache.org> on 2021/10/20 13:02:00 UTC
[jira] [Commented] (IMPALA-10569) Impala should determine Iceberg
data file format from Iceberg metadata
[ https://issues.apache.org/jira/browse/IMPALA-10569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17431218#comment-17431218 ]
Tamas Mate commented on IMPALA-10569:
-------------------------------------
In the current implementation the IcebergTable encapsulates an HdfsTable while the LocalIcebergTable encapsulates a LocalFsTable. While local catalog mode is active the local catalog could receive the file format from the catalog, but it could not be propagated to the internal HdfsTable and its partition which would need it later. Due to this limitation, to resolve this Jira we would possibly need some refactoring around this area.
> Impala should determine Iceberg data file format from Iceberg metadata
> ----------------------------------------------------------------------
>
> Key: IMPALA-10569
> URL: https://issues.apache.org/jira/browse/IMPALA-10569
> Project: IMPALA
> Issue Type: Bug
> Components: Frontend
> Reporter: Zoltán Borók-Nagy
> Assignee: Tamas Mate
> Priority: Major
> Labels: impala-iceberg
>
> When Impala creates an Iceberg table it sets HMS table property 'iceberg.file_format' to indicate the underlying data file format.
> However, when the table was created by Hive or Spark, we don't have this property and Impala assumes that the data file format is PARQUET. This assumption is just a wild guess, and when it's wrong Impala raises an error during query execution.
> Instead of only checking the table property, Impala could also try to determine the file format based on Iceberg metadata.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org