You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@impala.apache.org by "Zoltán Borók-Nagy (Jira)" <ji...@apache.org> on 2022/11/28 15:00:00 UTC
[jira] [Created] (IMPALA-11752) Handle s3:// paths in Iceberg tables
Zoltán Borók-Nagy created IMPALA-11752:
------------------------------------------
Summary: Handle s3:// paths in Iceberg tables
Key: IMPALA-11752
URL: https://issues.apache.org/jira/browse/IMPALA-11752
Project: IMPALA
Issue Type: Bug
Components: Backend, Frontend
Reporter: Zoltán Borók-Nagy
Components using [S3FileIO|https://iceberg.apache.org/docs/latest/aws/#s3-fileio] might write out file paths starting with 's3://' instead of 's3a://'. The latter is used by [HadoopFileIO|https://iceberg.apache.org/docs/latest/aws/#hadoop-s3a-filesystem] that Impala is using.
By default, HadoopFileIO doesn't interpret paths starting with 's3://'. (Probably this could be resolved by setting "fs.s3.impl" to "org.apache.hadoop.fs.s3a.S3AFileSystem" so that an s3a fs instance is created)
[FeIcebergTable.Utils.FeIcebergTable()|https://github.com/apache/impala/blob/2733d039ad4a830a1ea34c1a75d2b666788e39a9/fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java#L671-L689] depends on file paths returned by recursive file listing match the file paths in Iceberg metadata files. But the recursive listing returns s3a:// paths, while metadata contains s3:// paths, which means we'll load files one-by-one as we won't find the files in the hash map 'hdfsFileDescMap'.
Moreover, if position delete file processing is also based on exact matches of the file URIs. Therefore if entries with s3:// paths won't have the desired effects.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)