You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "Yong Yang (Jira)" <ji...@apache.org> on 2021/07/08 11:38:00 UTC
[jira] [Updated] (IMPALA-10780) Iceberg in Impala should support
combination storage
[ https://issues.apache.org/jira/browse/IMPALA-10780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Yong Yang updated IMPALA-10780:
-------------------------------
Description:
Currently, the filesystem of the metadata path is used to check the data file, that is blocking the following scenario:
# metadata is on hdfs
# data is on s3a or other object store.
Following code in FeIcebergTable.Utils fails this combination:
private static HdfsPartition.FileDescriptor getFileDescriptor(Path fileLoc,
Path tableLoc, ListMap<TNetworkAddress> hostIndex) throws IOException
{
FileSystem fs = FileSystemUtil.getFileSystemForPath(tableLoc); FileStatus fileStatus = fs.getFileStatus(fileLoc);
return getFileDescriptor(fs, tableLoc, fileStatus, hostIndex);
}
was:
Currently, the filesystem of the metadata path is used to check the data file, that is blocking the following scenario:
# metadata is on hdfs
# data is on s3a or other object store.
Following code in FeIcebergTable.Utils fails this combination:
private static HdfsPartition.FileDescriptor getFileDescriptor(Path fileLoc,
Path tableLoc, ListMap<TNetworkAddress> hostIndex) throws IOException {
FileSystem fs = FileSystemUtil.getFileSystemForPath(tableLoc);
FileStatus fileStatus = fs.getFileStatus(fileLoc);
return getFileDescriptor(fs, tableLoc, fileStatus, hostIndex);
}
> Iceberg in Impala should support combination storage
> ----------------------------------------------------
>
> Key: IMPALA-10780
> URL: https://issues.apache.org/jira/browse/IMPALA-10780
> Project: IMPALA
> Issue Type: Improvement
> Reporter: Yong Yang
> Priority: Major
>
> Currently, the filesystem of the metadata path is used to check the data file, that is blocking the following scenario:
> # metadata is on hdfs
> # data is on s3a or other object store.
>
> Following code in FeIcebergTable.Utils fails this combination:
> private static HdfsPartition.FileDescriptor getFileDescriptor(Path fileLoc,
> Path tableLoc, ListMap<TNetworkAddress> hostIndex) throws IOException
> {
> FileSystem fs = FileSystemUtil.getFileSystemForPath(tableLoc); FileStatus fileStatus = fs.getFileStatus(fileLoc);
> return getFileDescriptor(fs, tableLoc, fileStatus, hostIndex);
> }
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org