You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "ASF subversion and git services (Jira)" <ji...@apache.org> on 2025/08/08 05:30:00 UTC
[jira] [Commented] (IMPALA-14138) Detect if filesystem is not co-located in which case we shouldn't collect block location information
[ https://issues.apache.org/jira/browse/IMPALA-14138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18012735#comment-18012735 ]
ASF subversion and git services commented on IMPALA-14138:
----------------------------------------------------------
Commit 0dd6c154c9f05223768900bff7058b1cdc769c1c in impala's branch refs/heads/master from Peter Rozsa
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=0dd6c154c ]
IMPALA-14138: Include generated files in .gitignore
Change-Id: Ie5079c87dd27c2391bee13937d69a3340e2a4fb1
Reviewed-on: http://gerrit.cloudera.org:8080/23223
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
> Detect if filesystem is not co-located in which case we shouldn't collect block location information
> ----------------------------------------------------------------------------------------------------
>
> Key: IMPALA-14138
> URL: https://issues.apache.org/jira/browse/IMPALA-14138
> Project: IMPALA
> Issue Type: Bug
> Reporter: Zoltán Borók-Nagy
> Assignee: Zoltán Borók-Nagy
> Priority: Major
> Labels: impala-iceberg
> Fix For: Impala 5.0.0
>
>
> For storage systems that support block location information (HDFS, Ozone) we always retrieve it with the assumption that we can use it for scheduling, to do local reads:
> [https://github.com/apache/impala/blob/master/fe/src/main/java/org/apache/impala/catalog/IcebergFileMetadataLoader.java#L154]
> But it's also typical that Impala is not co-located with the storage system, not even in on-prem deployments. E.g. PVC DS Impala runs in containers, and even if they are co-located, I don't think we try to figure out which container runs on which machine. Also short-circuit reads are off the table.In such cases we should not reach out to the storage system to collect file information because it can be very expensive for large tables and we won't benefit from it at all.
>
> We could construct the file descriptors based on the information we have in the Iceberg manifests, and this is what we already do on cloud storage systems.
> Is there a good indicator that Impala is not co-located with the configured filesystems? E.g. if data cache is enabled? But of course there could be multiple filesystems configured, some of them co-located, some not..
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org