You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "Alex Rodoni (JIRA)" <ji...@apache.org> on 2019/01/23 00:50:00 UTC
[jira] [Commented] (IMPALA-7265) Cache remote file handles
[ https://issues.apache.org/jira/browse/IMPALA-7265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16749340#comment-16749340 ]
Alex Rodoni commented on IMPALA-7265:
-------------------------------------
[~joemcdonnell] Is there a doc impact for this feature?
> Cache remote file handles
> -------------------------
>
> Key: IMPALA-7265
> URL: https://issues.apache.org/jira/browse/IMPALA-7265
> Project: IMPALA
> Issue Type: Improvement
> Components: Backend
> Affects Versions: Impala 3.1.0
> Reporter: Joe McDonnell
> Assignee: Joe McDonnell
> Priority: Critical
>
> The file handle cache currently does not allow caching remote file handles. This means that clusters that have a lot of remote reads can suffer from overloading the NameNode. Impala should be able to cache remote file handles.
> There are some open questions about remote file handles and whether they behave differently from local file handles. In particular:
> # Is there any resource constraint on the number of remote file handles open? (e.g. do they maintain a network connection?)
> # Are there any semantic differences in how remote file handles behave when files are deleted, overwritten, or appended?
> # Are there any extra failure cases for remote file handles? (i.e. if a machine goes down or a remote file handle is left open for an extended period of time)
> The form of caching will depend on the answers, but at the very least, it should be possible to cache a remote file handle at the level of a query so that a Parquet file with multiple columns can share file handles.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org