[jira] [Resolved] (IMPALA-7265) Cache remote file handles

["Joe McDonnell (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/IMPALA-7265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Joe McDonnell resolved IMPALA-7265.
-----------------------------------
       Resolution: Fixed
    Fix Version/s: Impala 3.2.0

> Cache remote file handles
> -------------------------
>
>                 Key: IMPALA-7265
>                 URL: https://issues.apache.org/jira/browse/IMPALA-7265
>             Project: IMPALA
>          Issue Type: Improvement
>          Components: Backend
>    Affects Versions: Impala 3.1.0
>            Reporter: Joe McDonnell
>            Assignee: Joe McDonnell
>            Priority: Critical
>             Fix For: Impala 3.2.0
>
>
> The file handle cache currently does not allow caching remote file handles. This means that clusters that have a lot of remote reads can suffer from overloading the NameNode. Impala should be able to cache remote file handles.
> There are some open questions about remote file handles and whether they behave differently from local file handles. In particular:
>  # Is there any resource constraint on the number of remote file handles open? (e.g. do they maintain a network connection?)
>  # Are there any semantic differences in how remote file handles behave when files are deleted, overwritten, or appended?
>  # Are there any extra failure cases for remote file handles? (i.e. if a machine goes down or a remote file handle is left open for an extended period of time)
> The form of caching will depend on the answers, but at the very least, it should be possible to cache a remote file handle at the level of a query so that a Parquet file with multiple columns can share file handles.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)