[jira] [Comment Edited] (IMPALA-10214) Ozone support for file handle cache

["Michael Smith (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/IMPALA-10214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17572642#comment-17572642 ] 

Michael Smith edited comment on IMPALA-10214 at 7/28/22 8:38 PM:
-----------------------------------------------------------------

Local file handles are already cached because Ozone shares the HDFS implementation.

The file handle cache now supports local file handles, as well as remote HDFS/S3/ABFS handles. In evaluating Ozone, [~weichiu] identified that memory used by the file handle cache was substantially larger than HDFS when using remote handles. The original measurements behind setting {{max_cached_file_handles=20000}} probably no longer apply, but I'm not sure it's safe to change it.


was (Author: JIRAUSER288956):
Local file handles are already cached because Ozone shares the HDFS implementation. We likely want to assign a separate cache for Ozone local file handles ({{DiskIoMgr::file_handle_cache_}}).

> Ozone support for file handle cache
> -----------------------------------
>
>                 Key: IMPALA-10214
>                 URL: https://issues.apache.org/jira/browse/IMPALA-10214
>             Project: IMPALA
>          Issue Type: Sub-task
>          Components: Backend
>            Reporter: Sahil Takiar
>            Priority: Major
>
> This is dependent on the Ozone input streams supporting the {{CanUnbuffer}} interface first (last I checked, the input streams don't implement the interface).



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org