You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "ASF subversion and git services (Jira)" <ji...@apache.org> on 2022/08/31 19:47:00 UTC
[jira] [Commented] (IMPALA-10375) Lock down which filesystem types use the file handle cache
[ https://issues.apache.org/jira/browse/IMPALA-10375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17598606#comment-17598606 ]
ASF subversion and git services commented on IMPALA-10375:
----------------------------------------------------------
Commit 9b4a3ef7952c4c6aef5d33703d68bb7994644a0f in impala's branch refs/heads/master from Michael Smith
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=9b4a3ef79 ]
IMPALA-10214, IMPALA-10375: Ozone remote file handle caching
Enables support for caching remote file handles for Ozone. Local file
handles were already cached unintentionally, similar to HDFS. Updates
file handle cache enablement to be more stringent about enabling
caching.
File handle caching is enabled if a max_cached_file_handles is non-zero
and any of the following are true
- HDFS file is local
- HDFS file is remote and cache_remote_file_handles is enabled
- Ozone file is local or remote and cache_ozone_file_handles is enabled
- S3 file is remote and cache_s3_file_handles is enabled
- ABFS file is remote and cache_abfs_file_handles is enabled
Enables testing Ozone in test_hdfs_fd_caching, and adds tests that
remote caching can be disabled using individual flags.
Change-Id: I9df13208999c6d3b14f4c005a91ee2a92a05bdf9
Reviewed-on: http://gerrit.cloudera.org:8080/18853
Tested-by: Impala Public Jenkins <im...@cloudera.com>
Reviewed-by: Csaba Ringhofer <cs...@cloudera.com>
> Lock down which filesystem types use the file handle cache
> ----------------------------------------------------------
>
> Key: IMPALA-10375
> URL: https://issues.apache.org/jira/browse/IMPALA-10375
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
> Affects Versions: Impala 4.0.0
> Reporter: Joe McDonnell
> Assignee: Joe McDonnell
> Priority: Major
> Fix For: Impala 4.2.0
>
>
> When determining whether to cache file handles, the code uses this check:
> {noformat}
> bool use_file_handle_cache = false;
> if (is_file_handle_caching_enabled() &&
> (expected_local_ ||
> (FLAGS_cache_remote_file_handles && disk_id_ == io_mgr_->RemoteDfsDiskId()) ||
> (FLAGS_cache_s3_file_handles && disk_id_ == io_mgr_->RemoteS3DiskId()) ||
> (FLAGS_cache_abfs_file_handles && disk_id_ == io_mgr_->RemoteAbfsDiskId())) {
> use_file_handle_cache = true;
> }{noformat}
> If expected_local_ is specified, then we always would allow caching, regardless of filesystem type. While expected_local_ should be false for remote filesystems, the code should be more defensive and only use the file handle cache for specific supported filesystems. In particular, we currently don't support caching Ozone file handles, but if this somehow considered Ozone local, then it would cache them.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org