You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "David Dudley (Jira)" <ji...@apache.org> on 2020/01/15 19:46:00 UTC
[jira] [Commented] (HADOOP-16807) Enable Filesystem caching to
optionally include URI Path
[ https://issues.apache.org/jira/browse/HADOOP-16807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17016259#comment-17016259 ]
David Dudley commented on HADOOP-16807:
---------------------------------------
The current Key class in the Filesystem class only considers the Scheme, Authority, and User from the URI when generating the hashCode for filesystem caching. To allow users greater flexibility in determining how to cache the Filesystem, I propose adding optional functionality to incorporate configurable portions of the Path to the hashCode generated for filesystem caching.
Add optional hadoop config to define a split delimiter
Add optional hadoop config to define split depth
Modify Key class to
Read split delimiter
Read split depth
Split URI Path by delimiter
Add elements up to split depth to the hashCode generated
> Enable Filesystem caching to optionally include URI Path
> --------------------------------------------------------
>
> Key: HADOOP-16807
> URL: https://issues.apache.org/jira/browse/HADOOP-16807
> Project: Hadoop Common
> Issue Type: Improvement
> Reporter: David Dudley
> Priority: Major
>
> Implementing AWSCredentialsProviders that dynamically retrieve STS tokens based on the URI being accessed fail if Filesystem caching is enabled and the job accesses more than one URI Path within the same bucket.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org