You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Chris Douglas (JIRA)" <ji...@apache.org> on 2017/11/29 06:08:00 UTC
[jira] [Updated] (MAPREDUCE-7016) Avoid making separate RPC calls
for FileStatus and block locations in FileInputFormat
[ https://issues.apache.org/jira/browse/MAPREDUCE-7016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas updated MAPREDUCE-7016:
-------------------------------------
Description: {{FileInputFormat::getSplits}} uses {{FileSystem::globStatus}} to determine its inputs. When the glob returns directories, each is traversed and {{LocatedFileStatus}} instances are returned with the block locations. However, when the glob returns files, this is a {{FileStatus}} that requires a second RPC to obtain its locations. (was: {{FileInputFormat::getSplits}} uses {{FileSystem::globStatus}} to determine its inputs. When the glob returns directories, each is traversed and {{LocatedFileStatus}} instances are returned with the block locations. However, when the glob returns files, each requires a second RPC to obtain its locations.)
> Avoid making separate RPC calls for FileStatus and block locations in FileInputFormat
> -------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-7016
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7016
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Chris Douglas
>
> {{FileInputFormat::getSplits}} uses {{FileSystem::globStatus}} to determine its inputs. When the glob returns directories, each is traversed and {{LocatedFileStatus}} instances are returned with the block locations. However, when the glob returns files, this is a {{FileStatus}} that requires a second RPC to obtain its locations.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-help@hadoop.apache.org