You are viewing a plain text version of this content. The canonical link for it is here.
Posted to yarn-issues@hadoop.apache.org by "Angus Davis (JIRA)" <ji...@apache.org> on 2017/06/21 02:14:00 UTC
[jira] [Updated] (YARN-6725) NodeManager cannot localize files when
URL contains opaque authority
[ https://issues.apache.org/jira/browse/YARN-6725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Angus Davis updated YARN-6725:
------------------------------
Attachment: YARN-6721.001.patch
First pass at adding authority to YARN URL pb.
> NodeManager cannot localize files when URL contains opaque authority
> --------------------------------------------------------------------
>
> Key: YARN-6725
> URL: https://issues.apache.org/jira/browse/YARN-6725
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
> Affects Versions: 2.8.1
> Reporter: Angus Davis
> Priority: Minor
> Attachments: YARN-6721.001.patch
>
>
> When presented with a URI or URL that has a valid structure, but where the URI / URL authority cannot be interpreted as a valid hostname, the java URI#getHost method will return null. In the YARN api records the URL message is decomposed into hostname, username, port, etc, but it does not provide a means to transport an opaque authority. This becomes problematic when using object stores or other file systems that use something other than a hostname as an authority.
> In our particular case, Google Cloud Storage allows buckets to be named with underscores which is causing the NodeManagers to attempt to localize resources of the form 'gs:///path/to/object' when they should be attempting to localize 'gs://bucket/path/to/object'. Components that are written in terms of o.a.h.fs.Path instances seem to handle this case properly, as do most Path utility methods (toUri, etc).
> It seems that by transporting the authority along with the host, user info, etc, a YARN api URL can properly represent o.a.h.fs.Path instances of this particular form.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org