You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "ASF GitHub Bot (JIRA)" <ji...@apache.org> on 2017/01/25 02:04:26 UTC

[jira] [Commented] (FLINK-5631) [yarn] Support downloading additional jars from non-HDFS paths

    [ https://issues.apache.org/jira/browse/FLINK-5631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15837036#comment-15837036 ] 

ASF GitHub Bot commented on FLINK-5631:
---------------------------------------

GitHub user haohui opened a pull request:

    https://github.com/apache/flink/pull/3202

    [FLINK-5631] [yarn] Support downloading additional jars from non-HDFS…

    This PR enables Flink to support downloading additional jars from non-HDFS paths (e.g., S3, viewfs). More details are available in the jira.
    
    There are two changes in this PR:
    
      * Refactoring the code for localization into a common function
      * Use `Path.getFileSystem()` to obtain the FileSystem object instead of always using the default FileSystem constructed using YARN config.


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/haohui/flink FLINK-5631

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/flink/pull/3202.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #3202
    
----
commit f7d24c5a0559ca71cd49962c9dc0b4ab81997783
Author: Haohui Mai <wh...@apache.org>
Date:   2017-01-25T02:00:08Z

    [FLINK-5631] [yarn] Support downloading additional jars from non-HDFS paths.

----


> [yarn] Support downloading additional jars from non-HDFS paths
> --------------------------------------------------------------
>
>                 Key: FLINK-5631
>                 URL: https://issues.apache.org/jira/browse/FLINK-5631
>             Project: Flink
>          Issue Type: Bug
>            Reporter: Haohui Mai
>            Assignee: Haohui Mai
>
> Currently the {{YarnResourceManager}} and {{YarnApplicationMasterRunner}} always register the additional jars using the YARN filesystem object. This is problematic as the paths might require another filesystem.
> To support localizing from non-HDFS paths (e.g., s3, http or viewfs), the cleaner approach is to get the filesystem object from the path.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)