[jira] [Commented] (MESOS-1711) Create method for users to identify HDFS compatible protocols in fetcher.cpp

["Adam B (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/MESOS-1711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14193724#comment-14193724 ] 

Adam B commented on MESOS-1711:
-------------------------------

Ankur's proposed fix: https://reviews.apache.org/r/27483
[~tstclair] may want to take a look too.

> Create method for users to identify HDFS compatible protocols in fetcher.cpp
> ----------------------------------------------------------------------------
>
>                 Key: MESOS-1711
>                 URL: https://issues.apache.org/jira/browse/MESOS-1711
>             Project: Mesos
>          Issue Type: Improvement
>          Components: general
>    Affects Versions: 0.19.1
>         Environment: All
>            Reporter: John Omernik
>            Assignee: Timothy St. Clair
>            Priority: Minor
>              Labels: fetcher, hadoop, hdfs
>   Original Estimate: 6h
>  Remaining Estimate: 6h
>
> In fetcher.cpp, the code to get the Mesos packages uses a hard coded list of protocols to determine if the Hadoop copytoLocal method is used or if another method (such as standard filecopy).  This limits the addition of new protocols that are HDFS compatible until the next release of Mesos. Tachyon Filesystem (tachyonfs://), MapR FS (maprfs://) and glusterfs:// are three examples that could make use of this. 
> Instead of just adding those file systems in the hard coded list, I recommend following the lead of the Tachyon Project.  In tachyon-0.6.0-SNAPSHOT, they have added an environment variable of allowed hdfs compatible protocols. This comma-separated list allows the user/admin to specify which protocols are HDFS compatible, without hard coding it in the fetcher.cpp.   
> I don't have access to the Tachyon issues list for linking, but the code is on line 75 of 
> https://github.com/amplab/tachyon/blob/master/core/src/main/java/tachyon/UnderFileSystem.java



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)