You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@mesos.apache.org by Adam B <ad...@mesosphere.io> on 2016/01/03 11:50:56 UTC
Re: Review Request 41713: Added HDFS URI fetcher plugin.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41713/#review112444
-----------------------------------------------------------
Hardcoded hadoop schemes? Eek! Guess we'll have to reopen MESOS-1711?
src/uri/fetchers/hadoop.cpp (line 54)
<https://reviews.apache.org/r/41713/#comment172917>
Why is this hardcoded again? We moved away from that for https://issues.apache.org/jira/browse/MESOS-1711 but now tachyon/maprfs users will be broken (again).
Can we make the supported schemes a plugin flag/parameter, so users can adjust as needed?
- Adam B
On Dec. 30, 2015, 9:35 p.m., Jie Yu wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/41713/
> -----------------------------------------------------------
>
> (Updated Dec. 30, 2015, 9:35 p.m.)
>
>
> Review request for mesos, Adam B, Ben Mahler, Ian Downes, Timothy Chen, and Vinod Kone.
>
>
> Bugs: MESOS-3925
> https://issues.apache.org/jira/browse/MESOS-3925
>
>
> Repository: mesos
>
>
> Description
> -------
>
> Added HDFS URI fetcher plugin. This patch also adjusted the plugin initialization logic so that flags can be used to initialize each plugin.
>
>
> Diffs
> -----
>
> include/mesos/uri/fetcher.hpp 6cf7b37818ceff64ffe56138ebd4bc7e2abe5a7c
> include/mesos/uri/uri.hpp aa3ab5d24bb8c501dd19e93d7563cb0afd889f23
> src/Makefile.am bf71fe6ee65233bbc47a41f610d68ff9760e19f6
> src/tests/uri_fetcher_tests.cpp d54a7e7b08a741b1d5d371c11747681ae4ddef98
> src/uri/fetcher.hpp PRE-CREATION
> src/uri/fetcher.cpp 8bd7cf3d638a737c40183294cb4548a4851e0886
> src/uri/fetchers/curl.hpp 401829d1990ec817ea069ce45f902c521cd1a937
> src/uri/fetchers/curl.cpp 6c607a6a800374b6e14b37bfdc6b04e7a4e87892
> src/uri/fetchers/hadoop.hpp PRE-CREATION
> src/uri/fetchers/hadoop.cpp PRE-CREATION
>
> Diff: https://reviews.apache.org/r/41713/diff/
>
>
> Testing
> -------
>
> make check
>
>
> Thanks,
>
> Jie Yu
>
>
Re: Review Request 41713: Added HDFS URI fetcher plugin.
Posted by Jie Yu <yu...@gmail.com>.
> On Jan. 3, 2016, 10:50 a.m., Adam B wrote:
> > src/uri/fetchers/hadoop.cpp, line 54
> > <https://reviews.apache.org/r/41713/diff/2/?file=1180044#file1180044line54>
> >
> > Why is this hardcoded again? We moved away from that for https://issues.apache.org/jira/browse/MESOS-1711 but now tachyon/maprfs users will be broken (again).
> > Can we make the supported schemes a plugin flag/parameter, so users can adjust as needed?
Yeah, I'll follow up with a patch to allow users to configure schemes through a flag.
- Jie
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41713/#review112444
-----------------------------------------------------------
On Dec. 31, 2015, 5:35 a.m., Jie Yu wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/41713/
> -----------------------------------------------------------
>
> (Updated Dec. 31, 2015, 5:35 a.m.)
>
>
> Review request for mesos, Adam B, Ben Mahler, Ian Downes, Timothy Chen, and Vinod Kone.
>
>
> Bugs: MESOS-3925
> https://issues.apache.org/jira/browse/MESOS-3925
>
>
> Repository: mesos
>
>
> Description
> -------
>
> Added HDFS URI fetcher plugin. This patch also adjusted the plugin initialization logic so that flags can be used to initialize each plugin.
>
>
> Diffs
> -----
>
> include/mesos/uri/fetcher.hpp 6cf7b37818ceff64ffe56138ebd4bc7e2abe5a7c
> include/mesos/uri/uri.hpp aa3ab5d24bb8c501dd19e93d7563cb0afd889f23
> src/Makefile.am bf71fe6ee65233bbc47a41f610d68ff9760e19f6
> src/tests/uri_fetcher_tests.cpp d54a7e7b08a741b1d5d371c11747681ae4ddef98
> src/uri/fetcher.hpp PRE-CREATION
> src/uri/fetcher.cpp 8bd7cf3d638a737c40183294cb4548a4851e0886
> src/uri/fetchers/curl.hpp 401829d1990ec817ea069ce45f902c521cd1a937
> src/uri/fetchers/curl.cpp 6c607a6a800374b6e14b37bfdc6b04e7a4e87892
> src/uri/fetchers/hadoop.hpp PRE-CREATION
> src/uri/fetchers/hadoop.cpp PRE-CREATION
>
> Diff: https://reviews.apache.org/r/41713/diff/
>
>
> Testing
> -------
>
> make check
>
>
> Thanks,
>
> Jie Yu
>
>
Re: Review Request 41713: Added HDFS URI fetcher plugin.
Posted by haosdent huang <ha...@gmail.com>.
> On Jan. 3, 2016, 10:50 a.m., Adam B wrote:
> > src/uri/fetchers/hadoop.cpp, line 54
> > <https://reviews.apache.org/r/41713/diff/2/?file=1180044#file1180044line54>
> >
> > Why is this hardcoded again? We moved away from that for https://issues.apache.org/jira/browse/MESOS-1711 but now tachyon/maprfs users will be broken (again).
> > Can we make the supported schemes a plugin flag/parameter, so users can adjust as needed?
>
> Jie Yu wrote:
> Yeah, I'll follow up with a patch to allow users to configure schemes through a flag.
Seems hadoop also support below scheme:
file,hdfs,webhdfs,swebhdfs,har,viewfs,ftp,s3a,wasb,swift
- haosdent
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41713/#review112444
-----------------------------------------------------------
On Dec. 31, 2015, 5:35 a.m., Jie Yu wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/41713/
> -----------------------------------------------------------
>
> (Updated Dec. 31, 2015, 5:35 a.m.)
>
>
> Review request for mesos, Adam B, Ben Mahler, Ian Downes, Timothy Chen, and Vinod Kone.
>
>
> Bugs: MESOS-3925
> https://issues.apache.org/jira/browse/MESOS-3925
>
>
> Repository: mesos
>
>
> Description
> -------
>
> Added HDFS URI fetcher plugin. This patch also adjusted the plugin initialization logic so that flags can be used to initialize each plugin.
>
>
> Diffs
> -----
>
> include/mesos/uri/fetcher.hpp 6cf7b37818ceff64ffe56138ebd4bc7e2abe5a7c
> include/mesos/uri/uri.hpp aa3ab5d24bb8c501dd19e93d7563cb0afd889f23
> src/Makefile.am bf71fe6ee65233bbc47a41f610d68ff9760e19f6
> src/tests/uri_fetcher_tests.cpp d54a7e7b08a741b1d5d371c11747681ae4ddef98
> src/uri/fetcher.hpp PRE-CREATION
> src/uri/fetcher.cpp 8bd7cf3d638a737c40183294cb4548a4851e0886
> src/uri/fetchers/curl.hpp 401829d1990ec817ea069ce45f902c521cd1a937
> src/uri/fetchers/curl.cpp 6c607a6a800374b6e14b37bfdc6b04e7a4e87892
> src/uri/fetchers/hadoop.hpp PRE-CREATION
> src/uri/fetchers/hadoop.cpp PRE-CREATION
>
> Diff: https://reviews.apache.org/r/41713/diff/
>
>
> Testing
> -------
>
> make check
>
>
> Thanks,
>
> Jie Yu
>
>