You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Owen O'Malley (JIRA)" <ji...@apache.org> on 2009/10/08 20:17:31 UTC
[jira] Created: (HADOOP-6302) Move FileSystem and all of the
implementations to HDFS project
Move FileSystem and all of the implementations to HDFS project
--------------------------------------------------------------
Key: HADOOP-6302
URL: https://issues.apache.org/jira/browse/HADOOP-6302
Project: Hadoop Common
Issue Type: Improvement
Reporter: Owen O'Malley
Fix For: 0.21.0
Currently, the FileSystem and FileContext classes are in Common and the primary implementation is in HDFS. That means that many patches span between the subprojects. I think it will reduce the pain if we move FileSystem and the dependent classes into HDFS.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6302) Move FileSystem and all of the
implementations to HDFS project
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12763775#action_12763775 ]
Todd Lipcon commented on HADOOP-6302:
-------------------------------------
Yea, this seems a little odd. To me, it seems like this just points out the need for a better build-integration between the projects that doesn't involve committing jars after every patch. I think the ivy/maven work is a step in that direction, though I honestly have not been following it as closely as I should.
> Move FileSystem and all of the implementations to HDFS project
> --------------------------------------------------------------
>
> Key: HADOOP-6302
> URL: https://issues.apache.org/jira/browse/HADOOP-6302
> Project: Hadoop Common
> Issue Type: Improvement
> Reporter: Owen O'Malley
> Fix For: 0.21.0
>
>
> Currently, the FileSystem and FileContext classes are in Common and the primary implementation is in HDFS. That means that many patches span between the subprojects. I think it will reduce the pain if we move FileSystem and the dependent classes into HDFS.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6302) Move FileSystem and all of the
implementations to HDFS project
Posted by "Gregory Farnum (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12763668#action_12763668 ]
Gregory Farnum commented on HADOOP-6302:
----------------------------------------
Isn't the whole point of the FileSystem setup that MapRed, Common, et al are completely divorced from the details of HDFS? It seems like moving the interface classes into the HDFS subproject would make the other projects much more dependent on HDFS for their release cycle and make things harder on external developers trying to interface their own filesystems with the overall project.
Though my perspective on this is pretty narrow, to say the least...
> Move FileSystem and all of the implementations to HDFS project
> --------------------------------------------------------------
>
> Key: HADOOP-6302
> URL: https://issues.apache.org/jira/browse/HADOOP-6302
> Project: Hadoop Common
> Issue Type: Improvement
> Reporter: Owen O'Malley
> Fix For: 0.21.0
>
>
> Currently, the FileSystem and FileContext classes are in Common and the primary implementation is in HDFS. That means that many patches span between the subprojects. I think it will reduce the pain if we move FileSystem and the dependent classes into HDFS.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6302) Move FileSystem and all of the
implementations to HDFS project
Posted by "Sharad Agarwal (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12764543#action_12764543 ]
Sharad Agarwal commented on HADOOP-6302:
----------------------------------------
mapreduce dependency on HDFS is only runtime. Lot of tests depend on HDFS because HDFS is the primary filesystem which is used with mapreduce. We can easily consider a case where there may be additional tests in mapreduce which work with different filesystem implementations. I don't see any problem in this model.
> Move FileSystem and all of the implementations to HDFS project
> --------------------------------------------------------------
>
> Key: HADOOP-6302
> URL: https://issues.apache.org/jira/browse/HADOOP-6302
> Project: Hadoop Common
> Issue Type: Improvement
> Reporter: Owen O'Malley
> Fix For: 0.21.0
>
>
> Currently, the FileSystem and FileContext classes are in Common and the primary implementation is in HDFS. That means that many patches span between the subprojects. I think it will reduce the pain if we move FileSystem and the dependent classes into HDFS.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6302) Move FileSystem and all of the
implementations to HDFS project
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12764177#action_12764177 ]
Doug Cutting commented on HADOOP-6302:
--------------------------------------
I agree that this feels wrong. It puts HDFS in a privileged position that could be abused, if only inadvertently.
But we might still do this.
We've already embraced the dependency order of common<hdfs<mapreduce, right? The primary users who benefit from an abstract filesystem API are mapreduce users, who're already now dependent on HDFS. This change only affects filesystem API users who use neither HDFS nor MapReduce. Are there any such?
I wish we could have found a way to make MapReduce depend directly on Common. The problem as I understood it was tests: it's nice to keep tests with the code they test, and its also nice to test MapReduce with HDFS. I suggested that we might use a higher-level build that keeps tests with their projects but builds tests after both mapreduce and hdfs core, but that's perhaps not workable long-term, if we really want these to be able to evolve as separate projects with separate release schedules.
+0
> Move FileSystem and all of the implementations to HDFS project
> --------------------------------------------------------------
>
> Key: HADOOP-6302
> URL: https://issues.apache.org/jira/browse/HADOOP-6302
> Project: Hadoop Common
> Issue Type: Improvement
> Reporter: Owen O'Malley
> Fix For: 0.21.0
>
>
> Currently, the FileSystem and FileContext classes are in Common and the primary implementation is in HDFS. That means that many patches span between the subprojects. I think it will reduce the pain if we move FileSystem and the dependent classes into HDFS.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6302) Move FileSystem and all of the
implementations to HDFS project
Posted by "hadoopguy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12763734#action_12763734 ]
hadoopguy commented on HADOOP-6302:
-----------------------------------
I strongly agree with Gregory. Moving the FileSystem and FileContext into HDFS is not clean for other filesystem implementors. HDFS is one implementation of the FileSystem API - lets keep it separate.
Also won't the pain of the patches get reduced as we are striving hard to make the FileSystem API ready as 1.0 ?
> Move FileSystem and all of the implementations to HDFS project
> --------------------------------------------------------------
>
> Key: HADOOP-6302
> URL: https://issues.apache.org/jira/browse/HADOOP-6302
> Project: Hadoop Common
> Issue Type: Improvement
> Reporter: Owen O'Malley
> Fix For: 0.21.0
>
>
> Currently, the FileSystem and FileContext classes are in Common and the primary implementation is in HDFS. That means that many patches span between the subprojects. I think it will reduce the pain if we move FileSystem and the dependent classes into HDFS.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.