You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Eli Collins (Created) (JIRA)" <ji...@apache.org> on 2012/01/26 19:37:40 UTC
[jira] [Created] (MAPREDUCE-3736) Variable substitution depth too
large for fs.default.name causes jobs to fail
Variable substitution depth too large for fs.default.name causes jobs to fail
-----------------------------------------------------------------------------
Key: MAPREDUCE-3736
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3736
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 0.23.1
Reporter: Eli Collins
Priority: Blocker
I'm seeing the same failure as MAPREDUCE-3462 in downstream projects running against a recent build of branch-23. MR-3462 modified the tests rather than fixing the framework. In that jira Ravi mentioned "I'm still ignorant of the change which made the tests start to fail. I should probably understand better the reasons for that change before proposing a more generalized fix." Let's figure out the general fix (rather than require all projects to set mapreduce.job.hdfs-servers in their conf we should fix this in the framework). Perhaps we should not default this config to "$fs.default.name"?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3736) Variable substitution depth too
large for fs.default.name causes jobs to fail
Posted by "Eli Collins (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13194114#comment-13194114 ]
Eli Collins commented on MAPREDUCE-3736:
----------------------------------------
Btw setting "mapreduce.job.hdfs-servers" to the empty string in the test confs fixes it (like 3462) but upstream projects shouldn't have to do this.
> Variable substitution depth too large for fs.default.name causes jobs to fail
> -----------------------------------------------------------------------------
>
> Key: MAPREDUCE-3736
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3736
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.1
> Reporter: Eli Collins
> Priority: Blocker
>
> I'm seeing the same failure as MAPREDUCE-3462 in downstream projects running against a recent build of branch-23. MR-3462 modified the tests rather than fixing the framework. In that jira Ravi mentioned "I'm still ignorant of the change which made the tests start to fail. I should probably understand better the reasons for that change before proposing a more generalized fix." Let's figure out the general fix (rather than require all projects to set mapreduce.job.hdfs-servers in their conf we should fix this in the framework). Perhaps we should not default this config to "$fs.default.name"?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3736) Variable substitution depth too
large for fs.default.name causes jobs to fail
Posted by "Ahmed Radwan (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13195037#comment-13195037 ]
Ahmed Radwan commented on MAPREDUCE-3736:
-----------------------------------------
fs.default.name is added to core-default.xml, where fs.defaultFS lives now too.
I didn't change all MR uses of fs.default.name to use fs.defaultFS as I thought it is out of scope of the ticket. But I can add them, I'll update the patch momentarily. Thanks!
> Variable substitution depth too large for fs.default.name causes jobs to fail
> -----------------------------------------------------------------------------
>
> Key: MAPREDUCE-3736
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3736
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.1
> Reporter: Eli Collins
> Assignee: Ahmed Radwan
> Priority: Blocker
> Attachments: MAPREDUCE-3736.patch
>
>
> I'm seeing the same failure as MAPREDUCE-3462 in downstream projects running against a recent build of branch-23. MR-3462 modified the tests rather than fixing the framework. In that jira Ravi mentioned "I'm still ignorant of the change which made the tests start to fail. I should probably understand better the reasons for that change before proposing a more generalized fix." Let's figure out the general fix (rather than require all projects to set mapreduce.job.hdfs-servers in their conf we should fix this in the framework). Perhaps we should not default this config to "$fs.default.name"?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3736) Variable substitution depth too
large for fs.default.name causes jobs to fail
Posted by "Eli Collins (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13196319#comment-13196319 ]
Eli Collins commented on MAPREDUCE-3736:
----------------------------------------
HADOOP-7993 also looks related, there was a place in Configuration where deprecation was not being handled.
> Variable substitution depth too large for fs.default.name causes jobs to fail
> -----------------------------------------------------------------------------
>
> Key: MAPREDUCE-3736
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3736
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.1
> Reporter: Eli Collins
> Assignee: Ahmed Radwan
> Priority: Blocker
> Attachments: MAPREDUCE-3736.patch, MAPREDUCE-3736_rev2.patch
>
>
> I'm seeing the same failure as MAPREDUCE-3462 in downstream projects running against a recent build of branch-23. MR-3462 modified the tests rather than fixing the framework. In that jira Ravi mentioned "I'm still ignorant of the change which made the tests start to fail. I should probably understand better the reasons for that change before proposing a more generalized fix." Let's figure out the general fix (rather than require all projects to set mapreduce.job.hdfs-servers in their conf we should fix this in the framework). Perhaps we should not default this config to "$fs.default.name"?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3736) Variable substitution depth too
large for fs.default.name causes jobs to fail
Posted by "Ravi Prakash (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13194264#comment-13194264 ]
Ravi Prakash commented on MAPREDUCE-3736:
-----------------------------------------
I and Amar had come to a consensus that src/test/mapred-site.xml needs to have that property defined and it should be used.
> Variable substitution depth too large for fs.default.name causes jobs to fail
> -----------------------------------------------------------------------------
>
> Key: MAPREDUCE-3736
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3736
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.1
> Reporter: Eli Collins
> Assignee: Ahmed Radwan
> Priority: Blocker
>
> I'm seeing the same failure as MAPREDUCE-3462 in downstream projects running against a recent build of branch-23. MR-3462 modified the tests rather than fixing the framework. In that jira Ravi mentioned "I'm still ignorant of the change which made the tests start to fail. I should probably understand better the reasons for that change before proposing a more generalized fix." Let's figure out the general fix (rather than require all projects to set mapreduce.job.hdfs-servers in their conf we should fix this in the framework). Perhaps we should not default this config to "$fs.default.name"?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3736) Variable substitution depth too
large for fs.default.name causes jobs to fail
Posted by "Ahmed Radwan (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13194280#comment-13194280 ]
Ahmed Radwan commented on MAPREDUCE-3736:
-----------------------------------------
It seems that the property "fs.defaultFS" is now used instead of "fs.default.name"
So I think setting mapreduce.job.hdfs-servers to ${fs.defaultFS} instead of ${fs.default.name} in yarn-default.xml should solve this issue to all upstream projects. I'm running the failing tests again with this patch to confirm.
@Ravi What property needs to be in src/test/mapred-site.xml?
> Variable substitution depth too large for fs.default.name causes jobs to fail
> -----------------------------------------------------------------------------
>
> Key: MAPREDUCE-3736
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3736
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.1
> Reporter: Eli Collins
> Assignee: Ahmed Radwan
> Priority: Blocker
>
> I'm seeing the same failure as MAPREDUCE-3462 in downstream projects running against a recent build of branch-23. MR-3462 modified the tests rather than fixing the framework. In that jira Ravi mentioned "I'm still ignorant of the change which made the tests start to fail. I should probably understand better the reasons for that change before proposing a more generalized fix." Let's figure out the general fix (rather than require all projects to set mapreduce.job.hdfs-servers in their conf we should fix this in the framework). Perhaps we should not default this config to "$fs.default.name"?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3736) Variable substitution depth too
large for fs.default.name causes jobs to fail
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13208421#comment-13208421 ]
Hudson commented on MAPREDUCE-3736:
-----------------------------------
Integrated in Hadoop-Hdfs-trunk #956 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/956/])
MAPREDUCE-3736. Variable substitution depth too large for fs.default.name causes jobs to fail (ahmed via tucu) (Revision 1244264)
Result = FAILURE
tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1244264
Files :
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/resources/core-default.xml
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapred/TestMRWithDistributedCache.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/conf/TestNoDefaultsJobConf.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/fs/JHLogAnalyzer.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/io/FileBench.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestCombineFileInputFormat.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestConcatenatedCompressedInput.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestTextInputFormat.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/TestMapCollection.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/input/TestFileInputFormat.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/input/TestMRKeyValueTextInputFormat.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
> Variable substitution depth too large for fs.default.name causes jobs to fail
> -----------------------------------------------------------------------------
>
> Key: MAPREDUCE-3736
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3736
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.1
> Reporter: Eli Collins
> Assignee: Ahmed Radwan
> Priority: Blocker
> Attachments: MAPREDUCE-3736.patch, MAPREDUCE-3736_rev2.patch, MAPREDUCE-3736_rev3.patch
>
>
> I'm seeing the same failure as MAPREDUCE-3462 in downstream projects running against a recent build of branch-23. MR-3462 modified the tests rather than fixing the framework. In that jira Ravi mentioned "I'm still ignorant of the change which made the tests start to fail. I should probably understand better the reasons for that change before proposing a more generalized fix." Let's figure out the general fix (rather than require all projects to set mapreduce.job.hdfs-servers in their conf we should fix this in the framework). Perhaps we should not default this config to "$fs.default.name"?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3736) Variable substitution depth too
large for fs.default.name causes jobs to fail
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13208452#comment-13208452 ]
Hudson commented on MAPREDUCE-3736:
-----------------------------------
Integrated in Hadoop-Mapreduce-0.23-Build #197 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/197/])
Merge -r 1244263:1244264 from trunk to branch. FIXES: MAPREDUCE-3736 (Revision 1244265)
Result = FAILURE
tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1244265
Files :
* /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/resources/core-default.xml
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapred/TestMRWithDistributedCache.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/conf/TestNoDefaultsJobConf.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/fs/JHLogAnalyzer.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/io/FileBench.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestCombineFileInputFormat.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestConcatenatedCompressedInput.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestTextInputFormat.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/TestMapCollection.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/input/TestFileInputFormat.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/input/TestMRKeyValueTextInputFormat.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
> Variable substitution depth too large for fs.default.name causes jobs to fail
> -----------------------------------------------------------------------------
>
> Key: MAPREDUCE-3736
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3736
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.1
> Reporter: Eli Collins
> Assignee: Ahmed Radwan
> Priority: Blocker
> Attachments: MAPREDUCE-3736.patch, MAPREDUCE-3736_rev2.patch, MAPREDUCE-3736_rev3.patch
>
>
> I'm seeing the same failure as MAPREDUCE-3462 in downstream projects running against a recent build of branch-23. MR-3462 modified the tests rather than fixing the framework. In that jira Ravi mentioned "I'm still ignorant of the change which made the tests start to fail. I should probably understand better the reasons for that change before proposing a more generalized fix." Let's figure out the general fix (rather than require all projects to set mapreduce.job.hdfs-servers in their conf we should fix this in the framework). Perhaps we should not default this config to "$fs.default.name"?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3736) Variable substitution depth too
large for fs.default.name causes jobs to fail
Posted by "Eli Collins (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13195905#comment-13195905 ]
Eli Collins commented on MAPREDUCE-3736:
----------------------------------------
Hm, on 2nd thought using fs.defaultFS in mapreduce.job.hdfs-servers will mean people moving to a 23 release will have to update their configs to defaultFS or this feature will break. Seems like MR should be compatible with fs.default.name. Just updating core-default.xml to add back fs.default.name would fix the issue right? If so let's do that.
> Variable substitution depth too large for fs.default.name causes jobs to fail
> -----------------------------------------------------------------------------
>
> Key: MAPREDUCE-3736
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3736
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.1
> Reporter: Eli Collins
> Assignee: Ahmed Radwan
> Priority: Blocker
> Attachments: MAPREDUCE-3736.patch, MAPREDUCE-3736_rev2.patch
>
>
> I'm seeing the same failure as MAPREDUCE-3462 in downstream projects running against a recent build of branch-23. MR-3462 modified the tests rather than fixing the framework. In that jira Ravi mentioned "I'm still ignorant of the change which made the tests start to fail. I should probably understand better the reasons for that change before proposing a more generalized fix." Let's figure out the general fix (rather than require all projects to set mapreduce.job.hdfs-servers in their conf we should fix this in the framework). Perhaps we should not default this config to "$fs.default.name"?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3736) Variable substitution depth too
large for fs.default.name causes jobs to fail
Posted by "Ahmed Radwan (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13194169#comment-13194169 ]
Ahmed Radwan commented on MAPREDUCE-3736:
-----------------------------------------
I have checked for fs.default.name property and I didn't find it in any of *-default.xml.
In previous hadoop versions, it usually exists in core-default.xml:
<property>
<name>fs.default.name</name>
<value>file:///</value>
</property>
I'll create a patch to set this property in core-default.xml and test if this solves the problem.
> Variable substitution depth too large for fs.default.name causes jobs to fail
> -----------------------------------------------------------------------------
>
> Key: MAPREDUCE-3736
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3736
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.1
> Reporter: Eli Collins
> Assignee: Ahmed Radwan
> Priority: Blocker
>
> I'm seeing the same failure as MAPREDUCE-3462 in downstream projects running against a recent build of branch-23. MR-3462 modified the tests rather than fixing the framework. In that jira Ravi mentioned "I'm still ignorant of the change which made the tests start to fail. I should probably understand better the reasons for that change before proposing a more generalized fix." Let's figure out the general fix (rather than require all projects to set mapreduce.job.hdfs-servers in their conf we should fix this in the framework). Perhaps we should not default this config to "$fs.default.name"?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3736) Variable substitution depth too
large for fs.default.name causes jobs to fail
Posted by "Ahmed Radwan (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ahmed Radwan updated MAPREDUCE-3736:
------------------------------------
Assignee: Ahmed Radwan
> Variable substitution depth too large for fs.default.name causes jobs to fail
> -----------------------------------------------------------------------------
>
> Key: MAPREDUCE-3736
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3736
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.1
> Reporter: Eli Collins
> Assignee: Ahmed Radwan
> Priority: Blocker
>
> I'm seeing the same failure as MAPREDUCE-3462 in downstream projects running against a recent build of branch-23. MR-3462 modified the tests rather than fixing the framework. In that jira Ravi mentioned "I'm still ignorant of the change which made the tests start to fail. I should probably understand better the reasons for that change before proposing a more generalized fix." Let's figure out the general fix (rather than require all projects to set mapreduce.job.hdfs-servers in their conf we should fix this in the framework). Perhaps we should not default this config to "$fs.default.name"?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3736) Variable substitution depth too
large for fs.default.name causes jobs to fail
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13208134#comment-13208134 ]
Hudson commented on MAPREDUCE-3736:
-----------------------------------
Integrated in Hadoop-Hdfs-trunk-Commit #1802 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1802/])
MAPREDUCE-3736. Variable substitution depth too large for fs.default.name causes jobs to fail (ahmed via tucu) (Revision 1244264)
Result = SUCCESS
tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1244264
Files :
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/resources/core-default.xml
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapred/TestMRWithDistributedCache.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/conf/TestNoDefaultsJobConf.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/fs/JHLogAnalyzer.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/io/FileBench.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestCombineFileInputFormat.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestConcatenatedCompressedInput.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestTextInputFormat.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/TestMapCollection.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/input/TestFileInputFormat.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/input/TestMRKeyValueTextInputFormat.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
> Variable substitution depth too large for fs.default.name causes jobs to fail
> -----------------------------------------------------------------------------
>
> Key: MAPREDUCE-3736
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3736
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.1
> Reporter: Eli Collins
> Assignee: Ahmed Radwan
> Priority: Blocker
> Attachments: MAPREDUCE-3736.patch, MAPREDUCE-3736_rev2.patch, MAPREDUCE-3736_rev3.patch
>
>
> I'm seeing the same failure as MAPREDUCE-3462 in downstream projects running against a recent build of branch-23. MR-3462 modified the tests rather than fixing the framework. In that jira Ravi mentioned "I'm still ignorant of the change which made the tests start to fail. I should probably understand better the reasons for that change before proposing a more generalized fix." Let's figure out the general fix (rather than require all projects to set mapreduce.job.hdfs-servers in their conf we should fix this in the framework). Perhaps we should not default this config to "$fs.default.name"?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3736) Variable substitution depth too
large for fs.default.name causes jobs to fail
Posted by "Ahmed Radwan (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13196493#comment-13196493 ]
Ahmed Radwan commented on MAPREDUCE-3736:
-----------------------------------------
Sqoop build.xml sets:
<sysproperty key="fs.default.name" value="${fs.default.name}"/>
> Variable substitution depth too large for fs.default.name causes jobs to fail
> -----------------------------------------------------------------------------
>
> Key: MAPREDUCE-3736
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3736
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.1
> Reporter: Eli Collins
> Assignee: Ahmed Radwan
> Priority: Blocker
> Attachments: MAPREDUCE-3736.patch, MAPREDUCE-3736_rev2.patch
>
>
> I'm seeing the same failure as MAPREDUCE-3462 in downstream projects running against a recent build of branch-23. MR-3462 modified the tests rather than fixing the framework. In that jira Ravi mentioned "I'm still ignorant of the change which made the tests start to fail. I should probably understand better the reasons for that change before proposing a more generalized fix." Let's figure out the general fix (rather than require all projects to set mapreduce.job.hdfs-servers in their conf we should fix this in the framework). Perhaps we should not default this config to "$fs.default.name"?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3736) Variable substitution depth too
large for fs.default.name causes jobs to fail
Posted by "Ahmed Radwan (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13196513#comment-13196513 ]
Ahmed Radwan commented on MAPREDUCE-3736:
-----------------------------------------
No, I mean it describes and proposes a possible solution for such cyclic evaluation of variables causing "Variable substitution depth too large" problem.
> Variable substitution depth too large for fs.default.name causes jobs to fail
> -----------------------------------------------------------------------------
>
> Key: MAPREDUCE-3736
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3736
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.1
> Reporter: Eli Collins
> Assignee: Ahmed Radwan
> Priority: Blocker
> Attachments: MAPREDUCE-3736.patch, MAPREDUCE-3736_rev2.patch
>
>
> I'm seeing the same failure as MAPREDUCE-3462 in downstream projects running against a recent build of branch-23. MR-3462 modified the tests rather than fixing the framework. In that jira Ravi mentioned "I'm still ignorant of the change which made the tests start to fail. I should probably understand better the reasons for that change before proposing a more generalized fix." Let's figure out the general fix (rather than require all projects to set mapreduce.job.hdfs-servers in their conf we should fix this in the framework). Perhaps we should not default this config to "$fs.default.name"?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3736) Variable substitution depth too
large for fs.default.name causes jobs to fail
Posted by "Mahadev konar (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13196414#comment-13196414 ]
Mahadev konar commented on MAPREDUCE-3736:
------------------------------------------
Sorry, I meant MAPREDUCE-3758 and not MAPREDUCE-3696.
> Variable substitution depth too large for fs.default.name causes jobs to fail
> -----------------------------------------------------------------------------
>
> Key: MAPREDUCE-3736
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3736
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.1
> Reporter: Eli Collins
> Assignee: Ahmed Radwan
> Priority: Blocker
> Attachments: MAPREDUCE-3736.patch, MAPREDUCE-3736_rev2.patch
>
>
> I'm seeing the same failure as MAPREDUCE-3462 in downstream projects running against a recent build of branch-23. MR-3462 modified the tests rather than fixing the framework. In that jira Ravi mentioned "I'm still ignorant of the change which made the tests start to fail. I should probably understand better the reasons for that change before proposing a more generalized fix." Let's figure out the general fix (rather than require all projects to set mapreduce.job.hdfs-servers in their conf we should fix this in the framework). Perhaps we should not default this config to "$fs.default.name"?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3736) Variable substitution depth too
large for fs.default.name causes jobs to fail
Posted by "John George (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13196411#comment-13196411 ]
John George commented on MAPREDUCE-3736:
----------------------------------------
I filed MAPREDUCE-3758, but the real reason seems very similar to what is being discussed here. The value of APPLICATION_CLASSPATH seems to be in yarn-default.xml but does not seem to be loaded.
> Variable substitution depth too large for fs.default.name causes jobs to fail
> -----------------------------------------------------------------------------
>
> Key: MAPREDUCE-3736
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3736
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.1
> Reporter: Eli Collins
> Assignee: Ahmed Radwan
> Priority: Blocker
> Attachments: MAPREDUCE-3736.patch, MAPREDUCE-3736_rev2.patch
>
>
> I'm seeing the same failure as MAPREDUCE-3462 in downstream projects running against a recent build of branch-23. MR-3462 modified the tests rather than fixing the framework. In that jira Ravi mentioned "I'm still ignorant of the change which made the tests start to fail. I should probably understand better the reasons for that change before proposing a more generalized fix." Let's figure out the general fix (rather than require all projects to set mapreduce.job.hdfs-servers in their conf we should fix this in the framework). Perhaps we should not default this config to "$fs.default.name"?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3736) Variable substitution depth too
large for fs.default.name causes jobs to fail
Posted by "Eli Collins (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13194281#comment-13194281 ]
Eli Collins commented on MAPREDUCE-3736:
----------------------------------------
Ahmed and I investigated this some, it looks like core-default.xml has switched from fs.default.name to fs.defaultFS however MR is still using fs.default.name which is now undefined. Seems like we should either update MR to use fs.defaultFS or define both fs.default.name and fs.defaultFS in core-site.xml
> Variable substitution depth too large for fs.default.name causes jobs to fail
> -----------------------------------------------------------------------------
>
> Key: MAPREDUCE-3736
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3736
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.1
> Reporter: Eli Collins
> Assignee: Ahmed Radwan
> Priority: Blocker
>
> I'm seeing the same failure as MAPREDUCE-3462 in downstream projects running against a recent build of branch-23. MR-3462 modified the tests rather than fixing the framework. In that jira Ravi mentioned "I'm still ignorant of the change which made the tests start to fail. I should probably understand better the reasons for that change before proposing a more generalized fix." Let's figure out the general fix (rather than require all projects to set mapreduce.job.hdfs-servers in their conf we should fix this in the framework). Perhaps we should not default this config to "$fs.default.name"?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3736) Variable substitution depth too
large for fs.default.name causes jobs to fail
Posted by "Mahadev konar (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13196524#comment-13196524 ]
Mahadev konar commented on MAPREDUCE-3736:
------------------------------------------
Sorry, I should have looked at the jira. I was more worried abt the defaults not getting loaded as mentioned in MAPREDUCE-3758.
> Variable substitution depth too large for fs.default.name causes jobs to fail
> -----------------------------------------------------------------------------
>
> Key: MAPREDUCE-3736
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3736
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.1
> Reporter: Eli Collins
> Assignee: Ahmed Radwan
> Priority: Blocker
> Attachments: MAPREDUCE-3736.patch, MAPREDUCE-3736_rev2.patch
>
>
> I'm seeing the same failure as MAPREDUCE-3462 in downstream projects running against a recent build of branch-23. MR-3462 modified the tests rather than fixing the framework. In that jira Ravi mentioned "I'm still ignorant of the change which made the tests start to fail. I should probably understand better the reasons for that change before proposing a more generalized fix." Let's figure out the general fix (rather than require all projects to set mapreduce.job.hdfs-servers in their conf we should fix this in the framework). Perhaps we should not default this config to "$fs.default.name"?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3736) Variable substitution depth too
large for fs.default.name causes jobs to fail
Posted by "Eli Collins (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13196440#comment-13196440 ]
Eli Collins commented on MAPREDUCE-3736:
----------------------------------------
We hit this issue in a build before HADOOP-7993 was introduced.
> Variable substitution depth too large for fs.default.name causes jobs to fail
> -----------------------------------------------------------------------------
>
> Key: MAPREDUCE-3736
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3736
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.1
> Reporter: Eli Collins
> Assignee: Ahmed Radwan
> Priority: Blocker
> Attachments: MAPREDUCE-3736.patch, MAPREDUCE-3736_rev2.patch
>
>
> I'm seeing the same failure as MAPREDUCE-3462 in downstream projects running against a recent build of branch-23. MR-3462 modified the tests rather than fixing the framework. In that jira Ravi mentioned "I'm still ignorant of the change which made the tests start to fail. I should probably understand better the reasons for that change before proposing a more generalized fix." Let's figure out the general fix (rather than require all projects to set mapreduce.job.hdfs-servers in their conf we should fix this in the framework). Perhaps we should not default this config to "$fs.default.name"?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3736) Variable substitution depth too
large for fs.default.name causes jobs to fail
Posted by "Eli Collins (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13195907#comment-13195907 ]
Eli Collins commented on MAPREDUCE-3736:
----------------------------------------
Actually when fs.default.name was removed from core-site by HADOOP-6233 a deprecation was added so it would get mapped to fs.defaultFS, so updating MR to use fs.defaultFS should be fine, if defaultFS is not defined it should do the lookup with fs.default.name.
However, if doing a lookup on fs.default.name (as MR does currently) and it finds that this key has no value I think it's *supposed* to do a lookup using the new key (fs.defaultFS), so if HADOOP-6233 was done correctly as I understand it this change shouldn't be necessary, eg a lookup to fs.default.name should map to a lookup to fs.defaultFS (if fs.default.name was not defined) and we do have a value for it defined in core-default.xml. Seems like either the configuration deprecation is broken or something else going on here (eg in YarnConfiguration somehow fs.default.name is getting defined or overwritten).
> Variable substitution depth too large for fs.default.name causes jobs to fail
> -----------------------------------------------------------------------------
>
> Key: MAPREDUCE-3736
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3736
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.1
> Reporter: Eli Collins
> Assignee: Ahmed Radwan
> Priority: Blocker
> Attachments: MAPREDUCE-3736.patch, MAPREDUCE-3736_rev2.patch
>
>
> I'm seeing the same failure as MAPREDUCE-3462 in downstream projects running against a recent build of branch-23. MR-3462 modified the tests rather than fixing the framework. In that jira Ravi mentioned "I'm still ignorant of the change which made the tests start to fail. I should probably understand better the reasons for that change before proposing a more generalized fix." Let's figure out the general fix (rather than require all projects to set mapreduce.job.hdfs-servers in their conf we should fix this in the framework). Perhaps we should not default this config to "$fs.default.name"?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3736) Variable substitution depth too
large for fs.default.name causes jobs to fail
Posted by "Ravi Prakash (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13194270#comment-13194270 ]
Ravi Prakash commented on MAPREDUCE-3736:
-----------------------------------------
Just to make things more explicit. Amar found (https://issues.apache.org/jira/browse/MAPREDUCE-3462?focusedCommentId=13175250&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13175250)
bq. I tried setting this property in src/test/mapred-site.xml but the test still failed. Somehow, we should make sure that the contrib tests load the src/test/mapred-site.xml.
This is a pretty basic property and I don't think we can do without it. So even for tests it needs to be defined. I could do a git bisect to see what changed, but mehhh.....
{noformat}
_______________________
< I'm too lazy for that >
-----------------------
\ ^__^
\ (oo)\_______
(__)\ )\/\
||----w |
|| ||
{noformat}
> Variable substitution depth too large for fs.default.name causes jobs to fail
> -----------------------------------------------------------------------------
>
> Key: MAPREDUCE-3736
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3736
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.1
> Reporter: Eli Collins
> Assignee: Ahmed Radwan
> Priority: Blocker
>
> I'm seeing the same failure as MAPREDUCE-3462 in downstream projects running against a recent build of branch-23. MR-3462 modified the tests rather than fixing the framework. In that jira Ravi mentioned "I'm still ignorant of the change which made the tests start to fail. I should probably understand better the reasons for that change before proposing a more generalized fix." Let's figure out the general fix (rather than require all projects to set mapreduce.job.hdfs-servers in their conf we should fix this in the framework). Perhaps we should not default this config to "$fs.default.name"?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3736) Variable substitution depth too
large for fs.default.name causes jobs to fail
Posted by "Ahmed Radwan (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13195227#comment-13195227 ]
Ahmed Radwan commented on MAPREDUCE-3736:
-----------------------------------------
bq. why do we need to add fs.default.name back to core-default.xml
We don't need to, I just added it for backward compatibility, in case other dependent projects are still using it. I added (in the description) that it is deprecated, so fs.defaultFS will be used instead in the future.
bq. is the mapreduce.job.hdfs-servers change sufficient or do they also require fs.default.name be set?
Just setting mapreduce.job.hdfs-servers to fs.defaultFS is sufficient to address the failures described in this bug.
> Variable substitution depth too large for fs.default.name causes jobs to fail
> -----------------------------------------------------------------------------
>
> Key: MAPREDUCE-3736
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3736
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.1
> Reporter: Eli Collins
> Assignee: Ahmed Radwan
> Priority: Blocker
> Attachments: MAPREDUCE-3736.patch, MAPREDUCE-3736_rev2.patch
>
>
> I'm seeing the same failure as MAPREDUCE-3462 in downstream projects running against a recent build of branch-23. MR-3462 modified the tests rather than fixing the framework. In that jira Ravi mentioned "I'm still ignorant of the change which made the tests start to fail. I should probably understand better the reasons for that change before proposing a more generalized fix." Let's figure out the general fix (rather than require all projects to set mapreduce.job.hdfs-servers in their conf we should fix this in the framework). Perhaps we should not default this config to "$fs.default.name"?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3736) Variable substitution depth too
large for fs.default.name causes jobs to fail
Posted by "Ahmed Radwan (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ahmed Radwan updated MAPREDUCE-3736:
------------------------------------
Attachment: MAPREDUCE-3736.patch
Attaching the patch, confirmed that it fixes the issue for the sqoop tests.
> Variable substitution depth too large for fs.default.name causes jobs to fail
> -----------------------------------------------------------------------------
>
> Key: MAPREDUCE-3736
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3736
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.1
> Reporter: Eli Collins
> Assignee: Ahmed Radwan
> Priority: Blocker
> Attachments: MAPREDUCE-3736.patch
>
>
> I'm seeing the same failure as MAPREDUCE-3462 in downstream projects running against a recent build of branch-23. MR-3462 modified the tests rather than fixing the framework. In that jira Ravi mentioned "I'm still ignorant of the change which made the tests start to fail. I should probably understand better the reasons for that change before proposing a more generalized fix." Let's figure out the general fix (rather than require all projects to set mapreduce.job.hdfs-servers in their conf we should fix this in the framework). Perhaps we should not default this config to "$fs.default.name"?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3736) Variable substitution depth too
large for fs.default.name causes jobs to fail
Posted by "Alejandro Abdelnur (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13208100#comment-13208100 ]
Alejandro Abdelnur commented on MAPREDUCE-3736:
-----------------------------------------------
+1
> Variable substitution depth too large for fs.default.name causes jobs to fail
> -----------------------------------------------------------------------------
>
> Key: MAPREDUCE-3736
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3736
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.1
> Reporter: Eli Collins
> Assignee: Ahmed Radwan
> Priority: Blocker
> Attachments: MAPREDUCE-3736.patch, MAPREDUCE-3736_rev2.patch, MAPREDUCE-3736_rev3.patch
>
>
> I'm seeing the same failure as MAPREDUCE-3462 in downstream projects running against a recent build of branch-23. MR-3462 modified the tests rather than fixing the framework. In that jira Ravi mentioned "I'm still ignorant of the change which made the tests start to fail. I should probably understand better the reasons for that change before proposing a more generalized fix." Let's figure out the general fix (rather than require all projects to set mapreduce.job.hdfs-servers in their conf we should fix this in the framework). Perhaps we should not default this config to "$fs.default.name"?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3736) Variable substitution depth too
large for fs.default.name causes jobs to fail
Posted by "Alejandro Abdelnur (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alejandro Abdelnur updated MAPREDUCE-3736:
------------------------------------------
Resolution: Fixed
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Thanks Ahmed. Committed to trunk and branch-0.23
> Variable substitution depth too large for fs.default.name causes jobs to fail
> -----------------------------------------------------------------------------
>
> Key: MAPREDUCE-3736
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3736
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.1
> Reporter: Eli Collins
> Assignee: Ahmed Radwan
> Priority: Blocker
> Attachments: MAPREDUCE-3736.patch, MAPREDUCE-3736_rev2.patch, MAPREDUCE-3736_rev3.patch
>
>
> I'm seeing the same failure as MAPREDUCE-3462 in downstream projects running against a recent build of branch-23. MR-3462 modified the tests rather than fixing the framework. In that jira Ravi mentioned "I'm still ignorant of the change which made the tests start to fail. I should probably understand better the reasons for that change before proposing a more generalized fix." Let's figure out the general fix (rather than require all projects to set mapreduce.job.hdfs-servers in their conf we should fix this in the framework). Perhaps we should not default this config to "$fs.default.name"?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3736) Variable substitution depth too
large for fs.default.name causes jobs to fail
Posted by "Ahmed Radwan (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ahmed Radwan updated MAPREDUCE-3736:
------------------------------------
Attachment: MAPREDUCE-3736_rev3.patch
Updated patch, since location of yarn-default.xml was changed.
> Variable substitution depth too large for fs.default.name causes jobs to fail
> -----------------------------------------------------------------------------
>
> Key: MAPREDUCE-3736
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3736
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.1
> Reporter: Eli Collins
> Assignee: Ahmed Radwan
> Priority: Blocker
> Attachments: MAPREDUCE-3736.patch, MAPREDUCE-3736_rev2.patch, MAPREDUCE-3736_rev3.patch
>
>
> I'm seeing the same failure as MAPREDUCE-3462 in downstream projects running against a recent build of branch-23. MR-3462 modified the tests rather than fixing the framework. In that jira Ravi mentioned "I'm still ignorant of the change which made the tests start to fail. I should probably understand better the reasons for that change before proposing a more generalized fix." Let's figure out the general fix (rather than require all projects to set mapreduce.job.hdfs-servers in their conf we should fix this in the framework). Perhaps we should not default this config to "$fs.default.name"?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3736) Variable substitution depth too
large for fs.default.name causes jobs to fail
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13208471#comment-13208471 ]
Hudson commented on MAPREDUCE-3736:
-----------------------------------
Integrated in Hadoop-Mapreduce-trunk #991 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/991/])
MAPREDUCE-3736. Variable substitution depth too large for fs.default.name causes jobs to fail (ahmed via tucu) (Revision 1244264)
Result = SUCCESS
tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1244264
Files :
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/resources/core-default.xml
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapred/TestMRWithDistributedCache.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/conf/TestNoDefaultsJobConf.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/fs/JHLogAnalyzer.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/io/FileBench.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestCombineFileInputFormat.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestConcatenatedCompressedInput.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestTextInputFormat.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/TestMapCollection.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/input/TestFileInputFormat.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/input/TestMRKeyValueTextInputFormat.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
> Variable substitution depth too large for fs.default.name causes jobs to fail
> -----------------------------------------------------------------------------
>
> Key: MAPREDUCE-3736
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3736
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.1
> Reporter: Eli Collins
> Assignee: Ahmed Radwan
> Priority: Blocker
> Attachments: MAPREDUCE-3736.patch, MAPREDUCE-3736_rev2.patch, MAPREDUCE-3736_rev3.patch
>
>
> I'm seeing the same failure as MAPREDUCE-3462 in downstream projects running against a recent build of branch-23. MR-3462 modified the tests rather than fixing the framework. In that jira Ravi mentioned "I'm still ignorant of the change which made the tests start to fail. I should probably understand better the reasons for that change before proposing a more generalized fix." Let's figure out the general fix (rather than require all projects to set mapreduce.job.hdfs-servers in their conf we should fix this in the framework). Perhaps we should not default this config to "$fs.default.name"?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3736) Variable substitution depth too
large for fs.default.name causes jobs to fail
Posted by "Eli Collins (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13194064#comment-13194064 ]
Eli Collins commented on MAPREDUCE-3736:
----------------------------------------
Here's the stacktrace from Sqoop:
{noformat}
java.lang.IllegalStateException: Variable substitution depth too large: 20 ${fs.default.name}
at org.apache.hadoop.conf.Configuration.substituteVars(Configuration.java:551)
at org.apache.hadoop.conf.Configuration.get(Configuration.java:569)
at org.apache.hadoop.conf.Configuration.getStrings(Configuration.java:1047)
at org.apache.hadoop.mapreduce.JobSubmitter.populateTokenCache(JobSubmitter.java:564)
at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:353)
at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1215)
at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1212)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1167)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:1212)
at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1233)
at org.apache.sqoop.mapreduce.ImportJobBase.runJob(ImportJobBase.java:119)
at org.apache.sqoop.mapreduce.ImportJobBase.runImport(ImportJobBase.java:179)
at org.apache.sqoop.manager.SqlManager.importTable(SqlManager.java:413)
at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:380)
at org.apache.sqoop.tool.ImportAllTablesTool.run(ImportAllTablesTool.java:64)
at org.apache.sqoop.Sqoop.run(Sqoop.java:145)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:181)
{noformat}
> Variable substitution depth too large for fs.default.name causes jobs to fail
> -----------------------------------------------------------------------------
>
> Key: MAPREDUCE-3736
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3736
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.1
> Reporter: Eli Collins
> Priority: Blocker
>
> I'm seeing the same failure as MAPREDUCE-3462 in downstream projects running against a recent build of branch-23. MR-3462 modified the tests rather than fixing the framework. In that jira Ravi mentioned "I'm still ignorant of the change which made the tests start to fail. I should probably understand better the reasons for that change before proposing a more generalized fix." Let's figure out the general fix (rather than require all projects to set mapreduce.job.hdfs-servers in their conf we should fix this in the framework). Perhaps we should not default this config to "$fs.default.name"?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3736) Variable substitution depth too
large for fs.default.name causes jobs to fail
Posted by "Ravi Prakash (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13196222#comment-13196222 ]
Ravi Prakash commented on MAPREDUCE-3736:
-----------------------------------------
@Eli: I think Amar determined that the src/test/core-default.xml's values were not being used. I first thought it was only for gridmix unit tests, but the problem might be more pervasive.
Can we check that the values defined in src/test/core-default.xml are actually being read and used? Maybe all that mavenization messed things up.
> Variable substitution depth too large for fs.default.name causes jobs to fail
> -----------------------------------------------------------------------------
>
> Key: MAPREDUCE-3736
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3736
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.1
> Reporter: Eli Collins
> Assignee: Ahmed Radwan
> Priority: Blocker
> Attachments: MAPREDUCE-3736.patch, MAPREDUCE-3736_rev2.patch
>
>
> I'm seeing the same failure as MAPREDUCE-3462 in downstream projects running against a recent build of branch-23. MR-3462 modified the tests rather than fixing the framework. In that jira Ravi mentioned "I'm still ignorant of the change which made the tests start to fail. I should probably understand better the reasons for that change before proposing a more generalized fix." Let's figure out the general fix (rather than require all projects to set mapreduce.job.hdfs-servers in their conf we should fix this in the framework). Perhaps we should not default this config to "$fs.default.name"?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3736) Variable substitution depth too
large for fs.default.name causes jobs to fail
Posted by "Mahadev konar (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13196413#comment-13196413 ]
Mahadev konar commented on MAPREDUCE-3736:
------------------------------------------
@Eli/Ahmed,
It looks like this is causing a lot more issues. I think MAPREDUCE-3696 is also related to this. Any update on why this is happening? I'd suspect HADOOP-7993 to be the most risky one for this.
> Variable substitution depth too large for fs.default.name causes jobs to fail
> -----------------------------------------------------------------------------
>
> Key: MAPREDUCE-3736
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3736
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.1
> Reporter: Eli Collins
> Assignee: Ahmed Radwan
> Priority: Blocker
> Attachments: MAPREDUCE-3736.patch, MAPREDUCE-3736_rev2.patch
>
>
> I'm seeing the same failure as MAPREDUCE-3462 in downstream projects running against a recent build of branch-23. MR-3462 modified the tests rather than fixing the framework. In that jira Ravi mentioned "I'm still ignorant of the change which made the tests start to fail. I should probably understand better the reasons for that change before proposing a more generalized fix." Let's figure out the general fix (rather than require all projects to set mapreduce.job.hdfs-servers in their conf we should fix this in the framework). Perhaps we should not default this config to "$fs.default.name"?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3736) Variable substitution depth too
large for fs.default.name causes jobs to fail
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13208327#comment-13208327 ]
Hudson commented on MAPREDUCE-3736:
-----------------------------------
Integrated in Hadoop-Mapreduce-trunk-Commit #1740 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1740/])
MAPREDUCE-3736. Variable substitution depth too large for fs.default.name causes jobs to fail (ahmed via tucu) (Revision 1244264)
Result = ABORTED
tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1244264
Files :
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/resources/core-default.xml
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapred/TestMRWithDistributedCache.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/conf/TestNoDefaultsJobConf.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/fs/JHLogAnalyzer.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/io/FileBench.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestCombineFileInputFormat.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestConcatenatedCompressedInput.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestTextInputFormat.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/TestMapCollection.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/input/TestFileInputFormat.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/input/TestMRKeyValueTextInputFormat.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
> Variable substitution depth too large for fs.default.name causes jobs to fail
> -----------------------------------------------------------------------------
>
> Key: MAPREDUCE-3736
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3736
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.1
> Reporter: Eli Collins
> Assignee: Ahmed Radwan
> Priority: Blocker
> Attachments: MAPREDUCE-3736.patch, MAPREDUCE-3736_rev2.patch, MAPREDUCE-3736_rev3.patch
>
>
> I'm seeing the same failure as MAPREDUCE-3462 in downstream projects running against a recent build of branch-23. MR-3462 modified the tests rather than fixing the framework. In that jira Ravi mentioned "I'm still ignorant of the change which made the tests start to fail. I should probably understand better the reasons for that change before proposing a more generalized fix." Let's figure out the general fix (rather than require all projects to set mapreduce.job.hdfs-servers in their conf we should fix this in the framework). Perhaps we should not default this config to "$fs.default.name"?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3736) Variable substitution depth too
large for fs.default.name causes jobs to fail
Posted by "Mahadev konar (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13199386#comment-13199386 ]
Mahadev konar commented on MAPREDUCE-3736:
------------------------------------------
@Ahmed,
Any update on this?
> Variable substitution depth too large for fs.default.name causes jobs to fail
> -----------------------------------------------------------------------------
>
> Key: MAPREDUCE-3736
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3736
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.1
> Reporter: Eli Collins
> Assignee: Ahmed Radwan
> Priority: Blocker
> Attachments: MAPREDUCE-3736.patch, MAPREDUCE-3736_rev2.patch
>
>
> I'm seeing the same failure as MAPREDUCE-3462 in downstream projects running against a recent build of branch-23. MR-3462 modified the tests rather than fixing the framework. In that jira Ravi mentioned "I'm still ignorant of the change which made the tests start to fail. I should probably understand better the reasons for that change before proposing a more generalized fix." Let's figure out the general fix (rather than require all projects to set mapreduce.job.hdfs-servers in their conf we should fix this in the framework). Perhaps we should not default this config to "$fs.default.name"?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3736) Variable substitution depth too
large for fs.default.name causes jobs to fail
Posted by "Ahmed Radwan (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ahmed Radwan updated MAPREDUCE-3736:
------------------------------------
Status: Patch Available (was: Open)
> Variable substitution depth too large for fs.default.name causes jobs to fail
> -----------------------------------------------------------------------------
>
> Key: MAPREDUCE-3736
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3736
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.1
> Reporter: Eli Collins
> Assignee: Ahmed Radwan
> Priority: Blocker
> Attachments: MAPREDUCE-3736.patch
>
>
> I'm seeing the same failure as MAPREDUCE-3462 in downstream projects running against a recent build of branch-23. MR-3462 modified the tests rather than fixing the framework. In that jira Ravi mentioned "I'm still ignorant of the change which made the tests start to fail. I should probably understand better the reasons for that change before proposing a more generalized fix." Let's figure out the general fix (rather than require all projects to set mapreduce.job.hdfs-servers in their conf we should fix this in the framework). Perhaps we should not default this config to "$fs.default.name"?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3736) Variable substitution depth too
large for fs.default.name causes jobs to fail
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13208325#comment-13208325 ]
Hudson commented on MAPREDUCE-3736:
-----------------------------------
Integrated in Hadoop-Mapreduce-0.23-Commit #556 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/556/])
Merge -r 1244263:1244264 from trunk to branch. FIXES: MAPREDUCE-3736 (Revision 1244265)
Result = ABORTED
tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1244265
Files :
* /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/resources/core-default.xml
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapred/TestMRWithDistributedCache.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/conf/TestNoDefaultsJobConf.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/fs/JHLogAnalyzer.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/io/FileBench.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestCombineFileInputFormat.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestConcatenatedCompressedInput.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestTextInputFormat.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/TestMapCollection.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/input/TestFileInputFormat.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/input/TestMRKeyValueTextInputFormat.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
> Variable substitution depth too large for fs.default.name causes jobs to fail
> -----------------------------------------------------------------------------
>
> Key: MAPREDUCE-3736
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3736
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.1
> Reporter: Eli Collins
> Assignee: Ahmed Radwan
> Priority: Blocker
> Attachments: MAPREDUCE-3736.patch, MAPREDUCE-3736_rev2.patch, MAPREDUCE-3736_rev3.patch
>
>
> I'm seeing the same failure as MAPREDUCE-3462 in downstream projects running against a recent build of branch-23. MR-3462 modified the tests rather than fixing the framework. In that jira Ravi mentioned "I'm still ignorant of the change which made the tests start to fail. I should probably understand better the reasons for that change before proposing a more generalized fix." Let's figure out the general fix (rather than require all projects to set mapreduce.job.hdfs-servers in their conf we should fix this in the framework). Perhaps we should not default this config to "$fs.default.name"?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3736) Variable substitution depth too
large for fs.default.name causes jobs to fail
Posted by "Eli Collins (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13195017#comment-13195017 ]
Eli Collins commented on MAPREDUCE-3736:
----------------------------------------
Why add fs.default.name to core-site.xml? Also, don't we want to update all MR uses to use fs.defaultFS, not just mapreduce.job.hdfs-servers?
> Variable substitution depth too large for fs.default.name causes jobs to fail
> -----------------------------------------------------------------------------
>
> Key: MAPREDUCE-3736
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3736
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.1
> Reporter: Eli Collins
> Assignee: Ahmed Radwan
> Priority: Blocker
> Attachments: MAPREDUCE-3736.patch
>
>
> I'm seeing the same failure as MAPREDUCE-3462 in downstream projects running against a recent build of branch-23. MR-3462 modified the tests rather than fixing the framework. In that jira Ravi mentioned "I'm still ignorant of the change which made the tests start to fail. I should probably understand better the reasons for that change before proposing a more generalized fix." Let's figure out the general fix (rather than require all projects to set mapreduce.job.hdfs-servers in their conf we should fix this in the framework). Perhaps we should not default this config to "$fs.default.name"?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3736) Variable substitution depth too
large for fs.default.name causes jobs to fail
Posted by "Eli Collins (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13194284#comment-13194284 ]
Eli Collins commented on MAPREDUCE-3736:
----------------------------------------
We'll probably need to set both "fs.defaultFS" as well as "fs.default.name" and both to "file:///" otherwise when the default filesystem is read to get the current working directory it read fs.defaultFS and finds HDFS, not the local FS.
> Variable substitution depth too large for fs.default.name causes jobs to fail
> -----------------------------------------------------------------------------
>
> Key: MAPREDUCE-3736
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3736
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.1
> Reporter: Eli Collins
> Assignee: Ahmed Radwan
> Priority: Blocker
>
> I'm seeing the same failure as MAPREDUCE-3462 in downstream projects running against a recent build of branch-23. MR-3462 modified the tests rather than fixing the framework. In that jira Ravi mentioned "I'm still ignorant of the change which made the tests start to fail. I should probably understand better the reasons for that change before proposing a more generalized fix." Let's figure out the general fix (rather than require all projects to set mapreduce.job.hdfs-servers in their conf we should fix this in the framework). Perhaps we should not default this config to "$fs.default.name"?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3736) Variable substitution depth too
large for fs.default.name causes jobs to fail
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13208301#comment-13208301 ]
Hudson commented on MAPREDUCE-3736:
-----------------------------------
Integrated in Hadoop-Common-0.23-Commit #552 (See [https://builds.apache.org/job/Hadoop-Common-0.23-Commit/552/])
Merge -r 1244263:1244264 from trunk to branch. FIXES: MAPREDUCE-3736 (Revision 1244265)
Result = SUCCESS
tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1244265
Files :
* /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/resources/core-default.xml
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapred/TestMRWithDistributedCache.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/conf/TestNoDefaultsJobConf.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/fs/JHLogAnalyzer.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/io/FileBench.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestCombineFileInputFormat.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestConcatenatedCompressedInput.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestTextInputFormat.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/TestMapCollection.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/input/TestFileInputFormat.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/input/TestMRKeyValueTextInputFormat.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
> Variable substitution depth too large for fs.default.name causes jobs to fail
> -----------------------------------------------------------------------------
>
> Key: MAPREDUCE-3736
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3736
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.1
> Reporter: Eli Collins
> Assignee: Ahmed Radwan
> Priority: Blocker
> Attachments: MAPREDUCE-3736.patch, MAPREDUCE-3736_rev2.patch, MAPREDUCE-3736_rev3.patch
>
>
> I'm seeing the same failure as MAPREDUCE-3462 in downstream projects running against a recent build of branch-23. MR-3462 modified the tests rather than fixing the framework. In that jira Ravi mentioned "I'm still ignorant of the change which made the tests start to fail. I should probably understand better the reasons for that change before proposing a more generalized fix." Let's figure out the general fix (rather than require all projects to set mapreduce.job.hdfs-servers in their conf we should fix this in the framework). Perhaps we should not default this config to "$fs.default.name"?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3736) Variable substitution depth too
large for fs.default.name causes jobs to fail
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13208147#comment-13208147 ]
Hudson commented on MAPREDUCE-3736:
-----------------------------------
Integrated in Hadoop-Hdfs-0.23-Commit #539 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/539/])
Merge -r 1244263:1244264 from trunk to branch. FIXES: MAPREDUCE-3736 (Revision 1244265)
Result = SUCCESS
tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1244265
Files :
* /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/resources/core-default.xml
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapred/TestMRWithDistributedCache.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/conf/TestNoDefaultsJobConf.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/fs/JHLogAnalyzer.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/io/FileBench.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestCombineFileInputFormat.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestConcatenatedCompressedInput.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestTextInputFormat.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/TestMapCollection.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/input/TestFileInputFormat.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/input/TestMRKeyValueTextInputFormat.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
> Variable substitution depth too large for fs.default.name causes jobs to fail
> -----------------------------------------------------------------------------
>
> Key: MAPREDUCE-3736
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3736
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.1
> Reporter: Eli Collins
> Assignee: Ahmed Radwan
> Priority: Blocker
> Attachments: MAPREDUCE-3736.patch, MAPREDUCE-3736_rev2.patch, MAPREDUCE-3736_rev3.patch
>
>
> I'm seeing the same failure as MAPREDUCE-3462 in downstream projects running against a recent build of branch-23. MR-3462 modified the tests rather than fixing the framework. In that jira Ravi mentioned "I'm still ignorant of the change which made the tests start to fail. I should probably understand better the reasons for that change before proposing a more generalized fix." Let's figure out the general fix (rather than require all projects to set mapreduce.job.hdfs-servers in their conf we should fix this in the framework). Perhaps we should not default this config to "$fs.default.name"?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3736) Variable substitution depth too
large for fs.default.name causes jobs to fail
Posted by "Mahadev konar (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13196509#comment-13196509 ]
Mahadev konar commented on MAPREDUCE-3736:
------------------------------------------
@Ahmed,
When you say HADOOP-6871 is relevant, you mean its causing the issue?
> Variable substitution depth too large for fs.default.name causes jobs to fail
> -----------------------------------------------------------------------------
>
> Key: MAPREDUCE-3736
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3736
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.1
> Reporter: Eli Collins
> Assignee: Ahmed Radwan
> Priority: Blocker
> Attachments: MAPREDUCE-3736.patch, MAPREDUCE-3736_rev2.patch
>
>
> I'm seeing the same failure as MAPREDUCE-3462 in downstream projects running against a recent build of branch-23. MR-3462 modified the tests rather than fixing the framework. In that jira Ravi mentioned "I'm still ignorant of the change which made the tests start to fail. I should probably understand better the reasons for that change before proposing a more generalized fix." Let's figure out the general fix (rather than require all projects to set mapreduce.job.hdfs-servers in their conf we should fix this in the framework). Perhaps we should not default this config to "$fs.default.name"?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3736) Variable substitution depth too
large for fs.default.name causes jobs to fail
Posted by "Eli Collins (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13196305#comment-13196305 ]
Eli Collins commented on MAPREDUCE-3736:
----------------------------------------
Sounds like a good idea. This was working on branch 23 until recently. I suspect this could be fallout from one of these changes:
MAPREDUCE-3717. JobClient test jar has missing files to run all the test programs (mahadev)
MAPREDUCE-3712. The mapreduce tar does not contain the hadoop-mapreduce-client-jobclient-tests.jar. (mahadev)
MAPREDUCE-3505. yarn APPLICATION_CLASSPATH needs to be overridable. (ahmed via tucu)
> Variable substitution depth too large for fs.default.name causes jobs to fail
> -----------------------------------------------------------------------------
>
> Key: MAPREDUCE-3736
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3736
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.1
> Reporter: Eli Collins
> Assignee: Ahmed Radwan
> Priority: Blocker
> Attachments: MAPREDUCE-3736.patch, MAPREDUCE-3736_rev2.patch
>
>
> I'm seeing the same failure as MAPREDUCE-3462 in downstream projects running against a recent build of branch-23. MR-3462 modified the tests rather than fixing the framework. In that jira Ravi mentioned "I'm still ignorant of the change which made the tests start to fail. I should probably understand better the reasons for that change before proposing a more generalized fix." Let's figure out the general fix (rather than require all projects to set mapreduce.job.hdfs-servers in their conf we should fix this in the framework). Perhaps we should not default this config to "$fs.default.name"?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3736) Variable substitution depth too
large for fs.default.name causes jobs to fail
Posted by "Ahmed Radwan (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13196486#comment-13196486 ]
Ahmed Radwan commented on MAPREDUCE-3736:
-----------------------------------------
This issue is relevant: https://issues.apache.org/jira/browse/HADOOP-6871
> Variable substitution depth too large for fs.default.name causes jobs to fail
> -----------------------------------------------------------------------------
>
> Key: MAPREDUCE-3736
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3736
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.1
> Reporter: Eli Collins
> Assignee: Ahmed Radwan
> Priority: Blocker
> Attachments: MAPREDUCE-3736.patch, MAPREDUCE-3736_rev2.patch
>
>
> I'm seeing the same failure as MAPREDUCE-3462 in downstream projects running against a recent build of branch-23. MR-3462 modified the tests rather than fixing the framework. In that jira Ravi mentioned "I'm still ignorant of the change which made the tests start to fail. I should probably understand better the reasons for that change before proposing a more generalized fix." Let's figure out the general fix (rather than require all projects to set mapreduce.job.hdfs-servers in their conf we should fix this in the framework). Perhaps we should not default this config to "$fs.default.name"?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3736) Variable substitution depth too
large for fs.default.name causes jobs to fail
Posted by "Ahmed Radwan (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ahmed Radwan updated MAPREDUCE-3736:
------------------------------------
Attachment: MAPREDUCE-3736_rev2.patch
Updated patch changing all instances in Mapreduce using fs.default.name to use fs.defaultFS.
> Variable substitution depth too large for fs.default.name causes jobs to fail
> -----------------------------------------------------------------------------
>
> Key: MAPREDUCE-3736
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3736
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.1
> Reporter: Eli Collins
> Assignee: Ahmed Radwan
> Priority: Blocker
> Attachments: MAPREDUCE-3736.patch, MAPREDUCE-3736_rev2.patch
>
>
> I'm seeing the same failure as MAPREDUCE-3462 in downstream projects running against a recent build of branch-23. MR-3462 modified the tests rather than fixing the framework. In that jira Ravi mentioned "I'm still ignorant of the change which made the tests start to fail. I should probably understand better the reasons for that change before proposing a more generalized fix." Let's figure out the general fix (rather than require all projects to set mapreduce.job.hdfs-servers in their conf we should fix this in the framework). Perhaps we should not default this config to "$fs.default.name"?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3736) Variable substitution depth too
large for fs.default.name causes jobs to fail
Posted by "Eli Collins (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13195185#comment-13195185 ]
Eli Collins commented on MAPREDUCE-3736:
----------------------------------------
bq. fs.default.name is added to core-default.xml, where fs.defaultFS lives now too.
Sorry, I meant why do we need to add fs.default.name back to core-default.xml, so the non-MR users like Sqoop will have not have to modify their mini MR cluster configs to set it manually? Ie is the mapreduce.job.hdfs-servers change sufficient or do they also require fs.default.name be set?
> Variable substitution depth too large for fs.default.name causes jobs to fail
> -----------------------------------------------------------------------------
>
> Key: MAPREDUCE-3736
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3736
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.1
> Reporter: Eli Collins
> Assignee: Ahmed Radwan
> Priority: Blocker
> Attachments: MAPREDUCE-3736.patch, MAPREDUCE-3736_rev2.patch
>
>
> I'm seeing the same failure as MAPREDUCE-3462 in downstream projects running against a recent build of branch-23. MR-3462 modified the tests rather than fixing the framework. In that jira Ravi mentioned "I'm still ignorant of the change which made the tests start to fail. I should probably understand better the reasons for that change before proposing a more generalized fix." Let's figure out the general fix (rather than require all projects to set mapreduce.job.hdfs-servers in their conf we should fix this in the framework). Perhaps we should not default this config to "$fs.default.name"?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3736) Variable substitution depth too
large for fs.default.name causes jobs to fail
Posted by "Ahmed Radwan (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13202838#comment-13202838 ]
Ahmed Radwan commented on MAPREDUCE-3736:
-----------------------------------------
I have confirmed that all modified tests that now use "fs.defaulyFS" are running successfully with the v2 patch.
> Variable substitution depth too large for fs.default.name causes jobs to fail
> -----------------------------------------------------------------------------
>
> Key: MAPREDUCE-3736
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3736
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.1
> Reporter: Eli Collins
> Assignee: Ahmed Radwan
> Priority: Blocker
> Attachments: MAPREDUCE-3736.patch, MAPREDUCE-3736_rev2.patch
>
>
> I'm seeing the same failure as MAPREDUCE-3462 in downstream projects running against a recent build of branch-23. MR-3462 modified the tests rather than fixing the framework. In that jira Ravi mentioned "I'm still ignorant of the change which made the tests start to fail. I should probably understand better the reasons for that change before proposing a more generalized fix." Let's figure out the general fix (rather than require all projects to set mapreduce.job.hdfs-servers in their conf we should fix this in the framework). Perhaps we should not default this config to "$fs.default.name"?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3736) Variable substitution depth too
large for fs.default.name causes jobs to fail
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13208296#comment-13208296 ]
Hudson commented on MAPREDUCE-3736:
-----------------------------------
Integrated in Hadoop-Common-trunk-Commit #1728 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1728/])
MAPREDUCE-3736. Variable substitution depth too large for fs.default.name causes jobs to fail (ahmed via tucu) (Revision 1244264)
Result = SUCCESS
tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1244264
Files :
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/resources/core-default.xml
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapred/TestMRWithDistributedCache.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/conf/TestNoDefaultsJobConf.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/fs/JHLogAnalyzer.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/io/FileBench.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestCombineFileInputFormat.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestConcatenatedCompressedInput.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestTextInputFormat.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/TestMapCollection.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/input/TestFileInputFormat.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/input/TestMRKeyValueTextInputFormat.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
> Variable substitution depth too large for fs.default.name causes jobs to fail
> -----------------------------------------------------------------------------
>
> Key: MAPREDUCE-3736
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3736
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.1
> Reporter: Eli Collins
> Assignee: Ahmed Radwan
> Priority: Blocker
> Attachments: MAPREDUCE-3736.patch, MAPREDUCE-3736_rev2.patch, MAPREDUCE-3736_rev3.patch
>
>
> I'm seeing the same failure as MAPREDUCE-3462 in downstream projects running against a recent build of branch-23. MR-3462 modified the tests rather than fixing the framework. In that jira Ravi mentioned "I'm still ignorant of the change which made the tests start to fail. I should probably understand better the reasons for that change before proposing a more generalized fix." Let's figure out the general fix (rather than require all projects to set mapreduce.job.hdfs-servers in their conf we should fix this in the framework). Perhaps we should not default this config to "$fs.default.name"?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3736) Variable substitution depth too
large for fs.default.name causes jobs to fail
Posted by "Alejandro Abdelnur (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13208049#comment-13208049 ]
Alejandro Abdelnur commented on MAPREDUCE-3736:
-----------------------------------------------
+1. I'll be committing this later today.
> Variable substitution depth too large for fs.default.name causes jobs to fail
> -----------------------------------------------------------------------------
>
> Key: MAPREDUCE-3736
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3736
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.1
> Reporter: Eli Collins
> Assignee: Ahmed Radwan
> Priority: Blocker
> Attachments: MAPREDUCE-3736.patch, MAPREDUCE-3736_rev2.patch
>
>
> I'm seeing the same failure as MAPREDUCE-3462 in downstream projects running against a recent build of branch-23. MR-3462 modified the tests rather than fixing the framework. In that jira Ravi mentioned "I'm still ignorant of the change which made the tests start to fail. I should probably understand better the reasons for that change before proposing a more generalized fix." Let's figure out the general fix (rather than require all projects to set mapreduce.job.hdfs-servers in their conf we should fix this in the framework). Perhaps we should not default this config to "$fs.default.name"?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3736) Variable substitution depth too
large for fs.default.name causes jobs to fail
Posted by "Ahmed Radwan (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13199861#comment-13199861 ]
Ahmed Radwan commented on MAPREDUCE-3736:
-----------------------------------------
@Mahadev, The issue reported in this ticket is basically caused by the fact that: The "mapreduce.job.hdfs-servers" in yarn-default.xml is set to ${fs.default.name}, and the sqoop build.xml sets <sysproperty key="fs.default.name" value="${fs.default.name}"/> and this causes the indefinite circular variable substitution.
The latest patch I uploaded uses fs.defaultFS instead of fs.default.name (also updating few old places that were using fs.default.name) and also adds fs.default.name to core-default.xml for backward compatibility. I have confirmed that the patch solves the sqoop failures.
@Eli, Are you also OK with the current shape of the patch.
> Variable substitution depth too large for fs.default.name causes jobs to fail
> -----------------------------------------------------------------------------
>
> Key: MAPREDUCE-3736
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3736
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.1
> Reporter: Eli Collins
> Assignee: Ahmed Radwan
> Priority: Blocker
> Attachments: MAPREDUCE-3736.patch, MAPREDUCE-3736_rev2.patch
>
>
> I'm seeing the same failure as MAPREDUCE-3462 in downstream projects running against a recent build of branch-23. MR-3462 modified the tests rather than fixing the framework. In that jira Ravi mentioned "I'm still ignorant of the change which made the tests start to fail. I should probably understand better the reasons for that change before proposing a more generalized fix." Let's figure out the general fix (rather than require all projects to set mapreduce.job.hdfs-servers in their conf we should fix this in the framework). Perhaps we should not default this config to "$fs.default.name"?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3736) Variable substitution depth too
large for fs.default.name causes jobs to fail
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13208428#comment-13208428 ]
Hudson commented on MAPREDUCE-3736:
-----------------------------------
Integrated in Hadoop-Hdfs-0.23-Build #169 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/169/])
Merge -r 1244263:1244264 from trunk to branch. FIXES: MAPREDUCE-3736 (Revision 1244265)
Result = FAILURE
tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1244265
Files :
* /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/resources/core-default.xml
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapred/TestMRWithDistributedCache.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/conf/TestNoDefaultsJobConf.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/fs/JHLogAnalyzer.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/io/FileBench.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestCombineFileInputFormat.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestConcatenatedCompressedInput.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestTextInputFormat.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/TestMapCollection.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/input/TestFileInputFormat.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/input/TestMRKeyValueTextInputFormat.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
> Variable substitution depth too large for fs.default.name causes jobs to fail
> -----------------------------------------------------------------------------
>
> Key: MAPREDUCE-3736
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3736
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.1
> Reporter: Eli Collins
> Assignee: Ahmed Radwan
> Priority: Blocker
> Attachments: MAPREDUCE-3736.patch, MAPREDUCE-3736_rev2.patch, MAPREDUCE-3736_rev3.patch
>
>
> I'm seeing the same failure as MAPREDUCE-3462 in downstream projects running against a recent build of branch-23. MR-3462 modified the tests rather than fixing the framework. In that jira Ravi mentioned "I'm still ignorant of the change which made the tests start to fail. I should probably understand better the reasons for that change before proposing a more generalized fix." Let's figure out the general fix (rather than require all projects to set mapreduce.job.hdfs-servers in their conf we should fix this in the framework). Perhaps we should not default this config to "$fs.default.name"?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3736) Variable substitution depth too
large for fs.default.name causes jobs to fail
Posted by "Eli Collins (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13200133#comment-13200133 ]
Eli Collins commented on MAPREDUCE-3736:
----------------------------------------
@Ahmed, I think your v2 patch is good to go. +1
I was worried that using fs.defaultFS in MR would break old configs that just used fs.default.name but it looks like the Configuration deprecation should fix that (ie MR's uses of defaultFS should work even if defaultFS is not defined in their config). Would be good to test this for sanity.
> Variable substitution depth too large for fs.default.name causes jobs to fail
> -----------------------------------------------------------------------------
>
> Key: MAPREDUCE-3736
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3736
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.1
> Reporter: Eli Collins
> Assignee: Ahmed Radwan
> Priority: Blocker
> Attachments: MAPREDUCE-3736.patch, MAPREDUCE-3736_rev2.patch
>
>
> I'm seeing the same failure as MAPREDUCE-3462 in downstream projects running against a recent build of branch-23. MR-3462 modified the tests rather than fixing the framework. In that jira Ravi mentioned "I'm still ignorant of the change which made the tests start to fail. I should probably understand better the reasons for that change before proposing a more generalized fix." Let's figure out the general fix (rather than require all projects to set mapreduce.job.hdfs-servers in their conf we should fix this in the framework). Perhaps we should not default this config to "$fs.default.name"?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira