You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Jeffrey Naisbitt (JIRA)" <ji...@apache.org> on 2011/07/20 23:13:58 UTC
[jira] [Created] (MAPREDUCE-2716) The jobFile is missing from the
ApplicationReport, causing at least the MRReliabilityTest job to fail
The jobFile is missing from the ApplicationReport, causing at least the MRReliabilityTest job to fail
-----------------------------------------------------------------------------------------------------
Key: MAPREDUCE-2716
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2716
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 0.23.0
Reporter: Jeffrey Naisbitt
Assignee: Jeffrey Naisbitt
Fix For: 0.23.0
The ApplicationReport should have the jobFile (e.g. hdfs://localhost:9000/tmp/hadoop-<USER>/mapred/staging/<USER>/.staging/job_201107121640_0001/job.xml)
Without it, jobs such as MRReliabilityTest fail with the following error (caused by the fact that jobFile is hardcoded to "" in TypeConverter.java):
e.g. java.lang.IllegalArgumentException: Can not create a Path from an empty string
at org.apache.hadoop.fs.Path.checkPathArg(Path.java:88)
at org.apache.hadoop.fs.Path.<init>(Path.java:96)
at org.apache.hadoop.mapred.JobConf.<init>(JobConf.java:445)
at org.apache.hadoop.mapreduce.Cluster.getJobs(Cluster.java:104)
at org.apache.hadoop.mapreduce.Cluster.getAllJobs(Cluster.java:218)
at org.apache.hadoop.mapred.JobClient.getAllJobs(JobClient.java:757)
at org.apache.hadoop.mapred.JobClient.jobsToComplete(JobClient.java:741)
at org.apache.hadoop.mapred.ReliabilityTest.runTest(ReliabilityTest.java:219)
at org.apache.hadoop.mapred.ReliabilityTest.runSleepJobTest(ReliabilityTest.java:133)
at org.apache.hadoop.mapred.ReliabilityTest.run(ReliabilityTest.java:116)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69)
at org.apache.hadoop.mapred.ReliabilityTest.main(ReliabilityTest.java:504)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72)
at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:144)
at org.apache.hadoop.test.MapredTestDriver.run(MapredTestDriver.java:111)
at org.apache.hadoop.test.MapredTestDriver.main(MapredTestDriver.java:118)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:192)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2716) MR279: The jobFile is missing
from the ApplicationReport, causing at least the MRReliabilityTest job to
fail
Posted by "Jeffrey Naisbitt (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jeffrey Naisbitt updated MAPREDUCE-2716:
----------------------------------------
Attachment: MAPREDUCE-2716.patch
Initial draft of patch adding jobFile
> MR279: The jobFile is missing from the ApplicationReport, causing at least the MRReliabilityTest job to fail
> ------------------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-2716
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2716
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Jeffrey Naisbitt
> Assignee: Jeffrey Naisbitt
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2716.patch
>
>
> The ApplicationReport should have the jobFile (e.g. hdfs://localhost:9000/tmp/hadoop-<USER>/mapred/staging/<USER>/.staging/job_201107121640_0001/job.xml)
> Without it, jobs such as MRReliabilityTest fail with the following error (caused by the fact that jobFile is hardcoded to "" in TypeConverter.java):
> e.g. java.lang.IllegalArgumentException: Can not create a Path from an empty string
> at org.apache.hadoop.fs.Path.checkPathArg(Path.java:88)
> at org.apache.hadoop.fs.Path.<init>(Path.java:96)
> at org.apache.hadoop.mapred.JobConf.<init>(JobConf.java:445)
> at org.apache.hadoop.mapreduce.Cluster.getJobs(Cluster.java:104)
> at org.apache.hadoop.mapreduce.Cluster.getAllJobs(Cluster.java:218)
> at org.apache.hadoop.mapred.JobClient.getAllJobs(JobClient.java:757)
> at org.apache.hadoop.mapred.JobClient.jobsToComplete(JobClient.java:741)
> at org.apache.hadoop.mapred.ReliabilityTest.runTest(ReliabilityTest.java:219)
> at org.apache.hadoop.mapred.ReliabilityTest.runSleepJobTest(ReliabilityTest.java:133)
> at org.apache.hadoop.mapred.ReliabilityTest.run(ReliabilityTest.java:116)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69)
> at org.apache.hadoop.mapred.ReliabilityTest.main(ReliabilityTest.java:504)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72)
> at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:144)
> at org.apache.hadoop.test.MapredTestDriver.run(MapredTestDriver.java:111)
> at org.apache.hadoop.test.MapredTestDriver.main(MapredTestDriver.java:118)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:192)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2716) MR279: The jobFile is missing
from the ApplicationReport, causing at least the MRReliabilityTest job to
fail
Posted by "Jeffrey Naisbitt (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jeffrey Naisbitt updated MAPREDUCE-2716:
----------------------------------------
Status: Patch Available (was: Open)
> MR279: The jobFile is missing from the ApplicationReport, causing at least the MRReliabilityTest job to fail
> ------------------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-2716
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2716
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Jeffrey Naisbitt
> Assignee: Jeffrey Naisbitt
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2716-v2.patch, MAPREDUCE-2716.patch
>
>
> The ApplicationReport should have the jobFile (e.g. hdfs://localhost:9000/tmp/hadoop-<USER>/mapred/staging/<USER>/.staging/job_201107121640_0001/job.xml)
> Without it, jobs such as MRReliabilityTest fail with the following error (caused by the fact that jobFile is hardcoded to "" in TypeConverter.java):
> e.g. java.lang.IllegalArgumentException: Can not create a Path from an empty string
> at org.apache.hadoop.fs.Path.checkPathArg(Path.java:88)
> at org.apache.hadoop.fs.Path.<init>(Path.java:96)
> at org.apache.hadoop.mapred.JobConf.<init>(JobConf.java:445)
> at org.apache.hadoop.mapreduce.Cluster.getJobs(Cluster.java:104)
> at org.apache.hadoop.mapreduce.Cluster.getAllJobs(Cluster.java:218)
> at org.apache.hadoop.mapred.JobClient.getAllJobs(JobClient.java:757)
> at org.apache.hadoop.mapred.JobClient.jobsToComplete(JobClient.java:741)
> at org.apache.hadoop.mapred.ReliabilityTest.runTest(ReliabilityTest.java:219)
> at org.apache.hadoop.mapred.ReliabilityTest.runSleepJobTest(ReliabilityTest.java:133)
> at org.apache.hadoop.mapred.ReliabilityTest.run(ReliabilityTest.java:116)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69)
> at org.apache.hadoop.mapred.ReliabilityTest.main(ReliabilityTest.java:504)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72)
> at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:144)
> at org.apache.hadoop.test.MapredTestDriver.run(MapredTestDriver.java:111)
> at org.apache.hadoop.test.MapredTestDriver.main(MapredTestDriver.java:118)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:192)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2716) MR279: MRReliabilityTest job
fails because of missing job-file.
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13094874#comment-13094874 ]
Hadoop QA commented on MAPREDUCE-2716:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12492516/MAPREDUCE-2716-v7.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 6 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
-1 findbugs. The patch appears to introduce 8 new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests:
org.apache.hadoop.mapred.TestClientRedirect
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/578//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/578//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-hs.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/578//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-shuffle.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/578//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-common.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/578//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-app.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/578//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-jobclient.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/578//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-core.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/578//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-common.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/578//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/578//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-nodemanager.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/578//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-common.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/578//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-api.html
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/578//console
This message is automatically generated.
> MR279: MRReliabilityTest job fails because of missing job-file.
> ---------------------------------------------------------------
>
> Key: MAPREDUCE-2716
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2716
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Jeffrey Naisbitt
> Assignee: Jeffrey Naisbitt
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2716-v2.patch, MAPREDUCE-2716-v3.patch, MAPREDUCE-2716-v4.patch, MAPREDUCE-2716-v5.patch, MAPREDUCE-2716-v6.patch, MAPREDUCE-2716-v7.patch, MAPREDUCE-2716.patch
>
>
> The ApplicationReport should have the jobFile (e.g. hdfs://localhost:9000/tmp/hadoop-<USER>/mapred/staging/<USER>/.staging/job_201107121640_0001/job.xml)
> Without it, jobs such as MRReliabilityTest fail with the following error (caused by the fact that jobFile is hardcoded to "" in TypeConverter.java):
> e.g. java.lang.IllegalArgumentException: Can not create a Path from an empty string
> at org.apache.hadoop.fs.Path.checkPathArg(Path.java:88)
> at org.apache.hadoop.fs.Path.<init>(Path.java:96)
> at org.apache.hadoop.mapred.JobConf.<init>(JobConf.java:445)
> at org.apache.hadoop.mapreduce.Cluster.getJobs(Cluster.java:104)
> at org.apache.hadoop.mapreduce.Cluster.getAllJobs(Cluster.java:218)
> at org.apache.hadoop.mapred.JobClient.getAllJobs(JobClient.java:757)
> at org.apache.hadoop.mapred.JobClient.jobsToComplete(JobClient.java:741)
> at org.apache.hadoop.mapred.ReliabilityTest.runTest(ReliabilityTest.java:219)
> at org.apache.hadoop.mapred.ReliabilityTest.runSleepJobTest(ReliabilityTest.java:133)
> at org.apache.hadoop.mapred.ReliabilityTest.run(ReliabilityTest.java:116)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69)
> at org.apache.hadoop.mapred.ReliabilityTest.main(ReliabilityTest.java:504)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72)
> at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:144)
> at org.apache.hadoop.test.MapredTestDriver.run(MapredTestDriver.java:111)
> at org.apache.hadoop.test.MapredTestDriver.main(MapredTestDriver.java:118)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:192)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2716) MR279: MRReliabilityTest job
fails because of missing job-file.
Posted by "Vinod Kumar Vavilapalli (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vinod Kumar Vavilapalli updated MAPREDUCE-2716:
-----------------------------------------------
Attachment: MAPREDUCE-2716-20110903.txt
+1 for the patch overall.
Minor update to synchronize the access to {{notRunningJobs}} map in multi-threaded case.
> MR279: MRReliabilityTest job fails because of missing job-file.
> ---------------------------------------------------------------
>
> Key: MAPREDUCE-2716
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2716
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Jeffrey Naisbitt
> Assignee: Jeffrey Naisbitt
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2716-20110903.txt, MAPREDUCE-2716-v2.patch, MAPREDUCE-2716-v3.patch, MAPREDUCE-2716-v4.patch, MAPREDUCE-2716-v5.patch, MAPREDUCE-2716-v6.patch, MAPREDUCE-2716-v7.patch, MAPREDUCE-2716-v8.patch, MAPREDUCE-2716.patch
>
>
> The ApplicationReport should have the jobFile (e.g. hdfs://localhost:9000/tmp/hadoop-<USER>/mapred/staging/<USER>/.staging/job_201107121640_0001/job.xml)
> Without it, jobs such as MRReliabilityTest fail with the following error (caused by the fact that jobFile is hardcoded to "" in TypeConverter.java):
> e.g. java.lang.IllegalArgumentException: Can not create a Path from an empty string
> at org.apache.hadoop.fs.Path.checkPathArg(Path.java:88)
> at org.apache.hadoop.fs.Path.<init>(Path.java:96)
> at org.apache.hadoop.mapred.JobConf.<init>(JobConf.java:445)
> at org.apache.hadoop.mapreduce.Cluster.getJobs(Cluster.java:104)
> at org.apache.hadoop.mapreduce.Cluster.getAllJobs(Cluster.java:218)
> at org.apache.hadoop.mapred.JobClient.getAllJobs(JobClient.java:757)
> at org.apache.hadoop.mapred.JobClient.jobsToComplete(JobClient.java:741)
> at org.apache.hadoop.mapred.ReliabilityTest.runTest(ReliabilityTest.java:219)
> at org.apache.hadoop.mapred.ReliabilityTest.runSleepJobTest(ReliabilityTest.java:133)
> at org.apache.hadoop.mapred.ReliabilityTest.run(ReliabilityTest.java:116)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69)
> at org.apache.hadoop.mapred.ReliabilityTest.main(ReliabilityTest.java:504)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72)
> at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:144)
> at org.apache.hadoop.test.MapredTestDriver.run(MapredTestDriver.java:111)
> at org.apache.hadoop.test.MapredTestDriver.main(MapredTestDriver.java:118)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:192)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2716) MR279: MRReliabilityTest job
fails because of missing job-file.
Posted by "Jeffrey Naisbitt (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13085345#comment-13085345 ]
Jeffrey Naisbitt commented on MAPREDUCE-2716:
---------------------------------------------
After running some more tests, there are still some problems with the latest patch (v7).
First, the createFakeJobReport calls in ClientServiceDelegate.getJobStatus seem to need a valid jobfile since the MRReliability test uses that codepath and expects a valid jobFile.
Also, I noticed that with adding the ".staging" and user (the way MRApps generates the stagingAreaDir) to the current setting in yarn-default.xml (/tmp/hadoop-yarn/${user.name}/staging), we end up with duplicated information in the path. I get this:
/tmp/hadoop-yarn/<username>/staging/<username>/.staging/job_XXX/job.xml
(Notice the duplicated <username> and the fact that we have both staging and .staging. I'm not sure how this would work when another user's jobfile is requested)
It seems like either the default should not include "<username>/staging", or we shouldn't be adding both later (unless I'm missing something here). For consistency, shouldn't we have the jobfile path & stagingAreaDir match what they were in previous versions?
Looking at a log from a mapreduce job on trunk (not MR-279), I see this:
hdfs://<hostname>/user/<username>/.staging/job_XXXX/job.xml
> MR279: MRReliabilityTest job fails because of missing job-file.
> ---------------------------------------------------------------
>
> Key: MAPREDUCE-2716
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2716
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Jeffrey Naisbitt
> Assignee: Jeffrey Naisbitt
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2716-v2.patch, MAPREDUCE-2716-v3.patch, MAPREDUCE-2716-v4.patch, MAPREDUCE-2716-v5.patch, MAPREDUCE-2716-v6.patch, MAPREDUCE-2716.patch
>
>
> The ApplicationReport should have the jobFile (e.g. hdfs://localhost:9000/tmp/hadoop-<USER>/mapred/staging/<USER>/.staging/job_201107121640_0001/job.xml)
> Without it, jobs such as MRReliabilityTest fail with the following error (caused by the fact that jobFile is hardcoded to "" in TypeConverter.java):
> e.g. java.lang.IllegalArgumentException: Can not create a Path from an empty string
> at org.apache.hadoop.fs.Path.checkPathArg(Path.java:88)
> at org.apache.hadoop.fs.Path.<init>(Path.java:96)
> at org.apache.hadoop.mapred.JobConf.<init>(JobConf.java:445)
> at org.apache.hadoop.mapreduce.Cluster.getJobs(Cluster.java:104)
> at org.apache.hadoop.mapreduce.Cluster.getAllJobs(Cluster.java:218)
> at org.apache.hadoop.mapred.JobClient.getAllJobs(JobClient.java:757)
> at org.apache.hadoop.mapred.JobClient.jobsToComplete(JobClient.java:741)
> at org.apache.hadoop.mapred.ReliabilityTest.runTest(ReliabilityTest.java:219)
> at org.apache.hadoop.mapred.ReliabilityTest.runSleepJobTest(ReliabilityTest.java:133)
> at org.apache.hadoop.mapred.ReliabilityTest.run(ReliabilityTest.java:116)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69)
> at org.apache.hadoop.mapred.ReliabilityTest.main(ReliabilityTest.java:504)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72)
> at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:144)
> at org.apache.hadoop.test.MapredTestDriver.run(MapredTestDriver.java:111)
> at org.apache.hadoop.test.MapredTestDriver.main(MapredTestDriver.java:118)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:192)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2716) MR279: MRReliabilityTest job
fails because of missing job-file.
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13082798#comment-13082798 ]
Arun C Murthy commented on MAPREDUCE-2716:
------------------------------------------
Vinod, can you pls commit it after a final review? Thanks.
> MR279: MRReliabilityTest job fails because of missing job-file.
> ---------------------------------------------------------------
>
> Key: MAPREDUCE-2716
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2716
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Jeffrey Naisbitt
> Assignee: Jeffrey Naisbitt
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2716-v2.patch, MAPREDUCE-2716-v3.patch, MAPREDUCE-2716-v4.patch, MAPREDUCE-2716.patch
>
>
> The ApplicationReport should have the jobFile (e.g. hdfs://localhost:9000/tmp/hadoop-<USER>/mapred/staging/<USER>/.staging/job_201107121640_0001/job.xml)
> Without it, jobs such as MRReliabilityTest fail with the following error (caused by the fact that jobFile is hardcoded to "" in TypeConverter.java):
> e.g. java.lang.IllegalArgumentException: Can not create a Path from an empty string
> at org.apache.hadoop.fs.Path.checkPathArg(Path.java:88)
> at org.apache.hadoop.fs.Path.<init>(Path.java:96)
> at org.apache.hadoop.mapred.JobConf.<init>(JobConf.java:445)
> at org.apache.hadoop.mapreduce.Cluster.getJobs(Cluster.java:104)
> at org.apache.hadoop.mapreduce.Cluster.getAllJobs(Cluster.java:218)
> at org.apache.hadoop.mapred.JobClient.getAllJobs(JobClient.java:757)
> at org.apache.hadoop.mapred.JobClient.jobsToComplete(JobClient.java:741)
> at org.apache.hadoop.mapred.ReliabilityTest.runTest(ReliabilityTest.java:219)
> at org.apache.hadoop.mapred.ReliabilityTest.runSleepJobTest(ReliabilityTest.java:133)
> at org.apache.hadoop.mapred.ReliabilityTest.run(ReliabilityTest.java:116)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69)
> at org.apache.hadoop.mapred.ReliabilityTest.main(ReliabilityTest.java:504)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72)
> at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:144)
> at org.apache.hadoop.test.MapredTestDriver.run(MapredTestDriver.java:111)
> at org.apache.hadoop.test.MapredTestDriver.main(MapredTestDriver.java:118)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:192)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2716) MR279: The jobFile is missing
from the ApplicationReport, causing at least the MRReliabilityTest job to
fail
Posted by "Jeffrey Naisbitt (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13070642#comment-13070642 ]
Jeffrey Naisbitt commented on MAPREDUCE-2716:
---------------------------------------------
Patch is for MR-279, so the results above are N/A.
> MR279: The jobFile is missing from the ApplicationReport, causing at least the MRReliabilityTest job to fail
> ------------------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-2716
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2716
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Jeffrey Naisbitt
> Assignee: Jeffrey Naisbitt
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2716-v2.patch, MAPREDUCE-2716.patch
>
>
> The ApplicationReport should have the jobFile (e.g. hdfs://localhost:9000/tmp/hadoop-<USER>/mapred/staging/<USER>/.staging/job_201107121640_0001/job.xml)
> Without it, jobs such as MRReliabilityTest fail with the following error (caused by the fact that jobFile is hardcoded to "" in TypeConverter.java):
> e.g. java.lang.IllegalArgumentException: Can not create a Path from an empty string
> at org.apache.hadoop.fs.Path.checkPathArg(Path.java:88)
> at org.apache.hadoop.fs.Path.<init>(Path.java:96)
> at org.apache.hadoop.mapred.JobConf.<init>(JobConf.java:445)
> at org.apache.hadoop.mapreduce.Cluster.getJobs(Cluster.java:104)
> at org.apache.hadoop.mapreduce.Cluster.getAllJobs(Cluster.java:218)
> at org.apache.hadoop.mapred.JobClient.getAllJobs(JobClient.java:757)
> at org.apache.hadoop.mapred.JobClient.jobsToComplete(JobClient.java:741)
> at org.apache.hadoop.mapred.ReliabilityTest.runTest(ReliabilityTest.java:219)
> at org.apache.hadoop.mapred.ReliabilityTest.runSleepJobTest(ReliabilityTest.java:133)
> at org.apache.hadoop.mapred.ReliabilityTest.run(ReliabilityTest.java:116)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69)
> at org.apache.hadoop.mapred.ReliabilityTest.main(ReliabilityTest.java:504)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72)
> at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:144)
> at org.apache.hadoop.test.MapredTestDriver.run(MapredTestDriver.java:111)
> at org.apache.hadoop.test.MapredTestDriver.main(MapredTestDriver.java:118)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:192)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2716) MR279: MRReliabilityTest job
fails because of missing job-file.
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13096594#comment-13096594 ]
Hudson commented on MAPREDUCE-2716:
-----------------------------------
Integrated in Hadoop-Hdfs-trunk-Commit #904 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/904/])
MAPREDUCE-2716. MRReliabilityTest job fails because of missing job-file. Contributed by Jeffrey Naisbitt.
vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1164805
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/JobImpl.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/TypeConverter.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/api/records/JobReport.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/api/records/impl/pb/JobReportPBImpl.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/proto/mr_protos.proto
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/TestTypeConverter.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/v2/util/TestMRApps.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/CompletedJob.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/ClientServiceDelegate.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/NotRunningJob.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/ResourceMgrDelegate.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestClientRedirect.java
> MR279: MRReliabilityTest job fails because of missing job-file.
> ---------------------------------------------------------------
>
> Key: MAPREDUCE-2716
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2716
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Jeffrey Naisbitt
> Assignee: Jeffrey Naisbitt
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2716-20110903.txt, MAPREDUCE-2716-v2.patch, MAPREDUCE-2716-v3.patch, MAPREDUCE-2716-v4.patch, MAPREDUCE-2716-v5.patch, MAPREDUCE-2716-v6.patch, MAPREDUCE-2716-v7.patch, MAPREDUCE-2716-v8.patch, MAPREDUCE-2716.patch
>
>
> The ApplicationReport should have the jobFile (e.g. hdfs://localhost:9000/tmp/hadoop-<USER>/mapred/staging/<USER>/.staging/job_201107121640_0001/job.xml)
> Without it, jobs such as MRReliabilityTest fail with the following error (caused by the fact that jobFile is hardcoded to "" in TypeConverter.java):
> e.g. java.lang.IllegalArgumentException: Can not create a Path from an empty string
> at org.apache.hadoop.fs.Path.checkPathArg(Path.java:88)
> at org.apache.hadoop.fs.Path.<init>(Path.java:96)
> at org.apache.hadoop.mapred.JobConf.<init>(JobConf.java:445)
> at org.apache.hadoop.mapreduce.Cluster.getJobs(Cluster.java:104)
> at org.apache.hadoop.mapreduce.Cluster.getAllJobs(Cluster.java:218)
> at org.apache.hadoop.mapred.JobClient.getAllJobs(JobClient.java:757)
> at org.apache.hadoop.mapred.JobClient.jobsToComplete(JobClient.java:741)
> at org.apache.hadoop.mapred.ReliabilityTest.runTest(ReliabilityTest.java:219)
> at org.apache.hadoop.mapred.ReliabilityTest.runSleepJobTest(ReliabilityTest.java:133)
> at org.apache.hadoop.mapred.ReliabilityTest.run(ReliabilityTest.java:116)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69)
> at org.apache.hadoop.mapred.ReliabilityTest.main(ReliabilityTest.java:504)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72)
> at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:144)
> at org.apache.hadoop.test.MapredTestDriver.run(MapredTestDriver.java:111)
> at org.apache.hadoop.test.MapredTestDriver.main(MapredTestDriver.java:118)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:192)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2716) MR279: MRReliabilityTest job
fails because of missing job-file.
Posted by "Jeffrey Naisbitt (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jeffrey Naisbitt updated MAPREDUCE-2716:
----------------------------------------
Status: Patch Available (was: Open)
> MR279: MRReliabilityTest job fails because of missing job-file.
> ---------------------------------------------------------------
>
> Key: MAPREDUCE-2716
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2716
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Jeffrey Naisbitt
> Assignee: Jeffrey Naisbitt
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2716-v2.patch, MAPREDUCE-2716-v3.patch, MAPREDUCE-2716-v4.patch, MAPREDUCE-2716.patch
>
>
> The ApplicationReport should have the jobFile (e.g. hdfs://localhost:9000/tmp/hadoop-<USER>/mapred/staging/<USER>/.staging/job_201107121640_0001/job.xml)
> Without it, jobs such as MRReliabilityTest fail with the following error (caused by the fact that jobFile is hardcoded to "" in TypeConverter.java):
> e.g. java.lang.IllegalArgumentException: Can not create a Path from an empty string
> at org.apache.hadoop.fs.Path.checkPathArg(Path.java:88)
> at org.apache.hadoop.fs.Path.<init>(Path.java:96)
> at org.apache.hadoop.mapred.JobConf.<init>(JobConf.java:445)
> at org.apache.hadoop.mapreduce.Cluster.getJobs(Cluster.java:104)
> at org.apache.hadoop.mapreduce.Cluster.getAllJobs(Cluster.java:218)
> at org.apache.hadoop.mapred.JobClient.getAllJobs(JobClient.java:757)
> at org.apache.hadoop.mapred.JobClient.jobsToComplete(JobClient.java:741)
> at org.apache.hadoop.mapred.ReliabilityTest.runTest(ReliabilityTest.java:219)
> at org.apache.hadoop.mapred.ReliabilityTest.runSleepJobTest(ReliabilityTest.java:133)
> at org.apache.hadoop.mapred.ReliabilityTest.run(ReliabilityTest.java:116)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69)
> at org.apache.hadoop.mapred.ReliabilityTest.main(ReliabilityTest.java:504)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72)
> at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:144)
> at org.apache.hadoop.test.MapredTestDriver.run(MapredTestDriver.java:111)
> at org.apache.hadoop.test.MapredTestDriver.main(MapredTestDriver.java:118)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:192)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2716) MR279: MRReliabilityTest job
fails because of missing job-file.
Posted by "Mahadev konar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mahadev konar updated MAPREDUCE-2716:
-------------------------------------
Status: Open (was: Patch Available)
looks like the patch doesnt apply. Jeffrey, can you please rebase the patch?
> MR279: MRReliabilityTest job fails because of missing job-file.
> ---------------------------------------------------------------
>
> Key: MAPREDUCE-2716
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2716
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Jeffrey Naisbitt
> Assignee: Jeffrey Naisbitt
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2716-v2.patch, MAPREDUCE-2716-v3.patch, MAPREDUCE-2716-v4.patch, MAPREDUCE-2716-v5.patch, MAPREDUCE-2716-v6.patch, MAPREDUCE-2716.patch
>
>
> The ApplicationReport should have the jobFile (e.g. hdfs://localhost:9000/tmp/hadoop-<USER>/mapred/staging/<USER>/.staging/job_201107121640_0001/job.xml)
> Without it, jobs such as MRReliabilityTest fail with the following error (caused by the fact that jobFile is hardcoded to "" in TypeConverter.java):
> e.g. java.lang.IllegalArgumentException: Can not create a Path from an empty string
> at org.apache.hadoop.fs.Path.checkPathArg(Path.java:88)
> at org.apache.hadoop.fs.Path.<init>(Path.java:96)
> at org.apache.hadoop.mapred.JobConf.<init>(JobConf.java:445)
> at org.apache.hadoop.mapreduce.Cluster.getJobs(Cluster.java:104)
> at org.apache.hadoop.mapreduce.Cluster.getAllJobs(Cluster.java:218)
> at org.apache.hadoop.mapred.JobClient.getAllJobs(JobClient.java:757)
> at org.apache.hadoop.mapred.JobClient.jobsToComplete(JobClient.java:741)
> at org.apache.hadoop.mapred.ReliabilityTest.runTest(ReliabilityTest.java:219)
> at org.apache.hadoop.mapred.ReliabilityTest.runSleepJobTest(ReliabilityTest.java:133)
> at org.apache.hadoop.mapred.ReliabilityTest.run(ReliabilityTest.java:116)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69)
> at org.apache.hadoop.mapred.ReliabilityTest.main(ReliabilityTest.java:504)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72)
> at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:144)
> at org.apache.hadoop.test.MapredTestDriver.run(MapredTestDriver.java:111)
> at org.apache.hadoop.test.MapredTestDriver.main(MapredTestDriver.java:118)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:192)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2716) MR279: MRReliabilityTest job
fails because of missing job-file.
Posted by "Jeffrey Naisbitt (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jeffrey Naisbitt updated MAPREDUCE-2716:
----------------------------------------
Attachment: MAPREDUCE-2716-v5.patch
Aargh... I forgot to add the jobId, and I was assigning the same jobFile to multiple apps/jobs. This patch addresses these issues.
I'm currently using the JobID since that's what fromYarn(applicationId) returns, but I am curious if I should be trying to use JobId instead.
> MR279: MRReliabilityTest job fails because of missing job-file.
> ---------------------------------------------------------------
>
> Key: MAPREDUCE-2716
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2716
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Jeffrey Naisbitt
> Assignee: Jeffrey Naisbitt
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2716-v2.patch, MAPREDUCE-2716-v3.patch, MAPREDUCE-2716-v4.patch, MAPREDUCE-2716-v5.patch, MAPREDUCE-2716.patch
>
>
> The ApplicationReport should have the jobFile (e.g. hdfs://localhost:9000/tmp/hadoop-<USER>/mapred/staging/<USER>/.staging/job_201107121640_0001/job.xml)
> Without it, jobs such as MRReliabilityTest fail with the following error (caused by the fact that jobFile is hardcoded to "" in TypeConverter.java):
> e.g. java.lang.IllegalArgumentException: Can not create a Path from an empty string
> at org.apache.hadoop.fs.Path.checkPathArg(Path.java:88)
> at org.apache.hadoop.fs.Path.<init>(Path.java:96)
> at org.apache.hadoop.mapred.JobConf.<init>(JobConf.java:445)
> at org.apache.hadoop.mapreduce.Cluster.getJobs(Cluster.java:104)
> at org.apache.hadoop.mapreduce.Cluster.getAllJobs(Cluster.java:218)
> at org.apache.hadoop.mapred.JobClient.getAllJobs(JobClient.java:757)
> at org.apache.hadoop.mapred.JobClient.jobsToComplete(JobClient.java:741)
> at org.apache.hadoop.mapred.ReliabilityTest.runTest(ReliabilityTest.java:219)
> at org.apache.hadoop.mapred.ReliabilityTest.runSleepJobTest(ReliabilityTest.java:133)
> at org.apache.hadoop.mapred.ReliabilityTest.run(ReliabilityTest.java:116)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69)
> at org.apache.hadoop.mapred.ReliabilityTest.main(ReliabilityTest.java:504)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72)
> at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:144)
> at org.apache.hadoop.test.MapredTestDriver.run(MapredTestDriver.java:111)
> at org.apache.hadoop.test.MapredTestDriver.main(MapredTestDriver.java:118)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:192)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2716) MR279: MRReliabilityTest job
fails because of missing job-file.
Posted by "Jeffrey Naisbitt (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jeffrey Naisbitt updated MAPREDUCE-2716:
----------------------------------------
Status: Patch Available (was: Open)
> MR279: MRReliabilityTest job fails because of missing job-file.
> ---------------------------------------------------------------
>
> Key: MAPREDUCE-2716
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2716
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Jeffrey Naisbitt
> Assignee: Jeffrey Naisbitt
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2716-v2.patch, MAPREDUCE-2716-v3.patch, MAPREDUCE-2716-v4.patch, MAPREDUCE-2716-v5.patch, MAPREDUCE-2716-v6.patch, MAPREDUCE-2716-v7.patch, MAPREDUCE-2716.patch
>
>
> The ApplicationReport should have the jobFile (e.g. hdfs://localhost:9000/tmp/hadoop-<USER>/mapred/staging/<USER>/.staging/job_201107121640_0001/job.xml)
> Without it, jobs such as MRReliabilityTest fail with the following error (caused by the fact that jobFile is hardcoded to "" in TypeConverter.java):
> e.g. java.lang.IllegalArgumentException: Can not create a Path from an empty string
> at org.apache.hadoop.fs.Path.checkPathArg(Path.java:88)
> at org.apache.hadoop.fs.Path.<init>(Path.java:96)
> at org.apache.hadoop.mapred.JobConf.<init>(JobConf.java:445)
> at org.apache.hadoop.mapreduce.Cluster.getJobs(Cluster.java:104)
> at org.apache.hadoop.mapreduce.Cluster.getAllJobs(Cluster.java:218)
> at org.apache.hadoop.mapred.JobClient.getAllJobs(JobClient.java:757)
> at org.apache.hadoop.mapred.JobClient.jobsToComplete(JobClient.java:741)
> at org.apache.hadoop.mapred.ReliabilityTest.runTest(ReliabilityTest.java:219)
> at org.apache.hadoop.mapred.ReliabilityTest.runSleepJobTest(ReliabilityTest.java:133)
> at org.apache.hadoop.mapred.ReliabilityTest.run(ReliabilityTest.java:116)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69)
> at org.apache.hadoop.mapred.ReliabilityTest.main(ReliabilityTest.java:504)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72)
> at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:144)
> at org.apache.hadoop.test.MapredTestDriver.run(MapredTestDriver.java:111)
> at org.apache.hadoop.test.MapredTestDriver.main(MapredTestDriver.java:118)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:192)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2716) MR279: MRReliabilityTest job
fails because of missing job-file.
Posted by "Vinod Kumar Vavilapalli (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vinod Kumar Vavilapalli updated MAPREDUCE-2716:
-----------------------------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
I just committed this to trunk and branch 0.23. Thanks Jeffrey!
> MR279: MRReliabilityTest job fails because of missing job-file.
> ---------------------------------------------------------------
>
> Key: MAPREDUCE-2716
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2716
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Jeffrey Naisbitt
> Assignee: Jeffrey Naisbitt
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2716-20110903.txt, MAPREDUCE-2716-v2.patch, MAPREDUCE-2716-v3.patch, MAPREDUCE-2716-v4.patch, MAPREDUCE-2716-v5.patch, MAPREDUCE-2716-v6.patch, MAPREDUCE-2716-v7.patch, MAPREDUCE-2716-v8.patch, MAPREDUCE-2716.patch
>
>
> The ApplicationReport should have the jobFile (e.g. hdfs://localhost:9000/tmp/hadoop-<USER>/mapred/staging/<USER>/.staging/job_201107121640_0001/job.xml)
> Without it, jobs such as MRReliabilityTest fail with the following error (caused by the fact that jobFile is hardcoded to "" in TypeConverter.java):
> e.g. java.lang.IllegalArgumentException: Can not create a Path from an empty string
> at org.apache.hadoop.fs.Path.checkPathArg(Path.java:88)
> at org.apache.hadoop.fs.Path.<init>(Path.java:96)
> at org.apache.hadoop.mapred.JobConf.<init>(JobConf.java:445)
> at org.apache.hadoop.mapreduce.Cluster.getJobs(Cluster.java:104)
> at org.apache.hadoop.mapreduce.Cluster.getAllJobs(Cluster.java:218)
> at org.apache.hadoop.mapred.JobClient.getAllJobs(JobClient.java:757)
> at org.apache.hadoop.mapred.JobClient.jobsToComplete(JobClient.java:741)
> at org.apache.hadoop.mapred.ReliabilityTest.runTest(ReliabilityTest.java:219)
> at org.apache.hadoop.mapred.ReliabilityTest.runSleepJobTest(ReliabilityTest.java:133)
> at org.apache.hadoop.mapred.ReliabilityTest.run(ReliabilityTest.java:116)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69)
> at org.apache.hadoop.mapred.ReliabilityTest.main(ReliabilityTest.java:504)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72)
> at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:144)
> at org.apache.hadoop.test.MapredTestDriver.run(MapredTestDriver.java:111)
> at org.apache.hadoop.test.MapredTestDriver.main(MapredTestDriver.java:118)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:192)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2716) MR279: MRReliabilityTest job
fails because of missing job-file.
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13092007#comment-13092007 ]
Hadoop QA commented on MAPREDUCE-2716:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12490454/MAPREDUCE-2716-v6.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 6 new or modified tests.
-1 patch. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/534//console
This message is automatically generated.
> MR279: MRReliabilityTest job fails because of missing job-file.
> ---------------------------------------------------------------
>
> Key: MAPREDUCE-2716
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2716
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Jeffrey Naisbitt
> Assignee: Jeffrey Naisbitt
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2716-v2.patch, MAPREDUCE-2716-v3.patch, MAPREDUCE-2716-v4.patch, MAPREDUCE-2716-v5.patch, MAPREDUCE-2716-v6.patch, MAPREDUCE-2716.patch
>
>
> The ApplicationReport should have the jobFile (e.g. hdfs://localhost:9000/tmp/hadoop-<USER>/mapred/staging/<USER>/.staging/job_201107121640_0001/job.xml)
> Without it, jobs such as MRReliabilityTest fail with the following error (caused by the fact that jobFile is hardcoded to "" in TypeConverter.java):
> e.g. java.lang.IllegalArgumentException: Can not create a Path from an empty string
> at org.apache.hadoop.fs.Path.checkPathArg(Path.java:88)
> at org.apache.hadoop.fs.Path.<init>(Path.java:96)
> at org.apache.hadoop.mapred.JobConf.<init>(JobConf.java:445)
> at org.apache.hadoop.mapreduce.Cluster.getJobs(Cluster.java:104)
> at org.apache.hadoop.mapreduce.Cluster.getAllJobs(Cluster.java:218)
> at org.apache.hadoop.mapred.JobClient.getAllJobs(JobClient.java:757)
> at org.apache.hadoop.mapred.JobClient.jobsToComplete(JobClient.java:741)
> at org.apache.hadoop.mapred.ReliabilityTest.runTest(ReliabilityTest.java:219)
> at org.apache.hadoop.mapred.ReliabilityTest.runSleepJobTest(ReliabilityTest.java:133)
> at org.apache.hadoop.mapred.ReliabilityTest.run(ReliabilityTest.java:116)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69)
> at org.apache.hadoop.mapred.ReliabilityTest.main(ReliabilityTest.java:504)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72)
> at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:144)
> at org.apache.hadoop.test.MapredTestDriver.run(MapredTestDriver.java:111)
> at org.apache.hadoop.test.MapredTestDriver.main(MapredTestDriver.java:118)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:192)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2716) MR279: MRReliabilityTest job
fails because of missing job-file.
Posted by "Vinod Kumar Vavilapalli (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vinod Kumar Vavilapalli updated MAPREDUCE-2716:
-----------------------------------------------
Status: Open (was: Patch Available)
Cancelling the patch to fix the issue correctly.
> MR279: MRReliabilityTest job fails because of missing job-file.
> ---------------------------------------------------------------
>
> Key: MAPREDUCE-2716
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2716
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Jeffrey Naisbitt
> Assignee: Jeffrey Naisbitt
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2716-v2.patch, MAPREDUCE-2716.patch
>
>
> The ApplicationReport should have the jobFile (e.g. hdfs://localhost:9000/tmp/hadoop-<USER>/mapred/staging/<USER>/.staging/job_201107121640_0001/job.xml)
> Without it, jobs such as MRReliabilityTest fail with the following error (caused by the fact that jobFile is hardcoded to "" in TypeConverter.java):
> e.g. java.lang.IllegalArgumentException: Can not create a Path from an empty string
> at org.apache.hadoop.fs.Path.checkPathArg(Path.java:88)
> at org.apache.hadoop.fs.Path.<init>(Path.java:96)
> at org.apache.hadoop.mapred.JobConf.<init>(JobConf.java:445)
> at org.apache.hadoop.mapreduce.Cluster.getJobs(Cluster.java:104)
> at org.apache.hadoop.mapreduce.Cluster.getAllJobs(Cluster.java:218)
> at org.apache.hadoop.mapred.JobClient.getAllJobs(JobClient.java:757)
> at org.apache.hadoop.mapred.JobClient.jobsToComplete(JobClient.java:741)
> at org.apache.hadoop.mapred.ReliabilityTest.runTest(ReliabilityTest.java:219)
> at org.apache.hadoop.mapred.ReliabilityTest.runSleepJobTest(ReliabilityTest.java:133)
> at org.apache.hadoop.mapred.ReliabilityTest.run(ReliabilityTest.java:116)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69)
> at org.apache.hadoop.mapred.ReliabilityTest.main(ReliabilityTest.java:504)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72)
> at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:144)
> at org.apache.hadoop.test.MapredTestDriver.run(MapredTestDriver.java:111)
> at org.apache.hadoop.test.MapredTestDriver.main(MapredTestDriver.java:118)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:192)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2716) MR279: MRReliabilityTest job
fails because of missing job-file.
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13096590#comment-13096590 ]
Hadoop QA commented on MAPREDUCE-2716:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12492874/MAPREDUCE-2716-20110903.txt
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 9 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
-1 findbugs. The patch appears to introduce 8 new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed unit tests in hadoop-mapreduce-project.
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/595//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/595//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-hs.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/595//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-shuffle.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/595//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-common.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/595//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-app.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/595//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-jobclient.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/595//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-core.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/595//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-common.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/595//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/595//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-nodemanager.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/595//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-common.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/595//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-api.html
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/595//console
This message is automatically generated.
> MR279: MRReliabilityTest job fails because of missing job-file.
> ---------------------------------------------------------------
>
> Key: MAPREDUCE-2716
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2716
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Jeffrey Naisbitt
> Assignee: Jeffrey Naisbitt
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2716-20110903.txt, MAPREDUCE-2716-v2.patch, MAPREDUCE-2716-v3.patch, MAPREDUCE-2716-v4.patch, MAPREDUCE-2716-v5.patch, MAPREDUCE-2716-v6.patch, MAPREDUCE-2716-v7.patch, MAPREDUCE-2716-v8.patch, MAPREDUCE-2716.patch
>
>
> The ApplicationReport should have the jobFile (e.g. hdfs://localhost:9000/tmp/hadoop-<USER>/mapred/staging/<USER>/.staging/job_201107121640_0001/job.xml)
> Without it, jobs such as MRReliabilityTest fail with the following error (caused by the fact that jobFile is hardcoded to "" in TypeConverter.java):
> e.g. java.lang.IllegalArgumentException: Can not create a Path from an empty string
> at org.apache.hadoop.fs.Path.checkPathArg(Path.java:88)
> at org.apache.hadoop.fs.Path.<init>(Path.java:96)
> at org.apache.hadoop.mapred.JobConf.<init>(JobConf.java:445)
> at org.apache.hadoop.mapreduce.Cluster.getJobs(Cluster.java:104)
> at org.apache.hadoop.mapreduce.Cluster.getAllJobs(Cluster.java:218)
> at org.apache.hadoop.mapred.JobClient.getAllJobs(JobClient.java:757)
> at org.apache.hadoop.mapred.JobClient.jobsToComplete(JobClient.java:741)
> at org.apache.hadoop.mapred.ReliabilityTest.runTest(ReliabilityTest.java:219)
> at org.apache.hadoop.mapred.ReliabilityTest.runSleepJobTest(ReliabilityTest.java:133)
> at org.apache.hadoop.mapred.ReliabilityTest.run(ReliabilityTest.java:116)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69)
> at org.apache.hadoop.mapred.ReliabilityTest.main(ReliabilityTest.java:504)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72)
> at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:144)
> at org.apache.hadoop.test.MapredTestDriver.run(MapredTestDriver.java:111)
> at org.apache.hadoop.test.MapredTestDriver.main(MapredTestDriver.java:118)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:192)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2716) MR279: MRReliabilityTest job
fails because of missing job-file.
Posted by "Vinod Kumar Vavilapalli (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13095775#comment-13095775 ]
Vinod Kumar Vavilapalli commented on MAPREDUCE-2716:
----------------------------------------------------
Modified the log statements to print the exception trace in the test, and found this:
{code}
Call: protocol=org.apache.hadoop.yarn.proto.ClientRMProtocol.ClientRMProtocolService.BlockingInterface, method=getApplicationReport
2011-09-02 10:19:56,878 INFO mapred.ClientServiceDelegate (ClientServiceDelegate.java:invoke(234)) - Failed to contact AM for job job_201103121733_0001 Will retry..
java.lang.reflect.InvocationTargetException
at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.mapred.ClientServiceDelegate.invoke(ClientServiceDelegate.java:228)
at org.apache.hadoop.mapred.ClientServiceDelegate.getJobStatus(ClientServiceDelegate.java:293)
at org.apache.hadoop.mapred.YARNRunner.getJobStatus(YARNRunner.java:534)
at org.apache.hadoop.mapreduce.Cluster.getJob(Cluster.java:153)
at org.apache.hadoop.mapred.TestClientRedirect.testRedirect(TestClientRedirect.java:164)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
at org.junit.runners.BlockJUnit4ClassRunner.runNotIgnored(BlockJUnit4ClassRunner.java:79)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:71)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:49)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:193)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:52)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:191)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:42)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:184)
at org.junit.runners.ParentRunner.run(ParentRunner.java:236)
at org.apache.maven.surefire.junit4.JUnit4TestSet.execute(JUnit4TestSet.java:35)
at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:115)
at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:97)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.maven.surefire.booter.ProviderFactory$ClassLoaderProxy.invoke(ProviderFactory.java:103)
at $Proxy0.invoke(Unknown Source)
at org.apache.maven.surefire.booter.SurefireStarter.invokeProvider(SurefireStarter.java:150)
at org.apache.maven.surefire.booter.SurefireStarter.runSuitesInProcess(SurefireStarter.java:74)
at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:69)
Caused by: java.lang.NullPointerException
at org.apache.hadoop.mapreduce.v2.proto.MRProtos$JobReportProto$Builder.setUser(MRProtos.java:7747)
at org.apache.hadoop.mapreduce.v2.api.records.impl.pb.JobReportPBImpl.setUser(JobReportPBImpl.java:194)
at org.apache.hadoop.mapred.NotRunningJob.getJobReport(NotRunningJob.java:110)
... 37 more
{code}
> MR279: MRReliabilityTest job fails because of missing job-file.
> ---------------------------------------------------------------
>
> Key: MAPREDUCE-2716
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2716
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Jeffrey Naisbitt
> Assignee: Jeffrey Naisbitt
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2716-v2.patch, MAPREDUCE-2716-v3.patch, MAPREDUCE-2716-v4.patch, MAPREDUCE-2716-v5.patch, MAPREDUCE-2716-v6.patch, MAPREDUCE-2716-v7.patch, MAPREDUCE-2716.patch
>
>
> The ApplicationReport should have the jobFile (e.g. hdfs://localhost:9000/tmp/hadoop-<USER>/mapred/staging/<USER>/.staging/job_201107121640_0001/job.xml)
> Without it, jobs such as MRReliabilityTest fail with the following error (caused by the fact that jobFile is hardcoded to "" in TypeConverter.java):
> e.g. java.lang.IllegalArgumentException: Can not create a Path from an empty string
> at org.apache.hadoop.fs.Path.checkPathArg(Path.java:88)
> at org.apache.hadoop.fs.Path.<init>(Path.java:96)
> at org.apache.hadoop.mapred.JobConf.<init>(JobConf.java:445)
> at org.apache.hadoop.mapreduce.Cluster.getJobs(Cluster.java:104)
> at org.apache.hadoop.mapreduce.Cluster.getAllJobs(Cluster.java:218)
> at org.apache.hadoop.mapred.JobClient.getAllJobs(JobClient.java:757)
> at org.apache.hadoop.mapred.JobClient.jobsToComplete(JobClient.java:741)
> at org.apache.hadoop.mapred.ReliabilityTest.runTest(ReliabilityTest.java:219)
> at org.apache.hadoop.mapred.ReliabilityTest.runSleepJobTest(ReliabilityTest.java:133)
> at org.apache.hadoop.mapred.ReliabilityTest.run(ReliabilityTest.java:116)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69)
> at org.apache.hadoop.mapred.ReliabilityTest.main(ReliabilityTest.java:504)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72)
> at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:144)
> at org.apache.hadoop.test.MapredTestDriver.run(MapredTestDriver.java:111)
> at org.apache.hadoop.test.MapredTestDriver.main(MapredTestDriver.java:118)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:192)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2716) MR279: MRReliabilityTest job
fails because of missing job-file.
Posted by "Jeffrey Naisbitt (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jeffrey Naisbitt updated MAPREDUCE-2716:
----------------------------------------
Attachment: MAPREDUCE-2716-v3.patch
New patch that uses the getJobStatus() method to populate the jobFile and other values. MRReliabilityTest gets past the jobfile issue, but there are still some functions that call the fromYarn method that sets the jobFile to "", so I wonder if those need to be changed too.
> MR279: MRReliabilityTest job fails because of missing job-file.
> ---------------------------------------------------------------
>
> Key: MAPREDUCE-2716
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2716
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Jeffrey Naisbitt
> Assignee: Jeffrey Naisbitt
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2716-v2.patch, MAPREDUCE-2716-v3.patch, MAPREDUCE-2716.patch
>
>
> The ApplicationReport should have the jobFile (e.g. hdfs://localhost:9000/tmp/hadoop-<USER>/mapred/staging/<USER>/.staging/job_201107121640_0001/job.xml)
> Without it, jobs such as MRReliabilityTest fail with the following error (caused by the fact that jobFile is hardcoded to "" in TypeConverter.java):
> e.g. java.lang.IllegalArgumentException: Can not create a Path from an empty string
> at org.apache.hadoop.fs.Path.checkPathArg(Path.java:88)
> at org.apache.hadoop.fs.Path.<init>(Path.java:96)
> at org.apache.hadoop.mapred.JobConf.<init>(JobConf.java:445)
> at org.apache.hadoop.mapreduce.Cluster.getJobs(Cluster.java:104)
> at org.apache.hadoop.mapreduce.Cluster.getAllJobs(Cluster.java:218)
> at org.apache.hadoop.mapred.JobClient.getAllJobs(JobClient.java:757)
> at org.apache.hadoop.mapred.JobClient.jobsToComplete(JobClient.java:741)
> at org.apache.hadoop.mapred.ReliabilityTest.runTest(ReliabilityTest.java:219)
> at org.apache.hadoop.mapred.ReliabilityTest.runSleepJobTest(ReliabilityTest.java:133)
> at org.apache.hadoop.mapred.ReliabilityTest.run(ReliabilityTest.java:116)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69)
> at org.apache.hadoop.mapred.ReliabilityTest.main(ReliabilityTest.java:504)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72)
> at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:144)
> at org.apache.hadoop.test.MapredTestDriver.run(MapredTestDriver.java:111)
> at org.apache.hadoop.test.MapredTestDriver.main(MapredTestDriver.java:118)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:192)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2716) MR279: The jobFile is missing
from the ApplicationReport, causing at least the MRReliabilityTest job to
fail
Posted by "Jeffrey Naisbitt (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jeffrey Naisbitt updated MAPREDUCE-2716:
----------------------------------------
Attachment: MAPREDUCE-2716-v2.patch
Updated patch with some tests
> MR279: The jobFile is missing from the ApplicationReport, causing at least the MRReliabilityTest job to fail
> ------------------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-2716
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2716
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Jeffrey Naisbitt
> Assignee: Jeffrey Naisbitt
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2716-v2.patch, MAPREDUCE-2716.patch
>
>
> The ApplicationReport should have the jobFile (e.g. hdfs://localhost:9000/tmp/hadoop-<USER>/mapred/staging/<USER>/.staging/job_201107121640_0001/job.xml)
> Without it, jobs such as MRReliabilityTest fail with the following error (caused by the fact that jobFile is hardcoded to "" in TypeConverter.java):
> e.g. java.lang.IllegalArgumentException: Can not create a Path from an empty string
> at org.apache.hadoop.fs.Path.checkPathArg(Path.java:88)
> at org.apache.hadoop.fs.Path.<init>(Path.java:96)
> at org.apache.hadoop.mapred.JobConf.<init>(JobConf.java:445)
> at org.apache.hadoop.mapreduce.Cluster.getJobs(Cluster.java:104)
> at org.apache.hadoop.mapreduce.Cluster.getAllJobs(Cluster.java:218)
> at org.apache.hadoop.mapred.JobClient.getAllJobs(JobClient.java:757)
> at org.apache.hadoop.mapred.JobClient.jobsToComplete(JobClient.java:741)
> at org.apache.hadoop.mapred.ReliabilityTest.runTest(ReliabilityTest.java:219)
> at org.apache.hadoop.mapred.ReliabilityTest.runSleepJobTest(ReliabilityTest.java:133)
> at org.apache.hadoop.mapred.ReliabilityTest.run(ReliabilityTest.java:116)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69)
> at org.apache.hadoop.mapred.ReliabilityTest.main(ReliabilityTest.java:504)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72)
> at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:144)
> at org.apache.hadoop.test.MapredTestDriver.run(MapredTestDriver.java:111)
> at org.apache.hadoop.test.MapredTestDriver.main(MapredTestDriver.java:118)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:192)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2716) MR279: MRReliabilityTest job
fails because of missing job-file.
Posted by "Vinod Kumar Vavilapalli (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vinod Kumar Vavilapalli updated MAPREDUCE-2716:
-----------------------------------------------
Status: Open (was: Patch Available)
> MR279: MRReliabilityTest job fails because of missing job-file.
> ---------------------------------------------------------------
>
> Key: MAPREDUCE-2716
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2716
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Jeffrey Naisbitt
> Assignee: Jeffrey Naisbitt
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2716-v2.patch, MAPREDUCE-2716-v3.patch, MAPREDUCE-2716-v4.patch, MAPREDUCE-2716-v5.patch, MAPREDUCE-2716-v6.patch, MAPREDUCE-2716-v7.patch, MAPREDUCE-2716-v8.patch, MAPREDUCE-2716.patch
>
>
> The ApplicationReport should have the jobFile (e.g. hdfs://localhost:9000/tmp/hadoop-<USER>/mapred/staging/<USER>/.staging/job_201107121640_0001/job.xml)
> Without it, jobs such as MRReliabilityTest fail with the following error (caused by the fact that jobFile is hardcoded to "" in TypeConverter.java):
> e.g. java.lang.IllegalArgumentException: Can not create a Path from an empty string
> at org.apache.hadoop.fs.Path.checkPathArg(Path.java:88)
> at org.apache.hadoop.fs.Path.<init>(Path.java:96)
> at org.apache.hadoop.mapred.JobConf.<init>(JobConf.java:445)
> at org.apache.hadoop.mapreduce.Cluster.getJobs(Cluster.java:104)
> at org.apache.hadoop.mapreduce.Cluster.getAllJobs(Cluster.java:218)
> at org.apache.hadoop.mapred.JobClient.getAllJobs(JobClient.java:757)
> at org.apache.hadoop.mapred.JobClient.jobsToComplete(JobClient.java:741)
> at org.apache.hadoop.mapred.ReliabilityTest.runTest(ReliabilityTest.java:219)
> at org.apache.hadoop.mapred.ReliabilityTest.runSleepJobTest(ReliabilityTest.java:133)
> at org.apache.hadoop.mapred.ReliabilityTest.run(ReliabilityTest.java:116)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69)
> at org.apache.hadoop.mapred.ReliabilityTest.main(ReliabilityTest.java:504)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72)
> at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:144)
> at org.apache.hadoop.test.MapredTestDriver.run(MapredTestDriver.java:111)
> at org.apache.hadoop.test.MapredTestDriver.main(MapredTestDriver.java:118)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:192)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2716) MR279: MRReliabilityTest job
fails because of missing job-file.
Posted by "Jeffrey Naisbitt (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13073038#comment-13073038 ]
Jeffrey Naisbitt commented on MAPREDUCE-2716:
---------------------------------------------
Thanks for the review, Vinod. That makes sense.
Another question/concern though: I see that in JobSubmitter.java (submitJobInternal), it originally generates the "submitJobDir" based on a call to JobSubmissionFiles.getStagingDir & the jobId. (where getStagingDir ultimately goes off to YARNRunner.getStagingAreaDir(), which goes to the ResourceMgrDelegate & then MRApps). Anyway, that staging area directory seems to include the user name and ".staging" as well as the value from the "yarn.apps.stagingDir". Is that the value we should be using instead of just getting the setting from conf? If so, maybe it would make sense to just pass the actual staging directory in instead of the actual Configuration object.
> MR279: MRReliabilityTest job fails because of missing job-file.
> ---------------------------------------------------------------
>
> Key: MAPREDUCE-2716
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2716
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Jeffrey Naisbitt
> Assignee: Jeffrey Naisbitt
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2716-v2.patch, MAPREDUCE-2716-v3.patch, MAPREDUCE-2716.patch
>
>
> The ApplicationReport should have the jobFile (e.g. hdfs://localhost:9000/tmp/hadoop-<USER>/mapred/staging/<USER>/.staging/job_201107121640_0001/job.xml)
> Without it, jobs such as MRReliabilityTest fail with the following error (caused by the fact that jobFile is hardcoded to "" in TypeConverter.java):
> e.g. java.lang.IllegalArgumentException: Can not create a Path from an empty string
> at org.apache.hadoop.fs.Path.checkPathArg(Path.java:88)
> at org.apache.hadoop.fs.Path.<init>(Path.java:96)
> at org.apache.hadoop.mapred.JobConf.<init>(JobConf.java:445)
> at org.apache.hadoop.mapreduce.Cluster.getJobs(Cluster.java:104)
> at org.apache.hadoop.mapreduce.Cluster.getAllJobs(Cluster.java:218)
> at org.apache.hadoop.mapred.JobClient.getAllJobs(JobClient.java:757)
> at org.apache.hadoop.mapred.JobClient.jobsToComplete(JobClient.java:741)
> at org.apache.hadoop.mapred.ReliabilityTest.runTest(ReliabilityTest.java:219)
> at org.apache.hadoop.mapred.ReliabilityTest.runSleepJobTest(ReliabilityTest.java:133)
> at org.apache.hadoop.mapred.ReliabilityTest.run(ReliabilityTest.java:116)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69)
> at org.apache.hadoop.mapred.ReliabilityTest.main(ReliabilityTest.java:504)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72)
> at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:144)
> at org.apache.hadoop.test.MapredTestDriver.run(MapredTestDriver.java:111)
> at org.apache.hadoop.test.MapredTestDriver.main(MapredTestDriver.java:118)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:192)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2716) MR279: MRReliabilityTest job
fails because of missing job-file.
Posted by "Jeffrey Naisbitt (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13094939#comment-13094939 ]
Jeffrey Naisbitt commented on MAPREDUCE-2716:
---------------------------------------------
The findbugs warnings seem unrelated.
However, the TestClientRedirect failure may be related. I'm looking into it - not sure what's going on there.
> MR279: MRReliabilityTest job fails because of missing job-file.
> ---------------------------------------------------------------
>
> Key: MAPREDUCE-2716
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2716
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Jeffrey Naisbitt
> Assignee: Jeffrey Naisbitt
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2716-v2.patch, MAPREDUCE-2716-v3.patch, MAPREDUCE-2716-v4.patch, MAPREDUCE-2716-v5.patch, MAPREDUCE-2716-v6.patch, MAPREDUCE-2716-v7.patch, MAPREDUCE-2716.patch
>
>
> The ApplicationReport should have the jobFile (e.g. hdfs://localhost:9000/tmp/hadoop-<USER>/mapred/staging/<USER>/.staging/job_201107121640_0001/job.xml)
> Without it, jobs such as MRReliabilityTest fail with the following error (caused by the fact that jobFile is hardcoded to "" in TypeConverter.java):
> e.g. java.lang.IllegalArgumentException: Can not create a Path from an empty string
> at org.apache.hadoop.fs.Path.checkPathArg(Path.java:88)
> at org.apache.hadoop.fs.Path.<init>(Path.java:96)
> at org.apache.hadoop.mapred.JobConf.<init>(JobConf.java:445)
> at org.apache.hadoop.mapreduce.Cluster.getJobs(Cluster.java:104)
> at org.apache.hadoop.mapreduce.Cluster.getAllJobs(Cluster.java:218)
> at org.apache.hadoop.mapred.JobClient.getAllJobs(JobClient.java:757)
> at org.apache.hadoop.mapred.JobClient.jobsToComplete(JobClient.java:741)
> at org.apache.hadoop.mapred.ReliabilityTest.runTest(ReliabilityTest.java:219)
> at org.apache.hadoop.mapred.ReliabilityTest.runSleepJobTest(ReliabilityTest.java:133)
> at org.apache.hadoop.mapred.ReliabilityTest.run(ReliabilityTest.java:116)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69)
> at org.apache.hadoop.mapred.ReliabilityTest.main(ReliabilityTest.java:504)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72)
> at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:144)
> at org.apache.hadoop.test.MapredTestDriver.run(MapredTestDriver.java:111)
> at org.apache.hadoop.test.MapredTestDriver.main(MapredTestDriver.java:118)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:192)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2716) MR279: MRReliabilityTest job
fails because of missing job-file.
Posted by "Jeffrey Naisbitt (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jeffrey Naisbitt updated MAPREDUCE-2716:
----------------------------------------
Attachment: MAPREDUCE-2716-v4.patch
Updated patch, addressing Vinod's comments and fixing existing issue with jobfile generation.
> MR279: MRReliabilityTest job fails because of missing job-file.
> ---------------------------------------------------------------
>
> Key: MAPREDUCE-2716
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2716
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Jeffrey Naisbitt
> Assignee: Jeffrey Naisbitt
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2716-v2.patch, MAPREDUCE-2716-v3.patch, MAPREDUCE-2716-v4.patch, MAPREDUCE-2716.patch
>
>
> The ApplicationReport should have the jobFile (e.g. hdfs://localhost:9000/tmp/hadoop-<USER>/mapred/staging/<USER>/.staging/job_201107121640_0001/job.xml)
> Without it, jobs such as MRReliabilityTest fail with the following error (caused by the fact that jobFile is hardcoded to "" in TypeConverter.java):
> e.g. java.lang.IllegalArgumentException: Can not create a Path from an empty string
> at org.apache.hadoop.fs.Path.checkPathArg(Path.java:88)
> at org.apache.hadoop.fs.Path.<init>(Path.java:96)
> at org.apache.hadoop.mapred.JobConf.<init>(JobConf.java:445)
> at org.apache.hadoop.mapreduce.Cluster.getJobs(Cluster.java:104)
> at org.apache.hadoop.mapreduce.Cluster.getAllJobs(Cluster.java:218)
> at org.apache.hadoop.mapred.JobClient.getAllJobs(JobClient.java:757)
> at org.apache.hadoop.mapred.JobClient.jobsToComplete(JobClient.java:741)
> at org.apache.hadoop.mapred.ReliabilityTest.runTest(ReliabilityTest.java:219)
> at org.apache.hadoop.mapred.ReliabilityTest.runSleepJobTest(ReliabilityTest.java:133)
> at org.apache.hadoop.mapred.ReliabilityTest.run(ReliabilityTest.java:116)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69)
> at org.apache.hadoop.mapred.ReliabilityTest.main(ReliabilityTest.java:504)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72)
> at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:144)
> at org.apache.hadoop.test.MapredTestDriver.run(MapredTestDriver.java:111)
> at org.apache.hadoop.test.MapredTestDriver.main(MapredTestDriver.java:118)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:192)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2716) MR279: MRReliabilityTest job
fails because of missing job-file.
Posted by "Jeffrey Naisbitt (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13096027#comment-13096027 ]
Jeffrey Naisbitt commented on MAPREDUCE-2716:
---------------------------------------------
Yeah, it looks like the user is not actually set in the application report (I'm trying to use it in the getProxy() method in ClientServiceDelegate).
> MR279: MRReliabilityTest job fails because of missing job-file.
> ---------------------------------------------------------------
>
> Key: MAPREDUCE-2716
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2716
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Jeffrey Naisbitt
> Assignee: Jeffrey Naisbitt
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2716-v2.patch, MAPREDUCE-2716-v3.patch, MAPREDUCE-2716-v4.patch, MAPREDUCE-2716-v5.patch, MAPREDUCE-2716-v6.patch, MAPREDUCE-2716-v7.patch, MAPREDUCE-2716.patch
>
>
> The ApplicationReport should have the jobFile (e.g. hdfs://localhost:9000/tmp/hadoop-<USER>/mapred/staging/<USER>/.staging/job_201107121640_0001/job.xml)
> Without it, jobs such as MRReliabilityTest fail with the following error (caused by the fact that jobFile is hardcoded to "" in TypeConverter.java):
> e.g. java.lang.IllegalArgumentException: Can not create a Path from an empty string
> at org.apache.hadoop.fs.Path.checkPathArg(Path.java:88)
> at org.apache.hadoop.fs.Path.<init>(Path.java:96)
> at org.apache.hadoop.mapred.JobConf.<init>(JobConf.java:445)
> at org.apache.hadoop.mapreduce.Cluster.getJobs(Cluster.java:104)
> at org.apache.hadoop.mapreduce.Cluster.getAllJobs(Cluster.java:218)
> at org.apache.hadoop.mapred.JobClient.getAllJobs(JobClient.java:757)
> at org.apache.hadoop.mapred.JobClient.jobsToComplete(JobClient.java:741)
> at org.apache.hadoop.mapred.ReliabilityTest.runTest(ReliabilityTest.java:219)
> at org.apache.hadoop.mapred.ReliabilityTest.runSleepJobTest(ReliabilityTest.java:133)
> at org.apache.hadoop.mapred.ReliabilityTest.run(ReliabilityTest.java:116)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69)
> at org.apache.hadoop.mapred.ReliabilityTest.main(ReliabilityTest.java:504)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72)
> at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:144)
> at org.apache.hadoop.test.MapredTestDriver.run(MapredTestDriver.java:111)
> at org.apache.hadoop.test.MapredTestDriver.main(MapredTestDriver.java:118)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:192)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2716) MR279: MRReliabilityTest job
fails because of missing job-file.
Posted by "Vinod Kumar Vavilapalli (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vinod Kumar Vavilapalli updated MAPREDUCE-2716:
-----------------------------------------------
Hadoop Flags: [Reviewed]
Status: Patch Available (was: Open)
> MR279: MRReliabilityTest job fails because of missing job-file.
> ---------------------------------------------------------------
>
> Key: MAPREDUCE-2716
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2716
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Jeffrey Naisbitt
> Assignee: Jeffrey Naisbitt
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2716-20110903.txt, MAPREDUCE-2716-v2.patch, MAPREDUCE-2716-v3.patch, MAPREDUCE-2716-v4.patch, MAPREDUCE-2716-v5.patch, MAPREDUCE-2716-v6.patch, MAPREDUCE-2716-v7.patch, MAPREDUCE-2716-v8.patch, MAPREDUCE-2716.patch
>
>
> The ApplicationReport should have the jobFile (e.g. hdfs://localhost:9000/tmp/hadoop-<USER>/mapred/staging/<USER>/.staging/job_201107121640_0001/job.xml)
> Without it, jobs such as MRReliabilityTest fail with the following error (caused by the fact that jobFile is hardcoded to "" in TypeConverter.java):
> e.g. java.lang.IllegalArgumentException: Can not create a Path from an empty string
> at org.apache.hadoop.fs.Path.checkPathArg(Path.java:88)
> at org.apache.hadoop.fs.Path.<init>(Path.java:96)
> at org.apache.hadoop.mapred.JobConf.<init>(JobConf.java:445)
> at org.apache.hadoop.mapreduce.Cluster.getJobs(Cluster.java:104)
> at org.apache.hadoop.mapreduce.Cluster.getAllJobs(Cluster.java:218)
> at org.apache.hadoop.mapred.JobClient.getAllJobs(JobClient.java:757)
> at org.apache.hadoop.mapred.JobClient.jobsToComplete(JobClient.java:741)
> at org.apache.hadoop.mapred.ReliabilityTest.runTest(ReliabilityTest.java:219)
> at org.apache.hadoop.mapred.ReliabilityTest.runSleepJobTest(ReliabilityTest.java:133)
> at org.apache.hadoop.mapred.ReliabilityTest.run(ReliabilityTest.java:116)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69)
> at org.apache.hadoop.mapred.ReliabilityTest.main(ReliabilityTest.java:504)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72)
> at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:144)
> at org.apache.hadoop.test.MapredTestDriver.run(MapredTestDriver.java:111)
> at org.apache.hadoop.test.MapredTestDriver.main(MapredTestDriver.java:118)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:192)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2716) MR279: MRReliabilityTest job
fails because of missing job-file.
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13096593#comment-13096593 ]
Hudson commented on MAPREDUCE-2716:
-----------------------------------
Integrated in Hadoop-Common-trunk-Commit #827 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/827/])
MAPREDUCE-2716. MRReliabilityTest job fails because of missing job-file. Contributed by Jeffrey Naisbitt.
vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1164805
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/JobImpl.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/TypeConverter.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/api/records/JobReport.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/api/records/impl/pb/JobReportPBImpl.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/proto/mr_protos.proto
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/TestTypeConverter.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/v2/util/TestMRApps.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/CompletedJob.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/ClientServiceDelegate.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/NotRunningJob.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/ResourceMgrDelegate.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestClientRedirect.java
> MR279: MRReliabilityTest job fails because of missing job-file.
> ---------------------------------------------------------------
>
> Key: MAPREDUCE-2716
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2716
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Jeffrey Naisbitt
> Assignee: Jeffrey Naisbitt
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2716-20110903.txt, MAPREDUCE-2716-v2.patch, MAPREDUCE-2716-v3.patch, MAPREDUCE-2716-v4.patch, MAPREDUCE-2716-v5.patch, MAPREDUCE-2716-v6.patch, MAPREDUCE-2716-v7.patch, MAPREDUCE-2716-v8.patch, MAPREDUCE-2716.patch
>
>
> The ApplicationReport should have the jobFile (e.g. hdfs://localhost:9000/tmp/hadoop-<USER>/mapred/staging/<USER>/.staging/job_201107121640_0001/job.xml)
> Without it, jobs such as MRReliabilityTest fail with the following error (caused by the fact that jobFile is hardcoded to "" in TypeConverter.java):
> e.g. java.lang.IllegalArgumentException: Can not create a Path from an empty string
> at org.apache.hadoop.fs.Path.checkPathArg(Path.java:88)
> at org.apache.hadoop.fs.Path.<init>(Path.java:96)
> at org.apache.hadoop.mapred.JobConf.<init>(JobConf.java:445)
> at org.apache.hadoop.mapreduce.Cluster.getJobs(Cluster.java:104)
> at org.apache.hadoop.mapreduce.Cluster.getAllJobs(Cluster.java:218)
> at org.apache.hadoop.mapred.JobClient.getAllJobs(JobClient.java:757)
> at org.apache.hadoop.mapred.JobClient.jobsToComplete(JobClient.java:741)
> at org.apache.hadoop.mapred.ReliabilityTest.runTest(ReliabilityTest.java:219)
> at org.apache.hadoop.mapred.ReliabilityTest.runSleepJobTest(ReliabilityTest.java:133)
> at org.apache.hadoop.mapred.ReliabilityTest.run(ReliabilityTest.java:116)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69)
> at org.apache.hadoop.mapred.ReliabilityTest.main(ReliabilityTest.java:504)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72)
> at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:144)
> at org.apache.hadoop.test.MapredTestDriver.run(MapredTestDriver.java:111)
> at org.apache.hadoop.test.MapredTestDriver.main(MapredTestDriver.java:118)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:192)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2716) MR279: MRReliabilityTest job
fails because of missing job-file.
Posted by "Jeffrey Naisbitt (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jeffrey Naisbitt updated MAPREDUCE-2716:
----------------------------------------
Attachment: MAPREDUCE-2716-v7.patch
Rebased and updated for all the changes to trunk.
> MR279: MRReliabilityTest job fails because of missing job-file.
> ---------------------------------------------------------------
>
> Key: MAPREDUCE-2716
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2716
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Jeffrey Naisbitt
> Assignee: Jeffrey Naisbitt
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2716-v2.patch, MAPREDUCE-2716-v3.patch, MAPREDUCE-2716-v4.patch, MAPREDUCE-2716-v5.patch, MAPREDUCE-2716-v6.patch, MAPREDUCE-2716-v7.patch, MAPREDUCE-2716.patch
>
>
> The ApplicationReport should have the jobFile (e.g. hdfs://localhost:9000/tmp/hadoop-<USER>/mapred/staging/<USER>/.staging/job_201107121640_0001/job.xml)
> Without it, jobs such as MRReliabilityTest fail with the following error (caused by the fact that jobFile is hardcoded to "" in TypeConverter.java):
> e.g. java.lang.IllegalArgumentException: Can not create a Path from an empty string
> at org.apache.hadoop.fs.Path.checkPathArg(Path.java:88)
> at org.apache.hadoop.fs.Path.<init>(Path.java:96)
> at org.apache.hadoop.mapred.JobConf.<init>(JobConf.java:445)
> at org.apache.hadoop.mapreduce.Cluster.getJobs(Cluster.java:104)
> at org.apache.hadoop.mapreduce.Cluster.getAllJobs(Cluster.java:218)
> at org.apache.hadoop.mapred.JobClient.getAllJobs(JobClient.java:757)
> at org.apache.hadoop.mapred.JobClient.jobsToComplete(JobClient.java:741)
> at org.apache.hadoop.mapred.ReliabilityTest.runTest(ReliabilityTest.java:219)
> at org.apache.hadoop.mapred.ReliabilityTest.runSleepJobTest(ReliabilityTest.java:133)
> at org.apache.hadoop.mapred.ReliabilityTest.run(ReliabilityTest.java:116)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69)
> at org.apache.hadoop.mapred.ReliabilityTest.main(ReliabilityTest.java:504)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72)
> at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:144)
> at org.apache.hadoop.test.MapredTestDriver.run(MapredTestDriver.java:111)
> at org.apache.hadoop.test.MapredTestDriver.main(MapredTestDriver.java:118)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:192)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2716) MR279: MRReliabilityTest job
fails because of missing job-file.
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13096081#comment-13096081 ]
Hadoop QA commented on MAPREDUCE-2716:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12492763/MAPREDUCE-2716-v8.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 9 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
-1 findbugs. The patch appears to introduce 8 new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed unit tests in hadoop-mapreduce-project.
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/588//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/588//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-common.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/588//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-api.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/588//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-common.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/588//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/588//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-nodemanager.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/588//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-app.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/588//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-core.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/588//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-common.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/588//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-hs.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/588//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-shuffle.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/588//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-jobclient.html
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/588//console
This message is automatically generated.
> MR279: MRReliabilityTest job fails because of missing job-file.
> ---------------------------------------------------------------
>
> Key: MAPREDUCE-2716
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2716
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Jeffrey Naisbitt
> Assignee: Jeffrey Naisbitt
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2716-v2.patch, MAPREDUCE-2716-v3.patch, MAPREDUCE-2716-v4.patch, MAPREDUCE-2716-v5.patch, MAPREDUCE-2716-v6.patch, MAPREDUCE-2716-v7.patch, MAPREDUCE-2716-v8.patch, MAPREDUCE-2716.patch
>
>
> The ApplicationReport should have the jobFile (e.g. hdfs://localhost:9000/tmp/hadoop-<USER>/mapred/staging/<USER>/.staging/job_201107121640_0001/job.xml)
> Without it, jobs such as MRReliabilityTest fail with the following error (caused by the fact that jobFile is hardcoded to "" in TypeConverter.java):
> e.g. java.lang.IllegalArgumentException: Can not create a Path from an empty string
> at org.apache.hadoop.fs.Path.checkPathArg(Path.java:88)
> at org.apache.hadoop.fs.Path.<init>(Path.java:96)
> at org.apache.hadoop.mapred.JobConf.<init>(JobConf.java:445)
> at org.apache.hadoop.mapreduce.Cluster.getJobs(Cluster.java:104)
> at org.apache.hadoop.mapreduce.Cluster.getAllJobs(Cluster.java:218)
> at org.apache.hadoop.mapred.JobClient.getAllJobs(JobClient.java:757)
> at org.apache.hadoop.mapred.JobClient.jobsToComplete(JobClient.java:741)
> at org.apache.hadoop.mapred.ReliabilityTest.runTest(ReliabilityTest.java:219)
> at org.apache.hadoop.mapred.ReliabilityTest.runSleepJobTest(ReliabilityTest.java:133)
> at org.apache.hadoop.mapred.ReliabilityTest.run(ReliabilityTest.java:116)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69)
> at org.apache.hadoop.mapred.ReliabilityTest.main(ReliabilityTest.java:504)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72)
> at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:144)
> at org.apache.hadoop.test.MapredTestDriver.run(MapredTestDriver.java:111)
> at org.apache.hadoop.test.MapredTestDriver.main(MapredTestDriver.java:118)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:192)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2716) MR279: MRReliabilityTest job
fails because of missing job-file.
Posted by "Jeffrey Naisbitt (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13096113#comment-13096113 ]
Jeffrey Naisbitt commented on MAPREDUCE-2716:
---------------------------------------------
The findbugs warnings are still unrelated to this patch.
> MR279: MRReliabilityTest job fails because of missing job-file.
> ---------------------------------------------------------------
>
> Key: MAPREDUCE-2716
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2716
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Jeffrey Naisbitt
> Assignee: Jeffrey Naisbitt
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2716-v2.patch, MAPREDUCE-2716-v3.patch, MAPREDUCE-2716-v4.patch, MAPREDUCE-2716-v5.patch, MAPREDUCE-2716-v6.patch, MAPREDUCE-2716-v7.patch, MAPREDUCE-2716-v8.patch, MAPREDUCE-2716.patch
>
>
> The ApplicationReport should have the jobFile (e.g. hdfs://localhost:9000/tmp/hadoop-<USER>/mapred/staging/<USER>/.staging/job_201107121640_0001/job.xml)
> Without it, jobs such as MRReliabilityTest fail with the following error (caused by the fact that jobFile is hardcoded to "" in TypeConverter.java):
> e.g. java.lang.IllegalArgumentException: Can not create a Path from an empty string
> at org.apache.hadoop.fs.Path.checkPathArg(Path.java:88)
> at org.apache.hadoop.fs.Path.<init>(Path.java:96)
> at org.apache.hadoop.mapred.JobConf.<init>(JobConf.java:445)
> at org.apache.hadoop.mapreduce.Cluster.getJobs(Cluster.java:104)
> at org.apache.hadoop.mapreduce.Cluster.getAllJobs(Cluster.java:218)
> at org.apache.hadoop.mapred.JobClient.getAllJobs(JobClient.java:757)
> at org.apache.hadoop.mapred.JobClient.jobsToComplete(JobClient.java:741)
> at org.apache.hadoop.mapred.ReliabilityTest.runTest(ReliabilityTest.java:219)
> at org.apache.hadoop.mapred.ReliabilityTest.runSleepJobTest(ReliabilityTest.java:133)
> at org.apache.hadoop.mapred.ReliabilityTest.run(ReliabilityTest.java:116)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69)
> at org.apache.hadoop.mapred.ReliabilityTest.main(ReliabilityTest.java:504)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72)
> at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:144)
> at org.apache.hadoop.test.MapredTestDriver.run(MapredTestDriver.java:111)
> at org.apache.hadoop.test.MapredTestDriver.main(MapredTestDriver.java:118)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:192)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2716) MR279: MRReliabilityTest job
fails because of missing job-file.
Posted by "Jeffrey Naisbitt (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jeffrey Naisbitt updated MAPREDUCE-2716:
----------------------------------------
Status: Patch Available (was: Open)
> MR279: MRReliabilityTest job fails because of missing job-file.
> ---------------------------------------------------------------
>
> Key: MAPREDUCE-2716
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2716
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Jeffrey Naisbitt
> Assignee: Jeffrey Naisbitt
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2716-v2.patch, MAPREDUCE-2716-v3.patch, MAPREDUCE-2716-v4.patch, MAPREDUCE-2716-v5.patch, MAPREDUCE-2716-v6.patch, MAPREDUCE-2716-v7.patch, MAPREDUCE-2716-v8.patch, MAPREDUCE-2716.patch
>
>
> The ApplicationReport should have the jobFile (e.g. hdfs://localhost:9000/tmp/hadoop-<USER>/mapred/staging/<USER>/.staging/job_201107121640_0001/job.xml)
> Without it, jobs such as MRReliabilityTest fail with the following error (caused by the fact that jobFile is hardcoded to "" in TypeConverter.java):
> e.g. java.lang.IllegalArgumentException: Can not create a Path from an empty string
> at org.apache.hadoop.fs.Path.checkPathArg(Path.java:88)
> at org.apache.hadoop.fs.Path.<init>(Path.java:96)
> at org.apache.hadoop.mapred.JobConf.<init>(JobConf.java:445)
> at org.apache.hadoop.mapreduce.Cluster.getJobs(Cluster.java:104)
> at org.apache.hadoop.mapreduce.Cluster.getAllJobs(Cluster.java:218)
> at org.apache.hadoop.mapred.JobClient.getAllJobs(JobClient.java:757)
> at org.apache.hadoop.mapred.JobClient.jobsToComplete(JobClient.java:741)
> at org.apache.hadoop.mapred.ReliabilityTest.runTest(ReliabilityTest.java:219)
> at org.apache.hadoop.mapred.ReliabilityTest.runSleepJobTest(ReliabilityTest.java:133)
> at org.apache.hadoop.mapred.ReliabilityTest.run(ReliabilityTest.java:116)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69)
> at org.apache.hadoop.mapred.ReliabilityTest.main(ReliabilityTest.java:504)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72)
> at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:144)
> at org.apache.hadoop.test.MapredTestDriver.run(MapredTestDriver.java:111)
> at org.apache.hadoop.test.MapredTestDriver.main(MapredTestDriver.java:118)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:192)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2716) MR279: MRReliabilityTest job
fails because of missing job-file.
Posted by "Jeffrey Naisbitt (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jeffrey Naisbitt updated MAPREDUCE-2716:
----------------------------------------
Status: Patch Available (was: Open)
> MR279: MRReliabilityTest job fails because of missing job-file.
> ---------------------------------------------------------------
>
> Key: MAPREDUCE-2716
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2716
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Jeffrey Naisbitt
> Assignee: Jeffrey Naisbitt
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2716-v2.patch, MAPREDUCE-2716-v3.patch, MAPREDUCE-2716-v4.patch, MAPREDUCE-2716-v5.patch, MAPREDUCE-2716-v6.patch, MAPREDUCE-2716.patch
>
>
> The ApplicationReport should have the jobFile (e.g. hdfs://localhost:9000/tmp/hadoop-<USER>/mapred/staging/<USER>/.staging/job_201107121640_0001/job.xml)
> Without it, jobs such as MRReliabilityTest fail with the following error (caused by the fact that jobFile is hardcoded to "" in TypeConverter.java):
> e.g. java.lang.IllegalArgumentException: Can not create a Path from an empty string
> at org.apache.hadoop.fs.Path.checkPathArg(Path.java:88)
> at org.apache.hadoop.fs.Path.<init>(Path.java:96)
> at org.apache.hadoop.mapred.JobConf.<init>(JobConf.java:445)
> at org.apache.hadoop.mapreduce.Cluster.getJobs(Cluster.java:104)
> at org.apache.hadoop.mapreduce.Cluster.getAllJobs(Cluster.java:218)
> at org.apache.hadoop.mapred.JobClient.getAllJobs(JobClient.java:757)
> at org.apache.hadoop.mapred.JobClient.jobsToComplete(JobClient.java:741)
> at org.apache.hadoop.mapred.ReliabilityTest.runTest(ReliabilityTest.java:219)
> at org.apache.hadoop.mapred.ReliabilityTest.runSleepJobTest(ReliabilityTest.java:133)
> at org.apache.hadoop.mapred.ReliabilityTest.run(ReliabilityTest.java:116)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69)
> at org.apache.hadoop.mapred.ReliabilityTest.main(ReliabilityTest.java:504)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72)
> at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:144)
> at org.apache.hadoop.test.MapredTestDriver.run(MapredTestDriver.java:111)
> at org.apache.hadoop.test.MapredTestDriver.main(MapredTestDriver.java:118)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:192)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2716) MR279: MRReliabilityTest job
fails because of missing job-file.
Posted by "Vinod Kumar Vavilapalli (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13072789#comment-13072789 ]
Vinod Kumar Vavilapalli commented on MAPREDUCE-2716:
----------------------------------------------------
That's more like it.
But this patch has one issue in {{ClientServiceDelegate.getAllJobs()}}. After getting the app-list from the resource-manager, we also make more trips one per job to the RM via {{getJobState(JobId oldJobId)}} (ClientServiceDelegate.java +342 after applying your patch). Which is unnecessary.
Instead, we can use just get the JobStatus using the already present {{TypeConverter.fromYarn(ApplicationReport report)}} and set the job-file on the JobStatus to be {{conf.get("yarn.apps.stagingDir") + "/" + jobId.toString()}}. Once you do this, you will also not need the new {{fromYarn(ApplicationReport report, JobStatus status)}} utility in TypeConverter.
Regarding your other comment about other places that use {{TypeConverter.fromYarn(ApplicationReport report)}}. Yes that is a problem. So I propose we change it to {{TypeConverter.fromYarn(ApplicationReport report, Configuration conf)}} and set job-file using conf as described above and use this method everywhere.
> MR279: MRReliabilityTest job fails because of missing job-file.
> ---------------------------------------------------------------
>
> Key: MAPREDUCE-2716
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2716
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Jeffrey Naisbitt
> Assignee: Jeffrey Naisbitt
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2716-v2.patch, MAPREDUCE-2716-v3.patch, MAPREDUCE-2716.patch
>
>
> The ApplicationReport should have the jobFile (e.g. hdfs://localhost:9000/tmp/hadoop-<USER>/mapred/staging/<USER>/.staging/job_201107121640_0001/job.xml)
> Without it, jobs such as MRReliabilityTest fail with the following error (caused by the fact that jobFile is hardcoded to "" in TypeConverter.java):
> e.g. java.lang.IllegalArgumentException: Can not create a Path from an empty string
> at org.apache.hadoop.fs.Path.checkPathArg(Path.java:88)
> at org.apache.hadoop.fs.Path.<init>(Path.java:96)
> at org.apache.hadoop.mapred.JobConf.<init>(JobConf.java:445)
> at org.apache.hadoop.mapreduce.Cluster.getJobs(Cluster.java:104)
> at org.apache.hadoop.mapreduce.Cluster.getAllJobs(Cluster.java:218)
> at org.apache.hadoop.mapred.JobClient.getAllJobs(JobClient.java:757)
> at org.apache.hadoop.mapred.JobClient.jobsToComplete(JobClient.java:741)
> at org.apache.hadoop.mapred.ReliabilityTest.runTest(ReliabilityTest.java:219)
> at org.apache.hadoop.mapred.ReliabilityTest.runSleepJobTest(ReliabilityTest.java:133)
> at org.apache.hadoop.mapred.ReliabilityTest.run(ReliabilityTest.java:116)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69)
> at org.apache.hadoop.mapred.ReliabilityTest.main(ReliabilityTest.java:504)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72)
> at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:144)
> at org.apache.hadoop.test.MapredTestDriver.run(MapredTestDriver.java:111)
> at org.apache.hadoop.test.MapredTestDriver.main(MapredTestDriver.java:118)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:192)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2716) MR279: MRReliabilityTest job
fails because of missing job-file.
Posted by "Vinod Kumar Vavilapalli (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vinod Kumar Vavilapalli updated MAPREDUCE-2716:
-----------------------------------------------
Summary: MR279: MRReliabilityTest job fails because of missing job-file. (was: MR279: The jobFile is missing from the ApplicationReport, causing at least the MRReliabilityTest job to fail)
Hey Jeffrey, apologies for not noticing this earlier but this is an incorrect fix. YARN ( as opposed to mapreduce) doesn't have any notion of jobFile. Job-file is a concept of mapreduce and so should not be added to ApplicationReport (which is generic).
The correct fix is to do similar to how ClientServiceDelegate.getJobStatus() is constucting job-file and populating JobStatus.
> MR279: MRReliabilityTest job fails because of missing job-file.
> ---------------------------------------------------------------
>
> Key: MAPREDUCE-2716
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2716
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Jeffrey Naisbitt
> Assignee: Jeffrey Naisbitt
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2716-v2.patch, MAPREDUCE-2716.patch
>
>
> The ApplicationReport should have the jobFile (e.g. hdfs://localhost:9000/tmp/hadoop-<USER>/mapred/staging/<USER>/.staging/job_201107121640_0001/job.xml)
> Without it, jobs such as MRReliabilityTest fail with the following error (caused by the fact that jobFile is hardcoded to "" in TypeConverter.java):
> e.g. java.lang.IllegalArgumentException: Can not create a Path from an empty string
> at org.apache.hadoop.fs.Path.checkPathArg(Path.java:88)
> at org.apache.hadoop.fs.Path.<init>(Path.java:96)
> at org.apache.hadoop.mapred.JobConf.<init>(JobConf.java:445)
> at org.apache.hadoop.mapreduce.Cluster.getJobs(Cluster.java:104)
> at org.apache.hadoop.mapreduce.Cluster.getAllJobs(Cluster.java:218)
> at org.apache.hadoop.mapred.JobClient.getAllJobs(JobClient.java:757)
> at org.apache.hadoop.mapred.JobClient.jobsToComplete(JobClient.java:741)
> at org.apache.hadoop.mapred.ReliabilityTest.runTest(ReliabilityTest.java:219)
> at org.apache.hadoop.mapred.ReliabilityTest.runSleepJobTest(ReliabilityTest.java:133)
> at org.apache.hadoop.mapred.ReliabilityTest.run(ReliabilityTest.java:116)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69)
> at org.apache.hadoop.mapred.ReliabilityTest.main(ReliabilityTest.java:504)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72)
> at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:144)
> at org.apache.hadoop.test.MapredTestDriver.run(MapredTestDriver.java:111)
> at org.apache.hadoop.test.MapredTestDriver.main(MapredTestDriver.java:118)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:192)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2716) MR279: MRReliabilityTest job
fails because of missing job-file.
Posted by "Vinod Kumar Vavilapalli (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13078759#comment-13078759 ]
Vinod Kumar Vavilapalli commented on MAPREDUCE-2716:
----------------------------------------------------
Apologies for the delayed reply, I was tied up with one of those bigger patches.
bq. .. that staging area directory seems to include the user name and ".staging" as well as the value from the "yarn.apps.stagingDir"
Good catch, that's correct! In fact, given this, {{ClientServiceDelegate.getJobStatus()}} using the key from configuration directly is actually a bug.
For fixing this, we should be obtaining the job-file by simply using the same API from MRApps in both getJobStatus() and getAllJobs(). Or may be just add another API {{MRApps.getJobFile(Configuration conf, String user, JobID jobID)}}
> MR279: MRReliabilityTest job fails because of missing job-file.
> ---------------------------------------------------------------
>
> Key: MAPREDUCE-2716
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2716
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Jeffrey Naisbitt
> Assignee: Jeffrey Naisbitt
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2716-v2.patch, MAPREDUCE-2716-v3.patch, MAPREDUCE-2716.patch
>
>
> The ApplicationReport should have the jobFile (e.g. hdfs://localhost:9000/tmp/hadoop-<USER>/mapred/staging/<USER>/.staging/job_201107121640_0001/job.xml)
> Without it, jobs such as MRReliabilityTest fail with the following error (caused by the fact that jobFile is hardcoded to "" in TypeConverter.java):
> e.g. java.lang.IllegalArgumentException: Can not create a Path from an empty string
> at org.apache.hadoop.fs.Path.checkPathArg(Path.java:88)
> at org.apache.hadoop.fs.Path.<init>(Path.java:96)
> at org.apache.hadoop.mapred.JobConf.<init>(JobConf.java:445)
> at org.apache.hadoop.mapreduce.Cluster.getJobs(Cluster.java:104)
> at org.apache.hadoop.mapreduce.Cluster.getAllJobs(Cluster.java:218)
> at org.apache.hadoop.mapred.JobClient.getAllJobs(JobClient.java:757)
> at org.apache.hadoop.mapred.JobClient.jobsToComplete(JobClient.java:741)
> at org.apache.hadoop.mapred.ReliabilityTest.runTest(ReliabilityTest.java:219)
> at org.apache.hadoop.mapred.ReliabilityTest.runSleepJobTest(ReliabilityTest.java:133)
> at org.apache.hadoop.mapred.ReliabilityTest.run(ReliabilityTest.java:116)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69)
> at org.apache.hadoop.mapred.ReliabilityTest.main(ReliabilityTest.java:504)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72)
> at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:144)
> at org.apache.hadoop.test.MapredTestDriver.run(MapredTestDriver.java:111)
> at org.apache.hadoop.test.MapredTestDriver.main(MapredTestDriver.java:118)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:192)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2716) MR279: MRReliabilityTest job
fails because of missing job-file.
Posted by "Jeffrey Naisbitt (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13085278#comment-13085278 ]
Jeffrey Naisbitt commented on MAPREDUCE-2716:
---------------------------------------------
I meant to say that it adds the "job name" and user to the JobReport (not the jobFile)
> MR279: MRReliabilityTest job fails because of missing job-file.
> ---------------------------------------------------------------
>
> Key: MAPREDUCE-2716
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2716
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Jeffrey Naisbitt
> Assignee: Jeffrey Naisbitt
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2716-v2.patch, MAPREDUCE-2716-v3.patch, MAPREDUCE-2716-v4.patch, MAPREDUCE-2716-v5.patch, MAPREDUCE-2716-v6.patch, MAPREDUCE-2716.patch
>
>
> The ApplicationReport should have the jobFile (e.g. hdfs://localhost:9000/tmp/hadoop-<USER>/mapred/staging/<USER>/.staging/job_201107121640_0001/job.xml)
> Without it, jobs such as MRReliabilityTest fail with the following error (caused by the fact that jobFile is hardcoded to "" in TypeConverter.java):
> e.g. java.lang.IllegalArgumentException: Can not create a Path from an empty string
> at org.apache.hadoop.fs.Path.checkPathArg(Path.java:88)
> at org.apache.hadoop.fs.Path.<init>(Path.java:96)
> at org.apache.hadoop.mapred.JobConf.<init>(JobConf.java:445)
> at org.apache.hadoop.mapreduce.Cluster.getJobs(Cluster.java:104)
> at org.apache.hadoop.mapreduce.Cluster.getAllJobs(Cluster.java:218)
> at org.apache.hadoop.mapred.JobClient.getAllJobs(JobClient.java:757)
> at org.apache.hadoop.mapred.JobClient.jobsToComplete(JobClient.java:741)
> at org.apache.hadoop.mapred.ReliabilityTest.runTest(ReliabilityTest.java:219)
> at org.apache.hadoop.mapred.ReliabilityTest.runSleepJobTest(ReliabilityTest.java:133)
> at org.apache.hadoop.mapred.ReliabilityTest.run(ReliabilityTest.java:116)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69)
> at org.apache.hadoop.mapred.ReliabilityTest.main(ReliabilityTest.java:504)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72)
> at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:144)
> at org.apache.hadoop.test.MapredTestDriver.run(MapredTestDriver.java:111)
> at org.apache.hadoop.test.MapredTestDriver.main(MapredTestDriver.java:118)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:192)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2716) MR279: MRReliabilityTest job
fails because of missing job-file.
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13096595#comment-13096595 ]
Hudson commented on MAPREDUCE-2716:
-----------------------------------
Integrated in Hadoop-Mapreduce-trunk-Commit #837 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/837/])
MAPREDUCE-2716. MRReliabilityTest job fails because of missing job-file. Contributed by Jeffrey Naisbitt.
vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1164805
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/JobImpl.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/TypeConverter.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/api/records/JobReport.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/api/records/impl/pb/JobReportPBImpl.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/proto/mr_protos.proto
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/TestTypeConverter.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/v2/util/TestMRApps.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/CompletedJob.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/ClientServiceDelegate.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/NotRunningJob.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/ResourceMgrDelegate.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestClientRedirect.java
> MR279: MRReliabilityTest job fails because of missing job-file.
> ---------------------------------------------------------------
>
> Key: MAPREDUCE-2716
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2716
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Jeffrey Naisbitt
> Assignee: Jeffrey Naisbitt
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2716-20110903.txt, MAPREDUCE-2716-v2.patch, MAPREDUCE-2716-v3.patch, MAPREDUCE-2716-v4.patch, MAPREDUCE-2716-v5.patch, MAPREDUCE-2716-v6.patch, MAPREDUCE-2716-v7.patch, MAPREDUCE-2716-v8.patch, MAPREDUCE-2716.patch
>
>
> The ApplicationReport should have the jobFile (e.g. hdfs://localhost:9000/tmp/hadoop-<USER>/mapred/staging/<USER>/.staging/job_201107121640_0001/job.xml)
> Without it, jobs such as MRReliabilityTest fail with the following error (caused by the fact that jobFile is hardcoded to "" in TypeConverter.java):
> e.g. java.lang.IllegalArgumentException: Can not create a Path from an empty string
> at org.apache.hadoop.fs.Path.checkPathArg(Path.java:88)
> at org.apache.hadoop.fs.Path.<init>(Path.java:96)
> at org.apache.hadoop.mapred.JobConf.<init>(JobConf.java:445)
> at org.apache.hadoop.mapreduce.Cluster.getJobs(Cluster.java:104)
> at org.apache.hadoop.mapreduce.Cluster.getAllJobs(Cluster.java:218)
> at org.apache.hadoop.mapred.JobClient.getAllJobs(JobClient.java:757)
> at org.apache.hadoop.mapred.JobClient.jobsToComplete(JobClient.java:741)
> at org.apache.hadoop.mapred.ReliabilityTest.runTest(ReliabilityTest.java:219)
> at org.apache.hadoop.mapred.ReliabilityTest.runSleepJobTest(ReliabilityTest.java:133)
> at org.apache.hadoop.mapred.ReliabilityTest.run(ReliabilityTest.java:116)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69)
> at org.apache.hadoop.mapred.ReliabilityTest.main(ReliabilityTest.java:504)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72)
> at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:144)
> at org.apache.hadoop.test.MapredTestDriver.run(MapredTestDriver.java:111)
> at org.apache.hadoop.test.MapredTestDriver.main(MapredTestDriver.java:118)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:192)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2716) MR279: MRReliabilityTest job
fails because of missing job-file.
Posted by "Jeffrey Naisbitt (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13087263#comment-13087263 ]
Jeffrey Naisbitt commented on MAPREDUCE-2716:
---------------------------------------------
I think you can ignore the part above about adding ".staging" and user. The weird part is just because the default setting includes both, but I see on live clusters that they are still using "/user" for that setting (so it will look normal). I still need to fix the issue with createFakeJobReport needing a valid jobFile.
> MR279: MRReliabilityTest job fails because of missing job-file.
> ---------------------------------------------------------------
>
> Key: MAPREDUCE-2716
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2716
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Jeffrey Naisbitt
> Assignee: Jeffrey Naisbitt
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2716-v2.patch, MAPREDUCE-2716-v3.patch, MAPREDUCE-2716-v4.patch, MAPREDUCE-2716-v5.patch, MAPREDUCE-2716-v6.patch, MAPREDUCE-2716.patch
>
>
> The ApplicationReport should have the jobFile (e.g. hdfs://localhost:9000/tmp/hadoop-<USER>/mapred/staging/<USER>/.staging/job_201107121640_0001/job.xml)
> Without it, jobs such as MRReliabilityTest fail with the following error (caused by the fact that jobFile is hardcoded to "" in TypeConverter.java):
> e.g. java.lang.IllegalArgumentException: Can not create a Path from an empty string
> at org.apache.hadoop.fs.Path.checkPathArg(Path.java:88)
> at org.apache.hadoop.fs.Path.<init>(Path.java:96)
> at org.apache.hadoop.mapred.JobConf.<init>(JobConf.java:445)
> at org.apache.hadoop.mapreduce.Cluster.getJobs(Cluster.java:104)
> at org.apache.hadoop.mapreduce.Cluster.getAllJobs(Cluster.java:218)
> at org.apache.hadoop.mapred.JobClient.getAllJobs(JobClient.java:757)
> at org.apache.hadoop.mapred.JobClient.jobsToComplete(JobClient.java:741)
> at org.apache.hadoop.mapred.ReliabilityTest.runTest(ReliabilityTest.java:219)
> at org.apache.hadoop.mapred.ReliabilityTest.runSleepJobTest(ReliabilityTest.java:133)
> at org.apache.hadoop.mapred.ReliabilityTest.run(ReliabilityTest.java:116)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69)
> at org.apache.hadoop.mapred.ReliabilityTest.main(ReliabilityTest.java:504)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72)
> at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:144)
> at org.apache.hadoop.test.MapredTestDriver.run(MapredTestDriver.java:111)
> at org.apache.hadoop.test.MapredTestDriver.main(MapredTestDriver.java:118)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:192)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2716) MR279: The jobFile is missing
from the ApplicationReport, causing at least the MRReliabilityTest job to
fail
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13070639#comment-13070639 ]
Hadoop QA commented on MAPREDUCE-2716:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12487736/MAPREDUCE-2716-v2.patch
against trunk revision 1150533.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 12 new or modified tests.
-1 patch. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/501//console
This message is automatically generated.
> MR279: The jobFile is missing from the ApplicationReport, causing at least the MRReliabilityTest job to fail
> ------------------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-2716
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2716
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Jeffrey Naisbitt
> Assignee: Jeffrey Naisbitt
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2716-v2.patch, MAPREDUCE-2716.patch
>
>
> The ApplicationReport should have the jobFile (e.g. hdfs://localhost:9000/tmp/hadoop-<USER>/mapred/staging/<USER>/.staging/job_201107121640_0001/job.xml)
> Without it, jobs such as MRReliabilityTest fail with the following error (caused by the fact that jobFile is hardcoded to "" in TypeConverter.java):
> e.g. java.lang.IllegalArgumentException: Can not create a Path from an empty string
> at org.apache.hadoop.fs.Path.checkPathArg(Path.java:88)
> at org.apache.hadoop.fs.Path.<init>(Path.java:96)
> at org.apache.hadoop.mapred.JobConf.<init>(JobConf.java:445)
> at org.apache.hadoop.mapreduce.Cluster.getJobs(Cluster.java:104)
> at org.apache.hadoop.mapreduce.Cluster.getAllJobs(Cluster.java:218)
> at org.apache.hadoop.mapred.JobClient.getAllJobs(JobClient.java:757)
> at org.apache.hadoop.mapred.JobClient.jobsToComplete(JobClient.java:741)
> at org.apache.hadoop.mapred.ReliabilityTest.runTest(ReliabilityTest.java:219)
> at org.apache.hadoop.mapred.ReliabilityTest.runSleepJobTest(ReliabilityTest.java:133)
> at org.apache.hadoop.mapred.ReliabilityTest.run(ReliabilityTest.java:116)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69)
> at org.apache.hadoop.mapred.ReliabilityTest.main(ReliabilityTest.java:504)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72)
> at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:144)
> at org.apache.hadoop.test.MapredTestDriver.run(MapredTestDriver.java:111)
> at org.apache.hadoop.test.MapredTestDriver.main(MapredTestDriver.java:118)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:192)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2716) MR279: MRReliabilityTest job
fails because of missing job-file.
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13096645#comment-13096645 ]
Hudson commented on MAPREDUCE-2716:
-----------------------------------
Integrated in Hadoop-Hdfs-trunk #781 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/781/])
MAPREDUCE-2716. MRReliabilityTest job fails because of missing job-file. Contributed by Jeffrey Naisbitt.
vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1164805
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/JobImpl.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/TypeConverter.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/api/records/JobReport.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/api/records/impl/pb/JobReportPBImpl.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/proto/mr_protos.proto
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/TestTypeConverter.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/v2/util/TestMRApps.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/CompletedJob.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/ClientServiceDelegate.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/NotRunningJob.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/ResourceMgrDelegate.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestClientRedirect.java
> MR279: MRReliabilityTest job fails because of missing job-file.
> ---------------------------------------------------------------
>
> Key: MAPREDUCE-2716
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2716
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Jeffrey Naisbitt
> Assignee: Jeffrey Naisbitt
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2716-20110903.txt, MAPREDUCE-2716-v2.patch, MAPREDUCE-2716-v3.patch, MAPREDUCE-2716-v4.patch, MAPREDUCE-2716-v5.patch, MAPREDUCE-2716-v6.patch, MAPREDUCE-2716-v7.patch, MAPREDUCE-2716-v8.patch, MAPREDUCE-2716.patch
>
>
> The ApplicationReport should have the jobFile (e.g. hdfs://localhost:9000/tmp/hadoop-<USER>/mapred/staging/<USER>/.staging/job_201107121640_0001/job.xml)
> Without it, jobs such as MRReliabilityTest fail with the following error (caused by the fact that jobFile is hardcoded to "" in TypeConverter.java):
> e.g. java.lang.IllegalArgumentException: Can not create a Path from an empty string
> at org.apache.hadoop.fs.Path.checkPathArg(Path.java:88)
> at org.apache.hadoop.fs.Path.<init>(Path.java:96)
> at org.apache.hadoop.mapred.JobConf.<init>(JobConf.java:445)
> at org.apache.hadoop.mapreduce.Cluster.getJobs(Cluster.java:104)
> at org.apache.hadoop.mapreduce.Cluster.getAllJobs(Cluster.java:218)
> at org.apache.hadoop.mapred.JobClient.getAllJobs(JobClient.java:757)
> at org.apache.hadoop.mapred.JobClient.jobsToComplete(JobClient.java:741)
> at org.apache.hadoop.mapred.ReliabilityTest.runTest(ReliabilityTest.java:219)
> at org.apache.hadoop.mapred.ReliabilityTest.runSleepJobTest(ReliabilityTest.java:133)
> at org.apache.hadoop.mapred.ReliabilityTest.run(ReliabilityTest.java:116)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69)
> at org.apache.hadoop.mapred.ReliabilityTest.main(ReliabilityTest.java:504)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72)
> at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:144)
> at org.apache.hadoop.test.MapredTestDriver.run(MapredTestDriver.java:111)
> at org.apache.hadoop.test.MapredTestDriver.main(MapredTestDriver.java:118)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:192)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2716) MR279: MRReliabilityTest job
fails because of missing job-file.
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13096651#comment-13096651 ]
Hudson commented on MAPREDUCE-2716:
-----------------------------------
Integrated in Hadoop-Mapreduce-trunk #805 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/805/])
MAPREDUCE-2716. MRReliabilityTest job fails because of missing job-file. Contributed by Jeffrey Naisbitt.
vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1164805
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/JobImpl.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/TypeConverter.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/api/records/JobReport.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/api/records/impl/pb/JobReportPBImpl.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/proto/mr_protos.proto
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/TestTypeConverter.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/v2/util/TestMRApps.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/CompletedJob.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/ClientServiceDelegate.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/NotRunningJob.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/ResourceMgrDelegate.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestClientRedirect.java
> MR279: MRReliabilityTest job fails because of missing job-file.
> ---------------------------------------------------------------
>
> Key: MAPREDUCE-2716
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2716
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Jeffrey Naisbitt
> Assignee: Jeffrey Naisbitt
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2716-20110903.txt, MAPREDUCE-2716-v2.patch, MAPREDUCE-2716-v3.patch, MAPREDUCE-2716-v4.patch, MAPREDUCE-2716-v5.patch, MAPREDUCE-2716-v6.patch, MAPREDUCE-2716-v7.patch, MAPREDUCE-2716-v8.patch, MAPREDUCE-2716.patch
>
>
> The ApplicationReport should have the jobFile (e.g. hdfs://localhost:9000/tmp/hadoop-<USER>/mapred/staging/<USER>/.staging/job_201107121640_0001/job.xml)
> Without it, jobs such as MRReliabilityTest fail with the following error (caused by the fact that jobFile is hardcoded to "" in TypeConverter.java):
> e.g. java.lang.IllegalArgumentException: Can not create a Path from an empty string
> at org.apache.hadoop.fs.Path.checkPathArg(Path.java:88)
> at org.apache.hadoop.fs.Path.<init>(Path.java:96)
> at org.apache.hadoop.mapred.JobConf.<init>(JobConf.java:445)
> at org.apache.hadoop.mapreduce.Cluster.getJobs(Cluster.java:104)
> at org.apache.hadoop.mapreduce.Cluster.getAllJobs(Cluster.java:218)
> at org.apache.hadoop.mapred.JobClient.getAllJobs(JobClient.java:757)
> at org.apache.hadoop.mapred.JobClient.jobsToComplete(JobClient.java:741)
> at org.apache.hadoop.mapred.ReliabilityTest.runTest(ReliabilityTest.java:219)
> at org.apache.hadoop.mapred.ReliabilityTest.runSleepJobTest(ReliabilityTest.java:133)
> at org.apache.hadoop.mapred.ReliabilityTest.run(ReliabilityTest.java:116)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69)
> at org.apache.hadoop.mapred.ReliabilityTest.main(ReliabilityTest.java:504)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72)
> at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:144)
> at org.apache.hadoop.test.MapredTestDriver.run(MapredTestDriver.java:111)
> at org.apache.hadoop.test.MapredTestDriver.main(MapredTestDriver.java:118)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:192)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2716) MR279: MRReliabilityTest job
fails because of missing job-file.
Posted by "Jeffrey Naisbitt (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jeffrey Naisbitt updated MAPREDUCE-2716:
----------------------------------------
Attachment: MAPREDUCE-2716-v8.patch
v8 fixes the issue with TestClientRedirect's infinite loop and adds a null check to make future debugging easier and to prevent the loop in cases like this (where the application does not have the user set).
> MR279: MRReliabilityTest job fails because of missing job-file.
> ---------------------------------------------------------------
>
> Key: MAPREDUCE-2716
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2716
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Jeffrey Naisbitt
> Assignee: Jeffrey Naisbitt
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2716-v2.patch, MAPREDUCE-2716-v3.patch, MAPREDUCE-2716-v4.patch, MAPREDUCE-2716-v5.patch, MAPREDUCE-2716-v6.patch, MAPREDUCE-2716-v7.patch, MAPREDUCE-2716-v8.patch, MAPREDUCE-2716.patch
>
>
> The ApplicationReport should have the jobFile (e.g. hdfs://localhost:9000/tmp/hadoop-<USER>/mapred/staging/<USER>/.staging/job_201107121640_0001/job.xml)
> Without it, jobs such as MRReliabilityTest fail with the following error (caused by the fact that jobFile is hardcoded to "" in TypeConverter.java):
> e.g. java.lang.IllegalArgumentException: Can not create a Path from an empty string
> at org.apache.hadoop.fs.Path.checkPathArg(Path.java:88)
> at org.apache.hadoop.fs.Path.<init>(Path.java:96)
> at org.apache.hadoop.mapred.JobConf.<init>(JobConf.java:445)
> at org.apache.hadoop.mapreduce.Cluster.getJobs(Cluster.java:104)
> at org.apache.hadoop.mapreduce.Cluster.getAllJobs(Cluster.java:218)
> at org.apache.hadoop.mapred.JobClient.getAllJobs(JobClient.java:757)
> at org.apache.hadoop.mapred.JobClient.jobsToComplete(JobClient.java:741)
> at org.apache.hadoop.mapred.ReliabilityTest.runTest(ReliabilityTest.java:219)
> at org.apache.hadoop.mapred.ReliabilityTest.runSleepJobTest(ReliabilityTest.java:133)
> at org.apache.hadoop.mapred.ReliabilityTest.run(ReliabilityTest.java:116)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69)
> at org.apache.hadoop.mapred.ReliabilityTest.main(ReliabilityTest.java:504)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72)
> at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:144)
> at org.apache.hadoop.test.MapredTestDriver.run(MapredTestDriver.java:111)
> at org.apache.hadoop.test.MapredTestDriver.main(MapredTestDriver.java:118)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:192)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2716) MR279: MRReliabilityTest job
fails because of missing job-file.
Posted by "Jeffrey Naisbitt (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jeffrey Naisbitt updated MAPREDUCE-2716:
----------------------------------------
Attachment: MAPREDUCE-2716-v6.patch
New patch addresses user issues and adds the jobfile and user to the JobReport.
> MR279: MRReliabilityTest job fails because of missing job-file.
> ---------------------------------------------------------------
>
> Key: MAPREDUCE-2716
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2716
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Jeffrey Naisbitt
> Assignee: Jeffrey Naisbitt
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2716-v2.patch, MAPREDUCE-2716-v3.patch, MAPREDUCE-2716-v4.patch, MAPREDUCE-2716-v5.patch, MAPREDUCE-2716-v6.patch, MAPREDUCE-2716.patch
>
>
> The ApplicationReport should have the jobFile (e.g. hdfs://localhost:9000/tmp/hadoop-<USER>/mapred/staging/<USER>/.staging/job_201107121640_0001/job.xml)
> Without it, jobs such as MRReliabilityTest fail with the following error (caused by the fact that jobFile is hardcoded to "" in TypeConverter.java):
> e.g. java.lang.IllegalArgumentException: Can not create a Path from an empty string
> at org.apache.hadoop.fs.Path.checkPathArg(Path.java:88)
> at org.apache.hadoop.fs.Path.<init>(Path.java:96)
> at org.apache.hadoop.mapred.JobConf.<init>(JobConf.java:445)
> at org.apache.hadoop.mapreduce.Cluster.getJobs(Cluster.java:104)
> at org.apache.hadoop.mapreduce.Cluster.getAllJobs(Cluster.java:218)
> at org.apache.hadoop.mapred.JobClient.getAllJobs(JobClient.java:757)
> at org.apache.hadoop.mapred.JobClient.jobsToComplete(JobClient.java:741)
> at org.apache.hadoop.mapred.ReliabilityTest.runTest(ReliabilityTest.java:219)
> at org.apache.hadoop.mapred.ReliabilityTest.runSleepJobTest(ReliabilityTest.java:133)
> at org.apache.hadoop.mapred.ReliabilityTest.run(ReliabilityTest.java:116)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69)
> at org.apache.hadoop.mapred.ReliabilityTest.main(ReliabilityTest.java:504)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72)
> at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:144)
> at org.apache.hadoop.test.MapredTestDriver.run(MapredTestDriver.java:111)
> at org.apache.hadoop.test.MapredTestDriver.main(MapredTestDriver.java:118)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:192)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2716) MR279: The jobFile is missing
from the ApplicationReport, causing at least the MRReliabilityTest job to
fail
Posted by "Jeffrey Naisbitt (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jeffrey Naisbitt updated MAPREDUCE-2716:
----------------------------------------
Summary: MR279: The jobFile is missing from the ApplicationReport, causing at least the MRReliabilityTest job to fail (was: The jobFile is missing from the ApplicationReport, causing at least the MRReliabilityTest job to fail)
> MR279: The jobFile is missing from the ApplicationReport, causing at least the MRReliabilityTest job to fail
> ------------------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-2716
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2716
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Jeffrey Naisbitt
> Assignee: Jeffrey Naisbitt
> Fix For: 0.23.0
>
>
> The ApplicationReport should have the jobFile (e.g. hdfs://localhost:9000/tmp/hadoop-<USER>/mapred/staging/<USER>/.staging/job_201107121640_0001/job.xml)
> Without it, jobs such as MRReliabilityTest fail with the following error (caused by the fact that jobFile is hardcoded to "" in TypeConverter.java):
> e.g. java.lang.IllegalArgumentException: Can not create a Path from an empty string
> at org.apache.hadoop.fs.Path.checkPathArg(Path.java:88)
> at org.apache.hadoop.fs.Path.<init>(Path.java:96)
> at org.apache.hadoop.mapred.JobConf.<init>(JobConf.java:445)
> at org.apache.hadoop.mapreduce.Cluster.getJobs(Cluster.java:104)
> at org.apache.hadoop.mapreduce.Cluster.getAllJobs(Cluster.java:218)
> at org.apache.hadoop.mapred.JobClient.getAllJobs(JobClient.java:757)
> at org.apache.hadoop.mapred.JobClient.jobsToComplete(JobClient.java:741)
> at org.apache.hadoop.mapred.ReliabilityTest.runTest(ReliabilityTest.java:219)
> at org.apache.hadoop.mapred.ReliabilityTest.runSleepJobTest(ReliabilityTest.java:133)
> at org.apache.hadoop.mapred.ReliabilityTest.run(ReliabilityTest.java:116)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69)
> at org.apache.hadoop.mapred.ReliabilityTest.main(ReliabilityTest.java:504)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72)
> at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:144)
> at org.apache.hadoop.test.MapredTestDriver.run(MapredTestDriver.java:111)
> at org.apache.hadoop.test.MapredTestDriver.main(MapredTestDriver.java:118)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:192)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2716) MR279: MRReliabilityTest job
fails because of missing job-file.
Posted by "Jeffrey Naisbitt (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13083185#comment-13083185 ]
Jeffrey Naisbitt commented on MAPREDUCE-2716:
---------------------------------------------
By the way, all tests pass with 'mvn test', except for TestLeafQueue - which is unrelated to my patch and fails without my change.
> MR279: MRReliabilityTest job fails because of missing job-file.
> ---------------------------------------------------------------
>
> Key: MAPREDUCE-2716
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2716
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Jeffrey Naisbitt
> Assignee: Jeffrey Naisbitt
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2716-v2.patch, MAPREDUCE-2716-v3.patch, MAPREDUCE-2716-v4.patch, MAPREDUCE-2716-v5.patch, MAPREDUCE-2716.patch
>
>
> The ApplicationReport should have the jobFile (e.g. hdfs://localhost:9000/tmp/hadoop-<USER>/mapred/staging/<USER>/.staging/job_201107121640_0001/job.xml)
> Without it, jobs such as MRReliabilityTest fail with the following error (caused by the fact that jobFile is hardcoded to "" in TypeConverter.java):
> e.g. java.lang.IllegalArgumentException: Can not create a Path from an empty string
> at org.apache.hadoop.fs.Path.checkPathArg(Path.java:88)
> at org.apache.hadoop.fs.Path.<init>(Path.java:96)
> at org.apache.hadoop.mapred.JobConf.<init>(JobConf.java:445)
> at org.apache.hadoop.mapreduce.Cluster.getJobs(Cluster.java:104)
> at org.apache.hadoop.mapreduce.Cluster.getAllJobs(Cluster.java:218)
> at org.apache.hadoop.mapred.JobClient.getAllJobs(JobClient.java:757)
> at org.apache.hadoop.mapred.JobClient.jobsToComplete(JobClient.java:741)
> at org.apache.hadoop.mapred.ReliabilityTest.runTest(ReliabilityTest.java:219)
> at org.apache.hadoop.mapred.ReliabilityTest.runSleepJobTest(ReliabilityTest.java:133)
> at org.apache.hadoop.mapred.ReliabilityTest.run(ReliabilityTest.java:116)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69)
> at org.apache.hadoop.mapred.ReliabilityTest.main(ReliabilityTest.java:504)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72)
> at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:144)
> at org.apache.hadoop.test.MapredTestDriver.run(MapredTestDriver.java:111)
> at org.apache.hadoop.test.MapredTestDriver.main(MapredTestDriver.java:118)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:192)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2716) MR279: MRReliabilityTest job
fails because of missing job-file.
Posted by "Jeffrey Naisbitt (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jeffrey Naisbitt updated MAPREDUCE-2716:
----------------------------------------
Status: Open (was: Patch Available)
We found some more bugs in the existing code and new code that are causing the jobFile to use the incorrect user. I will update the patch to address these issues.
> MR279: MRReliabilityTest job fails because of missing job-file.
> ---------------------------------------------------------------
>
> Key: MAPREDUCE-2716
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2716
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Jeffrey Naisbitt
> Assignee: Jeffrey Naisbitt
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2716-v2.patch, MAPREDUCE-2716-v3.patch, MAPREDUCE-2716-v4.patch, MAPREDUCE-2716-v5.patch, MAPREDUCE-2716.patch
>
>
> The ApplicationReport should have the jobFile (e.g. hdfs://localhost:9000/tmp/hadoop-<USER>/mapred/staging/<USER>/.staging/job_201107121640_0001/job.xml)
> Without it, jobs such as MRReliabilityTest fail with the following error (caused by the fact that jobFile is hardcoded to "" in TypeConverter.java):
> e.g. java.lang.IllegalArgumentException: Can not create a Path from an empty string
> at org.apache.hadoop.fs.Path.checkPathArg(Path.java:88)
> at org.apache.hadoop.fs.Path.<init>(Path.java:96)
> at org.apache.hadoop.mapred.JobConf.<init>(JobConf.java:445)
> at org.apache.hadoop.mapreduce.Cluster.getJobs(Cluster.java:104)
> at org.apache.hadoop.mapreduce.Cluster.getAllJobs(Cluster.java:218)
> at org.apache.hadoop.mapred.JobClient.getAllJobs(JobClient.java:757)
> at org.apache.hadoop.mapred.JobClient.jobsToComplete(JobClient.java:741)
> at org.apache.hadoop.mapred.ReliabilityTest.runTest(ReliabilityTest.java:219)
> at org.apache.hadoop.mapred.ReliabilityTest.runSleepJobTest(ReliabilityTest.java:133)
> at org.apache.hadoop.mapred.ReliabilityTest.run(ReliabilityTest.java:116)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69)
> at org.apache.hadoop.mapred.ReliabilityTest.main(ReliabilityTest.java:504)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72)
> at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:144)
> at org.apache.hadoop.test.MapredTestDriver.run(MapredTestDriver.java:111)
> at org.apache.hadoop.test.MapredTestDriver.main(MapredTestDriver.java:118)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:192)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2716) MR279: MRReliabilityTest job
fails because of missing job-file.
Posted by "Vinod Kumar Vavilapalli (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13095320#comment-13095320 ]
Vinod Kumar Vavilapalli commented on MAPREDUCE-2716:
----------------------------------------------------
TestClientRedirect is going into an infinite loop on my box too with the patch. Didn't see this before, should be related to the patch.
> MR279: MRReliabilityTest job fails because of missing job-file.
> ---------------------------------------------------------------
>
> Key: MAPREDUCE-2716
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2716
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Jeffrey Naisbitt
> Assignee: Jeffrey Naisbitt
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2716-v2.patch, MAPREDUCE-2716-v3.patch, MAPREDUCE-2716-v4.patch, MAPREDUCE-2716-v5.patch, MAPREDUCE-2716-v6.patch, MAPREDUCE-2716-v7.patch, MAPREDUCE-2716.patch
>
>
> The ApplicationReport should have the jobFile (e.g. hdfs://localhost:9000/tmp/hadoop-<USER>/mapred/staging/<USER>/.staging/job_201107121640_0001/job.xml)
> Without it, jobs such as MRReliabilityTest fail with the following error (caused by the fact that jobFile is hardcoded to "" in TypeConverter.java):
> e.g. java.lang.IllegalArgumentException: Can not create a Path from an empty string
> at org.apache.hadoop.fs.Path.checkPathArg(Path.java:88)
> at org.apache.hadoop.fs.Path.<init>(Path.java:96)
> at org.apache.hadoop.mapred.JobConf.<init>(JobConf.java:445)
> at org.apache.hadoop.mapreduce.Cluster.getJobs(Cluster.java:104)
> at org.apache.hadoop.mapreduce.Cluster.getAllJobs(Cluster.java:218)
> at org.apache.hadoop.mapred.JobClient.getAllJobs(JobClient.java:757)
> at org.apache.hadoop.mapred.JobClient.jobsToComplete(JobClient.java:741)
> at org.apache.hadoop.mapred.ReliabilityTest.runTest(ReliabilityTest.java:219)
> at org.apache.hadoop.mapred.ReliabilityTest.runSleepJobTest(ReliabilityTest.java:133)
> at org.apache.hadoop.mapred.ReliabilityTest.run(ReliabilityTest.java:116)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69)
> at org.apache.hadoop.mapred.ReliabilityTest.main(ReliabilityTest.java:504)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72)
> at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:144)
> at org.apache.hadoop.test.MapredTestDriver.run(MapredTestDriver.java:111)
> at org.apache.hadoop.test.MapredTestDriver.main(MapredTestDriver.java:118)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:192)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2716) MR279: MRReliabilityTest job
fails because of missing job-file.
Posted by "Vinod Kumar Vavilapalli (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13092925#comment-13092925 ]
Vinod Kumar Vavilapalli commented on MAPREDUCE-2716:
----------------------------------------------------
Should replace "mapreduce/mr-client" with hadoop-mapreduce-client in the patch.
Even after that, because of MAPREDUCE-2807, ClientServiceDelegate changed much, so the patch needs regeneration w.r.t that.
bq. First, the createFakeJobReport calls in ClientServiceDelegate.getJobStatus seem to need a valid jobfile since the MRReliability test uses that codepath and expects a valid jobFile.
I guess this should be easy to fix, isn't it? Or is there is some complication? Note that {{NotRunningJob}} class is the new fake-report where this change should go.
> MR279: MRReliabilityTest job fails because of missing job-file.
> ---------------------------------------------------------------
>
> Key: MAPREDUCE-2716
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2716
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Jeffrey Naisbitt
> Assignee: Jeffrey Naisbitt
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2716-v2.patch, MAPREDUCE-2716-v3.patch, MAPREDUCE-2716-v4.patch, MAPREDUCE-2716-v5.patch, MAPREDUCE-2716-v6.patch, MAPREDUCE-2716.patch
>
>
> The ApplicationReport should have the jobFile (e.g. hdfs://localhost:9000/tmp/hadoop-<USER>/mapred/staging/<USER>/.staging/job_201107121640_0001/job.xml)
> Without it, jobs such as MRReliabilityTest fail with the following error (caused by the fact that jobFile is hardcoded to "" in TypeConverter.java):
> e.g. java.lang.IllegalArgumentException: Can not create a Path from an empty string
> at org.apache.hadoop.fs.Path.checkPathArg(Path.java:88)
> at org.apache.hadoop.fs.Path.<init>(Path.java:96)
> at org.apache.hadoop.mapred.JobConf.<init>(JobConf.java:445)
> at org.apache.hadoop.mapreduce.Cluster.getJobs(Cluster.java:104)
> at org.apache.hadoop.mapreduce.Cluster.getAllJobs(Cluster.java:218)
> at org.apache.hadoop.mapred.JobClient.getAllJobs(JobClient.java:757)
> at org.apache.hadoop.mapred.JobClient.jobsToComplete(JobClient.java:741)
> at org.apache.hadoop.mapred.ReliabilityTest.runTest(ReliabilityTest.java:219)
> at org.apache.hadoop.mapred.ReliabilityTest.runSleepJobTest(ReliabilityTest.java:133)
> at org.apache.hadoop.mapred.ReliabilityTest.run(ReliabilityTest.java:116)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69)
> at org.apache.hadoop.mapred.ReliabilityTest.main(ReliabilityTest.java:504)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72)
> at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:144)
> at org.apache.hadoop.test.MapredTestDriver.run(MapredTestDriver.java:111)
> at org.apache.hadoop.test.MapredTestDriver.main(MapredTestDriver.java:118)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:192)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2716) MR279: MRReliabilityTest job
fails because of missing job-file.
Posted by "Jeffrey Naisbitt (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jeffrey Naisbitt updated MAPREDUCE-2716:
----------------------------------------
Status: Open (was: Patch Available)
> MR279: MRReliabilityTest job fails because of missing job-file.
> ---------------------------------------------------------------
>
> Key: MAPREDUCE-2716
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2716
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Jeffrey Naisbitt
> Assignee: Jeffrey Naisbitt
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2716-v2.patch, MAPREDUCE-2716-v3.patch, MAPREDUCE-2716-v4.patch, MAPREDUCE-2716-v5.patch, MAPREDUCE-2716-v6.patch, MAPREDUCE-2716-v7.patch, MAPREDUCE-2716.patch
>
>
> The ApplicationReport should have the jobFile (e.g. hdfs://localhost:9000/tmp/hadoop-<USER>/mapred/staging/<USER>/.staging/job_201107121640_0001/job.xml)
> Without it, jobs such as MRReliabilityTest fail with the following error (caused by the fact that jobFile is hardcoded to "" in TypeConverter.java):
> e.g. java.lang.IllegalArgumentException: Can not create a Path from an empty string
> at org.apache.hadoop.fs.Path.checkPathArg(Path.java:88)
> at org.apache.hadoop.fs.Path.<init>(Path.java:96)
> at org.apache.hadoop.mapred.JobConf.<init>(JobConf.java:445)
> at org.apache.hadoop.mapreduce.Cluster.getJobs(Cluster.java:104)
> at org.apache.hadoop.mapreduce.Cluster.getAllJobs(Cluster.java:218)
> at org.apache.hadoop.mapred.JobClient.getAllJobs(JobClient.java:757)
> at org.apache.hadoop.mapred.JobClient.jobsToComplete(JobClient.java:741)
> at org.apache.hadoop.mapred.ReliabilityTest.runTest(ReliabilityTest.java:219)
> at org.apache.hadoop.mapred.ReliabilityTest.runSleepJobTest(ReliabilityTest.java:133)
> at org.apache.hadoop.mapred.ReliabilityTest.run(ReliabilityTest.java:116)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69)
> at org.apache.hadoop.mapred.ReliabilityTest.main(ReliabilityTest.java:504)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72)
> at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:144)
> at org.apache.hadoop.test.MapredTestDriver.run(MapredTestDriver.java:111)
> at org.apache.hadoop.test.MapredTestDriver.main(MapredTestDriver.java:118)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:192)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira