You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Anupam Seth (Created) (JIRA)" <ji...@apache.org> on 2012/02/09 21:46:57 UTC
[jira] [Created] (MAPREDUCE-3843) CLONE - JobHistoryServer should
be started by bin/mapred and not by bin/yarn
CLONE - JobHistoryServer should be started by bin/mapred and not by bin/yarn
----------------------------------------------------------------------------
Key: MAPREDUCE-3843
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3843
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: jobhistoryserver, mrv2
Affects Versions: 0.23.0
Reporter: Anupam Seth
Assignee: Jonathan Eagles
Priority: Blocker
Fix For: 0.23.1
JobHistoryServer belongs to mapreduce land.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3843) Job summary log file found
missing on the RM host
Posted by "Anupam Seth (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Anupam Seth updated MAPREDUCE-3843:
-----------------------------------
Attachment: MAPREDUCE-3843.patch
> Job summary log file found missing on the RM host
> -------------------------------------------------
>
> Key: MAPREDUCE-3843
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3843
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobhistoryserver, mrv2
> Affects Versions: 0.23.0
> Reporter: Anupam Seth
> Assignee: Anupam Seth
> Priority: Critical
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3843.patch, MAPREDUCE-3843.patch, MAPREDUCE-3843.patch
>
>
> This bug was found by Phil Su as part of our testing.
> After MAPREDUCE-3354 went in, the Job summary log file seems to have gone missing on the
> RM host.
> The job summary log appears to be interspersed in yarn-mapredqa-historyserver-<host>.out.
> e.g.
> 12/02/09 15:57:21 INFO jobhistory.JobSummary:
> jobId=job_1328658619341_0011,submitTime=1328802904381,launchTime=1328802909977,firstMapTaskLaunchTime=1328802912116,firstReduceTaskLaunchTime=1328802915074,finishTime=1328802933797,resourc
> esPerMap=1024,resourcesPerReduce=2048,numMaps=10,numReduces=10,user=hadoopqa,queue=default,status=KILLED,mapSlotSeconds=0,reduceSlotSeconds=0
> 1) On the RM with older hadoop version where the job summary log does not exist
> mapredqa 10903 0.0 1.2 1424404 210240 ? Sl Feb07 0:19 /home/gs/java/jdk64/current/bin/java -Xmx1000m
> -Djava.net.preferIPv4Stack=true
> -Djava.library.path=/home/gs/gridre/theoden/share/hadoop/lib/native/Linux-amd64-64:/
> home/gs/gridre/theoden/share/hadoop/lib/native/Linux-amd64-64 -Dhadoop.log.dir=/home/gs/var/log/mapredqa
> -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/home/gs/gridre/theoden/share/hadoop -Dhadoop.id.str=mapredqa
> -Dhadoop
> .root.logger=INFO,console
> -Djava.library.path=/home/gs/gridre/theoden/share/hadoop/lib/native/Linux-amd64-64:/home/gs/gridre/theoden/share/hadoop/lib/native/Linux-amd64-64:/home/gs/gridre/theoden/share/hadoop/lib/nat
> ive -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -Dmapred.jobsummary.logger=INFO,console
> -Dhadoop.security.logger=INFO,NullAppender org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer
> 2) On the RM with older hadoop version where the job summary log exists
> mapredqa 24851 0.0 0.5 1463280 90516 ? Sl Jan25 0:37 /home/gs/java/jdk64/current/bin/java
> -Dproc_historyserver -Xmx1000m -Dmapred.jobsummary.logger=INFO,JSA -Dyarn.log.dir=/home/gs/var/log/mapredqa
> -Dyarn.log.file=yarn.log -Dyarn.home.dir= -Dyarn.id.str= -Dyarn.root.logger=INFO,console
> -Djava.library.path=/home/gs/gridre/shelob/share/hadoop/lib/native/Linux-amd64-64
> -Dyarn.policy.file=hadoop-policy.xml -Dyarn.log.dir=/home/gs/var/log/mapredqa
> -Dyarn.log.file=yarn-mapredqa-historyserver-<host>.log -Dyarn.home.dir= -Dyarn.id.str=mapredqa
> -Dyarn.root.logger=INFO,DRFA -Djava.library.path=/home/gs/gridre/shelob/share/hadoop/lib/native/Linux-amd64-64
> -Dyarn.policy.file=hadoop-policy.xml -Dmapred.jobsummary.logger=INFO,JSA -Dhadoop.log.dir=/home/gs/var/log/mapredqa
> -Dyarn.log.dir=/home/gs/var/log/mapredqa
> -Dhadoop.log.file=yarn-mapredqa-historyserver-<host>.log
> -Dyarn.log.file=yarn-mapredqa-historyserver-<host>.log
> -Dyarn.home.dir=/home/gs/gridre/shelob/share/hadoop -Dhadoop.root.logger=INFO,DRFA -Dyarn.root.logger=INFO,DRFA
> -Djava.library.path=/home/gs/gridre/shelob/share/hadoop/lib/native/Linux-amd64-64 -classpath
> /home/gs/gridre/shelob/conf/hadoop:/home/gs/gridre/shelob/conf/hadoop:/home/gs/gridre/shelob/conf/hadoop:/home/gs/gridre/shelob/conf/hadoop:/home/gs/gridre/shelob/share/hadoop/share/hadoop/common/lib/*:/home/gs/gridre/shelob/share/hadoop/share/hadoop/common/*:/home/gs/gridre/shelob/share/hadoop/hadoop-*-capacity-scheduler.jar:/home/gs/gridre/shelob/share/hadoop/hadoop-*-capacity-scheduler.jar:/home/gs/gridre/shelob/share/hadoop/hadoop-*-capacity-scheduler.jar:/home/gs/gridre/shelob/share/hadoop/share/hadoop/hdfs:/home/gs/gridre/shelob/share/hadoop/share/hadoop/hdfs/lib/*:/home/gs/gridre/shelob/share/hadoop/share/hadoop/hdfs/*:/home/gs/gridre/shelob/share/hadoop/share/hadoop/mapreduce/lib/*:/home/gs/gridre/shelob/share/hadoop/share/hadoop/mapreduce/*:/home/gs/java/jdk64/current/lib/tools.jar:/home/gs/gridre/shelob/share/hadoop/share/hadoop/mapreduce/*:/home/gs/gridre/shelob/share/hadoop/share/hadoop/mapreduce/lib/*
> org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer
> 1) On the RM with older hadoop version where the job summary log does not exist
> jobhistory ps shows using the option:
> -Dmapred.jobsummary.logger=INFO,console
> 2) On the RM with older hadoop version where the job summary log exists
> jobhistory ps shows using the option:
> -Dmapred.jobsummary.logger=INFO,JSA
> -Dmapred.jobsummary.logger=INFO,JSA
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3843) Job summary log file found
missing on the RM host
Posted by "Anupam Seth (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Anupam Seth updated MAPREDUCE-3843:
-----------------------------------
Status: Patch Available (was: Open)
> Job summary log file found missing on the RM host
> -------------------------------------------------
>
> Key: MAPREDUCE-3843
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3843
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobhistoryserver, mrv2
> Affects Versions: 0.23.0
> Reporter: Anupam Seth
> Assignee: Anupam Seth
> Priority: Critical
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3843.patch, MAPREDUCE-3843.patch, MAPREDUCE-3843.patch, MAPREDUCE-3843.patch
>
>
> This bug was found by Phil Su as part of our testing.
> After MAPREDUCE-3354 went in, the Job summary log file seems to have gone missing on the
> RM host.
> The job summary log appears to be interspersed in yarn-mapredqa-historyserver-<host>.out.
> e.g.
> 12/02/09 15:57:21 INFO jobhistory.JobSummary:
> jobId=job_1328658619341_0011,submitTime=1328802904381,launchTime=1328802909977,firstMapTaskLaunchTime=1328802912116,firstReduceTaskLaunchTime=1328802915074,finishTime=1328802933797,resourc
> esPerMap=1024,resourcesPerReduce=2048,numMaps=10,numReduces=10,user=hadoopqa,queue=default,status=KILLED,mapSlotSeconds=0,reduceSlotSeconds=0
> 1) On the RM with older hadoop version where the job summary log does not exist
> mapredqa 10903 0.0 1.2 1424404 210240 ? Sl Feb07 0:19 /home/gs/java/jdk64/current/bin/java -Xmx1000m
> -Djava.net.preferIPv4Stack=true
> -Djava.library.path=/home/gs/gridre/theoden/share/hadoop/lib/native/Linux-amd64-64:/
> home/gs/gridre/theoden/share/hadoop/lib/native/Linux-amd64-64 -Dhadoop.log.dir=/home/gs/var/log/mapredqa
> -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/home/gs/gridre/theoden/share/hadoop -Dhadoop.id.str=mapredqa
> -Dhadoop
> .root.logger=INFO,console
> -Djava.library.path=/home/gs/gridre/theoden/share/hadoop/lib/native/Linux-amd64-64:/home/gs/gridre/theoden/share/hadoop/lib/native/Linux-amd64-64:/home/gs/gridre/theoden/share/hadoop/lib/nat
> ive -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -Dmapred.jobsummary.logger=INFO,console
> -Dhadoop.security.logger=INFO,NullAppender org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer
> 2) On the RM with older hadoop version where the job summary log exists
> mapredqa 24851 0.0 0.5 1463280 90516 ? Sl Jan25 0:37 /home/gs/java/jdk64/current/bin/java
> -Dproc_historyserver -Xmx1000m -Dmapred.jobsummary.logger=INFO,JSA -Dyarn.log.dir=/home/gs/var/log/mapredqa
> -Dyarn.log.file=yarn.log -Dyarn.home.dir= -Dyarn.id.str= -Dyarn.root.logger=INFO,console
> -Djava.library.path=/home/gs/gridre/shelob/share/hadoop/lib/native/Linux-amd64-64
> -Dyarn.policy.file=hadoop-policy.xml -Dyarn.log.dir=/home/gs/var/log/mapredqa
> -Dyarn.log.file=yarn-mapredqa-historyserver-<host>.log -Dyarn.home.dir= -Dyarn.id.str=mapredqa
> -Dyarn.root.logger=INFO,DRFA -Djava.library.path=/home/gs/gridre/shelob/share/hadoop/lib/native/Linux-amd64-64
> -Dyarn.policy.file=hadoop-policy.xml -Dmapred.jobsummary.logger=INFO,JSA -Dhadoop.log.dir=/home/gs/var/log/mapredqa
> -Dyarn.log.dir=/home/gs/var/log/mapredqa
> -Dhadoop.log.file=yarn-mapredqa-historyserver-<host>.log
> -Dyarn.log.file=yarn-mapredqa-historyserver-<host>.log
> -Dyarn.home.dir=/home/gs/gridre/shelob/share/hadoop -Dhadoop.root.logger=INFO,DRFA -Dyarn.root.logger=INFO,DRFA
> -Djava.library.path=/home/gs/gridre/shelob/share/hadoop/lib/native/Linux-amd64-64 -classpath
> /home/gs/gridre/shelob/conf/hadoop:/home/gs/gridre/shelob/conf/hadoop:/home/gs/gridre/shelob/conf/hadoop:/home/gs/gridre/shelob/conf/hadoop:/home/gs/gridre/shelob/share/hadoop/share/hadoop/common/lib/*:/home/gs/gridre/shelob/share/hadoop/share/hadoop/common/*:/home/gs/gridre/shelob/share/hadoop/hadoop-*-capacity-scheduler.jar:/home/gs/gridre/shelob/share/hadoop/hadoop-*-capacity-scheduler.jar:/home/gs/gridre/shelob/share/hadoop/hadoop-*-capacity-scheduler.jar:/home/gs/gridre/shelob/share/hadoop/share/hadoop/hdfs:/home/gs/gridre/shelob/share/hadoop/share/hadoop/hdfs/lib/*:/home/gs/gridre/shelob/share/hadoop/share/hadoop/hdfs/*:/home/gs/gridre/shelob/share/hadoop/share/hadoop/mapreduce/lib/*:/home/gs/gridre/shelob/share/hadoop/share/hadoop/mapreduce/*:/home/gs/java/jdk64/current/lib/tools.jar:/home/gs/gridre/shelob/share/hadoop/share/hadoop/mapreduce/*:/home/gs/gridre/shelob/share/hadoop/share/hadoop/mapreduce/lib/*
> org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer
> 1) On the RM with older hadoop version where the job summary log does not exist
> jobhistory ps shows using the option:
> -Dmapred.jobsummary.logger=INFO,console
> 2) On the RM with older hadoop version where the job summary log exists
> jobhistory ps shows using the option:
> -Dmapred.jobsummary.logger=INFO,JSA
> -Dmapred.jobsummary.logger=INFO,JSA
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3843) Job summary log file found
missing on the RM host
Posted by "Anupam Seth (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Anupam Seth updated MAPREDUCE-3843:
-----------------------------------
Status: Patch Available (was: Open)
> Job summary log file found missing on the RM host
> -------------------------------------------------
>
> Key: MAPREDUCE-3843
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3843
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobhistoryserver, mrv2
> Affects Versions: 0.23.0
> Reporter: Anupam Seth
> Assignee: Anupam Seth
> Priority: Critical
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3843.patch, MAPREDUCE-3843.patch, MAPREDUCE-3843.patch
>
>
> This bug was found by Phil Su as part of our testing.
> After MAPREDUCE-3354 went in, the Job summary log file seems to have gone missing on the
> RM host.
> The job summary log appears to be interspersed in yarn-mapredqa-historyserver-<host>.out.
> e.g.
> 12/02/09 15:57:21 INFO jobhistory.JobSummary:
> jobId=job_1328658619341_0011,submitTime=1328802904381,launchTime=1328802909977,firstMapTaskLaunchTime=1328802912116,firstReduceTaskLaunchTime=1328802915074,finishTime=1328802933797,resourc
> esPerMap=1024,resourcesPerReduce=2048,numMaps=10,numReduces=10,user=hadoopqa,queue=default,status=KILLED,mapSlotSeconds=0,reduceSlotSeconds=0
> 1) On the RM with older hadoop version where the job summary log does not exist
> mapredqa 10903 0.0 1.2 1424404 210240 ? Sl Feb07 0:19 /home/gs/java/jdk64/current/bin/java -Xmx1000m
> -Djava.net.preferIPv4Stack=true
> -Djava.library.path=/home/gs/gridre/theoden/share/hadoop/lib/native/Linux-amd64-64:/
> home/gs/gridre/theoden/share/hadoop/lib/native/Linux-amd64-64 -Dhadoop.log.dir=/home/gs/var/log/mapredqa
> -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/home/gs/gridre/theoden/share/hadoop -Dhadoop.id.str=mapredqa
> -Dhadoop
> .root.logger=INFO,console
> -Djava.library.path=/home/gs/gridre/theoden/share/hadoop/lib/native/Linux-amd64-64:/home/gs/gridre/theoden/share/hadoop/lib/native/Linux-amd64-64:/home/gs/gridre/theoden/share/hadoop/lib/nat
> ive -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -Dmapred.jobsummary.logger=INFO,console
> -Dhadoop.security.logger=INFO,NullAppender org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer
> 2) On the RM with older hadoop version where the job summary log exists
> mapredqa 24851 0.0 0.5 1463280 90516 ? Sl Jan25 0:37 /home/gs/java/jdk64/current/bin/java
> -Dproc_historyserver -Xmx1000m -Dmapred.jobsummary.logger=INFO,JSA -Dyarn.log.dir=/home/gs/var/log/mapredqa
> -Dyarn.log.file=yarn.log -Dyarn.home.dir= -Dyarn.id.str= -Dyarn.root.logger=INFO,console
> -Djava.library.path=/home/gs/gridre/shelob/share/hadoop/lib/native/Linux-amd64-64
> -Dyarn.policy.file=hadoop-policy.xml -Dyarn.log.dir=/home/gs/var/log/mapredqa
> -Dyarn.log.file=yarn-mapredqa-historyserver-<host>.log -Dyarn.home.dir= -Dyarn.id.str=mapredqa
> -Dyarn.root.logger=INFO,DRFA -Djava.library.path=/home/gs/gridre/shelob/share/hadoop/lib/native/Linux-amd64-64
> -Dyarn.policy.file=hadoop-policy.xml -Dmapred.jobsummary.logger=INFO,JSA -Dhadoop.log.dir=/home/gs/var/log/mapredqa
> -Dyarn.log.dir=/home/gs/var/log/mapredqa
> -Dhadoop.log.file=yarn-mapredqa-historyserver-<host>.log
> -Dyarn.log.file=yarn-mapredqa-historyserver-<host>.log
> -Dyarn.home.dir=/home/gs/gridre/shelob/share/hadoop -Dhadoop.root.logger=INFO,DRFA -Dyarn.root.logger=INFO,DRFA
> -Djava.library.path=/home/gs/gridre/shelob/share/hadoop/lib/native/Linux-amd64-64 -classpath
> /home/gs/gridre/shelob/conf/hadoop:/home/gs/gridre/shelob/conf/hadoop:/home/gs/gridre/shelob/conf/hadoop:/home/gs/gridre/shelob/conf/hadoop:/home/gs/gridre/shelob/share/hadoop/share/hadoop/common/lib/*:/home/gs/gridre/shelob/share/hadoop/share/hadoop/common/*:/home/gs/gridre/shelob/share/hadoop/hadoop-*-capacity-scheduler.jar:/home/gs/gridre/shelob/share/hadoop/hadoop-*-capacity-scheduler.jar:/home/gs/gridre/shelob/share/hadoop/hadoop-*-capacity-scheduler.jar:/home/gs/gridre/shelob/share/hadoop/share/hadoop/hdfs:/home/gs/gridre/shelob/share/hadoop/share/hadoop/hdfs/lib/*:/home/gs/gridre/shelob/share/hadoop/share/hadoop/hdfs/*:/home/gs/gridre/shelob/share/hadoop/share/hadoop/mapreduce/lib/*:/home/gs/gridre/shelob/share/hadoop/share/hadoop/mapreduce/*:/home/gs/java/jdk64/current/lib/tools.jar:/home/gs/gridre/shelob/share/hadoop/share/hadoop/mapreduce/*:/home/gs/gridre/shelob/share/hadoop/share/hadoop/mapreduce/lib/*
> org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer
> 1) On the RM with older hadoop version where the job summary log does not exist
> jobhistory ps shows using the option:
> -Dmapred.jobsummary.logger=INFO,console
> 2) On the RM with older hadoop version where the job summary log exists
> jobhistory ps shows using the option:
> -Dmapred.jobsummary.logger=INFO,JSA
> -Dmapred.jobsummary.logger=INFO,JSA
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3843) Job summary log file found
missing on the RM host
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13205896#comment-13205896 ]
Hudson commented on MAPREDUCE-3843:
-----------------------------------
Integrated in Hadoop-Common-0.23-Commit #537 (See [https://builds.apache.org/job/Hadoop-Common-0.23-Commit/537/])
merge -r 1242975:1242976 from trunk. FIXES: MAPREDUCE-3843
tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1242977
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/bin/mr-jobhistory-daemon.sh
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/ClusterSetup.apt.vm
> Job summary log file found missing on the RM host
> -------------------------------------------------
>
> Key: MAPREDUCE-3843
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3843
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobhistoryserver, mrv2
> Affects Versions: 0.23.0
> Reporter: Anupam Seth
> Assignee: Anupam Seth
> Priority: Critical
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3843.patch, MAPREDUCE-3843.patch, MAPREDUCE-3843.patch, MAPREDUCE-3843.patch, MAPREDUCE-3843.patch, MAPREDUCE-3843.patch
>
>
> This bug was found by Phil Su as part of our testing.
> After MAPREDUCE-3354 went in, the Job summary log file seems to have gone missing on the
> RM host.
> The job summary log appears to be interspersed in yarn-mapredqa-historyserver-<host>.out.
> e.g.
> 12/02/09 15:57:21 INFO jobhistory.JobSummary:
> jobId=job_1328658619341_0011,submitTime=1328802904381,launchTime=1328802909977,firstMapTaskLaunchTime=1328802912116,firstReduceTaskLaunchTime=1328802915074,finishTime=1328802933797,resourc
> esPerMap=1024,resourcesPerReduce=2048,numMaps=10,numReduces=10,user=hadoopqa,queue=default,status=KILLED,mapSlotSeconds=0,reduceSlotSeconds=0
> 1) On the RM with older hadoop version where the job summary log does not exist
> mapredqa 10903 0.0 1.2 1424404 210240 ? Sl Feb07 0:19 /home/gs/java/jdk64/current/bin/java -Xmx1000m
> -Djava.net.preferIPv4Stack=true
> -Djava.library.path=/home/gs/gridre/theoden/share/hadoop/lib/native/Linux-amd64-64:/
> home/gs/gridre/theoden/share/hadoop/lib/native/Linux-amd64-64 -Dhadoop.log.dir=/home/gs/var/log/mapredqa
> -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/home/gs/gridre/theoden/share/hadoop -Dhadoop.id.str=mapredqa
> -Dhadoop
> .root.logger=INFO,console
> -Djava.library.path=/home/gs/gridre/theoden/share/hadoop/lib/native/Linux-amd64-64:/home/gs/gridre/theoden/share/hadoop/lib/native/Linux-amd64-64:/home/gs/gridre/theoden/share/hadoop/lib/nat
> ive -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -Dmapred.jobsummary.logger=INFO,console
> -Dhadoop.security.logger=INFO,NullAppender org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer
> 2) On the RM with older hadoop version where the job summary log exists
> mapredqa 24851 0.0 0.5 1463280 90516 ? Sl Jan25 0:37 /home/gs/java/jdk64/current/bin/java
> -Dproc_historyserver -Xmx1000m -Dmapred.jobsummary.logger=INFO,JSA -Dyarn.log.dir=/home/gs/var/log/mapredqa
> -Dyarn.log.file=yarn.log -Dyarn.home.dir= -Dyarn.id.str= -Dyarn.root.logger=INFO,console
> -Djava.library.path=/home/gs/gridre/shelob/share/hadoop/lib/native/Linux-amd64-64
> -Dyarn.policy.file=hadoop-policy.xml -Dyarn.log.dir=/home/gs/var/log/mapredqa
> -Dyarn.log.file=yarn-mapredqa-historyserver-<host>.log -Dyarn.home.dir= -Dyarn.id.str=mapredqa
> -Dyarn.root.logger=INFO,DRFA -Djava.library.path=/home/gs/gridre/shelob/share/hadoop/lib/native/Linux-amd64-64
> -Dyarn.policy.file=hadoop-policy.xml -Dmapred.jobsummary.logger=INFO,JSA -Dhadoop.log.dir=/home/gs/var/log/mapredqa
> -Dyarn.log.dir=/home/gs/var/log/mapredqa
> -Dhadoop.log.file=yarn-mapredqa-historyserver-<host>.log
> -Dyarn.log.file=yarn-mapredqa-historyserver-<host>.log
> -Dyarn.home.dir=/home/gs/gridre/shelob/share/hadoop -Dhadoop.root.logger=INFO,DRFA -Dyarn.root.logger=INFO,DRFA
> -Djava.library.path=/home/gs/gridre/shelob/share/hadoop/lib/native/Linux-amd64-64 -classpath
> /home/gs/gridre/shelob/conf/hadoop:/home/gs/gridre/shelob/conf/hadoop:/home/gs/gridre/shelob/conf/hadoop:/home/gs/gridre/shelob/conf/hadoop:/home/gs/gridre/shelob/share/hadoop/share/hadoop/common/lib/*:/home/gs/gridre/shelob/share/hadoop/share/hadoop/common/*:/home/gs/gridre/shelob/share/hadoop/hadoop-*-capacity-scheduler.jar:/home/gs/gridre/shelob/share/hadoop/hadoop-*-capacity-scheduler.jar:/home/gs/gridre/shelob/share/hadoop/hadoop-*-capacity-scheduler.jar:/home/gs/gridre/shelob/share/hadoop/share/hadoop/hdfs:/home/gs/gridre/shelob/share/hadoop/share/hadoop/hdfs/lib/*:/home/gs/gridre/shelob/share/hadoop/share/hadoop/hdfs/*:/home/gs/gridre/shelob/share/hadoop/share/hadoop/mapreduce/lib/*:/home/gs/gridre/shelob/share/hadoop/share/hadoop/mapreduce/*:/home/gs/java/jdk64/current/lib/tools.jar:/home/gs/gridre/shelob/share/hadoop/share/hadoop/mapreduce/*:/home/gs/gridre/shelob/share/hadoop/share/hadoop/mapreduce/lib/*
> org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer
> 1) On the RM with older hadoop version where the job summary log does not exist
> jobhistory ps shows using the option:
> -Dmapred.jobsummary.logger=INFO,console
> 2) On the RM with older hadoop version where the job summary log exists
> jobhistory ps shows using the option:
> -Dmapred.jobsummary.logger=INFO,JSA
> -Dmapred.jobsummary.logger=INFO,JSA
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3843) Job summary log file found
missing on the RM host
Posted by "Thomas Graves (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Thomas Graves updated MAPREDUCE-3843:
-------------------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
Thanks Anupam!! I committed this to trunk, branch-0.23, and the branch-0.23.1
> Job summary log file found missing on the RM host
> -------------------------------------------------
>
> Key: MAPREDUCE-3843
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3843
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobhistoryserver, mrv2
> Affects Versions: 0.23.0
> Reporter: Anupam Seth
> Assignee: Anupam Seth
> Priority: Critical
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3843.patch, MAPREDUCE-3843.patch, MAPREDUCE-3843.patch, MAPREDUCE-3843.patch, MAPREDUCE-3843.patch, MAPREDUCE-3843.patch
>
>
> This bug was found by Phil Su as part of our testing.
> After MAPREDUCE-3354 went in, the Job summary log file seems to have gone missing on the
> RM host.
> The job summary log appears to be interspersed in yarn-mapredqa-historyserver-<host>.out.
> e.g.
> 12/02/09 15:57:21 INFO jobhistory.JobSummary:
> jobId=job_1328658619341_0011,submitTime=1328802904381,launchTime=1328802909977,firstMapTaskLaunchTime=1328802912116,firstReduceTaskLaunchTime=1328802915074,finishTime=1328802933797,resourc
> esPerMap=1024,resourcesPerReduce=2048,numMaps=10,numReduces=10,user=hadoopqa,queue=default,status=KILLED,mapSlotSeconds=0,reduceSlotSeconds=0
> 1) On the RM with older hadoop version where the job summary log does not exist
> mapredqa 10903 0.0 1.2 1424404 210240 ? Sl Feb07 0:19 /home/gs/java/jdk64/current/bin/java -Xmx1000m
> -Djava.net.preferIPv4Stack=true
> -Djava.library.path=/home/gs/gridre/theoden/share/hadoop/lib/native/Linux-amd64-64:/
> home/gs/gridre/theoden/share/hadoop/lib/native/Linux-amd64-64 -Dhadoop.log.dir=/home/gs/var/log/mapredqa
> -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/home/gs/gridre/theoden/share/hadoop -Dhadoop.id.str=mapredqa
> -Dhadoop
> .root.logger=INFO,console
> -Djava.library.path=/home/gs/gridre/theoden/share/hadoop/lib/native/Linux-amd64-64:/home/gs/gridre/theoden/share/hadoop/lib/native/Linux-amd64-64:/home/gs/gridre/theoden/share/hadoop/lib/nat
> ive -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -Dmapred.jobsummary.logger=INFO,console
> -Dhadoop.security.logger=INFO,NullAppender org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer
> 2) On the RM with older hadoop version where the job summary log exists
> mapredqa 24851 0.0 0.5 1463280 90516 ? Sl Jan25 0:37 /home/gs/java/jdk64/current/bin/java
> -Dproc_historyserver -Xmx1000m -Dmapred.jobsummary.logger=INFO,JSA -Dyarn.log.dir=/home/gs/var/log/mapredqa
> -Dyarn.log.file=yarn.log -Dyarn.home.dir= -Dyarn.id.str= -Dyarn.root.logger=INFO,console
> -Djava.library.path=/home/gs/gridre/shelob/share/hadoop/lib/native/Linux-amd64-64
> -Dyarn.policy.file=hadoop-policy.xml -Dyarn.log.dir=/home/gs/var/log/mapredqa
> -Dyarn.log.file=yarn-mapredqa-historyserver-<host>.log -Dyarn.home.dir= -Dyarn.id.str=mapredqa
> -Dyarn.root.logger=INFO,DRFA -Djava.library.path=/home/gs/gridre/shelob/share/hadoop/lib/native/Linux-amd64-64
> -Dyarn.policy.file=hadoop-policy.xml -Dmapred.jobsummary.logger=INFO,JSA -Dhadoop.log.dir=/home/gs/var/log/mapredqa
> -Dyarn.log.dir=/home/gs/var/log/mapredqa
> -Dhadoop.log.file=yarn-mapredqa-historyserver-<host>.log
> -Dyarn.log.file=yarn-mapredqa-historyserver-<host>.log
> -Dyarn.home.dir=/home/gs/gridre/shelob/share/hadoop -Dhadoop.root.logger=INFO,DRFA -Dyarn.root.logger=INFO,DRFA
> -Djava.library.path=/home/gs/gridre/shelob/share/hadoop/lib/native/Linux-amd64-64 -classpath
> /home/gs/gridre/shelob/conf/hadoop:/home/gs/gridre/shelob/conf/hadoop:/home/gs/gridre/shelob/conf/hadoop:/home/gs/gridre/shelob/conf/hadoop:/home/gs/gridre/shelob/share/hadoop/share/hadoop/common/lib/*:/home/gs/gridre/shelob/share/hadoop/share/hadoop/common/*:/home/gs/gridre/shelob/share/hadoop/hadoop-*-capacity-scheduler.jar:/home/gs/gridre/shelob/share/hadoop/hadoop-*-capacity-scheduler.jar:/home/gs/gridre/shelob/share/hadoop/hadoop-*-capacity-scheduler.jar:/home/gs/gridre/shelob/share/hadoop/share/hadoop/hdfs:/home/gs/gridre/shelob/share/hadoop/share/hadoop/hdfs/lib/*:/home/gs/gridre/shelob/share/hadoop/share/hadoop/hdfs/*:/home/gs/gridre/shelob/share/hadoop/share/hadoop/mapreduce/lib/*:/home/gs/gridre/shelob/share/hadoop/share/hadoop/mapreduce/*:/home/gs/java/jdk64/current/lib/tools.jar:/home/gs/gridre/shelob/share/hadoop/share/hadoop/mapreduce/*:/home/gs/gridre/shelob/share/hadoop/share/hadoop/mapreduce/lib/*
> org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer
> 1) On the RM with older hadoop version where the job summary log does not exist
> jobhistory ps shows using the option:
> -Dmapred.jobsummary.logger=INFO,console
> 2) On the RM with older hadoop version where the job summary log exists
> jobhistory ps shows using the option:
> -Dmapred.jobsummary.logger=INFO,JSA
> -Dmapred.jobsummary.logger=INFO,JSA
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3843) Job summary log file found
missing on the RM host
Posted by "Thomas Graves (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13205846#comment-13205846 ]
Thomas Graves commented on MAPREDUCE-3843:
------------------------------------------
+1, waiting for jenkins to +1 and then will commit. Note the previous test failure isn't related to this jira - see MAPREDUCE-3852
> Job summary log file found missing on the RM host
> -------------------------------------------------
>
> Key: MAPREDUCE-3843
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3843
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobhistoryserver, mrv2
> Affects Versions: 0.23.0
> Reporter: Anupam Seth
> Assignee: Anupam Seth
> Priority: Critical
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3843.patch, MAPREDUCE-3843.patch, MAPREDUCE-3843.patch, MAPREDUCE-3843.patch, MAPREDUCE-3843.patch, MAPREDUCE-3843.patch
>
>
> This bug was found by Phil Su as part of our testing.
> After MAPREDUCE-3354 went in, the Job summary log file seems to have gone missing on the
> RM host.
> The job summary log appears to be interspersed in yarn-mapredqa-historyserver-<host>.out.
> e.g.
> 12/02/09 15:57:21 INFO jobhistory.JobSummary:
> jobId=job_1328658619341_0011,submitTime=1328802904381,launchTime=1328802909977,firstMapTaskLaunchTime=1328802912116,firstReduceTaskLaunchTime=1328802915074,finishTime=1328802933797,resourc
> esPerMap=1024,resourcesPerReduce=2048,numMaps=10,numReduces=10,user=hadoopqa,queue=default,status=KILLED,mapSlotSeconds=0,reduceSlotSeconds=0
> 1) On the RM with older hadoop version where the job summary log does not exist
> mapredqa 10903 0.0 1.2 1424404 210240 ? Sl Feb07 0:19 /home/gs/java/jdk64/current/bin/java -Xmx1000m
> -Djava.net.preferIPv4Stack=true
> -Djava.library.path=/home/gs/gridre/theoden/share/hadoop/lib/native/Linux-amd64-64:/
> home/gs/gridre/theoden/share/hadoop/lib/native/Linux-amd64-64 -Dhadoop.log.dir=/home/gs/var/log/mapredqa
> -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/home/gs/gridre/theoden/share/hadoop -Dhadoop.id.str=mapredqa
> -Dhadoop
> .root.logger=INFO,console
> -Djava.library.path=/home/gs/gridre/theoden/share/hadoop/lib/native/Linux-amd64-64:/home/gs/gridre/theoden/share/hadoop/lib/native/Linux-amd64-64:/home/gs/gridre/theoden/share/hadoop/lib/nat
> ive -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -Dmapred.jobsummary.logger=INFO,console
> -Dhadoop.security.logger=INFO,NullAppender org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer
> 2) On the RM with older hadoop version where the job summary log exists
> mapredqa 24851 0.0 0.5 1463280 90516 ? Sl Jan25 0:37 /home/gs/java/jdk64/current/bin/java
> -Dproc_historyserver -Xmx1000m -Dmapred.jobsummary.logger=INFO,JSA -Dyarn.log.dir=/home/gs/var/log/mapredqa
> -Dyarn.log.file=yarn.log -Dyarn.home.dir= -Dyarn.id.str= -Dyarn.root.logger=INFO,console
> -Djava.library.path=/home/gs/gridre/shelob/share/hadoop/lib/native/Linux-amd64-64
> -Dyarn.policy.file=hadoop-policy.xml -Dyarn.log.dir=/home/gs/var/log/mapredqa
> -Dyarn.log.file=yarn-mapredqa-historyserver-<host>.log -Dyarn.home.dir= -Dyarn.id.str=mapredqa
> -Dyarn.root.logger=INFO,DRFA -Djava.library.path=/home/gs/gridre/shelob/share/hadoop/lib/native/Linux-amd64-64
> -Dyarn.policy.file=hadoop-policy.xml -Dmapred.jobsummary.logger=INFO,JSA -Dhadoop.log.dir=/home/gs/var/log/mapredqa
> -Dyarn.log.dir=/home/gs/var/log/mapredqa
> -Dhadoop.log.file=yarn-mapredqa-historyserver-<host>.log
> -Dyarn.log.file=yarn-mapredqa-historyserver-<host>.log
> -Dyarn.home.dir=/home/gs/gridre/shelob/share/hadoop -Dhadoop.root.logger=INFO,DRFA -Dyarn.root.logger=INFO,DRFA
> -Djava.library.path=/home/gs/gridre/shelob/share/hadoop/lib/native/Linux-amd64-64 -classpath
> /home/gs/gridre/shelob/conf/hadoop:/home/gs/gridre/shelob/conf/hadoop:/home/gs/gridre/shelob/conf/hadoop:/home/gs/gridre/shelob/conf/hadoop:/home/gs/gridre/shelob/share/hadoop/share/hadoop/common/lib/*:/home/gs/gridre/shelob/share/hadoop/share/hadoop/common/*:/home/gs/gridre/shelob/share/hadoop/hadoop-*-capacity-scheduler.jar:/home/gs/gridre/shelob/share/hadoop/hadoop-*-capacity-scheduler.jar:/home/gs/gridre/shelob/share/hadoop/hadoop-*-capacity-scheduler.jar:/home/gs/gridre/shelob/share/hadoop/share/hadoop/hdfs:/home/gs/gridre/shelob/share/hadoop/share/hadoop/hdfs/lib/*:/home/gs/gridre/shelob/share/hadoop/share/hadoop/hdfs/*:/home/gs/gridre/shelob/share/hadoop/share/hadoop/mapreduce/lib/*:/home/gs/gridre/shelob/share/hadoop/share/hadoop/mapreduce/*:/home/gs/java/jdk64/current/lib/tools.jar:/home/gs/gridre/shelob/share/hadoop/share/hadoop/mapreduce/*:/home/gs/gridre/shelob/share/hadoop/share/hadoop/mapreduce/lib/*
> org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer
> 1) On the RM with older hadoop version where the job summary log does not exist
> jobhistory ps shows using the option:
> -Dmapred.jobsummary.logger=INFO,console
> 2) On the RM with older hadoop version where the job summary log exists
> jobhistory ps shows using the option:
> -Dmapred.jobsummary.logger=INFO,JSA
> -Dmapred.jobsummary.logger=INFO,JSA
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3843) Job summary log file found
missing on the RM host
Posted by "Anupam Seth (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Anupam Seth updated MAPREDUCE-3843:
-----------------------------------
Status: Open (was: Patch Available)
> Job summary log file found missing on the RM host
> -------------------------------------------------
>
> Key: MAPREDUCE-3843
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3843
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobhistoryserver, mrv2
> Affects Versions: 0.23.0
> Reporter: Anupam Seth
> Assignee: Anupam Seth
> Priority: Critical
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3843.patch, MAPREDUCE-3843.patch, MAPREDUCE-3843.patch, MAPREDUCE-3843.patch
>
>
> This bug was found by Phil Su as part of our testing.
> After MAPREDUCE-3354 went in, the Job summary log file seems to have gone missing on the
> RM host.
> The job summary log appears to be interspersed in yarn-mapredqa-historyserver-<host>.out.
> e.g.
> 12/02/09 15:57:21 INFO jobhistory.JobSummary:
> jobId=job_1328658619341_0011,submitTime=1328802904381,launchTime=1328802909977,firstMapTaskLaunchTime=1328802912116,firstReduceTaskLaunchTime=1328802915074,finishTime=1328802933797,resourc
> esPerMap=1024,resourcesPerReduce=2048,numMaps=10,numReduces=10,user=hadoopqa,queue=default,status=KILLED,mapSlotSeconds=0,reduceSlotSeconds=0
> 1) On the RM with older hadoop version where the job summary log does not exist
> mapredqa 10903 0.0 1.2 1424404 210240 ? Sl Feb07 0:19 /home/gs/java/jdk64/current/bin/java -Xmx1000m
> -Djava.net.preferIPv4Stack=true
> -Djava.library.path=/home/gs/gridre/theoden/share/hadoop/lib/native/Linux-amd64-64:/
> home/gs/gridre/theoden/share/hadoop/lib/native/Linux-amd64-64 -Dhadoop.log.dir=/home/gs/var/log/mapredqa
> -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/home/gs/gridre/theoden/share/hadoop -Dhadoop.id.str=mapredqa
> -Dhadoop
> .root.logger=INFO,console
> -Djava.library.path=/home/gs/gridre/theoden/share/hadoop/lib/native/Linux-amd64-64:/home/gs/gridre/theoden/share/hadoop/lib/native/Linux-amd64-64:/home/gs/gridre/theoden/share/hadoop/lib/nat
> ive -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -Dmapred.jobsummary.logger=INFO,console
> -Dhadoop.security.logger=INFO,NullAppender org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer
> 2) On the RM with older hadoop version where the job summary log exists
> mapredqa 24851 0.0 0.5 1463280 90516 ? Sl Jan25 0:37 /home/gs/java/jdk64/current/bin/java
> -Dproc_historyserver -Xmx1000m -Dmapred.jobsummary.logger=INFO,JSA -Dyarn.log.dir=/home/gs/var/log/mapredqa
> -Dyarn.log.file=yarn.log -Dyarn.home.dir= -Dyarn.id.str= -Dyarn.root.logger=INFO,console
> -Djava.library.path=/home/gs/gridre/shelob/share/hadoop/lib/native/Linux-amd64-64
> -Dyarn.policy.file=hadoop-policy.xml -Dyarn.log.dir=/home/gs/var/log/mapredqa
> -Dyarn.log.file=yarn-mapredqa-historyserver-<host>.log -Dyarn.home.dir= -Dyarn.id.str=mapredqa
> -Dyarn.root.logger=INFO,DRFA -Djava.library.path=/home/gs/gridre/shelob/share/hadoop/lib/native/Linux-amd64-64
> -Dyarn.policy.file=hadoop-policy.xml -Dmapred.jobsummary.logger=INFO,JSA -Dhadoop.log.dir=/home/gs/var/log/mapredqa
> -Dyarn.log.dir=/home/gs/var/log/mapredqa
> -Dhadoop.log.file=yarn-mapredqa-historyserver-<host>.log
> -Dyarn.log.file=yarn-mapredqa-historyserver-<host>.log
> -Dyarn.home.dir=/home/gs/gridre/shelob/share/hadoop -Dhadoop.root.logger=INFO,DRFA -Dyarn.root.logger=INFO,DRFA
> -Djava.library.path=/home/gs/gridre/shelob/share/hadoop/lib/native/Linux-amd64-64 -classpath
> /home/gs/gridre/shelob/conf/hadoop:/home/gs/gridre/shelob/conf/hadoop:/home/gs/gridre/shelob/conf/hadoop:/home/gs/gridre/shelob/conf/hadoop:/home/gs/gridre/shelob/share/hadoop/share/hadoop/common/lib/*:/home/gs/gridre/shelob/share/hadoop/share/hadoop/common/*:/home/gs/gridre/shelob/share/hadoop/hadoop-*-capacity-scheduler.jar:/home/gs/gridre/shelob/share/hadoop/hadoop-*-capacity-scheduler.jar:/home/gs/gridre/shelob/share/hadoop/hadoop-*-capacity-scheduler.jar:/home/gs/gridre/shelob/share/hadoop/share/hadoop/hdfs:/home/gs/gridre/shelob/share/hadoop/share/hadoop/hdfs/lib/*:/home/gs/gridre/shelob/share/hadoop/share/hadoop/hdfs/*:/home/gs/gridre/shelob/share/hadoop/share/hadoop/mapreduce/lib/*:/home/gs/gridre/shelob/share/hadoop/share/hadoop/mapreduce/*:/home/gs/java/jdk64/current/lib/tools.jar:/home/gs/gridre/shelob/share/hadoop/share/hadoop/mapreduce/*:/home/gs/gridre/shelob/share/hadoop/share/hadoop/mapreduce/lib/*
> org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer
> 1) On the RM with older hadoop version where the job summary log does not exist
> jobhistory ps shows using the option:
> -Dmapred.jobsummary.logger=INFO,console
> 2) On the RM with older hadoop version where the job summary log exists
> jobhistory ps shows using the option:
> -Dmapred.jobsummary.logger=INFO,JSA
> -Dmapred.jobsummary.logger=INFO,JSA
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3843) Job summary log file found
missing on the RM host
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13205802#comment-13205802 ]
Hadoop QA commented on MAPREDUCE-3843:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12514166/MAPREDUCE-3843.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests:
org.apache.hadoop.yarn.util.TestLinuxResourceCalculatorPlugin
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1839//testReport/
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1839//console
This message is automatically generated.
> Job summary log file found missing on the RM host
> -------------------------------------------------
>
> Key: MAPREDUCE-3843
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3843
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobhistoryserver, mrv2
> Affects Versions: 0.23.0
> Reporter: Anupam Seth
> Assignee: Anupam Seth
> Priority: Critical
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3843.patch, MAPREDUCE-3843.patch, MAPREDUCE-3843.patch, MAPREDUCE-3843.patch, MAPREDUCE-3843.patch
>
>
> This bug was found by Phil Su as part of our testing.
> After MAPREDUCE-3354 went in, the Job summary log file seems to have gone missing on the
> RM host.
> The job summary log appears to be interspersed in yarn-mapredqa-historyserver-<host>.out.
> e.g.
> 12/02/09 15:57:21 INFO jobhistory.JobSummary:
> jobId=job_1328658619341_0011,submitTime=1328802904381,launchTime=1328802909977,firstMapTaskLaunchTime=1328802912116,firstReduceTaskLaunchTime=1328802915074,finishTime=1328802933797,resourc
> esPerMap=1024,resourcesPerReduce=2048,numMaps=10,numReduces=10,user=hadoopqa,queue=default,status=KILLED,mapSlotSeconds=0,reduceSlotSeconds=0
> 1) On the RM with older hadoop version where the job summary log does not exist
> mapredqa 10903 0.0 1.2 1424404 210240 ? Sl Feb07 0:19 /home/gs/java/jdk64/current/bin/java -Xmx1000m
> -Djava.net.preferIPv4Stack=true
> -Djava.library.path=/home/gs/gridre/theoden/share/hadoop/lib/native/Linux-amd64-64:/
> home/gs/gridre/theoden/share/hadoop/lib/native/Linux-amd64-64 -Dhadoop.log.dir=/home/gs/var/log/mapredqa
> -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/home/gs/gridre/theoden/share/hadoop -Dhadoop.id.str=mapredqa
> -Dhadoop
> .root.logger=INFO,console
> -Djava.library.path=/home/gs/gridre/theoden/share/hadoop/lib/native/Linux-amd64-64:/home/gs/gridre/theoden/share/hadoop/lib/native/Linux-amd64-64:/home/gs/gridre/theoden/share/hadoop/lib/nat
> ive -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -Dmapred.jobsummary.logger=INFO,console
> -Dhadoop.security.logger=INFO,NullAppender org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer
> 2) On the RM with older hadoop version where the job summary log exists
> mapredqa 24851 0.0 0.5 1463280 90516 ? Sl Jan25 0:37 /home/gs/java/jdk64/current/bin/java
> -Dproc_historyserver -Xmx1000m -Dmapred.jobsummary.logger=INFO,JSA -Dyarn.log.dir=/home/gs/var/log/mapredqa
> -Dyarn.log.file=yarn.log -Dyarn.home.dir= -Dyarn.id.str= -Dyarn.root.logger=INFO,console
> -Djava.library.path=/home/gs/gridre/shelob/share/hadoop/lib/native/Linux-amd64-64
> -Dyarn.policy.file=hadoop-policy.xml -Dyarn.log.dir=/home/gs/var/log/mapredqa
> -Dyarn.log.file=yarn-mapredqa-historyserver-<host>.log -Dyarn.home.dir= -Dyarn.id.str=mapredqa
> -Dyarn.root.logger=INFO,DRFA -Djava.library.path=/home/gs/gridre/shelob/share/hadoop/lib/native/Linux-amd64-64
> -Dyarn.policy.file=hadoop-policy.xml -Dmapred.jobsummary.logger=INFO,JSA -Dhadoop.log.dir=/home/gs/var/log/mapredqa
> -Dyarn.log.dir=/home/gs/var/log/mapredqa
> -Dhadoop.log.file=yarn-mapredqa-historyserver-<host>.log
> -Dyarn.log.file=yarn-mapredqa-historyserver-<host>.log
> -Dyarn.home.dir=/home/gs/gridre/shelob/share/hadoop -Dhadoop.root.logger=INFO,DRFA -Dyarn.root.logger=INFO,DRFA
> -Djava.library.path=/home/gs/gridre/shelob/share/hadoop/lib/native/Linux-amd64-64 -classpath
> /home/gs/gridre/shelob/conf/hadoop:/home/gs/gridre/shelob/conf/hadoop:/home/gs/gridre/shelob/conf/hadoop:/home/gs/gridre/shelob/conf/hadoop:/home/gs/gridre/shelob/share/hadoop/share/hadoop/common/lib/*:/home/gs/gridre/shelob/share/hadoop/share/hadoop/common/*:/home/gs/gridre/shelob/share/hadoop/hadoop-*-capacity-scheduler.jar:/home/gs/gridre/shelob/share/hadoop/hadoop-*-capacity-scheduler.jar:/home/gs/gridre/shelob/share/hadoop/hadoop-*-capacity-scheduler.jar:/home/gs/gridre/shelob/share/hadoop/share/hadoop/hdfs:/home/gs/gridre/shelob/share/hadoop/share/hadoop/hdfs/lib/*:/home/gs/gridre/shelob/share/hadoop/share/hadoop/hdfs/*:/home/gs/gridre/shelob/share/hadoop/share/hadoop/mapreduce/lib/*:/home/gs/gridre/shelob/share/hadoop/share/hadoop/mapreduce/*:/home/gs/java/jdk64/current/lib/tools.jar:/home/gs/gridre/shelob/share/hadoop/share/hadoop/mapreduce/*:/home/gs/gridre/shelob/share/hadoop/share/hadoop/mapreduce/lib/*
> org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer
> 1) On the RM with older hadoop version where the job summary log does not exist
> jobhistory ps shows using the option:
> -Dmapred.jobsummary.logger=INFO,console
> 2) On the RM with older hadoop version where the job summary log exists
> jobhistory ps shows using the option:
> -Dmapred.jobsummary.logger=INFO,JSA
> -Dmapred.jobsummary.logger=INFO,JSA
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3843) Job summary log file found
missing on the RM host
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13205557#comment-13205557 ]
Hadoop QA commented on MAPREDUCE-3843:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12514118/MAPREDUCE-3843.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests:
org.apache.hadoop.yarn.util.TestLinuxResourceCalculatorPlugin
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1834//testReport/
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1834//console
This message is automatically generated.
> Job summary log file found missing on the RM host
> -------------------------------------------------
>
> Key: MAPREDUCE-3843
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3843
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobhistoryserver, mrv2
> Affects Versions: 0.23.0
> Reporter: Anupam Seth
> Assignee: Anupam Seth
> Priority: Critical
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3843.patch
>
>
> This bug was found by Phil Su as part of our testing.
> After MAPREDUCE-3354 went in, the Job summary log file seems to have gone missing on the
> RM host.
> The job summary log appears to be interspersed in yarn-mapredqa-historyserver-<host>.out.
> e.g.
> 12/02/09 15:57:21 INFO jobhistory.JobSummary:
> jobId=job_1328658619341_0011,submitTime=1328802904381,launchTime=1328802909977,firstMapTaskLaunchTime=1328802912116,firstReduceTaskLaunchTime=1328802915074,finishTime=1328802933797,resourc
> esPerMap=1024,resourcesPerReduce=2048,numMaps=10,numReduces=10,user=hadoopqa,queue=default,status=KILLED,mapSlotSeconds=0,reduceSlotSeconds=0
> 1) On the RM with older hadoop version where the job summary log does not exist
> mapredqa 10903 0.0 1.2 1424404 210240 ? Sl Feb07 0:19 /home/gs/java/jdk64/current/bin/java -Xmx1000m
> -Djava.net.preferIPv4Stack=true
> -Djava.library.path=/home/gs/gridre/theoden/share/hadoop/lib/native/Linux-amd64-64:/
> home/gs/gridre/theoden/share/hadoop/lib/native/Linux-amd64-64 -Dhadoop.log.dir=/home/gs/var/log/mapredqa
> -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/home/gs/gridre/theoden/share/hadoop -Dhadoop.id.str=mapredqa
> -Dhadoop
> .root.logger=INFO,console
> -Djava.library.path=/home/gs/gridre/theoden/share/hadoop/lib/native/Linux-amd64-64:/home/gs/gridre/theoden/share/hadoop/lib/native/Linux-amd64-64:/home/gs/gridre/theoden/share/hadoop/lib/nat
> ive -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -Dmapred.jobsummary.logger=INFO,console
> -Dhadoop.security.logger=INFO,NullAppender org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer
> 2) On the RM with older hadoop version where the job summary log exists
> mapredqa 24851 0.0 0.5 1463280 90516 ? Sl Jan25 0:37 /home/gs/java/jdk64/current/bin/java
> -Dproc_historyserver -Xmx1000m -Dmapred.jobsummary.logger=INFO,JSA -Dyarn.log.dir=/home/gs/var/log/mapredqa
> -Dyarn.log.file=yarn.log -Dyarn.home.dir= -Dyarn.id.str= -Dyarn.root.logger=INFO,console
> -Djava.library.path=/home/gs/gridre/shelob/share/hadoop/lib/native/Linux-amd64-64
> -Dyarn.policy.file=hadoop-policy.xml -Dyarn.log.dir=/home/gs/var/log/mapredqa
> -Dyarn.log.file=yarn-mapredqa-historyserver-<host>.log -Dyarn.home.dir= -Dyarn.id.str=mapredqa
> -Dyarn.root.logger=INFO,DRFA -Djava.library.path=/home/gs/gridre/shelob/share/hadoop/lib/native/Linux-amd64-64
> -Dyarn.policy.file=hadoop-policy.xml -Dmapred.jobsummary.logger=INFO,JSA -Dhadoop.log.dir=/home/gs/var/log/mapredqa
> -Dyarn.log.dir=/home/gs/var/log/mapredqa
> -Dhadoop.log.file=yarn-mapredqa-historyserver-<host>.log
> -Dyarn.log.file=yarn-mapredqa-historyserver-<host>.log
> -Dyarn.home.dir=/home/gs/gridre/shelob/share/hadoop -Dhadoop.root.logger=INFO,DRFA -Dyarn.root.logger=INFO,DRFA
> -Djava.library.path=/home/gs/gridre/shelob/share/hadoop/lib/native/Linux-amd64-64 -classpath
> /home/gs/gridre/shelob/conf/hadoop:/home/gs/gridre/shelob/conf/hadoop:/home/gs/gridre/shelob/conf/hadoop:/home/gs/gridre/shelob/conf/hadoop:/home/gs/gridre/shelob/share/hadoop/share/hadoop/common/lib/*:/home/gs/gridre/shelob/share/hadoop/share/hadoop/common/*:/home/gs/gridre/shelob/share/hadoop/hadoop-*-capacity-scheduler.jar:/home/gs/gridre/shelob/share/hadoop/hadoop-*-capacity-scheduler.jar:/home/gs/gridre/shelob/share/hadoop/hadoop-*-capacity-scheduler.jar:/home/gs/gridre/shelob/share/hadoop/share/hadoop/hdfs:/home/gs/gridre/shelob/share/hadoop/share/hadoop/hdfs/lib/*:/home/gs/gridre/shelob/share/hadoop/share/hadoop/hdfs/*:/home/gs/gridre/shelob/share/hadoop/share/hadoop/mapreduce/lib/*:/home/gs/gridre/shelob/share/hadoop/share/hadoop/mapreduce/*:/home/gs/java/jdk64/current/lib/tools.jar:/home/gs/gridre/shelob/share/hadoop/share/hadoop/mapreduce/*:/home/gs/gridre/shelob/share/hadoop/share/hadoop/mapreduce/lib/*
> org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer
> 1) On the RM with older hadoop version where the job summary log does not exist
> jobhistory ps shows using the option:
> -Dmapred.jobsummary.logger=INFO,console
> 2) On the RM with older hadoop version where the job summary log exists
> jobhistory ps shows using the option:
> -Dmapred.jobsummary.logger=INFO,JSA
> -Dmapred.jobsummary.logger=INFO,JSA
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3843) Job summary log file found
missing on the RM host
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13205587#comment-13205587 ]
Hadoop QA commented on MAPREDUCE-3843:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12514126/MAPREDUCE-3843.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests:
org.apache.hadoop.yarn.util.TestLinuxResourceCalculatorPlugin
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1835//testReport/
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1835//console
This message is automatically generated.
> Job summary log file found missing on the RM host
> -------------------------------------------------
>
> Key: MAPREDUCE-3843
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3843
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobhistoryserver, mrv2
> Affects Versions: 0.23.0
> Reporter: Anupam Seth
> Assignee: Anupam Seth
> Priority: Critical
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3843.patch, MAPREDUCE-3843.patch, MAPREDUCE-3843.patch
>
>
> This bug was found by Phil Su as part of our testing.
> After MAPREDUCE-3354 went in, the Job summary log file seems to have gone missing on the
> RM host.
> The job summary log appears to be interspersed in yarn-mapredqa-historyserver-<host>.out.
> e.g.
> 12/02/09 15:57:21 INFO jobhistory.JobSummary:
> jobId=job_1328658619341_0011,submitTime=1328802904381,launchTime=1328802909977,firstMapTaskLaunchTime=1328802912116,firstReduceTaskLaunchTime=1328802915074,finishTime=1328802933797,resourc
> esPerMap=1024,resourcesPerReduce=2048,numMaps=10,numReduces=10,user=hadoopqa,queue=default,status=KILLED,mapSlotSeconds=0,reduceSlotSeconds=0
> 1) On the RM with older hadoop version where the job summary log does not exist
> mapredqa 10903 0.0 1.2 1424404 210240 ? Sl Feb07 0:19 /home/gs/java/jdk64/current/bin/java -Xmx1000m
> -Djava.net.preferIPv4Stack=true
> -Djava.library.path=/home/gs/gridre/theoden/share/hadoop/lib/native/Linux-amd64-64:/
> home/gs/gridre/theoden/share/hadoop/lib/native/Linux-amd64-64 -Dhadoop.log.dir=/home/gs/var/log/mapredqa
> -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/home/gs/gridre/theoden/share/hadoop -Dhadoop.id.str=mapredqa
> -Dhadoop
> .root.logger=INFO,console
> -Djava.library.path=/home/gs/gridre/theoden/share/hadoop/lib/native/Linux-amd64-64:/home/gs/gridre/theoden/share/hadoop/lib/native/Linux-amd64-64:/home/gs/gridre/theoden/share/hadoop/lib/nat
> ive -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -Dmapred.jobsummary.logger=INFO,console
> -Dhadoop.security.logger=INFO,NullAppender org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer
> 2) On the RM with older hadoop version where the job summary log exists
> mapredqa 24851 0.0 0.5 1463280 90516 ? Sl Jan25 0:37 /home/gs/java/jdk64/current/bin/java
> -Dproc_historyserver -Xmx1000m -Dmapred.jobsummary.logger=INFO,JSA -Dyarn.log.dir=/home/gs/var/log/mapredqa
> -Dyarn.log.file=yarn.log -Dyarn.home.dir= -Dyarn.id.str= -Dyarn.root.logger=INFO,console
> -Djava.library.path=/home/gs/gridre/shelob/share/hadoop/lib/native/Linux-amd64-64
> -Dyarn.policy.file=hadoop-policy.xml -Dyarn.log.dir=/home/gs/var/log/mapredqa
> -Dyarn.log.file=yarn-mapredqa-historyserver-<host>.log -Dyarn.home.dir= -Dyarn.id.str=mapredqa
> -Dyarn.root.logger=INFO,DRFA -Djava.library.path=/home/gs/gridre/shelob/share/hadoop/lib/native/Linux-amd64-64
> -Dyarn.policy.file=hadoop-policy.xml -Dmapred.jobsummary.logger=INFO,JSA -Dhadoop.log.dir=/home/gs/var/log/mapredqa
> -Dyarn.log.dir=/home/gs/var/log/mapredqa
> -Dhadoop.log.file=yarn-mapredqa-historyserver-<host>.log
> -Dyarn.log.file=yarn-mapredqa-historyserver-<host>.log
> -Dyarn.home.dir=/home/gs/gridre/shelob/share/hadoop -Dhadoop.root.logger=INFO,DRFA -Dyarn.root.logger=INFO,DRFA
> -Djava.library.path=/home/gs/gridre/shelob/share/hadoop/lib/native/Linux-amd64-64 -classpath
> /home/gs/gridre/shelob/conf/hadoop:/home/gs/gridre/shelob/conf/hadoop:/home/gs/gridre/shelob/conf/hadoop:/home/gs/gridre/shelob/conf/hadoop:/home/gs/gridre/shelob/share/hadoop/share/hadoop/common/lib/*:/home/gs/gridre/shelob/share/hadoop/share/hadoop/common/*:/home/gs/gridre/shelob/share/hadoop/hadoop-*-capacity-scheduler.jar:/home/gs/gridre/shelob/share/hadoop/hadoop-*-capacity-scheduler.jar:/home/gs/gridre/shelob/share/hadoop/hadoop-*-capacity-scheduler.jar:/home/gs/gridre/shelob/share/hadoop/share/hadoop/hdfs:/home/gs/gridre/shelob/share/hadoop/share/hadoop/hdfs/lib/*:/home/gs/gridre/shelob/share/hadoop/share/hadoop/hdfs/*:/home/gs/gridre/shelob/share/hadoop/share/hadoop/mapreduce/lib/*:/home/gs/gridre/shelob/share/hadoop/share/hadoop/mapreduce/*:/home/gs/java/jdk64/current/lib/tools.jar:/home/gs/gridre/shelob/share/hadoop/share/hadoop/mapreduce/*:/home/gs/gridre/shelob/share/hadoop/share/hadoop/mapreduce/lib/*
> org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer
> 1) On the RM with older hadoop version where the job summary log does not exist
> jobhistory ps shows using the option:
> -Dmapred.jobsummary.logger=INFO,console
> 2) On the RM with older hadoop version where the job summary log exists
> jobhistory ps shows using the option:
> -Dmapred.jobsummary.logger=INFO,JSA
> -Dmapred.jobsummary.logger=INFO,JSA
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3843) Job summary log file found
missing on the RM host
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13205811#comment-13205811 ]
Hadoop QA commented on MAPREDUCE-3843:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12514166/MAPREDUCE-3843.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests:
org.apache.hadoop.yarn.util.TestLinuxResourceCalculatorPlugin
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1840//testReport/
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1840//console
This message is automatically generated.
> Job summary log file found missing on the RM host
> -------------------------------------------------
>
> Key: MAPREDUCE-3843
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3843
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobhistoryserver, mrv2
> Affects Versions: 0.23.0
> Reporter: Anupam Seth
> Assignee: Anupam Seth
> Priority: Critical
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3843.patch, MAPREDUCE-3843.patch, MAPREDUCE-3843.patch, MAPREDUCE-3843.patch, MAPREDUCE-3843.patch
>
>
> This bug was found by Phil Su as part of our testing.
> After MAPREDUCE-3354 went in, the Job summary log file seems to have gone missing on the
> RM host.
> The job summary log appears to be interspersed in yarn-mapredqa-historyserver-<host>.out.
> e.g.
> 12/02/09 15:57:21 INFO jobhistory.JobSummary:
> jobId=job_1328658619341_0011,submitTime=1328802904381,launchTime=1328802909977,firstMapTaskLaunchTime=1328802912116,firstReduceTaskLaunchTime=1328802915074,finishTime=1328802933797,resourc
> esPerMap=1024,resourcesPerReduce=2048,numMaps=10,numReduces=10,user=hadoopqa,queue=default,status=KILLED,mapSlotSeconds=0,reduceSlotSeconds=0
> 1) On the RM with older hadoop version where the job summary log does not exist
> mapredqa 10903 0.0 1.2 1424404 210240 ? Sl Feb07 0:19 /home/gs/java/jdk64/current/bin/java -Xmx1000m
> -Djava.net.preferIPv4Stack=true
> -Djava.library.path=/home/gs/gridre/theoden/share/hadoop/lib/native/Linux-amd64-64:/
> home/gs/gridre/theoden/share/hadoop/lib/native/Linux-amd64-64 -Dhadoop.log.dir=/home/gs/var/log/mapredqa
> -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/home/gs/gridre/theoden/share/hadoop -Dhadoop.id.str=mapredqa
> -Dhadoop
> .root.logger=INFO,console
> -Djava.library.path=/home/gs/gridre/theoden/share/hadoop/lib/native/Linux-amd64-64:/home/gs/gridre/theoden/share/hadoop/lib/native/Linux-amd64-64:/home/gs/gridre/theoden/share/hadoop/lib/nat
> ive -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -Dmapred.jobsummary.logger=INFO,console
> -Dhadoop.security.logger=INFO,NullAppender org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer
> 2) On the RM with older hadoop version where the job summary log exists
> mapredqa 24851 0.0 0.5 1463280 90516 ? Sl Jan25 0:37 /home/gs/java/jdk64/current/bin/java
> -Dproc_historyserver -Xmx1000m -Dmapred.jobsummary.logger=INFO,JSA -Dyarn.log.dir=/home/gs/var/log/mapredqa
> -Dyarn.log.file=yarn.log -Dyarn.home.dir= -Dyarn.id.str= -Dyarn.root.logger=INFO,console
> -Djava.library.path=/home/gs/gridre/shelob/share/hadoop/lib/native/Linux-amd64-64
> -Dyarn.policy.file=hadoop-policy.xml -Dyarn.log.dir=/home/gs/var/log/mapredqa
> -Dyarn.log.file=yarn-mapredqa-historyserver-<host>.log -Dyarn.home.dir= -Dyarn.id.str=mapredqa
> -Dyarn.root.logger=INFO,DRFA -Djava.library.path=/home/gs/gridre/shelob/share/hadoop/lib/native/Linux-amd64-64
> -Dyarn.policy.file=hadoop-policy.xml -Dmapred.jobsummary.logger=INFO,JSA -Dhadoop.log.dir=/home/gs/var/log/mapredqa
> -Dyarn.log.dir=/home/gs/var/log/mapredqa
> -Dhadoop.log.file=yarn-mapredqa-historyserver-<host>.log
> -Dyarn.log.file=yarn-mapredqa-historyserver-<host>.log
> -Dyarn.home.dir=/home/gs/gridre/shelob/share/hadoop -Dhadoop.root.logger=INFO,DRFA -Dyarn.root.logger=INFO,DRFA
> -Djava.library.path=/home/gs/gridre/shelob/share/hadoop/lib/native/Linux-amd64-64 -classpath
> /home/gs/gridre/shelob/conf/hadoop:/home/gs/gridre/shelob/conf/hadoop:/home/gs/gridre/shelob/conf/hadoop:/home/gs/gridre/shelob/conf/hadoop:/home/gs/gridre/shelob/share/hadoop/share/hadoop/common/lib/*:/home/gs/gridre/shelob/share/hadoop/share/hadoop/common/*:/home/gs/gridre/shelob/share/hadoop/hadoop-*-capacity-scheduler.jar:/home/gs/gridre/shelob/share/hadoop/hadoop-*-capacity-scheduler.jar:/home/gs/gridre/shelob/share/hadoop/hadoop-*-capacity-scheduler.jar:/home/gs/gridre/shelob/share/hadoop/share/hadoop/hdfs:/home/gs/gridre/shelob/share/hadoop/share/hadoop/hdfs/lib/*:/home/gs/gridre/shelob/share/hadoop/share/hadoop/hdfs/*:/home/gs/gridre/shelob/share/hadoop/share/hadoop/mapreduce/lib/*:/home/gs/gridre/shelob/share/hadoop/share/hadoop/mapreduce/*:/home/gs/java/jdk64/current/lib/tools.jar:/home/gs/gridre/shelob/share/hadoop/share/hadoop/mapreduce/*:/home/gs/gridre/shelob/share/hadoop/share/hadoop/mapreduce/lib/*
> org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer
> 1) On the RM with older hadoop version where the job summary log does not exist
> jobhistory ps shows using the option:
> -Dmapred.jobsummary.logger=INFO,console
> 2) On the RM with older hadoop version where the job summary log exists
> jobhistory ps shows using the option:
> -Dmapred.jobsummary.logger=INFO,JSA
> -Dmapred.jobsummary.logger=INFO,JSA
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3843) Job summary log file found
missing on the RM host
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13206094#comment-13206094 ]
Hudson commented on MAPREDUCE-3843:
-----------------------------------
Integrated in Hadoop-Hdfs-trunk #953 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/953/])
MAPREDUCE-3843. Job summary log file found missing on the RM host (Anupam Seth via tgraves)
tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1242976
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/bin/mr-jobhistory-daemon.sh
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/ClusterSetup.apt.vm
> Job summary log file found missing on the RM host
> -------------------------------------------------
>
> Key: MAPREDUCE-3843
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3843
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobhistoryserver, mrv2
> Affects Versions: 0.23.0
> Reporter: Anupam Seth
> Assignee: Anupam Seth
> Priority: Critical
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3843.patch, MAPREDUCE-3843.patch, MAPREDUCE-3843.patch, MAPREDUCE-3843.patch, MAPREDUCE-3843.patch, MAPREDUCE-3843.patch
>
>
> This bug was found by Phil Su as part of our testing.
> After MAPREDUCE-3354 went in, the Job summary log file seems to have gone missing on the
> RM host.
> The job summary log appears to be interspersed in yarn-mapredqa-historyserver-<host>.out.
> e.g.
> 12/02/09 15:57:21 INFO jobhistory.JobSummary:
> jobId=job_1328658619341_0011,submitTime=1328802904381,launchTime=1328802909977,firstMapTaskLaunchTime=1328802912116,firstReduceTaskLaunchTime=1328802915074,finishTime=1328802933797,resourc
> esPerMap=1024,resourcesPerReduce=2048,numMaps=10,numReduces=10,user=hadoopqa,queue=default,status=KILLED,mapSlotSeconds=0,reduceSlotSeconds=0
> 1) On the RM with older hadoop version where the job summary log does not exist
> mapredqa 10903 0.0 1.2 1424404 210240 ? Sl Feb07 0:19 /home/gs/java/jdk64/current/bin/java -Xmx1000m
> -Djava.net.preferIPv4Stack=true
> -Djava.library.path=/home/gs/gridre/theoden/share/hadoop/lib/native/Linux-amd64-64:/
> home/gs/gridre/theoden/share/hadoop/lib/native/Linux-amd64-64 -Dhadoop.log.dir=/home/gs/var/log/mapredqa
> -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/home/gs/gridre/theoden/share/hadoop -Dhadoop.id.str=mapredqa
> -Dhadoop
> .root.logger=INFO,console
> -Djava.library.path=/home/gs/gridre/theoden/share/hadoop/lib/native/Linux-amd64-64:/home/gs/gridre/theoden/share/hadoop/lib/native/Linux-amd64-64:/home/gs/gridre/theoden/share/hadoop/lib/nat
> ive -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -Dmapred.jobsummary.logger=INFO,console
> -Dhadoop.security.logger=INFO,NullAppender org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer
> 2) On the RM with older hadoop version where the job summary log exists
> mapredqa 24851 0.0 0.5 1463280 90516 ? Sl Jan25 0:37 /home/gs/java/jdk64/current/bin/java
> -Dproc_historyserver -Xmx1000m -Dmapred.jobsummary.logger=INFO,JSA -Dyarn.log.dir=/home/gs/var/log/mapredqa
> -Dyarn.log.file=yarn.log -Dyarn.home.dir= -Dyarn.id.str= -Dyarn.root.logger=INFO,console
> -Djava.library.path=/home/gs/gridre/shelob/share/hadoop/lib/native/Linux-amd64-64
> -Dyarn.policy.file=hadoop-policy.xml -Dyarn.log.dir=/home/gs/var/log/mapredqa
> -Dyarn.log.file=yarn-mapredqa-historyserver-<host>.log -Dyarn.home.dir= -Dyarn.id.str=mapredqa
> -Dyarn.root.logger=INFO,DRFA -Djava.library.path=/home/gs/gridre/shelob/share/hadoop/lib/native/Linux-amd64-64
> -Dyarn.policy.file=hadoop-policy.xml -Dmapred.jobsummary.logger=INFO,JSA -Dhadoop.log.dir=/home/gs/var/log/mapredqa
> -Dyarn.log.dir=/home/gs/var/log/mapredqa
> -Dhadoop.log.file=yarn-mapredqa-historyserver-<host>.log
> -Dyarn.log.file=yarn-mapredqa-historyserver-<host>.log
> -Dyarn.home.dir=/home/gs/gridre/shelob/share/hadoop -Dhadoop.root.logger=INFO,DRFA -Dyarn.root.logger=INFO,DRFA
> -Djava.library.path=/home/gs/gridre/shelob/share/hadoop/lib/native/Linux-amd64-64 -classpath
> /home/gs/gridre/shelob/conf/hadoop:/home/gs/gridre/shelob/conf/hadoop:/home/gs/gridre/shelob/conf/hadoop:/home/gs/gridre/shelob/conf/hadoop:/home/gs/gridre/shelob/share/hadoop/share/hadoop/common/lib/*:/home/gs/gridre/shelob/share/hadoop/share/hadoop/common/*:/home/gs/gridre/shelob/share/hadoop/hadoop-*-capacity-scheduler.jar:/home/gs/gridre/shelob/share/hadoop/hadoop-*-capacity-scheduler.jar:/home/gs/gridre/shelob/share/hadoop/hadoop-*-capacity-scheduler.jar:/home/gs/gridre/shelob/share/hadoop/share/hadoop/hdfs:/home/gs/gridre/shelob/share/hadoop/share/hadoop/hdfs/lib/*:/home/gs/gridre/shelob/share/hadoop/share/hadoop/hdfs/*:/home/gs/gridre/shelob/share/hadoop/share/hadoop/mapreduce/lib/*:/home/gs/gridre/shelob/share/hadoop/share/hadoop/mapreduce/*:/home/gs/java/jdk64/current/lib/tools.jar:/home/gs/gridre/shelob/share/hadoop/share/hadoop/mapreduce/*:/home/gs/gridre/shelob/share/hadoop/share/hadoop/mapreduce/lib/*
> org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer
> 1) On the RM with older hadoop version where the job summary log does not exist
> jobhistory ps shows using the option:
> -Dmapred.jobsummary.logger=INFO,console
> 2) On the RM with older hadoop version where the job summary log exists
> jobhistory ps shows using the option:
> -Dmapred.jobsummary.logger=INFO,JSA
> -Dmapred.jobsummary.logger=INFO,JSA
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3843) Job summary log file found
missing on the RM host
Posted by "Anupam Seth (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Anupam Seth updated MAPREDUCE-3843:
-----------------------------------
Attachment: MAPREDUCE-3843.patch
> Job summary log file found missing on the RM host
> -------------------------------------------------
>
> Key: MAPREDUCE-3843
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3843
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobhistoryserver, mrv2
> Affects Versions: 0.23.0
> Reporter: Anupam Seth
> Assignee: Anupam Seth
> Priority: Critical
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3843.patch
>
>
> This bug was found by Phil Su as part of our testing.
> After MAPREDUCE-3354 went in, the Job summary log file seems to have gone missing on the
> RM host.
> The job summary log appears to be interspersed in yarn-mapredqa-historyserver-<host>.out.
> e.g.
> 12/02/09 15:57:21 INFO jobhistory.JobSummary:
> jobId=job_1328658619341_0011,submitTime=1328802904381,launchTime=1328802909977,firstMapTaskLaunchTime=1328802912116,firstReduceTaskLaunchTime=1328802915074,finishTime=1328802933797,resourc
> esPerMap=1024,resourcesPerReduce=2048,numMaps=10,numReduces=10,user=hadoopqa,queue=default,status=KILLED,mapSlotSeconds=0,reduceSlotSeconds=0
> 1) On the RM with older hadoop version where the job summary log does not exist
> mapredqa 10903 0.0 1.2 1424404 210240 ? Sl Feb07 0:19 /home/gs/java/jdk64/current/bin/java -Xmx1000m
> -Djava.net.preferIPv4Stack=true
> -Djava.library.path=/home/gs/gridre/theoden/share/hadoop/lib/native/Linux-amd64-64:/
> home/gs/gridre/theoden/share/hadoop/lib/native/Linux-amd64-64 -Dhadoop.log.dir=/home/gs/var/log/mapredqa
> -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/home/gs/gridre/theoden/share/hadoop -Dhadoop.id.str=mapredqa
> -Dhadoop
> .root.logger=INFO,console
> -Djava.library.path=/home/gs/gridre/theoden/share/hadoop/lib/native/Linux-amd64-64:/home/gs/gridre/theoden/share/hadoop/lib/native/Linux-amd64-64:/home/gs/gridre/theoden/share/hadoop/lib/nat
> ive -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -Dmapred.jobsummary.logger=INFO,console
> -Dhadoop.security.logger=INFO,NullAppender org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer
> 2) On the RM with older hadoop version where the job summary log exists
> mapredqa 24851 0.0 0.5 1463280 90516 ? Sl Jan25 0:37 /home/gs/java/jdk64/current/bin/java
> -Dproc_historyserver -Xmx1000m -Dmapred.jobsummary.logger=INFO,JSA -Dyarn.log.dir=/home/gs/var/log/mapredqa
> -Dyarn.log.file=yarn.log -Dyarn.home.dir= -Dyarn.id.str= -Dyarn.root.logger=INFO,console
> -Djava.library.path=/home/gs/gridre/shelob/share/hadoop/lib/native/Linux-amd64-64
> -Dyarn.policy.file=hadoop-policy.xml -Dyarn.log.dir=/home/gs/var/log/mapredqa
> -Dyarn.log.file=yarn-mapredqa-historyserver-<host>.log -Dyarn.home.dir= -Dyarn.id.str=mapredqa
> -Dyarn.root.logger=INFO,DRFA -Djava.library.path=/home/gs/gridre/shelob/share/hadoop/lib/native/Linux-amd64-64
> -Dyarn.policy.file=hadoop-policy.xml -Dmapred.jobsummary.logger=INFO,JSA -Dhadoop.log.dir=/home/gs/var/log/mapredqa
> -Dyarn.log.dir=/home/gs/var/log/mapredqa
> -Dhadoop.log.file=yarn-mapredqa-historyserver-<host>.log
> -Dyarn.log.file=yarn-mapredqa-historyserver-<host>.log
> -Dyarn.home.dir=/home/gs/gridre/shelob/share/hadoop -Dhadoop.root.logger=INFO,DRFA -Dyarn.root.logger=INFO,DRFA
> -Djava.library.path=/home/gs/gridre/shelob/share/hadoop/lib/native/Linux-amd64-64 -classpath
> /home/gs/gridre/shelob/conf/hadoop:/home/gs/gridre/shelob/conf/hadoop:/home/gs/gridre/shelob/conf/hadoop:/home/gs/gridre/shelob/conf/hadoop:/home/gs/gridre/shelob/share/hadoop/share/hadoop/common/lib/*:/home/gs/gridre/shelob/share/hadoop/share/hadoop/common/*:/home/gs/gridre/shelob/share/hadoop/hadoop-*-capacity-scheduler.jar:/home/gs/gridre/shelob/share/hadoop/hadoop-*-capacity-scheduler.jar:/home/gs/gridre/shelob/share/hadoop/hadoop-*-capacity-scheduler.jar:/home/gs/gridre/shelob/share/hadoop/share/hadoop/hdfs:/home/gs/gridre/shelob/share/hadoop/share/hadoop/hdfs/lib/*:/home/gs/gridre/shelob/share/hadoop/share/hadoop/hdfs/*:/home/gs/gridre/shelob/share/hadoop/share/hadoop/mapreduce/lib/*:/home/gs/gridre/shelob/share/hadoop/share/hadoop/mapreduce/*:/home/gs/java/jdk64/current/lib/tools.jar:/home/gs/gridre/shelob/share/hadoop/share/hadoop/mapreduce/*:/home/gs/gridre/shelob/share/hadoop/share/hadoop/mapreduce/lib/*
> org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer
> 1) On the RM with older hadoop version where the job summary log does not exist
> jobhistory ps shows using the option:
> -Dmapred.jobsummary.logger=INFO,console
> 2) On the RM with older hadoop version where the job summary log exists
> jobhistory ps shows using the option:
> -Dmapred.jobsummary.logger=INFO,JSA
> -Dmapred.jobsummary.logger=INFO,JSA
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3843) Job summary log file found
missing on the RM host
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13205907#comment-13205907 ]
Hudson commented on MAPREDUCE-3843:
-----------------------------------
Integrated in Hadoop-Hdfs-trunk-Commit #1787 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1787/])
MAPREDUCE-3843. Job summary log file found missing on the RM host (Anupam Seth via tgraves)
tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1242976
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/bin/mr-jobhistory-daemon.sh
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/ClusterSetup.apt.vm
> Job summary log file found missing on the RM host
> -------------------------------------------------
>
> Key: MAPREDUCE-3843
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3843
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobhistoryserver, mrv2
> Affects Versions: 0.23.0
> Reporter: Anupam Seth
> Assignee: Anupam Seth
> Priority: Critical
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3843.patch, MAPREDUCE-3843.patch, MAPREDUCE-3843.patch, MAPREDUCE-3843.patch, MAPREDUCE-3843.patch, MAPREDUCE-3843.patch
>
>
> This bug was found by Phil Su as part of our testing.
> After MAPREDUCE-3354 went in, the Job summary log file seems to have gone missing on the
> RM host.
> The job summary log appears to be interspersed in yarn-mapredqa-historyserver-<host>.out.
> e.g.
> 12/02/09 15:57:21 INFO jobhistory.JobSummary:
> jobId=job_1328658619341_0011,submitTime=1328802904381,launchTime=1328802909977,firstMapTaskLaunchTime=1328802912116,firstReduceTaskLaunchTime=1328802915074,finishTime=1328802933797,resourc
> esPerMap=1024,resourcesPerReduce=2048,numMaps=10,numReduces=10,user=hadoopqa,queue=default,status=KILLED,mapSlotSeconds=0,reduceSlotSeconds=0
> 1) On the RM with older hadoop version where the job summary log does not exist
> mapredqa 10903 0.0 1.2 1424404 210240 ? Sl Feb07 0:19 /home/gs/java/jdk64/current/bin/java -Xmx1000m
> -Djava.net.preferIPv4Stack=true
> -Djava.library.path=/home/gs/gridre/theoden/share/hadoop/lib/native/Linux-amd64-64:/
> home/gs/gridre/theoden/share/hadoop/lib/native/Linux-amd64-64 -Dhadoop.log.dir=/home/gs/var/log/mapredqa
> -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/home/gs/gridre/theoden/share/hadoop -Dhadoop.id.str=mapredqa
> -Dhadoop
> .root.logger=INFO,console
> -Djava.library.path=/home/gs/gridre/theoden/share/hadoop/lib/native/Linux-amd64-64:/home/gs/gridre/theoden/share/hadoop/lib/native/Linux-amd64-64:/home/gs/gridre/theoden/share/hadoop/lib/nat
> ive -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -Dmapred.jobsummary.logger=INFO,console
> -Dhadoop.security.logger=INFO,NullAppender org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer
> 2) On the RM with older hadoop version where the job summary log exists
> mapredqa 24851 0.0 0.5 1463280 90516 ? Sl Jan25 0:37 /home/gs/java/jdk64/current/bin/java
> -Dproc_historyserver -Xmx1000m -Dmapred.jobsummary.logger=INFO,JSA -Dyarn.log.dir=/home/gs/var/log/mapredqa
> -Dyarn.log.file=yarn.log -Dyarn.home.dir= -Dyarn.id.str= -Dyarn.root.logger=INFO,console
> -Djava.library.path=/home/gs/gridre/shelob/share/hadoop/lib/native/Linux-amd64-64
> -Dyarn.policy.file=hadoop-policy.xml -Dyarn.log.dir=/home/gs/var/log/mapredqa
> -Dyarn.log.file=yarn-mapredqa-historyserver-<host>.log -Dyarn.home.dir= -Dyarn.id.str=mapredqa
> -Dyarn.root.logger=INFO,DRFA -Djava.library.path=/home/gs/gridre/shelob/share/hadoop/lib/native/Linux-amd64-64
> -Dyarn.policy.file=hadoop-policy.xml -Dmapred.jobsummary.logger=INFO,JSA -Dhadoop.log.dir=/home/gs/var/log/mapredqa
> -Dyarn.log.dir=/home/gs/var/log/mapredqa
> -Dhadoop.log.file=yarn-mapredqa-historyserver-<host>.log
> -Dyarn.log.file=yarn-mapredqa-historyserver-<host>.log
> -Dyarn.home.dir=/home/gs/gridre/shelob/share/hadoop -Dhadoop.root.logger=INFO,DRFA -Dyarn.root.logger=INFO,DRFA
> -Djava.library.path=/home/gs/gridre/shelob/share/hadoop/lib/native/Linux-amd64-64 -classpath
> /home/gs/gridre/shelob/conf/hadoop:/home/gs/gridre/shelob/conf/hadoop:/home/gs/gridre/shelob/conf/hadoop:/home/gs/gridre/shelob/conf/hadoop:/home/gs/gridre/shelob/share/hadoop/share/hadoop/common/lib/*:/home/gs/gridre/shelob/share/hadoop/share/hadoop/common/*:/home/gs/gridre/shelob/share/hadoop/hadoop-*-capacity-scheduler.jar:/home/gs/gridre/shelob/share/hadoop/hadoop-*-capacity-scheduler.jar:/home/gs/gridre/shelob/share/hadoop/hadoop-*-capacity-scheduler.jar:/home/gs/gridre/shelob/share/hadoop/share/hadoop/hdfs:/home/gs/gridre/shelob/share/hadoop/share/hadoop/hdfs/lib/*:/home/gs/gridre/shelob/share/hadoop/share/hadoop/hdfs/*:/home/gs/gridre/shelob/share/hadoop/share/hadoop/mapreduce/lib/*:/home/gs/gridre/shelob/share/hadoop/share/hadoop/mapreduce/*:/home/gs/java/jdk64/current/lib/tools.jar:/home/gs/gridre/shelob/share/hadoop/share/hadoop/mapreduce/*:/home/gs/gridre/shelob/share/hadoop/share/hadoop/mapreduce/lib/*
> org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer
> 1) On the RM with older hadoop version where the job summary log does not exist
> jobhistory ps shows using the option:
> -Dmapred.jobsummary.logger=INFO,console
> 2) On the RM with older hadoop version where the job summary log exists
> jobhistory ps shows using the option:
> -Dmapred.jobsummary.logger=INFO,JSA
> -Dmapred.jobsummary.logger=INFO,JSA
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3843) Job summary log file found
missing on the RM host
Posted by "Anupam Seth (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Anupam Seth updated MAPREDUCE-3843:
-----------------------------------
Attachment: MAPREDUCE-3843.patch
> Job summary log file found missing on the RM host
> -------------------------------------------------
>
> Key: MAPREDUCE-3843
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3843
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobhistoryserver, mrv2
> Affects Versions: 0.23.0
> Reporter: Anupam Seth
> Assignee: Anupam Seth
> Priority: Critical
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3843.patch, MAPREDUCE-3843.patch, MAPREDUCE-3843.patch, MAPREDUCE-3843.patch, MAPREDUCE-3843.patch
>
>
> This bug was found by Phil Su as part of our testing.
> After MAPREDUCE-3354 went in, the Job summary log file seems to have gone missing on the
> RM host.
> The job summary log appears to be interspersed in yarn-mapredqa-historyserver-<host>.out.
> e.g.
> 12/02/09 15:57:21 INFO jobhistory.JobSummary:
> jobId=job_1328658619341_0011,submitTime=1328802904381,launchTime=1328802909977,firstMapTaskLaunchTime=1328802912116,firstReduceTaskLaunchTime=1328802915074,finishTime=1328802933797,resourc
> esPerMap=1024,resourcesPerReduce=2048,numMaps=10,numReduces=10,user=hadoopqa,queue=default,status=KILLED,mapSlotSeconds=0,reduceSlotSeconds=0
> 1) On the RM with older hadoop version where the job summary log does not exist
> mapredqa 10903 0.0 1.2 1424404 210240 ? Sl Feb07 0:19 /home/gs/java/jdk64/current/bin/java -Xmx1000m
> -Djava.net.preferIPv4Stack=true
> -Djava.library.path=/home/gs/gridre/theoden/share/hadoop/lib/native/Linux-amd64-64:/
> home/gs/gridre/theoden/share/hadoop/lib/native/Linux-amd64-64 -Dhadoop.log.dir=/home/gs/var/log/mapredqa
> -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/home/gs/gridre/theoden/share/hadoop -Dhadoop.id.str=mapredqa
> -Dhadoop
> .root.logger=INFO,console
> -Djava.library.path=/home/gs/gridre/theoden/share/hadoop/lib/native/Linux-amd64-64:/home/gs/gridre/theoden/share/hadoop/lib/native/Linux-amd64-64:/home/gs/gridre/theoden/share/hadoop/lib/nat
> ive -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -Dmapred.jobsummary.logger=INFO,console
> -Dhadoop.security.logger=INFO,NullAppender org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer
> 2) On the RM with older hadoop version where the job summary log exists
> mapredqa 24851 0.0 0.5 1463280 90516 ? Sl Jan25 0:37 /home/gs/java/jdk64/current/bin/java
> -Dproc_historyserver -Xmx1000m -Dmapred.jobsummary.logger=INFO,JSA -Dyarn.log.dir=/home/gs/var/log/mapredqa
> -Dyarn.log.file=yarn.log -Dyarn.home.dir= -Dyarn.id.str= -Dyarn.root.logger=INFO,console
> -Djava.library.path=/home/gs/gridre/shelob/share/hadoop/lib/native/Linux-amd64-64
> -Dyarn.policy.file=hadoop-policy.xml -Dyarn.log.dir=/home/gs/var/log/mapredqa
> -Dyarn.log.file=yarn-mapredqa-historyserver-<host>.log -Dyarn.home.dir= -Dyarn.id.str=mapredqa
> -Dyarn.root.logger=INFO,DRFA -Djava.library.path=/home/gs/gridre/shelob/share/hadoop/lib/native/Linux-amd64-64
> -Dyarn.policy.file=hadoop-policy.xml -Dmapred.jobsummary.logger=INFO,JSA -Dhadoop.log.dir=/home/gs/var/log/mapredqa
> -Dyarn.log.dir=/home/gs/var/log/mapredqa
> -Dhadoop.log.file=yarn-mapredqa-historyserver-<host>.log
> -Dyarn.log.file=yarn-mapredqa-historyserver-<host>.log
> -Dyarn.home.dir=/home/gs/gridre/shelob/share/hadoop -Dhadoop.root.logger=INFO,DRFA -Dyarn.root.logger=INFO,DRFA
> -Djava.library.path=/home/gs/gridre/shelob/share/hadoop/lib/native/Linux-amd64-64 -classpath
> /home/gs/gridre/shelob/conf/hadoop:/home/gs/gridre/shelob/conf/hadoop:/home/gs/gridre/shelob/conf/hadoop:/home/gs/gridre/shelob/conf/hadoop:/home/gs/gridre/shelob/share/hadoop/share/hadoop/common/lib/*:/home/gs/gridre/shelob/share/hadoop/share/hadoop/common/*:/home/gs/gridre/shelob/share/hadoop/hadoop-*-capacity-scheduler.jar:/home/gs/gridre/shelob/share/hadoop/hadoop-*-capacity-scheduler.jar:/home/gs/gridre/shelob/share/hadoop/hadoop-*-capacity-scheduler.jar:/home/gs/gridre/shelob/share/hadoop/share/hadoop/hdfs:/home/gs/gridre/shelob/share/hadoop/share/hadoop/hdfs/lib/*:/home/gs/gridre/shelob/share/hadoop/share/hadoop/hdfs/*:/home/gs/gridre/shelob/share/hadoop/share/hadoop/mapreduce/lib/*:/home/gs/gridre/shelob/share/hadoop/share/hadoop/mapreduce/*:/home/gs/java/jdk64/current/lib/tools.jar:/home/gs/gridre/shelob/share/hadoop/share/hadoop/mapreduce/*:/home/gs/gridre/shelob/share/hadoop/share/hadoop/mapreduce/lib/*
> org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer
> 1) On the RM with older hadoop version where the job summary log does not exist
> jobhistory ps shows using the option:
> -Dmapred.jobsummary.logger=INFO,console
> 2) On the RM with older hadoop version where the job summary log exists
> jobhistory ps shows using the option:
> -Dmapred.jobsummary.logger=INFO,JSA
> -Dmapred.jobsummary.logger=INFO,JSA
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3843) Job summary log file found
missing on the RM host
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13206111#comment-13206111 ]
Hudson commented on MAPREDUCE-3843:
-----------------------------------
Integrated in Hadoop-Mapreduce-0.23-Build #189 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/189/])
merge -r 1242975:1242976 from trunk. FIXES: MAPREDUCE-3843
tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1242977
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/bin/mr-jobhistory-daemon.sh
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/ClusterSetup.apt.vm
> Job summary log file found missing on the RM host
> -------------------------------------------------
>
> Key: MAPREDUCE-3843
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3843
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobhistoryserver, mrv2
> Affects Versions: 0.23.0
> Reporter: Anupam Seth
> Assignee: Anupam Seth
> Priority: Critical
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3843.patch, MAPREDUCE-3843.patch, MAPREDUCE-3843.patch, MAPREDUCE-3843.patch, MAPREDUCE-3843.patch, MAPREDUCE-3843.patch
>
>
> This bug was found by Phil Su as part of our testing.
> After MAPREDUCE-3354 went in, the Job summary log file seems to have gone missing on the
> RM host.
> The job summary log appears to be interspersed in yarn-mapredqa-historyserver-<host>.out.
> e.g.
> 12/02/09 15:57:21 INFO jobhistory.JobSummary:
> jobId=job_1328658619341_0011,submitTime=1328802904381,launchTime=1328802909977,firstMapTaskLaunchTime=1328802912116,firstReduceTaskLaunchTime=1328802915074,finishTime=1328802933797,resourc
> esPerMap=1024,resourcesPerReduce=2048,numMaps=10,numReduces=10,user=hadoopqa,queue=default,status=KILLED,mapSlotSeconds=0,reduceSlotSeconds=0
> 1) On the RM with older hadoop version where the job summary log does not exist
> mapredqa 10903 0.0 1.2 1424404 210240 ? Sl Feb07 0:19 /home/gs/java/jdk64/current/bin/java -Xmx1000m
> -Djava.net.preferIPv4Stack=true
> -Djava.library.path=/home/gs/gridre/theoden/share/hadoop/lib/native/Linux-amd64-64:/
> home/gs/gridre/theoden/share/hadoop/lib/native/Linux-amd64-64 -Dhadoop.log.dir=/home/gs/var/log/mapredqa
> -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/home/gs/gridre/theoden/share/hadoop -Dhadoop.id.str=mapredqa
> -Dhadoop
> .root.logger=INFO,console
> -Djava.library.path=/home/gs/gridre/theoden/share/hadoop/lib/native/Linux-amd64-64:/home/gs/gridre/theoden/share/hadoop/lib/native/Linux-amd64-64:/home/gs/gridre/theoden/share/hadoop/lib/nat
> ive -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -Dmapred.jobsummary.logger=INFO,console
> -Dhadoop.security.logger=INFO,NullAppender org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer
> 2) On the RM with older hadoop version where the job summary log exists
> mapredqa 24851 0.0 0.5 1463280 90516 ? Sl Jan25 0:37 /home/gs/java/jdk64/current/bin/java
> -Dproc_historyserver -Xmx1000m -Dmapred.jobsummary.logger=INFO,JSA -Dyarn.log.dir=/home/gs/var/log/mapredqa
> -Dyarn.log.file=yarn.log -Dyarn.home.dir= -Dyarn.id.str= -Dyarn.root.logger=INFO,console
> -Djava.library.path=/home/gs/gridre/shelob/share/hadoop/lib/native/Linux-amd64-64
> -Dyarn.policy.file=hadoop-policy.xml -Dyarn.log.dir=/home/gs/var/log/mapredqa
> -Dyarn.log.file=yarn-mapredqa-historyserver-<host>.log -Dyarn.home.dir= -Dyarn.id.str=mapredqa
> -Dyarn.root.logger=INFO,DRFA -Djava.library.path=/home/gs/gridre/shelob/share/hadoop/lib/native/Linux-amd64-64
> -Dyarn.policy.file=hadoop-policy.xml -Dmapred.jobsummary.logger=INFO,JSA -Dhadoop.log.dir=/home/gs/var/log/mapredqa
> -Dyarn.log.dir=/home/gs/var/log/mapredqa
> -Dhadoop.log.file=yarn-mapredqa-historyserver-<host>.log
> -Dyarn.log.file=yarn-mapredqa-historyserver-<host>.log
> -Dyarn.home.dir=/home/gs/gridre/shelob/share/hadoop -Dhadoop.root.logger=INFO,DRFA -Dyarn.root.logger=INFO,DRFA
> -Djava.library.path=/home/gs/gridre/shelob/share/hadoop/lib/native/Linux-amd64-64 -classpath
> /home/gs/gridre/shelob/conf/hadoop:/home/gs/gridre/shelob/conf/hadoop:/home/gs/gridre/shelob/conf/hadoop:/home/gs/gridre/shelob/conf/hadoop:/home/gs/gridre/shelob/share/hadoop/share/hadoop/common/lib/*:/home/gs/gridre/shelob/share/hadoop/share/hadoop/common/*:/home/gs/gridre/shelob/share/hadoop/hadoop-*-capacity-scheduler.jar:/home/gs/gridre/shelob/share/hadoop/hadoop-*-capacity-scheduler.jar:/home/gs/gridre/shelob/share/hadoop/hadoop-*-capacity-scheduler.jar:/home/gs/gridre/shelob/share/hadoop/share/hadoop/hdfs:/home/gs/gridre/shelob/share/hadoop/share/hadoop/hdfs/lib/*:/home/gs/gridre/shelob/share/hadoop/share/hadoop/hdfs/*:/home/gs/gridre/shelob/share/hadoop/share/hadoop/mapreduce/lib/*:/home/gs/gridre/shelob/share/hadoop/share/hadoop/mapreduce/*:/home/gs/java/jdk64/current/lib/tools.jar:/home/gs/gridre/shelob/share/hadoop/share/hadoop/mapreduce/*:/home/gs/gridre/shelob/share/hadoop/share/hadoop/mapreduce/lib/*
> org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer
> 1) On the RM with older hadoop version where the job summary log does not exist
> jobhistory ps shows using the option:
> -Dmapred.jobsummary.logger=INFO,console
> 2) On the RM with older hadoop version where the job summary log exists
> jobhistory ps shows using the option:
> -Dmapred.jobsummary.logger=INFO,JSA
> -Dmapred.jobsummary.logger=INFO,JSA
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3843) Job summary log file found
missing on the RM host
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13205897#comment-13205897 ]
Hudson commented on MAPREDUCE-3843:
-----------------------------------
Integrated in Hadoop-Common-trunk-Commit #1712 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1712/])
MAPREDUCE-3843. Job summary log file found missing on the RM host (Anupam Seth via tgraves)
tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1242976
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/bin/mr-jobhistory-daemon.sh
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/ClusterSetup.apt.vm
> Job summary log file found missing on the RM host
> -------------------------------------------------
>
> Key: MAPREDUCE-3843
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3843
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobhistoryserver, mrv2
> Affects Versions: 0.23.0
> Reporter: Anupam Seth
> Assignee: Anupam Seth
> Priority: Critical
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3843.patch, MAPREDUCE-3843.patch, MAPREDUCE-3843.patch, MAPREDUCE-3843.patch, MAPREDUCE-3843.patch, MAPREDUCE-3843.patch
>
>
> This bug was found by Phil Su as part of our testing.
> After MAPREDUCE-3354 went in, the Job summary log file seems to have gone missing on the
> RM host.
> The job summary log appears to be interspersed in yarn-mapredqa-historyserver-<host>.out.
> e.g.
> 12/02/09 15:57:21 INFO jobhistory.JobSummary:
> jobId=job_1328658619341_0011,submitTime=1328802904381,launchTime=1328802909977,firstMapTaskLaunchTime=1328802912116,firstReduceTaskLaunchTime=1328802915074,finishTime=1328802933797,resourc
> esPerMap=1024,resourcesPerReduce=2048,numMaps=10,numReduces=10,user=hadoopqa,queue=default,status=KILLED,mapSlotSeconds=0,reduceSlotSeconds=0
> 1) On the RM with older hadoop version where the job summary log does not exist
> mapredqa 10903 0.0 1.2 1424404 210240 ? Sl Feb07 0:19 /home/gs/java/jdk64/current/bin/java -Xmx1000m
> -Djava.net.preferIPv4Stack=true
> -Djava.library.path=/home/gs/gridre/theoden/share/hadoop/lib/native/Linux-amd64-64:/
> home/gs/gridre/theoden/share/hadoop/lib/native/Linux-amd64-64 -Dhadoop.log.dir=/home/gs/var/log/mapredqa
> -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/home/gs/gridre/theoden/share/hadoop -Dhadoop.id.str=mapredqa
> -Dhadoop
> .root.logger=INFO,console
> -Djava.library.path=/home/gs/gridre/theoden/share/hadoop/lib/native/Linux-amd64-64:/home/gs/gridre/theoden/share/hadoop/lib/native/Linux-amd64-64:/home/gs/gridre/theoden/share/hadoop/lib/nat
> ive -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -Dmapred.jobsummary.logger=INFO,console
> -Dhadoop.security.logger=INFO,NullAppender org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer
> 2) On the RM with older hadoop version where the job summary log exists
> mapredqa 24851 0.0 0.5 1463280 90516 ? Sl Jan25 0:37 /home/gs/java/jdk64/current/bin/java
> -Dproc_historyserver -Xmx1000m -Dmapred.jobsummary.logger=INFO,JSA -Dyarn.log.dir=/home/gs/var/log/mapredqa
> -Dyarn.log.file=yarn.log -Dyarn.home.dir= -Dyarn.id.str= -Dyarn.root.logger=INFO,console
> -Djava.library.path=/home/gs/gridre/shelob/share/hadoop/lib/native/Linux-amd64-64
> -Dyarn.policy.file=hadoop-policy.xml -Dyarn.log.dir=/home/gs/var/log/mapredqa
> -Dyarn.log.file=yarn-mapredqa-historyserver-<host>.log -Dyarn.home.dir= -Dyarn.id.str=mapredqa
> -Dyarn.root.logger=INFO,DRFA -Djava.library.path=/home/gs/gridre/shelob/share/hadoop/lib/native/Linux-amd64-64
> -Dyarn.policy.file=hadoop-policy.xml -Dmapred.jobsummary.logger=INFO,JSA -Dhadoop.log.dir=/home/gs/var/log/mapredqa
> -Dyarn.log.dir=/home/gs/var/log/mapredqa
> -Dhadoop.log.file=yarn-mapredqa-historyserver-<host>.log
> -Dyarn.log.file=yarn-mapredqa-historyserver-<host>.log
> -Dyarn.home.dir=/home/gs/gridre/shelob/share/hadoop -Dhadoop.root.logger=INFO,DRFA -Dyarn.root.logger=INFO,DRFA
> -Djava.library.path=/home/gs/gridre/shelob/share/hadoop/lib/native/Linux-amd64-64 -classpath
> /home/gs/gridre/shelob/conf/hadoop:/home/gs/gridre/shelob/conf/hadoop:/home/gs/gridre/shelob/conf/hadoop:/home/gs/gridre/shelob/conf/hadoop:/home/gs/gridre/shelob/share/hadoop/share/hadoop/common/lib/*:/home/gs/gridre/shelob/share/hadoop/share/hadoop/common/*:/home/gs/gridre/shelob/share/hadoop/hadoop-*-capacity-scheduler.jar:/home/gs/gridre/shelob/share/hadoop/hadoop-*-capacity-scheduler.jar:/home/gs/gridre/shelob/share/hadoop/hadoop-*-capacity-scheduler.jar:/home/gs/gridre/shelob/share/hadoop/share/hadoop/hdfs:/home/gs/gridre/shelob/share/hadoop/share/hadoop/hdfs/lib/*:/home/gs/gridre/shelob/share/hadoop/share/hadoop/hdfs/*:/home/gs/gridre/shelob/share/hadoop/share/hadoop/mapreduce/lib/*:/home/gs/gridre/shelob/share/hadoop/share/hadoop/mapreduce/*:/home/gs/java/jdk64/current/lib/tools.jar:/home/gs/gridre/shelob/share/hadoop/share/hadoop/mapreduce/*:/home/gs/gridre/shelob/share/hadoop/share/hadoop/mapreduce/lib/*
> org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer
> 1) On the RM with older hadoop version where the job summary log does not exist
> jobhistory ps shows using the option:
> -Dmapred.jobsummary.logger=INFO,console
> 2) On the RM with older hadoop version where the job summary log exists
> jobhistory ps shows using the option:
> -Dmapred.jobsummary.logger=INFO,JSA
> -Dmapred.jobsummary.logger=INFO,JSA
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3843) Job summary log file found
missing on the RM host
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13206109#comment-13206109 ]
Hudson commented on MAPREDUCE-3843:
-----------------------------------
Integrated in Hadoop-Mapreduce-trunk #986 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/986/])
MAPREDUCE-3843. Job summary log file found missing on the RM host (Anupam Seth via tgraves)
tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1242976
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/bin/mr-jobhistory-daemon.sh
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/ClusterSetup.apt.vm
> Job summary log file found missing on the RM host
> -------------------------------------------------
>
> Key: MAPREDUCE-3843
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3843
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobhistoryserver, mrv2
> Affects Versions: 0.23.0
> Reporter: Anupam Seth
> Assignee: Anupam Seth
> Priority: Critical
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3843.patch, MAPREDUCE-3843.patch, MAPREDUCE-3843.patch, MAPREDUCE-3843.patch, MAPREDUCE-3843.patch, MAPREDUCE-3843.patch
>
>
> This bug was found by Phil Su as part of our testing.
> After MAPREDUCE-3354 went in, the Job summary log file seems to have gone missing on the
> RM host.
> The job summary log appears to be interspersed in yarn-mapredqa-historyserver-<host>.out.
> e.g.
> 12/02/09 15:57:21 INFO jobhistory.JobSummary:
> jobId=job_1328658619341_0011,submitTime=1328802904381,launchTime=1328802909977,firstMapTaskLaunchTime=1328802912116,firstReduceTaskLaunchTime=1328802915074,finishTime=1328802933797,resourc
> esPerMap=1024,resourcesPerReduce=2048,numMaps=10,numReduces=10,user=hadoopqa,queue=default,status=KILLED,mapSlotSeconds=0,reduceSlotSeconds=0
> 1) On the RM with older hadoop version where the job summary log does not exist
> mapredqa 10903 0.0 1.2 1424404 210240 ? Sl Feb07 0:19 /home/gs/java/jdk64/current/bin/java -Xmx1000m
> -Djava.net.preferIPv4Stack=true
> -Djava.library.path=/home/gs/gridre/theoden/share/hadoop/lib/native/Linux-amd64-64:/
> home/gs/gridre/theoden/share/hadoop/lib/native/Linux-amd64-64 -Dhadoop.log.dir=/home/gs/var/log/mapredqa
> -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/home/gs/gridre/theoden/share/hadoop -Dhadoop.id.str=mapredqa
> -Dhadoop
> .root.logger=INFO,console
> -Djava.library.path=/home/gs/gridre/theoden/share/hadoop/lib/native/Linux-amd64-64:/home/gs/gridre/theoden/share/hadoop/lib/native/Linux-amd64-64:/home/gs/gridre/theoden/share/hadoop/lib/nat
> ive -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -Dmapred.jobsummary.logger=INFO,console
> -Dhadoop.security.logger=INFO,NullAppender org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer
> 2) On the RM with older hadoop version where the job summary log exists
> mapredqa 24851 0.0 0.5 1463280 90516 ? Sl Jan25 0:37 /home/gs/java/jdk64/current/bin/java
> -Dproc_historyserver -Xmx1000m -Dmapred.jobsummary.logger=INFO,JSA -Dyarn.log.dir=/home/gs/var/log/mapredqa
> -Dyarn.log.file=yarn.log -Dyarn.home.dir= -Dyarn.id.str= -Dyarn.root.logger=INFO,console
> -Djava.library.path=/home/gs/gridre/shelob/share/hadoop/lib/native/Linux-amd64-64
> -Dyarn.policy.file=hadoop-policy.xml -Dyarn.log.dir=/home/gs/var/log/mapredqa
> -Dyarn.log.file=yarn-mapredqa-historyserver-<host>.log -Dyarn.home.dir= -Dyarn.id.str=mapredqa
> -Dyarn.root.logger=INFO,DRFA -Djava.library.path=/home/gs/gridre/shelob/share/hadoop/lib/native/Linux-amd64-64
> -Dyarn.policy.file=hadoop-policy.xml -Dmapred.jobsummary.logger=INFO,JSA -Dhadoop.log.dir=/home/gs/var/log/mapredqa
> -Dyarn.log.dir=/home/gs/var/log/mapredqa
> -Dhadoop.log.file=yarn-mapredqa-historyserver-<host>.log
> -Dyarn.log.file=yarn-mapredqa-historyserver-<host>.log
> -Dyarn.home.dir=/home/gs/gridre/shelob/share/hadoop -Dhadoop.root.logger=INFO,DRFA -Dyarn.root.logger=INFO,DRFA
> -Djava.library.path=/home/gs/gridre/shelob/share/hadoop/lib/native/Linux-amd64-64 -classpath
> /home/gs/gridre/shelob/conf/hadoop:/home/gs/gridre/shelob/conf/hadoop:/home/gs/gridre/shelob/conf/hadoop:/home/gs/gridre/shelob/conf/hadoop:/home/gs/gridre/shelob/share/hadoop/share/hadoop/common/lib/*:/home/gs/gridre/shelob/share/hadoop/share/hadoop/common/*:/home/gs/gridre/shelob/share/hadoop/hadoop-*-capacity-scheduler.jar:/home/gs/gridre/shelob/share/hadoop/hadoop-*-capacity-scheduler.jar:/home/gs/gridre/shelob/share/hadoop/hadoop-*-capacity-scheduler.jar:/home/gs/gridre/shelob/share/hadoop/share/hadoop/hdfs:/home/gs/gridre/shelob/share/hadoop/share/hadoop/hdfs/lib/*:/home/gs/gridre/shelob/share/hadoop/share/hadoop/hdfs/*:/home/gs/gridre/shelob/share/hadoop/share/hadoop/mapreduce/lib/*:/home/gs/gridre/shelob/share/hadoop/share/hadoop/mapreduce/*:/home/gs/java/jdk64/current/lib/tools.jar:/home/gs/gridre/shelob/share/hadoop/share/hadoop/mapreduce/*:/home/gs/gridre/shelob/share/hadoop/share/hadoop/mapreduce/lib/*
> org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer
> 1) On the RM with older hadoop version where the job summary log does not exist
> jobhistory ps shows using the option:
> -Dmapred.jobsummary.logger=INFO,console
> 2) On the RM with older hadoop version where the job summary log exists
> jobhistory ps shows using the option:
> -Dmapred.jobsummary.logger=INFO,JSA
> -Dmapred.jobsummary.logger=INFO,JSA
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3843) Job summary log file found
missing on the RM host
Posted by "Anupam Seth (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Anupam Seth updated MAPREDUCE-3843:
-----------------------------------
Status: Open (was: Patch Available)
> Job summary log file found missing on the RM host
> -------------------------------------------------
>
> Key: MAPREDUCE-3843
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3843
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobhistoryserver, mrv2
> Affects Versions: 0.23.0
> Reporter: Anupam Seth
> Assignee: Anupam Seth
> Priority: Critical
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3843.patch
>
>
> This bug was found by Phil Su as part of our testing.
> After MAPREDUCE-3354 went in, the Job summary log file seems to have gone missing on the
> RM host.
> The job summary log appears to be interspersed in yarn-mapredqa-historyserver-<host>.out.
> e.g.
> 12/02/09 15:57:21 INFO jobhistory.JobSummary:
> jobId=job_1328658619341_0011,submitTime=1328802904381,launchTime=1328802909977,firstMapTaskLaunchTime=1328802912116,firstReduceTaskLaunchTime=1328802915074,finishTime=1328802933797,resourc
> esPerMap=1024,resourcesPerReduce=2048,numMaps=10,numReduces=10,user=hadoopqa,queue=default,status=KILLED,mapSlotSeconds=0,reduceSlotSeconds=0
> 1) On the RM with older hadoop version where the job summary log does not exist
> mapredqa 10903 0.0 1.2 1424404 210240 ? Sl Feb07 0:19 /home/gs/java/jdk64/current/bin/java -Xmx1000m
> -Djava.net.preferIPv4Stack=true
> -Djava.library.path=/home/gs/gridre/theoden/share/hadoop/lib/native/Linux-amd64-64:/
> home/gs/gridre/theoden/share/hadoop/lib/native/Linux-amd64-64 -Dhadoop.log.dir=/home/gs/var/log/mapredqa
> -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/home/gs/gridre/theoden/share/hadoop -Dhadoop.id.str=mapredqa
> -Dhadoop
> .root.logger=INFO,console
> -Djava.library.path=/home/gs/gridre/theoden/share/hadoop/lib/native/Linux-amd64-64:/home/gs/gridre/theoden/share/hadoop/lib/native/Linux-amd64-64:/home/gs/gridre/theoden/share/hadoop/lib/nat
> ive -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -Dmapred.jobsummary.logger=INFO,console
> -Dhadoop.security.logger=INFO,NullAppender org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer
> 2) On the RM with older hadoop version where the job summary log exists
> mapredqa 24851 0.0 0.5 1463280 90516 ? Sl Jan25 0:37 /home/gs/java/jdk64/current/bin/java
> -Dproc_historyserver -Xmx1000m -Dmapred.jobsummary.logger=INFO,JSA -Dyarn.log.dir=/home/gs/var/log/mapredqa
> -Dyarn.log.file=yarn.log -Dyarn.home.dir= -Dyarn.id.str= -Dyarn.root.logger=INFO,console
> -Djava.library.path=/home/gs/gridre/shelob/share/hadoop/lib/native/Linux-amd64-64
> -Dyarn.policy.file=hadoop-policy.xml -Dyarn.log.dir=/home/gs/var/log/mapredqa
> -Dyarn.log.file=yarn-mapredqa-historyserver-<host>.log -Dyarn.home.dir= -Dyarn.id.str=mapredqa
> -Dyarn.root.logger=INFO,DRFA -Djava.library.path=/home/gs/gridre/shelob/share/hadoop/lib/native/Linux-amd64-64
> -Dyarn.policy.file=hadoop-policy.xml -Dmapred.jobsummary.logger=INFO,JSA -Dhadoop.log.dir=/home/gs/var/log/mapredqa
> -Dyarn.log.dir=/home/gs/var/log/mapredqa
> -Dhadoop.log.file=yarn-mapredqa-historyserver-<host>.log
> -Dyarn.log.file=yarn-mapredqa-historyserver-<host>.log
> -Dyarn.home.dir=/home/gs/gridre/shelob/share/hadoop -Dhadoop.root.logger=INFO,DRFA -Dyarn.root.logger=INFO,DRFA
> -Djava.library.path=/home/gs/gridre/shelob/share/hadoop/lib/native/Linux-amd64-64 -classpath
> /home/gs/gridre/shelob/conf/hadoop:/home/gs/gridre/shelob/conf/hadoop:/home/gs/gridre/shelob/conf/hadoop:/home/gs/gridre/shelob/conf/hadoop:/home/gs/gridre/shelob/share/hadoop/share/hadoop/common/lib/*:/home/gs/gridre/shelob/share/hadoop/share/hadoop/common/*:/home/gs/gridre/shelob/share/hadoop/hadoop-*-capacity-scheduler.jar:/home/gs/gridre/shelob/share/hadoop/hadoop-*-capacity-scheduler.jar:/home/gs/gridre/shelob/share/hadoop/hadoop-*-capacity-scheduler.jar:/home/gs/gridre/shelob/share/hadoop/share/hadoop/hdfs:/home/gs/gridre/shelob/share/hadoop/share/hadoop/hdfs/lib/*:/home/gs/gridre/shelob/share/hadoop/share/hadoop/hdfs/*:/home/gs/gridre/shelob/share/hadoop/share/hadoop/mapreduce/lib/*:/home/gs/gridre/shelob/share/hadoop/share/hadoop/mapreduce/*:/home/gs/java/jdk64/current/lib/tools.jar:/home/gs/gridre/shelob/share/hadoop/share/hadoop/mapreduce/*:/home/gs/gridre/shelob/share/hadoop/share/hadoop/mapreduce/lib/*
> org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer
> 1) On the RM with older hadoop version where the job summary log does not exist
> jobhistory ps shows using the option:
> -Dmapred.jobsummary.logger=INFO,console
> 2) On the RM with older hadoop version where the job summary log exists
> jobhistory ps shows using the option:
> -Dmapred.jobsummary.logger=INFO,JSA
> -Dmapred.jobsummary.logger=INFO,JSA
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3843) Job summary log file found
missing on the RM host
Posted by "Anupam Seth (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Anupam Seth updated MAPREDUCE-3843:
-----------------------------------
Description:
This bug was found by Phil Su as part of our testing.
After MAPREDUCE-3354 went in, the Job summary log file seems to have gone missing on the
RM host.
The job summary log appears to be interspersed in yarn-mapredqa-historyserver-<host>.out.
e.g.
12/02/09 15:57:21 INFO jobhistory.JobSummary:
jobId=job_1328658619341_0011,submitTime=1328802904381,launchTime=1328802909977,firstMapTaskLaunchTime=1328802912116,firstReduceTaskLaunchTime=1328802915074,finishTime=1328802933797,resourc
esPerMap=1024,resourcesPerReduce=2048,numMaps=10,numReduces=10,user=hadoopqa,queue=default,status=KILLED,mapSlotSeconds=0,reduceSlotSeconds=0
1) On the RM with older hadoop version where the job summary log does not exist
mapredqa 10903 0.0 1.2 1424404 210240 ? Sl Feb07 0:19 /home/gs/java/jdk64/current/bin/java -Xmx1000m
-Djava.net.preferIPv4Stack=true
-Djava.library.path=/home/gs/gridre/theoden/share/hadoop/lib/native/Linux-amd64-64:/
home/gs/gridre/theoden/share/hadoop/lib/native/Linux-amd64-64 -Dhadoop.log.dir=/home/gs/var/log/mapredqa
-Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/home/gs/gridre/theoden/share/hadoop -Dhadoop.id.str=mapredqa
-Dhadoop
.root.logger=INFO,console
-Djava.library.path=/home/gs/gridre/theoden/share/hadoop/lib/native/Linux-amd64-64:/home/gs/gridre/theoden/share/hadoop/lib/native/Linux-amd64-64:/home/gs/gridre/theoden/share/hadoop/lib/nat
ive -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -Dmapred.jobsummary.logger=INFO,console
-Dhadoop.security.logger=INFO,NullAppender org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer
2) On the RM with older hadoop version where the job summary log exists
mapredqa 24851 0.0 0.5 1463280 90516 ? Sl Jan25 0:37 /home/gs/java/jdk64/current/bin/java
-Dproc_historyserver -Xmx1000m -Dmapred.jobsummary.logger=INFO,JSA -Dyarn.log.dir=/home/gs/var/log/mapredqa
-Dyarn.log.file=yarn.log -Dyarn.home.dir= -Dyarn.id.str= -Dyarn.root.logger=INFO,console
-Djava.library.path=/home/gs/gridre/shelob/share/hadoop/lib/native/Linux-amd64-64
-Dyarn.policy.file=hadoop-policy.xml -Dyarn.log.dir=/home/gs/var/log/mapredqa
-Dyarn.log.file=yarn-mapredqa-historyserver-<host>.log -Dyarn.home.dir= -Dyarn.id.str=mapredqa
-Dyarn.root.logger=INFO,DRFA -Djava.library.path=/home/gs/gridre/shelob/share/hadoop/lib/native/Linux-amd64-64
-Dyarn.policy.file=hadoop-policy.xml -Dmapred.jobsummary.logger=INFO,JSA -Dhadoop.log.dir=/home/gs/var/log/mapredqa
-Dyarn.log.dir=/home/gs/var/log/mapredqa
-Dhadoop.log.file=yarn-mapredqa-historyserver-<host>.log
-Dyarn.log.file=yarn-mapredqa-historyserver-<host>.log
-Dyarn.home.dir=/home/gs/gridre/shelob/share/hadoop -Dhadoop.root.logger=INFO,DRFA -Dyarn.root.logger=INFO,DRFA
-Djava.library.path=/home/gs/gridre/shelob/share/hadoop/lib/native/Linux-amd64-64 -classpath
/home/gs/gridre/shelob/conf/hadoop:/home/gs/gridre/shelob/conf/hadoop:/home/gs/gridre/shelob/conf/hadoop:/home/gs/gridre/shelob/conf/hadoop:/home/gs/gridre/shelob/share/hadoop/share/hadoop/common/lib/*:/home/gs/gridre/shelob/share/hadoop/share/hadoop/common/*:/home/gs/gridre/shelob/share/hadoop/hadoop-*-capacity-scheduler.jar:/home/gs/gridre/shelob/share/hadoop/hadoop-*-capacity-scheduler.jar:/home/gs/gridre/shelob/share/hadoop/hadoop-*-capacity-scheduler.jar:/home/gs/gridre/shelob/share/hadoop/share/hadoop/hdfs:/home/gs/gridre/shelob/share/hadoop/share/hadoop/hdfs/lib/*:/home/gs/gridre/shelob/share/hadoop/share/hadoop/hdfs/*:/home/gs/gridre/shelob/share/hadoop/share/hadoop/mapreduce/lib/*:/home/gs/gridre/shelob/share/hadoop/share/hadoop/mapreduce/*:/home/gs/java/jdk64/current/lib/tools.jar:/home/gs/gridre/shelob/share/hadoop/share/hadoop/mapreduce/*:/home/gs/gridre/shelob/share/hadoop/share/hadoop/mapreduce/lib/*
org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer
1) On the RM with older hadoop version where the job summary log does not exist
jobhistory ps shows using the option:
-Dmapred.jobsummary.logger=INFO,console
2) On the RM with older hadoop version where the job summary log exists
jobhistory ps shows using the option:
-Dmapred.jobsummary.logger=INFO,JSA
-Dmapred.jobsummary.logger=INFO,JSA
was:JobHistoryServer belongs to mapreduce land.
Priority: Critical (was: Blocker)
Assignee: Anupam Seth (was: Jonathan Eagles)
Summary: Job summary log file found missing on the RM host (was: CLONE - JobHistoryServer should be started by bin/mapred and not by bin/yarn)
> Job summary log file found missing on the RM host
> -------------------------------------------------
>
> Key: MAPREDUCE-3843
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3843
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobhistoryserver, mrv2
> Affects Versions: 0.23.0
> Reporter: Anupam Seth
> Assignee: Anupam Seth
> Priority: Critical
> Fix For: 0.23.1
>
>
> This bug was found by Phil Su as part of our testing.
> After MAPREDUCE-3354 went in, the Job summary log file seems to have gone missing on the
> RM host.
> The job summary log appears to be interspersed in yarn-mapredqa-historyserver-<host>.out.
> e.g.
> 12/02/09 15:57:21 INFO jobhistory.JobSummary:
> jobId=job_1328658619341_0011,submitTime=1328802904381,launchTime=1328802909977,firstMapTaskLaunchTime=1328802912116,firstReduceTaskLaunchTime=1328802915074,finishTime=1328802933797,resourc
> esPerMap=1024,resourcesPerReduce=2048,numMaps=10,numReduces=10,user=hadoopqa,queue=default,status=KILLED,mapSlotSeconds=0,reduceSlotSeconds=0
> 1) On the RM with older hadoop version where the job summary log does not exist
> mapredqa 10903 0.0 1.2 1424404 210240 ? Sl Feb07 0:19 /home/gs/java/jdk64/current/bin/java -Xmx1000m
> -Djava.net.preferIPv4Stack=true
> -Djava.library.path=/home/gs/gridre/theoden/share/hadoop/lib/native/Linux-amd64-64:/
> home/gs/gridre/theoden/share/hadoop/lib/native/Linux-amd64-64 -Dhadoop.log.dir=/home/gs/var/log/mapredqa
> -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/home/gs/gridre/theoden/share/hadoop -Dhadoop.id.str=mapredqa
> -Dhadoop
> .root.logger=INFO,console
> -Djava.library.path=/home/gs/gridre/theoden/share/hadoop/lib/native/Linux-amd64-64:/home/gs/gridre/theoden/share/hadoop/lib/native/Linux-amd64-64:/home/gs/gridre/theoden/share/hadoop/lib/nat
> ive -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -Dmapred.jobsummary.logger=INFO,console
> -Dhadoop.security.logger=INFO,NullAppender org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer
> 2) On the RM with older hadoop version where the job summary log exists
> mapredqa 24851 0.0 0.5 1463280 90516 ? Sl Jan25 0:37 /home/gs/java/jdk64/current/bin/java
> -Dproc_historyserver -Xmx1000m -Dmapred.jobsummary.logger=INFO,JSA -Dyarn.log.dir=/home/gs/var/log/mapredqa
> -Dyarn.log.file=yarn.log -Dyarn.home.dir= -Dyarn.id.str= -Dyarn.root.logger=INFO,console
> -Djava.library.path=/home/gs/gridre/shelob/share/hadoop/lib/native/Linux-amd64-64
> -Dyarn.policy.file=hadoop-policy.xml -Dyarn.log.dir=/home/gs/var/log/mapredqa
> -Dyarn.log.file=yarn-mapredqa-historyserver-<host>.log -Dyarn.home.dir= -Dyarn.id.str=mapredqa
> -Dyarn.root.logger=INFO,DRFA -Djava.library.path=/home/gs/gridre/shelob/share/hadoop/lib/native/Linux-amd64-64
> -Dyarn.policy.file=hadoop-policy.xml -Dmapred.jobsummary.logger=INFO,JSA -Dhadoop.log.dir=/home/gs/var/log/mapredqa
> -Dyarn.log.dir=/home/gs/var/log/mapredqa
> -Dhadoop.log.file=yarn-mapredqa-historyserver-<host>.log
> -Dyarn.log.file=yarn-mapredqa-historyserver-<host>.log
> -Dyarn.home.dir=/home/gs/gridre/shelob/share/hadoop -Dhadoop.root.logger=INFO,DRFA -Dyarn.root.logger=INFO,DRFA
> -Djava.library.path=/home/gs/gridre/shelob/share/hadoop/lib/native/Linux-amd64-64 -classpath
> /home/gs/gridre/shelob/conf/hadoop:/home/gs/gridre/shelob/conf/hadoop:/home/gs/gridre/shelob/conf/hadoop:/home/gs/gridre/shelob/conf/hadoop:/home/gs/gridre/shelob/share/hadoop/share/hadoop/common/lib/*:/home/gs/gridre/shelob/share/hadoop/share/hadoop/common/*:/home/gs/gridre/shelob/share/hadoop/hadoop-*-capacity-scheduler.jar:/home/gs/gridre/shelob/share/hadoop/hadoop-*-capacity-scheduler.jar:/home/gs/gridre/shelob/share/hadoop/hadoop-*-capacity-scheduler.jar:/home/gs/gridre/shelob/share/hadoop/share/hadoop/hdfs:/home/gs/gridre/shelob/share/hadoop/share/hadoop/hdfs/lib/*:/home/gs/gridre/shelob/share/hadoop/share/hadoop/hdfs/*:/home/gs/gridre/shelob/share/hadoop/share/hadoop/mapreduce/lib/*:/home/gs/gridre/shelob/share/hadoop/share/hadoop/mapreduce/*:/home/gs/java/jdk64/current/lib/tools.jar:/home/gs/gridre/shelob/share/hadoop/share/hadoop/mapreduce/*:/home/gs/gridre/shelob/share/hadoop/share/hadoop/mapreduce/lib/*
> org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer
> 1) On the RM with older hadoop version where the job summary log does not exist
> jobhistory ps shows using the option:
> -Dmapred.jobsummary.logger=INFO,console
> 2) On the RM with older hadoop version where the job summary log exists
> jobhistory ps shows using the option:
> -Dmapred.jobsummary.logger=INFO,JSA
> -Dmapred.jobsummary.logger=INFO,JSA
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3843) Job summary log file found
missing on the RM host
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13205851#comment-13205851 ]
Hadoop QA commented on MAPREDUCE-3843:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12514175/MAPREDUCE-3843.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests:
org.apache.hadoop.yarn.util.TestLinuxResourceCalculatorPlugin
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1842//testReport/
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1842//console
This message is automatically generated.
> Job summary log file found missing on the RM host
> -------------------------------------------------
>
> Key: MAPREDUCE-3843
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3843
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobhistoryserver, mrv2
> Affects Versions: 0.23.0
> Reporter: Anupam Seth
> Assignee: Anupam Seth
> Priority: Critical
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3843.patch, MAPREDUCE-3843.patch, MAPREDUCE-3843.patch, MAPREDUCE-3843.patch, MAPREDUCE-3843.patch, MAPREDUCE-3843.patch
>
>
> This bug was found by Phil Su as part of our testing.
> After MAPREDUCE-3354 went in, the Job summary log file seems to have gone missing on the
> RM host.
> The job summary log appears to be interspersed in yarn-mapredqa-historyserver-<host>.out.
> e.g.
> 12/02/09 15:57:21 INFO jobhistory.JobSummary:
> jobId=job_1328658619341_0011,submitTime=1328802904381,launchTime=1328802909977,firstMapTaskLaunchTime=1328802912116,firstReduceTaskLaunchTime=1328802915074,finishTime=1328802933797,resourc
> esPerMap=1024,resourcesPerReduce=2048,numMaps=10,numReduces=10,user=hadoopqa,queue=default,status=KILLED,mapSlotSeconds=0,reduceSlotSeconds=0
> 1) On the RM with older hadoop version where the job summary log does not exist
> mapredqa 10903 0.0 1.2 1424404 210240 ? Sl Feb07 0:19 /home/gs/java/jdk64/current/bin/java -Xmx1000m
> -Djava.net.preferIPv4Stack=true
> -Djava.library.path=/home/gs/gridre/theoden/share/hadoop/lib/native/Linux-amd64-64:/
> home/gs/gridre/theoden/share/hadoop/lib/native/Linux-amd64-64 -Dhadoop.log.dir=/home/gs/var/log/mapredqa
> -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/home/gs/gridre/theoden/share/hadoop -Dhadoop.id.str=mapredqa
> -Dhadoop
> .root.logger=INFO,console
> -Djava.library.path=/home/gs/gridre/theoden/share/hadoop/lib/native/Linux-amd64-64:/home/gs/gridre/theoden/share/hadoop/lib/native/Linux-amd64-64:/home/gs/gridre/theoden/share/hadoop/lib/nat
> ive -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -Dmapred.jobsummary.logger=INFO,console
> -Dhadoop.security.logger=INFO,NullAppender org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer
> 2) On the RM with older hadoop version where the job summary log exists
> mapredqa 24851 0.0 0.5 1463280 90516 ? Sl Jan25 0:37 /home/gs/java/jdk64/current/bin/java
> -Dproc_historyserver -Xmx1000m -Dmapred.jobsummary.logger=INFO,JSA -Dyarn.log.dir=/home/gs/var/log/mapredqa
> -Dyarn.log.file=yarn.log -Dyarn.home.dir= -Dyarn.id.str= -Dyarn.root.logger=INFO,console
> -Djava.library.path=/home/gs/gridre/shelob/share/hadoop/lib/native/Linux-amd64-64
> -Dyarn.policy.file=hadoop-policy.xml -Dyarn.log.dir=/home/gs/var/log/mapredqa
> -Dyarn.log.file=yarn-mapredqa-historyserver-<host>.log -Dyarn.home.dir= -Dyarn.id.str=mapredqa
> -Dyarn.root.logger=INFO,DRFA -Djava.library.path=/home/gs/gridre/shelob/share/hadoop/lib/native/Linux-amd64-64
> -Dyarn.policy.file=hadoop-policy.xml -Dmapred.jobsummary.logger=INFO,JSA -Dhadoop.log.dir=/home/gs/var/log/mapredqa
> -Dyarn.log.dir=/home/gs/var/log/mapredqa
> -Dhadoop.log.file=yarn-mapredqa-historyserver-<host>.log
> -Dyarn.log.file=yarn-mapredqa-historyserver-<host>.log
> -Dyarn.home.dir=/home/gs/gridre/shelob/share/hadoop -Dhadoop.root.logger=INFO,DRFA -Dyarn.root.logger=INFO,DRFA
> -Djava.library.path=/home/gs/gridre/shelob/share/hadoop/lib/native/Linux-amd64-64 -classpath
> /home/gs/gridre/shelob/conf/hadoop:/home/gs/gridre/shelob/conf/hadoop:/home/gs/gridre/shelob/conf/hadoop:/home/gs/gridre/shelob/conf/hadoop:/home/gs/gridre/shelob/share/hadoop/share/hadoop/common/lib/*:/home/gs/gridre/shelob/share/hadoop/share/hadoop/common/*:/home/gs/gridre/shelob/share/hadoop/hadoop-*-capacity-scheduler.jar:/home/gs/gridre/shelob/share/hadoop/hadoop-*-capacity-scheduler.jar:/home/gs/gridre/shelob/share/hadoop/hadoop-*-capacity-scheduler.jar:/home/gs/gridre/shelob/share/hadoop/share/hadoop/hdfs:/home/gs/gridre/shelob/share/hadoop/share/hadoop/hdfs/lib/*:/home/gs/gridre/shelob/share/hadoop/share/hadoop/hdfs/*:/home/gs/gridre/shelob/share/hadoop/share/hadoop/mapreduce/lib/*:/home/gs/gridre/shelob/share/hadoop/share/hadoop/mapreduce/*:/home/gs/java/jdk64/current/lib/tools.jar:/home/gs/gridre/shelob/share/hadoop/share/hadoop/mapreduce/*:/home/gs/gridre/shelob/share/hadoop/share/hadoop/mapreduce/lib/*
> org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer
> 1) On the RM with older hadoop version where the job summary log does not exist
> jobhistory ps shows using the option:
> -Dmapred.jobsummary.logger=INFO,console
> 2) On the RM with older hadoop version where the job summary log exists
> jobhistory ps shows using the option:
> -Dmapred.jobsummary.logger=INFO,JSA
> -Dmapred.jobsummary.logger=INFO,JSA
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3843) Job summary log file found
missing on the RM host
Posted by "Thomas Graves (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Thomas Graves updated MAPREDUCE-3843:
-------------------------------------
Attachment: MAPREDUCE-3843.patch
minor update to change HADOOP_PREFIX_HOME to just HADOOP_PREFIX.
> Job summary log file found missing on the RM host
> -------------------------------------------------
>
> Key: MAPREDUCE-3843
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3843
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobhistoryserver, mrv2
> Affects Versions: 0.23.0
> Reporter: Anupam Seth
> Assignee: Anupam Seth
> Priority: Critical
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3843.patch, MAPREDUCE-3843.patch, MAPREDUCE-3843.patch, MAPREDUCE-3843.patch, MAPREDUCE-3843.patch, MAPREDUCE-3843.patch
>
>
> This bug was found by Phil Su as part of our testing.
> After MAPREDUCE-3354 went in, the Job summary log file seems to have gone missing on the
> RM host.
> The job summary log appears to be interspersed in yarn-mapredqa-historyserver-<host>.out.
> e.g.
> 12/02/09 15:57:21 INFO jobhistory.JobSummary:
> jobId=job_1328658619341_0011,submitTime=1328802904381,launchTime=1328802909977,firstMapTaskLaunchTime=1328802912116,firstReduceTaskLaunchTime=1328802915074,finishTime=1328802933797,resourc
> esPerMap=1024,resourcesPerReduce=2048,numMaps=10,numReduces=10,user=hadoopqa,queue=default,status=KILLED,mapSlotSeconds=0,reduceSlotSeconds=0
> 1) On the RM with older hadoop version where the job summary log does not exist
> mapredqa 10903 0.0 1.2 1424404 210240 ? Sl Feb07 0:19 /home/gs/java/jdk64/current/bin/java -Xmx1000m
> -Djava.net.preferIPv4Stack=true
> -Djava.library.path=/home/gs/gridre/theoden/share/hadoop/lib/native/Linux-amd64-64:/
> home/gs/gridre/theoden/share/hadoop/lib/native/Linux-amd64-64 -Dhadoop.log.dir=/home/gs/var/log/mapredqa
> -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/home/gs/gridre/theoden/share/hadoop -Dhadoop.id.str=mapredqa
> -Dhadoop
> .root.logger=INFO,console
> -Djava.library.path=/home/gs/gridre/theoden/share/hadoop/lib/native/Linux-amd64-64:/home/gs/gridre/theoden/share/hadoop/lib/native/Linux-amd64-64:/home/gs/gridre/theoden/share/hadoop/lib/nat
> ive -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -Dmapred.jobsummary.logger=INFO,console
> -Dhadoop.security.logger=INFO,NullAppender org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer
> 2) On the RM with older hadoop version where the job summary log exists
> mapredqa 24851 0.0 0.5 1463280 90516 ? Sl Jan25 0:37 /home/gs/java/jdk64/current/bin/java
> -Dproc_historyserver -Xmx1000m -Dmapred.jobsummary.logger=INFO,JSA -Dyarn.log.dir=/home/gs/var/log/mapredqa
> -Dyarn.log.file=yarn.log -Dyarn.home.dir= -Dyarn.id.str= -Dyarn.root.logger=INFO,console
> -Djava.library.path=/home/gs/gridre/shelob/share/hadoop/lib/native/Linux-amd64-64
> -Dyarn.policy.file=hadoop-policy.xml -Dyarn.log.dir=/home/gs/var/log/mapredqa
> -Dyarn.log.file=yarn-mapredqa-historyserver-<host>.log -Dyarn.home.dir= -Dyarn.id.str=mapredqa
> -Dyarn.root.logger=INFO,DRFA -Djava.library.path=/home/gs/gridre/shelob/share/hadoop/lib/native/Linux-amd64-64
> -Dyarn.policy.file=hadoop-policy.xml -Dmapred.jobsummary.logger=INFO,JSA -Dhadoop.log.dir=/home/gs/var/log/mapredqa
> -Dyarn.log.dir=/home/gs/var/log/mapredqa
> -Dhadoop.log.file=yarn-mapredqa-historyserver-<host>.log
> -Dyarn.log.file=yarn-mapredqa-historyserver-<host>.log
> -Dyarn.home.dir=/home/gs/gridre/shelob/share/hadoop -Dhadoop.root.logger=INFO,DRFA -Dyarn.root.logger=INFO,DRFA
> -Djava.library.path=/home/gs/gridre/shelob/share/hadoop/lib/native/Linux-amd64-64 -classpath
> /home/gs/gridre/shelob/conf/hadoop:/home/gs/gridre/shelob/conf/hadoop:/home/gs/gridre/shelob/conf/hadoop:/home/gs/gridre/shelob/conf/hadoop:/home/gs/gridre/shelob/share/hadoop/share/hadoop/common/lib/*:/home/gs/gridre/shelob/share/hadoop/share/hadoop/common/*:/home/gs/gridre/shelob/share/hadoop/hadoop-*-capacity-scheduler.jar:/home/gs/gridre/shelob/share/hadoop/hadoop-*-capacity-scheduler.jar:/home/gs/gridre/shelob/share/hadoop/hadoop-*-capacity-scheduler.jar:/home/gs/gridre/shelob/share/hadoop/share/hadoop/hdfs:/home/gs/gridre/shelob/share/hadoop/share/hadoop/hdfs/lib/*:/home/gs/gridre/shelob/share/hadoop/share/hadoop/hdfs/*:/home/gs/gridre/shelob/share/hadoop/share/hadoop/mapreduce/lib/*:/home/gs/gridre/shelob/share/hadoop/share/hadoop/mapreduce/*:/home/gs/java/jdk64/current/lib/tools.jar:/home/gs/gridre/shelob/share/hadoop/share/hadoop/mapreduce/*:/home/gs/gridre/shelob/share/hadoop/share/hadoop/mapreduce/lib/*
> org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer
> 1) On the RM with older hadoop version where the job summary log does not exist
> jobhistory ps shows using the option:
> -Dmapred.jobsummary.logger=INFO,console
> 2) On the RM with older hadoop version where the job summary log exists
> jobhistory ps shows using the option:
> -Dmapred.jobsummary.logger=INFO,JSA
> -Dmapred.jobsummary.logger=INFO,JSA
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3843) Job summary log file found
missing on the RM host
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13205879#comment-13205879 ]
Hudson commented on MAPREDUCE-3843:
-----------------------------------
Integrated in Hadoop-Mapreduce-0.23-Commit #541 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/541/])
merge -r 1242975:1242976 from trunk. FIXES: MAPREDUCE-3843
tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1242977
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/bin/mr-jobhistory-daemon.sh
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/ClusterSetup.apt.vm
> Job summary log file found missing on the RM host
> -------------------------------------------------
>
> Key: MAPREDUCE-3843
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3843
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobhistoryserver, mrv2
> Affects Versions: 0.23.0
> Reporter: Anupam Seth
> Assignee: Anupam Seth
> Priority: Critical
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3843.patch, MAPREDUCE-3843.patch, MAPREDUCE-3843.patch, MAPREDUCE-3843.patch, MAPREDUCE-3843.patch, MAPREDUCE-3843.patch
>
>
> This bug was found by Phil Su as part of our testing.
> After MAPREDUCE-3354 went in, the Job summary log file seems to have gone missing on the
> RM host.
> The job summary log appears to be interspersed in yarn-mapredqa-historyserver-<host>.out.
> e.g.
> 12/02/09 15:57:21 INFO jobhistory.JobSummary:
> jobId=job_1328658619341_0011,submitTime=1328802904381,launchTime=1328802909977,firstMapTaskLaunchTime=1328802912116,firstReduceTaskLaunchTime=1328802915074,finishTime=1328802933797,resourc
> esPerMap=1024,resourcesPerReduce=2048,numMaps=10,numReduces=10,user=hadoopqa,queue=default,status=KILLED,mapSlotSeconds=0,reduceSlotSeconds=0
> 1) On the RM with older hadoop version where the job summary log does not exist
> mapredqa 10903 0.0 1.2 1424404 210240 ? Sl Feb07 0:19 /home/gs/java/jdk64/current/bin/java -Xmx1000m
> -Djava.net.preferIPv4Stack=true
> -Djava.library.path=/home/gs/gridre/theoden/share/hadoop/lib/native/Linux-amd64-64:/
> home/gs/gridre/theoden/share/hadoop/lib/native/Linux-amd64-64 -Dhadoop.log.dir=/home/gs/var/log/mapredqa
> -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/home/gs/gridre/theoden/share/hadoop -Dhadoop.id.str=mapredqa
> -Dhadoop
> .root.logger=INFO,console
> -Djava.library.path=/home/gs/gridre/theoden/share/hadoop/lib/native/Linux-amd64-64:/home/gs/gridre/theoden/share/hadoop/lib/native/Linux-amd64-64:/home/gs/gridre/theoden/share/hadoop/lib/nat
> ive -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -Dmapred.jobsummary.logger=INFO,console
> -Dhadoop.security.logger=INFO,NullAppender org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer
> 2) On the RM with older hadoop version where the job summary log exists
> mapredqa 24851 0.0 0.5 1463280 90516 ? Sl Jan25 0:37 /home/gs/java/jdk64/current/bin/java
> -Dproc_historyserver -Xmx1000m -Dmapred.jobsummary.logger=INFO,JSA -Dyarn.log.dir=/home/gs/var/log/mapredqa
> -Dyarn.log.file=yarn.log -Dyarn.home.dir= -Dyarn.id.str= -Dyarn.root.logger=INFO,console
> -Djava.library.path=/home/gs/gridre/shelob/share/hadoop/lib/native/Linux-amd64-64
> -Dyarn.policy.file=hadoop-policy.xml -Dyarn.log.dir=/home/gs/var/log/mapredqa
> -Dyarn.log.file=yarn-mapredqa-historyserver-<host>.log -Dyarn.home.dir= -Dyarn.id.str=mapredqa
> -Dyarn.root.logger=INFO,DRFA -Djava.library.path=/home/gs/gridre/shelob/share/hadoop/lib/native/Linux-amd64-64
> -Dyarn.policy.file=hadoop-policy.xml -Dmapred.jobsummary.logger=INFO,JSA -Dhadoop.log.dir=/home/gs/var/log/mapredqa
> -Dyarn.log.dir=/home/gs/var/log/mapredqa
> -Dhadoop.log.file=yarn-mapredqa-historyserver-<host>.log
> -Dyarn.log.file=yarn-mapredqa-historyserver-<host>.log
> -Dyarn.home.dir=/home/gs/gridre/shelob/share/hadoop -Dhadoop.root.logger=INFO,DRFA -Dyarn.root.logger=INFO,DRFA
> -Djava.library.path=/home/gs/gridre/shelob/share/hadoop/lib/native/Linux-amd64-64 -classpath
> /home/gs/gridre/shelob/conf/hadoop:/home/gs/gridre/shelob/conf/hadoop:/home/gs/gridre/shelob/conf/hadoop:/home/gs/gridre/shelob/conf/hadoop:/home/gs/gridre/shelob/share/hadoop/share/hadoop/common/lib/*:/home/gs/gridre/shelob/share/hadoop/share/hadoop/common/*:/home/gs/gridre/shelob/share/hadoop/hadoop-*-capacity-scheduler.jar:/home/gs/gridre/shelob/share/hadoop/hadoop-*-capacity-scheduler.jar:/home/gs/gridre/shelob/share/hadoop/hadoop-*-capacity-scheduler.jar:/home/gs/gridre/shelob/share/hadoop/share/hadoop/hdfs:/home/gs/gridre/shelob/share/hadoop/share/hadoop/hdfs/lib/*:/home/gs/gridre/shelob/share/hadoop/share/hadoop/hdfs/*:/home/gs/gridre/shelob/share/hadoop/share/hadoop/mapreduce/lib/*:/home/gs/gridre/shelob/share/hadoop/share/hadoop/mapreduce/*:/home/gs/java/jdk64/current/lib/tools.jar:/home/gs/gridre/shelob/share/hadoop/share/hadoop/mapreduce/*:/home/gs/gridre/shelob/share/hadoop/share/hadoop/mapreduce/lib/*
> org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer
> 1) On the RM with older hadoop version where the job summary log does not exist
> jobhistory ps shows using the option:
> -Dmapred.jobsummary.logger=INFO,console
> 2) On the RM with older hadoop version where the job summary log exists
> jobhistory ps shows using the option:
> -Dmapred.jobsummary.logger=INFO,JSA
> -Dmapred.jobsummary.logger=INFO,JSA
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3843) Job summary log file found
missing on the RM host
Posted by "Anupam Seth (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Anupam Seth updated MAPREDUCE-3843:
-----------------------------------
Attachment: MAPREDUCE-3843.patch
Fixing doc stuff
> Job summary log file found missing on the RM host
> -------------------------------------------------
>
> Key: MAPREDUCE-3843
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3843
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobhistoryserver, mrv2
> Affects Versions: 0.23.0
> Reporter: Anupam Seth
> Assignee: Anupam Seth
> Priority: Critical
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3843.patch, MAPREDUCE-3843.patch, MAPREDUCE-3843.patch
>
>
> This bug was found by Phil Su as part of our testing.
> After MAPREDUCE-3354 went in, the Job summary log file seems to have gone missing on the
> RM host.
> The job summary log appears to be interspersed in yarn-mapredqa-historyserver-<host>.out.
> e.g.
> 12/02/09 15:57:21 INFO jobhistory.JobSummary:
> jobId=job_1328658619341_0011,submitTime=1328802904381,launchTime=1328802909977,firstMapTaskLaunchTime=1328802912116,firstReduceTaskLaunchTime=1328802915074,finishTime=1328802933797,resourc
> esPerMap=1024,resourcesPerReduce=2048,numMaps=10,numReduces=10,user=hadoopqa,queue=default,status=KILLED,mapSlotSeconds=0,reduceSlotSeconds=0
> 1) On the RM with older hadoop version where the job summary log does not exist
> mapredqa 10903 0.0 1.2 1424404 210240 ? Sl Feb07 0:19 /home/gs/java/jdk64/current/bin/java -Xmx1000m
> -Djava.net.preferIPv4Stack=true
> -Djava.library.path=/home/gs/gridre/theoden/share/hadoop/lib/native/Linux-amd64-64:/
> home/gs/gridre/theoden/share/hadoop/lib/native/Linux-amd64-64 -Dhadoop.log.dir=/home/gs/var/log/mapredqa
> -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/home/gs/gridre/theoden/share/hadoop -Dhadoop.id.str=mapredqa
> -Dhadoop
> .root.logger=INFO,console
> -Djava.library.path=/home/gs/gridre/theoden/share/hadoop/lib/native/Linux-amd64-64:/home/gs/gridre/theoden/share/hadoop/lib/native/Linux-amd64-64:/home/gs/gridre/theoden/share/hadoop/lib/nat
> ive -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -Dmapred.jobsummary.logger=INFO,console
> -Dhadoop.security.logger=INFO,NullAppender org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer
> 2) On the RM with older hadoop version where the job summary log exists
> mapredqa 24851 0.0 0.5 1463280 90516 ? Sl Jan25 0:37 /home/gs/java/jdk64/current/bin/java
> -Dproc_historyserver -Xmx1000m -Dmapred.jobsummary.logger=INFO,JSA -Dyarn.log.dir=/home/gs/var/log/mapredqa
> -Dyarn.log.file=yarn.log -Dyarn.home.dir= -Dyarn.id.str= -Dyarn.root.logger=INFO,console
> -Djava.library.path=/home/gs/gridre/shelob/share/hadoop/lib/native/Linux-amd64-64
> -Dyarn.policy.file=hadoop-policy.xml -Dyarn.log.dir=/home/gs/var/log/mapredqa
> -Dyarn.log.file=yarn-mapredqa-historyserver-<host>.log -Dyarn.home.dir= -Dyarn.id.str=mapredqa
> -Dyarn.root.logger=INFO,DRFA -Djava.library.path=/home/gs/gridre/shelob/share/hadoop/lib/native/Linux-amd64-64
> -Dyarn.policy.file=hadoop-policy.xml -Dmapred.jobsummary.logger=INFO,JSA -Dhadoop.log.dir=/home/gs/var/log/mapredqa
> -Dyarn.log.dir=/home/gs/var/log/mapredqa
> -Dhadoop.log.file=yarn-mapredqa-historyserver-<host>.log
> -Dyarn.log.file=yarn-mapredqa-historyserver-<host>.log
> -Dyarn.home.dir=/home/gs/gridre/shelob/share/hadoop -Dhadoop.root.logger=INFO,DRFA -Dyarn.root.logger=INFO,DRFA
> -Djava.library.path=/home/gs/gridre/shelob/share/hadoop/lib/native/Linux-amd64-64 -classpath
> /home/gs/gridre/shelob/conf/hadoop:/home/gs/gridre/shelob/conf/hadoop:/home/gs/gridre/shelob/conf/hadoop:/home/gs/gridre/shelob/conf/hadoop:/home/gs/gridre/shelob/share/hadoop/share/hadoop/common/lib/*:/home/gs/gridre/shelob/share/hadoop/share/hadoop/common/*:/home/gs/gridre/shelob/share/hadoop/hadoop-*-capacity-scheduler.jar:/home/gs/gridre/shelob/share/hadoop/hadoop-*-capacity-scheduler.jar:/home/gs/gridre/shelob/share/hadoop/hadoop-*-capacity-scheduler.jar:/home/gs/gridre/shelob/share/hadoop/share/hadoop/hdfs:/home/gs/gridre/shelob/share/hadoop/share/hadoop/hdfs/lib/*:/home/gs/gridre/shelob/share/hadoop/share/hadoop/hdfs/*:/home/gs/gridre/shelob/share/hadoop/share/hadoop/mapreduce/lib/*:/home/gs/gridre/shelob/share/hadoop/share/hadoop/mapreduce/*:/home/gs/java/jdk64/current/lib/tools.jar:/home/gs/gridre/shelob/share/hadoop/share/hadoop/mapreduce/*:/home/gs/gridre/shelob/share/hadoop/share/hadoop/mapreduce/lib/*
> org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer
> 1) On the RM with older hadoop version where the job summary log does not exist
> jobhistory ps shows using the option:
> -Dmapred.jobsummary.logger=INFO,console
> 2) On the RM with older hadoop version where the job summary log exists
> jobhistory ps shows using the option:
> -Dmapred.jobsummary.logger=INFO,JSA
> -Dmapred.jobsummary.logger=INFO,JSA
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3843) Job summary log file found
missing on the RM host
Posted by "Thomas Graves (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13205748#comment-13205748 ]
Thomas Graves commented on MAPREDUCE-3843:
------------------------------------------
Anupam, if you don't mind could you change the rest of the commands in the ClusterSetup to use hadoop-daemon.sh and yarn-daemon.sh appropriately?
I tested starting and stopping the job history server. Everything worked and the log files (both out and log) and job summary file (mapred-jobsummary.log) were all created properly using the log4j.properties config below. it was started with the command: $HADOOP_COMMON_HOME/sbin/mr-jobhistory-daemon.sh --config $HADOOP_CONF_DIR/ start historyserver.
I think we need to do some cleanup of the mr-jobhistory-daemon.sh and other scripts since right now its intermixing HADOOP and YARN variables. I think the log4j.properties template conf file in the tree needs to also be updated. But I think that can be done in follow up jiras.
hadoop.mapreduce.jobsummary.logger=${hadoop.root.logger}
hadoop.mapreduce.jobsummary.log.file=hadoop-mapreduce.jobsummary.log
mapred.jobsummary.logger=INFO,console
log4j.logger.org.apache.hadoop.mapreduce.jobhistory.JobSummary=${mapred.jobsummary.logger}
log4j.additivity.org.apache.hadoop.mapreduce.jobhistory.JobSummary=false
log4j.appender.JSA=org.apache.log4j.DailyRollingFileAppender
log4j.appender.JSA.File=${hadoop.log.dir}/mapred-jobsummary.log
log4j.appender.JSA.layout=org.apache.log4j.PatternLayout
log4j.appender.JSA.layout.ConversionPattern=%d{ISO8601} %p %c{2}: %m%n
log4j.appender.JSA.DatePattern=.yyyy-MM-dd
> Job summary log file found missing on the RM host
> -------------------------------------------------
>
> Key: MAPREDUCE-3843
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3843
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobhistoryserver, mrv2
> Affects Versions: 0.23.0
> Reporter: Anupam Seth
> Assignee: Anupam Seth
> Priority: Critical
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3843.patch, MAPREDUCE-3843.patch, MAPREDUCE-3843.patch
>
>
> This bug was found by Phil Su as part of our testing.
> After MAPREDUCE-3354 went in, the Job summary log file seems to have gone missing on the
> RM host.
> The job summary log appears to be interspersed in yarn-mapredqa-historyserver-<host>.out.
> e.g.
> 12/02/09 15:57:21 INFO jobhistory.JobSummary:
> jobId=job_1328658619341_0011,submitTime=1328802904381,launchTime=1328802909977,firstMapTaskLaunchTime=1328802912116,firstReduceTaskLaunchTime=1328802915074,finishTime=1328802933797,resourc
> esPerMap=1024,resourcesPerReduce=2048,numMaps=10,numReduces=10,user=hadoopqa,queue=default,status=KILLED,mapSlotSeconds=0,reduceSlotSeconds=0
> 1) On the RM with older hadoop version where the job summary log does not exist
> mapredqa 10903 0.0 1.2 1424404 210240 ? Sl Feb07 0:19 /home/gs/java/jdk64/current/bin/java -Xmx1000m
> -Djava.net.preferIPv4Stack=true
> -Djava.library.path=/home/gs/gridre/theoden/share/hadoop/lib/native/Linux-amd64-64:/
> home/gs/gridre/theoden/share/hadoop/lib/native/Linux-amd64-64 -Dhadoop.log.dir=/home/gs/var/log/mapredqa
> -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/home/gs/gridre/theoden/share/hadoop -Dhadoop.id.str=mapredqa
> -Dhadoop
> .root.logger=INFO,console
> -Djava.library.path=/home/gs/gridre/theoden/share/hadoop/lib/native/Linux-amd64-64:/home/gs/gridre/theoden/share/hadoop/lib/native/Linux-amd64-64:/home/gs/gridre/theoden/share/hadoop/lib/nat
> ive -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -Dmapred.jobsummary.logger=INFO,console
> -Dhadoop.security.logger=INFO,NullAppender org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer
> 2) On the RM with older hadoop version where the job summary log exists
> mapredqa 24851 0.0 0.5 1463280 90516 ? Sl Jan25 0:37 /home/gs/java/jdk64/current/bin/java
> -Dproc_historyserver -Xmx1000m -Dmapred.jobsummary.logger=INFO,JSA -Dyarn.log.dir=/home/gs/var/log/mapredqa
> -Dyarn.log.file=yarn.log -Dyarn.home.dir= -Dyarn.id.str= -Dyarn.root.logger=INFO,console
> -Djava.library.path=/home/gs/gridre/shelob/share/hadoop/lib/native/Linux-amd64-64
> -Dyarn.policy.file=hadoop-policy.xml -Dyarn.log.dir=/home/gs/var/log/mapredqa
> -Dyarn.log.file=yarn-mapredqa-historyserver-<host>.log -Dyarn.home.dir= -Dyarn.id.str=mapredqa
> -Dyarn.root.logger=INFO,DRFA -Djava.library.path=/home/gs/gridre/shelob/share/hadoop/lib/native/Linux-amd64-64
> -Dyarn.policy.file=hadoop-policy.xml -Dmapred.jobsummary.logger=INFO,JSA -Dhadoop.log.dir=/home/gs/var/log/mapredqa
> -Dyarn.log.dir=/home/gs/var/log/mapredqa
> -Dhadoop.log.file=yarn-mapredqa-historyserver-<host>.log
> -Dyarn.log.file=yarn-mapredqa-historyserver-<host>.log
> -Dyarn.home.dir=/home/gs/gridre/shelob/share/hadoop -Dhadoop.root.logger=INFO,DRFA -Dyarn.root.logger=INFO,DRFA
> -Djava.library.path=/home/gs/gridre/shelob/share/hadoop/lib/native/Linux-amd64-64 -classpath
> /home/gs/gridre/shelob/conf/hadoop:/home/gs/gridre/shelob/conf/hadoop:/home/gs/gridre/shelob/conf/hadoop:/home/gs/gridre/shelob/conf/hadoop:/home/gs/gridre/shelob/share/hadoop/share/hadoop/common/lib/*:/home/gs/gridre/shelob/share/hadoop/share/hadoop/common/*:/home/gs/gridre/shelob/share/hadoop/hadoop-*-capacity-scheduler.jar:/home/gs/gridre/shelob/share/hadoop/hadoop-*-capacity-scheduler.jar:/home/gs/gridre/shelob/share/hadoop/hadoop-*-capacity-scheduler.jar:/home/gs/gridre/shelob/share/hadoop/share/hadoop/hdfs:/home/gs/gridre/shelob/share/hadoop/share/hadoop/hdfs/lib/*:/home/gs/gridre/shelob/share/hadoop/share/hadoop/hdfs/*:/home/gs/gridre/shelob/share/hadoop/share/hadoop/mapreduce/lib/*:/home/gs/gridre/shelob/share/hadoop/share/hadoop/mapreduce/*:/home/gs/java/jdk64/current/lib/tools.jar:/home/gs/gridre/shelob/share/hadoop/share/hadoop/mapreduce/*:/home/gs/gridre/shelob/share/hadoop/share/hadoop/mapreduce/lib/*
> org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer
> 1) On the RM with older hadoop version where the job summary log does not exist
> jobhistory ps shows using the option:
> -Dmapred.jobsummary.logger=INFO,console
> 2) On the RM with older hadoop version where the job summary log exists
> jobhistory ps shows using the option:
> -Dmapred.jobsummary.logger=INFO,JSA
> -Dmapred.jobsummary.logger=INFO,JSA
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3843) Job summary log file found
missing on the RM host
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13205905#comment-13205905 ]
Hudson commented on MAPREDUCE-3843:
-----------------------------------
Integrated in Hadoop-Hdfs-0.23-Commit #526 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/526/])
merge -r 1242975:1242976 from trunk. FIXES: MAPREDUCE-3843
tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1242977
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/bin/mr-jobhistory-daemon.sh
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/ClusterSetup.apt.vm
> Job summary log file found missing on the RM host
> -------------------------------------------------
>
> Key: MAPREDUCE-3843
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3843
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobhistoryserver, mrv2
> Affects Versions: 0.23.0
> Reporter: Anupam Seth
> Assignee: Anupam Seth
> Priority: Critical
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3843.patch, MAPREDUCE-3843.patch, MAPREDUCE-3843.patch, MAPREDUCE-3843.patch, MAPREDUCE-3843.patch, MAPREDUCE-3843.patch
>
>
> This bug was found by Phil Su as part of our testing.
> After MAPREDUCE-3354 went in, the Job summary log file seems to have gone missing on the
> RM host.
> The job summary log appears to be interspersed in yarn-mapredqa-historyserver-<host>.out.
> e.g.
> 12/02/09 15:57:21 INFO jobhistory.JobSummary:
> jobId=job_1328658619341_0011,submitTime=1328802904381,launchTime=1328802909977,firstMapTaskLaunchTime=1328802912116,firstReduceTaskLaunchTime=1328802915074,finishTime=1328802933797,resourc
> esPerMap=1024,resourcesPerReduce=2048,numMaps=10,numReduces=10,user=hadoopqa,queue=default,status=KILLED,mapSlotSeconds=0,reduceSlotSeconds=0
> 1) On the RM with older hadoop version where the job summary log does not exist
> mapredqa 10903 0.0 1.2 1424404 210240 ? Sl Feb07 0:19 /home/gs/java/jdk64/current/bin/java -Xmx1000m
> -Djava.net.preferIPv4Stack=true
> -Djava.library.path=/home/gs/gridre/theoden/share/hadoop/lib/native/Linux-amd64-64:/
> home/gs/gridre/theoden/share/hadoop/lib/native/Linux-amd64-64 -Dhadoop.log.dir=/home/gs/var/log/mapredqa
> -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/home/gs/gridre/theoden/share/hadoop -Dhadoop.id.str=mapredqa
> -Dhadoop
> .root.logger=INFO,console
> -Djava.library.path=/home/gs/gridre/theoden/share/hadoop/lib/native/Linux-amd64-64:/home/gs/gridre/theoden/share/hadoop/lib/native/Linux-amd64-64:/home/gs/gridre/theoden/share/hadoop/lib/nat
> ive -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -Dmapred.jobsummary.logger=INFO,console
> -Dhadoop.security.logger=INFO,NullAppender org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer
> 2) On the RM with older hadoop version where the job summary log exists
> mapredqa 24851 0.0 0.5 1463280 90516 ? Sl Jan25 0:37 /home/gs/java/jdk64/current/bin/java
> -Dproc_historyserver -Xmx1000m -Dmapred.jobsummary.logger=INFO,JSA -Dyarn.log.dir=/home/gs/var/log/mapredqa
> -Dyarn.log.file=yarn.log -Dyarn.home.dir= -Dyarn.id.str= -Dyarn.root.logger=INFO,console
> -Djava.library.path=/home/gs/gridre/shelob/share/hadoop/lib/native/Linux-amd64-64
> -Dyarn.policy.file=hadoop-policy.xml -Dyarn.log.dir=/home/gs/var/log/mapredqa
> -Dyarn.log.file=yarn-mapredqa-historyserver-<host>.log -Dyarn.home.dir= -Dyarn.id.str=mapredqa
> -Dyarn.root.logger=INFO,DRFA -Djava.library.path=/home/gs/gridre/shelob/share/hadoop/lib/native/Linux-amd64-64
> -Dyarn.policy.file=hadoop-policy.xml -Dmapred.jobsummary.logger=INFO,JSA -Dhadoop.log.dir=/home/gs/var/log/mapredqa
> -Dyarn.log.dir=/home/gs/var/log/mapredqa
> -Dhadoop.log.file=yarn-mapredqa-historyserver-<host>.log
> -Dyarn.log.file=yarn-mapredqa-historyserver-<host>.log
> -Dyarn.home.dir=/home/gs/gridre/shelob/share/hadoop -Dhadoop.root.logger=INFO,DRFA -Dyarn.root.logger=INFO,DRFA
> -Djava.library.path=/home/gs/gridre/shelob/share/hadoop/lib/native/Linux-amd64-64 -classpath
> /home/gs/gridre/shelob/conf/hadoop:/home/gs/gridre/shelob/conf/hadoop:/home/gs/gridre/shelob/conf/hadoop:/home/gs/gridre/shelob/conf/hadoop:/home/gs/gridre/shelob/share/hadoop/share/hadoop/common/lib/*:/home/gs/gridre/shelob/share/hadoop/share/hadoop/common/*:/home/gs/gridre/shelob/share/hadoop/hadoop-*-capacity-scheduler.jar:/home/gs/gridre/shelob/share/hadoop/hadoop-*-capacity-scheduler.jar:/home/gs/gridre/shelob/share/hadoop/hadoop-*-capacity-scheduler.jar:/home/gs/gridre/shelob/share/hadoop/share/hadoop/hdfs:/home/gs/gridre/shelob/share/hadoop/share/hadoop/hdfs/lib/*:/home/gs/gridre/shelob/share/hadoop/share/hadoop/hdfs/*:/home/gs/gridre/shelob/share/hadoop/share/hadoop/mapreduce/lib/*:/home/gs/gridre/shelob/share/hadoop/share/hadoop/mapreduce/*:/home/gs/java/jdk64/current/lib/tools.jar:/home/gs/gridre/shelob/share/hadoop/share/hadoop/mapreduce/*:/home/gs/gridre/shelob/share/hadoop/share/hadoop/mapreduce/lib/*
> org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer
> 1) On the RM with older hadoop version where the job summary log does not exist
> jobhistory ps shows using the option:
> -Dmapred.jobsummary.logger=INFO,console
> 2) On the RM with older hadoop version where the job summary log exists
> jobhistory ps shows using the option:
> -Dmapred.jobsummary.logger=INFO,JSA
> -Dmapred.jobsummary.logger=INFO,JSA
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3843) Job summary log file found
missing on the RM host
Posted by "Anupam Seth (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Anupam Seth updated MAPREDUCE-3843:
-----------------------------------
Attachment: MAPREDUCE-3843.patch
Thanks Tom for your feedback. I have incorporated your suggestion to change the rest of the start-up / stop cmds in the docs also.
> Job summary log file found missing on the RM host
> -------------------------------------------------
>
> Key: MAPREDUCE-3843
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3843
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobhistoryserver, mrv2
> Affects Versions: 0.23.0
> Reporter: Anupam Seth
> Assignee: Anupam Seth
> Priority: Critical
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3843.patch, MAPREDUCE-3843.patch, MAPREDUCE-3843.patch, MAPREDUCE-3843.patch
>
>
> This bug was found by Phil Su as part of our testing.
> After MAPREDUCE-3354 went in, the Job summary log file seems to have gone missing on the
> RM host.
> The job summary log appears to be interspersed in yarn-mapredqa-historyserver-<host>.out.
> e.g.
> 12/02/09 15:57:21 INFO jobhistory.JobSummary:
> jobId=job_1328658619341_0011,submitTime=1328802904381,launchTime=1328802909977,firstMapTaskLaunchTime=1328802912116,firstReduceTaskLaunchTime=1328802915074,finishTime=1328802933797,resourc
> esPerMap=1024,resourcesPerReduce=2048,numMaps=10,numReduces=10,user=hadoopqa,queue=default,status=KILLED,mapSlotSeconds=0,reduceSlotSeconds=0
> 1) On the RM with older hadoop version where the job summary log does not exist
> mapredqa 10903 0.0 1.2 1424404 210240 ? Sl Feb07 0:19 /home/gs/java/jdk64/current/bin/java -Xmx1000m
> -Djava.net.preferIPv4Stack=true
> -Djava.library.path=/home/gs/gridre/theoden/share/hadoop/lib/native/Linux-amd64-64:/
> home/gs/gridre/theoden/share/hadoop/lib/native/Linux-amd64-64 -Dhadoop.log.dir=/home/gs/var/log/mapredqa
> -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/home/gs/gridre/theoden/share/hadoop -Dhadoop.id.str=mapredqa
> -Dhadoop
> .root.logger=INFO,console
> -Djava.library.path=/home/gs/gridre/theoden/share/hadoop/lib/native/Linux-amd64-64:/home/gs/gridre/theoden/share/hadoop/lib/native/Linux-amd64-64:/home/gs/gridre/theoden/share/hadoop/lib/nat
> ive -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -Dmapred.jobsummary.logger=INFO,console
> -Dhadoop.security.logger=INFO,NullAppender org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer
> 2) On the RM with older hadoop version where the job summary log exists
> mapredqa 24851 0.0 0.5 1463280 90516 ? Sl Jan25 0:37 /home/gs/java/jdk64/current/bin/java
> -Dproc_historyserver -Xmx1000m -Dmapred.jobsummary.logger=INFO,JSA -Dyarn.log.dir=/home/gs/var/log/mapredqa
> -Dyarn.log.file=yarn.log -Dyarn.home.dir= -Dyarn.id.str= -Dyarn.root.logger=INFO,console
> -Djava.library.path=/home/gs/gridre/shelob/share/hadoop/lib/native/Linux-amd64-64
> -Dyarn.policy.file=hadoop-policy.xml -Dyarn.log.dir=/home/gs/var/log/mapredqa
> -Dyarn.log.file=yarn-mapredqa-historyserver-<host>.log -Dyarn.home.dir= -Dyarn.id.str=mapredqa
> -Dyarn.root.logger=INFO,DRFA -Djava.library.path=/home/gs/gridre/shelob/share/hadoop/lib/native/Linux-amd64-64
> -Dyarn.policy.file=hadoop-policy.xml -Dmapred.jobsummary.logger=INFO,JSA -Dhadoop.log.dir=/home/gs/var/log/mapredqa
> -Dyarn.log.dir=/home/gs/var/log/mapredqa
> -Dhadoop.log.file=yarn-mapredqa-historyserver-<host>.log
> -Dyarn.log.file=yarn-mapredqa-historyserver-<host>.log
> -Dyarn.home.dir=/home/gs/gridre/shelob/share/hadoop -Dhadoop.root.logger=INFO,DRFA -Dyarn.root.logger=INFO,DRFA
> -Djava.library.path=/home/gs/gridre/shelob/share/hadoop/lib/native/Linux-amd64-64 -classpath
> /home/gs/gridre/shelob/conf/hadoop:/home/gs/gridre/shelob/conf/hadoop:/home/gs/gridre/shelob/conf/hadoop:/home/gs/gridre/shelob/conf/hadoop:/home/gs/gridre/shelob/share/hadoop/share/hadoop/common/lib/*:/home/gs/gridre/shelob/share/hadoop/share/hadoop/common/*:/home/gs/gridre/shelob/share/hadoop/hadoop-*-capacity-scheduler.jar:/home/gs/gridre/shelob/share/hadoop/hadoop-*-capacity-scheduler.jar:/home/gs/gridre/shelob/share/hadoop/hadoop-*-capacity-scheduler.jar:/home/gs/gridre/shelob/share/hadoop/share/hadoop/hdfs:/home/gs/gridre/shelob/share/hadoop/share/hadoop/hdfs/lib/*:/home/gs/gridre/shelob/share/hadoop/share/hadoop/hdfs/*:/home/gs/gridre/shelob/share/hadoop/share/hadoop/mapreduce/lib/*:/home/gs/gridre/shelob/share/hadoop/share/hadoop/mapreduce/*:/home/gs/java/jdk64/current/lib/tools.jar:/home/gs/gridre/shelob/share/hadoop/share/hadoop/mapreduce/*:/home/gs/gridre/shelob/share/hadoop/share/hadoop/mapreduce/lib/*
> org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer
> 1) On the RM with older hadoop version where the job summary log does not exist
> jobhistory ps shows using the option:
> -Dmapred.jobsummary.logger=INFO,console
> 2) On the RM with older hadoop version where the job summary log exists
> jobhistory ps shows using the option:
> -Dmapred.jobsummary.logger=INFO,JSA
> -Dmapred.jobsummary.logger=INFO,JSA
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3843) Job summary log file found
missing on the RM host
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13205880#comment-13205880 ]
Hudson commented on MAPREDUCE-3843:
-----------------------------------
Integrated in Hadoop-Mapreduce-trunk-Commit #1723 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1723/])
MAPREDUCE-3843. Job summary log file found missing on the RM host (Anupam Seth via tgraves)
tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1242976
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/bin/mr-jobhistory-daemon.sh
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/ClusterSetup.apt.vm
> Job summary log file found missing on the RM host
> -------------------------------------------------
>
> Key: MAPREDUCE-3843
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3843
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobhistoryserver, mrv2
> Affects Versions: 0.23.0
> Reporter: Anupam Seth
> Assignee: Anupam Seth
> Priority: Critical
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3843.patch, MAPREDUCE-3843.patch, MAPREDUCE-3843.patch, MAPREDUCE-3843.patch, MAPREDUCE-3843.patch, MAPREDUCE-3843.patch
>
>
> This bug was found by Phil Su as part of our testing.
> After MAPREDUCE-3354 went in, the Job summary log file seems to have gone missing on the
> RM host.
> The job summary log appears to be interspersed in yarn-mapredqa-historyserver-<host>.out.
> e.g.
> 12/02/09 15:57:21 INFO jobhistory.JobSummary:
> jobId=job_1328658619341_0011,submitTime=1328802904381,launchTime=1328802909977,firstMapTaskLaunchTime=1328802912116,firstReduceTaskLaunchTime=1328802915074,finishTime=1328802933797,resourc
> esPerMap=1024,resourcesPerReduce=2048,numMaps=10,numReduces=10,user=hadoopqa,queue=default,status=KILLED,mapSlotSeconds=0,reduceSlotSeconds=0
> 1) On the RM with older hadoop version where the job summary log does not exist
> mapredqa 10903 0.0 1.2 1424404 210240 ? Sl Feb07 0:19 /home/gs/java/jdk64/current/bin/java -Xmx1000m
> -Djava.net.preferIPv4Stack=true
> -Djava.library.path=/home/gs/gridre/theoden/share/hadoop/lib/native/Linux-amd64-64:/
> home/gs/gridre/theoden/share/hadoop/lib/native/Linux-amd64-64 -Dhadoop.log.dir=/home/gs/var/log/mapredqa
> -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/home/gs/gridre/theoden/share/hadoop -Dhadoop.id.str=mapredqa
> -Dhadoop
> .root.logger=INFO,console
> -Djava.library.path=/home/gs/gridre/theoden/share/hadoop/lib/native/Linux-amd64-64:/home/gs/gridre/theoden/share/hadoop/lib/native/Linux-amd64-64:/home/gs/gridre/theoden/share/hadoop/lib/nat
> ive -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -Dmapred.jobsummary.logger=INFO,console
> -Dhadoop.security.logger=INFO,NullAppender org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer
> 2) On the RM with older hadoop version where the job summary log exists
> mapredqa 24851 0.0 0.5 1463280 90516 ? Sl Jan25 0:37 /home/gs/java/jdk64/current/bin/java
> -Dproc_historyserver -Xmx1000m -Dmapred.jobsummary.logger=INFO,JSA -Dyarn.log.dir=/home/gs/var/log/mapredqa
> -Dyarn.log.file=yarn.log -Dyarn.home.dir= -Dyarn.id.str= -Dyarn.root.logger=INFO,console
> -Djava.library.path=/home/gs/gridre/shelob/share/hadoop/lib/native/Linux-amd64-64
> -Dyarn.policy.file=hadoop-policy.xml -Dyarn.log.dir=/home/gs/var/log/mapredqa
> -Dyarn.log.file=yarn-mapredqa-historyserver-<host>.log -Dyarn.home.dir= -Dyarn.id.str=mapredqa
> -Dyarn.root.logger=INFO,DRFA -Djava.library.path=/home/gs/gridre/shelob/share/hadoop/lib/native/Linux-amd64-64
> -Dyarn.policy.file=hadoop-policy.xml -Dmapred.jobsummary.logger=INFO,JSA -Dhadoop.log.dir=/home/gs/var/log/mapredqa
> -Dyarn.log.dir=/home/gs/var/log/mapredqa
> -Dhadoop.log.file=yarn-mapredqa-historyserver-<host>.log
> -Dyarn.log.file=yarn-mapredqa-historyserver-<host>.log
> -Dyarn.home.dir=/home/gs/gridre/shelob/share/hadoop -Dhadoop.root.logger=INFO,DRFA -Dyarn.root.logger=INFO,DRFA
> -Djava.library.path=/home/gs/gridre/shelob/share/hadoop/lib/native/Linux-amd64-64 -classpath
> /home/gs/gridre/shelob/conf/hadoop:/home/gs/gridre/shelob/conf/hadoop:/home/gs/gridre/shelob/conf/hadoop:/home/gs/gridre/shelob/conf/hadoop:/home/gs/gridre/shelob/share/hadoop/share/hadoop/common/lib/*:/home/gs/gridre/shelob/share/hadoop/share/hadoop/common/*:/home/gs/gridre/shelob/share/hadoop/hadoop-*-capacity-scheduler.jar:/home/gs/gridre/shelob/share/hadoop/hadoop-*-capacity-scheduler.jar:/home/gs/gridre/shelob/share/hadoop/hadoop-*-capacity-scheduler.jar:/home/gs/gridre/shelob/share/hadoop/share/hadoop/hdfs:/home/gs/gridre/shelob/share/hadoop/share/hadoop/hdfs/lib/*:/home/gs/gridre/shelob/share/hadoop/share/hadoop/hdfs/*:/home/gs/gridre/shelob/share/hadoop/share/hadoop/mapreduce/lib/*:/home/gs/gridre/shelob/share/hadoop/share/hadoop/mapreduce/*:/home/gs/java/jdk64/current/lib/tools.jar:/home/gs/gridre/shelob/share/hadoop/share/hadoop/mapreduce/*:/home/gs/gridre/shelob/share/hadoop/share/hadoop/mapreduce/lib/*
> org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer
> 1) On the RM with older hadoop version where the job summary log does not exist
> jobhistory ps shows using the option:
> -Dmapred.jobsummary.logger=INFO,console
> 2) On the RM with older hadoop version where the job summary log exists
> jobhistory ps shows using the option:
> -Dmapred.jobsummary.logger=INFO,JSA
> -Dmapred.jobsummary.logger=INFO,JSA
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3843) Job summary log file found
missing on the RM host
Posted by "Anupam Seth (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Anupam Seth updated MAPREDUCE-3843:
-----------------------------------
Status: Patch Available (was: Open)
Unit tests not applicable as simple script change. Have manually tested on a single node cluster.
> Job summary log file found missing on the RM host
> -------------------------------------------------
>
> Key: MAPREDUCE-3843
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3843
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobhistoryserver, mrv2
> Affects Versions: 0.23.0
> Reporter: Anupam Seth
> Assignee: Anupam Seth
> Priority: Critical
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3843.patch
>
>
> This bug was found by Phil Su as part of our testing.
> After MAPREDUCE-3354 went in, the Job summary log file seems to have gone missing on the
> RM host.
> The job summary log appears to be interspersed in yarn-mapredqa-historyserver-<host>.out.
> e.g.
> 12/02/09 15:57:21 INFO jobhistory.JobSummary:
> jobId=job_1328658619341_0011,submitTime=1328802904381,launchTime=1328802909977,firstMapTaskLaunchTime=1328802912116,firstReduceTaskLaunchTime=1328802915074,finishTime=1328802933797,resourc
> esPerMap=1024,resourcesPerReduce=2048,numMaps=10,numReduces=10,user=hadoopqa,queue=default,status=KILLED,mapSlotSeconds=0,reduceSlotSeconds=0
> 1) On the RM with older hadoop version where the job summary log does not exist
> mapredqa 10903 0.0 1.2 1424404 210240 ? Sl Feb07 0:19 /home/gs/java/jdk64/current/bin/java -Xmx1000m
> -Djava.net.preferIPv4Stack=true
> -Djava.library.path=/home/gs/gridre/theoden/share/hadoop/lib/native/Linux-amd64-64:/
> home/gs/gridre/theoden/share/hadoop/lib/native/Linux-amd64-64 -Dhadoop.log.dir=/home/gs/var/log/mapredqa
> -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/home/gs/gridre/theoden/share/hadoop -Dhadoop.id.str=mapredqa
> -Dhadoop
> .root.logger=INFO,console
> -Djava.library.path=/home/gs/gridre/theoden/share/hadoop/lib/native/Linux-amd64-64:/home/gs/gridre/theoden/share/hadoop/lib/native/Linux-amd64-64:/home/gs/gridre/theoden/share/hadoop/lib/nat
> ive -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -Dmapred.jobsummary.logger=INFO,console
> -Dhadoop.security.logger=INFO,NullAppender org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer
> 2) On the RM with older hadoop version where the job summary log exists
> mapredqa 24851 0.0 0.5 1463280 90516 ? Sl Jan25 0:37 /home/gs/java/jdk64/current/bin/java
> -Dproc_historyserver -Xmx1000m -Dmapred.jobsummary.logger=INFO,JSA -Dyarn.log.dir=/home/gs/var/log/mapredqa
> -Dyarn.log.file=yarn.log -Dyarn.home.dir= -Dyarn.id.str= -Dyarn.root.logger=INFO,console
> -Djava.library.path=/home/gs/gridre/shelob/share/hadoop/lib/native/Linux-amd64-64
> -Dyarn.policy.file=hadoop-policy.xml -Dyarn.log.dir=/home/gs/var/log/mapredqa
> -Dyarn.log.file=yarn-mapredqa-historyserver-<host>.log -Dyarn.home.dir= -Dyarn.id.str=mapredqa
> -Dyarn.root.logger=INFO,DRFA -Djava.library.path=/home/gs/gridre/shelob/share/hadoop/lib/native/Linux-amd64-64
> -Dyarn.policy.file=hadoop-policy.xml -Dmapred.jobsummary.logger=INFO,JSA -Dhadoop.log.dir=/home/gs/var/log/mapredqa
> -Dyarn.log.dir=/home/gs/var/log/mapredqa
> -Dhadoop.log.file=yarn-mapredqa-historyserver-<host>.log
> -Dyarn.log.file=yarn-mapredqa-historyserver-<host>.log
> -Dyarn.home.dir=/home/gs/gridre/shelob/share/hadoop -Dhadoop.root.logger=INFO,DRFA -Dyarn.root.logger=INFO,DRFA
> -Djava.library.path=/home/gs/gridre/shelob/share/hadoop/lib/native/Linux-amd64-64 -classpath
> /home/gs/gridre/shelob/conf/hadoop:/home/gs/gridre/shelob/conf/hadoop:/home/gs/gridre/shelob/conf/hadoop:/home/gs/gridre/shelob/conf/hadoop:/home/gs/gridre/shelob/share/hadoop/share/hadoop/common/lib/*:/home/gs/gridre/shelob/share/hadoop/share/hadoop/common/*:/home/gs/gridre/shelob/share/hadoop/hadoop-*-capacity-scheduler.jar:/home/gs/gridre/shelob/share/hadoop/hadoop-*-capacity-scheduler.jar:/home/gs/gridre/shelob/share/hadoop/hadoop-*-capacity-scheduler.jar:/home/gs/gridre/shelob/share/hadoop/share/hadoop/hdfs:/home/gs/gridre/shelob/share/hadoop/share/hadoop/hdfs/lib/*:/home/gs/gridre/shelob/share/hadoop/share/hadoop/hdfs/*:/home/gs/gridre/shelob/share/hadoop/share/hadoop/mapreduce/lib/*:/home/gs/gridre/shelob/share/hadoop/share/hadoop/mapreduce/*:/home/gs/java/jdk64/current/lib/tools.jar:/home/gs/gridre/shelob/share/hadoop/share/hadoop/mapreduce/*:/home/gs/gridre/shelob/share/hadoop/share/hadoop/mapreduce/lib/*
> org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer
> 1) On the RM with older hadoop version where the job summary log does not exist
> jobhistory ps shows using the option:
> -Dmapred.jobsummary.logger=INFO,console
> 2) On the RM with older hadoop version where the job summary log exists
> jobhistory ps shows using the option:
> -Dmapred.jobsummary.logger=INFO,JSA
> -Dmapred.jobsummary.logger=INFO,JSA
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3843) Job summary log file found
missing on the RM host
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13206100#comment-13206100 ]
Hudson commented on MAPREDUCE-3843:
-----------------------------------
Integrated in Hadoop-Hdfs-0.23-Build #166 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/166/])
merge -r 1242975:1242976 from trunk. FIXES: MAPREDUCE-3843
tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1242977
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/bin/mr-jobhistory-daemon.sh
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/ClusterSetup.apt.vm
> Job summary log file found missing on the RM host
> -------------------------------------------------
>
> Key: MAPREDUCE-3843
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3843
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobhistoryserver, mrv2
> Affects Versions: 0.23.0
> Reporter: Anupam Seth
> Assignee: Anupam Seth
> Priority: Critical
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3843.patch, MAPREDUCE-3843.patch, MAPREDUCE-3843.patch, MAPREDUCE-3843.patch, MAPREDUCE-3843.patch, MAPREDUCE-3843.patch
>
>
> This bug was found by Phil Su as part of our testing.
> After MAPREDUCE-3354 went in, the Job summary log file seems to have gone missing on the
> RM host.
> The job summary log appears to be interspersed in yarn-mapredqa-historyserver-<host>.out.
> e.g.
> 12/02/09 15:57:21 INFO jobhistory.JobSummary:
> jobId=job_1328658619341_0011,submitTime=1328802904381,launchTime=1328802909977,firstMapTaskLaunchTime=1328802912116,firstReduceTaskLaunchTime=1328802915074,finishTime=1328802933797,resourc
> esPerMap=1024,resourcesPerReduce=2048,numMaps=10,numReduces=10,user=hadoopqa,queue=default,status=KILLED,mapSlotSeconds=0,reduceSlotSeconds=0
> 1) On the RM with older hadoop version where the job summary log does not exist
> mapredqa 10903 0.0 1.2 1424404 210240 ? Sl Feb07 0:19 /home/gs/java/jdk64/current/bin/java -Xmx1000m
> -Djava.net.preferIPv4Stack=true
> -Djava.library.path=/home/gs/gridre/theoden/share/hadoop/lib/native/Linux-amd64-64:/
> home/gs/gridre/theoden/share/hadoop/lib/native/Linux-amd64-64 -Dhadoop.log.dir=/home/gs/var/log/mapredqa
> -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/home/gs/gridre/theoden/share/hadoop -Dhadoop.id.str=mapredqa
> -Dhadoop
> .root.logger=INFO,console
> -Djava.library.path=/home/gs/gridre/theoden/share/hadoop/lib/native/Linux-amd64-64:/home/gs/gridre/theoden/share/hadoop/lib/native/Linux-amd64-64:/home/gs/gridre/theoden/share/hadoop/lib/nat
> ive -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -Dmapred.jobsummary.logger=INFO,console
> -Dhadoop.security.logger=INFO,NullAppender org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer
> 2) On the RM with older hadoop version where the job summary log exists
> mapredqa 24851 0.0 0.5 1463280 90516 ? Sl Jan25 0:37 /home/gs/java/jdk64/current/bin/java
> -Dproc_historyserver -Xmx1000m -Dmapred.jobsummary.logger=INFO,JSA -Dyarn.log.dir=/home/gs/var/log/mapredqa
> -Dyarn.log.file=yarn.log -Dyarn.home.dir= -Dyarn.id.str= -Dyarn.root.logger=INFO,console
> -Djava.library.path=/home/gs/gridre/shelob/share/hadoop/lib/native/Linux-amd64-64
> -Dyarn.policy.file=hadoop-policy.xml -Dyarn.log.dir=/home/gs/var/log/mapredqa
> -Dyarn.log.file=yarn-mapredqa-historyserver-<host>.log -Dyarn.home.dir= -Dyarn.id.str=mapredqa
> -Dyarn.root.logger=INFO,DRFA -Djava.library.path=/home/gs/gridre/shelob/share/hadoop/lib/native/Linux-amd64-64
> -Dyarn.policy.file=hadoop-policy.xml -Dmapred.jobsummary.logger=INFO,JSA -Dhadoop.log.dir=/home/gs/var/log/mapredqa
> -Dyarn.log.dir=/home/gs/var/log/mapredqa
> -Dhadoop.log.file=yarn-mapredqa-historyserver-<host>.log
> -Dyarn.log.file=yarn-mapredqa-historyserver-<host>.log
> -Dyarn.home.dir=/home/gs/gridre/shelob/share/hadoop -Dhadoop.root.logger=INFO,DRFA -Dyarn.root.logger=INFO,DRFA
> -Djava.library.path=/home/gs/gridre/shelob/share/hadoop/lib/native/Linux-amd64-64 -classpath
> /home/gs/gridre/shelob/conf/hadoop:/home/gs/gridre/shelob/conf/hadoop:/home/gs/gridre/shelob/conf/hadoop:/home/gs/gridre/shelob/conf/hadoop:/home/gs/gridre/shelob/share/hadoop/share/hadoop/common/lib/*:/home/gs/gridre/shelob/share/hadoop/share/hadoop/common/*:/home/gs/gridre/shelob/share/hadoop/hadoop-*-capacity-scheduler.jar:/home/gs/gridre/shelob/share/hadoop/hadoop-*-capacity-scheduler.jar:/home/gs/gridre/shelob/share/hadoop/hadoop-*-capacity-scheduler.jar:/home/gs/gridre/shelob/share/hadoop/share/hadoop/hdfs:/home/gs/gridre/shelob/share/hadoop/share/hadoop/hdfs/lib/*:/home/gs/gridre/shelob/share/hadoop/share/hadoop/hdfs/*:/home/gs/gridre/shelob/share/hadoop/share/hadoop/mapreduce/lib/*:/home/gs/gridre/shelob/share/hadoop/share/hadoop/mapreduce/*:/home/gs/java/jdk64/current/lib/tools.jar:/home/gs/gridre/shelob/share/hadoop/share/hadoop/mapreduce/*:/home/gs/gridre/shelob/share/hadoop/share/hadoop/mapreduce/lib/*
> org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer
> 1) On the RM with older hadoop version where the job summary log does not exist
> jobhistory ps shows using the option:
> -Dmapred.jobsummary.logger=INFO,console
> 2) On the RM with older hadoop version where the job summary log exists
> jobhistory ps shows using the option:
> -Dmapred.jobsummary.logger=INFO,JSA
> -Dmapred.jobsummary.logger=INFO,JSA
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira