You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-user@hadoop.apache.org by arun k <ar...@gmail.com> on 2011/12/16 07:52:04 UTC
Generating job and topology traces from history folder of multinode
cluster using Rumen
Hai guys !
I have set up 5 node cluster with each of them in different racks.
I have hadoop-0.20.2 set up on my Eclipse Helios. So, i ran Tracebuilder
using
Main Class: org.apache.hadoop.tools.rumen.TraceBuilder
I ran some jobs on cluster and used copy of /usr/local/hadoop/logs/history
folder of *master *node as input to TraceBuilder.
Arguments: /home/arun/job.json /home/arun/topology.json
/home/ubuntu/Documents/testlog
But i get
11/12/16 12:02:37 WARN util.NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable
11/12/16 12:02:38 WARN rumen.TraceBuilder: TraceBuilder got an error while
processing the [possibly virtual] file
master_1324011575958_job_201112161029_0001_hduser_word+count within Path
file:/home/ubuntu/Documents/testlog/master_1324011575958_job_201112161029_0001_hduser_word+count
java.lang.NullPointerException
at
org.apache.hadoop.tools.rumen.JobBuilder.processTaskAttemptFinishedEvent(JobBuilder.java:492)
at
org.apache.hadoop.tools.rumen.JobBuilder.process(JobBuilder.java:149)
at
org.apache.hadoop.tools.rumen.TraceBuilder.processJobHistory(TraceBuilder.java:310)
at
org.apache.hadoop.tools.rumen.TraceBuilder.run(TraceBuilder.java:264)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:83)
at
org.apache.hadoop.tools.rumen.TraceBuilder.main(TraceBuilder.java:142)
11/12/16 12:02:39 WARN rumen.TraceBuilder: TraceBuilder got an error while
processing the [possibly virtual] file
master_1324011575958_job_201112161029_0004_hduser_word+count within Path
file:/home/ubuntu/Documents/testlog/master_1324011575958_job_201112161029_0004_hduser_word+count
java.lang.NullPointerException
at
org.apache.hadoop.tools.rumen.JobBuilder.processTaskAttemptFinishedEvent(JobBuilder.java:492)
at org.apache.hadoop.tools.rumen.JobBuilder.process(JobBuilde
.........
....................
It generates job trace json file but the topology trace json file doesn't
have 5 node's info and is like this :
{
"name" : "<root>",
"children" : [ ]
}
> Which history folder do i need to use ? masters or slaves ? how do i use
it ?
> I have a single node cluster from which i got logs and gave input
TraceBuilder and it works.
> How do i run tracebuilder from terminal and do i need to use only from
version 21 ?
Can anyone help me out ?
Thanks,
Arun
Re: Generating job and topology traces from history folder of
multinode cluster using Rumen
Posted by arun k <ar...@gmail.com>.
Ravi,
Thanks for the info.
Arun
On Fri, Dec 16, 2011 at 12:27 PM, Ravi Gummadi <gr...@yahoo-inc.com> wrote:
> Amar is working on this issue MAPREDUCE-3349. The patch is not comiited to
> trunk yet. Feel free to try it out while it gets reviewed and committed.
>
> -Ravi
> ________________________________________
> From: arun k [arunk786@gmail.com]
> Sent: Friday, December 16, 2011 12:22 PM
> To: mapreduce-user@hadoop.apache.org
> Subject: Generating job and topology traces from history folder of
> multinode cluster using Rumen
>
> Hai guys !
>
> I have set up 5 node cluster with each of them in different racks.
>
> I have hadoop-0.20.2 set up on my Eclipse Helios. So, i ran Tracebuilder
> using
> Main Class: org.apache.hadoop.tools.rumen.TraceBuilder
> I ran some jobs on cluster and used copy of
> /usr/local/hadoop/logs/history folder of master node as input to
> TraceBuilder.
> Arguments: /home/arun/job.json /home/arun/topology.json
> /home/ubuntu/Documents/testlog
>
> But i get
> 11/12/16 12:02:37 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> 11/12/16 12:02:38 WARN rumen.TraceBuilder: TraceBuilder got an error while
> processing the [possibly virtual] file
> master_1324011575958_job_201112161029_0001_hduser_word+count within Path
> file:/home/ubuntu/Documents/testlog/master_1324011575958_job_201112161029_0001_hduser_word+count
> java.lang.NullPointerException
> at
> org.apache.hadoop.tools.rumen.JobBuilder.processTaskAttemptFinishedEvent(JobBuilder.java:492)
> at
> org.apache.hadoop.tools.rumen.JobBuilder.process(JobBuilder.java:149)
> at
> org.apache.hadoop.tools.rumen.TraceBuilder.processJobHistory(TraceBuilder.java:310)
> at
> org.apache.hadoop.tools.rumen.TraceBuilder.run(TraceBuilder.java:264)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:83)
> at
> org.apache.hadoop.tools.rumen.TraceBuilder.main(TraceBuilder.java:142)
> 11/12/16 12:02:39 WARN rumen.TraceBuilder: TraceBuilder got an error while
> processing the [possibly virtual] file
> master_1324011575958_job_201112161029_0004_hduser_word+count within Path
> file:/home/ubuntu/Documents/testlog/master_1324011575958_job_201112161029_0004_hduser_word+count
> java.lang.NullPointerException
> at
> org.apache.hadoop.tools.rumen.JobBuilder.processTaskAttemptFinishedEvent(JobBuilder.java:492)
> at org.apache.hadoop.tools.rumen.JobBuilder.process(JobBuilde
> .........
> ....................
>
> It generates job trace json file but the topology trace json file doesn't
> have 5 node's info and is like this :
>
> {
> "name" : "<root>",
> "children" : [ ]
> }
>
> > Which history folder do i need to use ? masters or slaves ? how do i
> use it ?
> > I have a single node cluster from which i got logs and gave input
> TraceBuilder and it works.
> > How do i run tracebuilder from terminal and do i need to use only from
> version 21 ?
>
> Can anyone help me out ?
>
>
> Thanks,
> Arun
>
RE: Generating job and topology traces from history folder of
multinode cluster using Rumen
Posted by Ravi Gummadi <gr...@yahoo-inc.com>.
Amar is working on this issue MAPREDUCE-3349. The patch is not comiited to trunk yet. Feel free to try it out while it gets reviewed and committed.
-Ravi
________________________________________
From: arun k [arunk786@gmail.com]
Sent: Friday, December 16, 2011 12:22 PM
To: mapreduce-user@hadoop.apache.org
Subject: Generating job and topology traces from history folder of multinode cluster using Rumen
Hai guys !
I have set up 5 node cluster with each of them in different racks.
I have hadoop-0.20.2 set up on my Eclipse Helios. So, i ran Tracebuilder using
Main Class: org.apache.hadoop.tools.rumen.TraceBuilder
I ran some jobs on cluster and used copy of /usr/local/hadoop/logs/history folder of master node as input to TraceBuilder.
Arguments: /home/arun/job.json /home/arun/topology.json /home/ubuntu/Documents/testlog
But i get
11/12/16 12:02:37 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
11/12/16 12:02:38 WARN rumen.TraceBuilder: TraceBuilder got an error while processing the [possibly virtual] file master_1324011575958_job_201112161029_0001_hduser_word+count within Path file:/home/ubuntu/Documents/testlog/master_1324011575958_job_201112161029_0001_hduser_word+count
java.lang.NullPointerException
at org.apache.hadoop.tools.rumen.JobBuilder.processTaskAttemptFinishedEvent(JobBuilder.java:492)
at org.apache.hadoop.tools.rumen.JobBuilder.process(JobBuilder.java:149)
at org.apache.hadoop.tools.rumen.TraceBuilder.processJobHistory(TraceBuilder.java:310)
at org.apache.hadoop.tools.rumen.TraceBuilder.run(TraceBuilder.java:264)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:83)
at org.apache.hadoop.tools.rumen.TraceBuilder.main(TraceBuilder.java:142)
11/12/16 12:02:39 WARN rumen.TraceBuilder: TraceBuilder got an error while processing the [possibly virtual] file master_1324011575958_job_201112161029_0004_hduser_word+count within Path file:/home/ubuntu/Documents/testlog/master_1324011575958_job_201112161029_0004_hduser_word+count
java.lang.NullPointerException
at org.apache.hadoop.tools.rumen.JobBuilder.processTaskAttemptFinishedEvent(JobBuilder.java:492)
at org.apache.hadoop.tools.rumen.JobBuilder.process(JobBuilde .........
....................
It generates job trace json file but the topology trace json file doesn't have 5 node's info and is like this :
{
"name" : "<root>",
"children" : [ ]
}
> Which history folder do i need to use ? masters or slaves ? how do i use it ?
> I have a single node cluster from which i got logs and gave input TraceBuilder and it works.
> How do i run tracebuilder from terminal and do i need to use only from version 21 ?
Can anyone help me out ?
Thanks,
Arun