You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-user@hadoop.apache.org by arun k <ar...@gmail.com> on 2011/12/08 07:33:12 UTC
Checksum error during trace generation using Rumen
Hi guys !
I was trying to generate job trace and topology trace.
I have hadoop set up for hduser at /usr/local/hadoop and ran wordcount
program as hduser .
I have mapreduce component set up in eclipse for user "arun".
I set for a configuration :
Class: org.apache.hadoop.tools.rumen.TraceBuilder
Args : /home/arun/Documents/jh.json /home/arun/Documents/top.json
/usr/local/hadoop/logs/history
When i run it i get the following error even when i have given :
${HADOOP_HOME} chmod -R 777 /logs
11/12/08 11:55:17 WARN util.NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable
11/12/08 11:55:18 WARN fs.FSInputChecker: Problem opening checksum file:
file:/usr/local/hadoop/logs/history/job_201109191122_0002_1316411924369_hduser_word+count.
Ignoring exception: java.io.EOFException
at java.io.DataInputStream.readFully(DataInputStream.java:180)
at java.io.DataInputStream.readFully(DataInputStream.java:152)
at
org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.<init>(ChecksumFileSystem.java:145)
at org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:311)
at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:534)
at
org.apache.hadoop.tools.rumen.PossiblyDecompressedInputStream.<init>(PossiblyDecompressedInputStream.java:42)
at
org.apache.hadoop.tools.rumen.DefaultInputDemuxer.bindTo(DefaultInputDemuxer.java:42)
at org.apache.hadoop.tools.rumen.TraceBuilder.run(TraceBuilder.java:225)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:83)
at org.apache.hadoop.tools.rumen.TraceBuilder.main(TraceBuilder.java:142)
Any help ?
Arun
Re: Checksum error during trace generation using Rumen
Posted by arun k <ar...@gmail.com>.
Amar,
Job trace and topology trace are generated but the TraceBuilder program
terminates after displaying
11/12/08 11:55:17 WARN util.NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable
It doesn't indicate whether it was a success and doesn't give any other
info. Is it Ok ?
I have used the generated trace with Mumak and it works fine.
Arun
On Thu, Dec 8, 2011 at 7:43 PM, arun k <ar...@gmail.com> wrote:
> Amar,
>
> I didn't modfiy Jobhistory file but i deleted hidden checksumfiels and it
> worked.
> I have a 3 node cluster set up.
> For generating a single job i am running say wordcount. Is there any way
> where i can generate good number of jobs(say 50,100,..) at a time ?
> How can i generate jobs trace for 50, 100, so on by keeping two nodes
> under one rack and the other in second rack in my cluster ?
> HOw do i control the placement of nodes in different rack and generate job
> and topology trace ?
>
>
> Arun
>
>
> On Thu, Dec 8, 2011 at 12:11 PM, Amar Kamat <am...@yahoo-inc.com> wrote:
>
>> Arun,
>> Did you modify the job history file manually? Looks like HDFS is not able
>> to match the job history file contents to its checksum. Try deleting the
>> checksum file. Note that the checksum file is a hidden file.
>> Amar
>>
>>
>>
>> On 12/8/11 12:03 PM, "arun k" <ar...@gmail.com> wrote:
>>
>> Hi guys !
>>
>> I was trying to generate job trace and topology trace.
>> I have hadoop set up for hduser at /usr/local/hadoop and ran wordcount
>> program as hduser .
>> I have mapreduce component set up in eclipse for user "arun".
>> I set for a configuration :
>> Class: org.apache.hadoop.tools.rumen.TraceBuilder
>> Args : /home/arun/Documents/jh.json /home/arun/Documents/top.json
>> /usr/local/hadoop/logs/history
>>
>> When i run it i get the following error even when i have given :
>> ${HADOOP_HOME} chmod -R 777 /logs
>>
>> 11/12/08 11:55:17 WARN util.NativeCodeLoader: Unable to load
>> native-hadoop library for your platform... using builtin-java classes where
>> applicable
>> 11/12/08 11:55:18 WARN fs.FSInputChecker: Problem opening checksum file:
>> file:/usr/local/hadoop/logs/history/job_201109191122_0002_1316411924369_hduser_word+count.
>> Ignoring exception: java.io.EOFException
>> at java.io.DataInputStream.readFully(DataInputStream.java:180)
>> at java.io.DataInputStream.readFully(DataInputStream.java:152)
>> at
>> org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.<init>(ChecksumFileSystem.java:145)
>> at
>> org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:311)
>> at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:534)
>> at
>> org.apache.hadoop.tools.rumen.PossiblyDecompressedInputStream.<init>(PossiblyDecompressedInputStream.java:42)
>> at
>> org.apache.hadoop.tools.rumen.DefaultInputDemuxer.bindTo(DefaultInputDemuxer.java:42)
>> at org.apache.hadoop.tools.rumen.TraceBuilder.run(TraceBuilder.java:225)
>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69)
>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:83)
>> at org.apache.hadoop.tools.rumen.TraceBuilder.main(TraceBuilder.java:142)
>>
>> Any help ?
>>
>> Arun
>>
>>
>>
>>
>
Re: Checksum error during trace generation using Rumen
Posted by arun k <ar...@gmail.com>.
Amar,
I didn't modfiy Jobhistory file but i deleted hidden checksumfiels and it
worked.
I have a 3 node cluster set up.
For generating a single job i am running say wordcount. Is there any way
where i can generate good number of jobs(say 50,100,..) at a time ?
How can i generate jobs trace for 50, 100, so on by keeping two nodes under
one rack and the other in second rack in my cluster ?
HOw do i control the placement of nodes in different rack and generate job
and topology trace ?
Arun
On Thu, Dec 8, 2011 at 12:11 PM, Amar Kamat <am...@yahoo-inc.com> wrote:
> Arun,
> Did you modify the job history file manually? Looks like HDFS is not able
> to match the job history file contents to its checksum. Try deleting the
> checksum file. Note that the checksum file is a hidden file.
> Amar
>
>
>
> On 12/8/11 12:03 PM, "arun k" <ar...@gmail.com> wrote:
>
> Hi guys !
>
> I was trying to generate job trace and topology trace.
> I have hadoop set up for hduser at /usr/local/hadoop and ran wordcount
> program as hduser .
> I have mapreduce component set up in eclipse for user "arun".
> I set for a configuration :
> Class: org.apache.hadoop.tools.rumen.TraceBuilder
> Args : /home/arun/Documents/jh.json /home/arun/Documents/top.json
> /usr/local/hadoop/logs/history
>
> When i run it i get the following error even when i have given :
> ${HADOOP_HOME} chmod -R 777 /logs
>
> 11/12/08 11:55:17 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> 11/12/08 11:55:18 WARN fs.FSInputChecker: Problem opening checksum file:
> file:/usr/local/hadoop/logs/history/job_201109191122_0002_1316411924369_hduser_word+count.
> Ignoring exception: java.io.EOFException
> at java.io.DataInputStream.readFully(DataInputStream.java:180)
> at java.io.DataInputStream.readFully(DataInputStream.java:152)
> at
> org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.<init>(ChecksumFileSystem.java:145)
> at
> org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:311)
> at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:534)
> at
> org.apache.hadoop.tools.rumen.PossiblyDecompressedInputStream.<init>(PossiblyDecompressedInputStream.java:42)
> at
> org.apache.hadoop.tools.rumen.DefaultInputDemuxer.bindTo(DefaultInputDemuxer.java:42)
> at org.apache.hadoop.tools.rumen.TraceBuilder.run(TraceBuilder.java:225)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:83)
> at org.apache.hadoop.tools.rumen.TraceBuilder.main(TraceBuilder.java:142)
>
> Any help ?
>
> Arun
>
>
>
>
Re: Checksum error during trace generation using Rumen
Posted by Amar Kamat <am...@yahoo-inc.com>.
Arun,
Did you modify the job history file manually? Looks like HDFS is not able to match the job history file contents to its checksum. Try deleting the checksum file. Note that the checksum file is a hidden file.
Amar
On 12/8/11 12:03 PM, "arun k" <ar...@gmail.com> wrote:
Hi guys !
I was trying to generate job trace and topology trace.
I have hadoop set up for hduser at /usr/local/hadoop and ran wordcount program as hduser .
I have mapreduce component set up in eclipse for user "arun".
I set for a configuration :
Class: org.apache.hadoop.tools.rumen.TraceBuilder
Args : /home/arun/Documents/jh.json /home/arun/Documents/top.json /usr/local/hadoop/logs/history
When i run it i get the following error even when i have given : ${HADOOP_HOME} chmod -R 777 /logs
11/12/08 11:55:17 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
11/12/08 11:55:18 WARN fs.FSInputChecker: Problem opening checksum file: file:/usr/local/hadoop/logs/history/job_201109191122_0002_1316411924369_hduser_word+count. Ignoring exception: java.io.EOFException
at java.io.DataInputStream.readFully(DataInputStream.java:180)
at java.io.DataInputStream.readFully(DataInputStream.java:152)
at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.<init>(ChecksumFileSystem.java:145)
at org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:311)
at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:534)
at org.apache.hadoop.tools.rumen.PossiblyDecompressedInputStream.<init>(PossiblyDecompressedInputStream.java:42)
at org.apache.hadoop.tools.rumen.DefaultInputDemuxer.bindTo(DefaultInputDemuxer.java:42)
at org.apache.hadoop.tools.rumen.TraceBuilder.run(TraceBuilder.java:225)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:83)
at org.apache.hadoop.tools.rumen.TraceBuilder.main(TraceBuilder.java:142)
Any help ?
Arun