You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Siddharth Shah <ia...@gmail.com> on 2006/08/16 18:02:03 UTC
Reporting Glitches....
Hi,
I am attaching a Hadoop job, there are some reporting glitches by
which Map phase reaches 100% completition it again goes back to to
54%. As suggested by one of my friend i used setStatus() method in
Reporter, but still I am experiencing the same problem.
Please look at where I am wrong and help me get out of this problem.
Thanks in advace,
Sidd.
RE: Hadoop 0.5.0 Issues
Posted by Jagadeesh <ja...@gmail.com>.
Thanks Benjamin, I will check and will revert.
--Jugs
-----Original Message-----
From: Benjamin Reed [mailto:breed@yahoo-inc.com]
Sent: Thursday, August 17, 2006 8:06 PM
To: hadoop-user@lucene.apache.org
Subject: Re: Hadoop 0.5.0 Issues
There was a protocol change in 0.5.0 are you sure you are using the 0.5.0
libraries with your sample app?
ben
On Thursday 17 August 2006 05:55, Jagadeesh wrote:
> Hi All,
>
> I have been using Hadoop 0.4.0 for the past 1 month with 22 slaves and 1
> master node. So far there are no issues except for the log which I had
> already mentioned and I rectified it by configuring log4j. I am very
> impressed with this piece of software.
>
> However when I installed 0.5.0 on a test machine, I noticed some weird
> behaviors. I could start the server in stand-alone mode and the app works
> if I use the DFS Client to upload and download files. But it just hangs
> when I tried connecting from another box using a sample Java app I wrote.
> The sample app works in 0.4.0 without any issues. I thought there is
> something wrong with the way I have configured Hadoop and tried with
> different configurations. Then I understood that, downloading a file using
> sample app works in 0.5.0, but it's taking ages for downloading a file of
> size 1 KB and most of the time that too doesn't work.
>
> So I strongly feel the configurations are right as I was able to download
> the file even though it took some time. I verified log files and
everything
> seems to be fine except for the below exception.
>
> 2006-08-16 12:35:07,214 WARN org.apache.hadoop.dfs.DataNode: DataXCeiver
> java.net.SocketException: Connection reset
> at java.net.SocketInputStream.read(SocketInputStream.java:168)
> at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:256)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:313)
> at java.io.DataInputStream.read(DataInputStream.java:134)
> at
> org.apache.hadoop.dfs.DataNode$DataXceiver.writeBlock(DataNode.java:720)
> at
> org.apache.hadoop.dfs.DataNode$DataXceiver.run(DataNode.java:507) at
> java.lang.Thread.run(Thread.java:595)
>
>
> I will appreciate any help or pointers to resolve this. I really want to
> use 0.5.0 but this error is holding me back.
>
> Thanks
> Jugs
Re: Hadoop 0.5.0 Issues
Posted by Benjamin Reed <br...@yahoo-inc.com>.
There was a protocol change in 0.5.0 are you sure you are using the 0.5.0
libraries with your sample app?
ben
On Thursday 17 August 2006 05:55, Jagadeesh wrote:
> Hi All,
>
> I have been using Hadoop 0.4.0 for the past 1 month with 22 slaves and 1
> master node. So far there are no issues except for the log which I had
> already mentioned and I rectified it by configuring log4j. I am very
> impressed with this piece of software.
>
> However when I installed 0.5.0 on a test machine, I noticed some weird
> behaviors. I could start the server in stand-alone mode and the app works
> if I use the DFS Client to upload and download files. But it just hangs
> when I tried connecting from another box using a sample Java app I wrote.
> The sample app works in 0.4.0 without any issues. I thought there is
> something wrong with the way I have configured Hadoop and tried with
> different configurations. Then I understood that, downloading a file using
> sample app works in 0.5.0, but it's taking ages for downloading a file of
> size 1 KB and most of the time that too doesn't work.
>
> So I strongly feel the configurations are right as I was able to download
> the file even though it took some time. I verified log files and everything
> seems to be fine except for the below exception.
>
> 2006-08-16 12:35:07,214 WARN org.apache.hadoop.dfs.DataNode: DataXCeiver
> java.net.SocketException: Connection reset
> at java.net.SocketInputStream.read(SocketInputStream.java:168)
> at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:256)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:313)
> at java.io.DataInputStream.read(DataInputStream.java:134)
> at
> org.apache.hadoop.dfs.DataNode$DataXceiver.writeBlock(DataNode.java:720)
> at
> org.apache.hadoop.dfs.DataNode$DataXceiver.run(DataNode.java:507) at
> java.lang.Thread.run(Thread.java:595)
>
>
> I will appreciate any help or pointers to resolve this. I really want to
> use 0.5.0 but this error is holding me back.
>
> Thanks
> Jugs
Hadoop 0.5.0 Issues
Posted by Jagadeesh <ja...@gmail.com>.
Hi All,
I have been using Hadoop 0.4.0 for the past 1 month with 22 slaves and 1
master node. So far there are no issues except for the log which I had
already mentioned and I rectified it by configuring log4j. I am very
impressed with this piece of software.
However when I installed 0.5.0 on a test machine, I noticed some weird
behaviors. I could start the server in stand-alone mode and the app works if
I use the DFS Client to upload and download files. But it just hangs when I
tried connecting from another box using a sample Java app I wrote. The
sample app works in 0.4.0 without any issues. I thought there is something
wrong with the way I have configured Hadoop and tried with different
configurations. Then I understood that, downloading a file using sample app
works in 0.5.0, but it's taking ages for downloading a file of size 1 KB and
most of the time that too doesn't work.
So I strongly feel the configurations are right as I was able to download
the file even though it took some time. I verified log files and everything
seems to be fine except for the below exception.
2006-08-16 12:35:07,214 WARN org.apache.hadoop.dfs.DataNode: DataXCeiver
java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(SocketInputStream.java:168)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:256)
at java.io.BufferedInputStream.read(BufferedInputStream.java:313)
at java.io.DataInputStream.read(DataInputStream.java:134)
at
org.apache.hadoop.dfs.DataNode$DataXceiver.writeBlock(DataNode.java:720)
at org.apache.hadoop.dfs.DataNode$DataXceiver.run(DataNode.java:507)
at java.lang.Thread.run(Thread.java:595)
I will appreciate any help or pointers to resolve this. I really want to use
0.5.0 but this error is holding me back.
Thanks
Jugs
Re: Reporting Glitches....
Posted by "Bryan A. Pendleton" <bp...@geekdom.net>.
There doesn't seem to be an attachment to the message....
Common things to know:
1) The "percent complete" is a rounded number. You'll start seeing 100% when
99.5% is done.
2) Even after you're really 100% done, map tasks can still get lost.
3) Reporting of job status is a sum of the parts. If all tasks are running,
and each thinks its 90% done, then Maps will show 90% done. However, if any
of those tasks changes it mind (some strange logic in the input format,
perhaps), you'll see backtracking. If any of those tasks *fails*, it's
portion of percent completion will revert to 0% immediately. I.e., if you
have 10 tasks, and each thinks its 90% done, and one fails, you'll revert to
81% progress.
On 8/16/06, Siddharth Shah <ia...@gmail.com> wrote:
>
> Hi,
> I am attaching a Hadoop job, there are some reporting glitches by
> which Map phase reaches 100% completition it again goes back to to
> 54%. As suggested by one of my friend i used setStatus() method in
> Reporter, but still I am experiencing the same problem.
>
> Please look at where I am wrong and help me get out of this problem.
>
>
> Thanks in advace,
>
> Sidd.
>
>
--
Bryan A. P. Pendleton
Ph: (877) geek-1-bp