You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by "Mahajan, Neeraj" <ne...@ebay.com> on 2007/06/15 02:50:45 UTC
Getting EOF exceptions while executing wordcount example
Hi,
I am trying to setup hadoop on two machines running solaris 10. After
fixing the scripts in bin to conform to bourne shell scripting standard,
I was able to start the jobtracker and tasktrackers. I am not using
HDFS. All the paths are common on all the machines as they are available
via NFS. I have made sure that the tmp directories are mapped to
seperate locations for both the machines. When I execute the wordcount
example, I get following errors
2007-06-14 16:51:57,605 WARN org.apache.hadoop.mapred.TaskTracker:
getMapOutput(task_0001_m_000018_0,5) failed :
java.io.EOFException
at java.io.DataInputStream.readFully(DataInputStream.java:178)
at java.io.DataInputStream.readLong(DataInputStream.java:380)
at
org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.
java:1911)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:689)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
at
org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:427)
at
org.mortbay.jetty.servlet.WebApplicationHandler.dispatch(WebApplicationH
andler.java:475)
at
org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:567)
at org.mortbay.http.HttpContext.handle(HttpContext.java:1565)
at
org.mortbay.jetty.servlet.WebApplicationContext.handle(WebApplicationCon
text.java:635)
at org.mortbay.http.HttpContext.handle(HttpContext.java:1517)
at org.mortbay.http.HttpServer.service(HttpServer.java:954)
at
org.mortbay.http.HttpConnection.service(HttpConnection.java:814)
at
org.mortbay.http.HttpConnection.handleNext(HttpConnection.java:981)
at
org.mortbay.http.HttpConnection.handle(HttpConnection.java:831)
at
org.mortbay.http.SocketListener.handleConnection(SocketListener.java:244
)
at
org.mortbay.util.ThreadedServer.handle(ThreadedServer.java:357)
at
org.mortbay.util.ThreadPool$PoolThread.run(ThreadPool.java:534)
These errors come when map task status is 100% and reduce is 76%. After
this the the map % drops to 95 and again both increase. This happens a
few times and then the job completes.
I have increased the number of tasks per node as well as the tasks per
job. Any idea what the problem is?
Please let me know if you need more information.
Thanks,
Neeraj
RE: Getting EOF exceptions while executing wordcount example
Posted by "Mahajan, Neeraj" <ne...@ebay.com>.
Got it!
My mistake. Had put "mapred.reduce.tasks" in hadoop-site.xml with
different values. I moved this parameter to mapred-default.xml and the
problem vanished.
~ Neeraj
-----Original Message-----
From: Mahajan, Neeraj [mailto:nemahajan@ebay.com]
Sent: Friday, June 15, 2007 11:58 AM
To: hadoop-user@lucene.apache.org
Subject: RE: Getting EOF exceptions while executing wordcount example
Guys help needed.. Please reply.
I started debugging this and found out that
TaskTracker.MapOutputServlet.doGet() is called with a reduce value of 6
and then the index file is tried to be readt at offset 6 * 16 = 96. But
I can see that the size of the index file is just 80 bytes. I am not
sure why this is happening. Can someone tell me who is issues a HTTP GET
request to MapOutputServlet so that I can debug that part and see what
is happening??
~ Neeraj
-----Original Message-----
From: Mahajan, Neeraj [mailto:nemahajan@ebay.com]
Sent: Thursday, June 14, 2007 5:51 PM
To: hadoop-user@lucene.apache.org
Subject: Getting EOF exceptions while executing wordcount example
Hi,
I am trying to setup hadoop on two machines running solaris 10. After
fixing the scripts in bin to conform to bourne shell scripting standard,
I was able to start the jobtracker and tasktrackers. I am not using
HDFS. All the paths are common on all the machines as they are available
via NFS. I have made sure that the tmp directories are mapped to
seperate locations for both the machines. When I execute the wordcount
example, I get following errors
2007-06-14 16:51:57,605 WARN org.apache.hadoop.mapred.TaskTracker:
getMapOutput(task_0001_m_000018_0,5) failed :
java.io.EOFException
at java.io.DataInputStream.readFully(DataInputStream.java:178)
at java.io.DataInputStream.readLong(DataInputStream.java:380)
at
org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.
java:1911)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:689)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
at
org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:427)
at
org.mortbay.jetty.servlet.WebApplicationHandler.dispatch(WebApplicationH
andler.java:475)
at
org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:567)
at org.mortbay.http.HttpContext.handle(HttpContext.java:1565)
at
org.mortbay.jetty.servlet.WebApplicationContext.handle(WebApplicationCon
text.java:635)
at org.mortbay.http.HttpContext.handle(HttpContext.java:1517)
at org.mortbay.http.HttpServer.service(HttpServer.java:954)
at
org.mortbay.http.HttpConnection.service(HttpConnection.java:814)
at
org.mortbay.http.HttpConnection.handleNext(HttpConnection.java:981)
at
org.mortbay.http.HttpConnection.handle(HttpConnection.java:831)
at
org.mortbay.http.SocketListener.handleConnection(SocketListener.java:244
)
at
org.mortbay.util.ThreadedServer.handle(ThreadedServer.java:357)
at
org.mortbay.util.ThreadPool$PoolThread.run(ThreadPool.java:534)
These errors come when map task status is 100% and reduce is 76%. After
this the the map % drops to 95 and again both increase. This happens a
few times and then the job completes.
I have increased the number of tasks per node as well as the tasks per
job. Any idea what the problem is?
Please let me know if you need more information.
Thanks,
Neeraj
RE: Getting EOF exceptions while executing wordcount example
Posted by "Mahajan, Neeraj" <ne...@ebay.com>.
Guys help needed.. Please reply.
I started debugging this and found out that
TaskTracker.MapOutputServlet.doGet() is called with a reduce value of 6
and then the index file is tried to be readt at offset 6 * 16 = 96. But
I can see that the size of the index file is just 80 bytes. I am not
sure why this is happening. Can someone tell me who is issues a HTTP GET
request to MapOutputServlet so that I can debug that part and see what
is happening??
~ Neeraj
-----Original Message-----
From: Mahajan, Neeraj [mailto:nemahajan@ebay.com]
Sent: Thursday, June 14, 2007 5:51 PM
To: hadoop-user@lucene.apache.org
Subject: Getting EOF exceptions while executing wordcount example
Hi,
I am trying to setup hadoop on two machines running solaris 10. After
fixing the scripts in bin to conform to bourne shell scripting standard,
I was able to start the jobtracker and tasktrackers. I am not using
HDFS. All the paths are common on all the machines as they are available
via NFS. I have made sure that the tmp directories are mapped to
seperate locations for both the machines. When I execute the wordcount
example, I get following errors
2007-06-14 16:51:57,605 WARN org.apache.hadoop.mapred.TaskTracker:
getMapOutput(task_0001_m_000018_0,5) failed :
java.io.EOFException
at java.io.DataInputStream.readFully(DataInputStream.java:178)
at java.io.DataInputStream.readLong(DataInputStream.java:380)
at
org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.
java:1911)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:689)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
at
org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:427)
at
org.mortbay.jetty.servlet.WebApplicationHandler.dispatch(WebApplicationH
andler.java:475)
at
org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:567)
at org.mortbay.http.HttpContext.handle(HttpContext.java:1565)
at
org.mortbay.jetty.servlet.WebApplicationContext.handle(WebApplicationCon
text.java:635)
at org.mortbay.http.HttpContext.handle(HttpContext.java:1517)
at org.mortbay.http.HttpServer.service(HttpServer.java:954)
at
org.mortbay.http.HttpConnection.service(HttpConnection.java:814)
at
org.mortbay.http.HttpConnection.handleNext(HttpConnection.java:981)
at
org.mortbay.http.HttpConnection.handle(HttpConnection.java:831)
at
org.mortbay.http.SocketListener.handleConnection(SocketListener.java:244
)
at
org.mortbay.util.ThreadedServer.handle(ThreadedServer.java:357)
at
org.mortbay.util.ThreadPool$PoolThread.run(ThreadPool.java:534)
These errors come when map task status is 100% and reduce is 76%. After
this the the map % drops to 95 and again both increase. This happens a
few times and then the job completes.
I have increased the number of tasks per node as well as the tasks per
job. Any idea what the problem is?
Please let me know if you need more information.
Thanks,
Neeraj