You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-user@hadoop.apache.org by David Parks <da...@yahoo.com> on 2013/05/14 08:19:38 UTC
JobClient: Error reading task output - after instituting a DNS server
So we just configured a local DNS server for hostname resolution and stopped
using a hosts file and now jobs fail on us. But I can't figure out why.
You can see the error below, but if I run curl to any of those URLs they
come back "Failed to retrieve stdout log", which doesn't look much like a
DNS issue.
I can ping and do nslookup from any host to any other host. This is a CDH4
cluster and the host inspector is happy as could be; also Cloudera Manager
indicates all is well.
When I open the task tracker website I see the first task attempt show up on
the site there for maybe 10 seconds or so before it fails.
Any idea what I need to look at here?
Job:
====
13/05/14 05:13:40 INFO input.FileInputFormat: Total input paths to process :
131
13/05/14 05:13:41 INFO input.FileInputFormat: Total input paths to process :
1
13/05/14 05:13:42 INFO mapred.JobClient: Running job: job_201305131758_0003
13/05/14 05:13:43 INFO mapred.JobClient: map 0% reduce 0%
13/05/14 05:13:47 INFO mapred.JobClient: Task Id :
attempt_201305131758_0003_m_000353_0, Status : FAILED
java.lang.Throwable: Child Error
at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:250)
Caused by: java.io.IOException: Task process exit with nonzero status of 1.
at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:237)
13/05/14 05:13:47 WARN mapred.JobClient: Error reading task
outputhttp://hadoop-fullslot2:50060/tasklog?plaintext=true&attemptid=attempt
_201305131758_0003_m_000353_0&filter=stdout
13/05/14 05:13:47 WARN mapred.JobClient: Error reading task
outputhttp://hadoop-fullslot2:50060/tasklog?plaintext=true&attemptid=attempt
_201305131758_0003_m_000353_0&filter=stderr
13/05/14 05:13:50 INFO mapred.JobClient: Task Id :
attempt_201305131758_0003_r_000521_0, Status : FAILED
java.lang.Throwable: Child Error
at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:250)
Caused by: java.io.IOException: Task process exit with nonzero status of 1.
at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:237)
13/05/14 05:13:50 WARN mapred.JobClient: Error reading task
outputhttp://hadoop-fullslot2:50060/tasklog?plaintext=true&attemptid=attempt
_201305131758_0003_r_000521_0&filter=stdout
13/05/14 05:13:50 WARN mapred.JobClient: Error reading task
outputhttp://hadoop-fullslot2:50060/tasklog?plaintext=true&attemptid=attempt
_201305131758_0003_r_000521_0&filter=stderr
curl of above URL:
====================
davidparks21@hadoop-meta1:~$ curl
'http://hadoop-fullslot2:50060/tasklog?plaintext=true&attemptid=attempt_2013
05131758_0003_m_000353_0&filter=stdout'
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1"/>
<title>Error 410 Failed to retrieve stdout log for task:
attempt_201305131758_0003_m_000353_0</title>
</head>
<body><h2>HTTP ERROR 410</h2>
<p>Problem accessing /tasklog. Reason:
<pre> Failed to retrieve stdout log for task:
attempt_201305131758_0003_m_000353_0</pre></p><hr /><i><small>Powered by
Jetty://</small></i><br/>
<br/>
<br/>
<br/>
<br/>
RE: JobClient: Error reading task output - after instituting a DNS server
Posted by David Parks <da...@yahoo.com>.
So simple I was hoping to avoid admitting to it. ;-)
I had set the tasks java options at -Xmx1.5g, that needed to be -Xmx1500m,
the telltale output of a mistake like that is rather tricky to find, I had
to dig into the task tracker UI/logs, it doesn't show up on the job
tracker's normal logs.
The timing perfectly coincided with a DNS change, and Googles first hit, on
the error that I *could* see in the jobtracker logs, suggested DNS, so I
went down that rabbit hole for quite a while.
Dave
From: Shahab Yunus [mailto:shahab.yunus@gmail.com]
Sent: Tuesday, May 14, 2013 6:56 PM
To: user@hadoop.apache.org
Subject: Re: JobClient: Error reading task output - after instituting a DNS
server
HI David. an you explain in a bit more detail what was the issue? Thanks.
Shahab
On Tue, May 14, 2013 at 2:29 AM, David Parks <da...@yahoo.com> wrote:
I just hate it when I figure out a problem right after asking for help.
Finding the task logs via the task tracker website identified the problem
which didn't show up elsewhere. Simple mis-configuration which I did
concurrently with the DNS update that threw me off track.
Dave
From: David Parks [mailto:davidparks21@yahoo.com]
Sent: Tuesday, May 14, 2013 1:20 PM
To: user@hadoop.apache.org
Subject: JobClient: Error reading task output - after instituting a DNS
server
So we just configured a local DNS server for hostname resolution and stopped
using a hosts file and now jobs fail on us. But I can't figure out why.
You can see the error below, but if I run curl to any of those URLs they
come back "Failed to retrieve stdout log", which doesn't look much like a
DNS issue.
I can ping and do nslookup from any host to any other host. This is a CDH4
cluster and the host inspector is happy as could be; also Cloudera Manager
indicates all is well.
When I open the task tracker website I see the first task attempt show up on
the site there for maybe 10 seconds or so before it fails.
Any idea what I need to look at here?
Job:
====
13/05/14 05:13:40 INFO input.FileInputFormat: Total input paths to process :
131
13/05/14 05:13:41 INFO input.FileInputFormat: Total input paths to process :
1
13/05/14 05:13:42 INFO mapred.JobClient: Running job: job_201305131758_0003
13/05/14 05:13:43 INFO mapred.JobClient: map 0% reduce 0%
13/05/14 05:13:47 INFO mapred.JobClient: Task Id :
attempt_201305131758_0003_m_000353_0, Status : FAILED
java.lang.Throwable: Child Error
at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:250)
Caused by: java.io.IOException: Task process exit with nonzero status of 1.
at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:237)
13/05/14 05:13:47 WARN mapred.JobClient: Error reading task
outputhttp://hadoop-fullslot2:50060/tasklog?plaintext=true&attemptid=attempt
_201305131758_0003_m_000353_0&filter=stdout
13/05/14 05:13:47 WARN mapred.JobClient: Error reading task
outputhttp://hadoop-fullslot2:50060/tasklog?plaintext=true&attemptid=attempt
_201305131758_0003_m_000353_0&filter=stderr
13/05/14 05:13:50 INFO mapred.JobClient: Task Id :
attempt_201305131758_0003_r_000521_0, Status : FAILED
java.lang.Throwable: Child Error
at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:250)
Caused by: java.io.IOException: Task process exit with nonzero status of 1.
at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:237)
13/05/14 05:13:50 WARN mapred.JobClient: Error reading task
outputhttp://hadoop-fullslot2:50060/tasklog?plaintext=true&attemptid=attempt
_201305131758_0003_r_000521_0&filter=stdout
13/05/14 05:13:50 WARN mapred.JobClient: Error reading task
outputhttp://hadoop-fullslot2:50060/tasklog?plaintext=true&attemptid=attempt
_201305131758_0003_r_000521_0&filter=stderr
curl of above URL:
====================
davidparks21@hadoop-meta1:~$ curl
'http://hadoop-fullslot2:50060/tasklog?plaintext=true
<http://hadoop-fullslot2:50060/tasklog?plaintext=true&attemptid=attempt_2013
05131758_0003_m_000353_0&filter=stdout>
&attemptid=attempt_201305131758_0003_m_000353_0&filter=stdout'
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1"/>
<title>Error 410 Failed to retrieve stdout log for task:
attempt_201305131758_0003_m_000353_0</title>
</head>
<body><h2>HTTP ERROR 410</h2>
<p>Problem accessing /tasklog. Reason:
<pre> Failed to retrieve stdout log for task:
attempt_201305131758_0003_m_000353_0</pre></p><hr /><i><small>Powered by
Jetty://</small></i><br/>
<br/>
<br/>
<br/>
<br/>
RE: JobClient: Error reading task output - after instituting a DNS server
Posted by David Parks <da...@yahoo.com>.
So simple I was hoping to avoid admitting to it. ;-)
I had set the tasks java options at -Xmx1.5g, that needed to be -Xmx1500m,
the telltale output of a mistake like that is rather tricky to find, I had
to dig into the task tracker UI/logs, it doesn't show up on the job
tracker's normal logs.
The timing perfectly coincided with a DNS change, and Googles first hit, on
the error that I *could* see in the jobtracker logs, suggested DNS, so I
went down that rabbit hole for quite a while.
Dave
From: Shahab Yunus [mailto:shahab.yunus@gmail.com]
Sent: Tuesday, May 14, 2013 6:56 PM
To: user@hadoop.apache.org
Subject: Re: JobClient: Error reading task output - after instituting a DNS
server
HI David. an you explain in a bit more detail what was the issue? Thanks.
Shahab
On Tue, May 14, 2013 at 2:29 AM, David Parks <da...@yahoo.com> wrote:
I just hate it when I figure out a problem right after asking for help.
Finding the task logs via the task tracker website identified the problem
which didn't show up elsewhere. Simple mis-configuration which I did
concurrently with the DNS update that threw me off track.
Dave
From: David Parks [mailto:davidparks21@yahoo.com]
Sent: Tuesday, May 14, 2013 1:20 PM
To: user@hadoop.apache.org
Subject: JobClient: Error reading task output - after instituting a DNS
server
So we just configured a local DNS server for hostname resolution and stopped
using a hosts file and now jobs fail on us. But I can't figure out why.
You can see the error below, but if I run curl to any of those URLs they
come back "Failed to retrieve stdout log", which doesn't look much like a
DNS issue.
I can ping and do nslookup from any host to any other host. This is a CDH4
cluster and the host inspector is happy as could be; also Cloudera Manager
indicates all is well.
When I open the task tracker website I see the first task attempt show up on
the site there for maybe 10 seconds or so before it fails.
Any idea what I need to look at here?
Job:
====
13/05/14 05:13:40 INFO input.FileInputFormat: Total input paths to process :
131
13/05/14 05:13:41 INFO input.FileInputFormat: Total input paths to process :
1
13/05/14 05:13:42 INFO mapred.JobClient: Running job: job_201305131758_0003
13/05/14 05:13:43 INFO mapred.JobClient: map 0% reduce 0%
13/05/14 05:13:47 INFO mapred.JobClient: Task Id :
attempt_201305131758_0003_m_000353_0, Status : FAILED
java.lang.Throwable: Child Error
at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:250)
Caused by: java.io.IOException: Task process exit with nonzero status of 1.
at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:237)
13/05/14 05:13:47 WARN mapred.JobClient: Error reading task
outputhttp://hadoop-fullslot2:50060/tasklog?plaintext=true&attemptid=attempt
_201305131758_0003_m_000353_0&filter=stdout
13/05/14 05:13:47 WARN mapred.JobClient: Error reading task
outputhttp://hadoop-fullslot2:50060/tasklog?plaintext=true&attemptid=attempt
_201305131758_0003_m_000353_0&filter=stderr
13/05/14 05:13:50 INFO mapred.JobClient: Task Id :
attempt_201305131758_0003_r_000521_0, Status : FAILED
java.lang.Throwable: Child Error
at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:250)
Caused by: java.io.IOException: Task process exit with nonzero status of 1.
at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:237)
13/05/14 05:13:50 WARN mapred.JobClient: Error reading task
outputhttp://hadoop-fullslot2:50060/tasklog?plaintext=true&attemptid=attempt
_201305131758_0003_r_000521_0&filter=stdout
13/05/14 05:13:50 WARN mapred.JobClient: Error reading task
outputhttp://hadoop-fullslot2:50060/tasklog?plaintext=true&attemptid=attempt
_201305131758_0003_r_000521_0&filter=stderr
curl of above URL:
====================
davidparks21@hadoop-meta1:~$ curl
'http://hadoop-fullslot2:50060/tasklog?plaintext=true
<http://hadoop-fullslot2:50060/tasklog?plaintext=true&attemptid=attempt_2013
05131758_0003_m_000353_0&filter=stdout>
&attemptid=attempt_201305131758_0003_m_000353_0&filter=stdout'
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1"/>
<title>Error 410 Failed to retrieve stdout log for task:
attempt_201305131758_0003_m_000353_0</title>
</head>
<body><h2>HTTP ERROR 410</h2>
<p>Problem accessing /tasklog. Reason:
<pre> Failed to retrieve stdout log for task:
attempt_201305131758_0003_m_000353_0</pre></p><hr /><i><small>Powered by
Jetty://</small></i><br/>
<br/>
<br/>
<br/>
<br/>
RE: JobClient: Error reading task output - after instituting a DNS server
Posted by David Parks <da...@yahoo.com>.
So simple I was hoping to avoid admitting to it. ;-)
I had set the tasks java options at -Xmx1.5g, that needed to be -Xmx1500m,
the telltale output of a mistake like that is rather tricky to find, I had
to dig into the task tracker UI/logs, it doesn't show up on the job
tracker's normal logs.
The timing perfectly coincided with a DNS change, and Googles first hit, on
the error that I *could* see in the jobtracker logs, suggested DNS, so I
went down that rabbit hole for quite a while.
Dave
From: Shahab Yunus [mailto:shahab.yunus@gmail.com]
Sent: Tuesday, May 14, 2013 6:56 PM
To: user@hadoop.apache.org
Subject: Re: JobClient: Error reading task output - after instituting a DNS
server
HI David. an you explain in a bit more detail what was the issue? Thanks.
Shahab
On Tue, May 14, 2013 at 2:29 AM, David Parks <da...@yahoo.com> wrote:
I just hate it when I figure out a problem right after asking for help.
Finding the task logs via the task tracker website identified the problem
which didn't show up elsewhere. Simple mis-configuration which I did
concurrently with the DNS update that threw me off track.
Dave
From: David Parks [mailto:davidparks21@yahoo.com]
Sent: Tuesday, May 14, 2013 1:20 PM
To: user@hadoop.apache.org
Subject: JobClient: Error reading task output - after instituting a DNS
server
So we just configured a local DNS server for hostname resolution and stopped
using a hosts file and now jobs fail on us. But I can't figure out why.
You can see the error below, but if I run curl to any of those URLs they
come back "Failed to retrieve stdout log", which doesn't look much like a
DNS issue.
I can ping and do nslookup from any host to any other host. This is a CDH4
cluster and the host inspector is happy as could be; also Cloudera Manager
indicates all is well.
When I open the task tracker website I see the first task attempt show up on
the site there for maybe 10 seconds or so before it fails.
Any idea what I need to look at here?
Job:
====
13/05/14 05:13:40 INFO input.FileInputFormat: Total input paths to process :
131
13/05/14 05:13:41 INFO input.FileInputFormat: Total input paths to process :
1
13/05/14 05:13:42 INFO mapred.JobClient: Running job: job_201305131758_0003
13/05/14 05:13:43 INFO mapred.JobClient: map 0% reduce 0%
13/05/14 05:13:47 INFO mapred.JobClient: Task Id :
attempt_201305131758_0003_m_000353_0, Status : FAILED
java.lang.Throwable: Child Error
at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:250)
Caused by: java.io.IOException: Task process exit with nonzero status of 1.
at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:237)
13/05/14 05:13:47 WARN mapred.JobClient: Error reading task
outputhttp://hadoop-fullslot2:50060/tasklog?plaintext=true&attemptid=attempt
_201305131758_0003_m_000353_0&filter=stdout
13/05/14 05:13:47 WARN mapred.JobClient: Error reading task
outputhttp://hadoop-fullslot2:50060/tasklog?plaintext=true&attemptid=attempt
_201305131758_0003_m_000353_0&filter=stderr
13/05/14 05:13:50 INFO mapred.JobClient: Task Id :
attempt_201305131758_0003_r_000521_0, Status : FAILED
java.lang.Throwable: Child Error
at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:250)
Caused by: java.io.IOException: Task process exit with nonzero status of 1.
at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:237)
13/05/14 05:13:50 WARN mapred.JobClient: Error reading task
outputhttp://hadoop-fullslot2:50060/tasklog?plaintext=true&attemptid=attempt
_201305131758_0003_r_000521_0&filter=stdout
13/05/14 05:13:50 WARN mapred.JobClient: Error reading task
outputhttp://hadoop-fullslot2:50060/tasklog?plaintext=true&attemptid=attempt
_201305131758_0003_r_000521_0&filter=stderr
curl of above URL:
====================
davidparks21@hadoop-meta1:~$ curl
'http://hadoop-fullslot2:50060/tasklog?plaintext=true
<http://hadoop-fullslot2:50060/tasklog?plaintext=true&attemptid=attempt_2013
05131758_0003_m_000353_0&filter=stdout>
&attemptid=attempt_201305131758_0003_m_000353_0&filter=stdout'
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1"/>
<title>Error 410 Failed to retrieve stdout log for task:
attempt_201305131758_0003_m_000353_0</title>
</head>
<body><h2>HTTP ERROR 410</h2>
<p>Problem accessing /tasklog. Reason:
<pre> Failed to retrieve stdout log for task:
attempt_201305131758_0003_m_000353_0</pre></p><hr /><i><small>Powered by
Jetty://</small></i><br/>
<br/>
<br/>
<br/>
<br/>
RE: JobClient: Error reading task output - after instituting a DNS server
Posted by David Parks <da...@yahoo.com>.
So simple I was hoping to avoid admitting to it. ;-)
I had set the tasks java options at -Xmx1.5g, that needed to be -Xmx1500m,
the telltale output of a mistake like that is rather tricky to find, I had
to dig into the task tracker UI/logs, it doesn't show up on the job
tracker's normal logs.
The timing perfectly coincided with a DNS change, and Googles first hit, on
the error that I *could* see in the jobtracker logs, suggested DNS, so I
went down that rabbit hole for quite a while.
Dave
From: Shahab Yunus [mailto:shahab.yunus@gmail.com]
Sent: Tuesday, May 14, 2013 6:56 PM
To: user@hadoop.apache.org
Subject: Re: JobClient: Error reading task output - after instituting a DNS
server
HI David. an you explain in a bit more detail what was the issue? Thanks.
Shahab
On Tue, May 14, 2013 at 2:29 AM, David Parks <da...@yahoo.com> wrote:
I just hate it when I figure out a problem right after asking for help.
Finding the task logs via the task tracker website identified the problem
which didn't show up elsewhere. Simple mis-configuration which I did
concurrently with the DNS update that threw me off track.
Dave
From: David Parks [mailto:davidparks21@yahoo.com]
Sent: Tuesday, May 14, 2013 1:20 PM
To: user@hadoop.apache.org
Subject: JobClient: Error reading task output - after instituting a DNS
server
So we just configured a local DNS server for hostname resolution and stopped
using a hosts file and now jobs fail on us. But I can't figure out why.
You can see the error below, but if I run curl to any of those URLs they
come back "Failed to retrieve stdout log", which doesn't look much like a
DNS issue.
I can ping and do nslookup from any host to any other host. This is a CDH4
cluster and the host inspector is happy as could be; also Cloudera Manager
indicates all is well.
When I open the task tracker website I see the first task attempt show up on
the site there for maybe 10 seconds or so before it fails.
Any idea what I need to look at here?
Job:
====
13/05/14 05:13:40 INFO input.FileInputFormat: Total input paths to process :
131
13/05/14 05:13:41 INFO input.FileInputFormat: Total input paths to process :
1
13/05/14 05:13:42 INFO mapred.JobClient: Running job: job_201305131758_0003
13/05/14 05:13:43 INFO mapred.JobClient: map 0% reduce 0%
13/05/14 05:13:47 INFO mapred.JobClient: Task Id :
attempt_201305131758_0003_m_000353_0, Status : FAILED
java.lang.Throwable: Child Error
at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:250)
Caused by: java.io.IOException: Task process exit with nonzero status of 1.
at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:237)
13/05/14 05:13:47 WARN mapred.JobClient: Error reading task
outputhttp://hadoop-fullslot2:50060/tasklog?plaintext=true&attemptid=attempt
_201305131758_0003_m_000353_0&filter=stdout
13/05/14 05:13:47 WARN mapred.JobClient: Error reading task
outputhttp://hadoop-fullslot2:50060/tasklog?plaintext=true&attemptid=attempt
_201305131758_0003_m_000353_0&filter=stderr
13/05/14 05:13:50 INFO mapred.JobClient: Task Id :
attempt_201305131758_0003_r_000521_0, Status : FAILED
java.lang.Throwable: Child Error
at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:250)
Caused by: java.io.IOException: Task process exit with nonzero status of 1.
at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:237)
13/05/14 05:13:50 WARN mapred.JobClient: Error reading task
outputhttp://hadoop-fullslot2:50060/tasklog?plaintext=true&attemptid=attempt
_201305131758_0003_r_000521_0&filter=stdout
13/05/14 05:13:50 WARN mapred.JobClient: Error reading task
outputhttp://hadoop-fullslot2:50060/tasklog?plaintext=true&attemptid=attempt
_201305131758_0003_r_000521_0&filter=stderr
curl of above URL:
====================
davidparks21@hadoop-meta1:~$ curl
'http://hadoop-fullslot2:50060/tasklog?plaintext=true
<http://hadoop-fullslot2:50060/tasklog?plaintext=true&attemptid=attempt_2013
05131758_0003_m_000353_0&filter=stdout>
&attemptid=attempt_201305131758_0003_m_000353_0&filter=stdout'
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1"/>
<title>Error 410 Failed to retrieve stdout log for task:
attempt_201305131758_0003_m_000353_0</title>
</head>
<body><h2>HTTP ERROR 410</h2>
<p>Problem accessing /tasklog. Reason:
<pre> Failed to retrieve stdout log for task:
attempt_201305131758_0003_m_000353_0</pre></p><hr /><i><small>Powered by
Jetty://</small></i><br/>
<br/>
<br/>
<br/>
<br/>
Re: JobClient: Error reading task output - after instituting a DNS server
Posted by Shahab Yunus <sh...@gmail.com>.
HI David. an you explain in a bit more detail what was the issue? Thanks.
Shahab
On Tue, May 14, 2013 at 2:29 AM, David Parks <da...@yahoo.com> wrote:
> I just hate it when I figure out a problem right after asking for help.***
> *
>
> ** **
>
> Finding the task logs via the task tracker website identified the problem
> which didn’t show up elsewhere. Simple mis-configuration which I did
> concurrently with the DNS update that threw me off track.****
>
> ** **
>
> Dave****
>
> ** **
>
> ** **
>
> *From:* David Parks [mailto:davidparks21@yahoo.com]
> *Sent:* Tuesday, May 14, 2013 1:20 PM
> *To:* user@hadoop.apache.org
> *Subject:* JobClient: Error reading task output - after instituting a DNS
> server****
>
> ** **
>
> So we just configured a local DNS server for hostname resolution and
> stopped using a hosts file and now jobs fail on us. But I can’t figure out
> why.****
>
> ** **
>
> You can see the error below, but if I run curl to any of those URLs they
> come back “Failed to retrieve stdout log”, which doesn’t look much like a
> DNS issue.****
>
> ** **
>
> I can ping and do nslookup from any host to any other host. This is a CDH4
> cluster and the host inspector is happy as could be; also Cloudera Manager
> indicates all is well.****
>
> ** **
>
> When I open the task tracker website I see the first task attempt show up
> on the site there for maybe 10 seconds or so before it fails.****
>
> ** **
>
> Any idea what I need to look at here?****
>
> ** **
>
> Job:****
>
> ====****
>
> 13/05/14 05:13:40 INFO input.FileInputFormat: Total input paths to process
> : 131****
>
> 13/05/14 05:13:41 INFO input.FileInputFormat: Total input paths to process
> : 1****
>
> 13/05/14 05:13:42 INFO mapred.JobClient: Running job: job_201305131758_0003
> ****
>
> 13/05/14 05:13:43 INFO mapred.JobClient: map 0% reduce 0%****
>
> 13/05/14 05:13:47 INFO mapred.JobClient: Task Id :
> attempt_201305131758_0003_m_000353_0, Status : FAILED****
>
> java.lang.Throwable: Child Error****
>
> at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:250)***
> *
>
> Caused by: java.io.IOException: Task process exit with nonzero status of 1.
> ****
>
> at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:237)***
> *
>
> ** **
>
> 13/05/14 05:13:47 WARN mapred.JobClient: Error reading task
> outputhttp://hadoop-fullslot2:50060/tasklog?plaintext=true&attemptid=attempt_201305131758_0003_m_000353_0&filter=stdout
> ****
>
> 13/05/14 05:13:47 WARN mapred.JobClient: Error reading task
> outputhttp://hadoop-fullslot2:50060/tasklog?plaintext=true&attemptid=attempt_201305131758_0003_m_000353_0&filter=stderr
> ****
>
> 13/05/14 05:13:50 INFO mapred.JobClient: Task Id :
> attempt_201305131758_0003_r_000521_0, Status : FAILED****
>
> java.lang.Throwable: Child Error****
>
> at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:250)***
> *
>
> Caused by: java.io.IOException: Task process exit with nonzero status of 1.
> ****
>
> at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:237)***
> *
>
> ** **
>
> 13/05/14 05:13:50 WARN mapred.JobClient: Error reading task
> outputhttp://hadoop-fullslot2:50060/tasklog?plaintext=true&attemptid=attempt_201305131758_0003_r_000521_0&filter=stdout
> ****
>
> 13/05/14 05:13:50 WARN mapred.JobClient: Error reading task
> outputhttp://hadoop-fullslot2:50060/tasklog?plaintext=true&attemptid=attempt_201305131758_0003_r_000521_0&filter=stderr
> ****
>
> ** **
>
> ** **
>
> curl of above URL:****
>
> ====================****
>
> davidparks21@hadoop-meta1:~$ curl '
> http://hadoop-fullslot2:50060/tasklog?plaintext=true&attemptid=attempt_201305131758_0003_m_000353_0&filter=stdout
> '****
>
> <html>****
>
> <head>****
>
> <meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1"/>*
> ***
>
> <title>Error 410 Failed to retrieve stdout log for task:
> attempt_201305131758_0003_m_000353_0</title>****
>
> </head>****
>
> <body><h2>HTTP ERROR 410</h2>****
>
> <p>Problem accessing /tasklog. Reason:****
>
> <pre> Failed to retrieve stdout log for task:
> attempt_201305131758_0003_m_000353_0</pre></p><hr /><i><small>Powered by
> Jetty://</small></i><br/> ****
>
> <br/>****
>
> <br/>****
>
> <br/>****
>
> <br/>****
>
Re: JobClient: Error reading task output - after instituting a DNS server
Posted by Shahab Yunus <sh...@gmail.com>.
HI David. an you explain in a bit more detail what was the issue? Thanks.
Shahab
On Tue, May 14, 2013 at 2:29 AM, David Parks <da...@yahoo.com> wrote:
> I just hate it when I figure out a problem right after asking for help.***
> *
>
> ** **
>
> Finding the task logs via the task tracker website identified the problem
> which didn’t show up elsewhere. Simple mis-configuration which I did
> concurrently with the DNS update that threw me off track.****
>
> ** **
>
> Dave****
>
> ** **
>
> ** **
>
> *From:* David Parks [mailto:davidparks21@yahoo.com]
> *Sent:* Tuesday, May 14, 2013 1:20 PM
> *To:* user@hadoop.apache.org
> *Subject:* JobClient: Error reading task output - after instituting a DNS
> server****
>
> ** **
>
> So we just configured a local DNS server for hostname resolution and
> stopped using a hosts file and now jobs fail on us. But I can’t figure out
> why.****
>
> ** **
>
> You can see the error below, but if I run curl to any of those URLs they
> come back “Failed to retrieve stdout log”, which doesn’t look much like a
> DNS issue.****
>
> ** **
>
> I can ping and do nslookup from any host to any other host. This is a CDH4
> cluster and the host inspector is happy as could be; also Cloudera Manager
> indicates all is well.****
>
> ** **
>
> When I open the task tracker website I see the first task attempt show up
> on the site there for maybe 10 seconds or so before it fails.****
>
> ** **
>
> Any idea what I need to look at here?****
>
> ** **
>
> Job:****
>
> ====****
>
> 13/05/14 05:13:40 INFO input.FileInputFormat: Total input paths to process
> : 131****
>
> 13/05/14 05:13:41 INFO input.FileInputFormat: Total input paths to process
> : 1****
>
> 13/05/14 05:13:42 INFO mapred.JobClient: Running job: job_201305131758_0003
> ****
>
> 13/05/14 05:13:43 INFO mapred.JobClient: map 0% reduce 0%****
>
> 13/05/14 05:13:47 INFO mapred.JobClient: Task Id :
> attempt_201305131758_0003_m_000353_0, Status : FAILED****
>
> java.lang.Throwable: Child Error****
>
> at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:250)***
> *
>
> Caused by: java.io.IOException: Task process exit with nonzero status of 1.
> ****
>
> at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:237)***
> *
>
> ** **
>
> 13/05/14 05:13:47 WARN mapred.JobClient: Error reading task
> outputhttp://hadoop-fullslot2:50060/tasklog?plaintext=true&attemptid=attempt_201305131758_0003_m_000353_0&filter=stdout
> ****
>
> 13/05/14 05:13:47 WARN mapred.JobClient: Error reading task
> outputhttp://hadoop-fullslot2:50060/tasklog?plaintext=true&attemptid=attempt_201305131758_0003_m_000353_0&filter=stderr
> ****
>
> 13/05/14 05:13:50 INFO mapred.JobClient: Task Id :
> attempt_201305131758_0003_r_000521_0, Status : FAILED****
>
> java.lang.Throwable: Child Error****
>
> at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:250)***
> *
>
> Caused by: java.io.IOException: Task process exit with nonzero status of 1.
> ****
>
> at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:237)***
> *
>
> ** **
>
> 13/05/14 05:13:50 WARN mapred.JobClient: Error reading task
> outputhttp://hadoop-fullslot2:50060/tasklog?plaintext=true&attemptid=attempt_201305131758_0003_r_000521_0&filter=stdout
> ****
>
> 13/05/14 05:13:50 WARN mapred.JobClient: Error reading task
> outputhttp://hadoop-fullslot2:50060/tasklog?plaintext=true&attemptid=attempt_201305131758_0003_r_000521_0&filter=stderr
> ****
>
> ** **
>
> ** **
>
> curl of above URL:****
>
> ====================****
>
> davidparks21@hadoop-meta1:~$ curl '
> http://hadoop-fullslot2:50060/tasklog?plaintext=true&attemptid=attempt_201305131758_0003_m_000353_0&filter=stdout
> '****
>
> <html>****
>
> <head>****
>
> <meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1"/>*
> ***
>
> <title>Error 410 Failed to retrieve stdout log for task:
> attempt_201305131758_0003_m_000353_0</title>****
>
> </head>****
>
> <body><h2>HTTP ERROR 410</h2>****
>
> <p>Problem accessing /tasklog. Reason:****
>
> <pre> Failed to retrieve stdout log for task:
> attempt_201305131758_0003_m_000353_0</pre></p><hr /><i><small>Powered by
> Jetty://</small></i><br/> ****
>
> <br/>****
>
> <br/>****
>
> <br/>****
>
> <br/>****
>
Re: JobClient: Error reading task output - after instituting a DNS server
Posted by Shahab Yunus <sh...@gmail.com>.
HI David. an you explain in a bit more detail what was the issue? Thanks.
Shahab
On Tue, May 14, 2013 at 2:29 AM, David Parks <da...@yahoo.com> wrote:
> I just hate it when I figure out a problem right after asking for help.***
> *
>
> ** **
>
> Finding the task logs via the task tracker website identified the problem
> which didn’t show up elsewhere. Simple mis-configuration which I did
> concurrently with the DNS update that threw me off track.****
>
> ** **
>
> Dave****
>
> ** **
>
> ** **
>
> *From:* David Parks [mailto:davidparks21@yahoo.com]
> *Sent:* Tuesday, May 14, 2013 1:20 PM
> *To:* user@hadoop.apache.org
> *Subject:* JobClient: Error reading task output - after instituting a DNS
> server****
>
> ** **
>
> So we just configured a local DNS server for hostname resolution and
> stopped using a hosts file and now jobs fail on us. But I can’t figure out
> why.****
>
> ** **
>
> You can see the error below, but if I run curl to any of those URLs they
> come back “Failed to retrieve stdout log”, which doesn’t look much like a
> DNS issue.****
>
> ** **
>
> I can ping and do nslookup from any host to any other host. This is a CDH4
> cluster and the host inspector is happy as could be; also Cloudera Manager
> indicates all is well.****
>
> ** **
>
> When I open the task tracker website I see the first task attempt show up
> on the site there for maybe 10 seconds or so before it fails.****
>
> ** **
>
> Any idea what I need to look at here?****
>
> ** **
>
> Job:****
>
> ====****
>
> 13/05/14 05:13:40 INFO input.FileInputFormat: Total input paths to process
> : 131****
>
> 13/05/14 05:13:41 INFO input.FileInputFormat: Total input paths to process
> : 1****
>
> 13/05/14 05:13:42 INFO mapred.JobClient: Running job: job_201305131758_0003
> ****
>
> 13/05/14 05:13:43 INFO mapred.JobClient: map 0% reduce 0%****
>
> 13/05/14 05:13:47 INFO mapred.JobClient: Task Id :
> attempt_201305131758_0003_m_000353_0, Status : FAILED****
>
> java.lang.Throwable: Child Error****
>
> at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:250)***
> *
>
> Caused by: java.io.IOException: Task process exit with nonzero status of 1.
> ****
>
> at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:237)***
> *
>
> ** **
>
> 13/05/14 05:13:47 WARN mapred.JobClient: Error reading task
> outputhttp://hadoop-fullslot2:50060/tasklog?plaintext=true&attemptid=attempt_201305131758_0003_m_000353_0&filter=stdout
> ****
>
> 13/05/14 05:13:47 WARN mapred.JobClient: Error reading task
> outputhttp://hadoop-fullslot2:50060/tasklog?plaintext=true&attemptid=attempt_201305131758_0003_m_000353_0&filter=stderr
> ****
>
> 13/05/14 05:13:50 INFO mapred.JobClient: Task Id :
> attempt_201305131758_0003_r_000521_0, Status : FAILED****
>
> java.lang.Throwable: Child Error****
>
> at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:250)***
> *
>
> Caused by: java.io.IOException: Task process exit with nonzero status of 1.
> ****
>
> at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:237)***
> *
>
> ** **
>
> 13/05/14 05:13:50 WARN mapred.JobClient: Error reading task
> outputhttp://hadoop-fullslot2:50060/tasklog?plaintext=true&attemptid=attempt_201305131758_0003_r_000521_0&filter=stdout
> ****
>
> 13/05/14 05:13:50 WARN mapred.JobClient: Error reading task
> outputhttp://hadoop-fullslot2:50060/tasklog?plaintext=true&attemptid=attempt_201305131758_0003_r_000521_0&filter=stderr
> ****
>
> ** **
>
> ** **
>
> curl of above URL:****
>
> ====================****
>
> davidparks21@hadoop-meta1:~$ curl '
> http://hadoop-fullslot2:50060/tasklog?plaintext=true&attemptid=attempt_201305131758_0003_m_000353_0&filter=stdout
> '****
>
> <html>****
>
> <head>****
>
> <meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1"/>*
> ***
>
> <title>Error 410 Failed to retrieve stdout log for task:
> attempt_201305131758_0003_m_000353_0</title>****
>
> </head>****
>
> <body><h2>HTTP ERROR 410</h2>****
>
> <p>Problem accessing /tasklog. Reason:****
>
> <pre> Failed to retrieve stdout log for task:
> attempt_201305131758_0003_m_000353_0</pre></p><hr /><i><small>Powered by
> Jetty://</small></i><br/> ****
>
> <br/>****
>
> <br/>****
>
> <br/>****
>
> <br/>****
>
Re: JobClient: Error reading task output - after instituting a DNS server
Posted by Shahab Yunus <sh...@gmail.com>.
HI David. an you explain in a bit more detail what was the issue? Thanks.
Shahab
On Tue, May 14, 2013 at 2:29 AM, David Parks <da...@yahoo.com> wrote:
> I just hate it when I figure out a problem right after asking for help.***
> *
>
> ** **
>
> Finding the task logs via the task tracker website identified the problem
> which didn’t show up elsewhere. Simple mis-configuration which I did
> concurrently with the DNS update that threw me off track.****
>
> ** **
>
> Dave****
>
> ** **
>
> ** **
>
> *From:* David Parks [mailto:davidparks21@yahoo.com]
> *Sent:* Tuesday, May 14, 2013 1:20 PM
> *To:* user@hadoop.apache.org
> *Subject:* JobClient: Error reading task output - after instituting a DNS
> server****
>
> ** **
>
> So we just configured a local DNS server for hostname resolution and
> stopped using a hosts file and now jobs fail on us. But I can’t figure out
> why.****
>
> ** **
>
> You can see the error below, but if I run curl to any of those URLs they
> come back “Failed to retrieve stdout log”, which doesn’t look much like a
> DNS issue.****
>
> ** **
>
> I can ping and do nslookup from any host to any other host. This is a CDH4
> cluster and the host inspector is happy as could be; also Cloudera Manager
> indicates all is well.****
>
> ** **
>
> When I open the task tracker website I see the first task attempt show up
> on the site there for maybe 10 seconds or so before it fails.****
>
> ** **
>
> Any idea what I need to look at here?****
>
> ** **
>
> Job:****
>
> ====****
>
> 13/05/14 05:13:40 INFO input.FileInputFormat: Total input paths to process
> : 131****
>
> 13/05/14 05:13:41 INFO input.FileInputFormat: Total input paths to process
> : 1****
>
> 13/05/14 05:13:42 INFO mapred.JobClient: Running job: job_201305131758_0003
> ****
>
> 13/05/14 05:13:43 INFO mapred.JobClient: map 0% reduce 0%****
>
> 13/05/14 05:13:47 INFO mapred.JobClient: Task Id :
> attempt_201305131758_0003_m_000353_0, Status : FAILED****
>
> java.lang.Throwable: Child Error****
>
> at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:250)***
> *
>
> Caused by: java.io.IOException: Task process exit with nonzero status of 1.
> ****
>
> at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:237)***
> *
>
> ** **
>
> 13/05/14 05:13:47 WARN mapred.JobClient: Error reading task
> outputhttp://hadoop-fullslot2:50060/tasklog?plaintext=true&attemptid=attempt_201305131758_0003_m_000353_0&filter=stdout
> ****
>
> 13/05/14 05:13:47 WARN mapred.JobClient: Error reading task
> outputhttp://hadoop-fullslot2:50060/tasklog?plaintext=true&attemptid=attempt_201305131758_0003_m_000353_0&filter=stderr
> ****
>
> 13/05/14 05:13:50 INFO mapred.JobClient: Task Id :
> attempt_201305131758_0003_r_000521_0, Status : FAILED****
>
> java.lang.Throwable: Child Error****
>
> at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:250)***
> *
>
> Caused by: java.io.IOException: Task process exit with nonzero status of 1.
> ****
>
> at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:237)***
> *
>
> ** **
>
> 13/05/14 05:13:50 WARN mapred.JobClient: Error reading task
> outputhttp://hadoop-fullslot2:50060/tasklog?plaintext=true&attemptid=attempt_201305131758_0003_r_000521_0&filter=stdout
> ****
>
> 13/05/14 05:13:50 WARN mapred.JobClient: Error reading task
> outputhttp://hadoop-fullslot2:50060/tasklog?plaintext=true&attemptid=attempt_201305131758_0003_r_000521_0&filter=stderr
> ****
>
> ** **
>
> ** **
>
> curl of above URL:****
>
> ====================****
>
> davidparks21@hadoop-meta1:~$ curl '
> http://hadoop-fullslot2:50060/tasklog?plaintext=true&attemptid=attempt_201305131758_0003_m_000353_0&filter=stdout
> '****
>
> <html>****
>
> <head>****
>
> <meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1"/>*
> ***
>
> <title>Error 410 Failed to retrieve stdout log for task:
> attempt_201305131758_0003_m_000353_0</title>****
>
> </head>****
>
> <body><h2>HTTP ERROR 410</h2>****
>
> <p>Problem accessing /tasklog. Reason:****
>
> <pre> Failed to retrieve stdout log for task:
> attempt_201305131758_0003_m_000353_0</pre></p><hr /><i><small>Powered by
> Jetty://</small></i><br/> ****
>
> <br/>****
>
> <br/>****
>
> <br/>****
>
> <br/>****
>
RE: JobClient: Error reading task output - after instituting a DNS server
Posted by David Parks <da...@yahoo.com>.
I just hate it when I figure out a problem right after asking for help.
Finding the task logs via the task tracker website identified the problem
which didn't show up elsewhere. Simple mis-configuration which I did
concurrently with the DNS update that threw me off track.
Dave
From: David Parks [mailto:davidparks21@yahoo.com]
Sent: Tuesday, May 14, 2013 1:20 PM
To: user@hadoop.apache.org
Subject: JobClient: Error reading task output - after instituting a DNS
server
So we just configured a local DNS server for hostname resolution and stopped
using a hosts file and now jobs fail on us. But I can't figure out why.
You can see the error below, but if I run curl to any of those URLs they
come back "Failed to retrieve stdout log", which doesn't look much like a
DNS issue.
I can ping and do nslookup from any host to any other host. This is a CDH4
cluster and the host inspector is happy as could be; also Cloudera Manager
indicates all is well.
When I open the task tracker website I see the first task attempt show up on
the site there for maybe 10 seconds or so before it fails.
Any idea what I need to look at here?
Job:
====
13/05/14 05:13:40 INFO input.FileInputFormat: Total input paths to process :
131
13/05/14 05:13:41 INFO input.FileInputFormat: Total input paths to process :
1
13/05/14 05:13:42 INFO mapred.JobClient: Running job: job_201305131758_0003
13/05/14 05:13:43 INFO mapred.JobClient: map 0% reduce 0%
13/05/14 05:13:47 INFO mapred.JobClient: Task Id :
attempt_201305131758_0003_m_000353_0, Status : FAILED
java.lang.Throwable: Child Error
at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:250)
Caused by: java.io.IOException: Task process exit with nonzero status of 1.
at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:237)
13/05/14 05:13:47 WARN mapred.JobClient: Error reading task
outputhttp://hadoop-fullslot2:50060/tasklog?plaintext=true&attemptid=attempt
_201305131758_0003_m_000353_0&filter=stdout
13/05/14 05:13:47 WARN mapred.JobClient: Error reading task
outputhttp://hadoop-fullslot2:50060/tasklog?plaintext=true&attemptid=attempt
_201305131758_0003_m_000353_0&filter=stderr
13/05/14 05:13:50 INFO mapred.JobClient: Task Id :
attempt_201305131758_0003_r_000521_0, Status : FAILED
java.lang.Throwable: Child Error
at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:250)
Caused by: java.io.IOException: Task process exit with nonzero status of 1.
at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:237)
13/05/14 05:13:50 WARN mapred.JobClient: Error reading task
outputhttp://hadoop-fullslot2:50060/tasklog?plaintext=true&attemptid=attempt
_201305131758_0003_r_000521_0&filter=stdout
13/05/14 05:13:50 WARN mapred.JobClient: Error reading task
outputhttp://hadoop-fullslot2:50060/tasklog?plaintext=true&attemptid=attempt
_201305131758_0003_r_000521_0&filter=stderr
curl of above URL:
====================
davidparks21@hadoop-meta1:~$ curl
'http://hadoop-fullslot2:50060/tasklog?plaintext=true&attemptid=attempt_2013
05131758_0003_m_000353_0&filter=stdout'
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1"/>
<title>Error 410 Failed to retrieve stdout log for task:
attempt_201305131758_0003_m_000353_0</title>
</head>
<body><h2>HTTP ERROR 410</h2>
<p>Problem accessing /tasklog. Reason:
<pre> Failed to retrieve stdout log for task:
attempt_201305131758_0003_m_000353_0</pre></p><hr /><i><small>Powered by
Jetty://</small></i><br/>
<br/>
<br/>
<br/>
<br/>
RE: JobClient: Error reading task output - after instituting a DNS server
Posted by David Parks <da...@yahoo.com>.
I just hate it when I figure out a problem right after asking for help.
Finding the task logs via the task tracker website identified the problem
which didn't show up elsewhere. Simple mis-configuration which I did
concurrently with the DNS update that threw me off track.
Dave
From: David Parks [mailto:davidparks21@yahoo.com]
Sent: Tuesday, May 14, 2013 1:20 PM
To: user@hadoop.apache.org
Subject: JobClient: Error reading task output - after instituting a DNS
server
So we just configured a local DNS server for hostname resolution and stopped
using a hosts file and now jobs fail on us. But I can't figure out why.
You can see the error below, but if I run curl to any of those URLs they
come back "Failed to retrieve stdout log", which doesn't look much like a
DNS issue.
I can ping and do nslookup from any host to any other host. This is a CDH4
cluster and the host inspector is happy as could be; also Cloudera Manager
indicates all is well.
When I open the task tracker website I see the first task attempt show up on
the site there for maybe 10 seconds or so before it fails.
Any idea what I need to look at here?
Job:
====
13/05/14 05:13:40 INFO input.FileInputFormat: Total input paths to process :
131
13/05/14 05:13:41 INFO input.FileInputFormat: Total input paths to process :
1
13/05/14 05:13:42 INFO mapred.JobClient: Running job: job_201305131758_0003
13/05/14 05:13:43 INFO mapred.JobClient: map 0% reduce 0%
13/05/14 05:13:47 INFO mapred.JobClient: Task Id :
attempt_201305131758_0003_m_000353_0, Status : FAILED
java.lang.Throwable: Child Error
at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:250)
Caused by: java.io.IOException: Task process exit with nonzero status of 1.
at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:237)
13/05/14 05:13:47 WARN mapred.JobClient: Error reading task
outputhttp://hadoop-fullslot2:50060/tasklog?plaintext=true&attemptid=attempt
_201305131758_0003_m_000353_0&filter=stdout
13/05/14 05:13:47 WARN mapred.JobClient: Error reading task
outputhttp://hadoop-fullslot2:50060/tasklog?plaintext=true&attemptid=attempt
_201305131758_0003_m_000353_0&filter=stderr
13/05/14 05:13:50 INFO mapred.JobClient: Task Id :
attempt_201305131758_0003_r_000521_0, Status : FAILED
java.lang.Throwable: Child Error
at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:250)
Caused by: java.io.IOException: Task process exit with nonzero status of 1.
at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:237)
13/05/14 05:13:50 WARN mapred.JobClient: Error reading task
outputhttp://hadoop-fullslot2:50060/tasklog?plaintext=true&attemptid=attempt
_201305131758_0003_r_000521_0&filter=stdout
13/05/14 05:13:50 WARN mapred.JobClient: Error reading task
outputhttp://hadoop-fullslot2:50060/tasklog?plaintext=true&attemptid=attempt
_201305131758_0003_r_000521_0&filter=stderr
curl of above URL:
====================
davidparks21@hadoop-meta1:~$ curl
'http://hadoop-fullslot2:50060/tasklog?plaintext=true&attemptid=attempt_2013
05131758_0003_m_000353_0&filter=stdout'
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1"/>
<title>Error 410 Failed to retrieve stdout log for task:
attempt_201305131758_0003_m_000353_0</title>
</head>
<body><h2>HTTP ERROR 410</h2>
<p>Problem accessing /tasklog. Reason:
<pre> Failed to retrieve stdout log for task:
attempt_201305131758_0003_m_000353_0</pre></p><hr /><i><small>Powered by
Jetty://</small></i><br/>
<br/>
<br/>
<br/>
<br/>
RE: JobClient: Error reading task output - after instituting a DNS server
Posted by David Parks <da...@yahoo.com>.
I just hate it when I figure out a problem right after asking for help.
Finding the task logs via the task tracker website identified the problem
which didn't show up elsewhere. Simple mis-configuration which I did
concurrently with the DNS update that threw me off track.
Dave
From: David Parks [mailto:davidparks21@yahoo.com]
Sent: Tuesday, May 14, 2013 1:20 PM
To: user@hadoop.apache.org
Subject: JobClient: Error reading task output - after instituting a DNS
server
So we just configured a local DNS server for hostname resolution and stopped
using a hosts file and now jobs fail on us. But I can't figure out why.
You can see the error below, but if I run curl to any of those URLs they
come back "Failed to retrieve stdout log", which doesn't look much like a
DNS issue.
I can ping and do nslookup from any host to any other host. This is a CDH4
cluster and the host inspector is happy as could be; also Cloudera Manager
indicates all is well.
When I open the task tracker website I see the first task attempt show up on
the site there for maybe 10 seconds or so before it fails.
Any idea what I need to look at here?
Job:
====
13/05/14 05:13:40 INFO input.FileInputFormat: Total input paths to process :
131
13/05/14 05:13:41 INFO input.FileInputFormat: Total input paths to process :
1
13/05/14 05:13:42 INFO mapred.JobClient: Running job: job_201305131758_0003
13/05/14 05:13:43 INFO mapred.JobClient: map 0% reduce 0%
13/05/14 05:13:47 INFO mapred.JobClient: Task Id :
attempt_201305131758_0003_m_000353_0, Status : FAILED
java.lang.Throwable: Child Error
at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:250)
Caused by: java.io.IOException: Task process exit with nonzero status of 1.
at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:237)
13/05/14 05:13:47 WARN mapred.JobClient: Error reading task
outputhttp://hadoop-fullslot2:50060/tasklog?plaintext=true&attemptid=attempt
_201305131758_0003_m_000353_0&filter=stdout
13/05/14 05:13:47 WARN mapred.JobClient: Error reading task
outputhttp://hadoop-fullslot2:50060/tasklog?plaintext=true&attemptid=attempt
_201305131758_0003_m_000353_0&filter=stderr
13/05/14 05:13:50 INFO mapred.JobClient: Task Id :
attempt_201305131758_0003_r_000521_0, Status : FAILED
java.lang.Throwable: Child Error
at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:250)
Caused by: java.io.IOException: Task process exit with nonzero status of 1.
at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:237)
13/05/14 05:13:50 WARN mapred.JobClient: Error reading task
outputhttp://hadoop-fullslot2:50060/tasklog?plaintext=true&attemptid=attempt
_201305131758_0003_r_000521_0&filter=stdout
13/05/14 05:13:50 WARN mapred.JobClient: Error reading task
outputhttp://hadoop-fullslot2:50060/tasklog?plaintext=true&attemptid=attempt
_201305131758_0003_r_000521_0&filter=stderr
curl of above URL:
====================
davidparks21@hadoop-meta1:~$ curl
'http://hadoop-fullslot2:50060/tasklog?plaintext=true&attemptid=attempt_2013
05131758_0003_m_000353_0&filter=stdout'
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1"/>
<title>Error 410 Failed to retrieve stdout log for task:
attempt_201305131758_0003_m_000353_0</title>
</head>
<body><h2>HTTP ERROR 410</h2>
<p>Problem accessing /tasklog. Reason:
<pre> Failed to retrieve stdout log for task:
attempt_201305131758_0003_m_000353_0</pre></p><hr /><i><small>Powered by
Jetty://</small></i><br/>
<br/>
<br/>
<br/>
<br/>
RE: JobClient: Error reading task output - after instituting a DNS server
Posted by David Parks <da...@yahoo.com>.
I just hate it when I figure out a problem right after asking for help.
Finding the task logs via the task tracker website identified the problem
which didn't show up elsewhere. Simple mis-configuration which I did
concurrently with the DNS update that threw me off track.
Dave
From: David Parks [mailto:davidparks21@yahoo.com]
Sent: Tuesday, May 14, 2013 1:20 PM
To: user@hadoop.apache.org
Subject: JobClient: Error reading task output - after instituting a DNS
server
So we just configured a local DNS server for hostname resolution and stopped
using a hosts file and now jobs fail on us. But I can't figure out why.
You can see the error below, but if I run curl to any of those URLs they
come back "Failed to retrieve stdout log", which doesn't look much like a
DNS issue.
I can ping and do nslookup from any host to any other host. This is a CDH4
cluster and the host inspector is happy as could be; also Cloudera Manager
indicates all is well.
When I open the task tracker website I see the first task attempt show up on
the site there for maybe 10 seconds or so before it fails.
Any idea what I need to look at here?
Job:
====
13/05/14 05:13:40 INFO input.FileInputFormat: Total input paths to process :
131
13/05/14 05:13:41 INFO input.FileInputFormat: Total input paths to process :
1
13/05/14 05:13:42 INFO mapred.JobClient: Running job: job_201305131758_0003
13/05/14 05:13:43 INFO mapred.JobClient: map 0% reduce 0%
13/05/14 05:13:47 INFO mapred.JobClient: Task Id :
attempt_201305131758_0003_m_000353_0, Status : FAILED
java.lang.Throwable: Child Error
at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:250)
Caused by: java.io.IOException: Task process exit with nonzero status of 1.
at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:237)
13/05/14 05:13:47 WARN mapred.JobClient: Error reading task
outputhttp://hadoop-fullslot2:50060/tasklog?plaintext=true&attemptid=attempt
_201305131758_0003_m_000353_0&filter=stdout
13/05/14 05:13:47 WARN mapred.JobClient: Error reading task
outputhttp://hadoop-fullslot2:50060/tasklog?plaintext=true&attemptid=attempt
_201305131758_0003_m_000353_0&filter=stderr
13/05/14 05:13:50 INFO mapred.JobClient: Task Id :
attempt_201305131758_0003_r_000521_0, Status : FAILED
java.lang.Throwable: Child Error
at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:250)
Caused by: java.io.IOException: Task process exit with nonzero status of 1.
at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:237)
13/05/14 05:13:50 WARN mapred.JobClient: Error reading task
outputhttp://hadoop-fullslot2:50060/tasklog?plaintext=true&attemptid=attempt
_201305131758_0003_r_000521_0&filter=stdout
13/05/14 05:13:50 WARN mapred.JobClient: Error reading task
outputhttp://hadoop-fullslot2:50060/tasklog?plaintext=true&attemptid=attempt
_201305131758_0003_r_000521_0&filter=stderr
curl of above URL:
====================
davidparks21@hadoop-meta1:~$ curl
'http://hadoop-fullslot2:50060/tasklog?plaintext=true&attemptid=attempt_2013
05131758_0003_m_000353_0&filter=stdout'
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1"/>
<title>Error 410 Failed to retrieve stdout log for task:
attempt_201305131758_0003_m_000353_0</title>
</head>
<body><h2>HTTP ERROR 410</h2>
<p>Problem accessing /tasklog. Reason:
<pre> Failed to retrieve stdout log for task:
attempt_201305131758_0003_m_000353_0</pre></p><hr /><i><small>Powered by
Jetty://</small></i><br/>
<br/>
<br/>
<br/>
<br/>