You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Patrick Angeles <pa...@gmail.com> on 2009/10/15 16:24:17 UTC
normal hadoop errors?
I got the following error while running the example sort program (hadoop
0.20) on a brand new Hadoop cluster (using the Cloudera distro). The job
seems to have recovered. However I'm wondering if this is normal or should I
be checking for something.
attempt_200910051513_0009_r_000005_0: 09/10/15 09:53:52 INFO hdfs.DFSClient:
Exception in createBlockOutputStream java.io.IOException: Bad connect ack
with firstBadLink 10.10.10.52:50010
attempt_200910051513_0009_r_000005_0: 09/10/15 09:53:52 INFO hdfs.DFSClient:
Abandoning block blk_-7778196938228518311_13172
attempt_200910051513_0009_r_000005_0: 09/10/15 09:53:52 INFO hdfs.DFSClient:
Waiting to find target node: 10.10.10.56:50010
attempt_200910051513_0009_r_000005_0: 09/10/15 09:54:01 INFO hdfs.DFSClient:
Exception in createBlockOutputStream java.io.IOException: Bad connect ack
with firstBadLink 10.10.10.55:50010
attempt_200910051513_0009_r_000005_0: 09/10/15 09:54:01 INFO hdfs.DFSClient:
Abandoning block blk_-7309503129247220072_13172
attempt_200910051513_0009_r_000005_0: 09/10/15 09:54:09 INFO hdfs.DFSClient:
Exception in createBlockOutputStream java.io.IOException: Bad connect ack
with firstBadLink 10.10.10.52:50010
attempt_200910051513_0009_r_000005_0: 09/10/15 09:54:09 INFO hdfs.DFSClient:
Abandoning block blk_3948102363851753370_13172
attempt_200910051513_0009_r_000005_0: 09/10/15 09:54:15 INFO hdfs.DFSClient:
Exception in createBlockOutputStream java.io.IOException: Bad connect ack
with firstBadLink 10.10.10.55:50010
attempt_200910051513_0009_r_000005_0: 09/10/15 09:54:15 INFO hdfs.DFSClient:
Abandoning block blk_-9105283762069697302_13172
attempt_200910051513_0009_r_000005_0: 09/10/15 09:54:21 WARN hdfs.DFSClient:
DataStreamer Exception: java.io.IOException: Unable to create new block.
attempt_200910051513_0009_r_000005_0: at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2813)
attempt_200910051513_0009_r_000005_0: at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:2077)
attempt_200910051513_0009_r_000005_0: at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2263)
attempt_200910051513_0009_r_000005_0: 09/10/15 09:54:21 WARN hdfs.DFSClient:
Error Recovery for block blk_-9105283762069697302_13172 bad datanode[1]
nodes == null
attempt_200910051513_0009_r_000005_0: 09/10/15 09:54:21 WARN hdfs.DFSClient:
Could not get block locations. Source file
"/user/hadoop/sort_out/_temporary/_attempt_200910051513_0009_r_000005_0/part-00005"
- Aborting...
attempt_200910051513_0009_r_000005_0: 09/10/15 09:54:21 WARN
mapred.TaskTracker: Error running child
attempt_200910051513_0009_r_000005_0: java.io.IOException: Bad connect ack
with firstBadLink 10.10.10.55:50010
attempt_200910051513_0009_r_000005_0: at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.createBlockOutputStream(DFSClient.java:2871)
attempt_200910051513_0009_r_000005_0: at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2794)
attempt_200910051513_0009_r_000005_0: at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:2077)
attempt_200910051513_0009_r_000005_0: at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2263)
attempt_200910051513_0009_r_000005_0: 09/10/15 09:54:21 INFO
mapred.TaskRunner: Runnning cleanup for the task
09/10/15 09:54:37 INFO mapred.JobClient: map 100% reduce 82%
09/10/15 09:54:41 INFO mapred.JobClient: map 100% reduce 83%
09/10/15 09:54:43 INFO mapred.JobClient: Task Id :
attempt_200910051513_0009_r_000017_0, Status : FAILED
java.io.IOException: Bad connect ack with firstBadLink 10.10.10.56:50010
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.createBlockOutputStream(DFSClient.java:2871)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2794)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:2077)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2263)
attempt_200910051513_0009_r_000017_0: 09/10/15 09:47:48 WARN
conf.Configuration:
/h3/hadoop/mapred/hadoop/taskTracker/jobcache/job_200910051513_0009/attempt_200910051513_0009_r_000017_0/job.xml:a
attempt to override final parameter: hadoop.tmp.dir; Ignoring.
attempt_200910051513_0009_r_000017_0: 09/10/15 09:47:48 WARN
conf.Configuration:
/h3/hadoop/mapred/hadoop/taskTracker/jobcache/job_200910051513_0009/attempt_200910051513_0009_r_000017_0/job.xml:a
attempt to override final parameter:
hadoop.rpc.socket.factory.class.default; Ignoring.
attempt_200910051513_0009_r_000017_0: 09/10/15 09:47:48 WARN
conf.Configuration:
/h3/hadoop/mapred/hadoop/taskTracker/jobcache/job_200910051513_0009/attempt_200910051513_0009_r_000017_0/job.xml:a
attempt to override final parameter: tasktracker.http.threads; Ignoring.
attempt_200910051513_0009_r_000017_0: 09/10/15 09:47:48 WARN
conf.Configuration:
/h3/hadoop/mapred/hadoop/taskTracker/jobcache/job_200910051513_0009/attempt_200910051513_0009_r_000017_0/job.xml:a
attempt to override final parameter: mapred.child.ulimit; Ignoring.
attempt_200910051513_0009_r_000017_0: 09/10/15 09:47:48 WARN
conf.Configuration:
/h3/hadoop/mapred/hadoop/taskTracker/jobcache/job_200910051513_0009/attempt_200910051513_0009_r_000017_0/job.xml:a
attempt to override final parameter: fs.checkpoint.dir; Ignoring.
attempt_200910051513_0009_r_000017_0: 09/10/15 09:47:48 WARN
conf.Configuration:
/h3/hadoop/mapred/hadoop/taskTracker/jobcache/job_200910051513_0009/attempt_200910051513_0009_r_000017_0/job.xml:a
attempt to override final parameter: mapred.job.tracker.handler.count;
Ignoring.
attempt_200910051513_0009_r_000017_0: 09/10/15 09:47:48 WARN
conf.Configuration:
/h3/hadoop/mapred/hadoop/taskTracker/jobcache/job_200910051513_0009/attempt_200910051513_0009_r_000017_0/job.xml:a
attempt to override final parameter: mapred.local.dir; Ignoring.
attempt_200910051513_0009_r_000017_0: 09/10/15 09:47:48 WARN
conf.Configuration:
/h3/hadoop/mapred/hadoop/taskTracker/jobcache/job_200910051513_0009/attempt_200910051513_0009_r_000017_0/job.xml:a
attempt to override final parameter: fs.trash.interval; Ignoring.
attempt_200910051513_0009_r_000017_0: 09/10/15 09:47:48 WARN
conf.Configuration:
/h3/hadoop/mapred/hadoop/taskTracker/jobcache/job_200910051513_0009/attempt_200910051513_0009_r_000017_0/job.xml:a
attempt to override final parameter:
mapred.tasktracker.reduce.tasks.maximum; Ignoring.
attempt_200910051513_0009_r_000017_0: 09/10/15 09:47:48 WARN
conf.Configuration:
/h3/hadoop/mapred/hadoop/taskTracker/jobcache/job_200910051513_0009/attempt_200910051513_0009_r_000017_0/job.xml:a
attempt to override final parameter: mapred.tasktracker.map.tasks.maximum;
Ignoring.
attempt_200910051513_0009_r_000017_0: 09/10/15 09:47:48 INFO jvm.JvmMetrics:
Initializing JVM Metrics with processName=SHUFFLE, sessionId=
attempt_200910051513_0009_r_000017_0: 09/10/15 09:47:48 WARN
conf.Configuration:
/h3/hadoop/mapred/hadoop/taskTracker/jobcache/job_200910051513_0009/attempt_200910051513_0009_r_000017_0/job.xml:a
attempt to override final parameter: hadoop.tmp.dir; Ignoring.
attempt_200910051513_0009_r_000017_0: 09/10/15 09:47:48 WARN
conf.Configuration:
/h3/hadoop/mapred/hadoop/taskTracker/jobcache/job_200910051513_0009/attempt_200910051513_0009_r_000017_0/job.xml:a
attempt to override final parameter: dfs.namenode.handler.count; Ignoring.
attempt_200910051513_0009_r_000017_0: 09/10/15 09:47:48 WARN
conf.Configuration:
/h3/hadoop/mapred/hadoop/taskTracker/jobcache/job_200910051513_0009/attempt_200910051513_0009_r_000017_0/job.xml:a
attempt to override final parameter:
hadoop.rpc.socket.factory.class.default; Ignoring.
attempt_200910051513_0009_r_000017_0: 09/10/15 09:47:48 WARN
conf.Configuration:
/h3/hadoop/mapred/hadoop/taskTracker/jobcache/job_200910051513_0009/attempt_200910051513_0009_r_000017_0/job.xml:a
attempt to override final parameter: tasktracker.http.threads; Ignoring.
attempt_200910051513_0009_r_000017_0: 09/10/15 09:47:48 WARN
conf.Configuration:
/h3/hadoop/mapred/hadoop/taskTracker/jobcache/job_200910051513_0009/attempt_200910051513_0009_r_000017_0/job.xml:a
attempt to override final parameter: dfs.block.size; Ignoring.
attempt_200910051513_0009_r_000017_0: 09/10/15 09:47:48 WARN
conf.Configuration:
/h3/hadoop/mapred/hadoop/taskTracker/jobcache/job_200910051513_0009/attempt_200910051513_0009_r_000017_0/job.xml:a
attempt to override final parameter: dfs.permissions; Ignoring.
attempt_200910051513_0009_r_000017_0: 09/10/15 09:47:48 WARN
conf.Configuration:
/h3/hadoop/mapred/hadoop/taskTracker/jobcache/job_200910051513_0009/attempt_200910051513_0009_r_000017_0/job.xml:a
attempt to override final parameter: mapred.child.ulimit; Ignoring.
attempt_200910051513_0009_r_000017_0: 09/10/15 09:47:48 WARN
conf.Configuration:
/h3/hadoop/mapred/hadoop/taskTracker/jobcache/job_200910051513_0009/attempt_200910051513_0009_r_000017_0/job.xml:a
attempt to override final parameter: fs.checkpoint.dir; Ignoring.
attempt_200910051513_0009_r_000017_0: 09/10/15 09:47:48 WARN
conf.Configuration:
/h3/hadoop/mapred/hadoop/taskTracker/jobcache/job_200910051513_0009/attempt_200910051513_0009_r_000017_0/job.xml:a
attempt to override final parameter: mapred.job.tracker.handler.count;
Ignoring.
attempt_200910051513_0009_r_000017_0: 09/10/15 09:47:48 WARN
conf.Configuration:
/h3/hadoop/mapred/hadoop/taskTracker/jobcache/job_200910051513_0009/attempt_200910051513_0009_r_000017_0/job.xml:a
attempt to override final parameter: dfs.datanode.du.reserved; Ignoring.
attempt_200910051513_0009_r_000017_0: 09/10/15 09:47:48 WARN
conf.Configuration:
/h3/hadoop/mapred/hadoop/taskTracker/jobcache/job_200910051513_0009/attempt_200910051513_0009_r_000017_0/job.xml:a
attempt to override final parameter: dfs.data.dir; Ignoring.
attempt_200910051513_0009_r_000017_0: 09/10/15 09:47:48 WARN
conf.Configuration:
/h3/hadoop/mapred/hadoop/taskTracker/jobcache/job_200910051513_0009/attempt_200910051513_0009_r_000017_0/job.xml:a
attempt to override final parameter: mapred.local.dir; Ignoring.
attempt_200910051513_0009_r_000017_0: 09/10/15 09:47:48 WARN
conf.Configuration:
/h3/hadoop/mapred/hadoop/taskTracker/jobcache/job_200910051513_0009/attempt_200910051513_0009_r_000017_0/job.xml:a
attempt to override final parameter: fs.trash.interval; Ignoring.
attempt_200910051513_0009_r_000017_0: 09/10/15 09:47:48 WARN
conf.Configuration:
/h3/hadoop/mapred/hadoop/taskTracker/jobcache/job_200910051513_0009/attempt_200910051513_0009_r_000017_0/job.xml:a
attempt to override final parameter: dfs.name.dir; Ignoring.
attempt_200910051513_0009_r_000017_0: 09/10/15 09:47:48 WARN
conf.Configuration:
/h3/hadoop/mapred/hadoop/taskTracker/jobcache/job_200910051513_0009/attempt_200910051513_0009_r_000017_0/job.xml:a
attempt to override final parameter:
mapred.tasktracker.reduce.tasks.maximum; Ignoring.
attempt_200910051513_0009_r_000017_0: 09/10/15 09:47:48 WARN
conf.Configuration:
/h3/hadoop/mapred/hadoop/taskTracker/jobcache/job_200910051513_0009/attempt_200910051513_0009_r_000017_0/job.xml:a
attempt to override final parameter: dfs.datanode.handler.count; Ignoring.
attempt_200910051513_0009_r_000017_0: 09/10/15 09:47:48 WARN
conf.Configuration:
/h3/hadoop/mapred/hadoop/taskTracker/jobcache/job_200910051513_0009/attempt_200910051513_0009_r_000017_0/job.xml:a
attempt to override final parameter: mapred.tasktracker.map.tasks.maximum;
Ignoring.
attempt_200910051513_0009_r_000017_0: 09/10/15 09:47:48 INFO
mapred.ReduceTask: ShuffleRamManager: MemoryLimit=762078784,
MaxSingleShuffleLimit=190519696
attempt_200910051513_0009_r_000017_0: 09/10/15 09:47:48 INFO
mapred.ReduceTask: attempt_200910051513_0009_r_000017_0 Thread started:
Thread for merging on-disk files
attempt_200910051513_0009_r_000017_0: 09/10/15 09:47:48 INFO
mapred.ReduceTask: attempt_200910051513_0009_r_000017_0 Thread waiting:
Thread for merging on-disk files
attempt_200910051513_0009_r_000017_0: 09/10/15 09:47:48 INFO
mapred.ReduceTask: attempt_200910051513_0009_r_000017_0 Thread started:
Thread for merging in memory files
attempt_200910051513_0009_r_000017_0: 09/10/15 09:47:48 INFO
mapred.ReduceTask: attempt_200910051513_0009_r_000017_0 Need another 560 map
output(s) where 0 is already in progress
attempt_200910051513_0009_r_000017_0: 09/10/15 09:47:48 INFO
mapred.ReduceTask: attempt_200910051513_0009_r_000017_0 Thread started:
Thread for polling Map Completion Events
attempt_200910051513_0009_r_000017_0: 09/10/15 09:47:48 INFO
mapred.ReduceTask: attempt_200910051513_0009_r_000017_0 Scheduled 0 outputs
(0 slow hosts and0 dup hosts)
attempt_200910051513_0009_r_000017_0: 09/10/15 09:47:48 INFO
mapred.ReduceTask: attempt_200910051513_0009_r_000017_0: Got 79 new
map-outputs
attempt_200910051513_0009_r_000017_0: 09/10/15 09:47:51 INFO
mapred.ReduceTask: attempt_200910051513_0009_r_000017_0: Got 6 new
map-outputs