You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by Brady Chang <bc...@atipa.com> on 2009/04/24 01:45:39 UTC
ioexception during reduce.
Hello all,
One of my customer's getting the following when increase input size.
same process runs fine with smaller input size.
/tmp has plenty of space,
each data node has 64 Gig memory.
hadoop 0-19.1
Please let me know if you need any additional info.
[root@head attempt_200904181818_0020_r_000002_1]# cat syslog
2009-04-20 00:52:52,898 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
Initializing JVM Metrics with processName=SHUFFLE, sessionId=
2009-04-20 00:52:53,005 INFO org.apache.hadoop.mapred.ReduceTask:
ShuffleRamManager: MemoryLimit=130514944, MaxSingleShuffleLimit=32628736
2009-04-20 00:52:53,013 INFO org.apache.hadoop.mapred.ReduceTask:
attempt_200904181818_0020_r_000002_1 Thread started: Thread for merging
on-disk files
2009-04-20 00:52:53,013 INFO org.apache.hadoop.mapred.ReduceTask:
attempt_200904181818_0020_r_000002_1 Thread started: Thread for merging in
memory files
2009-04-20 00:52:53,013 INFO org.apache.hadoop.mapred.ReduceTask:
attempt_200904181818_0020_r_000002_1 Thread waiting: Thread for merging
on-disk files
2009-04-20 00:52:53,015 INFO org.apache.hadoop.mapred.ReduceTask:
attempt_200904181818_0020_r_000002_1 Need another 168 map output(s) where 0
is already in progress
2009-04-20 00:52:53,042 INFO org.apache.hadoop.mapred.ReduceTask:
attempt_200904181818_0020_r_000002_1: Got 168 new map-outputs
2009-04-20 00:52:53,042 INFO org.apache.hadoop.mapred.ReduceTask:
attempt_200904181818_0020_r_000002_1 Scheduled 8 outputs (0 slow hosts and0
dup hosts)
2009-04-20 00:52:53,068 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling
2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000002_0
2009-04-20 00:52:53,069 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling
2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000006_0
2009-04-20 00:52:53,069 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling
2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000003_0
2009-04-20 00:52:53,069 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling
2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000016_0
2009-04-20 00:52:53,069 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling
2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000012_0
2009-04-20 00:52:53,073 INFO org.apache.hadoop.mapred.ReduceTask: Read 2
bytes from map-output for attempt_200904181818_0020_m_000016_0
2009-04-20 00:52:53,074 INFO org.apache.hadoop.mapred.ReduceTask: Read 2
bytes from map-output for attempt_200904181818_0020_m_000006_0
2009-04-20 00:52:53,074 INFO org.apache.hadoop.mapred.ReduceTask: Read 2
bytes from map-output for attempt_200904181818_0020_m_000012_0
2009-04-20 00:52:53,074 INFO org.apache.hadoop.mapred.ReduceTask: Read 2
bytes from map-output for attempt_200904181818_0020_m_000003_0
2009-04-20 00:52:53,073 INFO org.apache.hadoop.mapred.ReduceTask: Read 2
bytes from map-output for attempt_200904181818_0020_m_000002_0
2009-04-20 00:52:53,074 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1
from attempt_200904181818_0020_m_000012_0 -> (-1, -1) from n2
2009-04-20 00:52:53,074 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1
from attempt_200904181818_0020_m_000016_0 -> (-1, -1) from n8
2009-04-20 00:52:53,074 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1
from attempt_200904181818_0020_m_000006_0 -> (-1, -1) from n4
2009-04-20 00:52:53,074 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1
from attempt_200904181818_0020_m_000002_0 -> (-1, -1) from n5
2009-04-20 00:52:53,074 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1
from attempt_200904181818_0020_m_000003_0 -> (-1, -1) from n7
2009-04-20 00:52:53,078 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling
2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000007_0
2009-04-20 00:52:53,078 INFO org.apache.hadoop.mapred.ReduceTask: Read 2
bytes from map-output for attempt_200904181818_0020_m_000007_0
2009-04-20 00:52:53,078 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1
from attempt_200904181818_0020_m_000007_0 -> (-1, -1) from n3
2009-04-20 00:52:53,078 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling
2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000037_0
2009-04-20 00:52:53,078 INFO org.apache.hadoop.mapred.ReduceTask: Read 2
bytes from map-output for attempt_200904181818_0020_m_000037_0
2009-04-20 00:52:53,079 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1
from attempt_200904181818_0020_m_000037_0 -> (-1, -1) from n1
2009-04-20 00:52:53,079 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling
2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000033_0
2009-04-20 00:52:53,079 INFO org.apache.hadoop.mapred.ReduceTask: Read 2
bytes from map-output for attempt_200904181818_0020_m_000033_0
2009-04-20 00:52:53,079 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1
from attempt_200904181818_0020_m_000033_0 -> (-1, -1) from n6
2009-04-20 00:52:55,021 INFO org.apache.hadoop.mapred.ReduceTask: Ignoring
obsolete output of KILLED map-task: 'attempt_200904181818_0020_m_000166_1'
2009-04-20 00:52:55,021 INFO org.apache.hadoop.mapred.ReduceTask:
attempt_200904181818_0020_r_000002_1: Got 1 obsolete map-outputs from
tasktracker
2009-04-20 00:52:55,022 INFO org.apache.hadoop.mapred.ReduceTask:
attempt_200904181818_0020_r_000002_1 Scheduled 8 outputs (0 slow hosts and0
dup hosts)
2009-04-20 00:52:55,024 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling
2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000005_0
2009-04-20 00:52:55,024 INFO org.apache.hadoop.mapred.ReduceTask: Read 2
bytes from map-output for attempt_200904181818_0020_m_000005_0
2009-04-20 00:52:55,024 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1
from attempt_200904181818_0020_m_000005_0 -> (-1, -1) from n1
2009-04-20 00:52:55,024 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling
2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000014_0
2009-04-20 00:52:55,024 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling
2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000004_0
2009-04-20 00:52:55,024 INFO org.apache.hadoop.mapred.ReduceTask: Read 2
bytes from map-output for attempt_200904181818_0020_m_000014_0
2009-04-20 00:52:55,024 INFO org.apache.hadoop.mapred.ReduceTask: Read 2
bytes from map-output for attempt_200904181818_0020_m_000004_0
2009-04-20 00:52:55,024 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1
from attempt_200904181818_0020_m_000014_0 -> (-1, -1) from n4
2009-04-20 00:52:55,024 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1
from attempt_200904181818_0020_m_000004_0 -> (-1, -1) from n2
2009-04-20 00:52:55,025 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling
2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000019_0
2009-04-20 00:52:55,025 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling
2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000018_0
2009-04-20 00:52:55,025 INFO org.apache.hadoop.mapred.ReduceTask: Read 2
bytes from map-output for attempt_200904181818_0020_m_000019_0
2009-04-20 00:52:55,025 INFO org.apache.hadoop.mapred.ReduceTask: Read 2
bytes from map-output for attempt_200904181818_0020_m_000018_0
2009-04-20 00:52:55,025 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1
from attempt_200904181818_0020_m_000019_0 -> (-1, -1) from n7
2009-04-20 00:52:55,025 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1
from attempt_200904181818_0020_m_000018_0 -> (-1, -1) from n5
2009-04-20 00:52:55,025 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling
2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000008_0
2009-04-20 00:52:55,026 INFO org.apache.hadoop.mapred.ReduceTask: Read 2
bytes from map-output for attempt_200904181818_0020_m_000008_0
2009-04-20 00:52:55,026 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1
from attempt_200904181818_0020_m_000008_0 -> (-1, -1) from n8
2009-04-20 00:52:55,026 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling
2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000015_0
2009-04-20 00:52:55,026 INFO org.apache.hadoop.mapred.ReduceTask: Read 2
bytes from map-output for attempt_200904181818_0020_m_000015_0
2009-04-20 00:52:55,026 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1
from attempt_200904181818_0020_m_000015_0 -> (-1, -1) from n3
2009-04-20 00:52:55,026 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling
2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000001_0
2009-04-20 00:52:55,027 INFO org.apache.hadoop.mapred.ReduceTask: Read 2
bytes from map-output for attempt_200904181818_0020_m_000001_0
2009-04-20 00:52:55,027 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1
from attempt_200904181818_0020_m_000001_0 -> (-1, -1) from n6
2009-04-20 00:52:57,026 INFO org.apache.hadoop.mapred.ReduceTask:
attempt_200904181818_0020_r_000002_1 Scheduled 8 outputs (0 slow hosts and0
dup hosts)
2009-04-20 00:52:57,028 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling
2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000029_0
2009-04-20 00:52:57,028 INFO org.apache.hadoop.mapred.ReduceTask: Read 2
bytes from map-output for attempt_200904181818_0020_m_000029_0
2009-04-20 00:52:57,028 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1
from attempt_200904181818_0020_m_000029_0 -> (-1, -1) from n1
2009-04-20 00:52:57,028 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling
2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000011_0
2009-04-20 00:52:57,028 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling
2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000020_0
2009-04-20 00:52:57,029 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling
2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000026_0
2009-04-20 00:52:57,029 INFO org.apache.hadoop.mapred.ReduceTask: Read 2
bytes from map-output for attempt_200904181818_0020_m_000011_0
2009-04-20 00:52:57,029 INFO org.apache.hadoop.mapred.ReduceTask: Read 2
bytes from map-output for attempt_200904181818_0020_m_000020_0
2009-04-20 00:52:57,029 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1
from attempt_200904181818_0020_m_000011_0 -> (-1, -1) from n7
2009-04-20 00:52:57,029 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1
from attempt_200904181818_0020_m_000020_0 -> (-1, -1) from n2
2009-04-20 00:52:57,029 INFO org.apache.hadoop.mapred.ReduceTask: Read 2
bytes from map-output for attempt_200904181818_0020_m_000026_0
2009-04-20 00:52:57,029 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling
2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000047_0
2009-04-20 00:52:57,029 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1
from attempt_200904181818_0020_m_000026_0 -> (-1, -1) from n5
2009-04-20 00:52:57,029 INFO org.apache.hadoop.mapred.ReduceTask: Read 2
bytes from map-output for attempt_200904181818_0020_m_000047_0
2009-04-20 00:52:57,029 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1
from attempt_200904181818_0020_m_000047_0 -> (-1, -1) from n3
2009-04-20 00:52:57,030 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling
2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000022_0
2009-04-20 00:52:57,030 INFO org.apache.hadoop.mapred.ReduceTask: Read 2
bytes from map-output for attempt_200904181818_0020_m_000022_0
2009-04-20 00:52:57,030 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1
from attempt_200904181818_0020_m_000022_0 -> (-1, -1) from n4
2009-04-20 00:52:57,031 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling
2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000049_0
2009-04-20 00:52:57,031 INFO org.apache.hadoop.mapred.ReduceTask: Read 2
bytes from map-output for attempt_200904181818_0020_m_000049_0
2009-04-20 00:52:57,031 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1
from attempt_200904181818_0020_m_000049_0 -> (-1, -1) from n6
2009-04-20 00:52:57,031 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling
2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000000_0
2009-04-20 00:52:57,031 INFO org.apache.hadoop.mapred.ReduceTask: Read 2
bytes from map-output for attempt_200904181818_0020_m_000000_0
2009-04-20 00:52:57,031 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1
from attempt_200904181818_0020_m_000000_0 -> (-1, -1) from n8
2009-04-20 00:52:59,029 INFO org.apache.hadoop.mapred.ReduceTask:
attempt_200904181818_0020_r_000002_1 Scheduled 8 outputs (0 slow hosts and0
dup hosts)
2009-04-20 00:52:59,031 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling
2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000052_0
2009-04-20 00:52:59,031 INFO org.apache.hadoop.mapred.ReduceTask: Read 2
bytes from map-output for attempt_200904181818_0020_m_000052_0
2009-04-20 00:52:59,031 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling
2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000074_0
2009-04-20 00:52:59,031 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1
from attempt_200904181818_0020_m_000052_0 -> (-1, -1) from n2
2009-04-20 00:52:59,032 INFO org.apache.hadoop.mapred.ReduceTask: Read 2
bytes from map-output for attempt_200904181818_0020_m_000074_0
2009-04-20 00:52:59,032 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1
from attempt_200904181818_0020_m_000074_0 -> (-1, -1) from n5
2009-04-20 00:52:59,032 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling
2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000017_0
2009-04-20 00:52:59,032 INFO org.apache.hadoop.mapred.ReduceTask: Read 2
bytes from map-output for attempt_200904181818_0020_m_000017_0
2009-04-20 00:52:59,032 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1
from attempt_200904181818_0020_m_000017_0 -> (-1, -1) from n6
2009-04-20 00:52:59,031 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling
2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000038_0
2009-04-20 00:52:59,032 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling
2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000056_0
2009-04-20 00:52:59,032 INFO org.apache.hadoop.mapred.ReduceTask: Read 2
bytes from map-output for attempt_200904181818_0020_m_000038_0
2009-04-20 00:52:59,032 INFO org.apache.hadoop.mapred.ReduceTask: Read 2
bytes from map-output for attempt_200904181818_0020_m_000056_0
2009-04-20 00:52:59,033 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1
from attempt_200904181818_0020_m_000038_0 -> (-1, -1) from n4
2009-04-20 00:52:59,033 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1
from attempt_200904181818_0020_m_000056_0 -> (-1, -1) from n8
2009-04-20 00:52:59,033 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling
2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000023_0
2009-04-20 00:52:59,033 INFO org.apache.hadoop.mapred.ReduceTask: Read 2
bytes from map-output for attempt_200904181818_0020_m_000023_0
2009-04-20 00:52:59,033 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1
from attempt_200904181818_0020_m_000023_0 -> (-1, -1) from n3
2009-04-20 00:52:59,033 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling
2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000013_0
2009-04-20 00:52:59,034 INFO org.apache.hadoop.mapred.ReduceTask: Read 2
bytes from map-output for attempt_200904181818_0020_m_000013_0
2009-04-20 00:52:59,034 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1
from attempt_200904181818_0020_m_000013_0 -> (-1, -1) from n1
2009-04-20 00:52:59,034 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling
2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000027_0
2009-04-20 00:52:59,034 INFO org.apache.hadoop.mapred.ReduceTask: Read 2
bytes from map-output for attempt_200904181818_0020_m_000027_0
2009-04-20 00:52:59,034 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1
from attempt_200904181818_0020_m_000027_0 -> (-1, -1) from n7
2009-04-20 00:53:01,033 INFO org.apache.hadoop.mapred.ReduceTask:
attempt_200904181818_0020_r_000002_1 Scheduled 8 outputs (0 slow hosts and0
dup hosts)
2009-04-20 00:53:01,035 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling
2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000055_0
2009-04-20 00:53:01,035 INFO org.apache.hadoop.mapred.ReduceTask: Read 2
bytes from map-output for attempt_200904181818_0020_m_000055_0
2009-04-20 00:53:01,035 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1
from attempt_200904181818_0020_m_000055_0 -> (-1, -1) from n3
2009-04-20 00:53:01,035 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling
2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000025_0
2009-04-20 00:53:01,036 INFO org.apache.hadoop.mapred.ReduceTask: Read 2
bytes from map-output for attempt_200904181818_0020_m_000025_0
2009-04-20 00:53:01,036 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1
from attempt_200904181818_0020_m_000025_0 -> (-1, -1) from n6
2009-04-20 00:53:01,036 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling
2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000035_0
2009-04-20 00:53:01,036 INFO org.apache.hadoop.mapred.ReduceTask: Read 2
bytes from map-output for attempt_200904181818_0020_m_000035_0
2009-04-20 00:53:01,036 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1
from attempt_200904181818_0020_m_000035_0 -> (-1, -1) from n7
2009-04-20 00:53:01,037 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling
2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000021_0
2009-04-20 00:53:01,037 INFO org.apache.hadoop.mapred.ReduceTask: Read 2
bytes from map-output for attempt_200904181818_0020_m_000021_0
2009-04-20 00:53:01,037 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1
from attempt_200904181818_0020_m_000021_0 -> (-1, -1) from n1
2009-04-20 00:53:01,038 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling
2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000034_0
2009-04-20 00:53:01,038 INFO org.apache.hadoop.mapred.ReduceTask: Read 2
bytes from map-output for attempt_200904181818_0020_m_000034_0
2009-04-20 00:53:01,038 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1
from attempt_200904181818_0020_m_000034_0 -> (-1, -1) from n5
2009-04-20 00:53:01,041 WARN org.apache.hadoop.mapred.ReduceTask:
attempt_200904181818_0020_r_000002_1 copy failed:
attempt_200904181818_0020_m_000044_0 from n2
2009-04-20 00:53:01,041 WARN org.apache.hadoop.mapred.ReduceTask:
attempt_200904181818_0020_r_000002_1 copy failed:
attempt_200904181818_0020_m_000054_0 from n4
2009-04-20 00:53:01,041 WARN org.apache.hadoop.mapred.ReduceTask:
attempt_200904181818_0020_r_000002_1 copy failed:
attempt_200904181818_0020_m_000032_0 from n8
2009-04-20 00:53:01,042 WARN org.apache.hadoop.mapred.ReduceTask:
java.io.FileNotFoundException:
http://n2:50060/mapOutput?job=job_200904181818_0020&map=attempt_200904181818
_0020_m_000044_0&reduce=2
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAcces
sorImpl.java:39)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstruc
torAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at
sun.net.www.protocol.http.HttpURLConnection$6.run(HttpURLConnection.java:136
0)
at java.security.AccessController.doPrivileged(Native Method)
at
sun.net.www.protocol.http.HttpURLConnection.getChainedException(HttpURLConne
ction.java:1354)
at
sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection
.java:1008)
at
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.getInputStr
eam(ReduceTask.java:1293)
at
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.getMapOutpu
t(ReduceTask.java:1231)
at
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.copyOutput(
ReduceTask.java:1144)
at
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.run(ReduceT
ask.java:1084)
Caused by: java.io.FileNotFoundException:
http://n2:50060/mapOutput?job=job_200904181818_0020&map=attempt_200904181818
_0020_m_000044_0&reduce=2
at
sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection
.java:1303)
... 4 more
2009-04-20 00:53:01,042 WARN org.apache.hadoop.mapred.ReduceTask:
java.io.FileNotFoundException:
http://n8:50060/mapOutput?job=job_200904181818_0020&map=attempt_200904181818
_0020_m_000032_0&reduce=2
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAcces
sorImpl.java:39)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstruc
torAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at
sun.net.www.protocol.http.HttpURLConnection$6.run(HttpURLConnection.java:136
0)
at java.security.AccessController.doPrivileged(Native Method)
at
sun.net.www.protocol.http.HttpURLConnection.getChainedException(HttpURLConne
ction.java:1354)
at
sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection
.java:1008)
at
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.getInputStr
eam(ReduceTask.java:1293)
at
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.getMapOutpu
t(ReduceTask.java:1231)
at
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.copyOutput(
ReduceTask.java:1144)
at
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.run(ReduceT
ask.java:1084)
Caused by: java.io.FileNotFoundException:
http://n8:50060/mapOutput?job=job_200904181818_0020&map=attempt_200904181818
_0020_m_000032_0&reduce=2
at
sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection
.java:1303)
... 4 more
2009-04-20 00:53:01,042 WARN org.apache.hadoop.mapred.ReduceTask:
java.io.FileNotFoundException:
http://n4:50060/mapOutput?job=job_200904181818_0020&map=attempt_200904181818
_0020_m_000054_0&reduce=2
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAcces
sorImpl.java:39)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstruc
torAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at
sun.net.www.protocol.http.HttpURLConnection$6.run(HttpURLConnection.java:136
0)
at java.security.AccessController.doPrivileged(Native Method)
at
sun.net.www.protocol.http.HttpURLConnection.getChainedException(HttpURLConne
ction.java:1354)
at
sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection
.java:1008)
at
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.getInputStr
eam(ReduceTask.java:1293)
at
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.getMapOutpu
t(ReduceTask.java:1231)
at
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.copyOutput(
ReduceTask.java:1144)
at
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.run(ReduceT
ask.java:1084)
Caused by: java.io.FileNotFoundException:
http://n4:50060/mapOutput?job=job_200904181818_0020&map=attempt_200904181818
_0020_m_000054_0&reduce=2
at
sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection
.java:1303)
... 4 more
Re: ioexception during reduce.
Posted by Jothi Padmanabhan <jo...@yahoo-inc.com>.
Hi Brady,
Could you please check the TaskTracker logs (TTs that ran the maps that are
failing copy) and see what error you are seeing there?
Jothi
On 4/24/09 5:15 AM, "Brady Chang" <bc...@atipa.com> wrote:
> Hello all,
>
> One of my customer's getting the following when increase input size.
>
> same process runs fine with smaller input size.
>
> /tmp has plenty of space,
>
> each data node has 64 Gig memory.
>
> hadoop 0-19.1
>
> Please let me know if you need any additional info.
>
>
>
> [root@head attempt_200904181818_0020_r_000002_1]# cat syslog
>
> 2009-04-20 00:52:52,898 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
> Initializing JVM Metrics with processName=SHUFFLE, sessionId=
>
> 2009-04-20 00:52:53,005 INFO org.apache.hadoop.mapred.ReduceTask:
> ShuffleRamManager: MemoryLimit=130514944, MaxSingleShuffleLimit=32628736
>
> 2009-04-20 00:52:53,013 INFO org.apache.hadoop.mapred.ReduceTask:
> attempt_200904181818_0020_r_000002_1 Thread started: Thread for merging
> on-disk files
>
> 2009-04-20 00:52:53,013 INFO org.apache.hadoop.mapred.ReduceTask:
> attempt_200904181818_0020_r_000002_1 Thread started: Thread for merging in
> memory files
>
> 2009-04-20 00:52:53,013 INFO org.apache.hadoop.mapred.ReduceTask:
> attempt_200904181818_0020_r_000002_1 Thread waiting: Thread for merging
> on-disk files
>
> 2009-04-20 00:52:53,015 INFO org.apache.hadoop.mapred.ReduceTask:
> attempt_200904181818_0020_r_000002_1 Need another 168 map output(s) where 0
> is already in progress
>
> 2009-04-20 00:52:53,042 INFO org.apache.hadoop.mapred.ReduceTask:
> attempt_200904181818_0020_r_000002_1: Got 168 new map-outputs
>
> 2009-04-20 00:52:53,042 INFO org.apache.hadoop.mapred.ReduceTask:
> attempt_200904181818_0020_r_000002_1 Scheduled 8 outputs (0 slow hosts and0
> dup hosts)
>
> 2009-04-20 00:52:53,068 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling
> 2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000002_0
>
> 2009-04-20 00:52:53,069 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling
> 2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000006_0
>
> 2009-04-20 00:52:53,069 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling
> 2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000003_0
>
> 2009-04-20 00:52:53,069 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling
> 2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000016_0
>
> 2009-04-20 00:52:53,069 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling
> 2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000012_0
>
> 2009-04-20 00:52:53,073 INFO org.apache.hadoop.mapred.ReduceTask: Read 2
> bytes from map-output for attempt_200904181818_0020_m_000016_0
>
> 2009-04-20 00:52:53,074 INFO org.apache.hadoop.mapred.ReduceTask: Read 2
> bytes from map-output for attempt_200904181818_0020_m_000006_0
>
> 2009-04-20 00:52:53,074 INFO org.apache.hadoop.mapred.ReduceTask: Read 2
> bytes from map-output for attempt_200904181818_0020_m_000012_0
>
> 2009-04-20 00:52:53,074 INFO org.apache.hadoop.mapred.ReduceTask: Read 2
> bytes from map-output for attempt_200904181818_0020_m_000003_0
>
> 2009-04-20 00:52:53,073 INFO org.apache.hadoop.mapred.ReduceTask: Read 2
> bytes from map-output for attempt_200904181818_0020_m_000002_0
>
> 2009-04-20 00:52:53,074 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1
> from attempt_200904181818_0020_m_000012_0 -> (-1, -1) from n2
>
> 2009-04-20 00:52:53,074 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1
> from attempt_200904181818_0020_m_000016_0 -> (-1, -1) from n8
>
> 2009-04-20 00:52:53,074 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1
> from attempt_200904181818_0020_m_000006_0 -> (-1, -1) from n4
>
> 2009-04-20 00:52:53,074 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1
> from attempt_200904181818_0020_m_000002_0 -> (-1, -1) from n5
>
> 2009-04-20 00:52:53,074 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1
> from attempt_200904181818_0020_m_000003_0 -> (-1, -1) from n7
>
> 2009-04-20 00:52:53,078 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling
> 2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000007_0
>
> 2009-04-20 00:52:53,078 INFO org.apache.hadoop.mapred.ReduceTask: Read 2
> bytes from map-output for attempt_200904181818_0020_m_000007_0
>
> 2009-04-20 00:52:53,078 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1
> from attempt_200904181818_0020_m_000007_0 -> (-1, -1) from n3
>
> 2009-04-20 00:52:53,078 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling
> 2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000037_0
>
> 2009-04-20 00:52:53,078 INFO org.apache.hadoop.mapred.ReduceTask: Read 2
> bytes from map-output for attempt_200904181818_0020_m_000037_0
>
> 2009-04-20 00:52:53,079 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1
> from attempt_200904181818_0020_m_000037_0 -> (-1, -1) from n1
>
> 2009-04-20 00:52:53,079 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling
> 2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000033_0
>
> 2009-04-20 00:52:53,079 INFO org.apache.hadoop.mapred.ReduceTask: Read 2
> bytes from map-output for attempt_200904181818_0020_m_000033_0
>
> 2009-04-20 00:52:53,079 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1
> from attempt_200904181818_0020_m_000033_0 -> (-1, -1) from n6
>
> 2009-04-20 00:52:55,021 INFO org.apache.hadoop.mapred.ReduceTask: Ignoring
> obsolete output of KILLED map-task: 'attempt_200904181818_0020_m_000166_1'
>
> 2009-04-20 00:52:55,021 INFO org.apache.hadoop.mapred.ReduceTask:
> attempt_200904181818_0020_r_000002_1: Got 1 obsolete map-outputs from
> tasktracker
>
> 2009-04-20 00:52:55,022 INFO org.apache.hadoop.mapred.ReduceTask:
> attempt_200904181818_0020_r_000002_1 Scheduled 8 outputs (0 slow hosts and0
> dup hosts)
>
> 2009-04-20 00:52:55,024 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling
> 2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000005_0
>
> 2009-04-20 00:52:55,024 INFO org.apache.hadoop.mapred.ReduceTask: Read 2
> bytes from map-output for attempt_200904181818_0020_m_000005_0
>
> 2009-04-20 00:52:55,024 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1
> from attempt_200904181818_0020_m_000005_0 -> (-1, -1) from n1
>
> 2009-04-20 00:52:55,024 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling
> 2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000014_0
>
> 2009-04-20 00:52:55,024 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling
> 2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000004_0
>
> 2009-04-20 00:52:55,024 INFO org.apache.hadoop.mapred.ReduceTask: Read 2
> bytes from map-output for attempt_200904181818_0020_m_000014_0
>
> 2009-04-20 00:52:55,024 INFO org.apache.hadoop.mapred.ReduceTask: Read 2
> bytes from map-output for attempt_200904181818_0020_m_000004_0
>
> 2009-04-20 00:52:55,024 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1
> from attempt_200904181818_0020_m_000014_0 -> (-1, -1) from n4
>
> 2009-04-20 00:52:55,024 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1
> from attempt_200904181818_0020_m_000004_0 -> (-1, -1) from n2
>
> 2009-04-20 00:52:55,025 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling
> 2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000019_0
>
> 2009-04-20 00:52:55,025 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling
> 2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000018_0
>
> 2009-04-20 00:52:55,025 INFO org.apache.hadoop.mapred.ReduceTask: Read 2
> bytes from map-output for attempt_200904181818_0020_m_000019_0
>
> 2009-04-20 00:52:55,025 INFO org.apache.hadoop.mapred.ReduceTask: Read 2
> bytes from map-output for attempt_200904181818_0020_m_000018_0
>
> 2009-04-20 00:52:55,025 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1
> from attempt_200904181818_0020_m_000019_0 -> (-1, -1) from n7
>
> 2009-04-20 00:52:55,025 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1
> from attempt_200904181818_0020_m_000018_0 -> (-1, -1) from n5
>
> 2009-04-20 00:52:55,025 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling
> 2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000008_0
>
> 2009-04-20 00:52:55,026 INFO org.apache.hadoop.mapred.ReduceTask: Read 2
> bytes from map-output for attempt_200904181818_0020_m_000008_0
>
> 2009-04-20 00:52:55,026 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1
> from attempt_200904181818_0020_m_000008_0 -> (-1, -1) from n8
>
> 2009-04-20 00:52:55,026 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling
> 2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000015_0
>
> 2009-04-20 00:52:55,026 INFO org.apache.hadoop.mapred.ReduceTask: Read 2
> bytes from map-output for attempt_200904181818_0020_m_000015_0
>
> 2009-04-20 00:52:55,026 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1
> from attempt_200904181818_0020_m_000015_0 -> (-1, -1) from n3
>
> 2009-04-20 00:52:55,026 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling
> 2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000001_0
>
> 2009-04-20 00:52:55,027 INFO org.apache.hadoop.mapred.ReduceTask: Read 2
> bytes from map-output for attempt_200904181818_0020_m_000001_0
>
> 2009-04-20 00:52:55,027 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1
> from attempt_200904181818_0020_m_000001_0 -> (-1, -1) from n6
>
> 2009-04-20 00:52:57,026 INFO org.apache.hadoop.mapred.ReduceTask:
> attempt_200904181818_0020_r_000002_1 Scheduled 8 outputs (0 slow hosts and0
> dup hosts)
>
> 2009-04-20 00:52:57,028 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling
> 2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000029_0
>
> 2009-04-20 00:52:57,028 INFO org.apache.hadoop.mapred.ReduceTask: Read 2
> bytes from map-output for attempt_200904181818_0020_m_000029_0
>
> 2009-04-20 00:52:57,028 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1
> from attempt_200904181818_0020_m_000029_0 -> (-1, -1) from n1
>
> 2009-04-20 00:52:57,028 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling
> 2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000011_0
>
> 2009-04-20 00:52:57,028 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling
> 2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000020_0
>
> 2009-04-20 00:52:57,029 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling
> 2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000026_0
>
> 2009-04-20 00:52:57,029 INFO org.apache.hadoop.mapred.ReduceTask: Read 2
> bytes from map-output for attempt_200904181818_0020_m_000011_0
>
> 2009-04-20 00:52:57,029 INFO org.apache.hadoop.mapred.ReduceTask: Read 2
> bytes from map-output for attempt_200904181818_0020_m_000020_0
>
> 2009-04-20 00:52:57,029 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1
> from attempt_200904181818_0020_m_000011_0 -> (-1, -1) from n7
>
> 2009-04-20 00:52:57,029 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1
> from attempt_200904181818_0020_m_000020_0 -> (-1, -1) from n2
>
> 2009-04-20 00:52:57,029 INFO org.apache.hadoop.mapred.ReduceTask: Read 2
> bytes from map-output for attempt_200904181818_0020_m_000026_0
>
> 2009-04-20 00:52:57,029 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling
> 2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000047_0
>
> 2009-04-20 00:52:57,029 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1
> from attempt_200904181818_0020_m_000026_0 -> (-1, -1) from n5
>
> 2009-04-20 00:52:57,029 INFO org.apache.hadoop.mapred.ReduceTask: Read 2
> bytes from map-output for attempt_200904181818_0020_m_000047_0
>
> 2009-04-20 00:52:57,029 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1
> from attempt_200904181818_0020_m_000047_0 -> (-1, -1) from n3
>
> 2009-04-20 00:52:57,030 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling
> 2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000022_0
>
> 2009-04-20 00:52:57,030 INFO org.apache.hadoop.mapred.ReduceTask: Read 2
> bytes from map-output for attempt_200904181818_0020_m_000022_0
>
> 2009-04-20 00:52:57,030 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1
> from attempt_200904181818_0020_m_000022_0 -> (-1, -1) from n4
>
> 2009-04-20 00:52:57,031 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling
> 2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000049_0
>
> 2009-04-20 00:52:57,031 INFO org.apache.hadoop.mapred.ReduceTask: Read 2
> bytes from map-output for attempt_200904181818_0020_m_000049_0
>
> 2009-04-20 00:52:57,031 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1
> from attempt_200904181818_0020_m_000049_0 -> (-1, -1) from n6
>
> 2009-04-20 00:52:57,031 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling
> 2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000000_0
>
> 2009-04-20 00:52:57,031 INFO org.apache.hadoop.mapred.ReduceTask: Read 2
> bytes from map-output for attempt_200904181818_0020_m_000000_0
>
> 2009-04-20 00:52:57,031 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1
> from attempt_200904181818_0020_m_000000_0 -> (-1, -1) from n8
>
> 2009-04-20 00:52:59,029 INFO org.apache.hadoop.mapred.ReduceTask:
> attempt_200904181818_0020_r_000002_1 Scheduled 8 outputs (0 slow hosts and0
> dup hosts)
>
> 2009-04-20 00:52:59,031 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling
> 2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000052_0
>
> 2009-04-20 00:52:59,031 INFO org.apache.hadoop.mapred.ReduceTask: Read 2
> bytes from map-output for attempt_200904181818_0020_m_000052_0
>
> 2009-04-20 00:52:59,031 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling
> 2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000074_0
>
> 2009-04-20 00:52:59,031 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1
> from attempt_200904181818_0020_m_000052_0 -> (-1, -1) from n2
>
> 2009-04-20 00:52:59,032 INFO org.apache.hadoop.mapred.ReduceTask: Read 2
> bytes from map-output for attempt_200904181818_0020_m_000074_0
>
> 2009-04-20 00:52:59,032 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1
> from attempt_200904181818_0020_m_000074_0 -> (-1, -1) from n5
>
> 2009-04-20 00:52:59,032 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling
> 2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000017_0
>
> 2009-04-20 00:52:59,032 INFO org.apache.hadoop.mapred.ReduceTask: Read 2
> bytes from map-output for attempt_200904181818_0020_m_000017_0
>
> 2009-04-20 00:52:59,032 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1
> from attempt_200904181818_0020_m_000017_0 -> (-1, -1) from n6
>
> 2009-04-20 00:52:59,031 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling
> 2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000038_0
>
> 2009-04-20 00:52:59,032 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling
> 2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000056_0
>
> 2009-04-20 00:52:59,032 INFO org.apache.hadoop.mapred.ReduceTask: Read 2
> bytes from map-output for attempt_200904181818_0020_m_000038_0
>
> 2009-04-20 00:52:59,032 INFO org.apache.hadoop.mapred.ReduceTask: Read 2
> bytes from map-output for attempt_200904181818_0020_m_000056_0
>
> 2009-04-20 00:52:59,033 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1
> from attempt_200904181818_0020_m_000038_0 -> (-1, -1) from n4
>
> 2009-04-20 00:52:59,033 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1
> from attempt_200904181818_0020_m_000056_0 -> (-1, -1) from n8
>
> 2009-04-20 00:52:59,033 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling
> 2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000023_0
>
> 2009-04-20 00:52:59,033 INFO org.apache.hadoop.mapred.ReduceTask: Read 2
> bytes from map-output for attempt_200904181818_0020_m_000023_0
>
> 2009-04-20 00:52:59,033 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1
> from attempt_200904181818_0020_m_000023_0 -> (-1, -1) from n3
>
> 2009-04-20 00:52:59,033 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling
> 2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000013_0
>
> 2009-04-20 00:52:59,034 INFO org.apache.hadoop.mapred.ReduceTask: Read 2
> bytes from map-output for attempt_200904181818_0020_m_000013_0
>
> 2009-04-20 00:52:59,034 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1
> from attempt_200904181818_0020_m_000013_0 -> (-1, -1) from n1
>
> 2009-04-20 00:52:59,034 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling
> 2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000027_0
>
> 2009-04-20 00:52:59,034 INFO org.apache.hadoop.mapred.ReduceTask: Read 2
> bytes from map-output for attempt_200904181818_0020_m_000027_0
>
> 2009-04-20 00:52:59,034 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1
> from attempt_200904181818_0020_m_000027_0 -> (-1, -1) from n7
>
> 2009-04-20 00:53:01,033 INFO org.apache.hadoop.mapred.ReduceTask:
> attempt_200904181818_0020_r_000002_1 Scheduled 8 outputs (0 slow hosts and0
> dup hosts)
>
> 2009-04-20 00:53:01,035 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling
> 2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000055_0
>
> 2009-04-20 00:53:01,035 INFO org.apache.hadoop.mapred.ReduceTask: Read 2
> bytes from map-output for attempt_200904181818_0020_m_000055_0
>
> 2009-04-20 00:53:01,035 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1
> from attempt_200904181818_0020_m_000055_0 -> (-1, -1) from n3
>
> 2009-04-20 00:53:01,035 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling
> 2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000025_0
>
> 2009-04-20 00:53:01,036 INFO org.apache.hadoop.mapred.ReduceTask: Read 2
> bytes from map-output for attempt_200904181818_0020_m_000025_0
>
> 2009-04-20 00:53:01,036 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1
> from attempt_200904181818_0020_m_000025_0 -> (-1, -1) from n6
>
> 2009-04-20 00:53:01,036 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling
> 2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000035_0
>
> 2009-04-20 00:53:01,036 INFO org.apache.hadoop.mapred.ReduceTask: Read 2
> bytes from map-output for attempt_200904181818_0020_m_000035_0
>
> 2009-04-20 00:53:01,036 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1
> from attempt_200904181818_0020_m_000035_0 -> (-1, -1) from n7
>
> 2009-04-20 00:53:01,037 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling
> 2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000021_0
>
> 2009-04-20 00:53:01,037 INFO org.apache.hadoop.mapred.ReduceTask: Read 2
> bytes from map-output for attempt_200904181818_0020_m_000021_0
>
> 2009-04-20 00:53:01,037 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1
> from attempt_200904181818_0020_m_000021_0 -> (-1, -1) from n1
>
> 2009-04-20 00:53:01,038 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling
> 2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000034_0
>
> 2009-04-20 00:53:01,038 INFO org.apache.hadoop.mapred.ReduceTask: Read 2
> bytes from map-output for attempt_200904181818_0020_m_000034_0
>
> 2009-04-20 00:53:01,038 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1
> from attempt_200904181818_0020_m_000034_0 -> (-1, -1) from n5
>
> 2009-04-20 00:53:01,041 WARN org.apache.hadoop.mapred.ReduceTask:
> attempt_200904181818_0020_r_000002_1 copy failed:
> attempt_200904181818_0020_m_000044_0 from n2
>
> 2009-04-20 00:53:01,041 WARN org.apache.hadoop.mapred.ReduceTask:
> attempt_200904181818_0020_r_000002_1 copy failed:
> attempt_200904181818_0020_m_000054_0 from n4
>
> 2009-04-20 00:53:01,041 WARN org.apache.hadoop.mapred.ReduceTask:
> attempt_200904181818_0020_r_000002_1 copy failed:
> attempt_200904181818_0020_m_000032_0 from n8
>
> 2009-04-20 00:53:01,042 WARN org.apache.hadoop.mapred.ReduceTask:
> java.io.FileNotFoundException:
> http://n2:50060/mapOutput?job=job_200904181818_0020&map=attempt_200904181818
> _0020_m_000044_0&reduce=2
>
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
>
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAcces
> sorImpl.java:39)
>
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstruc
> torAccessorImpl.java:27)
>
> at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
>
> at
> sun.net.www.protocol.http.HttpURLConnection$6.run(HttpURLConnection.java:136
> 0)
>
> at java.security.AccessController.doPrivileged(Native Method)
>
> at
> sun.net.www.protocol.http.HttpURLConnection.getChainedException(HttpURLConne
> ction.java:1354)
>
> at
> sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection
> .java:1008)
>
> at
> org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.getInputStr
> eam(ReduceTask.java:1293)
>
> at
> org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.getMapOutpu
> t(ReduceTask.java:1231)
>
> at
> org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.copyOutput(
> ReduceTask.java:1144)
>
> at
> org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.run(ReduceT
> ask.java:1084)
>
> Caused by: java.io.FileNotFoundException:
> http://n2:50060/mapOutput?job=job_200904181818_0020&map=attempt_200904181818
> _0020_m_000044_0&reduce=2
>
> at
> sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection
> .java:1303)
>
> ... 4 more
>
>
>
> 2009-04-20 00:53:01,042 WARN org.apache.hadoop.mapred.ReduceTask:
> java.io.FileNotFoundException:
> http://n8:50060/mapOutput?job=job_200904181818_0020&map=attempt_200904181818
> _0020_m_000032_0&reduce=2
>
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
>
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAcces
> sorImpl.java:39)
>
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstruc
> torAccessorImpl.java:27)
>
> at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
>
> at
> sun.net.www.protocol.http.HttpURLConnection$6.run(HttpURLConnection.java:136
> 0)
>
> at java.security.AccessController.doPrivileged(Native Method)
>
> at
> sun.net.www.protocol.http.HttpURLConnection.getChainedException(HttpURLConne
> ction.java:1354)
>
> at
> sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection
> .java:1008)
>
> at
> org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.getInputStr
> eam(ReduceTask.java:1293)
>
> at
> org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.getMapOutpu
> t(ReduceTask.java:1231)
>
> at
> org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.copyOutput(
> ReduceTask.java:1144)
>
> at
> org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.run(ReduceT
> ask.java:1084)
>
> Caused by: java.io.FileNotFoundException:
> http://n8:50060/mapOutput?job=job_200904181818_0020&map=attempt_200904181818
> _0020_m_000032_0&reduce=2
>
> at
> sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection
> .java:1303)
>
> ... 4 more
>
>
>
> 2009-04-20 00:53:01,042 WARN org.apache.hadoop.mapred.ReduceTask:
> java.io.FileNotFoundException:
> http://n4:50060/mapOutput?job=job_200904181818_0020&map=attempt_200904181818
> _0020_m_000054_0&reduce=2
>
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
>
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAcces
> sorImpl.java:39)
>
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstruc
> torAccessorImpl.java:27)
>
> at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
>
> at
> sun.net.www.protocol.http.HttpURLConnection$6.run(HttpURLConnection.java:136
> 0)
>
> at java.security.AccessController.doPrivileged(Native Method)
>
> at
> sun.net.www.protocol.http.HttpURLConnection.getChainedException(HttpURLConne
> ction.java:1354)
>
> at
> sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection
> .java:1008)
>
> at
> org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.getInputStr
> eam(ReduceTask.java:1293)
>
> at
> org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.getMapOutpu
> t(ReduceTask.java:1231)
>
> at
> org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.copyOutput(
> ReduceTask.java:1144)
>
> at
> org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.run(ReduceT
> ask.java:1084)
>
> Caused by: java.io.FileNotFoundException:
> http://n4:50060/mapOutput?job=job_200904181818_0020&map=attempt_200904181818
> _0020_m_000054_0&reduce=2
>
> at
> sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection
> .java:1303)
>
> ... 4 more
>