You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Adarsh Sharma <ad...@orkash.com> on 2011/04/04 07:35:39 UTC
Running wordcount-nopipe program

Dear all,

Does anyone run the *wordcount-nopipe *program * *in C++ in Hadoop 
Cluster & encountered the below error :-


bash-3.2$ bin/hadoop pipes -D hadoop.pipes.java.recordreader=true -D 
hadoop.pipes.java.recordwriter=true -input gutenberg -output 
gutenberg-out1 -program bin/wordcount-nopipe
11/04/04 10:59:13 WARN mapred.JobClient: No job jar file set.  User 
classes may not be found. See JobConf(Class) or JobConf#setJar(String).
11/04/04 10:59:13 INFO mapred.FileInputFormat: Total input paths to 
process : 3
11/04/04 10:59:13 INFO mapred.JobClient: Running job: job_201104041057_0002
11/04/04 10:59:14 INFO mapred.JobClient:  map 0% reduce 0%
11/04/04 10:59:24 INFO mapred.JobClient: Task Id : 
attempt_201104041057_0002_m_000000_0, Status : FAILED
java.io.IOException: pipe child exception
        at 
org.apache.hadoop.mapred.pipes.Application.abort(Application.java:151)
        at 
org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java:101)
        at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:358)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
        at org.apache.hadoop.mapred.Child.main(Child.java:170)
Caused by: java.net.SocketException: Broken pipe
        at java.net.SocketOutputStream.socketWrite0(Native Method)
        at 
java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
        at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
        at 
java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
        at java.io.BufferedOutputStream.write(BufferedOutputStream.java:109)
        at java.io.DataOutputStream.write(DataOutputStream.java:90)
        at 
org.apache.hadoop.mapred.pipes.BinaryProtocol.writeObject(BinaryProtocol.java:333)
        at 
org.apache.hadoop.mapred.pipes.BinaryProtocol.mapItem(BinaryProtocol.java:286)
        at 
org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java:90)
        ... 3 more

11/04/04 10:59:27 WARN mapred.JobClient: Error reading task outputNo 
route to host
11/04/04 10:59:30 WARN mapred.JobClient: Error reading task outputNo 
route to host
11/04/04 10:59:30 INFO mapred.JobClient: Task Id : 
attempt_201104041057_0002_m_000001_0, Status : FAILED
java.io.IOException: pipe child exception
        at 
org.apache.hadoop.mapred.pipes.Application.abort(Application.java:151)
        at 
org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java:101)
        at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:358)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
        at org.apache.hadoop.mapred.Child.main(Child.java:170)
Caused by: java.net.SocketException: Broken pipe
        at java.net.SocketOutputStream.socketWrite0(Native Method)
        at 
java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
        at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
        at 
java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
        at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
        at java.io.DataOutputStream.flush(DataOutputStream.java:106)
        at 
org.apache.hadoop.mapred.pipes.BinaryProtocol.flush(BinaryProtocol.java:316)
        at 
org.apache.hadoop.mapred.pipes.Application.waitForFinish(Application.java:129)
        at 
org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java:99)
        ... 3 more

11/04/04 10:59:33 WARN mapred.JobClient: Error reading task outputNo 
route to host
11/04/04 10:59:36 WARN mapred.JobClient: Error reading task outputNo 
route to host
11/04/04 10:59:37 INFO mapred.JobClient: Task Id : 
attempt_201104041057_0002_m_000002_0, Status : FAILED
java.io.IOException: pipe child exception
        at 
org.apache.hadoop.mapred.pipes.Application.abort(Application.java:151)
        at 
org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java:101)
        at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:358)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
        at org.apache.hadoop.mapred.Child.main(Child.java:170)
Caused by: java.net.SocketException: Connection reset
        at java.net.SocketInputStream.read(SocketInputStream.java:168)
        at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
        at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
        at java.io.DataInputStream.readByte(DataInputStream.java:248)
        at 
org.apache.hadoop.io.WritableUtils.readVLong(WritableUtils.java:298)
        at 
org.apache.hadoop.io.WritableUtils.readVInt(WritableUtils.java:319)
        at 
org.apache.hadoop.mapred.pipes.BinaryProtocol$UplinkReaderThread.run(BinaryProtocol.java:114)

attempt_201104041057_0002_m_000002_0: Hadoop Pipes Exception: failed to 
open  at wordcount-nopipe.cc:82 in 
WordCountReader::WordCountReader(HadoopPipes::MapContext&)
11/04/04 10:59:37 INFO mapred.JobClient: Task Id : 
attempt_201104041057_0002_m_000003_0, Status : FAILED
java.io.IOException: pipe child exception
        at 
org.apache.hadoop.mapred.pipes.Application.abort(Application.java:151)
        at 
org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java:101)
        at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:358)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
        at org.apache.hadoop.mapred.Child.main(Child.java:170)
Caused by: java.net.SocketException: Broken pipe
        at java.net.SocketOutputStream.socketWrite0(Native Method)
        at 
java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
        at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
        at 
java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
        at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
        at java.io.DataOutputStream.flush(DataOutputStream.java:106)
        at 
org.apache.hadoop.mapred.pipes.BinaryProtocol.flush(BinaryProtocol.java:316)
        at 
org.apache.hadoop.mapred.pipes.Application.waitForFinish(Application.java:129)
        at 
org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java:99)
        ... 3 more

attempt_201104041057_0002_m_000003_0: Hadoop Pipes Exception: failed to 
open  at wordcount-nopipe.cc:82 in 
WordCountReader::WordCountReader(HadoopPipes::MapContext&)
11/04/04 10:59:37 INFO mapred.JobClient: Task Id : 
attempt_201104041057_0002_m_000006_0, Status : FAILED
java.io.IOException: pipe child exception
        at 
org.apache.hadoop.mapred.pipes.Application.abort(Application.java:151)
        at 
org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java:101)
        at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:358)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
        at org.apache.hadoop.mapred.Child.main(Child.java:170)
Caused by: java.net.SocketException: Connection reset
        at 
java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:96)
        at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
        at 
java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
        at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
        at java.io.DataOutputStream.flush(DataOutputStream.java:106)
        at 
org.apache.hadoop.mapred.pipes.BinaryProtocol.flush(BinaryProtocol.java:316)
        at 
org.apache.hadoop.mapred.pipes.Application.waitForFinish(Application.java:129)
        at 
org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java:99)
        ... 3 more

11/04/04 10:59:40 WARN mapred.JobClient: Error reading task outputNo 
route to host
11/04/04 10:59:43 WARN mapred.JobClient: Error reading task outputNo 
route to host
11/04/04 10:59:43 INFO mapred.JobClient: Task Id : 
attempt_201104041057_0002_m_000007_0, Status : FAILED
java.io.IOException: pipe child exception
        at 
org.apache.hadoop.mapred.pipes.Application.abort(Application.java:151)
        at 
org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java:101)
        at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:358)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
        at org.apache.hadoop.mapred.Child.main(Child.java:170)
Caused by: java.net.SocketException: Broken pipe
        at java.net.SocketOutputStream.socketWrite0(Native Method)
        at 
java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
        at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
        at 
java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
        at java.io.BufferedOutputStream.write(BufferedOutputStream.java:109)
        at java.io.DataOutputStream.write(DataOutputStream.java:90)
        at 
org.apache.hadoop.mapred.pipes.BinaryProtocol.writeObject(BinaryProtocol.java:333)
        at 
org.apache.hadoop.mapred.pipes.BinaryProtocol.mapItem(BinaryProtocol.java:286)
        at 
org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java:90)
        ... 3 more

11/04/04 10:59:46 WARN mapred.JobClient: Error reading task outputNo 
route to host
11/04/04 10:59:49 WARN mapred.JobClient: Error reading task outputNo 
route to host
11/04/04 10:59:49 INFO mapred.JobClient: Task Id : 
attempt_201104041057_0002_m_000004_0, Status : FAILED
java.io.IOException: pipe child exception
        at 
org.apache.hadoop.mapred.pipes.Application.abort(Application.java:151)
        at 
org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java:101)
        at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:358)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
        at org.apache.hadoop.mapred.Child.main(Child.java:170)
Caused by: java.net.SocketException: Broken pipe
        at java.net.SocketOutputStream.socketWrite0(Native Method)
        at 
java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
        at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
        at 
java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
        at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
        at java.io.DataOutputStream.flush(DataOutputStream.java:106)
        at 
org.apache.hadoop.mapred.pipes.BinaryProtocol.flush(BinaryProtocol.java:316)
        at 
org.apache.hadoop.mapred.pipes.Application.waitForFinish(Application.java:129)
        at 
org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java:99)
        ... 3 more

attempt_201104041057_0002_m_000004_0: Hadoop Pipes Exception: failed to 
open  at wordcount-nopipe.cc:82 in 
WordCountReader::WordCountReader(HadoopPipes::MapContext&)
11/04/04 10:59:49 INFO mapred.JobClient: Task Id : 
attempt_201104041057_0002_m_000005_0, Status : FAILED
java.io.IOException: pipe child exception
        at 
org.apache.hadoop.mapred.pipes.Application.abort(Application.java:151)
        at 
org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java:101)
        at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:358)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
        at org.apache.hadoop.mapred.Child.main(Child.java:170)
Caused by: java.net.SocketException: Connection reset
        at java.net.SocketInputStream.read(SocketInputStream.java:168)
        at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
        at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
        at java.io.DataInputStream.readByte(DataInputStream.java:248)
        at 
org.apache.hadoop.io.WritableUtils.readVLong(WritableUtils.java:298)
        at 
org.apache.hadoop.io.WritableUtils.readVInt(WritableUtils.java:319)
        at 
org.apache.hadoop.mapred.pipes.BinaryProtocol$UplinkReaderThread.run(BinaryProtocol.java:114)

attempt_201104041057_0002_m_000005_0: Hadoop Pipes Exception: failed to 
open  at wordcount-nopipe.cc:82 in 
WordCountReader::WordCountReader(HadoopPipes::MapContext&)
11/04/04 10:59:49 INFO mapred.JobClient: Task Id : 
attempt_201104041057_0002_m_000003_1, Status : FAILED
java.io.IOException: pipe child exception
        at 
org.apache.hadoop.mapred.pipes.Application.abort(Application.java:151)
        at 
org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java:101)
        at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:358)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
        at org.apache.hadoop.mapred.Child.main(Child.java:170)
Caused by: java.net.SocketException: Connection reset
        at 
java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:96)
        at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
        at 
java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
        at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
        at java.io.DataOutputStream.flush(DataOutputStream.java:106)
        at 
org.apache.hadoop.mapred.pipes.BinaryProtocol.flush(BinaryProtocol.java:316)
        at 
org.apache.hadoop.mapred.pipes.Application.waitForFinish(Application.java:129)
        at 
org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java:99)
        ... 3 more

11/04/04 10:59:52 WARN mapred.JobClient: Error reading task outputNo 
route to host
11/04/04 10:59:55 WARN mapred.JobClient: Error reading task outputNo 
route to host
11/04/04 10:59:55 INFO mapred.JobClient: Task Id : 
attempt_201104041057_0002_m_000002_1, Status : FAILED
java.io.IOException: pipe child exception
        at 
org.apache.hadoop.mapred.pipes.Application.abort(Application.java:151)
        at 
org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java:101)
        at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:358)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
        at org.apache.hadoop.mapred.Child.main(Child.java:170)
Caused by: java.net.SocketException: Connection reset
        at 
java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:96)
        at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
        at 
java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
        at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
        at java.io.DataOutputStream.flush(DataOutputStream.java:106)
        at 
org.apache.hadoop.mapred.pipes.BinaryProtocol.flush(BinaryProtocol.java:316)
        at 
org.apache.hadoop.mapred.pipes.Application.waitForFinish(Application.java:129)
        at 
org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java:99)
        ... 3 more

11/04/04 10:59:58 WARN mapred.JobClient: Error reading task outputNo 
route to host
11/04/04 11:00:01 WARN mapred.JobClient: Error reading task outputNo 
route to host
11/04/04 11:00:01 INFO mapred.JobClient: Task Id : 
attempt_201104041057_0002_m_000007_1, Status : FAILED
java.io.IOException: pipe child exception
        at 
org.apache.hadoop.mapred.pipes.Application.abort(Application.java:151)
        at 
org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java:101)
        at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:358)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
        at org.apache.hadoop.mapred.Child.main(Child.java:170)
Caused by: java.net.SocketException: Connection reset
        at java.net.SocketInputStream.read(SocketInputStream.java:168)
        at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
        at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
        at java.io.DataInputStream.readByte(DataInputStream.java:248)
        at 
org.apache.hadoop.io.WritableUtils.readVLong(WritableUtils.java:298)
        at 
org.apache.hadoop.io.WritableUtils.readVInt(WritableUtils.java:319)
        at 
org.apache.hadoop.mapred.pipes.BinaryProtocol$UplinkReaderThread.run(BinaryProtocol.java:114)

attempt_201104041057_0002_m_000007_1: Hadoop Pipes Exception: failed to 
open  at wordcount-nopipe.cc:82 in 
WordCountReader::WordCountReader(HadoopPipes::MapContext&)
11/04/04 11:00:01 INFO mapred.JobClient: Task Id : 
attempt_201104041057_0002_m_000006_1, Status : FAILED
java.io.IOException: pipe child exception
        at 
org.apache.hadoop.mapred.pipes.Application.abort(Application.java:151)
        at 
org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java:101)
        at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:358)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
        at org.apache.hadoop.mapred.Child.main(Child.java:170)
Caused by: java.io.EOFException
        at java.io.DataInputStream.readByte(DataInputStream.java:250)
        at 
org.apache.hadoop.io.WritableUtils.readVLong(WritableUtils.java:298)
        at 
org.apache.hadoop.io.WritableUtils.readVInt(WritableUtils.java:319)
        at 
org.apache.hadoop.mapred.pipes.BinaryProtocol$UplinkReaderThread.run(BinaryProtocol.java:114)

attempt_201104041057_0002_m_000006_1: Hadoop Pipes Exception: failed to 
open  at wordcount-nopipe.cc:82 in 
WordCountReader::WordCountReader(HadoopPipes::MapContext&)
11/04/04 11:00:01 INFO mapred.JobClient: Job complete: job_201104041057_0002
11/04/04 11:00:01 INFO mapred.JobClient: Counters: 4
11/04/04 11:00:01 INFO mapred.JobClient:   Job Counters
11/04/04 11:00:01 INFO mapred.JobClient:     Rack-local map tasks=8
11/04/04 11:00:01 INFO mapred.JobClient:     Launched map tasks=28
11/04/04 11:00:01 INFO mapred.JobClient:     Data-local map tasks=20
11/04/04 11:00:01 INFO mapred.JobClient:     Failed map tasks=1
Exception in thread "main" java.io.IOException: Job failed!
        at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1252)
        at 
org.apache.hadoop.mapred.pipes.Submitter.runJob(Submitter.java:248)
        at org.apache.hadoop.mapred.pipes.Submitter.run(Submitter.java:479)
        at org.apache.hadoop.mapred.pipes.Submitter.main(Submitter.java:494)
bash-3.2$


Please  help me to find the reasons & possible solutions to resolve the 
above error.




Thanks & best Regards,
Adarsh Sharma