You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "Arun C Murthy (JIRA)" <ji...@apache.org> on 2008/04/06 20:36:24 UTC
[jira] Reopened: (PIG-187) Got "broken pipe" with 2 shipped files
[ https://issues.apache.org/jira/browse/PIG-187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy reopened PIG-187:
-------------------------------
> Got "broken pipe" with 2 shipped files
> --------------------------------------
>
> Key: PIG-187
> URL: https://issues.apache.org/jira/browse/PIG-187
> Project: Pig
> Issue Type: Bug
> Reporter: Xu Zhang
> Assignee: Arun C Murthy
>
> Currently, it seems the "broken pipe" error for the case where only one file is shipped and executed has been fixed. I still get "broken pipe" error when 2 files are shipped and then one of them is used the command argument of the other.
> Here is my Pig script:
> {code}
> set stream.skippath '/home/xu/testdata/';
> define X `MySimpleStreamApp.pl copyofstudenttab10k` ship('./streamingscript/MySimpleStreamApp.pl', '/home/xu/testdata/copyofstudenttab10k');
> A = load '/user/pig/tests/data/singlefile/studenttab10k';
> B = stream A through X as (name, age, gpa);
> C = group B by name;
> D = foreach C generate COUNT(B.$0);
> store D into 'results_18';
> {code}
> Here is Pig's console output:
> {noformat}
> 2008-04-04 18:34:15,418 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to hadoop file system at: wilbur11.labs.corp.sp1.yahoo.com:8020
> 2008-04-04 18:34:16,306 [main] INFO org.apache.pig.backend.hadoop.executionengine.POMapreduce - ----- MapReduce Job -----
> 2008-04-04 18:34:16,306 [main] INFO org.apache.pig.backend.hadoop.executionengine.POMapreduce - Input: [/user/pig/tests/data/singlefile/studenttab10k:org.apache.pig.builtin.PigStorage()]
> 2008-04-04 18:34:16,307 [main] INFO org.apache.pig.backend.hadoop.executionengine.POMapreduce - Map: [[*]->org.apache.pig.impl.eval.StreamSpec@121ab80]
> 2008-04-04 18:34:16,307 [main] INFO org.apache.pig.backend.hadoop.executionengine.POMapreduce - Group: [GENERATE {[PROJECT $0],[*]}]
> 2008-04-04 18:34:16,307 [main] INFO org.apache.pig.backend.hadoop.executionengine.POMapreduce - Combine: null
> 2008-04-04 18:34:16,307 [main] INFO org.apache.pig.backend.hadoop.executionengine.POMapreduce - Reduce: GENERATE {[COUNT(GENERATE {[PROJECT $1]->[PROJECT $0]})]}
> 2008-04-04 18:34:16,308 [main] INFO org.apache.pig.backend.hadoop.executionengine.POMapreduce - Output: results_18:org.apache.pig.builtin.PigStorage
> 2008-04-04 18:34:16,308 [main] INFO org.apache.pig.backend.hadoop.executionengine.POMapreduce - Split: null
> 2008-04-04 18:34:16,308 [main] INFO org.apache.pig.backend.hadoop.executionengine.POMapreduce - Map parallelism: -1
> 2008-04-04 18:34:16,308 [main] INFO org.apache.pig.backend.hadoop.executionengine.POMapreduce - Reduce parallelism: -1
> 219190 hdfs://wilbur11.labs.corp.sp1.yahoo.com:8020/user/pig/tests/data/singlefile/studenttab10k
> 2008-04-04 18:34:19,383 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapreduceExec.MapReduceLauncher - Pig progress = 0%
> 2008-04-04 18:34:44,491 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapreduceExec.MapReduceLauncher - Error message from task (map) tip_200804041056_0168_m_000000 java.lang.RuntimeException: java.lang.RuntimeException: java.io.IOException: Broken pipe
> at org.apache.pig.impl.eval.collector.DataCollector.finishPipe(DataCollector.java:147)
> at org.apache.pig.backend.hadoop.executionengine.mapreduceExec.PigMapReduce.run(PigMapReduce.java:119)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:208)
> at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2071)
> Caused by: java.lang.RuntimeException: java.io.IOException: Broken pipe
> at org.apache.pig.impl.eval.StreamSpec$StreamDataCollector.finish(StreamSpec.java:152)
> at org.apache.pig.impl.eval.collector.DataCollector.finishPipe(DataCollector.java:131)
> ... 3 more
> Caused by: java.io.IOException: Broken pipe
> at java.io.FileOutputStream.writeBytes(Native Method)
> at java.io.FileOutputStream.write(FileOutputStream.java:260)
> at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
> at java.io.BufferedOutputStream.write(BufferedOutputStream.java:109)
> at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
> at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
> at java.io.DataOutputStream.flush(DataOutputStream.java:106)
> at org.apache.pig.impl.streaming.DefaultInputHandler.close(DefaultInputHandler.java:56)
> at org.apache.pig.impl.streaming.ExecutableManager.close(ExecutableManager.java:134)
> at org.apache.pig.backend.hadoop.streaming.HadoopExecutableManager.close(HadoopExecutableManager.java:115)
> at org.apache.pig.impl.eval.StreamSpec$StreamDataCollector.finish(StreamSpec.java:148)
> ... 4 more
> java.lang.RuntimeException: java.lang.RuntimeException: java.io.IOException: Broken pipe
> at org.apache.pig.impl.eval.collector.DataCollector.finishPipe(DataCollector.java:147)
> at org.apache.pig.backend.hadoop.executionengine.mapreduceExec.PigMapReduce.run(PigMapReduce.java:119)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:208)
> at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2071)
> Caused by: java.lang.RuntimeException: java.io.IOException: Broken pipe
> at org.apache.pig.impl.eval.StreamSpec$StreamDataCollector.finish(StreamSpec.java:152)
> at org.apache.pig.impl.eval.collector.DataCollector.finishPipe(DataCollector.java:131)
> ... 3 more
> Caused by: java.io.IOException: Broken pipe
> at java.io.FileOutputStream.writeBytes(Native Method)
> at java.io.FileOutputStream.write(FileOutputStream.java:260)
> at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
> at java.io.BufferedOutputStream.write(BufferedOutputStream.java:109)
> at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
> at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
> at java.io.DataOutputStream.flush(DataOutputStream.java:106)
> at org.apache.pig.impl.streaming.DefaultInputHandler.close(DefaultInputHandler.java:56)
> at org.apache.pig.impl.streaming.ExecutableManager.close(ExecutableManager.java:134)
> at org.apache.pig.backend.hadoop.streaming.HadoopExecutableManager.close(HadoopExecutableManager.java:115)
> at org.apache.pig.impl.eval.StreamSpec$StreamDataCollector.finish(StreamSpec.java:148)
> ... 4 more
> java.lang.RuntimeException: java.lang.RuntimeException: java.io.IOException: Broken pipe
> at org.apache.pig.impl.eval.collector.DataCollector.finishPipe(DataCollector.java:147)
> at org.apache.pig.backend.hadoop.executionengine.mapreduceExec.PigMapReduce.run(PigMapReduce.java:119)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:208)
> at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2071)
> Caused by: java.lang.RuntimeException: java.io.IOException: Broken pipe
> at org.apache.pig.impl.eval.StreamSpec$StreamDataCollector.finish(StreamSpec.java:152)
> at org.apache.pig.impl.eval.collector.DataCollector.finishPipe(DataCollector.java:131)
> ... 3 more
> Caused by: java.io.IOException: Broken pipe
> at java.io.FileOutputStream.writeBytes(Native Method)
> at java.io.FileOutputStream.write(FileOutputStream.java:260)
> at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
> at java.io.BufferedOutputStream.write(BufferedOutputStream.java:109)
> at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
> at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
> at java.io.DataOutputStream.flush(DataOutputStream.java:106)
> at org.apache.pig.impl.streaming.DefaultInputHandler.close(DefaultInputHandler.java:56)
> at org.apache.pig.impl.streaming.ExecutableManager.close(ExecutableManager.java:134)
> at org.apache.pig.backend.hadoop.streaming.HadoopExecutableManager.close(HadoopExecutableManager.java:115)
> at org.apache.pig.impl.eval.StreamSpec$StreamDataCollector.finish(StreamSpec.java:148)
> ... 4 more
> java.lang.RuntimeException: java.lang.RuntimeException: java.io.IOException: Broken pipe
> at org.apache.pig.impl.eval.collector.DataCollector.finishPipe(DataCollector.java:147)
> at org.apache.pig.backend.hadoop.executionengine.mapreduceExec.PigMapReduce.run(PigMapReduce.java:119)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:208)
> at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2071)
> Caused by: java.lang.RuntimeException: java.io.IOException: Broken pipe
> at org.apache.pig.impl.eval.StreamSpec$StreamDataCollector.finish(StreamSpec.java:152)
> at org.apache.pig.impl.eval.collector.DataCollector.finishPipe(DataCollector.java:131)
> ... 3 more
> Caused by: java.io.IOException: Broken pipe
> at java.io.FileOutputStream.writeBytes(Native Method)
> at java.io.FileOutputStream.write(FileOutputStream.java:260)
> at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
> at java.io.BufferedOutputStream.write(BufferedOutputStream.java:109)
> at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
> at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
> at java.io.DataOutputStream.flush(DataOutputStream.java:106)
> at org.apache.pig.impl.streaming.DefaultInputHandler.close(DefaultInputHandler.java:56)
> at org.apache.pig.impl.streaming.ExecutableManager.close(ExecutableManager.java:134)
> at org.apache.pig.backend.hadoop.streaming.HadoopExecutableManager.close(HadoopExecutableManager.java:115)
> at org.apache.pig.impl.eval.StreamSpec$StreamDataCollector.finish(StreamSpec.java:148)
> ... 4 more
> 2008-04-04 18:34:44,501 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapreduceExec.MapReduceLauncher - Error message from task (reduce) tip_200804041056_0168_r_000000
> 2008-04-04 18:34:44,501 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapreduceExec.MapReduceLauncher - Error message from task (reduce) tip_200804041056_0168_r_000001
> 2008-04-04 18:34:44,501 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapreduceExec.MapReduceLauncher - Error message from task (reduce) tip_200804041056_0168_r_000002
> 2008-04-04 18:34:44,501 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapreduceExec.MapReduceLauncher - Error message from task (reduce) tip_200804041056_0168_r_000003
> 2008-04-04 18:34:44,501 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapreduceExec.MapReduceLauncher - Error message from task (reduce) tip_200804041056_0168_r_000004
> 2008-04-04 18:34:44,501 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapreduceExec.MapReduceLauncher - Error message from task (reduce) tip_200804041056_0168_r_000005
> 2008-04-04 18:34:44,501 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapreduceExec.MapReduceLauncher - Error message from task (reduce) tip_200804041056_0168_r_000006
> 2008-04-04 18:34:44,501 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapreduceExec.MapReduceLauncher - Error message from task (reduce) tip_200804041056_0168_r_000007
> 2008-04-04 18:34:44,502 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapreduceExec.MapReduceLauncher - Error message from task (reduce) tip_200804041056_0168_r_000008
> 2008-04-04 18:34:44,502 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapreduceExec.MapReduceLauncher - Error message from task (reduce) tip_200804041056_0168_r_000009
> 2008-04-04 18:34:44,502 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapreduceExec.MapReduceLauncher - Error message from task (reduce) tip_200804041056_0168_r_000010
> 2008-04-04 18:34:44,502 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapreduceExec.MapReduceLauncher - Error message from task (reduce) tip_200804041056_0168_r_000011
> 2008-04-04 18:34:44,502 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapreduceExec.MapReduceLauncher - Error message from task (reduce) tip_200804041056_0168_r_000012
> 2008-04-04 18:34:44,502 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapreduceExec.MapReduceLauncher - Error message from task (reduce) tip_200804041056_0168_r_000013
> 2008-04-04 18:34:44,502 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapreduceExec.MapReduceLauncher - Error message from task (reduce) tip_200804041056_0168_r_000014
> 2008-04-04 18:34:44,502 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapreduceExec.MapReduceLauncher - Error message from task (reduce) tip_200804041056_0168_r_000015
> 2008-04-04 18:34:44,502 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapreduceExec.MapReduceLauncher - Error message from task (reduce) tip_200804041056_0168_r_000016
> 2008-04-04 18:34:44,502 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapreduceExec.MapReduceLauncher - Error message from task (reduce) tip_200804041056_0168_r_000017
> 2008-04-04 18:34:44,502 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapreduceExec.MapReduceLauncher - Error message from task (reduce) tip_200804041056_0168_r_000018
> 2008-04-04 18:34:44,502 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapreduceExec.MapReduceLauncher - Error message from task (reduce) tip_200804041056_0168_r_000019
> 2008-04-04 18:34:44,507 [main] ERROR org.apache.pig.tools.grunt.Grunt - java.io.IOException: Unable to store alias null
> at org.apache.pig.impl.util.WrappedIOException.wrap(WrappedIOException.java:16)
> at org.apache.pig.PigServer.registerQuery(PigServer.java:283)
> at org.apache.pig.tools.grunt.GruntParser.processPig(GruntParser.java:446)
> at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:226)
> at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:62)
> at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:60)
> at org.apache.pig.Main.main(Main.java:265)
> Caused by: org.apache.pig.backend.executionengine.ExecException: java.io.IOException: Job failed
> at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.execute(HExecutionEngine.java:288)
> at org.apache.pig.PigServer.optimizeAndRunQuery(PigServer.java:400)
> at org.apache.pig.PigServer.registerQuery(PigServer.java:280)
> ... 5 more
> Caused by: java.io.IOException: Job failed
> at org.apache.pig.backend.hadoop.executionengine.POMapreduce.open(POMapreduce.java:180)
> at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.execute(HExecutionEngine.java:274)
> ... 7 more
> 2008-04-04 18:34:44,508 [main] ERROR org.apache.pig.tools.grunt.Grunt - Unable to store alias null
> {noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.