You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Karl Anderson (JIRA)" <ji...@apache.org> on 2008/10/21 01:34:44 UTC
[jira] Created: (HADOOP-4468) writing to status reporter before
consuming standard input causes task failure.
writing to status reporter before consuming standard input causes task failure.
-------------------------------------------------------------------------------
Key: HADOOP-4468
URL: https://issues.apache.org/jira/browse/HADOOP-4468
Project: Hadoop Core
Issue Type: Bug
Components: contrib/streaming
Affects Versions: 0.18.1
Environment: amazon ec2 instance created with the given scripts (fedora, small)
Reporter: Karl Anderson
A Hadoop Streaming task which writes a status reporter line before consuming input causes the task to fail. Writing after consuming input does not fail.
I caused this failure using a Python reducer and writing a "reporter:status:foo\n" line to stderr. Didn't try writing anything else.
The reducer script which fails:
#!/usr/bin/env python
import sys
if __name__ == "__main__":
sys.stderr.write('reporter:status:foo\n')
sys.stderr.flush()
for line in sys.stdin:
print line
The reducer script which succeeds:
#!/usr/bin/env python
import sys
if __name__ == "__main__":
for line in sys.stdin:
sys.stderr.write('reporter:status:foo\n')
sys.stderr.flush()
print line
The hadoop invocation which I used:
hadoop jar /usr/local/hadoop-0.18.1/contrib/streaming/hadoop-0.18.1-streaming.jar -mapper cat -reducer ./reducer_foo.py -input vectors -output clusters_1 -jobconf mapred.map.tasks=512 -jobconf mapred.reduce.tasks=512 -file ./reducer_foo.py
This is on a 64 node hadoop-ec2 cluster.
One of the errors listed on the failures page (they all appear to be the same):
java.io.IOException: subprocess exited successfully
R/W/S=1/0/0 in:0=1/41 [rec/s] out:0=0/41 [rec/s]
minRecWrittenToEnableSkip_=9223372036854775807 LOGNAME=null
HOST=null
USER=root
HADOOP_USER=null
last Hadoop input: |null|
last tool output: |null|
Date: Mon Oct 20 19:13:38 EDT 2008
MROutput/MRErrThread failed:java.lang.NullPointerException
at org.apache.hadoop.streaming.PipeMapRed$MRErrorThread.setStatus(PipeMapRed.java:497)
at org.apache.hadoop.streaming.PipeMapRed$MRErrorThread.run(PipeMapRed.java:429)
at org.apache.hadoop.streaming.PipeReducer.reduce(PipeReducer.java:103)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:318)
at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2207)
The stderr log for a failed task:
Exception in thread "Timer thread for monitoring mapred" java.lang.NullPointerException
at org.apache.hadoop.metrics.ganglia.GangliaContext.xdr_string(GangliaContext.java:195)
at org.apache.hadoop.metrics.ganglia.GangliaContext.emitMetric(GangliaContext.java:138)
at org.apache.hadoop.metrics.ganglia.GangliaContext.emitRecord(GangliaContext.java:123)
at org.apache.hadoop.metrics.spi.AbstractMetricsContext.emitRecords(AbstractMetricsContext.java:304)
at org.apache.hadoop.metrics.spi.AbstractMetricsContext.timerEvent(AbstractMetricsContext.java:290)
at org.apache.hadoop.metrics.spi.AbstractMetricsContext.access$000(AbstractMetricsContext.java:50)
at org.apache.hadoop.metrics.spi.AbstractMetricsContext$1.run(AbstractMetricsContext.java:249)
at java.util.TimerThread.mainLoop(Timer.java:512)
at java.util.TimerThread.run(Timer.java:462)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4468) writing to status reporter before
consuming standard input causes task failure.
Posted by "Karl Anderson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12641211#action_12641211 ]
Karl Anderson commented on HADOOP-4468:
---------------------------------------
sorry, Jira ate the python whitespace formatting and under-under strings, it is hopefully clear enough. Failure example emitted output before "for line in sys.stdin" loop, suceeeding example emitted output within the loop (but before printing output).
> writing to status reporter before consuming standard input causes task failure.
> -------------------------------------------------------------------------------
>
> Key: HADOOP-4468
> URL: https://issues.apache.org/jira/browse/HADOOP-4468
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/streaming
> Affects Versions: 0.18.1
> Environment: amazon ec2 instance created with the given scripts (fedora, small)
> Reporter: Karl Anderson
>
> A Hadoop Streaming task which writes a status reporter line before consuming input causes the task to fail. Writing after consuming input does not fail.
> I caused this failure using a Python reducer and writing a "reporter:status:foo\n" line to stderr. Didn't try writing anything else.
> The reducer script which fails:
> #!/usr/bin/env python
> import sys
> if __name__ == "__main__":
> sys.stderr.write('reporter:status:foo\n')
> sys.stderr.flush()
> for line in sys.stdin:
> print line
> The reducer script which succeeds:
> #!/usr/bin/env python
> import sys
> if __name__ == "__main__":
> for line in sys.stdin:
> sys.stderr.write('reporter:status:foo\n')
> sys.stderr.flush()
> print line
> The hadoop invocation which I used:
> hadoop jar /usr/local/hadoop-0.18.1/contrib/streaming/hadoop-0.18.1-streaming.jar -mapper cat -reducer ./reducer_foo.py -input vectors -output clusters_1 -jobconf mapred.map.tasks=512 -jobconf mapred.reduce.tasks=512 -file ./reducer_foo.py
> This is on a 64 node hadoop-ec2 cluster.
> One of the errors listed on the failures page (they all appear to be the same):
> java.io.IOException: subprocess exited successfully
> R/W/S=1/0/0 in:0=1/41 [rec/s] out:0=0/41 [rec/s]
> minRecWrittenToEnableSkip_=9223372036854775807 LOGNAME=null
> HOST=null
> USER=root
> HADOOP_USER=null
> last Hadoop input: |null|
> last tool output: |null|
> Date: Mon Oct 20 19:13:38 EDT 2008
> MROutput/MRErrThread failed:java.lang.NullPointerException
> at org.apache.hadoop.streaming.PipeMapRed$MRErrorThread.setStatus(PipeMapRed.java:497)
> at org.apache.hadoop.streaming.PipeMapRed$MRErrorThread.run(PipeMapRed.java:429)
> at org.apache.hadoop.streaming.PipeReducer.reduce(PipeReducer.java:103)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:318)
> at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2207)
> The stderr log for a failed task:
> Exception in thread "Timer thread for monitoring mapred" java.lang.NullPointerException
> at org.apache.hadoop.metrics.ganglia.GangliaContext.xdr_string(GangliaContext.java:195)
> at org.apache.hadoop.metrics.ganglia.GangliaContext.emitMetric(GangliaContext.java:138)
> at org.apache.hadoop.metrics.ganglia.GangliaContext.emitRecord(GangliaContext.java:123)
> at org.apache.hadoop.metrics.spi.AbstractMetricsContext.emitRecords(AbstractMetricsContext.java:304)
> at org.apache.hadoop.metrics.spi.AbstractMetricsContext.timerEvent(AbstractMetricsContext.java:290)
> at org.apache.hadoop.metrics.spi.AbstractMetricsContext.access$000(AbstractMetricsContext.java:50)
> at org.apache.hadoop.metrics.spi.AbstractMetricsContext$1.run(AbstractMetricsContext.java:249)
> at java.util.TimerThread.mainLoop(Timer.java:512)
> at java.util.TimerThread.run(Timer.java:462)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.