(Unknown)

[Sheena O'Connell <sh...@gmail.com>]

Hi

I have a python script that throws an obvious error

            NameError: name 'asooasdhoasdhio' is not defined

and I'm using that script as both the mapper and reducer for a streaming
task.

/usr/local/hadoop/bin/hadoop jar
/usr/local/hadoop/share/hadoop/tools/lib/hadoop-streaming-2.4.0.jar \
-files "/fooo/simple_streaming/PrescriptionAlternateCosts.py" \
-files "some/file" \
-files another/file" \
-mapper "python PrescriptionAlternateCosts.py mapper 2014_04" \
-reducer "python PrescriptionAlternateCosts.py reducer" \
-input "/some/existing file" \
-output "/some/not/yet/existing/file"

As expected, the task fails. If I keep an eye on what is happening in my
userlogs directory then a folder is created for the application and it is
populated as it should be. But a few seconds after the application fails
the associated logs are deleted.

Any idea why my logs are being deleted? Any idea how I can stop this from
happening? It's making debugging non-obvious errors quite difficult.

Here is the error message I get:

14/09/10 14:08:08 INFO mapreduce.Job: Task Id :
attempt_1410178572567_0164_m_
000004_2, Status : FAILED
Error: java.lang.RuntimeException: PipeMapRed.waitOutputThreads():
subprocess failed with code 1
    at
org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:320)
    at
org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:533)
    at org.apache.hadoop.streaming.PipeMapper.close(PipeMapper.java:130)
    at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61)
    at org.apache.hadoop.streaming.PipeMapRunner.run(PipeMapRunner.java:34)
    at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:430)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
    at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:416)
    at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
    at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)

It continues like that for a while.

Any help would be appreciated

Sheena