[4/8] git commit: Log Python exceptions to stderr as well

[pw...@apache.org]

Log Python exceptions to stderr as well

This helps in case the exception happened while serializing a record to
be sent to Java, leaving the stream to Java in an inconsistent state
where PythonRDD won't be able to read the error.


Project: http://git-wip-us.apache.org/repos/asf/incubator-spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-spark/commit/5741078c
Tree: http://git-wip-us.apache.org/repos/asf/incubator-spark/tree/5741078c
Diff: http://git-wip-us.apache.org/repos/asf/incubator-spark/diff/5741078c

Branch: refs/heads/master
Commit: 5741078c46828f124bb8286919398a6c346b109c
Parents: f00e949
Author: Matei Zaharia <ma...@databricks.com>
Authored: Sun Jan 12 00:10:41 2014 -0800
Committer: Matei Zaharia <ma...@databricks.com>
Committed: Sun Jan 12 00:10:41 2014 -0800

----------------------------------------------------------------------
 python/pyspark/worker.py | 4 ++++
 1 file changed, 4 insertions(+)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/incubator-spark/blob/5741078c/python/pyspark/worker.py
----------------------------------------------------------------------
diff --git a/python/pyspark/worker.py b/python/pyspark/worker.py
index f2b3f3c..d77981f 100644
--- a/python/pyspark/worker.py
+++ b/python/pyspark/worker.py
@@ -76,6 +76,10 @@ def main(infile, outfile):
         iterator = deserializer.load_stream(infile)
         serializer.dump_stream(func(split_index, iterator), outfile)
     except Exception as e:
+        # Write the error to stderr in addition to trying to passi t back to
+        # Java, in case it happened while serializing a record
+        print >> sys.stderr, "PySpark worker failed with exception:"
+        print >> sys.stderr, traceback.format_exc()
         write_int(SpecialLengths.PYTHON_EXCEPTION_THROWN, outfile)
         write_with_length(traceback.format_exc(), outfile)
         sys.exit(-1)