You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2022/02/07 10:34:47 UTC
[GitHub] [spark] HyukjinKwon opened a new pull request #35423: [SPARK-38128][PYTHON][TESTS] Show full stacktrace in tests by default in PySpark tests
HyukjinKwon opened a new pull request #35423:
URL: https://github.com/apache/spark/pull/35423
### What changes were proposed in this pull request?
This PR proposes to show full stacktrace of Python worker and JVM in PySpark by controlling `spark.sql.pyspark.jvmStacktrace.enabled` and `spark.sql.execution.pyspark.udf.simplifiedTraceback.enabled` only in tests.
### Why are the changes needed?
[SPARK-33407](https://issues.apache.org/jira/browse/SPARK-33407) and [SPARK-31849](https://issues.apache.org/jira/browse/SPARK-31849) hide Java stacktrace and internal Python worker side traceback by default for simpler error messages to end users. However, specifically for unit tests, that makes a bit harder to debug the test failures. We should probably show the full stacktrace by default in tests.
### Does this PR introduce _any_ user-facing change?
No, this is test only.
### How was this patch tested?
Manually tested. Now the test failures show the logs as below:
```
======================================================================
ERROR [3.259s]: test (pyspark.sql.tests.test_functions.FunctionsTests)
----------------------------------------------------------------------
Traceback (most recent call last):
...
pyspark.sql.utils.PythonException:
An exception was thrown from the Python worker. Please see the stack trace below.
Traceback (most recent call last):
File "/.../lib/pyspark.zip/pyspark/worker.py", line 678, in main
process()
File "/.../lib/pyspark.zip/pyspark/worker.py", line 670, in process
serializer.dump_stream(out_iter, outfile)
File "/.../lib/pyspark.zip/pyspark/serializers.py", line 217, in dump_stream
self.serializer.dump_stream(self._batched(iterator), stream)
...
ZeroDivisionError: division by zero
JVM stacktrace:
...
at org.apache.spark.api.python.BasePythonRunner$ReaderIterator.handlePythonException(PythonRunner.scala:558)
at org.apache.spark.sql.execution.python.PythonUDFRunner$$anon$2.read(PythonUDFRunner.scala:86)
at org.apache.spark.sql.execution.python.PythonUDFRunner$$anon$2.read(PythonUDFRunner.scala:68)
at org.apache.spark.api.python.BasePythonRunner$ReaderIterator.hasNext(PythonRunner.scala:511)
at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:37)
...
Driver stacktrace:
...
Caused by: org.apache.spark.api.python.PythonException: Traceback (most recent call last):
... 1 more
----------------------------------------------------------------------
Ran 1 test in 12.610s
FAILED (errors=1)
Generating XML reports...
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #35423: [SPARK-38128][PYTHON][TESTS] Show full stacktrace in tests by default in PySpark tests
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #35423:
URL: https://github.com/apache/spark/pull/35423#issuecomment-1031405238
All tests passed.
Merged to master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon closed pull request #35423: [SPARK-38128][PYTHON][TESTS] Show full stacktrace in tests by default in PySpark tests
Posted by GitBox <gi...@apache.org>.
HyukjinKwon closed pull request #35423:
URL: https://github.com/apache/spark/pull/35423
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #35423: [SPARK-38128][PYTHON][TESTS] Show full stacktrace in tests by default in PySpark tests
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #35423:
URL: https://github.com/apache/spark/pull/35423#issuecomment-1031319536
@WeichenXu123 yup, it covers all
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org