You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by gu...@apache.org on 2021/11/09 11:04:33 UTC
[spark] branch master updated: [SPARK-37253][PYTHON]
`try_simplify_traceback` should not fail when `tb_frame.f_lineno` is None
This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 8ae88d0 [SPARK-37253][PYTHON] `try_simplify_traceback` should not fail when `tb_frame.f_lineno` is None
8ae88d0 is described below
commit 8ae88d01b46d581367d0047b50fcfb65078ab972
Author: Dongjoon Hyun <do...@apache.org>
AuthorDate: Tue Nov 9 20:03:34 2021 +0900
[SPARK-37253][PYTHON] `try_simplify_traceback` should not fail when `tb_frame.f_lineno` is None
### What changes were proposed in this pull request?
This PR aims to handle the corner case when `tb_frame.f_lineno` is `None` in `try_simplify_traceback` which was added by https://github.com/apache/spark/pull/30309 at Apache Spark 3.1.0.
### Why are the changes needed?
This will handle the following corner case.
```python
Traceback (most recent call last):
File "/Users/dongjoon/APACHE/spark-merge/python/lib/pyspark.zip/pyspark/worker.py", line 630, in main
tb = try_simplify_traceback(sys.exc_info()[-1])
File "/Users/dongjoon/APACHE/spark-merge/python/lib/pyspark.zip/pyspark/util.py", line 217, in try_simplify_traceback
new_tb = types.TracebackType(
TypeError: 'NoneType' object cannot be interpreted as an integer
```
Python GitHub Repo also has the test case for this corner case.
- https://github.com/python/cpython/blob/main/Lib/test/test_exceptions.py#L2373
```python
None if frame.f_lineno is None else
```
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
N/A
Closes #34530 from dongjoon-hyun/SPARK-37253.
Authored-by: Dongjoon Hyun <do...@apache.org>
Signed-off-by: Hyukjin Kwon <gu...@apache.org>
---
python/pyspark/util.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/python/pyspark/util.py b/python/pyspark/util.py
index e07474d..3f36fb3 100644
--- a/python/pyspark/util.py
+++ b/python/pyspark/util.py
@@ -218,7 +218,7 @@ def try_simplify_traceback(tb):
tb_next=tb_next,
tb_frame=cur_tb.tb_frame,
tb_lasti=cur_tb.tb_frame.f_lasti,
- tb_lineno=cur_tb.tb_frame.f_lineno)
+ tb_lineno=cur_tb.tb_frame.f_lineno if cur_tb.tb_frame.f_lineno is not None else -1)
tb_next = new_tb
return new_tb
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org