You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "linhongliu-db (via GitHub)" <gi...@apache.org> on 2026/07/24 20:02:57 UTC
[PR] [SPARK-57271][PYTHON] Propagate traceback locals to Python planner runner [spark]
linhongliu-db opened a new pull request, #57509:
URL: https://github.com/apache/spark/pull/57509
### What changes were proposed in this pull request?
This PR propagates `SPARK_TRACEBACK_WITH_LOCALS` from `PythonPlannerRunner` when `spark.sql.execution.pyspark.udf.tracebackWithLocals.enabled` is enabled.
It also adds a UDTF `analyze` regression test that raises from the planner-side Python worker and verifies the surfaced traceback includes the local variable.
### Why are the changes needed?
`PythonPlannerRunner` already reads `spark.sql.execution.pyspark.udf.tracebackWithLocals.enabled`, but it did not add `SPARK_TRACEBACK_WITH_LOCALS` to the Python worker environment. As a result, planner-driven Python paths such as UDTF `analyze` did not honor the traceback-locals config, unlike regular Python UDF execution.
### Does this PR introduce _any_ user-facing change?
Yes. When `spark.sql.execution.pyspark.udf.tracebackWithLocals.enabled` is enabled, Python exceptions raised through planner-driven Python paths can now include local variables in their tracebacks.
### How was this patch tested?
Passed:
```
python3 -m py_compile python/pyspark/sql/tests/test_udtf.py
python3 -m py_compile python/pyspark/sql/tests/test_udf.py
git diff --check
```
Attempted but not run locally because the Spark assembly could not be built in this environment:
```
./build/sbt assembly/package
JAVA_HOME=/usr/lib/jvm/java-11-openjdk-amd64 ./build/mvn -DskipTests -pl :spark-assembly_2.13 -am package
python3 python/run-tests.py --testnames 'pyspark.sql.tests.test_udtf UDTFTests.test_udtf_analyze_traceback_with_locals'
python3 python/run-tests.py --testnames 'pyspark.sql.tests.test_udf UDFTests.test_udf_traceback_with_locals'
```
The SBT launcher fetch failed, and Maven dependency resolution failed because Maven Central hostnames did not resolve from this local environment. The PySpark test runner then failed at startup with `Cannot find assembly build directory, please build Spark first.`
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: OpenAI Codex (GPT-5)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57271][PYTHON] Propagate traceback locals to Python planner runner [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan closed pull request #57509: [SPARK-57271][PYTHON] Propagate traceback locals to Python planner runner
URL: https://github.com/apache/spark/pull/57509
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57271][PYTHON] Propagate traceback locals to Python planner runner [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on PR #57509:
URL: https://github.com/apache/spark/pull/57509#issuecomment-5118008728
thanks, merging to master/4.x
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org