You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by ga...@apache.org on 2026/09/29 19:06:24 UTC
(spark) branch master updated: [SPARK-59762][PYTHON][FOLLOWUP] Stop UDTFWithPartitions from keeping udtf_info alive for the whole task
This is an automated email from the ASF dual-hosted git repository.
gaogaotiantian pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 0bebce96e477 [SPARK-59762][PYTHON][FOLLOWUP] Stop UDTFWithPartitions from keeping udtf_info alive for the whole task
0bebce96e477 is described below
commit 0bebce96e477fb4ecf893e72505a620644dc1f15
Author: Constantin-Tiberiu Craiu <57...@users.noreply.github.com>
AuthorDate: Tue Sep 29 12:06:08 2026 -0700
[SPARK-59762][PYTHON][FOLLOWUP] Stop UDTFWithPartitions from keeping udtf_info alive for the whole task
### What changes were proposed in this pull request?
Follow-up to https://github.com/apache/spark/pull/59013
Use the `partition_child_indexes` argument in `UDTFWithPartitions.__init__` instead of the enclosing `udtf_info` and avoid `udtf_info` being keppt by the closure.
### Why are the changes needed?
Reduces memory usage in UDTFs with PARTITION BY.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Manually.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes #59136 from craiuconstantintiberiu/udtf-partitions-release-init-info.
Authored-by: Constantin-Tiberiu Craiu <57...@users.noreply.github.com>
Signed-off-by: Tian Gao <ga...@hotmail.com>
---
python/pyspark/worker.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/python/pyspark/worker.py b/python/pyspark/worker.py
index a957a5d673d4..80ac5ab95e24 100644
--- a/python/pyspark/worker.py
+++ b/python/pyspark/worker.py
@@ -599,7 +599,7 @@ def read_udtf(pickleSer, udtf_info, eval_type, runner_conf, eval_conf):
self._create_udtf: Callable = create_udtf
self._udtf = create_udtf()
self._prev_arguments: list = list()
- self._partition_child_indexes: list = udtf_info.partition_child_indexes
+ self._partition_child_indexes: list = partition_child_indexes
self._eval_raised_skip_rest_of_input_table: bool = False
def eval(self, *args, **kwargs) -> Iterator:
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org