[jira] [Created] (SPARK-52809) Don't hold reader and iterator references for all partitions in task completion listeners for metric update

["L. C. Hsieh (Jira)" <ji...@apache.org>]

L. C. Hsieh created SPARK-52809:
-----------------------------------

             Summary: Don't hold reader and iterator references for all partitions in task completion listeners for metric update
                 Key: SPARK-52809
                 URL: https://issues.apache.org/jira/browse/SPARK-52809
             Project: Spark
          Issue Type: Bug
          Components: SQL
    Affects Versions: 3.4.4
            Reporter: L. C. Hsieh


For each partition iterator, currently we add one task completion listener used to update final metrics if the task is stopped early. In the listener, the reader and iterator are held. So if the partition is normally exhausted, the references cannot be released early. It is a problem especially if the references are heavy as reported by https://github.com/apache/iceberg/issues/13297.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org