You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Casimir Giesler (Jira)" <ji...@apache.org> on 2026/02/07 18:10:00 UTC

[jira] [Created] (SPARK-55416) Streaming Python Data Source memory leak when is offset not updated

Casimir Giesler created SPARK-55416:
---------------------------------------

             Summary: Streaming Python Data Source memory leak when is offset not updated
                 Key: SPARK-55416
                 URL: https://issues.apache.org/jira/browse/SPARK-55416
             Project: Spark
          Issue Type: Bug
          Components: PySpark, Structured Streaming
    Affects Versions: 4.1.1, 4.0.2, 4.0.1, 4.0.0, 4.1.0
            Reporter: Casimir Giesler


This only becomes a bug / memory leak if users implement the offset incorrectly, never increasing it.

The commit logic in [datasource_internal.py|https://github.com/apache/spark/blob/master/python/pyspark/sql/datasource_internal.py#L106-L114] fails to clean up cache if the offset never increases.

In this case, end will always hits at the first cache item, failing the {{end_idx > 0}} condition and not cleaning up cache, while the {{latestOffset}} function will continue to append to the cache.

This leads to an infinitely growing cache and will ultimately result in OOM.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org