You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by kineret M <ki...@gmail.com> on 2022/08/10 17:15:00 UTC

Memory leak while caching in foreachBatch block

Hi,

We have a structured streaming application, and we face a memory leak while
caching in the foreachBatch block.

We do unpersist every iteration, and we also verify via
"spark.sparkContext.getPersistentRDDs" that we don't have unnecessary
cached data.

We also noted in the profiler that many sparkSession objects are created
while we use cache (vs 2 sparkSession objects while not using cache).

Any idea what can cause this? We use spark 3.1.1