You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Eylon Ronen (Jira)" <ji...@apache.org> on 2024/03/31 10:11:00 UTC
[jira] [Created] (SPARK-47660) Memory leak in StructType column's .dropFields method
Eylon Ronen created SPARK-47660:
-----------------------------------
Summary: Memory leak in StructType column's .dropFields method
Key: SPARK-47660
URL: https://issues.apache.org/jira/browse/SPARK-47660
Project: Spark
Issue Type: Bug
Components: Java API, SQL
Affects Versions: 3.5.1
Reporter: Eylon Ronen
We noticed that when we're using the .dropFields method, for example by calling
df.withColum(column, col(column).dropFields("fieldToDrop")) the heap "explodes". we noticed it when our integration tests, for which 512MB heap is always enough started crashing with OutOfMemoryException and after a lot of trial and error we found that when we remove the .dropFields method the heap goes back downÂ
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org