You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Eylon Ronen (Jira)" <ji...@apache.org> on 2024/04/01 07:27:00 UTC
[jira] [Updated] (SPARK-47660) Memory leak in StructType column's .dropFields method
[ https://issues.apache.org/jira/browse/SPARK-47660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eylon Ronen updated SPARK-47660:
--------------------------------
Priority: Major (was: Minor)
> Memory leak in StructType column's .dropFields method
> -----------------------------------------------------
>
> Key: SPARK-47660
> URL: https://issues.apache.org/jira/browse/SPARK-47660
> Project: Spark
> Issue Type: Bug
> Components: Java API, SQL
> Affects Versions: 3.5.1
> Reporter: Eylon Ronen
> Priority: Major
>
> We noticed that when we're using the .dropFields method, for example by calling
> df.withColum(column, col(column).dropFields("fieldToDrop")) the heap "explodes". we noticed it when our integration tests, for which 512MB heap is always enough started crashing with OutOfMemoryException (even when we increased the heap size to 8GB) and after a lot of trial and error we found that when we remove the .dropFields method the heap goes back downÂ
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org