You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Wan Kun (Jira)" <ji...@apache.org> on 2023/06/29 05:05:00 UTC

[jira] [Updated] (SPARK-44239) Free memory allocated by huge column vector

     [ https://issues.apache.org/jira/browse/SPARK-44239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Wan Kun updated SPARK-44239:
----------------------------
    Summary: Free memory allocated by huge column vector  (was: Reclaim memory allocated by huge column vector)

> Free memory allocated by huge column vector
> -------------------------------------------
>
>                 Key: SPARK-44239
>                 URL: https://issues.apache.org/jira/browse/SPARK-44239
>             Project: Spark
>          Issue Type: Improvement
>          Components: SQL
>    Affects Versions: 3.5.0
>            Reporter: Wan Kun
>            Priority: Major
>         Attachments: image-2023-06-29-12-58-12-256.png, image-2023-06-29-13-03-15-470.png
>
>
> When spark reads a data file into a WritableColumnVector, the memory allocated by the WritableColumnVectors is not freed until the VectorizedColumnReader completes.
> It will save memory allocation time by reusing the allocated array objects. But it also takes up too many unused memory after the current large vector batch has been read.
> Add vector reserve policy for this scenario which will reuse the allocated array object for small column vectors and free the memory for huge column vectors.
> !image-2023-06-29-12-58-12-256.png!!image-2023-06-29-13-03-15-470.png!
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org