[GitHub] [iceberg] Kontinuation commented on a diff in pull request #5735: Core/Spark: Fix memory leaks in vectorized reader for parquet

[GitBox <gi...@apache.org>]

Kontinuation commented on code in PR #5735:
URL: https://github.com/apache/iceberg/pull/5735#discussion_r967570968


##########
arrow/src/main/java/org/apache/iceberg/arrow/vectorized/VectorizedArrowReader.java:
##########
@@ -210,6 +210,9 @@ public VectorHolder read(VectorHolder reuse, int numValsToRead) {
   }
 
   private void allocateFieldVector(boolean dictionaryEncodedVector) {
+    if (vec != null) {
+      vec.close();
+    }

Review Comment:
   This is the fix for memory leaks when reading parquet files containing interleaving plain/dictionary pages.
   `VectorizedArrowReader.read` allocates arrow vectors when encoding of pages in currently reading row group changes, it does not close the previously allocated vector before allocating new vector, which causes memory leaks.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org