You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "frroliveira (via GitHub)" <gi...@apache.org> on 2026/10/09 22:59:36 UTC
[PR] [SPARK-60098][SQL] Fix memory leaks in ParquetColumnVector [spark]
frroliveira opened a new pull request, #59329:
URL: https://github.com/apache/spark/pull/59329
### What changes were proposed in this pull request?
This tries to ensure that off-heap memory allocated by the Parquet reader is deallocated. The class `ParquetColumnVector` uses the method `WritableColumnVector.reserveNewColumn` in 4 scenarios:
- Repetition levels
- Definition levels
- Dummy vector to handle missing structs
- Variant type
Given these column vectors don't belong to `ColumnarBatch`es and the class is not calling `WritableColumnVector.close` when memory mode is OFF_HEAP these column vectors are leaking.
To fix the method `VectorizedParquetRecordReader.close` calls the new method in `ParquetColumnVector.close`. This last one mimics the logic in the constructor of `ParquetColumnVector` to deallocate memory of the 4 scenarios identified above.
### Why are the changes needed?
This will prevent OOMs for long running applications.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Given there is no way to track/debug memory allocations in `OffHeapColumnVector` the change was tested manually:
- Added println to invocations of `WritableColumnVector.reserveNewColumn` in `ParquetColumnVector`
- Added println to invocations of `WritableColumnVector.close` in `ParquetColumnVector`
- Run the suites
- org.apache.spark.sql.execution.datasources.parquet.ParquetIOSuite
- org.apache.spark.sql.VariantSuite
- Ensured that the number of occurrences of each scenario matched correctly
The rest of the test suite was able to catch issues (fixed now)
- org.apache.spark.sql.execution.datasources.PushVariantIntoScanV2VectorizedSuite
- org.apache.spark.sql.execution.WholeStageCodegenSuite
- org.apache.spark.sql.execution.datasources.parquet.ParquetStorageFilterSuite
### Was this patch authored or co-authored using generative AI tooling?
No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-60098][SQL] Fix memory leaks in ParquetColumnVector [spark]
Posted by "frroliveira (via GitHub)" <gi...@apache.org>.
frroliveira commented on PR #59329:
URL: https://github.com/apache/spark/pull/59329#issuecomment-6132944561
@HyukjinKwon Thank you for your review!
Given your feedback I concluded my approach is too error prone.
So I have taken a different approach in another branch https://github.com/apache/spark/compare/master...frroliveira:spark:fix-parquet-vector-closure?expand=1
Should I fix this PR or create a new one?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-60098][SQL] Fix memory leaks in ParquetColumnVector [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #59329:
URL: https://github.com/apache/spark/pull/59329#discussion_r4237879075
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedParquetRecordReader.java:
##########
@@ -231,6 +231,12 @@ public void initialize(
@Override
public void close() throws IOException {
+ if (columnVectors != null) {
+ for (ParquetColumnVector columnVector : columnVectors) {
+ columnVector.close();
Review Comment:
**Non-blocking (P2):** `initBatch` assigns `columnVectors = new ParquetColumnVector[n]` before the constructor loop fills it. If a `ParquetColumnVector` constructor throws partway through, the later slots stay null. Examples are `INVALID_VARIANT_SHREDDING_SCHEMA` from `buildVariantSchema` on a malformed shredded file, and the `IllegalArgumentException` for an unsupported existence default. `ParquetFileFormat.buildVectorizedIterator` then calls `iter.close()` in its SPARK-23457 catch block, and this loop throws `NullPointerException` on the null slot. The NPE replaces the original error before `throw e` runs. It also skips `columnarBatch.close()` and `super.close()`, so on exactly the failure path this cleanup exists for, the batch and the Parquet input stream leak.
Skipping null entries, and running the rest of the cleanup in a `finally` block as `LateMaterializationParquetRecordReader.close()` does, would keep both the original error and the existing cleanup.
**Verification:**
- **Regression:** When initBatch fails partway through (for example, an unsupported existence-default value or an invalid variant shredding schema), the original error is surfaced and no NullPointerException is raised from close.
- **Behavior:** Normal reads still release the batch and file reader exactly once, and repeated close() calls remain safe.
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/ParquetColumnVector.java:
##########
@@ -39,6 +39,11 @@ final class ParquetColumnVector {
private final List<ParquetColumnVector> children;
private final WritableColumnVector vector;
+ // Cache of number of children in vector for use in `close`. Call to `close` is not guaranteed to
+ // happen before `ColumnarBatch.close` which dereferences `WritableColumnVector.childColumns`
Review Comment:
**Nit (P3):** `WritableColumnVector.close()` sets `childColumns` to null rather than dereferencing it. It is the later `getNumChildren()` call that dereferences it and throws. Also, `VectorizedParquetRecordReader.close()` runs this before `columnarBatch.close()`. The caller that closes the batch's vectors first is `LateMaterializationParquetRecordReader.close()`, via `closeAll(persistentBatchColumns)` before `super.close()`. Naming that caller would make it clear why the count is cached.
**Verification:**
- **Inspection:** The comment accurately describes why the child count is cached: the late-materialization close ordering, and WritableColumnVector.close() nulling childColumns.
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/ParquetColumnVector.java:
##########
@@ -231,6 +238,30 @@ void reset() {
}
}
+ void close() {
+ // Calls `ColumnVector.close` for scenarios using `reserveNewColumn`.
+ if (column.variantFileType().isDefined()) {
+ ParquetColumnVector child = children.get(0);
Review Comment:
**Blocking (P1):** When a requested variant column is missing from the file and has no default, the constructor returns early at `vector.setMissing(); return;`, before the branch that adds this file-content child. `column.variantFileType()` is still defined in that case. `clipParquetGroupFields` synthesizes the missing field as a value/metadata group, and `ParquetToSparkSchemaConverter` sets `variantFileType` because `spark.sql.variant.allowReadingShredded` is on by default. So `children` is empty here, and `children.get(0)` throws `IndexOutOfBoundsException` when the reader is closed at the end of the split.
This is reachable with ordinary reads, for example `spark.read.schema("id INT, v VARIANT").parquet(<files written with only id>)`, a Parquet table that gained a `VARIANT` column after older files were written, or a variant field missing inside a struct that is present. Those reads return NULLs today and fail the task with this change. Because the exception escapes before `columnarBatch.close()` and `super.close()`, the batch memory and the file stream also leak.
`assemble()` already avoids this by checking `variantSchema != null`, which is only set after the child is added. Using the same condition here would make close() a no-op for the missing column.
**Verification:**
- **Regression:** Reading a file that lacks a requested variant column (top-level and nested in a present struct) through the vectorized reader succeeds and returns NULLs, in both on-heap and off-heap column vector modes.
- **Behavior:** Reading present shredded and unshredded variant columns returns the same results as before.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-60098][SQL] Fix memory leaks in ParquetColumnVector [spark]
Posted by "frroliveira (via GitHub)" <gi...@apache.org>.
frroliveira closed pull request #59329: [SPARK-60098][SQL] Fix memory leaks in ParquetColumnVector
URL: https://github.com/apache/spark/pull/59329
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org