You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "zhangkre (via GitHub)" <gi...@apache.org> on 2026/07/18 07:00:12 UTC
[I] [Java] VectorUnloader has multiple memory leak vulnerabilities [arrow-java]
zhangkre opened a new issue, #1234:
URL: https://github.com/apache/arrow-java/issues/1234
### Describe the bug, including details regarding any error messages, version, and platform.
### Problem Description
The VectorUnloader class has serious memory leak issues when serializing Vector data. Any exception thrown during the process will cause previously retained ArrowBuf instances to never be released.
Root Cause
ArrowBuf uses reference counting for memory management. After calling retain(), a corresponding release() must be called to free the buffer. The code has exception-safety defects in multiple locations:
---
#### Bug 1: OOM during compression causes leak
```
Location: appendNodes() method, around line 75
buf.getReferenceManager().retain();
buffers.add(codec.compress(vector.getAllocator(), buf)); // OOM here causes leak
Problem: If codec.compress() throws OutOfMemoryError, retain() has already been called but buffers.add() never executes, making the ArrowBuf unreleasable.
```
---
#### Bug 2: Compression failure in loop causes partial leak
```
Location: appendNodes() method, around lines 78-85
for (ArrowBuf buf : fieldBuffers) {
buf.getReferenceManager().retain();
buffers.add(codec.compress(vector.getAllocator(), buf)); // Any failure leaks all previously retained buffers
}
Problem: Any compression failure in the loop causes all previously retained buffers to leak.
```
---
#### Bug 3: Exception during recursive child vector processing causes leak
```
Location: appendNodes() method, around lines 87-89
for (FieldVector child : vector.getChildrenFromFields()) {
appendNodes(child, nodes, buffers, variadicBufferCounts); // Exception here leaks buffers from current level
}
Problem: An exception during recursive child processing prevents release of buffers already retained at the current level.
```
---
#### Bug 4: Iteration failure in top-level loop causes cumulative leaks
```
Location: getRecordBatch() method, around lines 53-58
for (FieldVector vector : root.getFieldVectors()) {
appendNodes(vector, nodes, buffers, variadicBufferCounts); // Any failure leaks all previously retained buffers
}
Problem: Each iteration retains buffers, and any failure causes cumulative leaks of all previously retained buffers. This is an accumulated leak.
```
---
### Impact
- All code using VectorUnloader.getRecordBatch()
- Leak probability increases significantly when processing large datasets or when compressor runs low on memory
- Leaked ArrowBuf instances persist until process termination
Suggested Fix
1. Use try-finally to ensure release() is called even when exceptions occur
2. Or add buffers to the list before calling retain() and compress()
3. Use transactional error handling for the entire getRecordBatch() operation, rolling back all buffers on failure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Java] VectorUnloader has multiple memory leak vulnerabilities [arrow-java]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on issue #1234:
URL: https://github.com/apache/arrow-java/issues/1234#issuecomment-5010846352
@zhangkre if you have vulnerabilities to disclose, please follow the ASF process: https://github.com/apache/arrow-java?tab=security-ov-file
That said, we've been discussing the security policy for Arrow: https://arrow.apache.org/docs/format/Security.html
It's not clear to me that we'd consider memory leaks as vulnerabilities, unless it can be shown that an attacker can cause excessive memory usage with a very small or trivial input. Given that the IPC format allows (de)compression, unbounded memory usage is always possible and is something that should be mitigated in other ways, e.g. bounding the allocator and tying it to a health check on a service (although we should absolutely fix cases like these where possible and document this more clearly).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Java] VectorUnloader has multiple memory leak issues [arrow-java]
Posted by "jbonofre (via GitHub)" <gi...@apache.org>.
jbonofre closed issue #1234: [Java] VectorUnloader has multiple memory leak issues
URL: https://github.com/apache/arrow-java/issues/1234
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Java] VectorUnloader has multiple memory leak vulnerabilities [arrow-java]
Posted by "zhangkre (via GitHub)" <gi...@apache.org>.
zhangkre commented on issue #1234:
URL: https://github.com/apache/arrow-java/issues/1234#issuecomment-5010380150
fixed code
```java
public ArrowRecordBatch getRecordBatch() {
List<ArrowFieldNode> nodes = new ArrayList<>();
List<ArrowBuf> buffers = new ArrayList<>();
List<Long> variadicBufferCounts = new ArrayList<>();
try {
for (FieldVector vector : root.getFieldVectors()) {
appendNodes(vector, nodes, buffers, variadicBufferCounts);
}
return new ArrowRecordBatch(
root.getRowCount(),
nodes,
buffers,
CompressionUtil.createBodyCompression(codec),
variadicBufferCounts,
alignBuffers,
false);
} catch (RuntimeException e) {
RuntimeException releaseException = releaseBuffers(buffers);
if (releaseException != null) {
e.addSuppressed(releaseException);
}
throw e;
}
}
private RuntimeException releaseBuffers(List<ArrowBuf> buffers) {
RuntimeException firstException = null;
for (ArrowBuf buf : buffers) {
try {
buf.getReferenceManager().release();
} catch (RuntimeException e) {
if (firstException == null) {
firstException = e;
} else {
firstException.addSuppressed(e);
}
}
}
return firstException;
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Java] VectorUnloader has multiple memory leak vulnerabilities [arrow-java]
Posted by "zhangkre (via GitHub)" <gi...@apache.org>.
zhangkre commented on issue #1234:
URL: https://github.com/apache/arrow-java/issues/1234#issuecomment-5011246494
> [@zhangkre](https://github.com/zhangkre) if you have vulnerabilities to disclose, please follow the ASF process: https://github.com/apache/arrow-java?tab=security-ov-file
>
> That said, we've been discussing the security policy for Arrow: https://arrow.apache.org/docs/format/Security.html
>
> It's not clear to me that we'd consider memory leaks as vulnerabilities, unless it can be shown that an attacker can cause excessive memory usage with a very small or trivial input. Given that the IPC format allows (de)compression, unbounded memory usage is always possible and is something that should be mitigated in other ways, e.g. bounding the allocator and tying it to a health check on a service (although we should absolutely fix cases like these where possible and document this more clearly).
I appreciate the clarification. In my view:
1. This is a functional issue, not a security vulnerability - The memory leak affects functionality (e.g., resource exhaustion, service degradation) rather than exposing any security boundary.
I suggest we:
- Fix when possible - Continue addressing avoidable memory leaks in the code
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org