You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@fluss.apache.org by "loserwang1024 (via GitHub)" <gi...@apache.org> on 2026/09/29 07:05:17 UTC
[I] Zstd Arrow codecs leak off-heap buffers when compression or decompression throws [fluss]
loserwang1024 opened a new issue, #4511:
URL: https://github.com/apache/fluss/issues/4511
### Search before asking
- [x] I searched in the [issues](https://github.com/apache/fluss/issues) and found nothing similar.
### Fluss version
0.9.0 (latest release)
### Please describe the bug 🐞
ZstdArrowCompressionCodec allocates an output ArrowBuf before calling
Zstd.compressDirectByteBuffer or Zstd.decompressDirectByteBuffer.
Currently, the output buffer is only closed when an error code is returned
or the decompressed length does not match. If the Zstd call throws directly,
execution bypasses these checks and the allocated output buffer is leaked.
UnshadedZstdArrowCompressionCodec has the same problem.
### Observed failure
A Flink sink write failed with:
```java
com.github.luben.zstd.ZstdException:Operation not authorized at current processing stage
at com.github.luben.zstd.ZstdCompressCtx.compressDirectByteBuffer(...)
at com.github.luben.zstd.Zstd.compressDirectByteBuffer(...)
at org.apache.fluss.compression.ZstdArrowCompressionCodec.doCompress(...)
```
The original Zstd failure and its trigger require separate investigation.
This issue concerns the output buffer leaked during exception handling.
### Solution
Use try/finally with a success flag immediately after output allocation
in both methods and both codec implementations. Close the output buffer
on failure; transfer ownership to the caller only on success.
### Are you willing to submit a PR?
- [x] I'm willing to submit a PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org