You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@uniffle.apache.org by "jerqi (via GitHub)" <gi...@apache.org> on 2024/12/04 12:11:00 UTC
[I] [Flaky Test] Throw an io.netty.util.IllegalReferenceCountException [incubator-uniffle]
jerqi opened a new issue, #2273:
URL: https://github.com/apache/incubator-uniffle/issues/2273
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
### Search before asking
- [X] I have searched in the [issues](https://github.com/apache/incubator-uniffle/issues?q=is%3Aissue) and found no similar issues.
### Describe the flaky test
Caused by: io.netty.util.IllegalReferenceCountException: refCnt: 0
at io.netty.buffer.AbstractByteBuf.ensureAccessible(AbstractByteBuf.java:1454)
at io.netty.buffer.AbstractByteBuf.checkIndex(AbstractByteBuf.java:1383)
at io.netty.buffer.PooledByteBuf.duplicateInternalNioBuffer(PooledByteBuf.java:196)
at io.netty.buffer.PooledByteBuf.nioBuffer(PooledByteBuf.java:213)
at io.netty.buffer.AbstractByteBuf.nioBuffer(AbstractByteBuf.java:1231)
at org.apache.uniffle.common.netty.buffer.NettyManagedBuffer.nioByteBuffer(NettyManagedBuffer.java:48)
at org.apache.uniffle.common.ShuffleIndexResult.getIndexData(ShuffleIndexResult.java:84)
### Actions URL
https://github.com/apache/incubator-uniffle/actions/runs/12158115239/attempts/1
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [I] [Flaky Test] Throw an io.netty.util.IllegalReferenceCountException [incubator-uniffle]
Posted by "jerqi (via GitHub)" <gi...@apache.org>.
jerqi commented on issue #2273:
URL: https://github.com/apache/incubator-uniffle/issues/2273#issuecomment-2518998873
> Users can choose whatever kind of bytebuf they want: client: #1767 server: #2145
>
> We basically will release every bytebuf, but it does not mean that we don't need PooledByteBuf. We use PooledByteBuf to reduce the number of interactions with the operating system for memory allocation and deallocation. This helps to avoid potential issues such as abnormal growth of RSS (Resident Set Size) memory, which might occur due to frequent and inefficient memory management (especially when using glibc, that's why we recommend using [mimalloc](https://github.com/apache/incubator-uniffle/blob/master/docs/server_guide.md#malloc-recommendation)).
But it will throw a ref count exception. I suspect that we released a reused bytebuf in the ByteBuf pool.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [I] [Flaky Test] Throw an io.netty.util.IllegalReferenceCountException [incubator-uniffle]
Posted by "jerqi (via GitHub)" <gi...@apache.org>.
jerqi commented on issue #2273:
URL: https://github.com/apache/incubator-uniffle/issues/2273#issuecomment-2519707538
> I didn't look into the code carefully. I suppose this should be a bug. @maobaolong
>
> Similar to #1628?
Yes, they are similar.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [I] [Flaky Test] Throw an io.netty.util.IllegalReferenceCountException [incubator-uniffle]
Posted by "jerqi (via GitHub)" <gi...@apache.org>.
jerqi commented on issue #2273:
URL: https://github.com/apache/incubator-uniffle/issues/2273#issuecomment-2517186016
Maybe we shouldn't use PooledByteBuf. Because we will release the memory after we used the memory. You can see https://github.com/apache/incubator-uniffle/blob/28475fced12f5a4b79152d146ead1917982f44b9/storage/src/main/java/org/apache/uniffle/storage/handler/impl/DataSkippableReadHandler.java#L82
cc @advancedxy @rickyma WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [I] [Flaky Test] Throw an io.netty.util.IllegalReferenceCountException [incubator-uniffle]
Posted by "jerqi (via GitHub)" <gi...@apache.org>.
jerqi commented on issue #2273:
URL: https://github.com/apache/incubator-uniffle/issues/2273#issuecomment-2519782123
I suspect that the issue is caused by `TransportFrameDecoder`
```
@Override
public void handlerRemoved(ChannelHandlerContext ctx) throws Exception {
// Release all buffers that are still in our ownership.
// Doing this in handlerRemoved(...) guarantees that this will happen in all cases:
// - When the Channel becomes inactive
// - When the decoder is removed from the ChannelPipeline
for (ByteBuf b : buffers) {
b.release();
}
buffers.clear();
headerBuf.release();
super.handlerRemoved(ctx);
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [I] [Flaky Test] Throw an io.netty.util.IllegalReferenceCountException [incubator-uniffle]
Posted by "rickyma (via GitHub)" <gi...@apache.org>.
rickyma commented on issue #2273:
URL: https://github.com/apache/incubator-uniffle/issues/2273#issuecomment-2518964197
Users can choose whatever kind of bytebuf they want:
client: https://github.com/apache/incubator-uniffle/issues/1767
server: https://github.com/apache/incubator-uniffle/issues/2145
We basically will release every bytebuf, but it does not mean that we don't need PooledByteBuf.
We use PooledByteBuf to reduce the number of interactions with the operating system for memory allocation and deallocation. This helps to avoid potential issues such as abnormal growth of RSS (Resident Set Size) memory, which might occur due to frequent and inefficient memory management (especially when using glibc, that's why we recommend using [mimalloc](https://github.com/apache/incubator-uniffle/blob/master/docs/server_guide.md#malloc-recommendation)).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [I] [Flaky Test] Throw an io.netty.util.IllegalReferenceCountException [incubator-uniffle]
Posted by "rickyma (via GitHub)" <gi...@apache.org>.
rickyma commented on issue #2273:
URL: https://github.com/apache/incubator-uniffle/issues/2273#issuecomment-2519335449
This is a bug. @maobaolong
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org