You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@uniffle.apache.org by "rickyma (via GitHub)" <gi...@apache.org> on 2024/01/19 04:04:45 UTC
[I] [Serious Bug] Inaccurate flow control leads to Shuffle server OOM [incubator-uniffle]
rickyma opened a new issue, #1472:
URL: https://github.com/apache/incubator-uniffle/issues/1472
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
### Search before asking
- [X] I have searched in the [issues](https://github.com/apache/incubator-uniffle/issues?q=is%3Aissue) and found no similar issues.
### Describe the bug
In high-pressure scenarios, inaccurate flow control(usedMemory? preAllocatedMemory?) leads to Shuffle server OOM.
The SQL used to reproduce the bug:
tpcds：
select * from (
select s.*,c.* from store_sales s join customer c on s.ss_customer_sk=c.c_customer_sk
) sc DISTRIBUTE BY sc.ss_customer_sk,sc.ss_item_sk;
### Affects Version(s)
master
### Uniffle Server Log Output
```logtalk
[2024-01-19 03:18:50.483] [Grpc-714] [DEBUG] org.apache.uniffle.server.buffer.ShuffleBufferManager.requireMemory - Require memory succeeded with 1023891 bytes, usedMemory[117888810505] include preAllocation[180869913], inFlushSize[110004009963]
[2024-01-19 03:18:50.485] [Grpc-19] [DEBUG] org.apache.uniffle.server.buffer.ShuffleBufferManager.requireMemory - Require memory succeeded with 3093288 bytes, usedMemory[117891903793] include preAllocation[183963201], inFlushSize[110004009963]
[2024-01-19 03:18:50.485] [epollEventLoopGroup-3-19] [DEBUG] org.apache.uniffle.server.netty.ShuffleServerNettyHandler.handleSendShuffleDataRequest - Cache Shuffle Data for appId[application_1703049085550_1651917_1705170927963], shuffleId[0], cost 0 ms with 52 blocks and 1189625 bytes
[2024-01-19 03:18:50.487] [epollEventLoopGroup-3-44] [WARN] org.apache.uniffle.common.netty.handle.TransportChannelHandler.exceptionCaught - Exception in connection from /9.23.12.144:45176
io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 4194304 byte(s) of direct memory (used: 171798691840, max: 171798691840)
at io.netty.util.internal.PlatformDependent.incrementMemoryCounter(PlatformDependent.java:843)
at io.netty.util.internal.PlatformDependent.allocateDirectNoCleaner(PlatformDependent.java:772)
at io.netty.buffer.PoolArena$DirectArena.allocateDirect(PoolArena.java:710)
at io.netty.buffer.PoolArena$DirectArena.newChunk(PoolArena.java:685)
at io.netty.buffer.PoolArena.allocateNormal(PoolArena.java:212)
at io.netty.buffer.PoolArena.tcacheAllocateNormal(PoolArena.java:194)
at io.netty.buffer.PoolArena.allocate(PoolArena.java:136)
at io.netty.buffer.PoolArena.allocate(PoolArena.java:126)
at io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:397)
at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:188)
at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:179)
at org.apache.uniffle.common.netty.protocol.Decoders.decodeShuffleBlockInfo(Decoders.java:50)
at org.apache.uniffle.common.netty.protocol.SendShuffleDataRequest.decodePartitionData(SendShuffleDataRequest.java:92)
at org.apache.uniffle.common.netty.protocol.SendShuffleDataRequest.decode(SendShuffleDataRequest.java:104)
at org.apache.uniffle.common.netty.protocol.Message.decode(Message.java:145)
at org.apache.uniffle.common.netty.TransportFrameDecoder.channelRead(TransportFrameDecoder.java:72)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:800)
at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:509)
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:407)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:750)
```
### Uniffle Engine Log Output
_No response_
### Uniffle Server Configurations
```yaml
xmx：120g
capacity：110g
read.capacity：20g
max_direct_mem：160g
rss.server.netty.epoll.enable true
rss.rpc.server.type GRPC_NETTY
```
### Uniffle Engine Configurations
```yaml
set spark.sql.files.maxPartitionBytes=1073741824;
set spark.executor.cores=8;
set spark.task.cpus=4;
set spark.executor.memory=49g;
set spark.driver.memory=20g;
set spark.dynamicAllocation.maxExecutors=150;
set spark.dynamicAllocation.minExecutors=150;
set spark.rss.client.type = GRPC_NETTY;
set spark.rss.client.netty.io.mode = EPOLL;
```
### Additional context
_No response_
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [I] [Serious Bug] Inaccurate flow control leads to Shuffle server OOM [incubator-uniffle]
Posted by "zuston (via GitHub)" <gi...@apache.org>.
zuston commented on issue #1472:
URL: https://github.com/apache/incubator-uniffle/issues/1472#issuecomment-1899797215
PTAL @leixm @jerqi
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [I] [Serious Bug] Inaccurate flow control leads to Shuffle server OOM when enabling Netty [incubator-uniffle]
Posted by "rickyma (via GitHub)" <gi...@apache.org>.
rickyma commented on issue #1472:
URL: https://github.com/apache/incubator-uniffle/issues/1472#issuecomment-1919567252
Progress: I already have a version that can solve this problem, and I am still testing it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [I] [Serious Bug] Inaccurate flow control leads to Shuffle server OOM [incubator-uniffle]
Posted by "jerqi (via GitHub)" <gi...@apache.org>.
jerqi commented on issue #1472:
URL: https://github.com/apache/incubator-uniffle/issues/1472#issuecomment-1909249809
> This only affects off heap Netty?
Currently, it only affects off heap Netty.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [I] [Serious Bug] Inaccurate flow control leads to Shuffle server OOM when enabling Netty [incubator-uniffle]
Posted by "rickyma (via GitHub)" <gi...@apache.org>.
rickyma closed issue #1472: [Serious Bug] Inaccurate flow control leads to Shuffle server OOM when enabling Netty
URL: https://github.com/apache/incubator-uniffle/issues/1472
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [I] [Serious Bug] Inaccurate flow control leads to Shuffle server OOM [incubator-uniffle]
Posted by "connorlwilkes (via GitHub)" <gi...@apache.org>.
connorlwilkes commented on issue #1472:
URL: https://github.com/apache/incubator-uniffle/issues/1472#issuecomment-1907982293
This only affects off heap Netty?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org