You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@shardingsphere.apache.org by GitBox <gi...@apache.org> on 2022/07/15 03:03:26 UTC
[GitHub] [shardingsphere] azexcy opened a new issue, #19183: Sometimes MySQLJsonBinlogProtocolValue.read() cause netty LEAK
azexcy opened a new issue, #19183:
URL: https://github.com/apache/shardingsphere/issues/19183
## Bug Report
Sometimes, when execute `stop scaling xxx` , Execute data insert at the same time. will cause netty leak error.
According to https://netty.io/wiki/reference-counted-objects.html I add the maven option `-Dio.netty.leakDetectionLevel=advanced`, then stack is
```
Error: 2022-07-15 02:11:43.485 [docker-java-stream--1043583587] :Scaling-Proxy - STDOUT: [ERROR] 2022-07-15 02:11:43.412 [nioEventLoopGroup-8-1] io.netty.util.ResourceLeakDetector - LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
[INFO ] 2022-07-15 02:11:43.485 [docker-java-stream--1043583587] :Scaling-Proxy - STDOUT: Recent access records:
[INFO ] 2022-07-15 02:11:43.485 [docker-java-stream--1043583587] :Scaling-Proxy - STDOUT: Created at:
[INFO ] 2022-07-15 02:11:43.485 [docker-java-stream--1043583587] :Scaling-Proxy - STDOUT: io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:401)
[INFO ] 2022-07-15 02:11:43.485 [docker-java-stream--1043583587] :Scaling-Proxy - STDOUT: io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:188)
[INFO ] 2022-07-15 02:11:43.485 [docker-java-stream--1043583587] :Scaling-Proxy - STDOUT: io.netty.buffer.AbstractByteBufAllocator.buffer(AbstractByteBufAllocator.java:124)
[INFO ] 2022-07-15 02:11:43.485 [docker-java-stream--1043583587] :Scaling-Proxy - STDOUT: io.netty.buffer.AbstractByteBuf.readBytes(AbstractByteBuf.java:871)
[INFO ] 2022-07-15 02:11:43.485 [docker-java-stream--1043583587] :Scaling-Proxy - STDOUT: org.apache.shardingsphere.db.protocol.mysql.packet.binlog.row.column.value.string.MySQLJsonBinlogProtocolValue.read(MySQLJsonBinlogProtocolValue.java:42)
[INFO ] 2022-07-15 02:11:43.485 [docker-java-stream--1043583587] :Scaling-Proxy - STDOUT: org.apache.shardingsphere.db.protocol.mysql.packet.binlog.row.MySQLBinlogRowsEventPacket.readRow(MySQLBinlogRowsEventPacket.java:106)
[INFO ] 2022-07-15 02:11:43.485 [docker-java-stream--1043583587] :Scaling-Proxy - STDOUT: org.apache.shardingsphere.db.protocol.mysql.packet.binlog.row.MySQLBinlogRowsEventPacket.readRows(MySQLBinlogRowsEventPacket.java:94)
[INFO ] 2022-07-15 02:11:43.485 [docker-java-stream--1043583587] :Scaling-Proxy - STDOUT: org.apache.shardingsphere.data.pipeline.mysql.ingest.client.netty.MySQLBinlogEventPacketDecoder.decodeWriteRowsEventV2(MySQLBinlogEventPacketDecoder.java:175)
[INFO ] 2022-07-15 02:11:43.485 [docker-java-stream--1043583587] :Scaling-Proxy - STDOUT: org.apache.shardingsphere.data.pipeline.mysql.ingest.client.netty.MySQLBinlogEventPacketDecoder.decodeEvent(MySQLBinlogEventPacketDecoder.java:99)
[INFO ] 2022-07-15 02:11:43.485 [docker-java-stream--1043583587] :Scaling-Proxy - STDOUT: org.apache.shardingsphere.data.pipeline.mysql.ingest.client.netty.MySQLBinlogEventPacketDecoder.decode(MySQLBinlogEventPacketDecoder.java:72)
[INFO ] 2022-07-15 02:11:43.485 [docker-java-stream--1043583587] :Scaling-Proxy - STDOUT: io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:510)
[INFO ] 2022-07-15 02:11:43.485 [docker-java-stream--1043583587] :Scaling-Proxy - STDOUT: io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:449)
[INFO ] 2022-07-15 02:11:43.485 [docker-java-stream--1043583587] :Scaling-Proxy - STDOUT: io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:279)
[INFO ] 2022-07-15 02:11:43.485 [docker-java-stream--1043583587] :Scaling-Proxy - STDOUT: io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
[INFO ] 2022-07-15 02:11:43.485 [docker-java-stream--1043583587] :Scaling-Proxy - STDOUT: io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
[INFO ] 2022-07-15 02:11:43.485 [docker-java-stream--1043583587] :Scaling-Proxy - STDOUT: io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
[INFO ] 2022-07-15 02:11:43.485 [docker-java-stream--1043583587] :Scaling-Proxy - STDOUT: io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:327)
[INFO ] 2022-07-15 02:11:43.485 [docker-java-stream--1043583587] :Scaling-Proxy - STDOUT: io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:314)
[INFO ] 2022-07-15 02:11:43.485 [docker-java-stream--1043583587] :Scaling-Proxy - STDOUT: io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:435)
[INFO ] 2022-07-15 02:11:43.485 [docker-java-stream--1043583587] :Scaling-Proxy - STDOUT: io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:279)
[INFO ] 2022-07-15 02:11:43.485 [docker-java-stream--1043583587] :Scaling-Proxy - STDOUT: io.netty.handler.codec.ByteToMessageCodec.channelRead(ByteToMessageCodec.java:103)
[INFO ] 2022-07-15 02:11:43.485 [docker-java-stream--1043583587] :Scaling-Proxy - STDOUT: io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
[INFO ] 2022-07-15 02:11:43.485 [docker-java-stream--1043583587] :Scaling-Proxy - STDOUT: io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
[INFO ] 2022-07-15 02:11:43.485 [docker-java-stream--1043583587] :Scaling-Proxy - STDOUT: io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
[INFO ] 2022-07-15 02:11:43.485 [docker-java-stream--1043583587] :Scaling-Proxy - STDOUT: io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
[INFO ] 2022-07-15 02:11:43.485 [docker-java-stream--1043583587] :Scaling-Proxy - STDOUT: io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
[INFO ] 2022-07-15 02:11:43.485 [docker-java-stream--1043583587] :Scaling-Proxy - STDOUT: io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
[INFO ] 2022-07-15 02:11:43.485 [docker-java-stream--1043583587] :Scaling-Proxy - STDOUT: io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
[INFO ] 2022-07-15 02:11:43.485 [docker-java-stream--1043583587] :Scaling-Proxy - STDOUT: io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
[INFO ] 2022-07-15 02:11:43.485 [docker-java-stream--1043583587] :Scaling-Proxy - STDOUT: io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:722)
[INFO ] 2022-07-15 02:11:43.485 [docker-java-stream--10435[835](https://github.com/apache/shardingsphere/runs/7350927587?check_suite_focus=true#step:6:836)87] :Scaling-Proxy - STDOUT: io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:658)
[INFO ] 2022-07-15 02:11:43.485 [docker-java-stream--1043583587] :Scaling-Proxy - STDOUT: io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:584)
[INFO ] 2022-07-15 02:11:43.485 [docker-java-stream--1043583587] :Scaling-Proxy - STDOUT: io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496)
[INFO ] 2022-07-15 02:11:43.485 [docker-java-stream--1043583587] :Scaling-Proxy - STDOUT: io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
[INFO ] 2022-07-15 02:11:43.486 [docker-java-stream--1043583587] :Scaling-Proxy - STDOUT: io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
[INFO ] 2022-07-15 02:11:43.486 [docker-java-stream--1043583587] :Scaling-Proxy - STDOUT: io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
[INFO ] 2022-07-15 02:11:43.486 [docker-java-stream--1043583587] :Scaling-Proxy - STDOUT: java.lang.Thread.run(Thread.java:748)
```
Relation ci link : https://github.com/apache/shardingsphere/runs/7350927587?check_suite_focus=true
But local runs can't reproduce consistently
### Which version of ShardingSphere did you use?
master
### Which project did you use? ShardingSphere-JDBC or ShardingSphere-Proxy?
ShardingSphere-Proxy
### Expected behavior
no netty error
### Actual behavior
throw leak error
### Reason analyze (If you can)
```
/**
* Transfers this buffer's data to a newly created buffer starting at
* the current {@code readerIndex} and increases the {@code readerIndex}
* by the number of the transferred bytes (= {@code length}).
* The returned buffer's {@code readerIndex} and {@code writerIndex} are
* {@code 0} and {@code length} respectively.
*
* @param length the number of bytes to transfer
*
* @return the newly created buffer which contains the transferred bytes
*
* @throws IndexOutOfBoundsException
* if {@code length} is greater than {@code this.readableBytes}
*/
public abstract ByteBuf readBytes(int length);
```
`readBytes()` wiil Returns: the newly created buffer, so need release()
According to the stack error, manual release ByteBuf
### Steps to reproduce the behavior, such as: SQL to execute, sharding rule configuration, when exception occur etc.
### Example codes for reproduce this issue (such as a github link).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] sandynz closed issue #19183: Sometimes MySQLJsonBinlogProtocolValue.read() cause netty LEAK
Posted by GitBox <gi...@apache.org>.
sandynz closed issue #19183: Sometimes MySQLJsonBinlogProtocolValue.read() cause netty LEAK
URL: https://github.com/apache/shardingsphere/issues/19183
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org