You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ratis.apache.org by "runzhiwang (Jira)" <ji...@apache.org> on 2020/12/06 10:45:00 UTC
[jira] [Comment Edited] (RATIS-1207) Fix duplicated StreamMap.Key
[ https://issues.apache.org/jira/browse/RATIS-1207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17244709#comment-17244709 ]
runzhiwang edited comment on RATIS-1207 at 12/6/20, 10:44 AM:
--------------------------------------------------------------
[~szetszwo]
{code:java}
//DataStreamManagement.java
throw new IllegalStateException("Failed to create a new stream for " + request
+ " since a stream already exists: " + info);
{code}
I log channelId when read.
{code:java}
void read(DataStreamRequestByteBuf request, ChannelHandlerContext ctx,
CheckedFunction<RaftClientRequest, List<DataStreamOutputRpc>, IOException> getDataStreamOutput) {
LOG.debug("{}: read {}", this, request);
final ByteBuf buf = request.slice();
+ LOG.info("read channelId:" + ctx.channel().id() + " streamId:" + request.getStreamId() + " request:" + request.hashCode());
final StreamMap.Key key = new StreamMap.Key(ctx.channel().id(), request.getStreamId());
final StreamInfo info;
{code}
And execute the following command two times.
{code:java}
${BIN}/client.sh filestore datastream --size 1000 --numFiles 2 --bufferSize 1000 --type DirectByteBuffer --peers ${PEERS}
{code}
The output of log as follows. You can find channelId is same when I execute the second command.
{code:java}
n1.log:90:2020-12-06 18:37:25 INFO DataStreamManagement:341 - read channelId:447fad2c streamId:0 request:1248961502
n1.log:91:2020-12-06 18:37:25 INFO DataStreamManagement:341 - read channelId:447fad2c streamId:0 request:2107066539
n1.log:92:2020-12-06 18:37:25 INFO DataStreamManagement:341 - read channelId:447fad2c streamId:1 request:1812599771
n1.log:93:2020-12-06 18:37:25 INFO DataStreamManagement:341 - read channelId:447fad2c streamId:1 request:1675898283
n1.log:94:2020-12-06 18:37:27 INFO DataStreamManagement:341 - read channelId:447fad2c streamId:0 request:1443703901
n1.log:120:2020-12-06 18:37:27 INFO DataStreamManagement:341 - read channelId:447fad2c streamId:0 request:370837989
n1.log:121:2020-12-06 18:37:27 INFO DataStreamManagement:341 - read channelId:447fad2c streamId:1 request:2089140397
n1.log:149:2020-12-06 18:37:27 INFO DataStreamManagement:341 - read channelId:447fad2c streamId:1 request:2062159344
{code}
There is exception in the server's log.
{code:java}
2020-12-06 18:37:25 INFO DataStreamManagement:341 - read channelId:447fad2c streamId:0 request:1248961502
2020-12-06 18:37:25 INFO DataStreamManagement:341 - read channelId:447fad2c streamId:0 request:2107066539
2020-12-06 18:37:25 INFO DataStreamManagement:341 - read channelId:447fad2c streamId:1 request:1812599771
2020-12-06 18:37:25 INFO DataStreamManagement:341 - read channelId:447fad2c streamId:1 request:1675898283
2020-12-06 18:37:27 INFO DataStreamManagement:341 - read channelId:447fad2c streamId:0 request:1443703901
2020-12-06 18:37:27 WARN DataStreamManagement:329 - Failed to process DataStreamRequestByteBuf:STREAM_HEADER,id=0,offset=0,length=68
java.lang.IllegalStateException: Failed to create a new stream for DataStreamRequestByteBuf:STREAM_HEADER,id=0,offset=0,length=68 since a stream already exists: StreamInfo:RaftClientRequest:client-3E5877A91C29->n0@group-6F7570313233, cid=0, seq=0, DataStream, Message:220e0a0966696c652d38...(size=16)
at org.apache.ratis.netty.server.DataStreamManagement.read(DataStreamManagement.java:348)
at org.apache.ratis.netty.server.NettyServerStreamRpc$1.channelRead(NettyServerStreamRpc.java:185)
at org.apache.ratis.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at org.apache.ratis.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at org.apache.ratis.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
at org.apache.ratis.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:324)
at org.apache.ratis.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:296)
at org.apache.ratis.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at org.apache.ratis.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at org.apache.ratis.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
at org.apache.ratis.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at org.apache.ratis.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at org.apache.ratis.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at org.apache.ratis.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at org.apache.ratis.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at org.apache.ratis.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at org.apache.ratis.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at org.apache.ratis.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at org.apache.ratis.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at org.apache.ratis.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at org.apache.ratis.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at org.apache.ratis.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)
{code}
I think Netty ChannelId is globally unique, but the channel in server maybe reused, the channelId will be same when execute different client command.
was (Author: yjxxtd):
[~szetszwo] I log channelId when read.
{code:java}
void read(DataStreamRequestByteBuf request, ChannelHandlerContext ctx,
CheckedFunction<RaftClientRequest, List<DataStreamOutputRpc>, IOException> getDataStreamOutput) {
LOG.debug("{}: read {}", this, request);
final ByteBuf buf = request.slice();
+ LOG.info("read channelId:" + ctx.channel().id() + " streamId:" + request.getStreamId() + " request:" + request.hashCode());
final StreamMap.Key key = new StreamMap.Key(ctx.channel().id(), request.getStreamId());
final StreamInfo info;
{code}
And execute the following command two times.
{code:java}
${BIN}/client.sh filestore datastream --size 1000 --numFiles 2 --bufferSize 1000 --type DirectByteBuffer --peers ${PEERS}
{code}
The output of log as follows. You can find channelId is same when I execute the second command.
{code:java}
n1.log:90:2020-12-06 18:37:25 INFO DataStreamManagement:341 - read channelId:447fad2c streamId:0 request:1248961502
n1.log:91:2020-12-06 18:37:25 INFO DataStreamManagement:341 - read channelId:447fad2c streamId:0 request:2107066539
n1.log:92:2020-12-06 18:37:25 INFO DataStreamManagement:341 - read channelId:447fad2c streamId:1 request:1812599771
n1.log:93:2020-12-06 18:37:25 INFO DataStreamManagement:341 - read channelId:447fad2c streamId:1 request:1675898283
n1.log:94:2020-12-06 18:37:27 INFO DataStreamManagement:341 - read channelId:447fad2c streamId:0 request:1443703901
n1.log:120:2020-12-06 18:37:27 INFO DataStreamManagement:341 - read channelId:447fad2c streamId:0 request:370837989
n1.log:121:2020-12-06 18:37:27 INFO DataStreamManagement:341 - read channelId:447fad2c streamId:1 request:2089140397
n1.log:149:2020-12-06 18:37:27 INFO DataStreamManagement:341 - read channelId:447fad2c streamId:1 request:2062159344
{code}
There is exception in the server's log.
{code:java}
2020-12-06 18:37:25 INFO DataStreamManagement:341 - read channelId:447fad2c streamId:0 request:1248961502
2020-12-06 18:37:25 INFO DataStreamManagement:341 - read channelId:447fad2c streamId:0 request:2107066539
2020-12-06 18:37:25 INFO DataStreamManagement:341 - read channelId:447fad2c streamId:1 request:1812599771
2020-12-06 18:37:25 INFO DataStreamManagement:341 - read channelId:447fad2c streamId:1 request:1675898283
2020-12-06 18:37:27 INFO DataStreamManagement:341 - read channelId:447fad2c streamId:0 request:1443703901
2020-12-06 18:37:27 WARN DataStreamManagement:329 - Failed to process DataStreamRequestByteBuf:STREAM_HEADER,id=0,offset=0,length=68
java.lang.IllegalStateException: Failed to create a new stream for DataStreamRequestByteBuf:STREAM_HEADER,id=0,offset=0,length=68 since a stream already exists: StreamInfo:RaftClientRequest:client-3E5877A91C29->n0@group-6F7570313233, cid=0, seq=0, DataStream, Message:220e0a0966696c652d38...(size=16)
at org.apache.ratis.netty.server.DataStreamManagement.read(DataStreamManagement.java:348)
at org.apache.ratis.netty.server.NettyServerStreamRpc$1.channelRead(NettyServerStreamRpc.java:185)
at org.apache.ratis.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at org.apache.ratis.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at org.apache.ratis.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
at org.apache.ratis.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:324)
at org.apache.ratis.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:296)
at org.apache.ratis.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at org.apache.ratis.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at org.apache.ratis.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
at org.apache.ratis.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at org.apache.ratis.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at org.apache.ratis.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at org.apache.ratis.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at org.apache.ratis.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at org.apache.ratis.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at org.apache.ratis.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at org.apache.ratis.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at org.apache.ratis.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at org.apache.ratis.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at org.apache.ratis.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at org.apache.ratis.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)
{code}
I think Netty ChannelId is globally unique, but the channel in server maybe reused, the channelId will be same when execute different client command.
> Fix duplicated StreamMap.Key
> ----------------------------
>
> Key: RATIS-1207
> URL: https://issues.apache.org/jira/browse/RATIS-1207
> Project: Ratis
> Issue Type: Sub-task
> Reporter: runzhiwang
> Assignee: runzhiwang
> Priority: Major
>
> Currently we use channelId_streamId as the stream key, but if we start the first client, and finish the streaming, then we start the second client, error happens because of duplicated stream key. Because the channel id of the second client is same as the first client, and streamId is both 0, so duplicated stream key happens. So we can use clientid_streamid as the stream key, to achieve this, we need put clientid in DataStreamPacketImpl
> {code:java}
> final StreamMap.Key key = new StreamMap.Key(ctx.channel().id(), request.getStreamId());
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)