You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by 李建奇 <li...@jd.com> on 2015/11/10 03:50:30 UTC
org.apache.cassandra.net.OutboundTcpConnection.run(OutboundTcpConnection.java:218 throws java.lang.AssertionError
Hi, All,
We have a 12 nodes cluster with 2.1.9 version for near one month. Last
week it have an exception . Cluster’s write and read latency will go up to
4 seconds from 0.4ms average after exception.
I suspect OutboundTcpConnection is broken .I try to disablegossip then
enablegossip to rescue OutboundTcpConnction ,but fail .
I have to restart every node to restore cluster to normal . Cluster’s
load is low. The client use datastax java driver 2.1.7.1 . CF with RF 3.
Question:
Which situation to trigger this AssertionError? I read
OutboundTcpConnection.java source code, line 228 comments
“writeConnected() is reasonably robust”.
Thanks
Attachment :
ERROR [MessagingService-Outgoing-/172.20.114.13] 2015-11-08 10:36:28,763
CassandraDaemon.java:223 - Exception in thread
Thread[MessagingService-Outgoing-/172.20.114.13,5,main]
java.lang.AssertionError: 78251
at
org.apache.cassandra.utils.ByteBufferUtil.writeWithShortLength(ByteBufferUti
l.java:290) ~[apache-cassandra-2.1.9.jar:2.1.9]
at
org.apache.cassandra.db.composites.AbstractCType$Serializer.serialize(Abstra
ctCType.java:392) ~[apache-cassandra-2.1.9.jar:2.1.9]
at
org.apache.cassandra.db.composites.AbstractCType$Serializer.serialize(Abstra
ctCType.java:381) ~[apache-cassandra-2.1.9.jar:2.1.9]
at
org.apache.cassandra.db.filter.ColumnSlice$Serializer.serialize(ColumnSlice.
java:271) ~[apache-cassandra-2.1.9.jar:2.1.9]
at
org.apache.cassandra.db.filter.ColumnSlice$Serializer.serialize(ColumnSlice.
java:259) ~[apache-cassandra-2.1.9.jar:2.1.9]
at
org.apache.cassandra.db.filter.SliceQueryFilter$Serializer.serialize(SliceQu
eryFilter.java:503) ~[apache-cassandra-2.1.9.jar:2.1.9]
at
org.apache.cassandra.db.filter.SliceQueryFilter$Serializer.serialize(SliceQu
eryFilter.java:490) ~[apache-cassandra-2.1.9.jar:2.1.9]
at
org.apache.cassandra.db.SliceFromReadCommandSerializer.serialize(SliceFromRe
adCommand.java:168) ~[apache-cassandra-2.1.9.jar:2.1.9]
at
org.apache.cassandra.db.ReadCommandSerializer.serialize(ReadCommand.java:143
) ~[apache-cassandra-2.1.9.jar:2.1.9]
at
org.apache.cassandra.db.ReadCommandSerializer.serialize(ReadCommand.java:132
) ~[apache-cassandra-2.1.9.jar:2.1.9]
at org.apache.cassandra.net.MessageOut.serialize(MessageOut.java:121)
~[apache-cassandra-2.1.9.jar:2.1.9]
at
org.apache.cassandra.net.OutboundTcpConnection.writeInternal(OutboundTcpConn
ection.java:330) ~[apache-cassandra-2.1.9.jar:2.1.9]
at
org.apache.cassandra.net.OutboundTcpConnection.writeConnected(OutboundTcpCon
nection.java:282) ~[apache-cassandra-2.1.9.jar:2.1.9]
at
org.apache.cassandra.net.OutboundTcpConnection.run(OutboundTcpConnection.jav
a:218) ~[apache-cassandra-2.1.9.jar:2.1.9]
ERROR [MessagingService-Outgoing-/172.20.114.19] 2015-11-08 10:36:28,763
CassandraDaemon.java:223 - Exception in thread
Thread[MessagingService-Outgoing-/172.20.114.19,5,main]
java.lang.AssertionError: 78251
at
org.apache.cassandra.utils.ByteBufferUtil.writeWithShortLength(ByteBufferUti
l.java:290) ~[apache-cassandra-2.1.9.jar:2.1.9]
at
org.apache.cassandra.db.composites.AbstractCType$Serializer.serialize(Abstra
ctCType.java:392) ~[apache-cassandra-2.1.9.jar:2.1.9]
at
org.apache.cassandra.db.composites.AbstractCType$Serializer.serialize(Abstra
ctCType.java:381) ~[apache-cassandra-2.1.9.jar:2.1.9]
at
org.apache.cassandra.db.filter.ColumnSlice$Serializer.serialize(ColumnSlice.
java:271) ~[apache-cassandra-2.1.9.jar:2.1.9]
at
org.apache.cassandra.db.filter.ColumnSlice$Serializer.serialize(ColumnSlice.
java:259) ~[apache-cassandra-2.1.9.jar:2.1.9]
at
org.apache.cassandra.db.filter.SliceQueryFilter$Serializer.serialize(SliceQu
eryFilter.java:503) ~[apache-cassandra-2.1.9.jar:2.1.9]
at
org.apache.cassandra.db.filter.SliceQueryFilter$Serializer.serialize(SliceQu
eryFilter.java:490) ~[apache-cassandra-2.1.9.jar:2.1.9]
at
org.apache.cassandra.db.SliceFromReadCommandSerializer.serialize(SliceFromRe
adCommand.java:168) ~[apache-cassandra-2.1.9.jar:2.1.9]
at
org.apache.cassandra.db.ReadCommandSerializer.serialize(ReadCommand.java:143
) ~[apache-cassandra-2.1.9.jar:2.1.9]
at
org.apache.cassandra.db.ReadCommandSerializer.serialize(ReadCommand.java:132
) ~[apache-cassandra-2.1.9.jar:2.1.9]
at org.apache.cassandra.net.MessageOut.serialize(MessageOut.java:121)
~[apache-cassandra-2.1.9.jar:2.1.9]
at
org.apache.cassandra.net.OutboundTcpConnection.writeInternal(OutboundTcpConn
ection.java:330) ~[apache-cassandra-2.1.9.jar:2.1.9]
at
org.apache.cassandra.net.OutboundTcpConnection.writeConnected(OutboundTcpCon
nection.java:282) ~[apache-cassandra-2.1.9.jar:2.1.9]
at
org.apache.cassandra.net.OutboundTcpConnection.run(OutboundTcpConnection.jav
a:218) ~[apache-cassandra-2.1.9.jar:2.1.9]
INFO [NativePoolCleaner] 2015-11-08 10:36:47,414
ColumnFamilyStore.java:1231 - Flushing largest CFS(Keyspace='qinglong',
ColumnFamily='package_state') to free up room. Used total: 0.04/0.14, live:
0.04/0.14, flushing: 0.00/0.00, this: 0.02/0.02
Best Regards
答复: org.apache.cassandra.net.OutboundTcpConnection.run(OutboundTcpConnection.java:218 throws java.lang.AssertionError
Posted by 李建奇 <li...@jd.com>.
Hello,
Reading more codes. I can reproduce the exception. Over long
slice parameter can cause the exception.
Code :
char[] chs = new char[70000]; //over 64k
Arrays.fill(chs,'b');
String end = String.valueOf(chs);
Composite c2 = nameType.make(new Object[] {2, end ,11});
ColumnSlice slice = new ColumnSlice(c1,c2);
ColumnSlice.Serializer ser = new ColumnSlice.Serializer(nameType);
DataOutputPlus out = new DataOutputBuffer();
SliceQueryFilter slf = new SliceQueryFilter(c1,c2,false,1000);
nameType.sliceQueryFilterSerializer().serialize(slf,out,
MessagingService.current_version);
Question:
I wonder if have some validators before send the ReadCommand to data
nodes .
Thanks
发件人: 李建奇 [mailto:lijianqi@jd.com]
发送时间: 2015年11月10日 10:51
收件人: user@cassandra.apache.org
主题:
org.apache.cassandra.net.OutboundTcpConnection.run(OutboundTcpConnection.jav
a:218 throws java.lang.AssertionError
Hi, All,
We have a 12 nodes cluster with 2.1.9 version for near one month. Last
week it have an exception . Cluster’s write and read latency will go up to
4 seconds from 0.4ms average after exception.
I suspect OutboundTcpConnection is broken .I try to disablegossip then
enablegossip to rescue OutboundTcpConnction ,but fail .
I have to restart every node to restore cluster to normal . Cluster’s
load is low. The client use datastax java driver 2.1.7.1 . CF with RF 3.
Question:
Which situation to trigger this AssertionError? I read
OutboundTcpConnection.java source code, line 228 comments
“writeConnected() is reasonably robust”.
Thanks
Attachment :
ERROR [MessagingService-Outgoing-/172.20.114.13] 2015-11-08 10:36:28,763
CassandraDaemon.java:223 - Exception in thread
Thread[MessagingService-Outgoing-/172.20.114.13,5,main]
java.lang.AssertionError: 78251
at
org.apache.cassandra.utils.ByteBufferUtil.writeWithShortLength(ByteBufferUti
l.java:290) ~[apache-cassandra-2.1.9.jar:2.1.9]
at
org.apache.cassandra.db.composites.AbstractCType$Serializer.serialize(Abstra
ctCType.java:392) ~[apache-cassandra-2.1.9.jar:2.1.9]
at
org.apache.cassandra.db.composites.AbstractCType$Serializer.serialize(Abstra
ctCType.java:381) ~[apache-cassandra-2.1.9.jar:2.1.9]
at
org.apache.cassandra.db.filter.ColumnSlice$Serializer.serialize(ColumnSlice.
java:271) ~[apache-cassandra-2.1.9.jar:2.1.9]
at
org.apache.cassandra.db.filter.ColumnSlice$Serializer.serialize(ColumnSlice.
java:259) ~[apache-cassandra-2.1.9.jar:2.1.9]
at
org.apache.cassandra.db.filter.SliceQueryFilter$Serializer.serialize(SliceQu
eryFilter.java:503) ~[apache-cassandra-2.1.9.jar:2.1.9]
at
org.apache.cassandra.db.filter.SliceQueryFilter$Serializer.serialize(SliceQu
eryFilter.java:490) ~[apache-cassandra-2.1.9.jar:2.1.9]
at
org.apache.cassandra.db.SliceFromReadCommandSerializer.serialize(SliceFromRe
adCommand.java:168) ~[apache-cassandra-2.1.9.jar:2.1.9]
at
org.apache.cassandra.db.ReadCommandSerializer.serialize(ReadCommand.java:143
) ~[apache-cassandra-2.1.9.jar:2.1.9]
at
org.apache.cassandra.db.ReadCommandSerializer.serialize(ReadCommand.java:132
) ~[apache-cassandra-2.1.9.jar:2.1.9]
at org.apache.cassandra.net.MessageOut.serialize(MessageOut.java:121)
~[apache-cassandra-2.1.9.jar:2.1.9]
at
org.apache.cassandra.net.OutboundTcpConnection.writeInternal(OutboundTcpConn
ection.java:330) ~[apache-cassandra-2.1.9.jar:2.1.9]
at
org.apache.cassandra.net.OutboundTcpConnection.writeConnected(OutboundTcpCon
nection.java:282) ~[apache-cassandra-2.1.9.jar:2.1.9]
at
org.apache.cassandra.net.OutboundTcpConnection.run(OutboundTcpConnection.jav
a:218) ~[apache-cassandra-2.1.9.jar:2.1.9]
ERROR [MessagingService-Outgoing-/172.20.114.19] 2015-11-08 10:36:28,763
CassandraDaemon.java:223 - Exception in thread
Thread[MessagingService-Outgoing-/172.20.114.19,5,main]
java.lang.AssertionError: 78251
at
org.apache.cassandra.utils.ByteBufferUtil.writeWithShortLength(ByteBufferUti
l.java:290) ~[apache-cassandra-2.1.9.jar:2.1.9]
at
org.apache.cassandra.db.composites.AbstractCType$Serializer.serialize(Abstra
ctCType.java:392) ~[apache-cassandra-2.1.9.jar:2.1.9]
at
org.apache.cassandra.db.composites.AbstractCType$Serializer.serialize(Abstra
ctCType.java:381) ~[apache-cassandra-2.1.9.jar:2.1.9]
at
org.apache.cassandra.db.filter.ColumnSlice$Serializer.serialize(ColumnSlice.
java:271) ~[apache-cassandra-2.1.9.jar:2.1.9]
at
org.apache.cassandra.db.filter.ColumnSlice$Serializer.serialize(ColumnSlice.
java:259) ~[apache-cassandra-2.1.9.jar:2.1.9]
at
org.apache.cassandra.db.filter.SliceQueryFilter$Serializer.serialize(SliceQu
eryFilter.java:503) ~[apache-cassandra-2.1.9.jar:2.1.9]
at
org.apache.cassandra.db.filter.SliceQueryFilter$Serializer.serialize(SliceQu
eryFilter.java:490) ~[apache-cassandra-2.1.9.jar:2.1.9]
at
org.apache.cassandra.db.SliceFromReadCommandSerializer.serialize(SliceFromRe
adCommand.java:168) ~[apache-cassandra-2.1.9.jar:2.1.9]
at
org.apache.cassandra.db.ReadCommandSerializer.serialize(ReadCommand.java:143
) ~[apache-cassandra-2.1.9.jar:2.1.9]
at
org.apache.cassandra.db.ReadCommandSerializer.serialize(ReadCommand.java:132
) ~[apache-cassandra-2.1.9.jar:2.1.9]
at org.apache.cassandra.net.MessageOut.serialize(MessageOut.java:121)
~[apache-cassandra-2.1.9.jar:2.1.9]
at
org.apache.cassandra.net.OutboundTcpConnection.writeInternal(OutboundTcpConn
ection.java:330) ~[apache-cassandra-2.1.9.jar:2.1.9]
at
org.apache.cassandra.net.OutboundTcpConnection.writeConnected(OutboundTcpCon
nection.java:282) ~[apache-cassandra-2.1.9.jar:2.1.9]
at
org.apache.cassandra.net.OutboundTcpConnection.run(OutboundTcpConnection.jav
a:218) ~[apache-cassandra-2.1.9.jar:2.1.9]
INFO [NativePoolCleaner] 2015-11-08 10:36:47,414
ColumnFamilyStore.java:1231 - Flushing largest CFS(Keyspace='qinglong',
ColumnFamily='package_state') to free up room. Used total: 0.04/0.14, live:
0.04/0.14, flushing: 0.00/0.00, this: 0.02/0.02
Best Regards