You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Alex Petrov (Jira)" <ji...@apache.org> on 2020/10/01 14:28:00 UTC
[jira] [Updated] (CASSANDRA-16156) Decomissioned nodes are picked
for gossip when unreachable nodes are considered for gossiping
[ https://issues.apache.org/jira/browse/CASSANDRA-16156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alex Petrov updated CASSANDRA-16156:
------------------------------------
Description:
After node is decommissioned, it is still considered for gossip via “unreachable” nodes, which results into following exceptions:
{code}
INFO [node4_Messaging-EventLoop-3-3] node4 2020-09-29 16:37:37,527 NoSpamLogger.java:91 - /127.0.0.4:7012->/127.0.0.1:7012-URGENT_MESSAGES-[no-channel] failed to connect
io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: /127.0.0.1:7012
Caused by: java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:330)
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:702)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)
{code}
Trace of the method that attempts to establish connection:
{code}
org.apache.cassandra.net.MessagingService.getOutbound(MessagingService.java:492)
at org.apache.cassandra.net.MessagingService.doSend(MessagingService.java:335)
at org.apache.cassandra.net.OutboundSink$Filtered.accept(OutboundSink.java:55)
at org.apache.cassandra.net.OutboundSink.accept(OutboundSink.java:70)
at org.apache.cassandra.net.MessagingService.send(MessagingService.java:327)
at org.apache.cassandra.net.MessagingService.send(MessagingService.java:314)
at org.apache.cassandra.gms.Gossiper.sendGossip(Gossiper.java:813)
at org.apache.cassandra.gms.Gossiper.maybeGossipToUnreachableMember(Gossiper.java:840)
at org.apache.cassandra.gms.Gossiper.access$400(Gossiper.java:86)
{code}
LEFT and other nodes that are considered dead should not be picked for gossip with unreachable nodes.
was:
After node is decommissioned, it is still considered for gossip via “unreachable” nodes, which results into following exceptions:
INFO [node4_Messaging-EventLoop-3-3] node4 2020-09-29 16:37:37,527 NoSpamLogger.java:91 - /127.0.0.4:7012->/127.0.0.1:7012-URGENT_MESSAGES-[no-channel] failed to connect
io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: /127.0.0.1:7012
Caused by: java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:330)
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:702)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)
Trace of the method that attempts to establish connection:
org.apache.cassandra.net.MessagingService.getOutbound(MessagingService.java:492)
at org.apache.cassandra.net.MessagingService.doSend(MessagingService.java:335)
at org.apache.cassandra.net.OutboundSink$Filtered.accept(OutboundSink.java:55)
at org.apache.cassandra.net.OutboundSink.accept(OutboundSink.java:70)
at org.apache.cassandra.net.MessagingService.send(MessagingService.java:327)
at org.apache.cassandra.net.MessagingService.send(MessagingService.java:314)
at org.apache.cassandra.gms.Gossiper.sendGossip(Gossiper.java:813)
at org.apache.cassandra.gms.Gossiper.maybeGossipToUnreachableMember(Gossiper.java:840)
at org.apache.cassandra.gms.Gossiper.access$400(Gossiper.java:86)
LEFT and other nodes that are considered dead should not be picked for gossip with unreachable nodes.
> Decomissioned nodes are picked for gossip when unreachable nodes are considered for gossiping
> ----------------------------------------------------------------------------------------------
>
> Key: CASSANDRA-16156
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16156
> Project: Cassandra
> Issue Type: Bug
> Reporter: Alex Petrov
> Assignee: Alex Petrov
> Priority: Normal
>
> After node is decommissioned, it is still considered for gossip via “unreachable” nodes, which results into following exceptions:
>
> {code}
> INFO [node4_Messaging-EventLoop-3-3] node4 2020-09-29 16:37:37,527 NoSpamLogger.java:91 - /127.0.0.4:7012->/127.0.0.1:7012-URGENT_MESSAGES-[no-channel] failed to connect
> io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: /127.0.0.1:7012
> Caused by: java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
> at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:330)
> at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334)
> at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:702)
> at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
> at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
> at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
> at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
> at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
> at java.lang.Thread.run(Thread.java:748)
> {code}
> Trace of the method that attempts to establish connection:
> {code}
> org.apache.cassandra.net.MessagingService.getOutbound(MessagingService.java:492)
> at org.apache.cassandra.net.MessagingService.doSend(MessagingService.java:335)
> at org.apache.cassandra.net.OutboundSink$Filtered.accept(OutboundSink.java:55)
> at org.apache.cassandra.net.OutboundSink.accept(OutboundSink.java:70)
> at org.apache.cassandra.net.MessagingService.send(MessagingService.java:327)
> at org.apache.cassandra.net.MessagingService.send(MessagingService.java:314)
> at org.apache.cassandra.gms.Gossiper.sendGossip(Gossiper.java:813)
> at org.apache.cassandra.gms.Gossiper.maybeGossipToUnreachableMember(Gossiper.java:840)
> at org.apache.cassandra.gms.Gossiper.access$400(Gossiper.java:86)
> {code}
> LEFT and other nodes that are considered dead should not be picked for gossip with unreachable nodes.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org