You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Yakir Gibraltar (Jira)" <ji...@apache.org> on 2020/11/19 16:52:00 UTC
[jira] [Updated] (CASSANDRA-16289) nodetool removenode failing
[ https://issues.apache.org/jira/browse/CASSANDRA-16289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Yakir Gibraltar updated CASSANDRA-16289:
----------------------------------------
Description:
Hi, i tied to remove node on Cassandra 4.0-beta3 and got a lot of errors and also nodetool command failed.
{code}
[root@foo ~]# nodetool removenode 2dfa2fde-15ee-40ad-8afc-0cb6db9aaad9
nodetool: Conflicting replica added (expected unique ranges): Full(/1.1.1.1:7000,(2012127519876180938,2107158795315043585]); existing: Full(/1.1.1.1:7000,(2012127519876180938,2107158795315043585])
See 'nodetool help' or 'nodetool help <command>'.
{code}
In log:
{code}
INFO [Messaging-EventLoop-3-31] 2020-11-19 15:56:48,949 NoSpamLogger.java:92 - /1.1.1.1:7000->/2.2.2.2:7000-URGENT_MESSAGES-[no-channel] failed to connect
io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: /2.2.2.2:7000
Caused by: java.net.ConnectException: finishConnect(..) failed: Connection refused
at io.netty.channel.unix.Errors.throwConnectException(Errors.java:124)
at io.netty.channel.unix.Socket.finishConnect(Socket.java:243)
at io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.doFinishConnect(AbstractEpollChannel.java:672)
at io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:649)
at io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:529)
at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:465)
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:834)
WARN [Messaging-EventLoop-3-31] 2020-11-19 16:01:18,973 NoSpamLogger.java:95 - /1.1.1.1:7000->/2.2.2.2:7000-URGENT_MESSAGES-[no-channel] dropping message of type GOSSIP_DIGEST_SYN whose timeout expired before reaching the network
{code}
How to reproduce:
Create 2 DC and remove one node.
Maybe it possible to reproduce with one DC but in my case was 2 DCs.
I had to remove with "force"
was:
Hi, i tied to remove node on Cassandra 4 and got a lot of errors and also nodetool command failed.
{code}
ot working well with C* 4:
[root@foo ~]# nodetool removenode 2dfa2fde-15ee-40ad-8afc-0cb6db9aaad9
nodetool: Conflicting replica added (expected unique ranges): Full(/1.1.1.1:7000,(2012127519876180938,2107158795315043585]); existing: Full(/1.1.1.1:7000,(2012127519876180938,2107158795315043585])
See 'nodetool help' or 'nodetool help <command>'.
{code}
In log:
{code}
INFO [Messaging-EventLoop-3-31] 2020-11-19 15:56:48,949 NoSpamLogger.java:92 - /1.1.1.1:7000->/2.2.2.2:7000-URGENT_MESSAGES-[no-channel] failed to connect
io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: /2.2.2.2:7000
Caused by: java.net.ConnectException: finishConnect(..) failed: Connection refused
at io.netty.channel.unix.Errors.throwConnectException(Errors.java:124)
at io.netty.channel.unix.Socket.finishConnect(Socket.java:243)
at io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.doFinishConnect(AbstractEpollChannel.java:672)
at io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:649)
at io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:529)
at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:465)
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:834)
WARN [Messaging-EventLoop-3-31] 2020-11-19 16:01:18,973 NoSpamLogger.java:95 - /1.1.1.1:7000->/2.2.2.2:7000-URGENT_MESSAGES-[no-channel] dropping message of type GOSSIP_DIGEST_SYN whose timeout expired before reaching the network
{code}
How to reproduce:
Create 2 DC and remove one node.
Maybe it possible to reproduce with one DC but in my case was 2 DCs.
I had to remove with "force"
> nodetool removenode failing
> ---------------------------
>
> Key: CASSANDRA-16289
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16289
> Project: Cassandra
> Issue Type: Bug
> Reporter: Yakir Gibraltar
> Priority: Normal
>
> Hi, i tied to remove node on Cassandra 4.0-beta3 and got a lot of errors and also nodetool command failed.
> {code}
> [root@foo ~]# nodetool removenode 2dfa2fde-15ee-40ad-8afc-0cb6db9aaad9
> nodetool: Conflicting replica added (expected unique ranges): Full(/1.1.1.1:7000,(2012127519876180938,2107158795315043585]); existing: Full(/1.1.1.1:7000,(2012127519876180938,2107158795315043585])
> See 'nodetool help' or 'nodetool help <command>'.
> {code}
> In log:
> {code}
> INFO [Messaging-EventLoop-3-31] 2020-11-19 15:56:48,949 NoSpamLogger.java:92 - /1.1.1.1:7000->/2.2.2.2:7000-URGENT_MESSAGES-[no-channel] failed to connect
> io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: /2.2.2.2:7000
> Caused by: java.net.ConnectException: finishConnect(..) failed: Connection refused
> at io.netty.channel.unix.Errors.throwConnectException(Errors.java:124)
> at io.netty.channel.unix.Socket.finishConnect(Socket.java:243)
> at io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.doFinishConnect(AbstractEpollChannel.java:672)
> at io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:649)
> at io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:529)
> at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:465)
> at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378)
> at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
> at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
> at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
> at java.base/java.lang.Thread.run(Thread.java:834)
> WARN [Messaging-EventLoop-3-31] 2020-11-19 16:01:18,973 NoSpamLogger.java:95 - /1.1.1.1:7000->/2.2.2.2:7000-URGENT_MESSAGES-[no-channel] dropping message of type GOSSIP_DIGEST_SYN whose timeout expired before reaching the network
> {code}
> How to reproduce:
> Create 2 DC and remove one node.
> Maybe it possible to reproduce with one DC but in my case was 2 DCs.
> I had to remove with "force"
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org