You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Gabriel Ciuloaica <gc...@gmail.com> on 2013/04/12 09:12:13 UTC
Adding new nodes to an existing cluster
Hi,
From yesterday, I'm trying to add a new node to an existing 3 nodes
Cassandra cluster, running version 1.2.3. Today I have started clean,
removed the node that I have tried to add yesterday and try to add
another node. Verified both cassandra.yaml and
cassandra-topology.properties are identical on all nodes (I'm using
NetworkTopologyStrategy and PropertyFile snitch).
I see the nodetool netstats on the new node shows that it is streaming
from the other nodes, but other than increasing number on Responses
Pool, there is no change in the report. Logs are set to Debug and most
of new lines are "Applying mutations".
Below is the only ERROR that I can find in both cassandra.log and
system.log. So, it seems that streaming is working but not
completing... For the sstable that I get the error, the percentage is
not increasing in netstats, not for any other sstables. Now it is
running for few hours and no progress. Any clue ?
ERROR 06:09:03,934 Exception in thread Thread[Thread-25,5,main]
java.lang.AssertionError: incorrect row data size 114969 written to
/data/woow_backend/offline/woow_backend-offline-tmp-ib-2-Data.db;
correct is 115006
at
org.apache.cassandra.io.sstable.SSTableWriter.appendFromStream(SSTableWriter.java:285)
at
org.apache.cassandra.streaming.IncomingStreamReader.streamIn(IncomingStreamReader.java:179)
at
org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:122)
at
org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:238)
at
org.apache.cassandra.net.IncomingTcpConnection.handleStream(IncomingTcpConnection.java:178)
at
org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:78)
nodetool netstats:
Mode: JOINING
Not sending any streams.
Streaming from: /10.11.1.108
woow_backend:
/data/woow_backend/user_rooms/woow_backend-user_rooms-ib-1-Data.db
sections=1 progress=0/566 - 0%
woow_backend:
/data/woow_backend/gc_archive/woow_backend-gc_archive-ib-2-Data.db
sections=1 progress=0/3478 - 0%
woow_backend:
/data/woow_backend/msg_archive/woow_backend-msg_archive-ib-258-Data.db
sections=1 progress=0/313 - 0%
woow_backend:
/data/woow_backend/roster_groups/woow_backend-roster_groups-ib-23-Data.db sections=1
progress=0/120 - 0%
woow_backend:
/data/woow_backend/roster_versions/woow_backend-roster_versions-ib-34-Data.db
sections=2 progress=0/195 - 0%
woow_backend:
/data/woow_backend/rosters/woow_backend-rosters-ib-27-Data.db
sections=93 progress=0/881196 - 0%
woow_backend:
/data/woow_backend/msg_archive/woow_backend-msg_archive-ib-257-Data.db
sections=104 progress=0/5816889 - 0%
woow_backend:
/data/woow_backend/avatars/woow_backend-avatars-ib-68-Data.db
sections=71 progress=0/5841975 - 0%
woow_backend:
/data/woow_backend/rosters/woow_backend-rosters-ib-28-Data.db sections=2
progress=0/234 - 0%
woow_backend:
/data/woow_backend/msg_archive/woow_backend-msg_archive-ib-259-Data.db
sections=3 progress=0/6910 - 0%
woow_backend:
/data/woow_backend/offline/woow_backend-offline-ib-129-Data.db
sections=77 progress=22499/462727 - 4%
woow_backend:
/data/woow_backend/rooms/woow_backend-rooms-ib-2-Data.db sections=1
progress=0/308 - 0%
woow_backend:
/data/woow_backend/roster_versions/woow_backend-roster_versions-ib-33-Data.db
sections=93 progress=0/34246 - 0%
woow_backend:
/data/woow_backend/rooms/woow_backend-rooms-ib-1-Data.db sections=1
progress=0/310 - 0%
Pool Name Active Pending Completed
Commands n/a 0 15
Responses n/a 0 44808
Thanks,
Gabi
Re: Adding new nodes to an existing cluster
Posted by Gabriel Ciuloaica <gc...@gmail.com>.
Thanks Marcus,
So, at this moment I have to wait until release of version 1.2.5 to be
able to extend the cluster ... I haven't seen mention any workaround ...
Thanks,
Gabi
On 4/12/13 11:11 AM, Marcus Eriksson wrote:
> looks very similar to this:
> https://issues.apache.org/jira/browse/CASSANDRA-5418
>
> /Marcus
>
>
> On Fri, Apr 12, 2013 at 9:12 AM, Gabriel Ciuloaica
> <gciuloaica@gmail.com <ma...@gmail.com>> wrote:
>
> Hi,
>
> From yesterday, I'm trying to add a new node to an existing 3
> nodes Cassandra cluster, running version 1.2.3. Today I have
> started clean, removed the node that I have tried to add yesterday
> and try to add another node. Verified both cassandra.yaml and
> cassandra-topology.properties are identical on all nodes (I'm
> using NetworkTopologyStrategy and PropertyFile snitch).
> I see the nodetool netstats on the new node shows that it is
> streaming from the other nodes, but other than increasing number
> on Responses Pool, there is no change in the report. Logs are set
> to Debug and most of new lines are "Applying mutations".
>
> Below is the only ERROR that I can find in both cassandra.log and
> system.log. So, it seems that streaming is working but not
> completing... For the sstable that I get the error, the percentage
> is not increasing in netstats, not for any other sstables. Now it
> is running for few hours and no progress. Any clue ?
> ERROR 06:09:03,934 Exception in thread Thread[Thread-25,5,main]
> java.lang.AssertionError: incorrect row data size 114969 written
> to
> /data/woow_backend/offline/woow_backend-offline-tmp-ib-2-Data.db;
> correct is 115006
> at org.apache.cassandra.io
> <http://org.apache.cassandra.io>.sstable.SSTableWriter.appendFromStream(SSTableWriter.java:285)
> at
> org.apache.cassandra.streaming.IncomingStreamReader.streamIn(IncomingStreamReader.java:179)
> at
> org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:122)
> at org.apache.cassandra.net
> <http://org.apache.cassandra.net>.IncomingTcpConnection.stream(IncomingTcpConnection.java:238)
> at org.apache.cassandra.net
> <http://org.apache.cassandra.net>.IncomingTcpConnection.handleStream(IncomingTcpConnection.java:178)
> at org.apache.cassandra.net
> <http://org.apache.cassandra.net>.IncomingTcpConnection.run(IncomingTcpConnection.java:78)
>
>
>
> nodetool netstats:
>
> Mode: JOINING
> Not sending any streams.
> Streaming from: /10.11.1.108 <http://10.11.1.108>
> woow_backend:
> /data/woow_backend/user_rooms/woow_backend-user_rooms-ib-1-Data.db
> sections=1 progress=0/566 - 0%
> woow_backend:
> /data/woow_backend/gc_archive/woow_backend-gc_archive-ib-2-Data.db
> sections=1 progress=0/3478 - 0%
> woow_backend:
> /data/woow_backend/msg_archive/woow_backend-msg_archive-ib-258-Data.db
> sections=1 progress=0/313 - 0%
> woow_backend:
> /data/woow_backend/roster_groups/woow_backend-roster_groups-ib-23-Data.db
> sections=1 progress=0/120 - 0%
> woow_backend:
> /data/woow_backend/roster_versions/woow_backend-roster_versions-ib-34-Data.db
> sections=2 progress=0/195 - 0%
> woow_backend:
> /data/woow_backend/rosters/woow_backend-rosters-ib-27-Data.db
> sections=93 progress=0/881196 - 0%
> woow_backend:
> /data/woow_backend/msg_archive/woow_backend-msg_archive-ib-257-Data.db
> sections=104 progress=0/5816889 - 0%
> woow_backend:
> /data/woow_backend/avatars/woow_backend-avatars-ib-68-Data.db
> sections=71 progress=0/5841975 - 0%
> woow_backend:
> /data/woow_backend/rosters/woow_backend-rosters-ib-28-Data.db
> sections=2 progress=0/234 - 0%
> woow_backend:
> /data/woow_backend/msg_archive/woow_backend-msg_archive-ib-259-Data.db
> sections=3 progress=0/6910 - 0%
> woow_backend:
> /data/woow_backend/offline/woow_backend-offline-ib-129-Data.db
> sections=77 progress=22499/462727 - 4%
> woow_backend:
> /data/woow_backend/rooms/woow_backend-rooms-ib-2-Data.db
> sections=1 progress=0/308 - 0%
> woow_backend:
> /data/woow_backend/roster_versions/woow_backend-roster_versions-ib-33-Data.db
> sections=93 progress=0/34246 - 0%
> woow_backend:
> /data/woow_backend/rooms/woow_backend-rooms-ib-1-Data.db
> sections=1 progress=0/310 - 0%
> Pool Name Active Pending Completed
> Commands n/a 0 15
> Responses n/a 0 44808
>
> Thanks,
> Gabi
>
>
Re: Adding new nodes to an existing cluster
Posted by Marcus Eriksson <kr...@gmail.com>.
looks very similar to this:
https://issues.apache.org/jira/browse/CASSANDRA-5418
/Marcus
On Fri, Apr 12, 2013 at 9:12 AM, Gabriel Ciuloaica <gc...@gmail.com>wrote:
> Hi,
>
> From yesterday, I'm trying to add a new node to an existing 3 nodes
> Cassandra cluster, running version 1.2.3. Today I have started clean,
> removed the node that I have tried to add yesterday and try to add another
> node. Verified both cassandra.yaml and cassandra-topology.properties are
> identical on all nodes (I'm using NetworkTopologyStrategy and PropertyFile
> snitch).
> I see the nodetool netstats on the new node shows that it is streaming
> from the other nodes, but other than increasing number on Responses Pool,
> there is no change in the report. Logs are set to Debug and most of new
> lines are "Applying mutations".
>
> Below is the only ERROR that I can find in both cassandra.log and
> system.log. So, it seems that streaming is working but not completing...
> For the sstable that I get the error, the percentage is not increasing in
> netstats, not for any other sstables. Now it is running for few hours and
> no progress. Any clue ?
> ERROR 06:09:03,934 Exception in thread Thread[Thread-25,5,main]
> java.lang.AssertionError: incorrect row data size 114969 written to
> /data/woow_backend/offline/**woow_backend-offline-tmp-ib-2-**Data.db;
> correct is 115006
> at org.apache.cassandra.io.**sstable.SSTableWriter.**
> appendFromStream(**SSTableWriter.java:285)
> at org.apache.cassandra.**streaming.**
> IncomingStreamReader.streamIn(**IncomingStreamReader.java:179)
> at org.apache.cassandra.**streaming.**IncomingStreamReader.read(**
> IncomingStreamReader.java:122)
> at org.apache.cassandra.net.**IncomingTcpConnection.stream(**
> IncomingTcpConnection.java:**238)
> at org.apache.cassandra.net.**IncomingTcpConnection.**
> handleStream(**IncomingTcpConnection.java:**178)
> at org.apache.cassandra.net.**IncomingTcpConnection.run(**
> IncomingTcpConnection.java:78)
>
>
>
> nodetool netstats:
>
> Mode: JOINING
> Not sending any streams.
> Streaming from: /10.11.1.108
> woow_backend: /data/woow_backend/user_rooms/**
> woow_backend-user_rooms-ib-1-**Data.db sections=1 progress=0/566 - 0%
> woow_backend: /data/woow_backend/gc_archive/**
> woow_backend-gc_archive-ib-2-**Data.db sections=1 progress=0/3478 - 0%
> woow_backend: /data/woow_backend/msg_**archive/woow_backend-msg_**archive-ib-258-Data.db
> sections=1 progress=0/313 - 0%
> woow_backend: /data/woow_backend/roster_**groups/woow_backend-roster_**groups-ib-23-Data.db
> sections=1 progress=0/120 - 0%
> woow_backend: /data/woow_backend/roster_**versions/woow_backend-roster_
> **versions-ib-34-Data.db sections=2 progress=0/195 - 0%
> woow_backend: /data/woow_backend/rosters/**woow_backend-rosters-ib-27-*
> *Data.db sections=93 progress=0/881196 - 0%
> woow_backend: /data/woow_backend/msg_**archive/woow_backend-msg_**archive-ib-257-Data.db
> sections=104 progress=0/5816889 - 0%
> woow_backend: /data/woow_backend/avatars/**woow_backend-avatars-ib-68-*
> *Data.db sections=71 progress=0/5841975 - 0%
> woow_backend: /data/woow_backend/rosters/**woow_backend-rosters-ib-28-*
> *Data.db sections=2 progress=0/234 - 0%
> woow_backend: /data/woow_backend/msg_**archive/woow_backend-msg_**archive-ib-259-Data.db
> sections=3 progress=0/6910 - 0%
> woow_backend: /data/woow_backend/offline/**woow_backend-offline-ib-129-
> **Data.db sections=77 progress=22499/462727 - 4%
> woow_backend: /data/woow_backend/rooms/woow_**backend-rooms-ib-2-Data.db
> sections=1 progress=0/308 - 0%
> woow_backend: /data/woow_backend/roster_**versions/woow_backend-roster_
> **versions-ib-33-Data.db sections=93 progress=0/34246 - 0%
> woow_backend: /data/woow_backend/rooms/woow_**backend-rooms-ib-1-Data.db
> sections=1 progress=0/310 - 0%
> Pool Name Active Pending Completed
> Commands n/a 0 15
> Responses n/a 0 44808
>
> Thanks,
> Gabi
>