You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Junaid Nasir <jn...@an10.io> on 2017/05/31 07:55:54 UTC
Convert single node C* to cluster (rebalancing problem)
Cassandra ensure that adding or removing nodes are very easy and that load
is balanced between nodes when a change is made. but it's not working in my
case.
I have a single node C* deployment (with 270 GB of data) and want to load
balance the data on multiple nodes, I followed this guide
<https://docs.datastax.com/en/cassandra/2.1/cassandra/operations/ops_add_node_to_cluster_t.html>
`nodetool status` shows 2 nodes but load is not balanced between them
Datacenter: dc1
===============
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host ID Rack
UN 10.128.0.7 270.75 GiB 256 48.6%
1a3f6faa-4376-45a8-9c20-11480ae5664c rack1
UN 10.128.0.14 414.36 KiB 256 51.4%
66a89fbf-08ba-4b5d-9f10-55d52a199b41 rack1
I also ran 'nodetool repair' on new node but result is same. any pointers
would be appreciated :)
conf file of new node
cluster_name: 'cluster1'
- seeds: "10.128.0.7"
num_tokens: 256
endpoint_snitch: GossipingPropertyFileSnitch
Thanks,
Junaid
Re: Convert single node C* to cluster (rebalancing problem)
Posted by Junaid Nasir <jn...@an10.io>.
not evenly, i have setup a new cluster with subset of data (around 5gb).
using the configuration above I am getting these results
Datacenter: datacenter1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host ID Rack
UN 10.128.2.1 4.86 GiB 256 44.9%
e4427611-c247-42ee-9404-371e177f5f17 rack1
UN 10.128.2.10 725.03 MiB 256 55.1%
690d5620-99d3-4ae3-aebe-8f33af54a08b rack1
is there anything else I can tweak/check to make the distribution even?
On Sat, Jun 3, 2017 at 3:30 AM, Akhil Mehra <ak...@gmail.com> wrote:
> So now the data is evenly balanced in both nodes?
>
> Refer to the following documentation to get a better understanding of the
> roc_address and the broadcast_rpc_address https://www.instaclustr.com/
> demystifying-cassandras-broadcast_address/. I am surprised that your node
> started up with rpc_broadcast_address set as this is an unsupported
> property. I am assuming you are using Cassandra version 3.10.
>
>
> Regards,
> Akhil
>
> On 2/06/2017, at 11:06 PM, Junaid Nasir <jn...@an10.io> wrote:
>
> I am able to get it working. I added a new node with following changes
>
> #rpc_address:0.0.0.0
> rpc_address: 10.128.1.11
> #rpc_broadcast_address:10.128.1.11
>
> rpc_address was set to 0.0.0.0, (I ran into a problem previously regarding
> remote connection and made these changes https://stackoverflow.com/
> questions/12236898/apache-cassandra-remote-access)
>
> should it be happening?
>
> On Thu, Jun 1, 2017 at 6:31 PM, Vladimir Yudovin <vl...@winguzone.com>
> wrote:
>
>> Did you run "nodetool cleanup" on first node after second was
>> bootstrapped? It should clean rows not belonging to node after tokens
>> changed.
>>
>> Best regards, Vladimir Yudovin,
>> *Winguzone <https://winguzone.com/?from=list> - Cloud Cassandra Hosting*
>>
>>
>> ---- On Wed, 31 May 2017 03:55:54 -0400 *Junaid Nasir <jnasir@an10.io
>> <jn...@an10.io>>* wrote ----
>>
>> Cassandra ensure that adding or removing nodes are very easy and that
>> load is balanced between nodes when a change is made. but it's not working
>> in my case.
>> I have a single node C* deployment (with 270 GB of data) and want to load
>> balance the data on multiple nodes, I followed this guide
>> <https://docs.datastax.com/en/cassandra/2.1/cassandra/operations/ops_add_node_to_cluster_t.html>
>>
>> `nodetool status` shows 2 nodes but load is not balanced between them
>>
>> Datacenter: dc1
>> ===============
>> Status=Up/Down
>> |/ State=Normal/Leaving/Joining/Moving
>> -- Address Load Tokens Owns (effective) Host ID Rack
>> UN 10.128.0.7 270.75 GiB 256 48.6% 1a3f6faa-4376-45a8-9c20-11480ae5664c rack1
>> UN 10.128.0.14 414.36 KiB 256 51.4% 66a89fbf-08ba-4b5d-9f10-55d52a199b41 rack1
>>
>> I also ran 'nodetool repair' on new node but result is same. any pointers
>> would be appreciated :)
>>
>> conf file of new node
>>
>> cluster_name: 'cluster1'
>> - seeds: "10.128.0.7"
>> num_tokens: 256
>> endpoint_snitch: GossipingPropertyFileSnitch
>>
>> Thanks,
>> Junaid
>>
>>
>>
>
>
Re: Convert single node C* to cluster (rebalancing problem)
Posted by Akhil Mehra <ak...@gmail.com>.
So now the data is evenly balanced in both nodes?
Refer to the following documentation to get a better understanding of the roc_address and the broadcast_rpc_address https://www.instaclustr.com/demystifying-cassandras-broadcast_address/ <https://www.instaclustr.com/demystifying-cassandras-broadcast_address/>. I am surprised that your node started up with rpc_broadcast_address set as this is an unsupported property. I am assuming you are using Cassandra version 3.10.
Regards,
Akhil
> On 2/06/2017, at 11:06 PM, Junaid Nasir <jn...@an10.io> wrote:
>
> I am able to get it working. I added a new node with following changes
> #rpc_address:0.0.0.0
> rpc_address: 10.128.1.11
> #rpc_broadcast_address:10.128.1.11
> rpc_address was set to 0.0.0.0, (I ran into a problem previously regarding remote connection and made these changes https://stackoverflow.com/questions/12236898/apache-cassandra-remote-access <https://stackoverflow.com/questions/12236898/apache-cassandra-remote-access>)
>
> should it be happening?
>
> On Thu, Jun 1, 2017 at 6:31 PM, Vladimir Yudovin <vladyu@winguzone.com <ma...@winguzone.com>> wrote:
> Did you run "nodetool cleanup" on first node after second was bootstrapped? It should clean rows not belonging to node after tokens changed.
>
> Best regards, Vladimir Yudovin,
> Winguzone <https://winguzone.com/?from=list> - Cloud Cassandra Hosting
>
>
> ---- On Wed, 31 May 2017 03:55:54 -0400 Junaid Nasir <jnasir@an10.io <ma...@an10.io>> wrote ----
>
> Cassandra ensure that adding or removing nodes are very easy and that load is balanced between nodes when a change is made. but it's not working in my case.
> I have a single node C* deployment (with 270 GB of data) and want to load balance the data on multiple nodes, I followed this guide <https://docs.datastax.com/en/cassandra/2.1/cassandra/operations/ops_add_node_to_cluster_t.html>
> `nodetool status` shows 2 nodes but load is not balanced between them
> Datacenter: dc1
> ===============
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> -- Address Load Tokens Owns (effective) Host ID Rack
> UN 10.128.0.7 270.75 GiB 256 48.6% 1a3f6faa-4376-45a8-9c20-11480ae5664c rack1
> UN 10.128.0.14 414.36 KiB 256 51.4% 66a89fbf-08ba-4b5d-9f10-55d52a199b41 rack1
> I also ran 'nodetool repair' on new node but result is same. any pointers would be appreciated :)
>
> conf file of new node
> cluster_name: 'cluster1'
> - seeds: "10.128.0.7"
> num_tokens: 256
> endpoint_snitch: GossipingPropertyFileSnitch
> Thanks,
> Junaid
>
>
Re: Convert single node C* to cluster (rebalancing problem)
Posted by Junaid Nasir <jn...@an10.io>.
I am able to get it working. I added a new node with following changes
#rpc_address:0.0.0.0
rpc_address: 10.128.1.11
#rpc_broadcast_address:10.128.1.11
rpc_address was set to 0.0.0.0, (I ran into a problem previously regarding
remote connection and made these changes
https://stackoverflow.com/questions/12236898/apache-cassandra-remote-access
)
should it be happening?
On Thu, Jun 1, 2017 at 6:31 PM, Vladimir Yudovin <vl...@winguzone.com>
wrote:
> Did you run "nodetool cleanup" on first node after second was
> bootstrapped? It should clean rows not belonging to node after tokens
> changed.
>
> Best regards, Vladimir Yudovin,
> *Winguzone <https://winguzone.com?from=list> - Cloud Cassandra Hosting*
>
>
> ---- On Wed, 31 May 2017 03:55:54 -0400 *Junaid Nasir <jnasir@an10.io
> <jn...@an10.io>>* wrote ----
>
> Cassandra ensure that adding or removing nodes are very easy and that load
> is balanced between nodes when a change is made. but it's not working in my
> case.
> I have a single node C* deployment (with 270 GB of data) and want to load
> balance the data on multiple nodes, I followed this guide
> <https://docs.datastax.com/en/cassandra/2.1/cassandra/operations/ops_add_node_to_cluster_t.html>
>
> `nodetool status` shows 2 nodes but load is not balanced between them
>
> Datacenter: dc1
> ===============
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> -- Address Load Tokens Owns (effective) Host ID Rack
> UN 10.128.0.7 270.75 GiB 256 48.6% 1a3f6faa-4376-45a8-9c20-11480ae5664c rack1
> UN 10.128.0.14 414.36 KiB 256 51.4% 66a89fbf-08ba-4b5d-9f10-55d52a199b41 rack1
>
> I also ran 'nodetool repair' on new node but result is same. any pointers
> would be appreciated :)
>
> conf file of new node
>
> cluster_name: 'cluster1'
> - seeds: "10.128.0.7"
> num_tokens: 256
> endpoint_snitch: GossipingPropertyFileSnitch
>
> Thanks,
> Junaid
>
>
>
Re: Convert single node C* to cluster (rebalancing problem)
Posted by Vladimir Yudovin <vl...@winguzone.com>.
Did you run "nodetool cleanup" on first node after second was bootstrapped? It should clean rows not belonging to node after tokens changed.
Best regards, Vladimir Yudovin,
Winguzone - Cloud Cassandra Hosting
---- On Wed, 31 May 2017 03:55:54 -0400 Junaid Nasir <jnasir@an10.io> wrote ----
Cassandra ensure that adding or removing nodes are very easy and that load is balanced between nodes when a change is made. but it's not working in my case.
I have a single node C* deployment (with 270 GB of data) and want to load balance the data on multiple nodes, I followed this guide
`nodetool status` shows 2 nodes but load is not balanced between them
Datacenter: dc1 =============== Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address Load Tokens Owns (effective) Host ID Rack UN 10.128.0.7 270.75 GiB 256 48.6% 1a3f6faa-4376-45a8-9c20-11480ae5664c rack1 UN 10.128.0.14 414.36 KiB 256 51.4% 66a89fbf-08ba-4b5d-9f10-55d52a199b41 rack1
I also ran 'nodetool repair' on new node but result is same. any pointers would be appreciated :)
conf file of new node
cluster_name: 'cluster1' - seeds: "10.128.0.7"
num_tokens: 256 endpoint_snitch: GossipingPropertyFileSnitch
Thanks,
Junaid
Re: Convert single node C* to cluster (rebalancing problem)
Posted by Akhil Mehra <ak...@gmail.com>.
When you bootstrapped the node for the first time did you see log similar to the following:
INFO [main] 2017-06-01 07:19:45,199 StorageService.java:1435 - JOINING: waiting for schema information to complete
INFO [main] 2017-06-01 07:19:45,250 StorageService.java:1435 - JOINING: schema complete, ready to bootstrap
INFO [main] 2017-06-01 07:19:45,251 StorageService.java:1435 - JOINING: waiting for pending range calculation
INFO [main] 2017-06-01 07:19:45,251 StorageService.java:1435 - JOINING: calculation complete, ready to bootstrap
INFO [main] 2017-06-01 07:19:45,251 StorageService.java:1435 - JOINING: getting bootstrap token
INFO [main] 2017-06-01 07:19:45,341 StorageService.java:1435 - JOINING: sleeping 30000 ms for pending range setup
INFO [main] 2017-06-01 07:20:15,342 StorageService.java:1435 - JOINING: Starting to bootstrap...
INFO [main] 2017-06-01 07:20:15,562 StreamResultFuture.java:90 - [Stream #c219d430-469a-11e7-8af3-81773e2a69ae] Executing streaming plan for Bootstrap
INFO [StreamConnectionEstablisher:1] 2017-06-01 07:20:15,568 StreamSession.java:266 - [Stream #c219d430-469a-11e7-8af3-81773e2a69ae] Starting streaming to /172.20.0.3
INFO [StreamConnectionEstablisher:1] 2017-06-01 07:20:15,591 StreamCoordinator.java:264 - [Stream #c219d430-469a-11e7-8af3-81773e2a69ae, ID#0] Beginning stream session with /172.20.0.3
INFO [STREAM-IN-/172.20.0.3:7000] 2017-06-01 07:20:16,369 StreamResultFuture.java:173 - [Stream #c219d430-469a-11e7-8af3-81773e2a69ae ID#0] Prepare completed. Receiving 4 files(4.046MiB), sending 0 files(0.000KiB)
INFO [StreamReceiveTask:1] 2017-06-01 07:20:32,489 StreamResultFuture.java:187 - [Stream #c219d430-469a-11e7-8af3-81773e2a69ae] Session with /172.20.0.3 is complete
INFO [StreamReceiveTask:1] 2017-06-01 07:20:32,529 StreamResultFuture.java:219 - [Stream #c219d430-469a-11e7-8af3-81773e2a69ae] All sessions completed
INFO [StreamReceiveTask:1] 2017-06-01 07:20:32,535 StorageService.java:1491 - Bootstrap completed! for the tokens [2170857698622202367, 8517072504425343717, -3254771037524187900, -1597835042001935502, 6878847904605480741, 6816916215341820068, 6291189887494617640, -6855333019196580358, -6353317035112065873, 8838974905234547016, 8539810544447438397, -2357950949959511387, 1242077960532340887, 2914039668080386735, 3548015300105653368, 8973388453035242795, -2325235809399362967, -7078812010537656277, 768585495224455336, 7153512700965912517, 8625819392009074153, -6138302849441936958, -2594051958993427953, -735827743339795655, -8202727571538912843, 2180751358288507888, -7872842094207074012, -2926504780761300623, -3197260822146229664, 3411052656191450941, -9049284186987733291, -157882351668930258, 454637839762305232, -2305675997627138050, 5785282040753174988, 8604531769609599767, 4363117061247143957, -7255854383313210529, -3497611663121502480, -6788457421774336480, -7809767930173770420, 6591540654522244365, 1773283733607350132, 1347769111173669066, -7242556233623424655, -1552552727731631642, -1226243976028310059, -8221762326275074149, -7963893314043006091, -850542197910474448, 4219437099703910566, -8039365343972054221, 7756456412568178996, 4057327843751741693, 7155628666873897485, -483058846775660782, 6968839681845709305, 6396337738827005745, -5285173481531605912, 7254663657455123842, 871654822989271789, -604574593420741277, -2446461444470484127, -3707613591745746278, -26727542030118959, -7190990795521107837, 5388348291571480415, 4249499356533972018, 82469082189512791, -6389351372873749061, 5138413916027470955, 2542233707258091740, -4057927973990056143, 552933169018893618, -8237860380097407047, 6917383508758068288, 543382311932406672, -5671560690999322491, -1240369858424929757, 7394536427227616773, 4716882285905136652, 8260705434779371419, 3259812719139852593, -73864539388331289, -3573980475038135246, -1047139059901238511, -1734886021153324482, 8674873751672827600, 3564384074427511950, 2754071903665103098, -1230493021099846761, -2731315467436512731, -7845984767828231726, -8082165594257396645, -2298177264815779081, -3645421111048544165, 9142633389925493379, 7206663288804675578, 2305939212045070856, -5101738026249032246, 6268847697773786891, 5903922100677671597, -2001787466557152206, 1318502870562311928, 5784020265166141829, 5385229217299505171, 6010414616247875068, -8080602674779008196, -9189764569651551963, -8969124116887255329, -9040482343274988119, -8575947267671214955, -1786409930636352174, -757203989676123224, -6640569567328853730, 8431839804447545665, 6781635966829972979, -8328382509754233304, -3181089993114819214, 3243262023331941781, 4213737472390389773, -4046361821170607634, 8877904009116429296, -6931048276693039052, 4838006612846181604, -5561480934050473057, -470112649587309682, 3175935810873308999, -1693695808908080717, -3753035103371291265, -2607412695999984337, -8454963020263227780, 2037428931895594762, 1158209127301347406, -8092787384269386871, -7741092217712244823, 3213269181965324853, 3972662756438857798, -1808499161350392500, -5552429155141285488, 2768019514490102470, -2381885168558935166, 7598271141891576988, -5968675860637356104, -2178161882622813874, -2782395662355757709, 4662660828871465894, 6726990970215445064, 691223925843765893, -3732536320705038428, 2169053732177722520, -3467691490997179851, -3201672755574011994, -706634586120752453, -7297234535099792750, 4195063085031070570, 1024797669903232596, -5102042883065498245, -4295412307398568491, -686079656172478689, -7652228004418103329, -5922734755429174917, 6562442130946482224, 3419893918407781185, -7840156446781283061, -3209525913297552052, -8254134338430272746, -7543559272928655856, -6413145215334169356, 8189387753488304279, 133576451402117013, 6859840908124654784, -480832477584575919, -4466949465409090307, 2224334850433074431, -1077941859365184025, 7694746877711316656, -1541425238506019058, -2694798376156556512, -752352477219592169, 1911593773128947549, 1053380063512932771, 2369074212175473237, 1511764544820953277, -916955813829019462, -3389255868702958135, -4853221732440365560, 436528405098383237, 1482375829041075317, -5200032867873743382, -2359116065936986284, 2323558012249686328, -8714456372962073495, 8264655970455461946, -6377550880012897671, 1904431293553435268, 4281876827769946039, 129146370702108712, -7734952979230886745, -8242033642929677698, 2814257954427384288, -3054904268940900861, -450367636631970004, 8109127225549270088, 8020800599127010921, -2679593348517582224, -1752859515012359396, 4479567094782500113, 6631436750259638425, -5598101192868431054, 2794504876633839794, -1756288804973602742, -534202438069841833, 7027493063976551471, 3982737257386418900, -930887821667260064, 273533908252754639, -1440739443874625754, -2507143240715618713, 2102474087594804091, -7697404848019572605, -8658346826860002424, -4596011304911479578, 6459167476271897881, 6805481881299404326, -7080476038981754572, 5767237378480651194, -978605465701070447, -2322705636227730109, -681083377956364436, -5892502600903861513, -3301298109594166057, 8752787692307586763, -7189731757057511702, 8976826014517372173, -6975924228178934185, 4829963829825594588, -2886015764818641366, -7753659165428942667, -5923615337534732994, 7275900434506691605, 4164818915231369384, 5154263935677458810, 301044058442370317, -7764756071469383634, 7327453618420416121, -2412303784295409238, 781599758351322350, 6625371277047208299, -7611076121968222555, 6127681154442724891, 7313542022544470677, 7222174446262831041, -9138442343184845349, -4110447459792797189, 2534623752979121051]
INFO [main] 2017-06-01 07:20:32,544 StorageService.java:1435 - JOINING: Finish joining ring
Regards,
Akhil
> On 1/06/2017, at 5:28 PM, Junaid Nasir <jn...@an10.io> wrote:
>
> sorry that was an old notetool status output (old cluster with a few configuration changes). new configuration is
>
> -- Address Load Tokens Owns (effective) Host ID Rack
> UN 10.128.1.2 270.75 GiB 256 52.6% e4427611-c247-42ee-9404-371e177f5f17 rack1
> UN 10.128.1.10 388.52 KiB 256 47.4% 7059e622-7861-495b-a69f-5a68876364dd rack1
>
> 10.128.1.2 is seed node, and no other C* node is running on the network.
> data_file_directories on seed node is not in default location, it's on another disk. if that helps
>
> On Thu, Jun 1, 2017 at 4:49 AM, Akhil Mehra <akhilmehra@gmail.com <ma...@gmail.com>> wrote:
> Hi Junaid,
>
> I noticed in the log files that data is being streamed from/to 10.128.1.2 but the address of your original node is 10.128.0.7.
>
> Are there any other Cassandra nodes on your local network the same cluster name.
>
> Regards,
> Akhil
>
> > 10.128.1.2
> > On 31/05/2017, at 10:15 PM, Junaid Nasir <jnasir@an10.io <ma...@an10.io>> wrote:
> >
> > after nodetool repair on new node following log appears in cassandra log
> >
> > INFO [StreamConnectionEstablisher:1] 2017-05-31 10:07:23,496 StreamSession.java:266 - [Stream #f0eb9670-45e8-11e7-a17b-81e9a18c6eac] Starting streaming to /10.128.1.2 <http://10.128.1.2/>
> > INFO [StreamConnectionEstablisher:1] 2017-05-31 10:07:23,501 StreamCoordinator.java:264 - [Stream #f0eb9670-45e8-11e7-a17b-81e9a18c6eac, ID#0] Beginning stream session with /10.128.1.2 <http://10.128.1.2/>
> > INFO [STREAM-IN-/10.128.1.2:7000 <http://10.128.1.2:7000/>] 2017-05-31 10:07:23,545 StreamResultFuture.java:173 - [Stream #f0eb9670-45e8-11e7-a17b-81e9a18c6eac ID#0] Prepare completed. Receiving 2 files(131.594KiB), sending 0 files(0.000KiB)
> > INFO [StreamReceiveTask:1] 2017-05-31 10:07:23,705 StreamResultFuture.java:187 - [Stream #f0eb9670-45e8-11e7-a17b-81e9a18c6eac] Session with /10.128.1.2 <http://10.128.1.2/> is complete
> > INFO [StreamReceiveTask:1] 2017-05-31 10:07:23,707 StreamResultFuture.java:219 - [Stream #f0eb9670-45e8-11e7-a17b-81e9a18c6eac] All sessions completed
> >
> >
> >
> > On Wed, May 31, 2017 at 2:50 PM, Junaid Nasir <jnasir@an10.io <ma...@an10.io>> wrote:
> > Cassandra version is 3.10, and yes its not a production server. i have seen some warnings in logs saying token exist on both servers. other than that nothing. if you need any more settings/details please ask. thank you for your time
> >
> >
> > On Wed, May 31, 2017 at 2:04 PM, Oleksandr Shulgin <oleksandr.shulgin@zalando.de <ma...@zalando.de>> wrote:
> > On Wed, May 31, 2017 at 9:55 AM, Junaid Nasir <jnasir@an10.io <ma...@an10.io>> wrote:
> > Cassandra ensure that adding or removing nodes are very easy and that load is balanced between nodes when a change is made. but it's not working in my case.
> > I have a single node C* deployment (with 270 GB of data) and want to load balance the data on multiple nodes,
> >
> > I guess it's fair to assume this is not a production "cluster"?
> >
> > I followed this guide
> > `nodetool status` shows 2 nodes but load is not balanced between them
> > Datacenter: dc1
> > ===============
> > Status=Up/Down
> > |/ State=Normal/Leaving/Joining/M
> > oving
> > -- Address Load Tokens Owns (effective) Host ID Rack
> > UN 10.128.0.7 270.75 GiB 256 48.6% 1a3f6faa-4376-45a8-9c20-11480a
> > e5664c rack1
> > UN 10.128.0.14 414.36 KiB 256 51.4% 66a89fbf-08ba-4b5d-9f10-55d52a
> > 199b41 rack1
> > I can imagine you are referring to Load column here, which shows only 400 KB for the new node. Did the newly added node actually bootstrap? Are there any error/warning messages in the Cassandra log?
> >
> > I also ran 'nodetool repair' on new node but result is same. any pointers would be appreciated :)
> >
> > Hm, this is not expected. Even if the node didn't bootstrap, repair should have streamed the data it is responsible for.
> >
> > conf file of new node
> > cluster_name: 'cluster1'
> > - seeds: "10.128.0.7"
> >
> > num_tokens: 256
> > endpoint_snitch: GossipingPropertyFileSnitch
> >
> > I see that you've tried to add the new node as a normal (non-seed) node, which supposed to work and it should have bootstrapped itself. Which version of Cassandra is it exactly?
> >
> > --
> > Oleksandr "Alex" Shulgin | Database Engineer | Zalando SE | Tel: +49 176 127-59-707
> >
> >
> >
>
>
Re: Convert single node C* to cluster (rebalancing problem)
Posted by Junaid Nasir <jn...@an10.io>.
sorry that was an old notetool status output (old cluster with a few
configuration changes). new configuration is
-- Address Load Tokens Owns (effective) Host ID
Rack
UN 10.128.1.2 270.75 GiB 256 52.6%
e4427611-c247-42ee-9404-371e177f5f17 rack1
UN 10.128.1.10 388.52 KiB 256 47.4%
7059e622-7861-495b-a69f-5a68876364dd rack1
10.128.1.2 is seed node, and no other C* node is running on the network.
data_file_directories on seed node is not in default location, it's on
another disk. if that helps
On Thu, Jun 1, 2017 at 4:49 AM, Akhil Mehra <ak...@gmail.com> wrote:
> Hi Junaid,
>
> I noticed in the log files that data is being streamed from/to 10.128.1.2
> but the address of your original node is 10.128.0.7.
>
> Are there any other Cassandra nodes on your local network the same cluster
> name.
>
> Regards,
> Akhil
>
> > 10.128.1.2
> > On 31/05/2017, at 10:15 PM, Junaid Nasir <jn...@an10.io> wrote:
> >
> > after nodetool repair on new node following log appears in cassandra log
> >
> > INFO [StreamConnectionEstablisher:1] 2017-05-31 10:07:23,496
> StreamSession.java:266 - [Stream #f0eb9670-45e8-11e7-a17b-81e9a18c6eac]
> Starting streaming to /10.128.1.2
> > INFO [StreamConnectionEstablisher:1] 2017-05-31 10:07:23,501
> StreamCoordinator.java:264 - [Stream #f0eb9670-45e8-11e7-a17b-81e9a18c6eac,
> ID#0] Beginning stream session with /10.128.1.2
> > INFO [STREAM-IN-/10.128.1.2:7000] 2017-05-31 10:07:23,545
> StreamResultFuture.java:173 - [Stream #f0eb9670-45e8-11e7-a17b-81e9a18c6eac
> ID#0] Prepare completed. Receiving 2 files(131.594KiB), sending 0
> files(0.000KiB)
> > INFO [StreamReceiveTask:1] 2017-05-31 10:07:23,705
> StreamResultFuture.java:187 - [Stream #f0eb9670-45e8-11e7-a17b-81e9a18c6eac]
> Session with /10.128.1.2 is complete
> > INFO [StreamReceiveTask:1] 2017-05-31 10:07:23,707
> StreamResultFuture.java:219 - [Stream #f0eb9670-45e8-11e7-a17b-81e9a18c6eac]
> All sessions completed
> >
> >
> >
> > On Wed, May 31, 2017 at 2:50 PM, Junaid Nasir <jn...@an10.io> wrote:
> > Cassandra version is 3.10, and yes its not a production server. i have
> seen some warnings in logs saying token exist on both servers. other than
> that nothing. if you need any more settings/details please ask. thank you
> for your time
> >
> >
> > On Wed, May 31, 2017 at 2:04 PM, Oleksandr Shulgin <
> oleksandr.shulgin@zalando.de> wrote:
> > On Wed, May 31, 2017 at 9:55 AM, Junaid Nasir <jn...@an10.io> wrote:
> > Cassandra ensure that adding or removing nodes are very easy and that
> load is balanced between nodes when a change is made. but it's not working
> in my case.
> > I have a single node C* deployment (with 270 GB of data) and want to
> load balance the data on multiple nodes,
> >
> > I guess it's fair to assume this is not a production "cluster"?
> >
> > I followed this guide
> > `nodetool status` shows 2 nodes but load is not balanced between them
> > Datacenter: dc1
> > ===============
> > Status=Up/Down
> > |/ State=Normal/Leaving/Joining/M
> > oving
> > -- Address Load Tokens Owns (effective) Host ID
> Rack
> > UN 10.128.0.7 270.75 GiB 256 48.6%
> 1a3f6faa-4376-45a8-9c20-11480a
> > e5664c rack1
> > UN 10.128.0.14 414.36 KiB 256 51.4%
> 66a89fbf-08ba-4b5d-9f10-55d52a
> > 199b41 rack1
> > I can imagine you are referring to Load column here, which shows only
> 400 KB for the new node. Did the newly added node actually bootstrap? Are
> there any error/warning messages in the Cassandra log?
> >
> > I also ran 'nodetool repair' on new node but result is same. any
> pointers would be appreciated :)
> >
> > Hm, this is not expected. Even if the node didn't bootstrap, repair
> should have streamed the data it is responsible for.
> >
> > conf file of new node
> > cluster_name: 'cluster1'
> > - seeds: "10.128.0.7"
> >
> > num_tokens: 256
> > endpoint_snitch: GossipingPropertyFileSnitch
> >
> > I see that you've tried to add the new node as a normal (non-seed) node,
> which supposed to work and it should have bootstrapped itself. Which
> version of Cassandra is it exactly?
> >
> > --
> > Oleksandr "Alex" Shulgin | Database Engineer | Zalando SE | Tel: +49 176
> 127-59-707
> >
> >
> >
>
>
Re: Convert single node C* to cluster (rebalancing problem)
Posted by Akhil Mehra <ak...@gmail.com>.
Hi Junaid,
I noticed in the log files that data is being streamed from/to 10.128.1.2 but the address of your original node is 10.128.0.7.
Are there any other Cassandra nodes on your local network the same cluster name.
Regards,
Akhil
> 10.128.1.2
> On 31/05/2017, at 10:15 PM, Junaid Nasir <jn...@an10.io> wrote:
>
> after nodetool repair on new node following log appears in cassandra log
>
> INFO [StreamConnectionEstablisher:1] 2017-05-31 10:07:23,496 StreamSession.java:266 - [Stream #f0eb9670-45e8-11e7-a17b-81e9a18c6eac] Starting streaming to /10.128.1.2
> INFO [StreamConnectionEstablisher:1] 2017-05-31 10:07:23,501 StreamCoordinator.java:264 - [Stream #f0eb9670-45e8-11e7-a17b-81e9a18c6eac, ID#0] Beginning stream session with /10.128.1.2
> INFO [STREAM-IN-/10.128.1.2:7000] 2017-05-31 10:07:23,545 StreamResultFuture.java:173 - [Stream #f0eb9670-45e8-11e7-a17b-81e9a18c6eac ID#0] Prepare completed. Receiving 2 files(131.594KiB), sending 0 files(0.000KiB)
> INFO [StreamReceiveTask:1] 2017-05-31 10:07:23,705 StreamResultFuture.java:187 - [Stream #f0eb9670-45e8-11e7-a17b-81e9a18c6eac] Session with /10.128.1.2 is complete
> INFO [StreamReceiveTask:1] 2017-05-31 10:07:23,707 StreamResultFuture.java:219 - [Stream #f0eb9670-45e8-11e7-a17b-81e9a18c6eac] All sessions completed
>
>
>
> On Wed, May 31, 2017 at 2:50 PM, Junaid Nasir <jn...@an10.io> wrote:
> Cassandra version is 3.10, and yes its not a production server. i have seen some warnings in logs saying token exist on both servers. other than that nothing. if you need any more settings/details please ask. thank you for your time
>
>
> On Wed, May 31, 2017 at 2:04 PM, Oleksandr Shulgin <ol...@zalando.de> wrote:
> On Wed, May 31, 2017 at 9:55 AM, Junaid Nasir <jn...@an10.io> wrote:
> Cassandra ensure that adding or removing nodes are very easy and that load is balanced between nodes when a change is made. but it's not working in my case.
> I have a single node C* deployment (with 270 GB of data) and want to load balance the data on multiple nodes,
>
> I guess it's fair to assume this is not a production "cluster"?
>
> I followed this guide
> `nodetool status` shows 2 nodes but load is not balanced between them
> Datacenter: dc1
> ===============
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/M
> oving
> -- Address Load Tokens Owns (effective) Host ID Rack
> UN 10.128.0.7 270.75 GiB 256 48.6% 1a3f6faa-4376-45a8-9c20-11480a
> e5664c rack1
> UN 10.128.0.14 414.36 KiB 256 51.4% 66a89fbf-08ba-4b5d-9f10-55d52a
> 199b41 rack1
> I can imagine you are referring to Load column here, which shows only 400 KB for the new node. Did the newly added node actually bootstrap? Are there any error/warning messages in the Cassandra log?
>
> I also ran 'nodetool repair' on new node but result is same. any pointers would be appreciated :)
>
> Hm, this is not expected. Even if the node didn't bootstrap, repair should have streamed the data it is responsible for.
>
> conf file of new node
> cluster_name: 'cluster1'
> - seeds: "10.128.0.7"
>
> num_tokens: 256
> endpoint_snitch: GossipingPropertyFileSnitch
>
> I see that you've tried to add the new node as a normal (non-seed) node, which supposed to work and it should have bootstrapped itself. Which version of Cassandra is it exactly?
>
> --
> Oleksandr "Alex" Shulgin | Database Engineer | Zalando SE | Tel: +49 176 127-59-707
>
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
For additional commands, e-mail: user-help@cassandra.apache.org
Re: Convert single node C* to cluster (rebalancing problem)
Posted by Junaid Nasir <jn...@an10.io>.
after nodetool repair on new node following log appears in cassandra log
INFO [StreamConnectionEstablisher:1] 2017-05-31 10:07:23,496
> StreamSession.java:266 - [Stream #f0eb9670-45e8-11e7-a17b-81e9a18c6eac]
> Starting streaming to /10.128.1.2
> INFO [StreamConnectionEstablisher:1] 2017-05-31 10:07:23,501
> StreamCoordinator.java:264 - [Stream #f0eb9670-45e8-11e7-a17b-81e9a18c6eac,
> ID#0] Beginning stream session with /10.128.1.2
> INFO [STREAM-IN-/10.128.1.2:7000] 2017-05-31 10:07:23,545
> StreamResultFuture.java:173 - [Stream #f0eb9670-45e8-11e7-a17b-81e9a18c6eac
> ID#0] Prepare completed. Receiving 2 files(131.594KiB), sending 0
> files(0.000KiB)
> INFO [StreamReceiveTask:1] 2017-05-31 10:07:23,705
> StreamResultFuture.java:187 - [Stream
> #f0eb9670-45e8-11e7-a17b-81e9a18c6eac] Session with /10.128.1.2 is
> complete
> INFO [StreamReceiveTask:1] 2017-05-31 10:07:23,707
> StreamResultFuture.java:219 - [Stream
> #f0eb9670-45e8-11e7-a17b-81e9a18c6eac] All sessions completed
On Wed, May 31, 2017 at 2:50 PM, Junaid Nasir <jn...@an10.io> wrote:
> Cassandra version is 3.10, and yes its not a production server. i have
> seen some warnings in logs saying token exist on both servers. other than
> that nothing. if you need any more settings/details please ask. thank you
> for your time
>
>
> On Wed, May 31, 2017 at 2:04 PM, Oleksandr Shulgin <
> oleksandr.shulgin@zalando.de> wrote:
>
>> On Wed, May 31, 2017 at 9:55 AM, Junaid Nasir <jn...@an10.io> wrote:
>>
>>> Cassandra ensure that adding or removing nodes are very easy and that
>>> load is balanced between nodes when a change is made. but it's not working
>>> in my case.
>>> I have a single node C* deployment (with 270 GB of data) and want to
>>> load balance the data on multiple nodes,
>>>
>>
>> I guess it's fair to assume this is not a production "cluster"?
>>
>>
>>> I followed this guide
>>> <https://docs.datastax.com/en/cassandra/2.1/cassandra/operations/ops_add_node_to_cluster_t.html>
>>>
>>> `nodetool status` shows 2 nodes but load is not balanced between them
>>>
>>> Datacenter: dc1
>>> ===============
>>> Status=Up/Down
>>> |/ State=Normal/Leaving/Joining/Moving
>>> -- Address Load Tokens Owns (effective) Host ID Rack
>>> UN 10.128.0.7 270.75 GiB 256 48.6% 1a3f6faa-4376-45a8-9c20-11480ae5664c rack1
>>> UN 10.128.0.14 414.36 KiB 256 51.4% 66a89fbf-08ba-4b5d-9f10-55d52a199b41 rack1
>>>
>>> I can imagine you are referring to Load column here, which shows only
>> 400 KB for the new node. Did the newly added node actually bootstrap? Are
>> there any error/warning messages in the Cassandra log?
>>
>>
>>> I also ran 'nodetool repair' on new node but result is same. any
>>> pointers would be appreciated :)
>>>
>>
>> Hm, this is not expected. Even if the node didn't bootstrap, repair
>> should have streamed the data it is responsible for.
>>
>> conf file of new node
>>>
>>> cluster_name: 'cluster1'
>>> - seeds: "10.128.0.7"
>>> num_tokens: 256
>>> endpoint_snitch: GossipingPropertyFileSnitch
>>>
>>> I see that you've tried to add the new node as a normal (non-seed) node,
>> which supposed to work and it should have bootstrapped itself. Which
>> version of Cassandra is it exactly?
>>
>> --
>> Oleksandr "Alex" Shulgin | Database Engineer | Zalando SE | Tel: +49 176
>> 127-59-707 <+49%20176%2012759707>
>>
>>
>
Re: Convert single node C* to cluster (rebalancing problem)
Posted by Junaid Nasir <jn...@an10.io>.
Cassandra version is 3.10, and yes its not a production server. i have seen
some warnings in logs saying token exist on both servers. other than that
nothing. if you need any more settings/details please ask. thank you for
your time
On Wed, May 31, 2017 at 2:04 PM, Oleksandr Shulgin <
oleksandr.shulgin@zalando.de> wrote:
> On Wed, May 31, 2017 at 9:55 AM, Junaid Nasir <jn...@an10.io> wrote:
>
>> Cassandra ensure that adding or removing nodes are very easy and that
>> load is balanced between nodes when a change is made. but it's not working
>> in my case.
>> I have a single node C* deployment (with 270 GB of data) and want to load
>> balance the data on multiple nodes,
>>
>
> I guess it's fair to assume this is not a production "cluster"?
>
>
>> I followed this guide
>> <https://docs.datastax.com/en/cassandra/2.1/cassandra/operations/ops_add_node_to_cluster_t.html>
>>
>> `nodetool status` shows 2 nodes but load is not balanced between them
>>
>> Datacenter: dc1
>> ===============
>> Status=Up/Down
>> |/ State=Normal/Leaving/Joining/Moving
>> -- Address Load Tokens Owns (effective) Host ID Rack
>> UN 10.128.0.7 270.75 GiB 256 48.6% 1a3f6faa-4376-45a8-9c20-11480ae5664c rack1
>> UN 10.128.0.14 414.36 KiB 256 51.4% 66a89fbf-08ba-4b5d-9f10-55d52a199b41 rack1
>>
>> I can imagine you are referring to Load column here, which shows only 400
> KB for the new node. Did the newly added node actually bootstrap? Are
> there any error/warning messages in the Cassandra log?
>
>
>> I also ran 'nodetool repair' on new node but result is same. any pointers
>> would be appreciated :)
>>
>
> Hm, this is not expected. Even if the node didn't bootstrap, repair
> should have streamed the data it is responsible for.
>
> conf file of new node
>>
>> cluster_name: 'cluster1'
>> - seeds: "10.128.0.7"
>> num_tokens: 256
>> endpoint_snitch: GossipingPropertyFileSnitch
>>
>> I see that you've tried to add the new node as a normal (non-seed) node,
> which supposed to work and it should have bootstrapped itself. Which
> version of Cassandra is it exactly?
>
> --
> Oleksandr "Alex" Shulgin | Database Engineer | Zalando SE | Tel: +49 176
> 127-59-707 <+49%20176%2012759707>
>
>
Re: Convert single node C* to cluster (rebalancing problem)
Posted by Oleksandr Shulgin <ol...@zalando.de>.
On Wed, May 31, 2017 at 9:55 AM, Junaid Nasir <jn...@an10.io> wrote:
> Cassandra ensure that adding or removing nodes are very easy and that load
> is balanced between nodes when a change is made. but it's not working in my
> case.
> I have a single node C* deployment (with 270 GB of data) and want to load
> balance the data on multiple nodes,
>
I guess it's fair to assume this is not a production "cluster"?
> I followed this guide
> <https://docs.datastax.com/en/cassandra/2.1/cassandra/operations/ops_add_node_to_cluster_t.html>
>
> `nodetool status` shows 2 nodes but load is not balanced between them
>
> Datacenter: dc1
> ===============
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> -- Address Load Tokens Owns (effective) Host ID Rack
> UN 10.128.0.7 270.75 GiB 256 48.6% 1a3f6faa-4376-45a8-9c20-11480ae5664c rack1
> UN 10.128.0.14 414.36 KiB 256 51.4% 66a89fbf-08ba-4b5d-9f10-55d52a199b41 rack1
>
> I can imagine you are referring to Load column here, which shows only 400
KB for the new node. Did the newly added node actually bootstrap? Are
there any error/warning messages in the Cassandra log?
> I also ran 'nodetool repair' on new node but result is same. any pointers
> would be appreciated :)
>
Hm, this is not expected. Even if the node didn't bootstrap, repair should
have streamed the data it is responsible for.
conf file of new node
>
> cluster_name: 'cluster1'
> - seeds: "10.128.0.7"
> num_tokens: 256
> endpoint_snitch: GossipingPropertyFileSnitch
>
> I see that you've tried to add the new node as a normal (non-seed) node,
which supposed to work and it should have bootstrapped itself. Which
version of Cassandra is it exactly?
--
Oleksandr "Alex" Shulgin | Database Engineer | Zalando SE | Tel: +49 176
127-59-707