You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@cassandra.apache.org by Bhuvan Rawal <bh...@gmail.com> on 2016/01/22 09:57:50 UTC

Getting error while issuing Cassandra stress

Hi,

i have created a POC cluster with 2 DC , each having 4 nodes with DSE 4.8.1
installed.

On issuing cassandra stress im getting an error  and data is not being
inserted:
*com.datastax.driver.core.exceptions.UnavailableException: Not enough
replica available for query at consistency LOCAL_ONE (1 required but only 0
alive)*

However, Im able to create keyspace, tables and insert data using cqlsh and
it is replicating fine to all the nodes.

Details of the cluster can be found below (all the nodes seem to be alive
and kicking):

$ nodetool status Datacenter: Analytics =====================
Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address Load
Tokens Owns Host ID Rack UN 10.41.55.17 428.5 KB 256 ?
39d6d585-e641-4046-9d0b-797356597b5e rack1 UN 10.41.55.19 404.44 KB 256 ?
69edf930-efd9-4d74-a798-f3d4ac02e516 rack1 UN 10.41.55.18 423.21 KB 256 ?
b74bab13-09b2-4760-bce9-c8ef05e50f6d rack1 UN 10.41.55.20 683.23 KB 256 ?
fb5c4fed-6e1e-4ea8-838d-358106906830 rack1 Datacenter: Cassandra
===================== Status=Up/Down |/ State=Normal/Leaving/Joining/Moving --
Address Load Tokens Owns Host ID Rack UN 10.41.55.15 209.4 KB 256 ?
ffc3b9a0-5d5c-4a3d-a99e-49d255731278 rack1 UN 10.41.55.21 227.44 KB 256 ?
c68deba4-b9a2-43fc-bb13-6af74c88c210 rack1 UN 10.41.55.23 222.71 KB 256 ?
8229aa87-af00-48fa-ad6b-3066d3dc0e58 rack1 UN 10.41.55.22 218.72 KB 256 ?
c7ba84fd-7992-41de-8c88-11574a72db99 rack1

Regards,
Bhuvan Rawal

Re: Getting error while issuing Cassandra stress

Posted by Alain RODRIGUEZ <ar...@gmail.com>.

Hi Bhuvan,

I guess this info will be useful -->
https://docs.datastax.com/en/cassandra/2.1/cassandra/tools/toolsCStress_t.html#task_ds_rnm_r53_gk__view-schema-help

You could try defining you own settings for the schema
(NetworkTopologyStrategy,
RF of your choice).

Yet SimpleStrategy should work just fine with local_one since
https://issues.apache.org/jira/browse/CASSANDRA-6238. So your version
should be ok. I might have missed a lot of info recently though. That's the
flip side of holidays...

Anyway, there is no point in not using NTS if you have 2 DC, as they will
be considered as one. Also you should change this
https://github.com/apache/cassandra/blob/cassandra-2.1.12/conf/cassandra.yaml#L696
before going to production. I think it is fine for testing purpose.

I still think something else might be wrong. Maybe about your network, just
a guess. You could try this on the node you run the stress tool from:

$ telnet 9042 <ip_node_you_try_to_reach>

There is another point of concern here which might be relevant to previous
> one as well, im not able to login to cqlsh directly, i.e. I have to specify
> ip as well even when im logged in to that machine.


This should help you with that (the "cqlsh environment variables" par at
the bottom)
http://docs.datastax.com/en/cql/3.1/cql/cql_reference/cqlsh.html#refCqlsh__using-files-as-input

You have to use your listen_address to connect with cqlsh / native off the
top of my head. You'll find out soon enough.

(C*heers :) - im so tempted to copy that)


Feel free, there is no trademark ™ © ® or whatever, I am glad to share.

Who knows, we might be starting today, a new trend :p.

C*heers ;-)

-----------------
Alain

The Last Pickle
http://www.thelastpickle.com

2016-01-22 22:40 GMT+01:00 Sebastian Estevez <sebastian.estevez@datastax.com
>:

> https://github.com/brianmhess/cassandra-loader
>
> All the best,
>
>
> [image: datastax_logo.png] <http://www.datastax.com/>
>
> Sebastián Estévez
>
> Solutions Architect | 954 905 8615 | sebastian.estevez@datastax.com
>
> [image: linkedin.png] <https://www.linkedin.com/company/datastax> [image:
> facebook.png] <https://www.facebook.com/datastax> [image: twitter.png]
> <https://twitter.com/datastax> [image: g+.png]
> <https://plus.google.com/+Datastax/about>
> <http://feeds.feedburner.com/datastax>
> <http://goog_410786983>
>
>
> <http://www.datastax.com/gartner-magic-quadrant-odbms>
>
> DataStax is the fastest, most scalable distributed database technology,
> delivering Apache Cassandra to the world’s most innovative enterprises.
> Datastax is built to be agile, always-on, and predictably scalable to any
> size. With more than 500 customers in 45 countries, DataStax is the
> database technology and transactional backbone of choice for the worlds
> most innovative companies such as Netflix, Adobe, Intuit, and eBay.
>
> On Fri, Jan 22, 2016 at 4:37 PM, Bhuvan Rawal <bh...@gmail.com> wrote:
>
>> Yes im specifying -node parameter to stress, otherwise it throws network
>> connection failed.
>>
>> Can you point me to a sample java application to test pushing data from
>> external server? Let's see if that works
>>
>> On Sat, Jan 23, 2016 at 2:55 AM, Sebastian Estevez <
>> sebastian.estevez@datastax.com> wrote:
>>
>>> when i opened my cassandra-rackdc.properties i saw that DC names were
>>>> DC1 & DC2, rack name was RAC1 . Please note that this is the default
>>>> configuration, I have not modified any file.
>>>
>>>
>>> cassandra-rackdc.properties is only respected based on your snitch
>>> <https://docs.datastax.com/en/cassandra/2.1/cassandra/architecture/architectureSnitchesAbout_c.html>
>>> .
>>>
>>> $ cqlsh
>>>> Connection error: ('Unable to connect to any servers', {'127.0.0.1':
>>>> error(111, "Tried connecting to [('127.0.0.1', 9042)]. Last error:
>>>> Connection refused")})
>>>> whereas
>>>> $ cqlsh <ip>
>>>> works fine
>>>> is that the reason why the cassandra-stress is not able to communicate
>>>> with other replicas?
>>>
>>>
>>> Are you providing the -node parameter to stress
>>> <http://docs.datastax.com/en/cassandra/2.1/cassandra/tools/toolsCStress_t.html>
>>> ?
>>>
>>>
>>>
>>> All the best,
>>>
>>>
>>> [image: datastax_logo.png] <http://www.datastax.com/>
>>>
>>> Sebastián Estévez
>>>
>>> Solutions Architect | 954 905 8615 | sebastian.estevez@datastax.com
>>>
>>> [image: linkedin.png] <https://www.linkedin.com/company/datastax> [image:
>>> facebook.png] <https://www.facebook.com/datastax> [image: twitter.png]
>>> <https://twitter.com/datastax> [image: g+.png]
>>> <https://plus.google.com/+Datastax/about>
>>> <http://feeds.feedburner.com/datastax>
>>> <http://goog_410786983>
>>>
>>>
>>> <http://www.datastax.com/gartner-magic-quadrant-odbms>
>>>
>>> DataStax is the fastest, most scalable distributed database technology,
>>> delivering Apache Cassandra to the world’s most innovative enterprises.
>>> Datastax is built to be agile, always-on, and predictably scalable to any
>>> size. With more than 500 customers in 45 countries, DataStax is the
>>> database technology and transactional backbone of choice for the worlds
>>> most innovative companies such as Netflix, Adobe, Intuit, and eBay.
>>>
>>> On Fri, Jan 22, 2016 at 4:07 PM, Bhuvan Rawal <bh...@gmail.com>
>>> wrote:
>>>
>>>> I had a look at the jira below:
>>>> https://issues.apache.org/jira/browse/CASSANDRA-7905
>>>>
>>>> when i opened my cassandra-rackdc.properties i saw that DC names were
>>>> DC1 & DC2, rack name was RAC1 . Please note that this is the default
>>>> configuration, I have not modified any file.
>>>>
>>>> There is another point of concern here which might be relevant to
>>>> previous one as well, im not able to login to cqlsh directly, i.e. I have
>>>> to specify ip as well even when im logged in to that machine.
>>>>
>>>> $ cqlsh
>>>> Connection error: ('Unable to connect to any servers', {'127.0.0.1':
>>>> error(111, "Tried connecting to [('127.0.0.1', 9042)]. Last error:
>>>> Connection refused")})
>>>>
>>>> whereas
>>>> $ cqlsh <ip>
>>>> works fine
>>>>
>>>> is that the reason why the cassandra-stress is not able to communicate
>>>> with other replicas?
>>>>
>>>> On Sat, Jan 23, 2016 at 1:37 AM, Sebastian Estevez <
>>>> sebastian.estevez@datastax.com> wrote:
>>>>
>>>>> Sorry I missed that.
>>>>>
>>>>> Both your nodetool status and keyspace replication settings say
>>>>> Cassandra and Analytics for the DC names. I'm not sure where you're seeing
>>>>> DC1, DC2, etc. and why you suspect that is the problem.
>>>>>
>>>>> All the best,
>>>>>
>>>>>
>>>>> [image: datastax_logo.png] <http://www.datastax.com/>
>>>>>
>>>>> Sebastián Estévez
>>>>>
>>>>> Solutions Architect | 954 905 8615 | sebastian.estevez@datastax.com
>>>>>
>>>>> [image: linkedin.png] <https://www.linkedin.com/company/datastax> [image:
>>>>> facebook.png] <https://www.facebook.com/datastax> [image: twitter.png]
>>>>> <https://twitter.com/datastax> [image: g+.png]
>>>>> <https://plus.google.com/+Datastax/about>
>>>>> <http://feeds.feedburner.com/datastax>
>>>>> <http://goog_410786983>
>>>>>
>>>>>
>>>>> <http://www.datastax.com/gartner-magic-quadrant-odbms>
>>>>>
>>>>> DataStax is the fastest, most scalable distributed database
>>>>> technology, delivering Apache Cassandra to the world’s most innovative
>>>>> enterprises. Datastax is built to be agile, always-on, and predictably
>>>>> scalable to any size. With more than 500 customers in 45 countries, DataStax
>>>>> is the database technology and transactional backbone of choice for the
>>>>> worlds most innovative companies such as Netflix, Adobe, Intuit, and eBay.
>>>>>
>>>>> On Fri, Jan 22, 2016 at 1:45 PM, Bhuvan Rawal <bh...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hi Sebastian,
>>>>>>
>>>>>> I had attached nodetool status output in previous mail, pasting it
>>>>>> again :
>>>>>>
>>>>>> $ nodetool status Datacenter: Analytics =====================
>>>>>> Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address
>>>>>> Load Tokens Owns Host ID Rack UN 10.41.55.17 428.5 KB 256 ?
>>>>>> 39d6d585-e641-4046-9d0b-797356597b5e rack1 UN 10.41.55.19 404.44 KB 256 ?
>>>>>> 69edf930-efd9-4d74-a798-f3d4ac02e516 rack1 UN 10.41.55.18 423.21 KB 256 ?
>>>>>> b74bab13-09b2-4760-bce9-c8ef05e50f6d rack1 UN 10.41.55.20 683.23 KB 256 ?
>>>>>> fb5c4fed-6e1e-4ea8-838d-358106906830 rack1 Datacenter: Cassandra
>>>>>> ===================== Status=Up/Down |/
>>>>>> State=Normal/Leaving/Joining/Moving -- Address Load Tokens Owns Host
>>>>>> ID Rack UN 10.41.55.15 209.4 KB 256 ? ffc3b9a0-5d5c-4a3d-a99e-49d255731278
>>>>>> rack1 UN 10.41.55.21 227.44 KB 256 ? c68deba4-b9a2-43fc-bb13-6af74c88c210
>>>>>> rack1 UN 10.41.55.23 222.71 KB 256 ? 8229aa87-af00-48fa-ad6b-3066d3dc0e58
>>>>>> rack1 UN 10.41.55.22 218.72 KB 256 ? c7ba84fd-7992-41de-8c88-11574a72db99
>>>>>> rack1
>>>>>>
>>>>>> Regards,
>>>>>> Bhuvan Rawal
>>>>>>
>>>>>> On Sat, Jan 23, 2016 at 12:11 AM, Sebastian Estevez <
>>>>>> sebastian.estevez@datastax.com> wrote:
>>>>>>
>>>>>>> The output of `nodetool status` would help us diagnose.
>>>>>>>
>>>>>>> All the best,
>>>>>>>
>>>>>>>
>>>>>>> [image: datastax_logo.png] <http://www.datastax.com/>
>>>>>>>
>>>>>>> Sebastián Estévez
>>>>>>>
>>>>>>> Solutions Architect | 954 905 8615 | sebastian.estevez@datastax.com
>>>>>>>
>>>>>>> [image: linkedin.png] <https://www.linkedin.com/company/datastax> [image:
>>>>>>> facebook.png] <https://www.facebook.com/datastax> [image:
>>>>>>> twitter.png] <https://twitter.com/datastax> [image: g+.png]
>>>>>>> <https://plus.google.com/+Datastax/about>
>>>>>>> <http://feeds.feedburner.com/datastax>
>>>>>>> <http://goog_410786983>
>>>>>>>
>>>>>>>
>>>>>>> <http://www.datastax.com/gartner-magic-quadrant-odbms>
>>>>>>>
>>>>>>> DataStax is the fastest, most scalable distributed database
>>>>>>> technology, delivering Apache Cassandra to the world’s most innovative
>>>>>>> enterprises. Datastax is built to be agile, always-on, and predictably
>>>>>>> scalable to any size. With more than 500 customers in 45 countries, DataStax
>>>>>>> is the database technology and transactional backbone of choice for the
>>>>>>> worlds most innovative companies such as Netflix, Adobe, Intuit, and eBay.
>>>>>>>
>>>>>>> On Fri, Jan 22, 2016 at 1:39 PM, Bhuvan Rawal <bh...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Thanks for the response Alain,
>>>>>>>>
>>>>>>>> cqlsh> create keyspace mykeyspace WITH replication =
>>>>>>>> {'class':'NetworkTopologyStrategy', 'Analytics':2, 'Cassandra':3}
>>>>>>>> cqlsh> use mykeyspace;
>>>>>>>> cqlsh:mykeyspace>create table mytable (id int primary key, name
>>>>>>>> text, address text, phone text);
>>>>>>>> cqlsh:mykeyspace> insert into mytable (id, name, address, phone)
>>>>>>>> values (1, 'Kiyu','Texas', '555-1212'); # and other similar statement
>>>>>>>> I then issued the below command from every node and found
>>>>>>>> consistent results.
>>>>>>>> cqlsh:mykeyspace> select * from mytable;
>>>>>>>>
>>>>>>>> // Then i repeated the above steps for NetworkTopologyStrategy and
>>>>>>>> found same results
>>>>>>>>
>>>>>>>> I ran basic cassandra stress
>>>>>>>> seed1 - seed of datacenter 1
>>>>>>>>  $ cassandra-stress write n=50000 -rate threads=4 -node
>>>>>>>> any_random_ip
>>>>>>>>  $ cassandra-stress write n=50000 -rate threads=4 -node seed1
>>>>>>>>  $ cassandra-stress write n=50000 -rate threads=4 -node seed1,seed2
>>>>>>>>  $ cassandra-stress write n=50000 -rate threads=4 -node
>>>>>>>> all_8_ip_comma_seperated
>>>>>>>>  $ cassandra-stress write n=1000000 cl=one -mode native cql3
>>>>>>>> -schema keyspace="keyspace1" -pop seq=1..1000000 -node ip1,ip2,ip3,ip4
>>>>>>>>
>>>>>>>> All of them threw the exception
>>>>>>>> *com.datastax.driver.core.exceptions.UnavailableException: Not
>>>>>>>> enough replica available for query at consistency LOCAL_ONE (1 required but
>>>>>>>> only 0 alive)*
>>>>>>>>
>>>>>>>>
>>>>>>>> I have a feeling that the issue is with datacenter name for some
>>>>>>>> reason, because in some config files I found DC name to be like DC1/DC2/DC3
>>>>>>>> in some it is like Cassandra/Analytics (The ones I had specified while
>>>>>>>> installation). Im unsure which yaml/property file to look for correct
>>>>>>>> inconsistency.
>>>>>>>>
>>>>>>>> (C*heers :) - im so tempted to copy that)
>>>>>>>>
>>>>>>>> Regards,
>>>>>>>> Bhuvan
>>>>>>>>
>>>>>>>> On Fri, Jan 22, 2016 at 8:47 PM, Alain RODRIGUEZ <
>>>>>>>> arodrime@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> The the exact command you ran (stress-tool with options) could be
>>>>>>>>> useful to help you on that.
>>>>>>>>>
>>>>>>>>> However, Im able to create keyspace, tables and insert data using
>>>>>>>>>> cqlsh and it is replicating fine to all the nodes.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Having the schema might be useful too.
>>>>>>>>>
>>>>>>>>> Did you ran the cqlsh and the stress-tool from the same server ?
>>>>>>>>> If not, you might want to check the port you use (9042/9160/...) are open.
>>>>>>>>> Also, cqlsh uses local_one by default too. If both commands were
>>>>>>>>> run against the same DC, from the same machine they should behave the same
>>>>>>>>> way. Are they ?
>>>>>>>>>
>>>>>>>>> C*heers,
>>>>>>>>>
>>>>>>>>> -----------------
>>>>>>>>> Alain
>>>>>>>>>
>>>>>>>>> The Last Pickle
>>>>>>>>> http://www.thelastpickle.com
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> 2016-01-22 9:57 GMT+01:00 Bhuvan Rawal <bh...@gmail.com>:
>>>>>>>>>
>>>>>>>>>> Hi,
>>>>>>>>>>
>>>>>>>>>> i have created a POC cluster with 2 DC , each having 4 nodes with
>>>>>>>>>> DSE 4.8.1 installed.
>>>>>>>>>>
>>>>>>>>>> On issuing cassandra stress im getting an error  and data is not
>>>>>>>>>> being inserted:
>>>>>>>>>> *com.datastax.driver.core.exceptions.UnavailableException: Not
>>>>>>>>>> enough replica available for query at consistency LOCAL_ONE (1 required but
>>>>>>>>>> only 0 alive)*
>>>>>>>>>>
>>>>>>>>>> However, Im able to create keyspace, tables and insert data using
>>>>>>>>>> cqlsh and it is replicating fine to all the nodes.
>>>>>>>>>>
>>>>>>>>>> Details of the cluster can be found below (all the nodes seem to
>>>>>>>>>> be alive and kicking):
>>>>>>>>>>
>>>>>>>>>> $ nodetool status Datacenter: Analytics =====================
>>>>>>>>>> Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address
>>>>>>>>>> Load Tokens Owns Host ID Rack UN 10.41.55.17 428.5 KB 256 ?
>>>>>>>>>> 39d6d585-e641-4046-9d0b-797356597b5e rack1 UN 10.41.55.19 404.44 KB 256 ?
>>>>>>>>>> 69edf930-efd9-4d74-a798-f3d4ac02e516 rack1 UN 10.41.55.18 423.21 KB 256 ?
>>>>>>>>>> b74bab13-09b2-4760-bce9-c8ef05e50f6d rack1 UN 10.41.55.20 683.23 KB 256 ?
>>>>>>>>>> fb5c4fed-6e1e-4ea8-838d-358106906830 rack1 Datacenter: Cassandra
>>>>>>>>>> ===================== Status=Up/Down |/
>>>>>>>>>> State=Normal/Leaving/Joining/Moving -- Address Load Tokens Owns
>>>>>>>>>> Host ID Rack UN 10.41.55.15 209.4 KB 256 ?
>>>>>>>>>> ffc3b9a0-5d5c-4a3d-a99e-49d255731278 rack1 UN 10.41.55.21 227.44 KB 256 ?
>>>>>>>>>> c68deba4-b9a2-43fc-bb13-6af74c88c210 rack1 UN 10.41.55.23 222.71 KB 256 ?
>>>>>>>>>> 8229aa87-af00-48fa-ad6b-3066d3dc0e58 rack1 UN 10.41.55.22 218.72 KB 256 ?
>>>>>>>>>> c7ba84fd-7992-41de-8c88-11574a72db99 rack1
>>>>>>>>>>
>>>>>>>>>> Regards,
>>>>>>>>>> Bhuvan Rawal
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>

Re: Getting error while issuing Cassandra stress

Posted by Bhuvan Rawal <bh...@gmail.com>.

Well this certainly looks like the case to me. Changing the RF did the
trick.

The test ran fine, but I could still observe the same exception here and
there when threads are high.
On 24 Jan 2016 10:12 pm, "Carlos Alonso" <in...@mrcalonso.com> wrote:

> Before closing this.
>
> Would the reason possibly be that with SimpleStrategy and RF=1 the data is
> only on one node on a single DC, and when the driver was connecting to the
> other DC, LOCAL_ONE could not be achieved?
>
> Regards.
>
> Carlos Alonso | Software Engineer | @calonso <https://twitter.com/calonso>
>
> On 23 January 2016 at 17:38, Bhuvan Rawal <bh...@gmail.com> wrote:
>
>> Hi Alan,
>>
>> You have hit bulls eye! I used a custom yaml file for configuring
>> keyspace, used NTT and multiple replications for each cluster, defined
>> table schema and query and it seemed to work great!!
>>
>> I have some apprehensions here though. The tests that I have done did not
>> yield results that I expected, Im not sure if I configured everything
>> correctly. Sharing results in a separate mail.
>>
>> Thanks & Regards,
>> Bhuvan
>>
>> On Sat, Jan 23, 2016 at 5:59 PM, Alain RODRIGUEZ <ar...@gmail.com>
>> wrote:
>>
>>> telnet 10.41.55.18 7199 # doesnt connect
>>>> Trying 10.41.55.18...
>>>> telnet: connect to address 10.41.55.18: Connection refused
>>>>
>>>> Could this be the reason?
>>>>
>>>
>>> I don't think so. But I have not a clue on what's going on...
>>>
>>> Did you tried this ?
>>>
>>> - Try increasing the RF / Strategy in the stress tool
>>>>
>>>
>>> -schema option, pass a file if it is easier, with a custom keyspace,
>>> defining NTS and a RF of 2 or 3. You don't want to use RF = 1 nor
>>> SimpleStrategy in Multi DC under real conditions anyway, so your test will
>>> be more relevant.
>>> You can paste us the command + schema + output / error (if any)
>>>
>>>
>>>> - Try with an other consistency level (not LOCAL_*, as mentioned here :
>>>> http://stackoverflow.com/questions/32055251/not-enough-replica-available-for-query-at-consistency-local-one-1-required-but
>>>> )
>>>>
>>>
>>> Shouldn't affect you, yet, it is easy enough to test, so it might be
>>> worth it. Maybe the first thing I would try, just out of curiosity.
>>>
>>> C*heers,
>>>
>>> -----------------
>>> Alain
>>>
>>> The Last Pickle
>>> http://www.thelastpickle.com
>>>
>>> 2016-01-23 11:13 GMT+01:00 Bhuvan Rawal <bh...@gmail.com>:
>>>
>>>> Alright, I uninstalled the DSE from all the nodes in the cluster and
>>>> reinstalled them again from scratch. Ran nodetool status and output is U/N
>>>> for all nodes again.
>>>>
>>>> Tried running cassandra-stress and it doesnt work.
>>>>
>>>> I can see in the system.peer table that token ranges has been
>>>> distributed, further, other columns seem normal too.
>>>> cqlsh> select peer, data_center, host_id,  rack, release_version,
>>>> rpc_address, workload from system.peers;
>>>>
>>>> peer | data_center | host_id | rack | release_version | rpc_address |
>>>> workload
>>>> -------------+-------------+--------------------------------------+-------+-----------------+-------------+----------- 10.41.55.23
>>>> | Cassandra | b48f7683-3f42-4c49-9ce8-694112a21d5d | rack1 | 2.1.12.1046 |
>>>> 10.41.55.23 | Cassandra 10.41.55.19 | Analytics |
>>>> f271657f-52e2-4f43-8bbe-03a979852679 | rack1 | 2.1.12.1046 | 10.41.55.19 |
>>>> Analytics 10.41.55.20 | Analytics |
>>>> e4f484c9-8e7f-48b7-bce8-2a91d70be4d1 | rack1 | 2.1.12.1046 | 10.41.55.20 |
>>>> Analytics 10.41.55.18 | Analytics |
>>>> 813c2610-0656-4714-9ec6-99d51c72da92 | rack1 | 2.1.12.1046 | 10.41.55.18 |
>>>> Analytics 10.41.55.15 | Cassandra |
>>>> 2b09a481-3e08-45e5-a2d0-4c3d5d611ef6 | rack1 | 2.1.12.1046 | 10.41.55.15 |
>>>> Cassandra 10.41.55.22 | Cassandra |
>>>> d16cc704-4453-4dc0-880e-4af6fbd0dc48 | rack1 | 2.1.12.1046 | 10.41.55.22 |
>>>> Cassandra 10.41.55.17 | Analytics |
>>>> dfbb4f0e-2ffa-4694-811c-1167662ad537 | rack1 | 2.1.12.1046 | 10.41.55.17 |
>>>> Analytics
>>>>
>>>> This seems to be fine too:
>>>> $ nodetool describecluster Cluster Information: Name: POC Cluster
>>>> Snitch: org.apache.cassandra.locator.DynamicEndpointSnitch Partitioner:
>>>> org.apache.cassandra.dht.Murmur3Partitioner Schema versions:
>>>> 07d395d2-e5b8-3e67-addd-3f8d1e8d7803: [10.41.55.15, 10.41.55.17,
>>>> 10.41.55.19, 10.41.55.18, 10.41.55.21, 10.41.55.20, 10.41.55.23,
>>>> 10.41.55.22]
>>>>
>>>> I tried to telnet to a node of other cluster which is not a seed, here
>>>> are the observations :
>>>> Telnet to ports 9042, 7000,9160 seem to work fine
>>>> telnet 10.41.55.18 9042 # connects
>>>> telnet 10.41.55.18 7000 # connects
>>>> telnet 10.41.55.18 9160 # connects
>>>> But telnet to JMX port is not working
>>>> telnet 10.41.55.18 7199 # doesnt connect
>>>> Trying 10.41.55.18...
>>>> telnet: connect to address 10.41.55.18: Connection refused
>>>>
>>>> Could this be the reason?
>>>>
>>>> On Sat, Jan 23, 2016 at 7:33 AM, Alain RODRIGUEZ <ar...@gmail.com>
>>>> wrote:
>>>>
>>>>> Should I use nodetool repair utility
>>>>>>
>>>>>
>>>>> That wouldn't help, this an anti-entropy mechanism (see
>>>>> https://docs.datastax.com/en/cassandra/2.1/cassandra/tools/toolsRepair.html#toolsRepair__description_unique_27
>>>>> ).
>>>>>
>>>>> It is something really important too often left aside.
>>>>>
>>>>> Yet, your issue here is not about consistency. The client can't find
>>>>> any node in charge of the read / written tokens in the ring. This depends
>>>>> on the topology, the replication factor and your network mainly. I think
>>>>> there is something wrong in your setup. I would try this:
>>>>>
>>>>> - Make sure connection / port are ok
>>>>> - Try increasing the RF / Strategy in the stress tool
>>>>> - Try with an other consistency level (not LOCAL_*, as mentioned here
>>>>> :
>>>>> http://stackoverflow.com/questions/32055251/not-enough-replica-available-for-query-at-consistency-local-one-1-required-but
>>>>> )
>>>>>
>>>>> Good luck,
>>>>>
>>>>> -----------------
>>>>> Alain
>>>>>
>>>>> The Last Pickle
>>>>> http://www.thelastpickle.com
>>>>>
>>>>> 2016-01-22 23:02 GMT+01:00 Bhuvan Rawal <bh...@gmail.com>:
>>>>>
>>>>>> Getting same exception again. Should I use nodetool repair utility?
>>>>>>
>>>>>> On Sat, Jan 23, 2016 at 3:10 AM, Sebastian Estevez <
>>>>>> sebastian.estevez@datastax.com> wrote:
>>>>>>
>>>>>>> https://github.com/brianmhess/cassandra-loader
>>>>>>>
>>>>>>> All the best,
>>>>>>>
>>>>>>>
>>>>>>> [image: datastax_logo.png] <http://www.datastax.com/>
>>>>>>>
>>>>>>> Sebastián Estévez
>>>>>>>
>>>>>>> Solutions Architect | 954 905 8615 | sebastian.estevez@datastax.com
>>>>>>>
>>>>>>> [image: linkedin.png] <https://www.linkedin.com/company/datastax> [image:
>>>>>>> facebook.png] <https://www.facebook.com/datastax> [image:
>>>>>>> twitter.png] <https://twitter.com/datastax> [image: g+.png]
>>>>>>> <https://plus.google.com/+Datastax/about>
>>>>>>> <http://feeds.feedburner.com/datastax>
>>>>>>> <http://goog_410786983>
>>>>>>>
>>>>>>>
>>>>>>> <http://www.datastax.com/gartner-magic-quadrant-odbms>
>>>>>>>
>>>>>>> DataStax is the fastest, most scalable distributed database
>>>>>>> technology, delivering Apache Cassandra to the world’s most innovative
>>>>>>> enterprises. Datastax is built to be agile, always-on, and predictably
>>>>>>> scalable to any size. With more than 500 customers in 45 countries, DataStax
>>>>>>> is the database technology and transactional backbone of choice for the
>>>>>>> worlds most innovative companies such as Netflix, Adobe, Intuit, and eBay.
>>>>>>>
>>>>>>> On Fri, Jan 22, 2016 at 4:37 PM, Bhuvan Rawal <bh...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Yes im specifying -node parameter to stress, otherwise it throws
>>>>>>>> network connection failed.
>>>>>>>>
>>>>>>>> Can you point me to a sample java application to test pushing data
>>>>>>>> from external server? Let's see if that works
>>>>>>>>
>>>>>>>> On Sat, Jan 23, 2016 at 2:55 AM, Sebastian Estevez <
>>>>>>>> sebastian.estevez@datastax.com> wrote:
>>>>>>>>
>>>>>>>>> when i opened my cassandra-rackdc.properties i saw that DC names
>>>>>>>>>> were DC1 & DC2, rack name was RAC1 . Please note that this is the default
>>>>>>>>>> configuration, I have not modified any file.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> cassandra-rackdc.properties is only respected based on your snitch
>>>>>>>>> <https://docs.datastax.com/en/cassandra/2.1/cassandra/architecture/architectureSnitchesAbout_c.html>
>>>>>>>>> .
>>>>>>>>>
>>>>>>>>> $ cqlsh
>>>>>>>>>> Connection error: ('Unable to connect to any servers',
>>>>>>>>>> {'127.0.0.1': error(111, "Tried connecting to [('127.0.0.1', 9042)]. Last
>>>>>>>>>> error: Connection refused")})
>>>>>>>>>> whereas
>>>>>>>>>> $ cqlsh <ip>
>>>>>>>>>> works fine
>>>>>>>>>> is that the reason why the cassandra-stress is not able to
>>>>>>>>>> communicate with other replicas?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Are you providing the -node parameter to stress
>>>>>>>>> <http://docs.datastax.com/en/cassandra/2.1/cassandra/tools/toolsCStress_t.html>
>>>>>>>>> ?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> All the best,
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> [image: datastax_logo.png] <http://www.datastax.com/>
>>>>>>>>>
>>>>>>>>> Sebastián Estévez
>>>>>>>>>
>>>>>>>>> Solutions Architect | 954 905 8615 |
>>>>>>>>> sebastian.estevez@datastax.com
>>>>>>>>>
>>>>>>>>> [image: linkedin.png] <https://www.linkedin.com/company/datastax> [image:
>>>>>>>>> facebook.png] <https://www.facebook.com/datastax> [image:
>>>>>>>>> twitter.png] <https://twitter.com/datastax> [image: g+.png]
>>>>>>>>> <https://plus.google.com/+Datastax/about>
>>>>>>>>> <http://feeds.feedburner.com/datastax>
>>>>>>>>> <http://goog_410786983>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> <http://www.datastax.com/gartner-magic-quadrant-odbms>
>>>>>>>>>
>>>>>>>>> DataStax is the fastest, most scalable distributed database
>>>>>>>>> technology, delivering Apache Cassandra to the world’s most innovative
>>>>>>>>> enterprises. Datastax is built to be agile, always-on, and predictably
>>>>>>>>> scalable to any size. With more than 500 customers in 45 countries, DataStax
>>>>>>>>> is the database technology and transactional backbone of choice for the
>>>>>>>>> worlds most innovative companies such as Netflix, Adobe, Intuit, and eBay.
>>>>>>>>>
>>>>>>>>> On Fri, Jan 22, 2016 at 4:07 PM, Bhuvan Rawal <bhu1rawal@gmail.com
>>>>>>>>> > wrote:
>>>>>>>>>
>>>>>>>>>> I had a look at the jira below:
>>>>>>>>>> https://issues.apache.org/jira/browse/CASSANDRA-7905
>>>>>>>>>>
>>>>>>>>>> when i opened my cassandra-rackdc.properties i saw that DC names
>>>>>>>>>> were DC1 & DC2, rack name was RAC1 . Please note that this is the default
>>>>>>>>>> configuration, I have not modified any file.
>>>>>>>>>>
>>>>>>>>>> There is another point of concern here which might be relevant to
>>>>>>>>>> previous one as well, im not able to login to cqlsh directly, i.e. I have
>>>>>>>>>> to specify ip as well even when im logged in to that machine.
>>>>>>>>>>
>>>>>>>>>> $ cqlsh
>>>>>>>>>> Connection error: ('Unable to connect to any servers',
>>>>>>>>>> {'127.0.0.1': error(111, "Tried connecting to [('127.0.0.1', 9042)]. Last
>>>>>>>>>> error: Connection refused")})
>>>>>>>>>>
>>>>>>>>>> whereas
>>>>>>>>>> $ cqlsh <ip>
>>>>>>>>>> works fine
>>>>>>>>>>
>>>>>>>>>> is that the reason why the cassandra-stress is not able to
>>>>>>>>>> communicate with other replicas?
>>>>>>>>>>
>>>>>>>>>> On Sat, Jan 23, 2016 at 1:37 AM, Sebastian Estevez <
>>>>>>>>>> sebastian.estevez@datastax.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Sorry I missed that.
>>>>>>>>>>>
>>>>>>>>>>> Both your nodetool status and keyspace replication settings say
>>>>>>>>>>> Cassandra and Analytics for the DC names. I'm not sure where you're seeing
>>>>>>>>>>> DC1, DC2, etc. and why you suspect that is the problem.
>>>>>>>>>>>
>>>>>>>>>>> All the best,
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> [image: datastax_logo.png] <http://www.datastax.com/>
>>>>>>>>>>>
>>>>>>>>>>> Sebastián Estévez
>>>>>>>>>>>
>>>>>>>>>>> Solutions Architect | 954 905 8615 |
>>>>>>>>>>> sebastian.estevez@datastax.com
>>>>>>>>>>>
>>>>>>>>>>> [image: linkedin.png]
>>>>>>>>>>> <https://www.linkedin.com/company/datastax> [image:
>>>>>>>>>>> facebook.png] <https://www.facebook.com/datastax> [image:
>>>>>>>>>>> twitter.png] <https://twitter.com/datastax> [image: g+.png]
>>>>>>>>>>> <https://plus.google.com/+Datastax/about>
>>>>>>>>>>> <http://feeds.feedburner.com/datastax>
>>>>>>>>>>> <http://goog_410786983>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> <http://www.datastax.com/gartner-magic-quadrant-odbms>
>>>>>>>>>>>
>>>>>>>>>>> DataStax is the fastest, most scalable distributed database
>>>>>>>>>>> technology, delivering Apache Cassandra to the world’s most innovative
>>>>>>>>>>> enterprises. Datastax is built to be agile, always-on, and predictably
>>>>>>>>>>> scalable to any size. With more than 500 customers in 45 countries, DataStax
>>>>>>>>>>> is the database technology and transactional backbone of choice for the
>>>>>>>>>>> worlds most innovative companies such as Netflix, Adobe, Intuit, and eBay.
>>>>>>>>>>>
>>>>>>>>>>> On Fri, Jan 22, 2016 at 1:45 PM, Bhuvan Rawal <
>>>>>>>>>>> bhu1rawal@gmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi Sebastian,
>>>>>>>>>>>>
>>>>>>>>>>>> I had attached nodetool status output in previous mail, pasting
>>>>>>>>>>>> it again :
>>>>>>>>>>>>
>>>>>>>>>>>> $ nodetool status Datacenter: Analytics =====================
>>>>>>>>>>>> Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address
>>>>>>>>>>>> Load Tokens Owns Host ID Rack UN 10.41.55.17 428.5 KB 256 ?
>>>>>>>>>>>> 39d6d585-e641-4046-9d0b-797356597b5e rack1 UN 10.41.55.19 404.44 KB 256 ?
>>>>>>>>>>>> 69edf930-efd9-4d74-a798-f3d4ac02e516 rack1 UN 10.41.55.18 423.21 KB 256 ?
>>>>>>>>>>>> b74bab13-09b2-4760-bce9-c8ef05e50f6d rack1 UN 10.41.55.20 683.23 KB 256 ?
>>>>>>>>>>>> fb5c4fed-6e1e-4ea8-838d-358106906830 rack1 Datacenter:
>>>>>>>>>>>> Cassandra ===================== Status=Up/Down |/
>>>>>>>>>>>> State=Normal/Leaving/Joining/Moving -- Address Load Tokens
>>>>>>>>>>>> Owns Host ID Rack UN 10.41.55.15 209.4 KB 256 ?
>>>>>>>>>>>> ffc3b9a0-5d5c-4a3d-a99e-49d255731278 rack1 UN 10.41.55.21 227.44 KB 256 ?
>>>>>>>>>>>> c68deba4-b9a2-43fc-bb13-6af74c88c210 rack1 UN 10.41.55.23 222.71 KB 256 ?
>>>>>>>>>>>> 8229aa87-af00-48fa-ad6b-3066d3dc0e58 rack1 UN 10.41.55.22 218.72 KB 256 ?
>>>>>>>>>>>> c7ba84fd-7992-41de-8c88-11574a72db99 rack1
>>>>>>>>>>>>
>>>>>>>>>>>> Regards,
>>>>>>>>>>>> Bhuvan Rawal
>>>>>>>>>>>>
>>>>>>>>>>>> On Sat, Jan 23, 2016 at 12:11 AM, Sebastian Estevez <
>>>>>>>>>>>> sebastian.estevez@datastax.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> The output of `nodetool status` would help us diagnose.
>>>>>>>>>>>>>
>>>>>>>>>>>>> All the best,
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> [image: datastax_logo.png] <http://www.datastax.com/>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Sebastián Estévez
>>>>>>>>>>>>>
>>>>>>>>>>>>> Solutions Architect | 954 905 8615 |
>>>>>>>>>>>>> sebastian.estevez@datastax.com
>>>>>>>>>>>>>
>>>>>>>>>>>>> [image: linkedin.png]
>>>>>>>>>>>>> <https://www.linkedin.com/company/datastax> [image:
>>>>>>>>>>>>> facebook.png] <https://www.facebook.com/datastax> [image:
>>>>>>>>>>>>> twitter.png] <https://twitter.com/datastax> [image: g+.png]
>>>>>>>>>>>>> <https://plus.google.com/+Datastax/about>
>>>>>>>>>>>>> <http://feeds.feedburner.com/datastax>
>>>>>>>>>>>>> <http://goog_410786983>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> <http://www.datastax.com/gartner-magic-quadrant-odbms>
>>>>>>>>>>>>>
>>>>>>>>>>>>> DataStax is the fastest, most scalable distributed database
>>>>>>>>>>>>> technology, delivering Apache Cassandra to the world’s most innovative
>>>>>>>>>>>>> enterprises. Datastax is built to be agile, always-on, and predictably
>>>>>>>>>>>>> scalable to any size. With more than 500 customers in 45 countries, DataStax
>>>>>>>>>>>>> is the database technology and transactional backbone of choice for the
>>>>>>>>>>>>> worlds most innovative companies such as Netflix, Adobe, Intuit, and eBay.
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Fri, Jan 22, 2016 at 1:39 PM, Bhuvan Rawal <
>>>>>>>>>>>>> bhu1rawal@gmail.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks for the response Alain,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> cqlsh> create keyspace mykeyspace WITH replication =
>>>>>>>>>>>>>> {'class':'NetworkTopologyStrategy', 'Analytics':2, 'Cassandra':3}
>>>>>>>>>>>>>> cqlsh> use mykeyspace;
>>>>>>>>>>>>>> cqlsh:mykeyspace>create table mytable (id int primary key,
>>>>>>>>>>>>>> name text, address text, phone text);
>>>>>>>>>>>>>> cqlsh:mykeyspace> insert into mytable (id, name, address,
>>>>>>>>>>>>>> phone) values (1, 'Kiyu','Texas', '555-1212'); # and other similar statement
>>>>>>>>>>>>>> I then issued the below command from every node and found
>>>>>>>>>>>>>> consistent results.
>>>>>>>>>>>>>> cqlsh:mykeyspace> select * from mytable;
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> // Then i repeated the above steps for
>>>>>>>>>>>>>> NetworkTopologyStrategy and found same results
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I ran basic cassandra stress
>>>>>>>>>>>>>> seed1 - seed of datacenter 1
>>>>>>>>>>>>>>  $ cassandra-stress write n=50000 -rate threads=4 -node
>>>>>>>>>>>>>> any_random_ip
>>>>>>>>>>>>>>  $ cassandra-stress write n=50000 -rate threads=4 -node seed1
>>>>>>>>>>>>>>  $ cassandra-stress write n=50000 -rate threads=4 -node
>>>>>>>>>>>>>> seed1,seed2
>>>>>>>>>>>>>>  $ cassandra-stress write n=50000 -rate threads=4 -node
>>>>>>>>>>>>>> all_8_ip_comma_seperated
>>>>>>>>>>>>>>  $ cassandra-stress write n=1000000 cl=one -mode native cql3
>>>>>>>>>>>>>> -schema keyspace="keyspace1" -pop seq=1..1000000 -node ip1,ip2,ip3,ip4
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> All of them threw the exception
>>>>>>>>>>>>>> *com.datastax.driver.core.exceptions.UnavailableException:
>>>>>>>>>>>>>> Not enough replica available for query at consistency LOCAL_ONE (1 required
>>>>>>>>>>>>>> but only 0 alive)*
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I have a feeling that the issue is with datacenter name for
>>>>>>>>>>>>>> some reason, because in some config files I found DC name to be like
>>>>>>>>>>>>>> DC1/DC2/DC3 in some it is like Cassandra/Analytics (The ones I had
>>>>>>>>>>>>>> specified while installation). Im unsure which yaml/property file to look
>>>>>>>>>>>>>> for correct inconsistency.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> (C*heers :) - im so tempted to copy that)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>> Bhuvan
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Fri, Jan 22, 2016 at 8:47 PM, Alain RODRIGUEZ <
>>>>>>>>>>>>>> arodrime@gmail.com> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> The the exact command you ran (stress-tool with options)
>>>>>>>>>>>>>>> could be useful to help you on that.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> However, Im able to create keyspace, tables and insert data
>>>>>>>>>>>>>>>> using cqlsh and it is replicating fine to all the nodes.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Having the schema might be useful too.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Did you ran the cqlsh and the stress-tool from the same
>>>>>>>>>>>>>>> server ? If not, you might want to check the port you use (9042/9160/...)
>>>>>>>>>>>>>>> are open.
>>>>>>>>>>>>>>> Also, cqlsh uses local_one by default too. If both commands
>>>>>>>>>>>>>>> were run against the same DC, from the same machine they should behave the
>>>>>>>>>>>>>>> same way. Are they ?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> C*heers,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> -----------------
>>>>>>>>>>>>>>> Alain
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> The Last Pickle
>>>>>>>>>>>>>>> http://www.thelastpickle.com
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> 2016-01-22 9:57 GMT+01:00 Bhuvan Rawal <bh...@gmail.com>
>>>>>>>>>>>>>>> :
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> i have created a POC cluster with 2 DC , each having 4
>>>>>>>>>>>>>>>> nodes with DSE 4.8.1 installed.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On issuing cassandra stress im getting an error  and data
>>>>>>>>>>>>>>>> is not being inserted:
>>>>>>>>>>>>>>>> *com.datastax.driver.core.exceptions.UnavailableException:
>>>>>>>>>>>>>>>> Not enough replica available for query at consistency LOCAL_ONE (1 required
>>>>>>>>>>>>>>>> but only 0 alive)*
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> However, Im able to create keyspace, tables and insert data
>>>>>>>>>>>>>>>> using cqlsh and it is replicating fine to all the nodes.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Details of the cluster can be found below (all the nodes
>>>>>>>>>>>>>>>> seem to be alive and kicking):
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> $ nodetool status Datacenter: Analytics
>>>>>>>>>>>>>>>> ===================== Status=Up/Down |/
>>>>>>>>>>>>>>>> State=Normal/Leaving/Joining/Moving -- Address Load Tokens
>>>>>>>>>>>>>>>> Owns Host ID Rack UN 10.41.55.17 428.5 KB 256 ?
>>>>>>>>>>>>>>>> 39d6d585-e641-4046-9d0b-797356597b5e rack1 UN 10.41.55.19 404.44 KB 256 ?
>>>>>>>>>>>>>>>> 69edf930-efd9-4d74-a798-f3d4ac02e516 rack1 UN 10.41.55.18 423.21 KB 256 ?
>>>>>>>>>>>>>>>> b74bab13-09b2-4760-bce9-c8ef05e50f6d rack1 UN 10.41.55.20 683.23 KB 256 ?
>>>>>>>>>>>>>>>> fb5c4fed-6e1e-4ea8-838d-358106906830 rack1 Datacenter:
>>>>>>>>>>>>>>>> Cassandra ===================== Status=Up/Down |/
>>>>>>>>>>>>>>>> State=Normal/Leaving/Joining/Moving -- Address Load Tokens
>>>>>>>>>>>>>>>> Owns Host ID Rack UN 10.41.55.15 209.4 KB 256 ?
>>>>>>>>>>>>>>>> ffc3b9a0-5d5c-4a3d-a99e-49d255731278 rack1 UN 10.41.55.21 227.44 KB 256 ?
>>>>>>>>>>>>>>>> c68deba4-b9a2-43fc-bb13-6af74c88c210 rack1 UN 10.41.55.23 222.71 KB 256 ?
>>>>>>>>>>>>>>>> 8229aa87-af00-48fa-ad6b-3066d3dc0e58 rack1 UN 10.41.55.22 218.72 KB 256 ?
>>>>>>>>>>>>>>>> c7ba84fd-7992-41de-8c88-11574a72db99 rack1
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>> Bhuvan Rawal
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>

Re: Getting error while issuing Cassandra stress

Posted by Carlos Alonso <in...@mrcalonso.com>.

Before closing this.

Would the reason possibly be that with SimpleStrategy and RF=1 the data is
only on one node on a single DC, and when the driver was connecting to the
other DC, LOCAL_ONE could not be achieved?

Regards.

Carlos Alonso | Software Engineer | @calonso <https://twitter.com/calonso>

On 23 January 2016 at 17:38, Bhuvan Rawal <bh...@gmail.com> wrote:

> Hi Alan,
>
> You have hit bulls eye! I used a custom yaml file for configuring
> keyspace, used NTT and multiple replications for each cluster, defined
> table schema and query and it seemed to work great!!
>
> I have some apprehensions here though. The tests that I have done did not
> yield results that I expected, Im not sure if I configured everything
> correctly. Sharing results in a separate mail.
>
> Thanks & Regards,
> Bhuvan
>
> On Sat, Jan 23, 2016 at 5:59 PM, Alain RODRIGUEZ <ar...@gmail.com>
> wrote:
>
>> telnet 10.41.55.18 7199 # doesnt connect
>>> Trying 10.41.55.18...
>>> telnet: connect to address 10.41.55.18: Connection refused
>>>
>>> Could this be the reason?
>>>
>>
>> I don't think so. But I have not a clue on what's going on...
>>
>> Did you tried this ?
>>
>> - Try increasing the RF / Strategy in the stress tool
>>>
>>
>> -schema option, pass a file if it is easier, with a custom keyspace,
>> defining NTS and a RF of 2 or 3. You don't want to use RF = 1 nor
>> SimpleStrategy in Multi DC under real conditions anyway, so your test will
>> be more relevant.
>> You can paste us the command + schema + output / error (if any)
>>
>>
>>> - Try with an other consistency level (not LOCAL_*, as mentioned here :
>>> http://stackoverflow.com/questions/32055251/not-enough-replica-available-for-query-at-consistency-local-one-1-required-but
>>> )
>>>
>>
>> Shouldn't affect you, yet, it is easy enough to test, so it might be
>> worth it. Maybe the first thing I would try, just out of curiosity.
>>
>> C*heers,
>>
>> -----------------
>> Alain
>>
>> The Last Pickle
>> http://www.thelastpickle.com
>>
>> 2016-01-23 11:13 GMT+01:00 Bhuvan Rawal <bh...@gmail.com>:
>>
>>> Alright, I uninstalled the DSE from all the nodes in the cluster and
>>> reinstalled them again from scratch. Ran nodetool status and output is U/N
>>> for all nodes again.
>>>
>>> Tried running cassandra-stress and it doesnt work.
>>>
>>> I can see in the system.peer table that token ranges has been
>>> distributed, further, other columns seem normal too.
>>> cqlsh> select peer, data_center, host_id,  rack, release_version,
>>> rpc_address, workload from system.peers;
>>>
>>> peer | data_center | host_id | rack | release_version | rpc_address |
>>> workload
>>> -------------+-------------+--------------------------------------+-------+-----------------+-------------+----------- 10.41.55.23
>>> | Cassandra | b48f7683-3f42-4c49-9ce8-694112a21d5d | rack1 | 2.1.12.1046 |
>>> 10.41.55.23 | Cassandra 10.41.55.19 | Analytics |
>>> f271657f-52e2-4f43-8bbe-03a979852679 | rack1 | 2.1.12.1046 | 10.41.55.19 |
>>> Analytics 10.41.55.20 | Analytics |
>>> e4f484c9-8e7f-48b7-bce8-2a91d70be4d1 | rack1 | 2.1.12.1046 | 10.41.55.20 |
>>> Analytics 10.41.55.18 | Analytics |
>>> 813c2610-0656-4714-9ec6-99d51c72da92 | rack1 | 2.1.12.1046 | 10.41.55.18 |
>>> Analytics 10.41.55.15 | Cassandra |
>>> 2b09a481-3e08-45e5-a2d0-4c3d5d611ef6 | rack1 | 2.1.12.1046 | 10.41.55.15 |
>>> Cassandra 10.41.55.22 | Cassandra |
>>> d16cc704-4453-4dc0-880e-4af6fbd0dc48 | rack1 | 2.1.12.1046 | 10.41.55.22 |
>>> Cassandra 10.41.55.17 | Analytics |
>>> dfbb4f0e-2ffa-4694-811c-1167662ad537 | rack1 | 2.1.12.1046 | 10.41.55.17 |
>>> Analytics
>>>
>>> This seems to be fine too:
>>> $ nodetool describecluster Cluster Information: Name: POC Cluster
>>> Snitch: org.apache.cassandra.locator.DynamicEndpointSnitch Partitioner:
>>> org.apache.cassandra.dht.Murmur3Partitioner Schema versions:
>>> 07d395d2-e5b8-3e67-addd-3f8d1e8d7803: [10.41.55.15, 10.41.55.17,
>>> 10.41.55.19, 10.41.55.18, 10.41.55.21, 10.41.55.20, 10.41.55.23,
>>> 10.41.55.22]
>>>
>>> I tried to telnet to a node of other cluster which is not a seed, here
>>> are the observations :
>>> Telnet to ports 9042, 7000,9160 seem to work fine
>>> telnet 10.41.55.18 9042 # connects
>>> telnet 10.41.55.18 7000 # connects
>>> telnet 10.41.55.18 9160 # connects
>>> But telnet to JMX port is not working
>>> telnet 10.41.55.18 7199 # doesnt connect
>>> Trying 10.41.55.18...
>>> telnet: connect to address 10.41.55.18: Connection refused
>>>
>>> Could this be the reason?
>>>
>>> On Sat, Jan 23, 2016 at 7:33 AM, Alain RODRIGUEZ <ar...@gmail.com>
>>> wrote:
>>>
>>>> Should I use nodetool repair utility
>>>>>
>>>>
>>>> That wouldn't help, this an anti-entropy mechanism (see
>>>> https://docs.datastax.com/en/cassandra/2.1/cassandra/tools/toolsRepair.html#toolsRepair__description_unique_27
>>>> ).
>>>>
>>>> It is something really important too often left aside.
>>>>
>>>> Yet, your issue here is not about consistency. The client can't find
>>>> any node in charge of the read / written tokens in the ring. This depends
>>>> on the topology, the replication factor and your network mainly. I think
>>>> there is something wrong in your setup. I would try this:
>>>>
>>>> - Make sure connection / port are ok
>>>> - Try increasing the RF / Strategy in the stress tool
>>>> - Try with an other consistency level (not LOCAL_*, as mentioned here :
>>>> http://stackoverflow.com/questions/32055251/not-enough-replica-available-for-query-at-consistency-local-one-1-required-but
>>>> )
>>>>
>>>> Good luck,
>>>>
>>>> -----------------
>>>> Alain
>>>>
>>>> The Last Pickle
>>>> http://www.thelastpickle.com
>>>>
>>>> 2016-01-22 23:02 GMT+01:00 Bhuvan Rawal <bh...@gmail.com>:
>>>>
>>>>> Getting same exception again. Should I use nodetool repair utility?
>>>>>
>>>>> On Sat, Jan 23, 2016 at 3:10 AM, Sebastian Estevez <
>>>>> sebastian.estevez@datastax.com> wrote:
>>>>>
>>>>>> https://github.com/brianmhess/cassandra-loader
>>>>>>
>>>>>> All the best,
>>>>>>
>>>>>>
>>>>>> [image: datastax_logo.png] <http://www.datastax.com/>
>>>>>>
>>>>>> Sebastián Estévez
>>>>>>
>>>>>> Solutions Architect | 954 905 8615 | sebastian.estevez@datastax.com
>>>>>>
>>>>>> [image: linkedin.png] <https://www.linkedin.com/company/datastax> [image:
>>>>>> facebook.png] <https://www.facebook.com/datastax> [image:
>>>>>> twitter.png] <https://twitter.com/datastax> [image: g+.png]
>>>>>> <https://plus.google.com/+Datastax/about>
>>>>>> <http://feeds.feedburner.com/datastax>
>>>>>> <http://goog_410786983>
>>>>>>
>>>>>>
>>>>>> <http://www.datastax.com/gartner-magic-quadrant-odbms>
>>>>>>
>>>>>> DataStax is the fastest, most scalable distributed database
>>>>>> technology, delivering Apache Cassandra to the world’s most innovative
>>>>>> enterprises. Datastax is built to be agile, always-on, and predictably
>>>>>> scalable to any size. With more than 500 customers in 45 countries, DataStax
>>>>>> is the database technology and transactional backbone of choice for the
>>>>>> worlds most innovative companies such as Netflix, Adobe, Intuit, and eBay.
>>>>>>
>>>>>> On Fri, Jan 22, 2016 at 4:37 PM, Bhuvan Rawal <bh...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Yes im specifying -node parameter to stress, otherwise it throws
>>>>>>> network connection failed.
>>>>>>>
>>>>>>> Can you point me to a sample java application to test pushing data
>>>>>>> from external server? Let's see if that works
>>>>>>>
>>>>>>> On Sat, Jan 23, 2016 at 2:55 AM, Sebastian Estevez <
>>>>>>> sebastian.estevez@datastax.com> wrote:
>>>>>>>
>>>>>>>> when i opened my cassandra-rackdc.properties i saw that DC names
>>>>>>>>> were DC1 & DC2, rack name was RAC1 . Please note that this is the default
>>>>>>>>> configuration, I have not modified any file.
>>>>>>>>
>>>>>>>>
>>>>>>>> cassandra-rackdc.properties is only respected based on your snitch
>>>>>>>> <https://docs.datastax.com/en/cassandra/2.1/cassandra/architecture/architectureSnitchesAbout_c.html>
>>>>>>>> .
>>>>>>>>
>>>>>>>> $ cqlsh
>>>>>>>>> Connection error: ('Unable to connect to any servers',
>>>>>>>>> {'127.0.0.1': error(111, "Tried connecting to [('127.0.0.1', 9042)]. Last
>>>>>>>>> error: Connection refused")})
>>>>>>>>> whereas
>>>>>>>>> $ cqlsh <ip>
>>>>>>>>> works fine
>>>>>>>>> is that the reason why the cassandra-stress is not able to
>>>>>>>>> communicate with other replicas?
>>>>>>>>
>>>>>>>>
>>>>>>>> Are you providing the -node parameter to stress
>>>>>>>> <http://docs.datastax.com/en/cassandra/2.1/cassandra/tools/toolsCStress_t.html>
>>>>>>>> ?
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> All the best,
>>>>>>>>
>>>>>>>>
>>>>>>>> [image: datastax_logo.png] <http://www.datastax.com/>
>>>>>>>>
>>>>>>>> Sebastián Estévez
>>>>>>>>
>>>>>>>> Solutions Architect | 954 905 8615 | sebastian.estevez@datastax.com
>>>>>>>>
>>>>>>>> [image: linkedin.png] <https://www.linkedin.com/company/datastax> [image:
>>>>>>>> facebook.png] <https://www.facebook.com/datastax> [image:
>>>>>>>> twitter.png] <https://twitter.com/datastax> [image: g+.png]
>>>>>>>> <https://plus.google.com/+Datastax/about>
>>>>>>>> <http://feeds.feedburner.com/datastax>
>>>>>>>> <http://goog_410786983>
>>>>>>>>
>>>>>>>>
>>>>>>>> <http://www.datastax.com/gartner-magic-quadrant-odbms>
>>>>>>>>
>>>>>>>> DataStax is the fastest, most scalable distributed database
>>>>>>>> technology, delivering Apache Cassandra to the world’s most innovative
>>>>>>>> enterprises. Datastax is built to be agile, always-on, and predictably
>>>>>>>> scalable to any size. With more than 500 customers in 45 countries, DataStax
>>>>>>>> is the database technology and transactional backbone of choice for the
>>>>>>>> worlds most innovative companies such as Netflix, Adobe, Intuit, and eBay.
>>>>>>>>
>>>>>>>> On Fri, Jan 22, 2016 at 4:07 PM, Bhuvan Rawal <bh...@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> I had a look at the jira below:
>>>>>>>>> https://issues.apache.org/jira/browse/CASSANDRA-7905
>>>>>>>>>
>>>>>>>>> when i opened my cassandra-rackdc.properties i saw that DC names
>>>>>>>>> were DC1 & DC2, rack name was RAC1 . Please note that this is the default
>>>>>>>>> configuration, I have not modified any file.
>>>>>>>>>
>>>>>>>>> There is another point of concern here which might be relevant to
>>>>>>>>> previous one as well, im not able to login to cqlsh directly, i.e. I have
>>>>>>>>> to specify ip as well even when im logged in to that machine.
>>>>>>>>>
>>>>>>>>> $ cqlsh
>>>>>>>>> Connection error: ('Unable to connect to any servers',
>>>>>>>>> {'127.0.0.1': error(111, "Tried connecting to [('127.0.0.1', 9042)]. Last
>>>>>>>>> error: Connection refused")})
>>>>>>>>>
>>>>>>>>> whereas
>>>>>>>>> $ cqlsh <ip>
>>>>>>>>> works fine
>>>>>>>>>
>>>>>>>>> is that the reason why the cassandra-stress is not able to
>>>>>>>>> communicate with other replicas?
>>>>>>>>>
>>>>>>>>> On Sat, Jan 23, 2016 at 1:37 AM, Sebastian Estevez <
>>>>>>>>> sebastian.estevez@datastax.com> wrote:
>>>>>>>>>
>>>>>>>>>> Sorry I missed that.
>>>>>>>>>>
>>>>>>>>>> Both your nodetool status and keyspace replication settings say
>>>>>>>>>> Cassandra and Analytics for the DC names. I'm not sure where you're seeing
>>>>>>>>>> DC1, DC2, etc. and why you suspect that is the problem.
>>>>>>>>>>
>>>>>>>>>> All the best,
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> [image: datastax_logo.png] <http://www.datastax.com/>
>>>>>>>>>>
>>>>>>>>>> Sebastián Estévez
>>>>>>>>>>
>>>>>>>>>> Solutions Architect | 954 905 8615 |
>>>>>>>>>> sebastian.estevez@datastax.com
>>>>>>>>>>
>>>>>>>>>> [image: linkedin.png] <https://www.linkedin.com/company/datastax> [image:
>>>>>>>>>> facebook.png] <https://www.facebook.com/datastax> [image:
>>>>>>>>>> twitter.png] <https://twitter.com/datastax> [image: g+.png]
>>>>>>>>>> <https://plus.google.com/+Datastax/about>
>>>>>>>>>> <http://feeds.feedburner.com/datastax>
>>>>>>>>>> <http://goog_410786983>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> <http://www.datastax.com/gartner-magic-quadrant-odbms>
>>>>>>>>>>
>>>>>>>>>> DataStax is the fastest, most scalable distributed database
>>>>>>>>>> technology, delivering Apache Cassandra to the world’s most innovative
>>>>>>>>>> enterprises. Datastax is built to be agile, always-on, and predictably
>>>>>>>>>> scalable to any size. With more than 500 customers in 45 countries, DataStax
>>>>>>>>>> is the database technology and transactional backbone of choice for the
>>>>>>>>>> worlds most innovative companies such as Netflix, Adobe, Intuit, and eBay.
>>>>>>>>>>
>>>>>>>>>> On Fri, Jan 22, 2016 at 1:45 PM, Bhuvan Rawal <
>>>>>>>>>> bhu1rawal@gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi Sebastian,
>>>>>>>>>>>
>>>>>>>>>>> I had attached nodetool status output in previous mail, pasting
>>>>>>>>>>> it again :
>>>>>>>>>>>
>>>>>>>>>>> $ nodetool status Datacenter: Analytics =====================
>>>>>>>>>>> Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address
>>>>>>>>>>> Load Tokens Owns Host ID Rack UN 10.41.55.17 428.5 KB 256 ?
>>>>>>>>>>> 39d6d585-e641-4046-9d0b-797356597b5e rack1 UN 10.41.55.19 404.44 KB 256 ?
>>>>>>>>>>> 69edf930-efd9-4d74-a798-f3d4ac02e516 rack1 UN 10.41.55.18 423.21 KB 256 ?
>>>>>>>>>>> b74bab13-09b2-4760-bce9-c8ef05e50f6d rack1 UN 10.41.55.20 683.23 KB 256 ?
>>>>>>>>>>> fb5c4fed-6e1e-4ea8-838d-358106906830 rack1 Datacenter:
>>>>>>>>>>> Cassandra ===================== Status=Up/Down |/
>>>>>>>>>>> State=Normal/Leaving/Joining/Moving -- Address Load Tokens Owns
>>>>>>>>>>> Host ID Rack UN 10.41.55.15 209.4 KB 256 ?
>>>>>>>>>>> ffc3b9a0-5d5c-4a3d-a99e-49d255731278 rack1 UN 10.41.55.21 227.44 KB 256 ?
>>>>>>>>>>> c68deba4-b9a2-43fc-bb13-6af74c88c210 rack1 UN 10.41.55.23 222.71 KB 256 ?
>>>>>>>>>>> 8229aa87-af00-48fa-ad6b-3066d3dc0e58 rack1 UN 10.41.55.22 218.72 KB 256 ?
>>>>>>>>>>> c7ba84fd-7992-41de-8c88-11574a72db99 rack1
>>>>>>>>>>>
>>>>>>>>>>> Regards,
>>>>>>>>>>> Bhuvan Rawal
>>>>>>>>>>>
>>>>>>>>>>> On Sat, Jan 23, 2016 at 12:11 AM, Sebastian Estevez <
>>>>>>>>>>> sebastian.estevez@datastax.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> The output of `nodetool status` would help us diagnose.
>>>>>>>>>>>>
>>>>>>>>>>>> All the best,
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> [image: datastax_logo.png] <http://www.datastax.com/>
>>>>>>>>>>>>
>>>>>>>>>>>> Sebastián Estévez
>>>>>>>>>>>>
>>>>>>>>>>>> Solutions Architect | 954 905 8615 |
>>>>>>>>>>>> sebastian.estevez@datastax.com
>>>>>>>>>>>>
>>>>>>>>>>>> [image: linkedin.png]
>>>>>>>>>>>> <https://www.linkedin.com/company/datastax> [image:
>>>>>>>>>>>> facebook.png] <https://www.facebook.com/datastax> [image:
>>>>>>>>>>>> twitter.png] <https://twitter.com/datastax> [image: g+.png]
>>>>>>>>>>>> <https://plus.google.com/+Datastax/about>
>>>>>>>>>>>> <http://feeds.feedburner.com/datastax>
>>>>>>>>>>>> <http://goog_410786983>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> <http://www.datastax.com/gartner-magic-quadrant-odbms>
>>>>>>>>>>>>
>>>>>>>>>>>> DataStax is the fastest, most scalable distributed database
>>>>>>>>>>>> technology, delivering Apache Cassandra to the world’s most innovative
>>>>>>>>>>>> enterprises. Datastax is built to be agile, always-on, and predictably
>>>>>>>>>>>> scalable to any size. With more than 500 customers in 45 countries, DataStax
>>>>>>>>>>>> is the database technology and transactional backbone of choice for the
>>>>>>>>>>>> worlds most innovative companies such as Netflix, Adobe, Intuit, and eBay.
>>>>>>>>>>>>
>>>>>>>>>>>> On Fri, Jan 22, 2016 at 1:39 PM, Bhuvan Rawal <
>>>>>>>>>>>> bhu1rawal@gmail.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks for the response Alain,
>>>>>>>>>>>>>
>>>>>>>>>>>>> cqlsh> create keyspace mykeyspace WITH replication =
>>>>>>>>>>>>> {'class':'NetworkTopologyStrategy', 'Analytics':2, 'Cassandra':3}
>>>>>>>>>>>>> cqlsh> use mykeyspace;
>>>>>>>>>>>>> cqlsh:mykeyspace>create table mytable (id int primary key,
>>>>>>>>>>>>> name text, address text, phone text);
>>>>>>>>>>>>> cqlsh:mykeyspace> insert into mytable (id, name, address,
>>>>>>>>>>>>> phone) values (1, 'Kiyu','Texas', '555-1212'); # and other similar statement
>>>>>>>>>>>>> I then issued the below command from every node and found
>>>>>>>>>>>>> consistent results.
>>>>>>>>>>>>> cqlsh:mykeyspace> select * from mytable;
>>>>>>>>>>>>>
>>>>>>>>>>>>> // Then i repeated the above steps for NetworkTopologyStrategy
>>>>>>>>>>>>> and found same results
>>>>>>>>>>>>>
>>>>>>>>>>>>> I ran basic cassandra stress
>>>>>>>>>>>>> seed1 - seed of datacenter 1
>>>>>>>>>>>>>  $ cassandra-stress write n=50000 -rate threads=4 -node
>>>>>>>>>>>>> any_random_ip
>>>>>>>>>>>>>  $ cassandra-stress write n=50000 -rate threads=4 -node seed1
>>>>>>>>>>>>>  $ cassandra-stress write n=50000 -rate threads=4 -node
>>>>>>>>>>>>> seed1,seed2
>>>>>>>>>>>>>  $ cassandra-stress write n=50000 -rate threads=4 -node
>>>>>>>>>>>>> all_8_ip_comma_seperated
>>>>>>>>>>>>>  $ cassandra-stress write n=1000000 cl=one -mode native cql3
>>>>>>>>>>>>> -schema keyspace="keyspace1" -pop seq=1..1000000 -node ip1,ip2,ip3,ip4
>>>>>>>>>>>>>
>>>>>>>>>>>>> All of them threw the exception
>>>>>>>>>>>>> *com.datastax.driver.core.exceptions.UnavailableException: Not
>>>>>>>>>>>>> enough replica available for query at consistency LOCAL_ONE (1 required but
>>>>>>>>>>>>> only 0 alive)*
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> I have a feeling that the issue is with datacenter name for
>>>>>>>>>>>>> some reason, because in some config files I found DC name to be like
>>>>>>>>>>>>> DC1/DC2/DC3 in some it is like Cassandra/Analytics (The ones I had
>>>>>>>>>>>>> specified while installation). Im unsure which yaml/property file to look
>>>>>>>>>>>>> for correct inconsistency.
>>>>>>>>>>>>>
>>>>>>>>>>>>> (C*heers :) - im so tempted to copy that)
>>>>>>>>>>>>>
>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>> Bhuvan
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Fri, Jan 22, 2016 at 8:47 PM, Alain RODRIGUEZ <
>>>>>>>>>>>>> arodrime@gmail.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> The the exact command you ran (stress-tool with options)
>>>>>>>>>>>>>> could be useful to help you on that.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> However, Im able to create keyspace, tables and insert data
>>>>>>>>>>>>>>> using cqlsh and it is replicating fine to all the nodes.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Having the schema might be useful too.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Did you ran the cqlsh and the stress-tool from the same
>>>>>>>>>>>>>> server ? If not, you might want to check the port you use (9042/9160/...)
>>>>>>>>>>>>>> are open.
>>>>>>>>>>>>>> Also, cqlsh uses local_one by default too. If both commands
>>>>>>>>>>>>>> were run against the same DC, from the same machine they should behave the
>>>>>>>>>>>>>> same way. Are they ?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> C*heers,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> -----------------
>>>>>>>>>>>>>> Alain
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> The Last Pickle
>>>>>>>>>>>>>> http://www.thelastpickle.com
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 2016-01-22 9:57 GMT+01:00 Bhuvan Rawal <bh...@gmail.com>:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> i have created a POC cluster with 2 DC , each having 4 nodes
>>>>>>>>>>>>>>> with DSE 4.8.1 installed.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On issuing cassandra stress im getting an error  and data is
>>>>>>>>>>>>>>> not being inserted:
>>>>>>>>>>>>>>> *com.datastax.driver.core.exceptions.UnavailableException:
>>>>>>>>>>>>>>> Not enough replica available for query at consistency LOCAL_ONE (1 required
>>>>>>>>>>>>>>> but only 0 alive)*
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> However, Im able to create keyspace, tables and insert data
>>>>>>>>>>>>>>> using cqlsh and it is replicating fine to all the nodes.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Details of the cluster can be found below (all the nodes
>>>>>>>>>>>>>>> seem to be alive and kicking):
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> $ nodetool status Datacenter: Analytics
>>>>>>>>>>>>>>> ===================== Status=Up/Down |/
>>>>>>>>>>>>>>> State=Normal/Leaving/Joining/Moving -- Address Load Tokens
>>>>>>>>>>>>>>> Owns Host ID Rack UN 10.41.55.17 428.5 KB 256 ?
>>>>>>>>>>>>>>> 39d6d585-e641-4046-9d0b-797356597b5e rack1 UN 10.41.55.19 404.44 KB 256 ?
>>>>>>>>>>>>>>> 69edf930-efd9-4d74-a798-f3d4ac02e516 rack1 UN 10.41.55.18 423.21 KB 256 ?
>>>>>>>>>>>>>>> b74bab13-09b2-4760-bce9-c8ef05e50f6d rack1 UN 10.41.55.20 683.23 KB 256 ?
>>>>>>>>>>>>>>> fb5c4fed-6e1e-4ea8-838d-358106906830 rack1 Datacenter:
>>>>>>>>>>>>>>> Cassandra ===================== Status=Up/Down |/
>>>>>>>>>>>>>>> State=Normal/Leaving/Joining/Moving -- Address Load Tokens
>>>>>>>>>>>>>>> Owns Host ID Rack UN 10.41.55.15 209.4 KB 256 ?
>>>>>>>>>>>>>>> ffc3b9a0-5d5c-4a3d-a99e-49d255731278 rack1 UN 10.41.55.21 227.44 KB 256 ?
>>>>>>>>>>>>>>> c68deba4-b9a2-43fc-bb13-6af74c88c210 rack1 UN 10.41.55.23 222.71 KB 256 ?
>>>>>>>>>>>>>>> 8229aa87-af00-48fa-ad6b-3066d3dc0e58 rack1 UN 10.41.55.22 218.72 KB 256 ?
>>>>>>>>>>>>>>> c7ba84fd-7992-41de-8c88-11574a72db99 rack1
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>> Bhuvan Rawal
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>

Re: Getting error while issuing Cassandra stress

Posted by Bhuvan Rawal <bh...@gmail.com>.

Hi Alan,

You have hit bulls eye! I used a custom yaml file for configuring keyspace,
used NTT and multiple replications for each cluster, defined table schema
and query and it seemed to work great!!

I have some apprehensions here though. The tests that I have done did not
yield results that I expected, Im not sure if I configured everything
correctly. Sharing results in a separate mail.

Thanks & Regards,
Bhuvan

On Sat, Jan 23, 2016 at 5:59 PM, Alain RODRIGUEZ <ar...@gmail.com> wrote:

> telnet 10.41.55.18 7199 # doesnt connect
>> Trying 10.41.55.18...
>> telnet: connect to address 10.41.55.18: Connection refused
>>
>> Could this be the reason?
>>
>
> I don't think so. But I have not a clue on what's going on...
>
> Did you tried this ?
>
> - Try increasing the RF / Strategy in the stress tool
>>
>
> -schema option, pass a file if it is easier, with a custom keyspace,
> defining NTS and a RF of 2 or 3. You don't want to use RF = 1 nor
> SimpleStrategy in Multi DC under real conditions anyway, so your test will
> be more relevant.
> You can paste us the command + schema + output / error (if any)
>
>
>> - Try with an other consistency level (not LOCAL_*, as mentioned here :
>> http://stackoverflow.com/questions/32055251/not-enough-replica-available-for-query-at-consistency-local-one-1-required-but
>> )
>>
>
> Shouldn't affect you, yet, it is easy enough to test, so it might be worth
> it. Maybe the first thing I would try, just out of curiosity.
>
> C*heers,
>
> -----------------
> Alain
>
> The Last Pickle
> http://www.thelastpickle.com
>
> 2016-01-23 11:13 GMT+01:00 Bhuvan Rawal <bh...@gmail.com>:
>
>> Alright, I uninstalled the DSE from all the nodes in the cluster and
>> reinstalled them again from scratch. Ran nodetool status and output is U/N
>> for all nodes again.
>>
>> Tried running cassandra-stress and it doesnt work.
>>
>> I can see in the system.peer table that token ranges has been
>> distributed, further, other columns seem normal too.
>> cqlsh> select peer, data_center, host_id,  rack, release_version,
>> rpc_address, workload from system.peers;
>>
>> peer | data_center | host_id | rack | release_version | rpc_address |
>> workload
>> -------------+-------------+--------------------------------------+-------+-----------------+-------------+----------- 10.41.55.23
>> | Cassandra | b48f7683-3f42-4c49-9ce8-694112a21d5d | rack1 | 2.1.12.1046 |
>> 10.41.55.23 | Cassandra 10.41.55.19 | Analytics |
>> f271657f-52e2-4f43-8bbe-03a979852679 | rack1 | 2.1.12.1046 | 10.41.55.19 |
>> Analytics 10.41.55.20 | Analytics | e4f484c9-8e7f-48b7-bce8-2a91d70be4d1
>> | rack1 | 2.1.12.1046 | 10.41.55.20 | Analytics 10.41.55.18 | Analytics
>> | 813c2610-0656-4714-9ec6-99d51c72da92 | rack1 | 2.1.12.1046 | 10.41.55.18
>> | Analytics 10.41.55.15 | Cassandra |
>> 2b09a481-3e08-45e5-a2d0-4c3d5d611ef6 | rack1 | 2.1.12.1046 | 10.41.55.15 |
>> Cassandra 10.41.55.22 | Cassandra | d16cc704-4453-4dc0-880e-4af6fbd0dc48
>> | rack1 | 2.1.12.1046 | 10.41.55.22 | Cassandra 10.41.55.17 | Analytics
>> | dfbb4f0e-2ffa-4694-811c-1167662ad537 | rack1 | 2.1.12.1046 | 10.41.55.17
>> | Analytics
>>
>> This seems to be fine too:
>> $ nodetool describecluster Cluster Information: Name: POC Cluster Snitch:
>> org.apache.cassandra.locator.DynamicEndpointSnitch Partitioner:
>> org.apache.cassandra.dht.Murmur3Partitioner Schema versions:
>> 07d395d2-e5b8-3e67-addd-3f8d1e8d7803: [10.41.55.15, 10.41.55.17,
>> 10.41.55.19, 10.41.55.18, 10.41.55.21, 10.41.55.20, 10.41.55.23,
>> 10.41.55.22]
>>
>> I tried to telnet to a node of other cluster which is not a seed, here
>> are the observations :
>> Telnet to ports 9042, 7000,9160 seem to work fine
>> telnet 10.41.55.18 9042 # connects
>> telnet 10.41.55.18 7000 # connects
>> telnet 10.41.55.18 9160 # connects
>> But telnet to JMX port is not working
>> telnet 10.41.55.18 7199 # doesnt connect
>> Trying 10.41.55.18...
>> telnet: connect to address 10.41.55.18: Connection refused
>>
>> Could this be the reason?
>>
>> On Sat, Jan 23, 2016 at 7:33 AM, Alain RODRIGUEZ <ar...@gmail.com>
>> wrote:
>>
>>> Should I use nodetool repair utility
>>>>
>>>
>>> That wouldn't help, this an anti-entropy mechanism (see
>>> https://docs.datastax.com/en/cassandra/2.1/cassandra/tools/toolsRepair.html#toolsRepair__description_unique_27
>>> ).
>>>
>>> It is something really important too often left aside.
>>>
>>> Yet, your issue here is not about consistency. The client can't find any
>>> node in charge of the read / written tokens in the ring. This depends on
>>> the topology, the replication factor and your network mainly. I think there
>>> is something wrong in your setup. I would try this:
>>>
>>> - Make sure connection / port are ok
>>> - Try increasing the RF / Strategy in the stress tool
>>> - Try with an other consistency level (not LOCAL_*, as mentioned here :
>>> http://stackoverflow.com/questions/32055251/not-enough-replica-available-for-query-at-consistency-local-one-1-required-but
>>> )
>>>
>>> Good luck,
>>>
>>> -----------------
>>> Alain
>>>
>>> The Last Pickle
>>> http://www.thelastpickle.com
>>>
>>> 2016-01-22 23:02 GMT+01:00 Bhuvan Rawal <bh...@gmail.com>:
>>>
>>>> Getting same exception again. Should I use nodetool repair utility?
>>>>
>>>> On Sat, Jan 23, 2016 at 3:10 AM, Sebastian Estevez <
>>>> sebastian.estevez@datastax.com> wrote:
>>>>
>>>>> https://github.com/brianmhess/cassandra-loader
>>>>>
>>>>> All the best,
>>>>>
>>>>>
>>>>> [image: datastax_logo.png] <http://www.datastax.com/>
>>>>>
>>>>> Sebastián Estévez
>>>>>
>>>>> Solutions Architect | 954 905 8615 | sebastian.estevez@datastax.com
>>>>>
>>>>> [image: linkedin.png] <https://www.linkedin.com/company/datastax> [image:
>>>>> facebook.png] <https://www.facebook.com/datastax> [image: twitter.png]
>>>>> <https://twitter.com/datastax> [image: g+.png]
>>>>> <https://plus.google.com/+Datastax/about>
>>>>> <http://feeds.feedburner.com/datastax>
>>>>> <http://goog_410786983>
>>>>>
>>>>>
>>>>> <http://www.datastax.com/gartner-magic-quadrant-odbms>
>>>>>
>>>>> DataStax is the fastest, most scalable distributed database
>>>>> technology, delivering Apache Cassandra to the world’s most innovative
>>>>> enterprises. Datastax is built to be agile, always-on, and predictably
>>>>> scalable to any size. With more than 500 customers in 45 countries, DataStax
>>>>> is the database technology and transactional backbone of choice for the
>>>>> worlds most innovative companies such as Netflix, Adobe, Intuit, and eBay.
>>>>>
>>>>> On Fri, Jan 22, 2016 at 4:37 PM, Bhuvan Rawal <bh...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Yes im specifying -node parameter to stress, otherwise it throws
>>>>>> network connection failed.
>>>>>>
>>>>>> Can you point me to a sample java application to test pushing data
>>>>>> from external server? Let's see if that works
>>>>>>
>>>>>> On Sat, Jan 23, 2016 at 2:55 AM, Sebastian Estevez <
>>>>>> sebastian.estevez@datastax.com> wrote:
>>>>>>
>>>>>>> when i opened my cassandra-rackdc.properties i saw that DC names
>>>>>>>> were DC1 & DC2, rack name was RAC1 . Please note that this is the default
>>>>>>>> configuration, I have not modified any file.
>>>>>>>
>>>>>>>
>>>>>>> cassandra-rackdc.properties is only respected based on your snitch
>>>>>>> <https://docs.datastax.com/en/cassandra/2.1/cassandra/architecture/architectureSnitchesAbout_c.html>
>>>>>>> .
>>>>>>>
>>>>>>> $ cqlsh
>>>>>>>> Connection error: ('Unable to connect to any servers',
>>>>>>>> {'127.0.0.1': error(111, "Tried connecting to [('127.0.0.1', 9042)]. Last
>>>>>>>> error: Connection refused")})
>>>>>>>> whereas
>>>>>>>> $ cqlsh <ip>
>>>>>>>> works fine
>>>>>>>> is that the reason why the cassandra-stress is not able to
>>>>>>>> communicate with other replicas?
>>>>>>>
>>>>>>>
>>>>>>> Are you providing the -node parameter to stress
>>>>>>> <http://docs.datastax.com/en/cassandra/2.1/cassandra/tools/toolsCStress_t.html>
>>>>>>> ?
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> All the best,
>>>>>>>
>>>>>>>
>>>>>>> [image: datastax_logo.png] <http://www.datastax.com/>
>>>>>>>
>>>>>>> Sebastián Estévez
>>>>>>>
>>>>>>> Solutions Architect | 954 905 8615 | sebastian.estevez@datastax.com
>>>>>>>
>>>>>>> [image: linkedin.png] <https://www.linkedin.com/company/datastax> [image:
>>>>>>> facebook.png] <https://www.facebook.com/datastax> [image:
>>>>>>> twitter.png] <https://twitter.com/datastax> [image: g+.png]
>>>>>>> <https://plus.google.com/+Datastax/about>
>>>>>>> <http://feeds.feedburner.com/datastax>
>>>>>>> <http://goog_410786983>
>>>>>>>
>>>>>>>
>>>>>>> <http://www.datastax.com/gartner-magic-quadrant-odbms>
>>>>>>>
>>>>>>> DataStax is the fastest, most scalable distributed database
>>>>>>> technology, delivering Apache Cassandra to the world’s most innovative
>>>>>>> enterprises. Datastax is built to be agile, always-on, and predictably
>>>>>>> scalable to any size. With more than 500 customers in 45 countries, DataStax
>>>>>>> is the database technology and transactional backbone of choice for the
>>>>>>> worlds most innovative companies such as Netflix, Adobe, Intuit, and eBay.
>>>>>>>
>>>>>>> On Fri, Jan 22, 2016 at 4:07 PM, Bhuvan Rawal <bh...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> I had a look at the jira below:
>>>>>>>> https://issues.apache.org/jira/browse/CASSANDRA-7905
>>>>>>>>
>>>>>>>> when i opened my cassandra-rackdc.properties i saw that DC names
>>>>>>>> were DC1 & DC2, rack name was RAC1 . Please note that this is the default
>>>>>>>> configuration, I have not modified any file.
>>>>>>>>
>>>>>>>> There is another point of concern here which might be relevant to
>>>>>>>> previous one as well, im not able to login to cqlsh directly, i.e. I have
>>>>>>>> to specify ip as well even when im logged in to that machine.
>>>>>>>>
>>>>>>>> $ cqlsh
>>>>>>>> Connection error: ('Unable to connect to any servers',
>>>>>>>> {'127.0.0.1': error(111, "Tried connecting to [('127.0.0.1', 9042)]. Last
>>>>>>>> error: Connection refused")})
>>>>>>>>
>>>>>>>> whereas
>>>>>>>> $ cqlsh <ip>
>>>>>>>> works fine
>>>>>>>>
>>>>>>>> is that the reason why the cassandra-stress is not able to
>>>>>>>> communicate with other replicas?
>>>>>>>>
>>>>>>>> On Sat, Jan 23, 2016 at 1:37 AM, Sebastian Estevez <
>>>>>>>> sebastian.estevez@datastax.com> wrote:
>>>>>>>>
>>>>>>>>> Sorry I missed that.
>>>>>>>>>
>>>>>>>>> Both your nodetool status and keyspace replication settings say
>>>>>>>>> Cassandra and Analytics for the DC names. I'm not sure where you're seeing
>>>>>>>>> DC1, DC2, etc. and why you suspect that is the problem.
>>>>>>>>>
>>>>>>>>> All the best,
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> [image: datastax_logo.png] <http://www.datastax.com/>
>>>>>>>>>
>>>>>>>>> Sebastián Estévez
>>>>>>>>>
>>>>>>>>> Solutions Architect | 954 905 8615 |
>>>>>>>>> sebastian.estevez@datastax.com
>>>>>>>>>
>>>>>>>>> [image: linkedin.png] <https://www.linkedin.com/company/datastax> [image:
>>>>>>>>> facebook.png] <https://www.facebook.com/datastax> [image:
>>>>>>>>> twitter.png] <https://twitter.com/datastax> [image: g+.png]
>>>>>>>>> <https://plus.google.com/+Datastax/about>
>>>>>>>>> <http://feeds.feedburner.com/datastax>
>>>>>>>>> <http://goog_410786983>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> <http://www.datastax.com/gartner-magic-quadrant-odbms>
>>>>>>>>>
>>>>>>>>> DataStax is the fastest, most scalable distributed database
>>>>>>>>> technology, delivering Apache Cassandra to the world’s most innovative
>>>>>>>>> enterprises. Datastax is built to be agile, always-on, and predictably
>>>>>>>>> scalable to any size. With more than 500 customers in 45 countries, DataStax
>>>>>>>>> is the database technology and transactional backbone of choice for the
>>>>>>>>> worlds most innovative companies such as Netflix, Adobe, Intuit, and eBay.
>>>>>>>>>
>>>>>>>>> On Fri, Jan 22, 2016 at 1:45 PM, Bhuvan Rawal <bhu1rawal@gmail.com
>>>>>>>>> > wrote:
>>>>>>>>>
>>>>>>>>>> Hi Sebastian,
>>>>>>>>>>
>>>>>>>>>> I had attached nodetool status output in previous mail, pasting
>>>>>>>>>> it again :
>>>>>>>>>>
>>>>>>>>>> $ nodetool status Datacenter: Analytics =====================
>>>>>>>>>> Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address
>>>>>>>>>> Load Tokens Owns Host ID Rack UN 10.41.55.17 428.5 KB 256 ?
>>>>>>>>>> 39d6d585-e641-4046-9d0b-797356597b5e rack1 UN 10.41.55.19 404.44 KB 256 ?
>>>>>>>>>> 69edf930-efd9-4d74-a798-f3d4ac02e516 rack1 UN 10.41.55.18 423.21 KB 256 ?
>>>>>>>>>> b74bab13-09b2-4760-bce9-c8ef05e50f6d rack1 UN 10.41.55.20 683.23 KB 256 ?
>>>>>>>>>> fb5c4fed-6e1e-4ea8-838d-358106906830 rack1 Datacenter: Cassandra
>>>>>>>>>> ===================== Status=Up/Down |/
>>>>>>>>>> State=Normal/Leaving/Joining/Moving -- Address Load Tokens Owns
>>>>>>>>>> Host ID Rack UN 10.41.55.15 209.4 KB 256 ?
>>>>>>>>>> ffc3b9a0-5d5c-4a3d-a99e-49d255731278 rack1 UN 10.41.55.21 227.44 KB 256 ?
>>>>>>>>>> c68deba4-b9a2-43fc-bb13-6af74c88c210 rack1 UN 10.41.55.23 222.71 KB 256 ?
>>>>>>>>>> 8229aa87-af00-48fa-ad6b-3066d3dc0e58 rack1 UN 10.41.55.22 218.72 KB 256 ?
>>>>>>>>>> c7ba84fd-7992-41de-8c88-11574a72db99 rack1
>>>>>>>>>>
>>>>>>>>>> Regards,
>>>>>>>>>> Bhuvan Rawal
>>>>>>>>>>
>>>>>>>>>> On Sat, Jan 23, 2016 at 12:11 AM, Sebastian Estevez <
>>>>>>>>>> sebastian.estevez@datastax.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> The output of `nodetool status` would help us diagnose.
>>>>>>>>>>>
>>>>>>>>>>> All the best,
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> [image: datastax_logo.png] <http://www.datastax.com/>
>>>>>>>>>>>
>>>>>>>>>>> Sebastián Estévez
>>>>>>>>>>>
>>>>>>>>>>> Solutions Architect | 954 905 8615 |
>>>>>>>>>>> sebastian.estevez@datastax.com
>>>>>>>>>>>
>>>>>>>>>>> [image: linkedin.png]
>>>>>>>>>>> <https://www.linkedin.com/company/datastax> [image:
>>>>>>>>>>> facebook.png] <https://www.facebook.com/datastax> [image:
>>>>>>>>>>> twitter.png] <https://twitter.com/datastax> [image: g+.png]
>>>>>>>>>>> <https://plus.google.com/+Datastax/about>
>>>>>>>>>>> <http://feeds.feedburner.com/datastax>
>>>>>>>>>>> <http://goog_410786983>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> <http://www.datastax.com/gartner-magic-quadrant-odbms>
>>>>>>>>>>>
>>>>>>>>>>> DataStax is the fastest, most scalable distributed database
>>>>>>>>>>> technology, delivering Apache Cassandra to the world’s most innovative
>>>>>>>>>>> enterprises. Datastax is built to be agile, always-on, and predictably
>>>>>>>>>>> scalable to any size. With more than 500 customers in 45 countries, DataStax
>>>>>>>>>>> is the database technology and transactional backbone of choice for the
>>>>>>>>>>> worlds most innovative companies such as Netflix, Adobe, Intuit, and eBay.
>>>>>>>>>>>
>>>>>>>>>>> On Fri, Jan 22, 2016 at 1:39 PM, Bhuvan Rawal <
>>>>>>>>>>> bhu1rawal@gmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Thanks for the response Alain,
>>>>>>>>>>>>
>>>>>>>>>>>> cqlsh> create keyspace mykeyspace WITH replication =
>>>>>>>>>>>> {'class':'NetworkTopologyStrategy', 'Analytics':2, 'Cassandra':3}
>>>>>>>>>>>> cqlsh> use mykeyspace;
>>>>>>>>>>>> cqlsh:mykeyspace>create table mytable (id int primary key, name
>>>>>>>>>>>> text, address text, phone text);
>>>>>>>>>>>> cqlsh:mykeyspace> insert into mytable (id, name, address,
>>>>>>>>>>>> phone) values (1, 'Kiyu','Texas', '555-1212'); # and other similar statement
>>>>>>>>>>>> I then issued the below command from every node and found
>>>>>>>>>>>> consistent results.
>>>>>>>>>>>> cqlsh:mykeyspace> select * from mytable;
>>>>>>>>>>>>
>>>>>>>>>>>> // Then i repeated the above steps for NetworkTopologyStrategy
>>>>>>>>>>>> and found same results
>>>>>>>>>>>>
>>>>>>>>>>>> I ran basic cassandra stress
>>>>>>>>>>>> seed1 - seed of datacenter 1
>>>>>>>>>>>>  $ cassandra-stress write n=50000 -rate threads=4 -node
>>>>>>>>>>>> any_random_ip
>>>>>>>>>>>>  $ cassandra-stress write n=50000 -rate threads=4 -node seed1
>>>>>>>>>>>>  $ cassandra-stress write n=50000 -rate threads=4 -node
>>>>>>>>>>>> seed1,seed2
>>>>>>>>>>>>  $ cassandra-stress write n=50000 -rate threads=4 -node
>>>>>>>>>>>> all_8_ip_comma_seperated
>>>>>>>>>>>>  $ cassandra-stress write n=1000000 cl=one -mode native cql3
>>>>>>>>>>>> -schema keyspace="keyspace1" -pop seq=1..1000000 -node ip1,ip2,ip3,ip4
>>>>>>>>>>>>
>>>>>>>>>>>> All of them threw the exception
>>>>>>>>>>>> *com.datastax.driver.core.exceptions.UnavailableException: Not
>>>>>>>>>>>> enough replica available for query at consistency LOCAL_ONE (1 required but
>>>>>>>>>>>> only 0 alive)*
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> I have a feeling that the issue is with datacenter name for
>>>>>>>>>>>> some reason, because in some config files I found DC name to be like
>>>>>>>>>>>> DC1/DC2/DC3 in some it is like Cassandra/Analytics (The ones I had
>>>>>>>>>>>> specified while installation). Im unsure which yaml/property file to look
>>>>>>>>>>>> for correct inconsistency.
>>>>>>>>>>>>
>>>>>>>>>>>> (C*heers :) - im so tempted to copy that)
>>>>>>>>>>>>
>>>>>>>>>>>> Regards,
>>>>>>>>>>>> Bhuvan
>>>>>>>>>>>>
>>>>>>>>>>>> On Fri, Jan 22, 2016 at 8:47 PM, Alain RODRIGUEZ <
>>>>>>>>>>>> arodrime@gmail.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>
>>>>>>>>>>>>> The the exact command you ran (stress-tool with options) could
>>>>>>>>>>>>> be useful to help you on that.
>>>>>>>>>>>>>
>>>>>>>>>>>>> However, Im able to create keyspace, tables and insert data
>>>>>>>>>>>>>> using cqlsh and it is replicating fine to all the nodes.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Having the schema might be useful too.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Did you ran the cqlsh and the stress-tool from the same
>>>>>>>>>>>>> server ? If not, you might want to check the port you use (9042/9160/...)
>>>>>>>>>>>>> are open.
>>>>>>>>>>>>> Also, cqlsh uses local_one by default too. If both commands
>>>>>>>>>>>>> were run against the same DC, from the same machine they should behave the
>>>>>>>>>>>>> same way. Are they ?
>>>>>>>>>>>>>
>>>>>>>>>>>>> C*heers,
>>>>>>>>>>>>>
>>>>>>>>>>>>> -----------------
>>>>>>>>>>>>> Alain
>>>>>>>>>>>>>
>>>>>>>>>>>>> The Last Pickle
>>>>>>>>>>>>> http://www.thelastpickle.com
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> 2016-01-22 9:57 GMT+01:00 Bhuvan Rawal <bh...@gmail.com>:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> i have created a POC cluster with 2 DC , each having 4 nodes
>>>>>>>>>>>>>> with DSE 4.8.1 installed.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On issuing cassandra stress im getting an error  and data is
>>>>>>>>>>>>>> not being inserted:
>>>>>>>>>>>>>> *com.datastax.driver.core.exceptions.UnavailableException:
>>>>>>>>>>>>>> Not enough replica available for query at consistency LOCAL_ONE (1 required
>>>>>>>>>>>>>> but only 0 alive)*
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> However, Im able to create keyspace, tables and insert data
>>>>>>>>>>>>>> using cqlsh and it is replicating fine to all the nodes.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Details of the cluster can be found below (all the nodes seem
>>>>>>>>>>>>>> to be alive and kicking):
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> $ nodetool status Datacenter: Analytics =====================
>>>>>>>>>>>>>> Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address
>>>>>>>>>>>>>> Load Tokens Owns Host ID Rack UN 10.41.55.17 428.5 KB 256 ?
>>>>>>>>>>>>>> 39d6d585-e641-4046-9d0b-797356597b5e rack1 UN 10.41.55.19 404.44 KB 256 ?
>>>>>>>>>>>>>> 69edf930-efd9-4d74-a798-f3d4ac02e516 rack1 UN 10.41.55.18 423.21 KB 256 ?
>>>>>>>>>>>>>> b74bab13-09b2-4760-bce9-c8ef05e50f6d rack1 UN 10.41.55.20 683.23 KB 256 ?
>>>>>>>>>>>>>> fb5c4fed-6e1e-4ea8-838d-358106906830 rack1 Datacenter:
>>>>>>>>>>>>>> Cassandra ===================== Status=Up/Down |/
>>>>>>>>>>>>>> State=Normal/Leaving/Joining/Moving -- Address Load Tokens
>>>>>>>>>>>>>> Owns Host ID Rack UN 10.41.55.15 209.4 KB 256 ?
>>>>>>>>>>>>>> ffc3b9a0-5d5c-4a3d-a99e-49d255731278 rack1 UN 10.41.55.21 227.44 KB 256 ?
>>>>>>>>>>>>>> c68deba4-b9a2-43fc-bb13-6af74c88c210 rack1 UN 10.41.55.23 222.71 KB 256 ?
>>>>>>>>>>>>>> 8229aa87-af00-48fa-ad6b-3066d3dc0e58 rack1 UN 10.41.55.22 218.72 KB 256 ?
>>>>>>>>>>>>>> c7ba84fd-7992-41de-8c88-11574a72db99 rack1
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>> Bhuvan Rawal
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>

Re: Getting error while issuing Cassandra stress

Posted by Alain RODRIGUEZ <ar...@gmail.com>.

>
> telnet 10.41.55.18 7199 # doesnt connect
> Trying 10.41.55.18...
> telnet: connect to address 10.41.55.18: Connection refused
>
> Could this be the reason?
>

I don't think so. But I have not a clue on what's going on...

Did you tried this ?

- Try increasing the RF / Strategy in the stress tool
>

-schema option, pass a file if it is easier, with a custom keyspace,
defining NTS and a RF of 2 or 3. You don't want to use RF = 1 nor
SimpleStrategy in Multi DC under real conditions anyway, so your test will
be more relevant.
You can paste us the command + schema + output / error (if any)


> - Try with an other consistency level (not LOCAL_*, as mentioned here :
> http://stackoverflow.com/questions/32055251/not-enough-replica-available-for-query-at-consistency-local-one-1-required-but
> )
>

Shouldn't affect you, yet, it is easy enough to test, so it might be worth
it. Maybe the first thing I would try, just out of curiosity.

C*heers,

-----------------
Alain

The Last Pickle
http://www.thelastpickle.com

2016-01-23 11:13 GMT+01:00 Bhuvan Rawal <bh...@gmail.com>:

> Alright, I uninstalled the DSE from all the nodes in the cluster and
> reinstalled them again from scratch. Ran nodetool status and output is U/N
> for all nodes again.
>
> Tried running cassandra-stress and it doesnt work.
>
> I can see in the system.peer table that token ranges has been distributed,
> further, other columns seem normal too.
> cqlsh> select peer, data_center, host_id,  rack, release_version,
> rpc_address, workload from system.peers;
>
> peer | data_center | host_id | rack | release_version | rpc_address |
> workload
> -------------+-------------+--------------------------------------+-------+-----------------+-------------+----------- 10.41.55.23
> | Cassandra | b48f7683-3f42-4c49-9ce8-694112a21d5d | rack1 | 2.1.12.1046 |
> 10.41.55.23 | Cassandra 10.41.55.19 | Analytics |
> f271657f-52e2-4f43-8bbe-03a979852679 | rack1 | 2.1.12.1046 | 10.41.55.19 |
> Analytics 10.41.55.20 | Analytics | e4f484c9-8e7f-48b7-bce8-2a91d70be4d1
> | rack1 | 2.1.12.1046 | 10.41.55.20 | Analytics 10.41.55.18 | Analytics |
> 813c2610-0656-4714-9ec6-99d51c72da92 | rack1 | 2.1.12.1046 | 10.41.55.18 |
> Analytics 10.41.55.15 | Cassandra | 2b09a481-3e08-45e5-a2d0-4c3d5d611ef6
> | rack1 | 2.1.12.1046 | 10.41.55.15 | Cassandra 10.41.55.22 | Cassandra |
> d16cc704-4453-4dc0-880e-4af6fbd0dc48 | rack1 | 2.1.12.1046 | 10.41.55.22 |
> Cassandra 10.41.55.17 | Analytics | dfbb4f0e-2ffa-4694-811c-1167662ad537
> | rack1 | 2.1.12.1046 | 10.41.55.17 | Analytics
>
> This seems to be fine too:
> $ nodetool describecluster Cluster Information: Name: POC Cluster Snitch:
> org.apache.cassandra.locator.DynamicEndpointSnitch Partitioner:
> org.apache.cassandra.dht.Murmur3Partitioner Schema versions:
> 07d395d2-e5b8-3e67-addd-3f8d1e8d7803: [10.41.55.15, 10.41.55.17,
> 10.41.55.19, 10.41.55.18, 10.41.55.21, 10.41.55.20, 10.41.55.23,
> 10.41.55.22]
>
> I tried to telnet to a node of other cluster which is not a seed, here are
> the observations :
> Telnet to ports 9042, 7000,9160 seem to work fine
> telnet 10.41.55.18 9042 # connects
> telnet 10.41.55.18 7000 # connects
> telnet 10.41.55.18 9160 # connects
> But telnet to JMX port is not working
> telnet 10.41.55.18 7199 # doesnt connect
> Trying 10.41.55.18...
> telnet: connect to address 10.41.55.18: Connection refused
>
> Could this be the reason?
>
> On Sat, Jan 23, 2016 at 7:33 AM, Alain RODRIGUEZ <ar...@gmail.com>
> wrote:
>
>> Should I use nodetool repair utility
>>>
>>
>> That wouldn't help, this an anti-entropy mechanism (see
>> https://docs.datastax.com/en/cassandra/2.1/cassandra/tools/toolsRepair.html#toolsRepair__description_unique_27
>> ).
>>
>> It is something really important too often left aside.
>>
>> Yet, your issue here is not about consistency. The client can't find any
>> node in charge of the read / written tokens in the ring. This depends on
>> the topology, the replication factor and your network mainly. I think there
>> is something wrong in your setup. I would try this:
>>
>> - Make sure connection / port are ok
>> - Try increasing the RF / Strategy in the stress tool
>> - Try with an other consistency level (not LOCAL_*, as mentioned here :
>> http://stackoverflow.com/questions/32055251/not-enough-replica-available-for-query-at-consistency-local-one-1-required-but
>> )
>>
>> Good luck,
>>
>> -----------------
>> Alain
>>
>> The Last Pickle
>> http://www.thelastpickle.com
>>
>> 2016-01-22 23:02 GMT+01:00 Bhuvan Rawal <bh...@gmail.com>:
>>
>>> Getting same exception again. Should I use nodetool repair utility?
>>>
>>> On Sat, Jan 23, 2016 at 3:10 AM, Sebastian Estevez <
>>> sebastian.estevez@datastax.com> wrote:
>>>
>>>> https://github.com/brianmhess/cassandra-loader
>>>>
>>>> All the best,
>>>>
>>>>
>>>> [image: datastax_logo.png] <http://www.datastax.com/>
>>>>
>>>> Sebastián Estévez
>>>>
>>>> Solutions Architect | 954 905 8615 | sebastian.estevez@datastax.com
>>>>
>>>> [image: linkedin.png] <https://www.linkedin.com/company/datastax> [image:
>>>> facebook.png] <https://www.facebook.com/datastax> [image: twitter.png]
>>>> <https://twitter.com/datastax> [image: g+.png]
>>>> <https://plus.google.com/+Datastax/about>
>>>> <http://feeds.feedburner.com/datastax>
>>>> <http://goog_410786983>
>>>>
>>>>
>>>> <http://www.datastax.com/gartner-magic-quadrant-odbms>
>>>>
>>>> DataStax is the fastest, most scalable distributed database
>>>> technology, delivering Apache Cassandra to the world’s most innovative
>>>> enterprises. Datastax is built to be agile, always-on, and predictably
>>>> scalable to any size. With more than 500 customers in 45 countries, DataStax
>>>> is the database technology and transactional backbone of choice for the
>>>> worlds most innovative companies such as Netflix, Adobe, Intuit, and eBay.
>>>>
>>>> On Fri, Jan 22, 2016 at 4:37 PM, Bhuvan Rawal <bh...@gmail.com>
>>>> wrote:
>>>>
>>>>> Yes im specifying -node parameter to stress, otherwise it throws
>>>>> network connection failed.
>>>>>
>>>>> Can you point me to a sample java application to test pushing data
>>>>> from external server? Let's see if that works
>>>>>
>>>>> On Sat, Jan 23, 2016 at 2:55 AM, Sebastian Estevez <
>>>>> sebastian.estevez@datastax.com> wrote:
>>>>>
>>>>>> when i opened my cassandra-rackdc.properties i saw that DC names were
>>>>>>> DC1 & DC2, rack name was RAC1 . Please note that this is the default
>>>>>>> configuration, I have not modified any file.
>>>>>>
>>>>>>
>>>>>> cassandra-rackdc.properties is only respected based on your snitch
>>>>>> <https://docs.datastax.com/en/cassandra/2.1/cassandra/architecture/architectureSnitchesAbout_c.html>
>>>>>> .
>>>>>>
>>>>>> $ cqlsh
>>>>>>> Connection error: ('Unable to connect to any servers', {'127.0.0.1':
>>>>>>> error(111, "Tried connecting to [('127.0.0.1', 9042)]. Last error:
>>>>>>> Connection refused")})
>>>>>>> whereas
>>>>>>> $ cqlsh <ip>
>>>>>>> works fine
>>>>>>> is that the reason why the cassandra-stress is not able to
>>>>>>> communicate with other replicas?
>>>>>>
>>>>>>
>>>>>> Are you providing the -node parameter to stress
>>>>>> <http://docs.datastax.com/en/cassandra/2.1/cassandra/tools/toolsCStress_t.html>
>>>>>> ?
>>>>>>
>>>>>>
>>>>>>
>>>>>> All the best,
>>>>>>
>>>>>>
>>>>>> [image: datastax_logo.png] <http://www.datastax.com/>
>>>>>>
>>>>>> Sebastián Estévez
>>>>>>
>>>>>> Solutions Architect | 954 905 8615 | sebastian.estevez@datastax.com
>>>>>>
>>>>>> [image: linkedin.png] <https://www.linkedin.com/company/datastax> [image:
>>>>>> facebook.png] <https://www.facebook.com/datastax> [image:
>>>>>> twitter.png] <https://twitter.com/datastax> [image: g+.png]
>>>>>> <https://plus.google.com/+Datastax/about>
>>>>>> <http://feeds.feedburner.com/datastax>
>>>>>> <http://goog_410786983>
>>>>>>
>>>>>>
>>>>>> <http://www.datastax.com/gartner-magic-quadrant-odbms>
>>>>>>
>>>>>> DataStax is the fastest, most scalable distributed database
>>>>>> technology, delivering Apache Cassandra to the world’s most innovative
>>>>>> enterprises. Datastax is built to be agile, always-on, and predictably
>>>>>> scalable to any size. With more than 500 customers in 45 countries, DataStax
>>>>>> is the database technology and transactional backbone of choice for the
>>>>>> worlds most innovative companies such as Netflix, Adobe, Intuit, and eBay.
>>>>>>
>>>>>> On Fri, Jan 22, 2016 at 4:07 PM, Bhuvan Rawal <bh...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> I had a look at the jira below:
>>>>>>> https://issues.apache.org/jira/browse/CASSANDRA-7905
>>>>>>>
>>>>>>> when i opened my cassandra-rackdc.properties i saw that DC names
>>>>>>> were DC1 & DC2, rack name was RAC1 . Please note that this is the default
>>>>>>> configuration, I have not modified any file.
>>>>>>>
>>>>>>> There is another point of concern here which might be relevant to
>>>>>>> previous one as well, im not able to login to cqlsh directly, i.e. I have
>>>>>>> to specify ip as well even when im logged in to that machine.
>>>>>>>
>>>>>>> $ cqlsh
>>>>>>> Connection error: ('Unable to connect to any servers', {'127.0.0.1':
>>>>>>> error(111, "Tried connecting to [('127.0.0.1', 9042)]. Last error:
>>>>>>> Connection refused")})
>>>>>>>
>>>>>>> whereas
>>>>>>> $ cqlsh <ip>
>>>>>>> works fine
>>>>>>>
>>>>>>> is that the reason why the cassandra-stress is not able to
>>>>>>> communicate with other replicas?
>>>>>>>
>>>>>>> On Sat, Jan 23, 2016 at 1:37 AM, Sebastian Estevez <
>>>>>>> sebastian.estevez@datastax.com> wrote:
>>>>>>>
>>>>>>>> Sorry I missed that.
>>>>>>>>
>>>>>>>> Both your nodetool status and keyspace replication settings say
>>>>>>>> Cassandra and Analytics for the DC names. I'm not sure where you're seeing
>>>>>>>> DC1, DC2, etc. and why you suspect that is the problem.
>>>>>>>>
>>>>>>>> All the best,
>>>>>>>>
>>>>>>>>
>>>>>>>> [image: datastax_logo.png] <http://www.datastax.com/>
>>>>>>>>
>>>>>>>> Sebastián Estévez
>>>>>>>>
>>>>>>>> Solutions Architect | 954 905 8615 | sebastian.estevez@datastax.com
>>>>>>>>
>>>>>>>> [image: linkedin.png] <https://www.linkedin.com/company/datastax> [image:
>>>>>>>> facebook.png] <https://www.facebook.com/datastax> [image:
>>>>>>>> twitter.png] <https://twitter.com/datastax> [image: g+.png]
>>>>>>>> <https://plus.google.com/+Datastax/about>
>>>>>>>> <http://feeds.feedburner.com/datastax>
>>>>>>>> <http://goog_410786983>
>>>>>>>>
>>>>>>>>
>>>>>>>> <http://www.datastax.com/gartner-magic-quadrant-odbms>
>>>>>>>>
>>>>>>>> DataStax is the fastest, most scalable distributed database
>>>>>>>> technology, delivering Apache Cassandra to the world’s most innovative
>>>>>>>> enterprises. Datastax is built to be agile, always-on, and predictably
>>>>>>>> scalable to any size. With more than 500 customers in 45 countries, DataStax
>>>>>>>> is the database technology and transactional backbone of choice for the
>>>>>>>> worlds most innovative companies such as Netflix, Adobe, Intuit, and eBay.
>>>>>>>>
>>>>>>>> On Fri, Jan 22, 2016 at 1:45 PM, Bhuvan Rawal <bh...@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Hi Sebastian,
>>>>>>>>>
>>>>>>>>> I had attached nodetool status output in previous mail, pasting it
>>>>>>>>> again :
>>>>>>>>>
>>>>>>>>> $ nodetool status Datacenter: Analytics =====================
>>>>>>>>> Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address
>>>>>>>>> Load Tokens Owns Host ID Rack UN 10.41.55.17 428.5 KB 256 ?
>>>>>>>>> 39d6d585-e641-4046-9d0b-797356597b5e rack1 UN 10.41.55.19 404.44 KB 256 ?
>>>>>>>>> 69edf930-efd9-4d74-a798-f3d4ac02e516 rack1 UN 10.41.55.18 423.21 KB 256 ?
>>>>>>>>> b74bab13-09b2-4760-bce9-c8ef05e50f6d rack1 UN 10.41.55.20 683.23 KB 256 ?
>>>>>>>>> fb5c4fed-6e1e-4ea8-838d-358106906830 rack1 Datacenter: Cassandra
>>>>>>>>> ===================== Status=Up/Down |/
>>>>>>>>> State=Normal/Leaving/Joining/Moving -- Address Load Tokens Owns
>>>>>>>>> Host ID Rack UN 10.41.55.15 209.4 KB 256 ?
>>>>>>>>> ffc3b9a0-5d5c-4a3d-a99e-49d255731278 rack1 UN 10.41.55.21 227.44 KB 256 ?
>>>>>>>>> c68deba4-b9a2-43fc-bb13-6af74c88c210 rack1 UN 10.41.55.23 222.71 KB 256 ?
>>>>>>>>> 8229aa87-af00-48fa-ad6b-3066d3dc0e58 rack1 UN 10.41.55.22 218.72 KB 256 ?
>>>>>>>>> c7ba84fd-7992-41de-8c88-11574a72db99 rack1
>>>>>>>>>
>>>>>>>>> Regards,
>>>>>>>>> Bhuvan Rawal
>>>>>>>>>
>>>>>>>>> On Sat, Jan 23, 2016 at 12:11 AM, Sebastian Estevez <
>>>>>>>>> sebastian.estevez@datastax.com> wrote:
>>>>>>>>>
>>>>>>>>>> The output of `nodetool status` would help us diagnose.
>>>>>>>>>>
>>>>>>>>>> All the best,
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> [image: datastax_logo.png] <http://www.datastax.com/>
>>>>>>>>>>
>>>>>>>>>> Sebastián Estévez
>>>>>>>>>>
>>>>>>>>>> Solutions Architect | 954 905 8615 |
>>>>>>>>>> sebastian.estevez@datastax.com
>>>>>>>>>>
>>>>>>>>>> [image: linkedin.png] <https://www.linkedin.com/company/datastax> [image:
>>>>>>>>>> facebook.png] <https://www.facebook.com/datastax> [image:
>>>>>>>>>> twitter.png] <https://twitter.com/datastax> [image: g+.png]
>>>>>>>>>> <https://plus.google.com/+Datastax/about>
>>>>>>>>>> <http://feeds.feedburner.com/datastax>
>>>>>>>>>> <http://goog_410786983>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> <http://www.datastax.com/gartner-magic-quadrant-odbms>
>>>>>>>>>>
>>>>>>>>>> DataStax is the fastest, most scalable distributed database
>>>>>>>>>> technology, delivering Apache Cassandra to the world’s most innovative
>>>>>>>>>> enterprises. Datastax is built to be agile, always-on, and predictably
>>>>>>>>>> scalable to any size. With more than 500 customers in 45 countries, DataStax
>>>>>>>>>> is the database technology and transactional backbone of choice for the
>>>>>>>>>> worlds most innovative companies such as Netflix, Adobe, Intuit, and eBay.
>>>>>>>>>>
>>>>>>>>>> On Fri, Jan 22, 2016 at 1:39 PM, Bhuvan Rawal <
>>>>>>>>>> bhu1rawal@gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Thanks for the response Alain,
>>>>>>>>>>>
>>>>>>>>>>> cqlsh> create keyspace mykeyspace WITH replication =
>>>>>>>>>>> {'class':'NetworkTopologyStrategy', 'Analytics':2, 'Cassandra':3}
>>>>>>>>>>> cqlsh> use mykeyspace;
>>>>>>>>>>> cqlsh:mykeyspace>create table mytable (id int primary key, name
>>>>>>>>>>> text, address text, phone text);
>>>>>>>>>>> cqlsh:mykeyspace> insert into mytable (id, name, address, phone)
>>>>>>>>>>> values (1, 'Kiyu','Texas', '555-1212'); # and other similar statement
>>>>>>>>>>> I then issued the below command from every node and found
>>>>>>>>>>> consistent results.
>>>>>>>>>>> cqlsh:mykeyspace> select * from mytable;
>>>>>>>>>>>
>>>>>>>>>>> // Then i repeated the above steps for NetworkTopologyStrategy
>>>>>>>>>>> and found same results
>>>>>>>>>>>
>>>>>>>>>>> I ran basic cassandra stress
>>>>>>>>>>> seed1 - seed of datacenter 1
>>>>>>>>>>>  $ cassandra-stress write n=50000 -rate threads=4 -node
>>>>>>>>>>> any_random_ip
>>>>>>>>>>>  $ cassandra-stress write n=50000 -rate threads=4 -node seed1
>>>>>>>>>>>  $ cassandra-stress write n=50000 -rate threads=4 -node
>>>>>>>>>>> seed1,seed2
>>>>>>>>>>>  $ cassandra-stress write n=50000 -rate threads=4 -node
>>>>>>>>>>> all_8_ip_comma_seperated
>>>>>>>>>>>  $ cassandra-stress write n=1000000 cl=one -mode native cql3
>>>>>>>>>>> -schema keyspace="keyspace1" -pop seq=1..1000000 -node ip1,ip2,ip3,ip4
>>>>>>>>>>>
>>>>>>>>>>> All of them threw the exception
>>>>>>>>>>> *com.datastax.driver.core.exceptions.UnavailableException: Not
>>>>>>>>>>> enough replica available for query at consistency LOCAL_ONE (1 required but
>>>>>>>>>>> only 0 alive)*
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> I have a feeling that the issue is with datacenter name for some
>>>>>>>>>>> reason, because in some config files I found DC name to be like DC1/DC2/DC3
>>>>>>>>>>> in some it is like Cassandra/Analytics (The ones I had specified while
>>>>>>>>>>> installation). Im unsure which yaml/property file to look for correct
>>>>>>>>>>> inconsistency.
>>>>>>>>>>>
>>>>>>>>>>> (C*heers :) - im so tempted to copy that)
>>>>>>>>>>>
>>>>>>>>>>> Regards,
>>>>>>>>>>> Bhuvan
>>>>>>>>>>>
>>>>>>>>>>> On Fri, Jan 22, 2016 at 8:47 PM, Alain RODRIGUEZ <
>>>>>>>>>>> arodrime@gmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi,
>>>>>>>>>>>>
>>>>>>>>>>>> The the exact command you ran (stress-tool with options) could
>>>>>>>>>>>> be useful to help you on that.
>>>>>>>>>>>>
>>>>>>>>>>>> However, Im able to create keyspace, tables and insert data
>>>>>>>>>>>>> using cqlsh and it is replicating fine to all the nodes.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Having the schema might be useful too.
>>>>>>>>>>>>
>>>>>>>>>>>> Did you ran the cqlsh and the stress-tool from the same server
>>>>>>>>>>>> ? If not, you might want to check the port you use (9042/9160/...) are open.
>>>>>>>>>>>> Also, cqlsh uses local_one by default too. If both commands
>>>>>>>>>>>> were run against the same DC, from the same machine they should behave the
>>>>>>>>>>>> same way. Are they ?
>>>>>>>>>>>>
>>>>>>>>>>>> C*heers,
>>>>>>>>>>>>
>>>>>>>>>>>> -----------------
>>>>>>>>>>>> Alain
>>>>>>>>>>>>
>>>>>>>>>>>> The Last Pickle
>>>>>>>>>>>> http://www.thelastpickle.com
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> 2016-01-22 9:57 GMT+01:00 Bhuvan Rawal <bh...@gmail.com>:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>
>>>>>>>>>>>>> i have created a POC cluster with 2 DC , each having 4 nodes
>>>>>>>>>>>>> with DSE 4.8.1 installed.
>>>>>>>>>>>>>
>>>>>>>>>>>>> On issuing cassandra stress im getting an error  and data is
>>>>>>>>>>>>> not being inserted:
>>>>>>>>>>>>> *com.datastax.driver.core.exceptions.UnavailableException: Not
>>>>>>>>>>>>> enough replica available for query at consistency LOCAL_ONE (1 required but
>>>>>>>>>>>>> only 0 alive)*
>>>>>>>>>>>>>
>>>>>>>>>>>>> However, Im able to create keyspace, tables and insert data
>>>>>>>>>>>>> using cqlsh and it is replicating fine to all the nodes.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Details of the cluster can be found below (all the nodes seem
>>>>>>>>>>>>> to be alive and kicking):
>>>>>>>>>>>>>
>>>>>>>>>>>>> $ nodetool status Datacenter: Analytics =====================
>>>>>>>>>>>>> Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address
>>>>>>>>>>>>> Load Tokens Owns Host ID Rack UN 10.41.55.17 428.5 KB 256 ?
>>>>>>>>>>>>> 39d6d585-e641-4046-9d0b-797356597b5e rack1 UN 10.41.55.19 404.44 KB 256 ?
>>>>>>>>>>>>> 69edf930-efd9-4d74-a798-f3d4ac02e516 rack1 UN 10.41.55.18 423.21 KB 256 ?
>>>>>>>>>>>>> b74bab13-09b2-4760-bce9-c8ef05e50f6d rack1 UN 10.41.55.20 683.23 KB 256 ?
>>>>>>>>>>>>> fb5c4fed-6e1e-4ea8-838d-358106906830 rack1 Datacenter:
>>>>>>>>>>>>> Cassandra ===================== Status=Up/Down |/
>>>>>>>>>>>>> State=Normal/Leaving/Joining/Moving -- Address Load Tokens
>>>>>>>>>>>>> Owns Host ID Rack UN 10.41.55.15 209.4 KB 256 ?
>>>>>>>>>>>>> ffc3b9a0-5d5c-4a3d-a99e-49d255731278 rack1 UN 10.41.55.21 227.44 KB 256 ?
>>>>>>>>>>>>> c68deba4-b9a2-43fc-bb13-6af74c88c210 rack1 UN 10.41.55.23 222.71 KB 256 ?
>>>>>>>>>>>>> 8229aa87-af00-48fa-ad6b-3066d3dc0e58 rack1 UN 10.41.55.22 218.72 KB 256 ?
>>>>>>>>>>>>> c7ba84fd-7992-41de-8c88-11574a72db99 rack1
>>>>>>>>>>>>>
>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>> Bhuvan Rawal
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>

Re: Getting error while issuing Cassandra stress

Posted by Bhuvan Rawal <bh...@gmail.com>.

Alright, I uninstalled the DSE from all the nodes in the cluster and
reinstalled them again from scratch. Ran nodetool status and output is U/N
for all nodes again.

Tried running cassandra-stress and it doesnt work.

I can see in the system.peer table that token ranges has been distributed,
further, other columns seem normal too.
cqlsh> select peer, data_center, host_id,  rack, release_version,
rpc_address, workload from system.peers;

peer | data_center | host_id | rack | release_version | rpc_address |
workload
-------------+-------------+--------------------------------------+-------+-----------------+-------------+-----------
10.41.55.23
| Cassandra | b48f7683-3f42-4c49-9ce8-694112a21d5d | rack1 | 2.1.12.1046 |
10.41.55.23 | Cassandra 10.41.55.19 | Analytics |
f271657f-52e2-4f43-8bbe-03a979852679 | rack1 | 2.1.12.1046 | 10.41.55.19 |
Analytics 10.41.55.20 | Analytics | e4f484c9-8e7f-48b7-bce8-2a91d70be4d1 |
rack1 | 2.1.12.1046 | 10.41.55.20 | Analytics 10.41.55.18 | Analytics |
813c2610-0656-4714-9ec6-99d51c72da92 | rack1 | 2.1.12.1046 | 10.41.55.18 |
Analytics 10.41.55.15 | Cassandra | 2b09a481-3e08-45e5-a2d0-4c3d5d611ef6 |
rack1 | 2.1.12.1046 | 10.41.55.15 | Cassandra 10.41.55.22 | Cassandra |
d16cc704-4453-4dc0-880e-4af6fbd0dc48 | rack1 | 2.1.12.1046 | 10.41.55.22 |
Cassandra 10.41.55.17 | Analytics | dfbb4f0e-2ffa-4694-811c-1167662ad537 |
rack1 | 2.1.12.1046 | 10.41.55.17 | Analytics

This seems to be fine too:
$ nodetool describecluster Cluster Information: Name: POC Cluster Snitch:
org.apache.cassandra.locator.DynamicEndpointSnitch Partitioner:
org.apache.cassandra.dht.Murmur3Partitioner Schema versions:
07d395d2-e5b8-3e67-addd-3f8d1e8d7803: [10.41.55.15, 10.41.55.17,
10.41.55.19, 10.41.55.18, 10.41.55.21, 10.41.55.20, 10.41.55.23,
10.41.55.22]

I tried to telnet to a node of other cluster which is not a seed, here are
the observations :
Telnet to ports 9042, 7000,9160 seem to work fine
telnet 10.41.55.18 9042 # connects
telnet 10.41.55.18 7000 # connects
telnet 10.41.55.18 9160 # connects
But telnet to JMX port is not working
telnet 10.41.55.18 7199 # doesnt connect
Trying 10.41.55.18...
telnet: connect to address 10.41.55.18: Connection refused

Could this be the reason?

On Sat, Jan 23, 2016 at 7:33 AM, Alain RODRIGUEZ <ar...@gmail.com> wrote:

> Should I use nodetool repair utility
>>
>
> That wouldn't help, this an anti-entropy mechanism (see
> https://docs.datastax.com/en/cassandra/2.1/cassandra/tools/toolsRepair.html#toolsRepair__description_unique_27
> ).
>
> It is something really important too often left aside.
>
> Yet, your issue here is not about consistency. The client can't find any
> node in charge of the read / written tokens in the ring. This depends on
> the topology, the replication factor and your network mainly. I think there
> is something wrong in your setup. I would try this:
>
> - Make sure connection / port are ok
> - Try increasing the RF / Strategy in the stress tool
> - Try with an other consistency level (not LOCAL_*, as mentioned here :
> http://stackoverflow.com/questions/32055251/not-enough-replica-available-for-query-at-consistency-local-one-1-required-but
> )
>
> Good luck,
>
> -----------------
> Alain
>
> The Last Pickle
> http://www.thelastpickle.com
>
> 2016-01-22 23:02 GMT+01:00 Bhuvan Rawal <bh...@gmail.com>:
>
>> Getting same exception again. Should I use nodetool repair utility?
>>
>> On Sat, Jan 23, 2016 at 3:10 AM, Sebastian Estevez <
>> sebastian.estevez@datastax.com> wrote:
>>
>>> https://github.com/brianmhess/cassandra-loader
>>>
>>> All the best,
>>>
>>>
>>> [image: datastax_logo.png] <http://www.datastax.com/>
>>>
>>> Sebastián Estévez
>>>
>>> Solutions Architect | 954 905 8615 | sebastian.estevez@datastax.com
>>>
>>> [image: linkedin.png] <https://www.linkedin.com/company/datastax> [image:
>>> facebook.png] <https://www.facebook.com/datastax> [image: twitter.png]
>>> <https://twitter.com/datastax> [image: g+.png]
>>> <https://plus.google.com/+Datastax/about>
>>> <http://feeds.feedburner.com/datastax>
>>> <http://goog_410786983>
>>>
>>>
>>> <http://www.datastax.com/gartner-magic-quadrant-odbms>
>>>
>>> DataStax is the fastest, most scalable distributed database technology,
>>> delivering Apache Cassandra to the world’s most innovative enterprises.
>>> Datastax is built to be agile, always-on, and predictably scalable to any
>>> size. With more than 500 customers in 45 countries, DataStax is the
>>> database technology and transactional backbone of choice for the worlds
>>> most innovative companies such as Netflix, Adobe, Intuit, and eBay.
>>>
>>> On Fri, Jan 22, 2016 at 4:37 PM, Bhuvan Rawal <bh...@gmail.com>
>>> wrote:
>>>
>>>> Yes im specifying -node parameter to stress, otherwise it throws
>>>> network connection failed.
>>>>
>>>> Can you point me to a sample java application to test pushing data from
>>>> external server? Let's see if that works
>>>>
>>>> On Sat, Jan 23, 2016 at 2:55 AM, Sebastian Estevez <
>>>> sebastian.estevez@datastax.com> wrote:
>>>>
>>>>> when i opened my cassandra-rackdc.properties i saw that DC names were
>>>>>> DC1 & DC2, rack name was RAC1 . Please note that this is the default
>>>>>> configuration, I have not modified any file.
>>>>>
>>>>>
>>>>> cassandra-rackdc.properties is only respected based on your snitch
>>>>> <https://docs.datastax.com/en/cassandra/2.1/cassandra/architecture/architectureSnitchesAbout_c.html>
>>>>> .
>>>>>
>>>>> $ cqlsh
>>>>>> Connection error: ('Unable to connect to any servers', {'127.0.0.1':
>>>>>> error(111, "Tried connecting to [('127.0.0.1', 9042)]. Last error:
>>>>>> Connection refused")})
>>>>>> whereas
>>>>>> $ cqlsh <ip>
>>>>>> works fine
>>>>>> is that the reason why the cassandra-stress is not able to
>>>>>> communicate with other replicas?
>>>>>
>>>>>
>>>>> Are you providing the -node parameter to stress
>>>>> <http://docs.datastax.com/en/cassandra/2.1/cassandra/tools/toolsCStress_t.html>
>>>>> ?
>>>>>
>>>>>
>>>>>
>>>>> All the best,
>>>>>
>>>>>
>>>>> [image: datastax_logo.png] <http://www.datastax.com/>
>>>>>
>>>>> Sebastián Estévez
>>>>>
>>>>> Solutions Architect | 954 905 8615 | sebastian.estevez@datastax.com
>>>>>
>>>>> [image: linkedin.png] <https://www.linkedin.com/company/datastax> [image:
>>>>> facebook.png] <https://www.facebook.com/datastax> [image: twitter.png]
>>>>> <https://twitter.com/datastax> [image: g+.png]
>>>>> <https://plus.google.com/+Datastax/about>
>>>>> <http://feeds.feedburner.com/datastax>
>>>>> <http://goog_410786983>
>>>>>
>>>>>
>>>>> <http://www.datastax.com/gartner-magic-quadrant-odbms>
>>>>>
>>>>> DataStax is the fastest, most scalable distributed database
>>>>> technology, delivering Apache Cassandra to the world’s most innovative
>>>>> enterprises. Datastax is built to be agile, always-on, and predictably
>>>>> scalable to any size. With more than 500 customers in 45 countries, DataStax
>>>>> is the database technology and transactional backbone of choice for the
>>>>> worlds most innovative companies such as Netflix, Adobe, Intuit, and eBay.
>>>>>
>>>>> On Fri, Jan 22, 2016 at 4:07 PM, Bhuvan Rawal <bh...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> I had a look at the jira below:
>>>>>> https://issues.apache.org/jira/browse/CASSANDRA-7905
>>>>>>
>>>>>> when i opened my cassandra-rackdc.properties i saw that DC names were
>>>>>> DC1 & DC2, rack name was RAC1 . Please note that this is the default
>>>>>> configuration, I have not modified any file.
>>>>>>
>>>>>> There is another point of concern here which might be relevant to
>>>>>> previous one as well, im not able to login to cqlsh directly, i.e. I have
>>>>>> to specify ip as well even when im logged in to that machine.
>>>>>>
>>>>>> $ cqlsh
>>>>>> Connection error: ('Unable to connect to any servers', {'127.0.0.1':
>>>>>> error(111, "Tried connecting to [('127.0.0.1', 9042)]. Last error:
>>>>>> Connection refused")})
>>>>>>
>>>>>> whereas
>>>>>> $ cqlsh <ip>
>>>>>> works fine
>>>>>>
>>>>>> is that the reason why the cassandra-stress is not able to
>>>>>> communicate with other replicas?
>>>>>>
>>>>>> On Sat, Jan 23, 2016 at 1:37 AM, Sebastian Estevez <
>>>>>> sebastian.estevez@datastax.com> wrote:
>>>>>>
>>>>>>> Sorry I missed that.
>>>>>>>
>>>>>>> Both your nodetool status and keyspace replication settings say
>>>>>>> Cassandra and Analytics for the DC names. I'm not sure where you're seeing
>>>>>>> DC1, DC2, etc. and why you suspect that is the problem.
>>>>>>>
>>>>>>> All the best,
>>>>>>>
>>>>>>>
>>>>>>> [image: datastax_logo.png] <http://www.datastax.com/>
>>>>>>>
>>>>>>> Sebastián Estévez
>>>>>>>
>>>>>>> Solutions Architect | 954 905 8615 | sebastian.estevez@datastax.com
>>>>>>>
>>>>>>> [image: linkedin.png] <https://www.linkedin.com/company/datastax> [image:
>>>>>>> facebook.png] <https://www.facebook.com/datastax> [image:
>>>>>>> twitter.png] <https://twitter.com/datastax> [image: g+.png]
>>>>>>> <https://plus.google.com/+Datastax/about>
>>>>>>> <http://feeds.feedburner.com/datastax>
>>>>>>> <http://goog_410786983>
>>>>>>>
>>>>>>>
>>>>>>> <http://www.datastax.com/gartner-magic-quadrant-odbms>
>>>>>>>
>>>>>>> DataStax is the fastest, most scalable distributed database
>>>>>>> technology, delivering Apache Cassandra to the world’s most innovative
>>>>>>> enterprises. Datastax is built to be agile, always-on, and predictably
>>>>>>> scalable to any size. With more than 500 customers in 45 countries, DataStax
>>>>>>> is the database technology and transactional backbone of choice for the
>>>>>>> worlds most innovative companies such as Netflix, Adobe, Intuit, and eBay.
>>>>>>>
>>>>>>> On Fri, Jan 22, 2016 at 1:45 PM, Bhuvan Rawal <bh...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hi Sebastian,
>>>>>>>>
>>>>>>>> I had attached nodetool status output in previous mail, pasting it
>>>>>>>> again :
>>>>>>>>
>>>>>>>> $ nodetool status Datacenter: Analytics =====================
>>>>>>>> Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address
>>>>>>>> Load Tokens Owns Host ID Rack UN 10.41.55.17 428.5 KB 256 ?
>>>>>>>> 39d6d585-e641-4046-9d0b-797356597b5e rack1 UN 10.41.55.19 404.44 KB 256 ?
>>>>>>>> 69edf930-efd9-4d74-a798-f3d4ac02e516 rack1 UN 10.41.55.18 423.21 KB 256 ?
>>>>>>>> b74bab13-09b2-4760-bce9-c8ef05e50f6d rack1 UN 10.41.55.20 683.23 KB 256 ?
>>>>>>>> fb5c4fed-6e1e-4ea8-838d-358106906830 rack1 Datacenter: Cassandra
>>>>>>>> ===================== Status=Up/Down |/
>>>>>>>> State=Normal/Leaving/Joining/Moving -- Address Load Tokens Owns
>>>>>>>> Host ID Rack UN 10.41.55.15 209.4 KB 256 ?
>>>>>>>> ffc3b9a0-5d5c-4a3d-a99e-49d255731278 rack1 UN 10.41.55.21 227.44 KB 256 ?
>>>>>>>> c68deba4-b9a2-43fc-bb13-6af74c88c210 rack1 UN 10.41.55.23 222.71 KB 256 ?
>>>>>>>> 8229aa87-af00-48fa-ad6b-3066d3dc0e58 rack1 UN 10.41.55.22 218.72 KB 256 ?
>>>>>>>> c7ba84fd-7992-41de-8c88-11574a72db99 rack1
>>>>>>>>
>>>>>>>> Regards,
>>>>>>>> Bhuvan Rawal
>>>>>>>>
>>>>>>>> On Sat, Jan 23, 2016 at 12:11 AM, Sebastian Estevez <
>>>>>>>> sebastian.estevez@datastax.com> wrote:
>>>>>>>>
>>>>>>>>> The output of `nodetool status` would help us diagnose.
>>>>>>>>>
>>>>>>>>> All the best,
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> [image: datastax_logo.png] <http://www.datastax.com/>
>>>>>>>>>
>>>>>>>>> Sebastián Estévez
>>>>>>>>>
>>>>>>>>> Solutions Architect | 954 905 8615 |
>>>>>>>>> sebastian.estevez@datastax.com
>>>>>>>>>
>>>>>>>>> [image: linkedin.png] <https://www.linkedin.com/company/datastax> [image:
>>>>>>>>> facebook.png] <https://www.facebook.com/datastax> [image:
>>>>>>>>> twitter.png] <https://twitter.com/datastax> [image: g+.png]
>>>>>>>>> <https://plus.google.com/+Datastax/about>
>>>>>>>>> <http://feeds.feedburner.com/datastax>
>>>>>>>>> <http://goog_410786983>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> <http://www.datastax.com/gartner-magic-quadrant-odbms>
>>>>>>>>>
>>>>>>>>> DataStax is the fastest, most scalable distributed database
>>>>>>>>> technology, delivering Apache Cassandra to the world’s most innovative
>>>>>>>>> enterprises. Datastax is built to be agile, always-on, and predictably
>>>>>>>>> scalable to any size. With more than 500 customers in 45 countries, DataStax
>>>>>>>>> is the database technology and transactional backbone of choice for the
>>>>>>>>> worlds most innovative companies such as Netflix, Adobe, Intuit, and eBay.
>>>>>>>>>
>>>>>>>>> On Fri, Jan 22, 2016 at 1:39 PM, Bhuvan Rawal <bhu1rawal@gmail.com
>>>>>>>>> > wrote:
>>>>>>>>>
>>>>>>>>>> Thanks for the response Alain,
>>>>>>>>>>
>>>>>>>>>> cqlsh> create keyspace mykeyspace WITH replication =
>>>>>>>>>> {'class':'NetworkTopologyStrategy', 'Analytics':2, 'Cassandra':3}
>>>>>>>>>> cqlsh> use mykeyspace;
>>>>>>>>>> cqlsh:mykeyspace>create table mytable (id int primary key, name
>>>>>>>>>> text, address text, phone text);
>>>>>>>>>> cqlsh:mykeyspace> insert into mytable (id, name, address, phone)
>>>>>>>>>> values (1, 'Kiyu','Texas', '555-1212'); # and other similar statement
>>>>>>>>>> I then issued the below command from every node and found
>>>>>>>>>> consistent results.
>>>>>>>>>> cqlsh:mykeyspace> select * from mytable;
>>>>>>>>>>
>>>>>>>>>> // Then i repeated the above steps for NetworkTopologyStrategy
>>>>>>>>>> and found same results
>>>>>>>>>>
>>>>>>>>>> I ran basic cassandra stress
>>>>>>>>>> seed1 - seed of datacenter 1
>>>>>>>>>>  $ cassandra-stress write n=50000 -rate threads=4 -node
>>>>>>>>>> any_random_ip
>>>>>>>>>>  $ cassandra-stress write n=50000 -rate threads=4 -node seed1
>>>>>>>>>>  $ cassandra-stress write n=50000 -rate threads=4 -node
>>>>>>>>>> seed1,seed2
>>>>>>>>>>  $ cassandra-stress write n=50000 -rate threads=4 -node
>>>>>>>>>> all_8_ip_comma_seperated
>>>>>>>>>>  $ cassandra-stress write n=1000000 cl=one -mode native cql3
>>>>>>>>>> -schema keyspace="keyspace1" -pop seq=1..1000000 -node ip1,ip2,ip3,ip4
>>>>>>>>>>
>>>>>>>>>> All of them threw the exception
>>>>>>>>>> *com.datastax.driver.core.exceptions.UnavailableException: Not
>>>>>>>>>> enough replica available for query at consistency LOCAL_ONE (1 required but
>>>>>>>>>> only 0 alive)*
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> I have a feeling that the issue is with datacenter name for some
>>>>>>>>>> reason, because in some config files I found DC name to be like DC1/DC2/DC3
>>>>>>>>>> in some it is like Cassandra/Analytics (The ones I had specified while
>>>>>>>>>> installation). Im unsure which yaml/property file to look for correct
>>>>>>>>>> inconsistency.
>>>>>>>>>>
>>>>>>>>>> (C*heers :) - im so tempted to copy that)
>>>>>>>>>>
>>>>>>>>>> Regards,
>>>>>>>>>> Bhuvan
>>>>>>>>>>
>>>>>>>>>> On Fri, Jan 22, 2016 at 8:47 PM, Alain RODRIGUEZ <
>>>>>>>>>> arodrime@gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi,
>>>>>>>>>>>
>>>>>>>>>>> The the exact command you ran (stress-tool with options) could
>>>>>>>>>>> be useful to help you on that.
>>>>>>>>>>>
>>>>>>>>>>> However, Im able to create keyspace, tables and insert data
>>>>>>>>>>>> using cqlsh and it is replicating fine to all the nodes.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Having the schema might be useful too.
>>>>>>>>>>>
>>>>>>>>>>> Did you ran the cqlsh and the stress-tool from the same server
>>>>>>>>>>> ? If not, you might want to check the port you use (9042/9160/...) are open.
>>>>>>>>>>> Also, cqlsh uses local_one by default too. If both commands were
>>>>>>>>>>> run against the same DC, from the same machine they should behave the same
>>>>>>>>>>> way. Are they ?
>>>>>>>>>>>
>>>>>>>>>>> C*heers,
>>>>>>>>>>>
>>>>>>>>>>> -----------------
>>>>>>>>>>> Alain
>>>>>>>>>>>
>>>>>>>>>>> The Last Pickle
>>>>>>>>>>> http://www.thelastpickle.com
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> 2016-01-22 9:57 GMT+01:00 Bhuvan Rawal <bh...@gmail.com>:
>>>>>>>>>>>
>>>>>>>>>>>> Hi,
>>>>>>>>>>>>
>>>>>>>>>>>> i have created a POC cluster with 2 DC , each having 4 nodes
>>>>>>>>>>>> with DSE 4.8.1 installed.
>>>>>>>>>>>>
>>>>>>>>>>>> On issuing cassandra stress im getting an error  and data is
>>>>>>>>>>>> not being inserted:
>>>>>>>>>>>> *com.datastax.driver.core.exceptions.UnavailableException: Not
>>>>>>>>>>>> enough replica available for query at consistency LOCAL_ONE (1 required but
>>>>>>>>>>>> only 0 alive)*
>>>>>>>>>>>>
>>>>>>>>>>>> However, Im able to create keyspace, tables and insert data
>>>>>>>>>>>> using cqlsh and it is replicating fine to all the nodes.
>>>>>>>>>>>>
>>>>>>>>>>>> Details of the cluster can be found below (all the nodes seem
>>>>>>>>>>>> to be alive and kicking):
>>>>>>>>>>>>
>>>>>>>>>>>> $ nodetool status Datacenter: Analytics =====================
>>>>>>>>>>>> Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address
>>>>>>>>>>>> Load Tokens Owns Host ID Rack UN 10.41.55.17 428.5 KB 256 ?
>>>>>>>>>>>> 39d6d585-e641-4046-9d0b-797356597b5e rack1 UN 10.41.55.19 404.44 KB 256 ?
>>>>>>>>>>>> 69edf930-efd9-4d74-a798-f3d4ac02e516 rack1 UN 10.41.55.18 423.21 KB 256 ?
>>>>>>>>>>>> b74bab13-09b2-4760-bce9-c8ef05e50f6d rack1 UN 10.41.55.20 683.23 KB 256 ?
>>>>>>>>>>>> fb5c4fed-6e1e-4ea8-838d-358106906830 rack1 Datacenter:
>>>>>>>>>>>> Cassandra ===================== Status=Up/Down |/
>>>>>>>>>>>> State=Normal/Leaving/Joining/Moving -- Address Load Tokens
>>>>>>>>>>>> Owns Host ID Rack UN 10.41.55.15 209.4 KB 256 ?
>>>>>>>>>>>> ffc3b9a0-5d5c-4a3d-a99e-49d255731278 rack1 UN 10.41.55.21 227.44 KB 256 ?
>>>>>>>>>>>> c68deba4-b9a2-43fc-bb13-6af74c88c210 rack1 UN 10.41.55.23 222.71 KB 256 ?
>>>>>>>>>>>> 8229aa87-af00-48fa-ad6b-3066d3dc0e58 rack1 UN 10.41.55.22 218.72 KB 256 ?
>>>>>>>>>>>> c7ba84fd-7992-41de-8c88-11574a72db99 rack1
>>>>>>>>>>>>
>>>>>>>>>>>> Regards,
>>>>>>>>>>>> Bhuvan Rawal
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>

Re: Getting error while issuing Cassandra stress

Posted by Alain RODRIGUEZ <ar...@gmail.com>.

>
> Should I use nodetool repair utility
>

That wouldn't help, this an anti-entropy mechanism (see
https://docs.datastax.com/en/cassandra/2.1/cassandra/tools/toolsRepair.html#toolsRepair__description_unique_27
).

It is something really important too often left aside.

Yet, your issue here is not about consistency. The client can't find any
node in charge of the read / written tokens in the ring. This depends on
the topology, the replication factor and your network mainly. I think there
is something wrong in your setup. I would try this:

- Make sure connection / port are ok
- Try increasing the RF / Strategy in the stress tool
- Try with an other consistency level (not LOCAL_*, as mentioned here :
http://stackoverflow.com/questions/32055251/not-enough-replica-available-for-query-at-consistency-local-one-1-required-but
)

Good luck,

-----------------
Alain

The Last Pickle
http://www.thelastpickle.com

2016-01-22 23:02 GMT+01:00 Bhuvan Rawal <bh...@gmail.com>:

> Getting same exception again. Should I use nodetool repair utility?
>
> On Sat, Jan 23, 2016 at 3:10 AM, Sebastian Estevez <
> sebastian.estevez@datastax.com> wrote:
>
>> https://github.com/brianmhess/cassandra-loader
>>
>> All the best,
>>
>>
>> [image: datastax_logo.png] <http://www.datastax.com/>
>>
>> Sebastián Estévez
>>
>> Solutions Architect | 954 905 8615 | sebastian.estevez@datastax.com
>>
>> [image: linkedin.png] <https://www.linkedin.com/company/datastax> [image:
>> facebook.png] <https://www.facebook.com/datastax> [image: twitter.png]
>> <https://twitter.com/datastax> [image: g+.png]
>> <https://plus.google.com/+Datastax/about>
>> <http://feeds.feedburner.com/datastax>
>> <http://goog_410786983>
>>
>>
>> <http://www.datastax.com/gartner-magic-quadrant-odbms>
>>
>> DataStax is the fastest, most scalable distributed database technology,
>> delivering Apache Cassandra to the world’s most innovative enterprises.
>> Datastax is built to be agile, always-on, and predictably scalable to any
>> size. With more than 500 customers in 45 countries, DataStax is the
>> database technology and transactional backbone of choice for the worlds
>> most innovative companies such as Netflix, Adobe, Intuit, and eBay.
>>
>> On Fri, Jan 22, 2016 at 4:37 PM, Bhuvan Rawal <bh...@gmail.com>
>> wrote:
>>
>>> Yes im specifying -node parameter to stress, otherwise it throws network
>>> connection failed.
>>>
>>> Can you point me to a sample java application to test pushing data from
>>> external server? Let's see if that works
>>>
>>> On Sat, Jan 23, 2016 at 2:55 AM, Sebastian Estevez <
>>> sebastian.estevez@datastax.com> wrote:
>>>
>>>> when i opened my cassandra-rackdc.properties i saw that DC names were
>>>>> DC1 & DC2, rack name was RAC1 . Please note that this is the default
>>>>> configuration, I have not modified any file.
>>>>
>>>>
>>>> cassandra-rackdc.properties is only respected based on your snitch
>>>> <https://docs.datastax.com/en/cassandra/2.1/cassandra/architecture/architectureSnitchesAbout_c.html>
>>>> .
>>>>
>>>> $ cqlsh
>>>>> Connection error: ('Unable to connect to any servers', {'127.0.0.1':
>>>>> error(111, "Tried connecting to [('127.0.0.1', 9042)]. Last error:
>>>>> Connection refused")})
>>>>> whereas
>>>>> $ cqlsh <ip>
>>>>> works fine
>>>>> is that the reason why the cassandra-stress is not able to communicate
>>>>> with other replicas?
>>>>
>>>>
>>>> Are you providing the -node parameter to stress
>>>> <http://docs.datastax.com/en/cassandra/2.1/cassandra/tools/toolsCStress_t.html>
>>>> ?
>>>>
>>>>
>>>>
>>>> All the best,
>>>>
>>>>
>>>> [image: datastax_logo.png] <http://www.datastax.com/>
>>>>
>>>> Sebastián Estévez
>>>>
>>>> Solutions Architect | 954 905 8615 | sebastian.estevez@datastax.com
>>>>
>>>> [image: linkedin.png] <https://www.linkedin.com/company/datastax> [image:
>>>> facebook.png] <https://www.facebook.com/datastax> [image: twitter.png]
>>>> <https://twitter.com/datastax> [image: g+.png]
>>>> <https://plus.google.com/+Datastax/about>
>>>> <http://feeds.feedburner.com/datastax>
>>>> <http://goog_410786983>
>>>>
>>>>
>>>> <http://www.datastax.com/gartner-magic-quadrant-odbms>
>>>>
>>>> DataStax is the fastest, most scalable distributed database
>>>> technology, delivering Apache Cassandra to the world’s most innovative
>>>> enterprises. Datastax is built to be agile, always-on, and predictably
>>>> scalable to any size. With more than 500 customers in 45 countries, DataStax
>>>> is the database technology and transactional backbone of choice for the
>>>> worlds most innovative companies such as Netflix, Adobe, Intuit, and eBay.
>>>>
>>>> On Fri, Jan 22, 2016 at 4:07 PM, Bhuvan Rawal <bh...@gmail.com>
>>>> wrote:
>>>>
>>>>> I had a look at the jira below:
>>>>> https://issues.apache.org/jira/browse/CASSANDRA-7905
>>>>>
>>>>> when i opened my cassandra-rackdc.properties i saw that DC names were
>>>>> DC1 & DC2, rack name was RAC1 . Please note that this is the default
>>>>> configuration, I have not modified any file.
>>>>>
>>>>> There is another point of concern here which might be relevant to
>>>>> previous one as well, im not able to login to cqlsh directly, i.e. I have
>>>>> to specify ip as well even when im logged in to that machine.
>>>>>
>>>>> $ cqlsh
>>>>> Connection error: ('Unable to connect to any servers', {'127.0.0.1':
>>>>> error(111, "Tried connecting to [('127.0.0.1', 9042)]. Last error:
>>>>> Connection refused")})
>>>>>
>>>>> whereas
>>>>> $ cqlsh <ip>
>>>>> works fine
>>>>>
>>>>> is that the reason why the cassandra-stress is not able to communicate
>>>>> with other replicas?
>>>>>
>>>>> On Sat, Jan 23, 2016 at 1:37 AM, Sebastian Estevez <
>>>>> sebastian.estevez@datastax.com> wrote:
>>>>>
>>>>>> Sorry I missed that.
>>>>>>
>>>>>> Both your nodetool status and keyspace replication settings say
>>>>>> Cassandra and Analytics for the DC names. I'm not sure where you're seeing
>>>>>> DC1, DC2, etc. and why you suspect that is the problem.
>>>>>>
>>>>>> All the best,
>>>>>>
>>>>>>
>>>>>> [image: datastax_logo.png] <http://www.datastax.com/>
>>>>>>
>>>>>> Sebastián Estévez
>>>>>>
>>>>>> Solutions Architect | 954 905 8615 | sebastian.estevez@datastax.com
>>>>>>
>>>>>> [image: linkedin.png] <https://www.linkedin.com/company/datastax> [image:
>>>>>> facebook.png] <https://www.facebook.com/datastax> [image:
>>>>>> twitter.png] <https://twitter.com/datastax> [image: g+.png]
>>>>>> <https://plus.google.com/+Datastax/about>
>>>>>> <http://feeds.feedburner.com/datastax>
>>>>>> <http://goog_410786983>
>>>>>>
>>>>>>
>>>>>> <http://www.datastax.com/gartner-magic-quadrant-odbms>
>>>>>>
>>>>>> DataStax is the fastest, most scalable distributed database
>>>>>> technology, delivering Apache Cassandra to the world’s most innovative
>>>>>> enterprises. Datastax is built to be agile, always-on, and predictably
>>>>>> scalable to any size. With more than 500 customers in 45 countries, DataStax
>>>>>> is the database technology and transactional backbone of choice for the
>>>>>> worlds most innovative companies such as Netflix, Adobe, Intuit, and eBay.
>>>>>>
>>>>>> On Fri, Jan 22, 2016 at 1:45 PM, Bhuvan Rawal <bh...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi Sebastian,
>>>>>>>
>>>>>>> I had attached nodetool status output in previous mail, pasting it
>>>>>>> again :
>>>>>>>
>>>>>>> $ nodetool status Datacenter: Analytics =====================
>>>>>>> Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address
>>>>>>> Load Tokens Owns Host ID Rack UN 10.41.55.17 428.5 KB 256 ?
>>>>>>> 39d6d585-e641-4046-9d0b-797356597b5e rack1 UN 10.41.55.19 404.44 KB 256 ?
>>>>>>> 69edf930-efd9-4d74-a798-f3d4ac02e516 rack1 UN 10.41.55.18 423.21 KB 256 ?
>>>>>>> b74bab13-09b2-4760-bce9-c8ef05e50f6d rack1 UN 10.41.55.20 683.23 KB 256 ?
>>>>>>> fb5c4fed-6e1e-4ea8-838d-358106906830 rack1 Datacenter: Cassandra
>>>>>>> ===================== Status=Up/Down |/
>>>>>>> State=Normal/Leaving/Joining/Moving -- Address Load Tokens Owns
>>>>>>> Host ID Rack UN 10.41.55.15 209.4 KB 256 ?
>>>>>>> ffc3b9a0-5d5c-4a3d-a99e-49d255731278 rack1 UN 10.41.55.21 227.44 KB 256 ?
>>>>>>> c68deba4-b9a2-43fc-bb13-6af74c88c210 rack1 UN 10.41.55.23 222.71 KB 256 ?
>>>>>>> 8229aa87-af00-48fa-ad6b-3066d3dc0e58 rack1 UN 10.41.55.22 218.72 KB 256 ?
>>>>>>> c7ba84fd-7992-41de-8c88-11574a72db99 rack1
>>>>>>>
>>>>>>> Regards,
>>>>>>> Bhuvan Rawal
>>>>>>>
>>>>>>> On Sat, Jan 23, 2016 at 12:11 AM, Sebastian Estevez <
>>>>>>> sebastian.estevez@datastax.com> wrote:
>>>>>>>
>>>>>>>> The output of `nodetool status` would help us diagnose.
>>>>>>>>
>>>>>>>> All the best,
>>>>>>>>
>>>>>>>>
>>>>>>>> [image: datastax_logo.png] <http://www.datastax.com/>
>>>>>>>>
>>>>>>>> Sebastián Estévez
>>>>>>>>
>>>>>>>> Solutions Architect | 954 905 8615 | sebastian.estevez@datastax.com
>>>>>>>>
>>>>>>>> [image: linkedin.png] <https://www.linkedin.com/company/datastax> [image:
>>>>>>>> facebook.png] <https://www.facebook.com/datastax> [image:
>>>>>>>> twitter.png] <https://twitter.com/datastax> [image: g+.png]
>>>>>>>> <https://plus.google.com/+Datastax/about>
>>>>>>>> <http://feeds.feedburner.com/datastax>
>>>>>>>> <http://goog_410786983>
>>>>>>>>
>>>>>>>>
>>>>>>>> <http://www.datastax.com/gartner-magic-quadrant-odbms>
>>>>>>>>
>>>>>>>> DataStax is the fastest, most scalable distributed database
>>>>>>>> technology, delivering Apache Cassandra to the world’s most innovative
>>>>>>>> enterprises. Datastax is built to be agile, always-on, and predictably
>>>>>>>> scalable to any size. With more than 500 customers in 45 countries, DataStax
>>>>>>>> is the database technology and transactional backbone of choice for the
>>>>>>>> worlds most innovative companies such as Netflix, Adobe, Intuit, and eBay.
>>>>>>>>
>>>>>>>> On Fri, Jan 22, 2016 at 1:39 PM, Bhuvan Rawal <bh...@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Thanks for the response Alain,
>>>>>>>>>
>>>>>>>>> cqlsh> create keyspace mykeyspace WITH replication =
>>>>>>>>> {'class':'NetworkTopologyStrategy', 'Analytics':2, 'Cassandra':3}
>>>>>>>>> cqlsh> use mykeyspace;
>>>>>>>>> cqlsh:mykeyspace>create table mytable (id int primary key, name
>>>>>>>>> text, address text, phone text);
>>>>>>>>> cqlsh:mykeyspace> insert into mytable (id, name, address, phone)
>>>>>>>>> values (1, 'Kiyu','Texas', '555-1212'); # and other similar statement
>>>>>>>>> I then issued the below command from every node and found
>>>>>>>>> consistent results.
>>>>>>>>> cqlsh:mykeyspace> select * from mytable;
>>>>>>>>>
>>>>>>>>> // Then i repeated the above steps for NetworkTopologyStrategy and
>>>>>>>>> found same results
>>>>>>>>>
>>>>>>>>> I ran basic cassandra stress
>>>>>>>>> seed1 - seed of datacenter 1
>>>>>>>>>  $ cassandra-stress write n=50000 -rate threads=4 -node
>>>>>>>>> any_random_ip
>>>>>>>>>  $ cassandra-stress write n=50000 -rate threads=4 -node seed1
>>>>>>>>>  $ cassandra-stress write n=50000 -rate threads=4 -node seed1,seed2
>>>>>>>>>  $ cassandra-stress write n=50000 -rate threads=4 -node
>>>>>>>>> all_8_ip_comma_seperated
>>>>>>>>>  $ cassandra-stress write n=1000000 cl=one -mode native cql3
>>>>>>>>> -schema keyspace="keyspace1" -pop seq=1..1000000 -node ip1,ip2,ip3,ip4
>>>>>>>>>
>>>>>>>>> All of them threw the exception
>>>>>>>>> *com.datastax.driver.core.exceptions.UnavailableException: Not
>>>>>>>>> enough replica available for query at consistency LOCAL_ONE (1 required but
>>>>>>>>> only 0 alive)*
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> I have a feeling that the issue is with datacenter name for some
>>>>>>>>> reason, because in some config files I found DC name to be like DC1/DC2/DC3
>>>>>>>>> in some it is like Cassandra/Analytics (The ones I had specified while
>>>>>>>>> installation). Im unsure which yaml/property file to look for correct
>>>>>>>>> inconsistency.
>>>>>>>>>
>>>>>>>>> (C*heers :) - im so tempted to copy that)
>>>>>>>>>
>>>>>>>>> Regards,
>>>>>>>>> Bhuvan
>>>>>>>>>
>>>>>>>>> On Fri, Jan 22, 2016 at 8:47 PM, Alain RODRIGUEZ <
>>>>>>>>> arodrime@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> Hi,
>>>>>>>>>>
>>>>>>>>>> The the exact command you ran (stress-tool with options) could be
>>>>>>>>>> useful to help you on that.
>>>>>>>>>>
>>>>>>>>>> However, Im able to create keyspace, tables and insert data using
>>>>>>>>>>> cqlsh and it is replicating fine to all the nodes.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Having the schema might be useful too.
>>>>>>>>>>
>>>>>>>>>> Did you ran the cqlsh and the stress-tool from the same server ?
>>>>>>>>>> If not, you might want to check the port you use (9042/9160/...) are open.
>>>>>>>>>> Also, cqlsh uses local_one by default too. If both commands were
>>>>>>>>>> run against the same DC, from the same machine they should behave the same
>>>>>>>>>> way. Are they ?
>>>>>>>>>>
>>>>>>>>>> C*heers,
>>>>>>>>>>
>>>>>>>>>> -----------------
>>>>>>>>>> Alain
>>>>>>>>>>
>>>>>>>>>> The Last Pickle
>>>>>>>>>> http://www.thelastpickle.com
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> 2016-01-22 9:57 GMT+01:00 Bhuvan Rawal <bh...@gmail.com>:
>>>>>>>>>>
>>>>>>>>>>> Hi,
>>>>>>>>>>>
>>>>>>>>>>> i have created a POC cluster with 2 DC , each having 4 nodes
>>>>>>>>>>> with DSE 4.8.1 installed.
>>>>>>>>>>>
>>>>>>>>>>> On issuing cassandra stress im getting an error  and data is not
>>>>>>>>>>> being inserted:
>>>>>>>>>>> *com.datastax.driver.core.exceptions.UnavailableException: Not
>>>>>>>>>>> enough replica available for query at consistency LOCAL_ONE (1 required but
>>>>>>>>>>> only 0 alive)*
>>>>>>>>>>>
>>>>>>>>>>> However, Im able to create keyspace, tables and insert data
>>>>>>>>>>> using cqlsh and it is replicating fine to all the nodes.
>>>>>>>>>>>
>>>>>>>>>>> Details of the cluster can be found below (all the nodes seem to
>>>>>>>>>>> be alive and kicking):
>>>>>>>>>>>
>>>>>>>>>>> $ nodetool status Datacenter: Analytics =====================
>>>>>>>>>>> Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address
>>>>>>>>>>> Load Tokens Owns Host ID Rack UN 10.41.55.17 428.5 KB 256 ?
>>>>>>>>>>> 39d6d585-e641-4046-9d0b-797356597b5e rack1 UN 10.41.55.19 404.44 KB 256 ?
>>>>>>>>>>> 69edf930-efd9-4d74-a798-f3d4ac02e516 rack1 UN 10.41.55.18 423.21 KB 256 ?
>>>>>>>>>>> b74bab13-09b2-4760-bce9-c8ef05e50f6d rack1 UN 10.41.55.20 683.23 KB 256 ?
>>>>>>>>>>> fb5c4fed-6e1e-4ea8-838d-358106906830 rack1 Datacenter:
>>>>>>>>>>> Cassandra ===================== Status=Up/Down |/
>>>>>>>>>>> State=Normal/Leaving/Joining/Moving -- Address Load Tokens Owns
>>>>>>>>>>> Host ID Rack UN 10.41.55.15 209.4 KB 256 ?
>>>>>>>>>>> ffc3b9a0-5d5c-4a3d-a99e-49d255731278 rack1 UN 10.41.55.21 227.44 KB 256 ?
>>>>>>>>>>> c68deba4-b9a2-43fc-bb13-6af74c88c210 rack1 UN 10.41.55.23 222.71 KB 256 ?
>>>>>>>>>>> 8229aa87-af00-48fa-ad6b-3066d3dc0e58 rack1 UN 10.41.55.22 218.72 KB 256 ?
>>>>>>>>>>> c7ba84fd-7992-41de-8c88-11574a72db99 rack1
>>>>>>>>>>>
>>>>>>>>>>> Regards,
>>>>>>>>>>> Bhuvan Rawal
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>

Re: Getting error while issuing Cassandra stress

Posted by Bhuvan Rawal <bh...@gmail.com>.

Getting same exception again. Should I use nodetool repair utility?

On Sat, Jan 23, 2016 at 3:10 AM, Sebastian Estevez <
sebastian.estevez@datastax.com> wrote:

> https://github.com/brianmhess/cassandra-loader
>
> All the best,
>
>
> [image: datastax_logo.png] <http://www.datastax.com/>
>
> Sebastián Estévez
>
> Solutions Architect | 954 905 8615 | sebastian.estevez@datastax.com
>
> [image: linkedin.png] <https://www.linkedin.com/company/datastax> [image:
> facebook.png] <https://www.facebook.com/datastax> [image: twitter.png]
> <https://twitter.com/datastax> [image: g+.png]
> <https://plus.google.com/+Datastax/about>
> <http://feeds.feedburner.com/datastax>
> <http://goog_410786983>
>
>
> <http://www.datastax.com/gartner-magic-quadrant-odbms>
>
> DataStax is the fastest, most scalable distributed database technology,
> delivering Apache Cassandra to the world’s most innovative enterprises.
> Datastax is built to be agile, always-on, and predictably scalable to any
> size. With more than 500 customers in 45 countries, DataStax is the
> database technology and transactional backbone of choice for the worlds
> most innovative companies such as Netflix, Adobe, Intuit, and eBay.
>
> On Fri, Jan 22, 2016 at 4:37 PM, Bhuvan Rawal <bh...@gmail.com> wrote:
>
>> Yes im specifying -node parameter to stress, otherwise it throws network
>> connection failed.
>>
>> Can you point me to a sample java application to test pushing data from
>> external server? Let's see if that works
>>
>> On Sat, Jan 23, 2016 at 2:55 AM, Sebastian Estevez <
>> sebastian.estevez@datastax.com> wrote:
>>
>>> when i opened my cassandra-rackdc.properties i saw that DC names were
>>>> DC1 & DC2, rack name was RAC1 . Please note that this is the default
>>>> configuration, I have not modified any file.
>>>
>>>
>>> cassandra-rackdc.properties is only respected based on your snitch
>>> <https://docs.datastax.com/en/cassandra/2.1/cassandra/architecture/architectureSnitchesAbout_c.html>
>>> .
>>>
>>> $ cqlsh
>>>> Connection error: ('Unable to connect to any servers', {'127.0.0.1':
>>>> error(111, "Tried connecting to [('127.0.0.1', 9042)]. Last error:
>>>> Connection refused")})
>>>> whereas
>>>> $ cqlsh <ip>
>>>> works fine
>>>> is that the reason why the cassandra-stress is not able to communicate
>>>> with other replicas?
>>>
>>>
>>> Are you providing the -node parameter to stress
>>> <http://docs.datastax.com/en/cassandra/2.1/cassandra/tools/toolsCStress_t.html>
>>> ?
>>>
>>>
>>>
>>> All the best,
>>>
>>>
>>> [image: datastax_logo.png] <http://www.datastax.com/>
>>>
>>> Sebastián Estévez
>>>
>>> Solutions Architect | 954 905 8615 | sebastian.estevez@datastax.com
>>>
>>> [image: linkedin.png] <https://www.linkedin.com/company/datastax> [image:
>>> facebook.png] <https://www.facebook.com/datastax> [image: twitter.png]
>>> <https://twitter.com/datastax> [image: g+.png]
>>> <https://plus.google.com/+Datastax/about>
>>> <http://feeds.feedburner.com/datastax>
>>> <http://goog_410786983>
>>>
>>>
>>> <http://www.datastax.com/gartner-magic-quadrant-odbms>
>>>
>>> DataStax is the fastest, most scalable distributed database technology,
>>> delivering Apache Cassandra to the world’s most innovative enterprises.
>>> Datastax is built to be agile, always-on, and predictably scalable to any
>>> size. With more than 500 customers in 45 countries, DataStax is the
>>> database technology and transactional backbone of choice for the worlds
>>> most innovative companies such as Netflix, Adobe, Intuit, and eBay.
>>>
>>> On Fri, Jan 22, 2016 at 4:07 PM, Bhuvan Rawal <bh...@gmail.com>
>>> wrote:
>>>
>>>> I had a look at the jira below:
>>>> https://issues.apache.org/jira/browse/CASSANDRA-7905
>>>>
>>>> when i opened my cassandra-rackdc.properties i saw that DC names were
>>>> DC1 & DC2, rack name was RAC1 . Please note that this is the default
>>>> configuration, I have not modified any file.
>>>>
>>>> There is another point of concern here which might be relevant to
>>>> previous one as well, im not able to login to cqlsh directly, i.e. I have
>>>> to specify ip as well even when im logged in to that machine.
>>>>
>>>> $ cqlsh
>>>> Connection error: ('Unable to connect to any servers', {'127.0.0.1':
>>>> error(111, "Tried connecting to [('127.0.0.1', 9042)]. Last error:
>>>> Connection refused")})
>>>>
>>>> whereas
>>>> $ cqlsh <ip>
>>>> works fine
>>>>
>>>> is that the reason why the cassandra-stress is not able to communicate
>>>> with other replicas?
>>>>
>>>> On Sat, Jan 23, 2016 at 1:37 AM, Sebastian Estevez <
>>>> sebastian.estevez@datastax.com> wrote:
>>>>
>>>>> Sorry I missed that.
>>>>>
>>>>> Both your nodetool status and keyspace replication settings say
>>>>> Cassandra and Analytics for the DC names. I'm not sure where you're seeing
>>>>> DC1, DC2, etc. and why you suspect that is the problem.
>>>>>
>>>>> All the best,
>>>>>
>>>>>
>>>>> [image: datastax_logo.png] <http://www.datastax.com/>
>>>>>
>>>>> Sebastián Estévez
>>>>>
>>>>> Solutions Architect | 954 905 8615 | sebastian.estevez@datastax.com
>>>>>
>>>>> [image: linkedin.png] <https://www.linkedin.com/company/datastax> [image:
>>>>> facebook.png] <https://www.facebook.com/datastax> [image: twitter.png]
>>>>> <https://twitter.com/datastax> [image: g+.png]
>>>>> <https://plus.google.com/+Datastax/about>
>>>>> <http://feeds.feedburner.com/datastax>
>>>>> <http://goog_410786983>
>>>>>
>>>>>
>>>>> <http://www.datastax.com/gartner-magic-quadrant-odbms>
>>>>>
>>>>> DataStax is the fastest, most scalable distributed database
>>>>> technology, delivering Apache Cassandra to the world’s most innovative
>>>>> enterprises. Datastax is built to be agile, always-on, and predictably
>>>>> scalable to any size. With more than 500 customers in 45 countries, DataStax
>>>>> is the database technology and transactional backbone of choice for the
>>>>> worlds most innovative companies such as Netflix, Adobe, Intuit, and eBay.
>>>>>
>>>>> On Fri, Jan 22, 2016 at 1:45 PM, Bhuvan Rawal <bh...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hi Sebastian,
>>>>>>
>>>>>> I had attached nodetool status output in previous mail, pasting it
>>>>>> again :
>>>>>>
>>>>>> $ nodetool status Datacenter: Analytics =====================
>>>>>> Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address
>>>>>> Load Tokens Owns Host ID Rack UN 10.41.55.17 428.5 KB 256 ?
>>>>>> 39d6d585-e641-4046-9d0b-797356597b5e rack1 UN 10.41.55.19 404.44 KB 256 ?
>>>>>> 69edf930-efd9-4d74-a798-f3d4ac02e516 rack1 UN 10.41.55.18 423.21 KB 256 ?
>>>>>> b74bab13-09b2-4760-bce9-c8ef05e50f6d rack1 UN 10.41.55.20 683.23 KB 256 ?
>>>>>> fb5c4fed-6e1e-4ea8-838d-358106906830 rack1 Datacenter: Cassandra
>>>>>> ===================== Status=Up/Down |/
>>>>>> State=Normal/Leaving/Joining/Moving -- Address Load Tokens Owns Host
>>>>>> ID Rack UN 10.41.55.15 209.4 KB 256 ? ffc3b9a0-5d5c-4a3d-a99e-49d255731278
>>>>>> rack1 UN 10.41.55.21 227.44 KB 256 ? c68deba4-b9a2-43fc-bb13-6af74c88c210
>>>>>> rack1 UN 10.41.55.23 222.71 KB 256 ? 8229aa87-af00-48fa-ad6b-3066d3dc0e58
>>>>>> rack1 UN 10.41.55.22 218.72 KB 256 ? c7ba84fd-7992-41de-8c88-11574a72db99
>>>>>> rack1
>>>>>>
>>>>>> Regards,
>>>>>> Bhuvan Rawal
>>>>>>
>>>>>> On Sat, Jan 23, 2016 at 12:11 AM, Sebastian Estevez <
>>>>>> sebastian.estevez@datastax.com> wrote:
>>>>>>
>>>>>>> The output of `nodetool status` would help us diagnose.
>>>>>>>
>>>>>>> All the best,
>>>>>>>
>>>>>>>
>>>>>>> [image: datastax_logo.png] <http://www.datastax.com/>
>>>>>>>
>>>>>>> Sebastián Estévez
>>>>>>>
>>>>>>> Solutions Architect | 954 905 8615 | sebastian.estevez@datastax.com
>>>>>>>
>>>>>>> [image: linkedin.png] <https://www.linkedin.com/company/datastax> [image:
>>>>>>> facebook.png] <https://www.facebook.com/datastax> [image:
>>>>>>> twitter.png] <https://twitter.com/datastax> [image: g+.png]
>>>>>>> <https://plus.google.com/+Datastax/about>
>>>>>>> <http://feeds.feedburner.com/datastax>
>>>>>>> <http://goog_410786983>
>>>>>>>
>>>>>>>
>>>>>>> <http://www.datastax.com/gartner-magic-quadrant-odbms>
>>>>>>>
>>>>>>> DataStax is the fastest, most scalable distributed database
>>>>>>> technology, delivering Apache Cassandra to the world’s most innovative
>>>>>>> enterprises. Datastax is built to be agile, always-on, and predictably
>>>>>>> scalable to any size. With more than 500 customers in 45 countries, DataStax
>>>>>>> is the database technology and transactional backbone of choice for the
>>>>>>> worlds most innovative companies such as Netflix, Adobe, Intuit, and eBay.
>>>>>>>
>>>>>>> On Fri, Jan 22, 2016 at 1:39 PM, Bhuvan Rawal <bh...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Thanks for the response Alain,
>>>>>>>>
>>>>>>>> cqlsh> create keyspace mykeyspace WITH replication =
>>>>>>>> {'class':'NetworkTopologyStrategy', 'Analytics':2, 'Cassandra':3}
>>>>>>>> cqlsh> use mykeyspace;
>>>>>>>> cqlsh:mykeyspace>create table mytable (id int primary key, name
>>>>>>>> text, address text, phone text);
>>>>>>>> cqlsh:mykeyspace> insert into mytable (id, name, address, phone)
>>>>>>>> values (1, 'Kiyu','Texas', '555-1212'); # and other similar statement
>>>>>>>> I then issued the below command from every node and found
>>>>>>>> consistent results.
>>>>>>>> cqlsh:mykeyspace> select * from mytable;
>>>>>>>>
>>>>>>>> // Then i repeated the above steps for NetworkTopologyStrategy and
>>>>>>>> found same results
>>>>>>>>
>>>>>>>> I ran basic cassandra stress
>>>>>>>> seed1 - seed of datacenter 1
>>>>>>>>  $ cassandra-stress write n=50000 -rate threads=4 -node
>>>>>>>> any_random_ip
>>>>>>>>  $ cassandra-stress write n=50000 -rate threads=4 -node seed1
>>>>>>>>  $ cassandra-stress write n=50000 -rate threads=4 -node seed1,seed2
>>>>>>>>  $ cassandra-stress write n=50000 -rate threads=4 -node
>>>>>>>> all_8_ip_comma_seperated
>>>>>>>>  $ cassandra-stress write n=1000000 cl=one -mode native cql3
>>>>>>>> -schema keyspace="keyspace1" -pop seq=1..1000000 -node ip1,ip2,ip3,ip4
>>>>>>>>
>>>>>>>> All of them threw the exception
>>>>>>>> *com.datastax.driver.core.exceptions.UnavailableException: Not
>>>>>>>> enough replica available for query at consistency LOCAL_ONE (1 required but
>>>>>>>> only 0 alive)*
>>>>>>>>
>>>>>>>>
>>>>>>>> I have a feeling that the issue is with datacenter name for some
>>>>>>>> reason, because in some config files I found DC name to be like DC1/DC2/DC3
>>>>>>>> in some it is like Cassandra/Analytics (The ones I had specified while
>>>>>>>> installation). Im unsure which yaml/property file to look for correct
>>>>>>>> inconsistency.
>>>>>>>>
>>>>>>>> (C*heers :) - im so tempted to copy that)
>>>>>>>>
>>>>>>>> Regards,
>>>>>>>> Bhuvan
>>>>>>>>
>>>>>>>> On Fri, Jan 22, 2016 at 8:47 PM, Alain RODRIGUEZ <
>>>>>>>> arodrime@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> The the exact command you ran (stress-tool with options) could be
>>>>>>>>> useful to help you on that.
>>>>>>>>>
>>>>>>>>> However, Im able to create keyspace, tables and insert data using
>>>>>>>>>> cqlsh and it is replicating fine to all the nodes.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Having the schema might be useful too.
>>>>>>>>>
>>>>>>>>> Did you ran the cqlsh and the stress-tool from the same server ?
>>>>>>>>> If not, you might want to check the port you use (9042/9160/...) are open.
>>>>>>>>> Also, cqlsh uses local_one by default too. If both commands were
>>>>>>>>> run against the same DC, from the same machine they should behave the same
>>>>>>>>> way. Are they ?
>>>>>>>>>
>>>>>>>>> C*heers,
>>>>>>>>>
>>>>>>>>> -----------------
>>>>>>>>> Alain
>>>>>>>>>
>>>>>>>>> The Last Pickle
>>>>>>>>> http://www.thelastpickle.com
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> 2016-01-22 9:57 GMT+01:00 Bhuvan Rawal <bh...@gmail.com>:
>>>>>>>>>
>>>>>>>>>> Hi,
>>>>>>>>>>
>>>>>>>>>> i have created a POC cluster with 2 DC , each having 4 nodes with
>>>>>>>>>> DSE 4.8.1 installed.
>>>>>>>>>>
>>>>>>>>>> On issuing cassandra stress im getting an error  and data is not
>>>>>>>>>> being inserted:
>>>>>>>>>> *com.datastax.driver.core.exceptions.UnavailableException: Not
>>>>>>>>>> enough replica available for query at consistency LOCAL_ONE (1 required but
>>>>>>>>>> only 0 alive)*
>>>>>>>>>>
>>>>>>>>>> However, Im able to create keyspace, tables and insert data using
>>>>>>>>>> cqlsh and it is replicating fine to all the nodes.
>>>>>>>>>>
>>>>>>>>>> Details of the cluster can be found below (all the nodes seem to
>>>>>>>>>> be alive and kicking):
>>>>>>>>>>
>>>>>>>>>> $ nodetool status Datacenter: Analytics =====================
>>>>>>>>>> Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address
>>>>>>>>>> Load Tokens Owns Host ID Rack UN 10.41.55.17 428.5 KB 256 ?
>>>>>>>>>> 39d6d585-e641-4046-9d0b-797356597b5e rack1 UN 10.41.55.19 404.44 KB 256 ?
>>>>>>>>>> 69edf930-efd9-4d74-a798-f3d4ac02e516 rack1 UN 10.41.55.18 423.21 KB 256 ?
>>>>>>>>>> b74bab13-09b2-4760-bce9-c8ef05e50f6d rack1 UN 10.41.55.20 683.23 KB 256 ?
>>>>>>>>>> fb5c4fed-6e1e-4ea8-838d-358106906830 rack1 Datacenter: Cassandra
>>>>>>>>>> ===================== Status=Up/Down |/
>>>>>>>>>> State=Normal/Leaving/Joining/Moving -- Address Load Tokens Owns
>>>>>>>>>> Host ID Rack UN 10.41.55.15 209.4 KB 256 ?
>>>>>>>>>> ffc3b9a0-5d5c-4a3d-a99e-49d255731278 rack1 UN 10.41.55.21 227.44 KB 256 ?
>>>>>>>>>> c68deba4-b9a2-43fc-bb13-6af74c88c210 rack1 UN 10.41.55.23 222.71 KB 256 ?
>>>>>>>>>> 8229aa87-af00-48fa-ad6b-3066d3dc0e58 rack1 UN 10.41.55.22 218.72 KB 256 ?
>>>>>>>>>> c7ba84fd-7992-41de-8c88-11574a72db99 rack1
>>>>>>>>>>
>>>>>>>>>> Regards,
>>>>>>>>>> Bhuvan Rawal
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>

Re: Getting error while issuing Cassandra stress

Posted by Sebastian Estevez <se...@datastax.com>.

https://github.com/brianmhess/cassandra-loader

All the best,


[image: datastax_logo.png] <http://www.datastax.com/>

Sebastián Estévez

Solutions Architect | 954 905 8615 | sebastian.estevez@datastax.com

[image: linkedin.png] <https://www.linkedin.com/company/datastax> [image:
facebook.png] <https://www.facebook.com/datastax> [image: twitter.png]
<https://twitter.com/datastax> [image: g+.png]
<https://plus.google.com/+Datastax/about>
<http://feeds.feedburner.com/datastax>
<http://goog_410786983>


<http://www.datastax.com/gartner-magic-quadrant-odbms>

DataStax is the fastest, most scalable distributed database technology,
delivering Apache Cassandra to the world’s most innovative enterprises.
Datastax is built to be agile, always-on, and predictably scalable to any
size. With more than 500 customers in 45 countries, DataStax is the
database technology and transactional backbone of choice for the worlds
most innovative companies such as Netflix, Adobe, Intuit, and eBay.

On Fri, Jan 22, 2016 at 4:37 PM, Bhuvan Rawal <bh...@gmail.com> wrote:

> Yes im specifying -node parameter to stress, otherwise it throws network
> connection failed.
>
> Can you point me to a sample java application to test pushing data from
> external server? Let's see if that works
>
> On Sat, Jan 23, 2016 at 2:55 AM, Sebastian Estevez <
> sebastian.estevez@datastax.com> wrote:
>
>> when i opened my cassandra-rackdc.properties i saw that DC names were DC1
>>> & DC2, rack name was RAC1 . Please note that this is the default
>>> configuration, I have not modified any file.
>>
>>
>> cassandra-rackdc.properties is only respected based on your snitch
>> <https://docs.datastax.com/en/cassandra/2.1/cassandra/architecture/architectureSnitchesAbout_c.html>
>> .
>>
>> $ cqlsh
>>> Connection error: ('Unable to connect to any servers', {'127.0.0.1':
>>> error(111, "Tried connecting to [('127.0.0.1', 9042)]. Last error:
>>> Connection refused")})
>>> whereas
>>> $ cqlsh <ip>
>>> works fine
>>> is that the reason why the cassandra-stress is not able to communicate
>>> with other replicas?
>>
>>
>> Are you providing the -node parameter to stress
>> <http://docs.datastax.com/en/cassandra/2.1/cassandra/tools/toolsCStress_t.html>
>> ?
>>
>>
>>
>> All the best,
>>
>>
>> [image: datastax_logo.png] <http://www.datastax.com/>
>>
>> Sebastián Estévez
>>
>> Solutions Architect | 954 905 8615 | sebastian.estevez@datastax.com
>>
>> [image: linkedin.png] <https://www.linkedin.com/company/datastax> [image:
>> facebook.png] <https://www.facebook.com/datastax> [image: twitter.png]
>> <https://twitter.com/datastax> [image: g+.png]
>> <https://plus.google.com/+Datastax/about>
>> <http://feeds.feedburner.com/datastax>
>> <http://goog_410786983>
>>
>>
>> <http://www.datastax.com/gartner-magic-quadrant-odbms>
>>
>> DataStax is the fastest, most scalable distributed database technology,
>> delivering Apache Cassandra to the world’s most innovative enterprises.
>> Datastax is built to be agile, always-on, and predictably scalable to any
>> size. With more than 500 customers in 45 countries, DataStax is the
>> database technology and transactional backbone of choice for the worlds
>> most innovative companies such as Netflix, Adobe, Intuit, and eBay.
>>
>> On Fri, Jan 22, 2016 at 4:07 PM, Bhuvan Rawal <bh...@gmail.com>
>> wrote:
>>
>>> I had a look at the jira below:
>>> https://issues.apache.org/jira/browse/CASSANDRA-7905
>>>
>>> when i opened my cassandra-rackdc.properties i saw that DC names were
>>> DC1 & DC2, rack name was RAC1 . Please note that this is the default
>>> configuration, I have not modified any file.
>>>
>>> There is another point of concern here which might be relevant to
>>> previous one as well, im not able to login to cqlsh directly, i.e. I have
>>> to specify ip as well even when im logged in to that machine.
>>>
>>> $ cqlsh
>>> Connection error: ('Unable to connect to any servers', {'127.0.0.1':
>>> error(111, "Tried connecting to [('127.0.0.1', 9042)]. Last error:
>>> Connection refused")})
>>>
>>> whereas
>>> $ cqlsh <ip>
>>> works fine
>>>
>>> is that the reason why the cassandra-stress is not able to communicate
>>> with other replicas?
>>>
>>> On Sat, Jan 23, 2016 at 1:37 AM, Sebastian Estevez <
>>> sebastian.estevez@datastax.com> wrote:
>>>
>>>> Sorry I missed that.
>>>>
>>>> Both your nodetool status and keyspace replication settings say
>>>> Cassandra and Analytics for the DC names. I'm not sure where you're seeing
>>>> DC1, DC2, etc. and why you suspect that is the problem.
>>>>
>>>> All the best,
>>>>
>>>>
>>>> [image: datastax_logo.png] <http://www.datastax.com/>
>>>>
>>>> Sebastián Estévez
>>>>
>>>> Solutions Architect | 954 905 8615 | sebastian.estevez@datastax.com
>>>>
>>>> [image: linkedin.png] <https://www.linkedin.com/company/datastax> [image:
>>>> facebook.png] <https://www.facebook.com/datastax> [image: twitter.png]
>>>> <https://twitter.com/datastax> [image: g+.png]
>>>> <https://plus.google.com/+Datastax/about>
>>>> <http://feeds.feedburner.com/datastax>
>>>> <http://goog_410786983>
>>>>
>>>>
>>>> <http://www.datastax.com/gartner-magic-quadrant-odbms>
>>>>
>>>> DataStax is the fastest, most scalable distributed database
>>>> technology, delivering Apache Cassandra to the world’s most innovative
>>>> enterprises. Datastax is built to be agile, always-on, and predictably
>>>> scalable to any size. With more than 500 customers in 45 countries, DataStax
>>>> is the database technology and transactional backbone of choice for the
>>>> worlds most innovative companies such as Netflix, Adobe, Intuit, and eBay.
>>>>
>>>> On Fri, Jan 22, 2016 at 1:45 PM, Bhuvan Rawal <bh...@gmail.com>
>>>> wrote:
>>>>
>>>>> Hi Sebastian,
>>>>>
>>>>> I had attached nodetool status output in previous mail, pasting it
>>>>> again :
>>>>>
>>>>> $ nodetool status Datacenter: Analytics =====================
>>>>> Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address Load
>>>>> Tokens Owns Host ID Rack UN 10.41.55.17 428.5 KB 256 ?
>>>>> 39d6d585-e641-4046-9d0b-797356597b5e rack1 UN 10.41.55.19 404.44 KB 256 ?
>>>>> 69edf930-efd9-4d74-a798-f3d4ac02e516 rack1 UN 10.41.55.18 423.21 KB 256 ?
>>>>> b74bab13-09b2-4760-bce9-c8ef05e50f6d rack1 UN 10.41.55.20 683.23 KB 256 ?
>>>>> fb5c4fed-6e1e-4ea8-838d-358106906830 rack1 Datacenter: Cassandra
>>>>> ===================== Status=Up/Down |/
>>>>> State=Normal/Leaving/Joining/Moving -- Address Load Tokens Owns Host
>>>>> ID Rack UN 10.41.55.15 209.4 KB 256 ? ffc3b9a0-5d5c-4a3d-a99e-49d255731278
>>>>> rack1 UN 10.41.55.21 227.44 KB 256 ? c68deba4-b9a2-43fc-bb13-6af74c88c210
>>>>> rack1 UN 10.41.55.23 222.71 KB 256 ? 8229aa87-af00-48fa-ad6b-3066d3dc0e58
>>>>> rack1 UN 10.41.55.22 218.72 KB 256 ? c7ba84fd-7992-41de-8c88-11574a72db99
>>>>> rack1
>>>>>
>>>>> Regards,
>>>>> Bhuvan Rawal
>>>>>
>>>>> On Sat, Jan 23, 2016 at 12:11 AM, Sebastian Estevez <
>>>>> sebastian.estevez@datastax.com> wrote:
>>>>>
>>>>>> The output of `nodetool status` would help us diagnose.
>>>>>>
>>>>>> All the best,
>>>>>>
>>>>>>
>>>>>> [image: datastax_logo.png] <http://www.datastax.com/>
>>>>>>
>>>>>> Sebastián Estévez
>>>>>>
>>>>>> Solutions Architect | 954 905 8615 | sebastian.estevez@datastax.com
>>>>>>
>>>>>> [image: linkedin.png] <https://www.linkedin.com/company/datastax> [image:
>>>>>> facebook.png] <https://www.facebook.com/datastax> [image:
>>>>>> twitter.png] <https://twitter.com/datastax> [image: g+.png]
>>>>>> <https://plus.google.com/+Datastax/about>
>>>>>> <http://feeds.feedburner.com/datastax>
>>>>>> <http://goog_410786983>
>>>>>>
>>>>>>
>>>>>> <http://www.datastax.com/gartner-magic-quadrant-odbms>
>>>>>>
>>>>>> DataStax is the fastest, most scalable distributed database
>>>>>> technology, delivering Apache Cassandra to the world’s most innovative
>>>>>> enterprises. Datastax is built to be agile, always-on, and predictably
>>>>>> scalable to any size. With more than 500 customers in 45 countries, DataStax
>>>>>> is the database technology and transactional backbone of choice for the
>>>>>> worlds most innovative companies such as Netflix, Adobe, Intuit, and eBay.
>>>>>>
>>>>>> On Fri, Jan 22, 2016 at 1:39 PM, Bhuvan Rawal <bh...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Thanks for the response Alain,
>>>>>>>
>>>>>>> cqlsh> create keyspace mykeyspace WITH replication =
>>>>>>> {'class':'NetworkTopologyStrategy', 'Analytics':2, 'Cassandra':3}
>>>>>>> cqlsh> use mykeyspace;
>>>>>>> cqlsh:mykeyspace>create table mytable (id int primary key, name
>>>>>>> text, address text, phone text);
>>>>>>> cqlsh:mykeyspace> insert into mytable (id, name, address, phone)
>>>>>>> values (1, 'Kiyu','Texas', '555-1212'); # and other similar statement
>>>>>>> I then issued the below command from every node and found consistent
>>>>>>> results.
>>>>>>> cqlsh:mykeyspace> select * from mytable;
>>>>>>>
>>>>>>> // Then i repeated the above steps for NetworkTopologyStrategy and
>>>>>>> found same results
>>>>>>>
>>>>>>> I ran basic cassandra stress
>>>>>>> seed1 - seed of datacenter 1
>>>>>>>  $ cassandra-stress write n=50000 -rate threads=4 -node any_random_ip
>>>>>>>  $ cassandra-stress write n=50000 -rate threads=4 -node seed1
>>>>>>>  $ cassandra-stress write n=50000 -rate threads=4 -node seed1,seed2
>>>>>>>  $ cassandra-stress write n=50000 -rate threads=4 -node
>>>>>>> all_8_ip_comma_seperated
>>>>>>>  $ cassandra-stress write n=1000000 cl=one -mode native cql3 -schema
>>>>>>> keyspace="keyspace1" -pop seq=1..1000000 -node ip1,ip2,ip3,ip4
>>>>>>>
>>>>>>> All of them threw the exception
>>>>>>> *com.datastax.driver.core.exceptions.UnavailableException: Not
>>>>>>> enough replica available for query at consistency LOCAL_ONE (1 required but
>>>>>>> only 0 alive)*
>>>>>>>
>>>>>>>
>>>>>>> I have a feeling that the issue is with datacenter name for some
>>>>>>> reason, because in some config files I found DC name to be like DC1/DC2/DC3
>>>>>>> in some it is like Cassandra/Analytics (The ones I had specified while
>>>>>>> installation). Im unsure which yaml/property file to look for correct
>>>>>>> inconsistency.
>>>>>>>
>>>>>>> (C*heers :) - im so tempted to copy that)
>>>>>>>
>>>>>>> Regards,
>>>>>>> Bhuvan
>>>>>>>
>>>>>>> On Fri, Jan 22, 2016 at 8:47 PM, Alain RODRIGUEZ <arodrime@gmail.com
>>>>>>> > wrote:
>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> The the exact command you ran (stress-tool with options) could be
>>>>>>>> useful to help you on that.
>>>>>>>>
>>>>>>>> However, Im able to create keyspace, tables and insert data using
>>>>>>>>> cqlsh and it is replicating fine to all the nodes.
>>>>>>>>
>>>>>>>>
>>>>>>>> Having the schema might be useful too.
>>>>>>>>
>>>>>>>> Did you ran the cqlsh and the stress-tool from the same server ?
>>>>>>>> If not, you might want to check the port you use (9042/9160/...) are open.
>>>>>>>> Also, cqlsh uses local_one by default too. If both commands were
>>>>>>>> run against the same DC, from the same machine they should behave the same
>>>>>>>> way. Are they ?
>>>>>>>>
>>>>>>>> C*heers,
>>>>>>>>
>>>>>>>> -----------------
>>>>>>>> Alain
>>>>>>>>
>>>>>>>> The Last Pickle
>>>>>>>> http://www.thelastpickle.com
>>>>>>>>
>>>>>>>>
>>>>>>>> 2016-01-22 9:57 GMT+01:00 Bhuvan Rawal <bh...@gmail.com>:
>>>>>>>>
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> i have created a POC cluster with 2 DC , each having 4 nodes with
>>>>>>>>> DSE 4.8.1 installed.
>>>>>>>>>
>>>>>>>>> On issuing cassandra stress im getting an error  and data is not
>>>>>>>>> being inserted:
>>>>>>>>> *com.datastax.driver.core.exceptions.UnavailableException: Not
>>>>>>>>> enough replica available for query at consistency LOCAL_ONE (1 required but
>>>>>>>>> only 0 alive)*
>>>>>>>>>
>>>>>>>>> However, Im able to create keyspace, tables and insert data using
>>>>>>>>> cqlsh and it is replicating fine to all the nodes.
>>>>>>>>>
>>>>>>>>> Details of the cluster can be found below (all the nodes seem to
>>>>>>>>> be alive and kicking):
>>>>>>>>>
>>>>>>>>> $ nodetool status Datacenter: Analytics =====================
>>>>>>>>> Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address
>>>>>>>>> Load Tokens Owns Host ID Rack UN 10.41.55.17 428.5 KB 256 ?
>>>>>>>>> 39d6d585-e641-4046-9d0b-797356597b5e rack1 UN 10.41.55.19 404.44 KB 256 ?
>>>>>>>>> 69edf930-efd9-4d74-a798-f3d4ac02e516 rack1 UN 10.41.55.18 423.21 KB 256 ?
>>>>>>>>> b74bab13-09b2-4760-bce9-c8ef05e50f6d rack1 UN 10.41.55.20 683.23 KB 256 ?
>>>>>>>>> fb5c4fed-6e1e-4ea8-838d-358106906830 rack1 Datacenter: Cassandra
>>>>>>>>> ===================== Status=Up/Down |/
>>>>>>>>> State=Normal/Leaving/Joining/Moving -- Address Load Tokens Owns
>>>>>>>>> Host ID Rack UN 10.41.55.15 209.4 KB 256 ?
>>>>>>>>> ffc3b9a0-5d5c-4a3d-a99e-49d255731278 rack1 UN 10.41.55.21 227.44 KB 256 ?
>>>>>>>>> c68deba4-b9a2-43fc-bb13-6af74c88c210 rack1 UN 10.41.55.23 222.71 KB 256 ?
>>>>>>>>> 8229aa87-af00-48fa-ad6b-3066d3dc0e58 rack1 UN 10.41.55.22 218.72 KB 256 ?
>>>>>>>>> c7ba84fd-7992-41de-8c88-11574a72db99 rack1
>>>>>>>>>
>>>>>>>>> Regards,
>>>>>>>>> Bhuvan Rawal
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>

Re: Getting error while issuing Cassandra stress

Posted by Bhuvan Rawal <bh...@gmail.com>.

Yes im specifying -node parameter to stress, otherwise it throws network
connection failed.

Can you point me to a sample java application to test pushing data from
external server? Let's see if that works

On Sat, Jan 23, 2016 at 2:55 AM, Sebastian Estevez <
sebastian.estevez@datastax.com> wrote:

> when i opened my cassandra-rackdc.properties i saw that DC names were DC1
>> & DC2, rack name was RAC1 . Please note that this is the default
>> configuration, I have not modified any file.
>
>
> cassandra-rackdc.properties is only respected based on your snitch
> <https://docs.datastax.com/en/cassandra/2.1/cassandra/architecture/architectureSnitchesAbout_c.html>
> .
>
> $ cqlsh
>> Connection error: ('Unable to connect to any servers', {'127.0.0.1':
>> error(111, "Tried connecting to [('127.0.0.1', 9042)]. Last error:
>> Connection refused")})
>> whereas
>> $ cqlsh <ip>
>> works fine
>> is that the reason why the cassandra-stress is not able to communicate
>> with other replicas?
>
>
> Are you providing the -node parameter to stress
> <http://docs.datastax.com/en/cassandra/2.1/cassandra/tools/toolsCStress_t.html>
> ?
>
>
>
> All the best,
>
>
> [image: datastax_logo.png] <http://www.datastax.com/>
>
> Sebastián Estévez
>
> Solutions Architect | 954 905 8615 | sebastian.estevez@datastax.com
>
> [image: linkedin.png] <https://www.linkedin.com/company/datastax> [image:
> facebook.png] <https://www.facebook.com/datastax> [image: twitter.png]
> <https://twitter.com/datastax> [image: g+.png]
> <https://plus.google.com/+Datastax/about>
> <http://feeds.feedburner.com/datastax>
> <http://goog_410786983>
>
>
> <http://www.datastax.com/gartner-magic-quadrant-odbms>
>
> DataStax is the fastest, most scalable distributed database technology,
> delivering Apache Cassandra to the world’s most innovative enterprises.
> Datastax is built to be agile, always-on, and predictably scalable to any
> size. With more than 500 customers in 45 countries, DataStax is the
> database technology and transactional backbone of choice for the worlds
> most innovative companies such as Netflix, Adobe, Intuit, and eBay.
>
> On Fri, Jan 22, 2016 at 4:07 PM, Bhuvan Rawal <bh...@gmail.com> wrote:
>
>> I had a look at the jira below:
>> https://issues.apache.org/jira/browse/CASSANDRA-7905
>>
>> when i opened my cassandra-rackdc.properties i saw that DC names were DC1
>> & DC2, rack name was RAC1 . Please note that this is the default
>> configuration, I have not modified any file.
>>
>> There is another point of concern here which might be relevant to
>> previous one as well, im not able to login to cqlsh directly, i.e. I have
>> to specify ip as well even when im logged in to that machine.
>>
>> $ cqlsh
>> Connection error: ('Unable to connect to any servers', {'127.0.0.1':
>> error(111, "Tried connecting to [('127.0.0.1', 9042)]. Last error:
>> Connection refused")})
>>
>> whereas
>> $ cqlsh <ip>
>> works fine
>>
>> is that the reason why the cassandra-stress is not able to communicate
>> with other replicas?
>>
>> On Sat, Jan 23, 2016 at 1:37 AM, Sebastian Estevez <
>> sebastian.estevez@datastax.com> wrote:
>>
>>> Sorry I missed that.
>>>
>>> Both your nodetool status and keyspace replication settings say
>>> Cassandra and Analytics for the DC names. I'm not sure where you're seeing
>>> DC1, DC2, etc. and why you suspect that is the problem.
>>>
>>> All the best,
>>>
>>>
>>> [image: datastax_logo.png] <http://www.datastax.com/>
>>>
>>> Sebastián Estévez
>>>
>>> Solutions Architect | 954 905 8615 | sebastian.estevez@datastax.com
>>>
>>> [image: linkedin.png] <https://www.linkedin.com/company/datastax> [image:
>>> facebook.png] <https://www.facebook.com/datastax> [image: twitter.png]
>>> <https://twitter.com/datastax> [image: g+.png]
>>> <https://plus.google.com/+Datastax/about>
>>> <http://feeds.feedburner.com/datastax>
>>> <http://goog_410786983>
>>>
>>>
>>> <http://www.datastax.com/gartner-magic-quadrant-odbms>
>>>
>>> DataStax is the fastest, most scalable distributed database technology,
>>> delivering Apache Cassandra to the world’s most innovative enterprises.
>>> Datastax is built to be agile, always-on, and predictably scalable to any
>>> size. With more than 500 customers in 45 countries, DataStax is the
>>> database technology and transactional backbone of choice for the worlds
>>> most innovative companies such as Netflix, Adobe, Intuit, and eBay.
>>>
>>> On Fri, Jan 22, 2016 at 1:45 PM, Bhuvan Rawal <bh...@gmail.com>
>>> wrote:
>>>
>>>> Hi Sebastian,
>>>>
>>>> I had attached nodetool status output in previous mail, pasting it
>>>> again :
>>>>
>>>> $ nodetool status Datacenter: Analytics =====================
>>>> Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address Load
>>>> Tokens Owns Host ID Rack UN 10.41.55.17 428.5 KB 256 ?
>>>> 39d6d585-e641-4046-9d0b-797356597b5e rack1 UN 10.41.55.19 404.44 KB 256 ?
>>>> 69edf930-efd9-4d74-a798-f3d4ac02e516 rack1 UN 10.41.55.18 423.21 KB 256 ?
>>>> b74bab13-09b2-4760-bce9-c8ef05e50f6d rack1 UN 10.41.55.20 683.23 KB 256 ?
>>>> fb5c4fed-6e1e-4ea8-838d-358106906830 rack1 Datacenter: Cassandra
>>>> ===================== Status=Up/Down |/
>>>> State=Normal/Leaving/Joining/Moving -- Address Load Tokens Owns Host
>>>> ID Rack UN 10.41.55.15 209.4 KB 256 ? ffc3b9a0-5d5c-4a3d-a99e-49d255731278
>>>> rack1 UN 10.41.55.21 227.44 KB 256 ? c68deba4-b9a2-43fc-bb13-6af74c88c210
>>>> rack1 UN 10.41.55.23 222.71 KB 256 ? 8229aa87-af00-48fa-ad6b-3066d3dc0e58
>>>> rack1 UN 10.41.55.22 218.72 KB 256 ? c7ba84fd-7992-41de-8c88-11574a72db99
>>>> rack1
>>>>
>>>> Regards,
>>>> Bhuvan Rawal
>>>>
>>>> On Sat, Jan 23, 2016 at 12:11 AM, Sebastian Estevez <
>>>> sebastian.estevez@datastax.com> wrote:
>>>>
>>>>> The output of `nodetool status` would help us diagnose.
>>>>>
>>>>> All the best,
>>>>>
>>>>>
>>>>> [image: datastax_logo.png] <http://www.datastax.com/>
>>>>>
>>>>> Sebastián Estévez
>>>>>
>>>>> Solutions Architect | 954 905 8615 | sebastian.estevez@datastax.com
>>>>>
>>>>> [image: linkedin.png] <https://www.linkedin.com/company/datastax> [image:
>>>>> facebook.png] <https://www.facebook.com/datastax> [image: twitter.png]
>>>>> <https://twitter.com/datastax> [image: g+.png]
>>>>> <https://plus.google.com/+Datastax/about>
>>>>> <http://feeds.feedburner.com/datastax>
>>>>> <http://goog_410786983>
>>>>>
>>>>>
>>>>> <http://www.datastax.com/gartner-magic-quadrant-odbms>
>>>>>
>>>>> DataStax is the fastest, most scalable distributed database
>>>>> technology, delivering Apache Cassandra to the world’s most innovative
>>>>> enterprises. Datastax is built to be agile, always-on, and predictably
>>>>> scalable to any size. With more than 500 customers in 45 countries, DataStax
>>>>> is the database technology and transactional backbone of choice for the
>>>>> worlds most innovative companies such as Netflix, Adobe, Intuit, and eBay.
>>>>>
>>>>> On Fri, Jan 22, 2016 at 1:39 PM, Bhuvan Rawal <bh...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Thanks for the response Alain,
>>>>>>
>>>>>> cqlsh> create keyspace mykeyspace WITH replication =
>>>>>> {'class':'NetworkTopologyStrategy', 'Analytics':2, 'Cassandra':3}
>>>>>> cqlsh> use mykeyspace;
>>>>>> cqlsh:mykeyspace>create table mytable (id int primary key, name text,
>>>>>> address text, phone text);
>>>>>> cqlsh:mykeyspace> insert into mytable (id, name, address, phone)
>>>>>> values (1, 'Kiyu','Texas', '555-1212'); # and other similar statement
>>>>>> I then issued the below command from every node and found consistent
>>>>>> results.
>>>>>> cqlsh:mykeyspace> select * from mytable;
>>>>>>
>>>>>> // Then i repeated the above steps for NetworkTopologyStrategy and
>>>>>> found same results
>>>>>>
>>>>>> I ran basic cassandra stress
>>>>>> seed1 - seed of datacenter 1
>>>>>>  $ cassandra-stress write n=50000 -rate threads=4 -node any_random_ip
>>>>>>  $ cassandra-stress write n=50000 -rate threads=4 -node seed1
>>>>>>  $ cassandra-stress write n=50000 -rate threads=4 -node seed1,seed2
>>>>>>  $ cassandra-stress write n=50000 -rate threads=4 -node
>>>>>> all_8_ip_comma_seperated
>>>>>>  $ cassandra-stress write n=1000000 cl=one -mode native cql3 -schema
>>>>>> keyspace="keyspace1" -pop seq=1..1000000 -node ip1,ip2,ip3,ip4
>>>>>>
>>>>>> All of them threw the exception
>>>>>> *com.datastax.driver.core.exceptions.UnavailableException: Not enough
>>>>>> replica available for query at consistency LOCAL_ONE (1 required but only 0
>>>>>> alive)*
>>>>>>
>>>>>>
>>>>>> I have a feeling that the issue is with datacenter name for some
>>>>>> reason, because in some config files I found DC name to be like DC1/DC2/DC3
>>>>>> in some it is like Cassandra/Analytics (The ones I had specified while
>>>>>> installation). Im unsure which yaml/property file to look for correct
>>>>>> inconsistency.
>>>>>>
>>>>>> (C*heers :) - im so tempted to copy that)
>>>>>>
>>>>>> Regards,
>>>>>> Bhuvan
>>>>>>
>>>>>> On Fri, Jan 22, 2016 at 8:47 PM, Alain RODRIGUEZ <ar...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> The the exact command you ran (stress-tool with options) could be
>>>>>>> useful to help you on that.
>>>>>>>
>>>>>>> However, Im able to create keyspace, tables and insert data using
>>>>>>>> cqlsh and it is replicating fine to all the nodes.
>>>>>>>
>>>>>>>
>>>>>>> Having the schema might be useful too.
>>>>>>>
>>>>>>> Did you ran the cqlsh and the stress-tool from the same server ? If
>>>>>>> not, you might want to check the port you use (9042/9160/...) are open.
>>>>>>> Also, cqlsh uses local_one by default too. If both commands were run
>>>>>>> against the same DC, from the same machine they should behave the same way.
>>>>>>> Are they ?
>>>>>>>
>>>>>>> C*heers,
>>>>>>>
>>>>>>> -----------------
>>>>>>> Alain
>>>>>>>
>>>>>>> The Last Pickle
>>>>>>> http://www.thelastpickle.com
>>>>>>>
>>>>>>>
>>>>>>> 2016-01-22 9:57 GMT+01:00 Bhuvan Rawal <bh...@gmail.com>:
>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> i have created a POC cluster with 2 DC , each having 4 nodes with
>>>>>>>> DSE 4.8.1 installed.
>>>>>>>>
>>>>>>>> On issuing cassandra stress im getting an error  and data is not
>>>>>>>> being inserted:
>>>>>>>> *com.datastax.driver.core.exceptions.UnavailableException: Not
>>>>>>>> enough replica available for query at consistency LOCAL_ONE (1 required but
>>>>>>>> only 0 alive)*
>>>>>>>>
>>>>>>>> However, Im able to create keyspace, tables and insert data using
>>>>>>>> cqlsh and it is replicating fine to all the nodes.
>>>>>>>>
>>>>>>>> Details of the cluster can be found below (all the nodes seem to be
>>>>>>>> alive and kicking):
>>>>>>>>
>>>>>>>> $ nodetool status Datacenter: Analytics =====================
>>>>>>>> Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address
>>>>>>>> Load Tokens Owns Host ID Rack UN 10.41.55.17 428.5 KB 256 ?
>>>>>>>> 39d6d585-e641-4046-9d0b-797356597b5e rack1 UN 10.41.55.19 404.44 KB 256 ?
>>>>>>>> 69edf930-efd9-4d74-a798-f3d4ac02e516 rack1 UN 10.41.55.18 423.21 KB 256 ?
>>>>>>>> b74bab13-09b2-4760-bce9-c8ef05e50f6d rack1 UN 10.41.55.20 683.23 KB 256 ?
>>>>>>>> fb5c4fed-6e1e-4ea8-838d-358106906830 rack1 Datacenter: Cassandra
>>>>>>>> ===================== Status=Up/Down |/
>>>>>>>> State=Normal/Leaving/Joining/Moving -- Address Load Tokens Owns
>>>>>>>> Host ID Rack UN 10.41.55.15 209.4 KB 256 ?
>>>>>>>> ffc3b9a0-5d5c-4a3d-a99e-49d255731278 rack1 UN 10.41.55.21 227.44 KB 256 ?
>>>>>>>> c68deba4-b9a2-43fc-bb13-6af74c88c210 rack1 UN 10.41.55.23 222.71 KB 256 ?
>>>>>>>> 8229aa87-af00-48fa-ad6b-3066d3dc0e58 rack1 UN 10.41.55.22 218.72 KB 256 ?
>>>>>>>> c7ba84fd-7992-41de-8c88-11574a72db99 rack1
>>>>>>>>
>>>>>>>> Regards,
>>>>>>>> Bhuvan Rawal
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>

Re: Getting error while issuing Cassandra stress

Posted by Sebastian Estevez <se...@datastax.com>.

>
> when i opened my cassandra-rackdc.properties i saw that DC names were DC1
> & DC2, rack name was RAC1 . Please note that this is the default
> configuration, I have not modified any file.


cassandra-rackdc.properties is only respected based on your snitch
<https://docs.datastax.com/en/cassandra/2.1/cassandra/architecture/architectureSnitchesAbout_c.html>
.

$ cqlsh
> Connection error: ('Unable to connect to any servers', {'127.0.0.1':
> error(111, "Tried connecting to [('127.0.0.1', 9042)]. Last error:
> Connection refused")})
> whereas
> $ cqlsh <ip>
> works fine
> is that the reason why the cassandra-stress is not able to communicate
> with other replicas?


Are you providing the -node parameter to stress
<http://docs.datastax.com/en/cassandra/2.1/cassandra/tools/toolsCStress_t.html>
?



All the best,


[image: datastax_logo.png] <http://www.datastax.com/>

Sebastián Estévez

Solutions Architect | 954 905 8615 | sebastian.estevez@datastax.com

[image: linkedin.png] <https://www.linkedin.com/company/datastax> [image:
facebook.png] <https://www.facebook.com/datastax> [image: twitter.png]
<https://twitter.com/datastax> [image: g+.png]
<https://plus.google.com/+Datastax/about>
<http://feeds.feedburner.com/datastax>
<http://goog_410786983>


<http://www.datastax.com/gartner-magic-quadrant-odbms>

DataStax is the fastest, most scalable distributed database technology,
delivering Apache Cassandra to the world’s most innovative enterprises.
Datastax is built to be agile, always-on, and predictably scalable to any
size. With more than 500 customers in 45 countries, DataStax is the
database technology and transactional backbone of choice for the worlds
most innovative companies such as Netflix, Adobe, Intuit, and eBay.

On Fri, Jan 22, 2016 at 4:07 PM, Bhuvan Rawal <bh...@gmail.com> wrote:

> I had a look at the jira below:
> https://issues.apache.org/jira/browse/CASSANDRA-7905
>
> when i opened my cassandra-rackdc.properties i saw that DC names were DC1
> & DC2, rack name was RAC1 . Please note that this is the default
> configuration, I have not modified any file.
>
> There is another point of concern here which might be relevant to previous
> one as well, im not able to login to cqlsh directly, i.e. I have to specify
> ip as well even when im logged in to that machine.
>
> $ cqlsh
> Connection error: ('Unable to connect to any servers', {'127.0.0.1':
> error(111, "Tried connecting to [('127.0.0.1', 9042)]. Last error:
> Connection refused")})
>
> whereas
> $ cqlsh <ip>
> works fine
>
> is that the reason why the cassandra-stress is not able to communicate
> with other replicas?
>
> On Sat, Jan 23, 2016 at 1:37 AM, Sebastian Estevez <
> sebastian.estevez@datastax.com> wrote:
>
>> Sorry I missed that.
>>
>> Both your nodetool status and keyspace replication settings say Cassandra
>> and Analytics for the DC names. I'm not sure where you're seeing DC1, DC2,
>> etc. and why you suspect that is the problem.
>>
>> All the best,
>>
>>
>> [image: datastax_logo.png] <http://www.datastax.com/>
>>
>> Sebastián Estévez
>>
>> Solutions Architect | 954 905 8615 | sebastian.estevez@datastax.com
>>
>> [image: linkedin.png] <https://www.linkedin.com/company/datastax> [image:
>> facebook.png] <https://www.facebook.com/datastax> [image: twitter.png]
>> <https://twitter.com/datastax> [image: g+.png]
>> <https://plus.google.com/+Datastax/about>
>> <http://feeds.feedburner.com/datastax>
>> <http://goog_410786983>
>>
>>
>> <http://www.datastax.com/gartner-magic-quadrant-odbms>
>>
>> DataStax is the fastest, most scalable distributed database technology,
>> delivering Apache Cassandra to the world’s most innovative enterprises.
>> Datastax is built to be agile, always-on, and predictably scalable to any
>> size. With more than 500 customers in 45 countries, DataStax is the
>> database technology and transactional backbone of choice for the worlds
>> most innovative companies such as Netflix, Adobe, Intuit, and eBay.
>>
>> On Fri, Jan 22, 2016 at 1:45 PM, Bhuvan Rawal <bh...@gmail.com>
>> wrote:
>>
>>> Hi Sebastian,
>>>
>>> I had attached nodetool status output in previous mail, pasting it again
>>> :
>>>
>>> $ nodetool status Datacenter: Analytics =====================
>>> Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address Load
>>> Tokens Owns Host ID Rack UN 10.41.55.17 428.5 KB 256 ?
>>> 39d6d585-e641-4046-9d0b-797356597b5e rack1 UN 10.41.55.19 404.44 KB 256 ?
>>> 69edf930-efd9-4d74-a798-f3d4ac02e516 rack1 UN 10.41.55.18 423.21 KB 256 ?
>>> b74bab13-09b2-4760-bce9-c8ef05e50f6d rack1 UN 10.41.55.20 683.23 KB 256 ?
>>> fb5c4fed-6e1e-4ea8-838d-358106906830 rack1 Datacenter: Cassandra
>>> ===================== Status=Up/Down |/
>>> State=Normal/Leaving/Joining/Moving -- Address Load Tokens Owns Host ID
>>> Rack UN 10.41.55.15 209.4 KB 256 ? ffc3b9a0-5d5c-4a3d-a99e-49d255731278
>>> rack1 UN 10.41.55.21 227.44 KB 256 ? c68deba4-b9a2-43fc-bb13-6af74c88c210
>>> rack1 UN 10.41.55.23 222.71 KB 256 ? 8229aa87-af00-48fa-ad6b-3066d3dc0e58
>>> rack1 UN 10.41.55.22 218.72 KB 256 ? c7ba84fd-7992-41de-8c88-11574a72db99
>>> rack1
>>>
>>> Regards,
>>> Bhuvan Rawal
>>>
>>> On Sat, Jan 23, 2016 at 12:11 AM, Sebastian Estevez <
>>> sebastian.estevez@datastax.com> wrote:
>>>
>>>> The output of `nodetool status` would help us diagnose.
>>>>
>>>> All the best,
>>>>
>>>>
>>>> [image: datastax_logo.png] <http://www.datastax.com/>
>>>>
>>>> Sebastián Estévez
>>>>
>>>> Solutions Architect | 954 905 8615 | sebastian.estevez@datastax.com
>>>>
>>>> [image: linkedin.png] <https://www.linkedin.com/company/datastax> [image:
>>>> facebook.png] <https://www.facebook.com/datastax> [image: twitter.png]
>>>> <https://twitter.com/datastax> [image: g+.png]
>>>> <https://plus.google.com/+Datastax/about>
>>>> <http://feeds.feedburner.com/datastax>
>>>> <http://goog_410786983>
>>>>
>>>>
>>>> <http://www.datastax.com/gartner-magic-quadrant-odbms>
>>>>
>>>> DataStax is the fastest, most scalable distributed database
>>>> technology, delivering Apache Cassandra to the world’s most innovative
>>>> enterprises. Datastax is built to be agile, always-on, and predictably
>>>> scalable to any size. With more than 500 customers in 45 countries, DataStax
>>>> is the database technology and transactional backbone of choice for the
>>>> worlds most innovative companies such as Netflix, Adobe, Intuit, and eBay.
>>>>
>>>> On Fri, Jan 22, 2016 at 1:39 PM, Bhuvan Rawal <bh...@gmail.com>
>>>> wrote:
>>>>
>>>>> Thanks for the response Alain,
>>>>>
>>>>> cqlsh> create keyspace mykeyspace WITH replication =
>>>>> {'class':'NetworkTopologyStrategy', 'Analytics':2, 'Cassandra':3}
>>>>> cqlsh> use mykeyspace;
>>>>> cqlsh:mykeyspace>create table mytable (id int primary key, name text,
>>>>> address text, phone text);
>>>>> cqlsh:mykeyspace> insert into mytable (id, name, address, phone)
>>>>> values (1, 'Kiyu','Texas', '555-1212'); # and other similar statement
>>>>> I then issued the below command from every node and found consistent
>>>>> results.
>>>>> cqlsh:mykeyspace> select * from mytable;
>>>>>
>>>>> // Then i repeated the above steps for NetworkTopologyStrategy and
>>>>> found same results
>>>>>
>>>>> I ran basic cassandra stress
>>>>> seed1 - seed of datacenter 1
>>>>>  $ cassandra-stress write n=50000 -rate threads=4 -node any_random_ip
>>>>>  $ cassandra-stress write n=50000 -rate threads=4 -node seed1
>>>>>  $ cassandra-stress write n=50000 -rate threads=4 -node seed1,seed2
>>>>>  $ cassandra-stress write n=50000 -rate threads=4 -node
>>>>> all_8_ip_comma_seperated
>>>>>  $ cassandra-stress write n=1000000 cl=one -mode native cql3 -schema
>>>>> keyspace="keyspace1" -pop seq=1..1000000 -node ip1,ip2,ip3,ip4
>>>>>
>>>>> All of them threw the exception
>>>>> *com.datastax.driver.core.exceptions.UnavailableException: Not enough
>>>>> replica available for query at consistency LOCAL_ONE (1 required but only 0
>>>>> alive)*
>>>>>
>>>>>
>>>>> I have a feeling that the issue is with datacenter name for some
>>>>> reason, because in some config files I found DC name to be like DC1/DC2/DC3
>>>>> in some it is like Cassandra/Analytics (The ones I had specified while
>>>>> installation). Im unsure which yaml/property file to look for correct
>>>>> inconsistency.
>>>>>
>>>>> (C*heers :) - im so tempted to copy that)
>>>>>
>>>>> Regards,
>>>>> Bhuvan
>>>>>
>>>>> On Fri, Jan 22, 2016 at 8:47 PM, Alain RODRIGUEZ <ar...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> The the exact command you ran (stress-tool with options) could be
>>>>>> useful to help you on that.
>>>>>>
>>>>>> However, Im able to create keyspace, tables and insert data using
>>>>>>> cqlsh and it is replicating fine to all the nodes.
>>>>>>
>>>>>>
>>>>>> Having the schema might be useful too.
>>>>>>
>>>>>> Did you ran the cqlsh and the stress-tool from the same server ? If
>>>>>> not, you might want to check the port you use (9042/9160/...) are open.
>>>>>> Also, cqlsh uses local_one by default too. If both commands were run
>>>>>> against the same DC, from the same machine they should behave the same way.
>>>>>> Are they ?
>>>>>>
>>>>>> C*heers,
>>>>>>
>>>>>> -----------------
>>>>>> Alain
>>>>>>
>>>>>> The Last Pickle
>>>>>> http://www.thelastpickle.com
>>>>>>
>>>>>>
>>>>>> 2016-01-22 9:57 GMT+01:00 Bhuvan Rawal <bh...@gmail.com>:
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> i have created a POC cluster with 2 DC , each having 4 nodes with
>>>>>>> DSE 4.8.1 installed.
>>>>>>>
>>>>>>> On issuing cassandra stress im getting an error  and data is not
>>>>>>> being inserted:
>>>>>>> *com.datastax.driver.core.exceptions.UnavailableException: Not
>>>>>>> enough replica available for query at consistency LOCAL_ONE (1 required but
>>>>>>> only 0 alive)*
>>>>>>>
>>>>>>> However, Im able to create keyspace, tables and insert data using
>>>>>>> cqlsh and it is replicating fine to all the nodes.
>>>>>>>
>>>>>>> Details of the cluster can be found below (all the nodes seem to be
>>>>>>> alive and kicking):
>>>>>>>
>>>>>>> $ nodetool status Datacenter: Analytics =====================
>>>>>>> Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address
>>>>>>> Load Tokens Owns Host ID Rack UN 10.41.55.17 428.5 KB 256 ?
>>>>>>> 39d6d585-e641-4046-9d0b-797356597b5e rack1 UN 10.41.55.19 404.44 KB 256 ?
>>>>>>> 69edf930-efd9-4d74-a798-f3d4ac02e516 rack1 UN 10.41.55.18 423.21 KB 256 ?
>>>>>>> b74bab13-09b2-4760-bce9-c8ef05e50f6d rack1 UN 10.41.55.20 683.23 KB 256 ?
>>>>>>> fb5c4fed-6e1e-4ea8-838d-358106906830 rack1 Datacenter: Cassandra
>>>>>>> ===================== Status=Up/Down |/
>>>>>>> State=Normal/Leaving/Joining/Moving -- Address Load Tokens Owns
>>>>>>> Host ID Rack UN 10.41.55.15 209.4 KB 256 ?
>>>>>>> ffc3b9a0-5d5c-4a3d-a99e-49d255731278 rack1 UN 10.41.55.21 227.44 KB 256 ?
>>>>>>> c68deba4-b9a2-43fc-bb13-6af74c88c210 rack1 UN 10.41.55.23 222.71 KB 256 ?
>>>>>>> 8229aa87-af00-48fa-ad6b-3066d3dc0e58 rack1 UN 10.41.55.22 218.72 KB 256 ?
>>>>>>> c7ba84fd-7992-41de-8c88-11574a72db99 rack1
>>>>>>>
>>>>>>> Regards,
>>>>>>> Bhuvan Rawal
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>

Re: Getting error while issuing Cassandra stress

Posted by Bhuvan Rawal <bh...@gmail.com>.

I had a look at the jira below:
https://issues.apache.org/jira/browse/CASSANDRA-7905

when i opened my cassandra-rackdc.properties i saw that DC names were DC1 &
DC2, rack name was RAC1 . Please note that this is the default
configuration, I have not modified any file.

There is another point of concern here which might be relevant to previous
one as well, im not able to login to cqlsh directly, i.e. I have to specify
ip as well even when im logged in to that machine.

$ cqlsh
Connection error: ('Unable to connect to any servers', {'127.0.0.1':
error(111, "Tried connecting to [('127.0.0.1', 9042)]. Last error:
Connection refused")})

whereas
$ cqlsh <ip>
works fine

is that the reason why the cassandra-stress is not able to communicate with
other replicas?

On Sat, Jan 23, 2016 at 1:37 AM, Sebastian Estevez <
sebastian.estevez@datastax.com> wrote:

> Sorry I missed that.
>
> Both your nodetool status and keyspace replication settings say Cassandra
> and Analytics for the DC names. I'm not sure where you're seeing DC1, DC2,
> etc. and why you suspect that is the problem.
>
> All the best,
>
>
> [image: datastax_logo.png] <http://www.datastax.com/>
>
> Sebastián Estévez
>
> Solutions Architect | 954 905 8615 | sebastian.estevez@datastax.com
>
> [image: linkedin.png] <https://www.linkedin.com/company/datastax> [image:
> facebook.png] <https://www.facebook.com/datastax> [image: twitter.png]
> <https://twitter.com/datastax> [image: g+.png]
> <https://plus.google.com/+Datastax/about>
> <http://feeds.feedburner.com/datastax>
> <http://goog_410786983>
>
>
> <http://www.datastax.com/gartner-magic-quadrant-odbms>
>
> DataStax is the fastest, most scalable distributed database technology,
> delivering Apache Cassandra to the world’s most innovative enterprises.
> Datastax is built to be agile, always-on, and predictably scalable to any
> size. With more than 500 customers in 45 countries, DataStax is the
> database technology and transactional backbone of choice for the worlds
> most innovative companies such as Netflix, Adobe, Intuit, and eBay.
>
> On Fri, Jan 22, 2016 at 1:45 PM, Bhuvan Rawal <bh...@gmail.com> wrote:
>
>> Hi Sebastian,
>>
>> I had attached nodetool status output in previous mail, pasting it again :
>>
>> $ nodetool status Datacenter: Analytics =====================
>> Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address Load
>> Tokens Owns Host ID Rack UN 10.41.55.17 428.5 KB 256 ?
>> 39d6d585-e641-4046-9d0b-797356597b5e rack1 UN 10.41.55.19 404.44 KB 256 ?
>> 69edf930-efd9-4d74-a798-f3d4ac02e516 rack1 UN 10.41.55.18 423.21 KB 256 ?
>> b74bab13-09b2-4760-bce9-c8ef05e50f6d rack1 UN 10.41.55.20 683.23 KB 256 ?
>> fb5c4fed-6e1e-4ea8-838d-358106906830 rack1 Datacenter: Cassandra
>> ===================== Status=Up/Down |/
>> State=Normal/Leaving/Joining/Moving -- Address Load Tokens Owns Host ID
>> Rack UN 10.41.55.15 209.4 KB 256 ? ffc3b9a0-5d5c-4a3d-a99e-49d255731278
>> rack1 UN 10.41.55.21 227.44 KB 256 ? c68deba4-b9a2-43fc-bb13-6af74c88c210
>> rack1 UN 10.41.55.23 222.71 KB 256 ? 8229aa87-af00-48fa-ad6b-3066d3dc0e58
>> rack1 UN 10.41.55.22 218.72 KB 256 ? c7ba84fd-7992-41de-8c88-11574a72db99
>> rack1
>>
>> Regards,
>> Bhuvan Rawal
>>
>> On Sat, Jan 23, 2016 at 12:11 AM, Sebastian Estevez <
>> sebastian.estevez@datastax.com> wrote:
>>
>>> The output of `nodetool status` would help us diagnose.
>>>
>>> All the best,
>>>
>>>
>>> [image: datastax_logo.png] <http://www.datastax.com/>
>>>
>>> Sebastián Estévez
>>>
>>> Solutions Architect | 954 905 8615 | sebastian.estevez@datastax.com
>>>
>>> [image: linkedin.png] <https://www.linkedin.com/company/datastax> [image:
>>> facebook.png] <https://www.facebook.com/datastax> [image: twitter.png]
>>> <https://twitter.com/datastax> [image: g+.png]
>>> <https://plus.google.com/+Datastax/about>
>>> <http://feeds.feedburner.com/datastax>
>>> <http://goog_410786983>
>>>
>>>
>>> <http://www.datastax.com/gartner-magic-quadrant-odbms>
>>>
>>> DataStax is the fastest, most scalable distributed database technology,
>>> delivering Apache Cassandra to the world’s most innovative enterprises.
>>> Datastax is built to be agile, always-on, and predictably scalable to any
>>> size. With more than 500 customers in 45 countries, DataStax is the
>>> database technology and transactional backbone of choice for the worlds
>>> most innovative companies such as Netflix, Adobe, Intuit, and eBay.
>>>
>>> On Fri, Jan 22, 2016 at 1:39 PM, Bhuvan Rawal <bh...@gmail.com>
>>> wrote:
>>>
>>>> Thanks for the response Alain,
>>>>
>>>> cqlsh> create keyspace mykeyspace WITH replication =
>>>> {'class':'NetworkTopologyStrategy', 'Analytics':2, 'Cassandra':3}
>>>> cqlsh> use mykeyspace;
>>>> cqlsh:mykeyspace>create table mytable (id int primary key, name text,
>>>> address text, phone text);
>>>> cqlsh:mykeyspace> insert into mytable (id, name, address, phone) values
>>>> (1, 'Kiyu','Texas', '555-1212'); # and other similar statement
>>>> I then issued the below command from every node and found consistent
>>>> results.
>>>> cqlsh:mykeyspace> select * from mytable;
>>>>
>>>> // Then i repeated the above steps for NetworkTopologyStrategy and
>>>> found same results
>>>>
>>>> I ran basic cassandra stress
>>>> seed1 - seed of datacenter 1
>>>>  $ cassandra-stress write n=50000 -rate threads=4 -node any_random_ip
>>>>  $ cassandra-stress write n=50000 -rate threads=4 -node seed1
>>>>  $ cassandra-stress write n=50000 -rate threads=4 -node seed1,seed2
>>>>  $ cassandra-stress write n=50000 -rate threads=4 -node
>>>> all_8_ip_comma_seperated
>>>>  $ cassandra-stress write n=1000000 cl=one -mode native cql3 -schema
>>>> keyspace="keyspace1" -pop seq=1..1000000 -node ip1,ip2,ip3,ip4
>>>>
>>>> All of them threw the exception
>>>> *com.datastax.driver.core.exceptions.UnavailableException: Not enough
>>>> replica available for query at consistency LOCAL_ONE (1 required but only 0
>>>> alive)*
>>>>
>>>>
>>>> I have a feeling that the issue is with datacenter name for some
>>>> reason, because in some config files I found DC name to be like DC1/DC2/DC3
>>>> in some it is like Cassandra/Analytics (The ones I had specified while
>>>> installation). Im unsure which yaml/property file to look for correct
>>>> inconsistency.
>>>>
>>>> (C*heers :) - im so tempted to copy that)
>>>>
>>>> Regards,
>>>> Bhuvan
>>>>
>>>> On Fri, Jan 22, 2016 at 8:47 PM, Alain RODRIGUEZ <ar...@gmail.com>
>>>> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> The the exact command you ran (stress-tool with options) could be
>>>>> useful to help you on that.
>>>>>
>>>>> However, Im able to create keyspace, tables and insert data using
>>>>>> cqlsh and it is replicating fine to all the nodes.
>>>>>
>>>>>
>>>>> Having the schema might be useful too.
>>>>>
>>>>> Did you ran the cqlsh and the stress-tool from the same server ? If
>>>>> not, you might want to check the port you use (9042/9160/...) are open.
>>>>> Also, cqlsh uses local_one by default too. If both commands were run
>>>>> against the same DC, from the same machine they should behave the same way.
>>>>> Are they ?
>>>>>
>>>>> C*heers,
>>>>>
>>>>> -----------------
>>>>> Alain
>>>>>
>>>>> The Last Pickle
>>>>> http://www.thelastpickle.com
>>>>>
>>>>>
>>>>> 2016-01-22 9:57 GMT+01:00 Bhuvan Rawal <bh...@gmail.com>:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> i have created a POC cluster with 2 DC , each having 4 nodes with DSE
>>>>>> 4.8.1 installed.
>>>>>>
>>>>>> On issuing cassandra stress im getting an error  and data is not
>>>>>> being inserted:
>>>>>> *com.datastax.driver.core.exceptions.UnavailableException: Not enough
>>>>>> replica available for query at consistency LOCAL_ONE (1 required but only 0
>>>>>> alive)*
>>>>>>
>>>>>> However, Im able to create keyspace, tables and insert data using
>>>>>> cqlsh and it is replicating fine to all the nodes.
>>>>>>
>>>>>> Details of the cluster can be found below (all the nodes seem to be
>>>>>> alive and kicking):
>>>>>>
>>>>>> $ nodetool status Datacenter: Analytics =====================
>>>>>> Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address
>>>>>> Load Tokens Owns Host ID Rack UN 10.41.55.17 428.5 KB 256 ?
>>>>>> 39d6d585-e641-4046-9d0b-797356597b5e rack1 UN 10.41.55.19 404.44 KB 256 ?
>>>>>> 69edf930-efd9-4d74-a798-f3d4ac02e516 rack1 UN 10.41.55.18 423.21 KB 256 ?
>>>>>> b74bab13-09b2-4760-bce9-c8ef05e50f6d rack1 UN 10.41.55.20 683.23 KB 256 ?
>>>>>> fb5c4fed-6e1e-4ea8-838d-358106906830 rack1 Datacenter: Cassandra
>>>>>> ===================== Status=Up/Down |/
>>>>>> State=Normal/Leaving/Joining/Moving -- Address Load Tokens Owns Host
>>>>>> ID Rack UN 10.41.55.15 209.4 KB 256 ? ffc3b9a0-5d5c-4a3d-a99e-49d255731278
>>>>>> rack1 UN 10.41.55.21 227.44 KB 256 ? c68deba4-b9a2-43fc-bb13-6af74c88c210
>>>>>> rack1 UN 10.41.55.23 222.71 KB 256 ? 8229aa87-af00-48fa-ad6b-3066d3dc0e58
>>>>>> rack1 UN 10.41.55.22 218.72 KB 256 ? c7ba84fd-7992-41de-8c88-11574a72db99
>>>>>> rack1
>>>>>>
>>>>>> Regards,
>>>>>> Bhuvan Rawal
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>

Re: Getting error while issuing Cassandra stress

Posted by Sebastian Estevez <se...@datastax.com>.

Sorry I missed that.

Both your nodetool status and keyspace replication settings say Cassandra
and Analytics for the DC names. I'm not sure where you're seeing DC1, DC2,
etc. and why you suspect that is the problem.

All the best,


[image: datastax_logo.png] <http://www.datastax.com/>

Sebastián Estévez

Solutions Architect | 954 905 8615 | sebastian.estevez@datastax.com

[image: linkedin.png] <https://www.linkedin.com/company/datastax> [image:
facebook.png] <https://www.facebook.com/datastax> [image: twitter.png]
<https://twitter.com/datastax> [image: g+.png]
<https://plus.google.com/+Datastax/about>
<http://feeds.feedburner.com/datastax>
<http://goog_410786983>


<http://www.datastax.com/gartner-magic-quadrant-odbms>

DataStax is the fastest, most scalable distributed database technology,
delivering Apache Cassandra to the world’s most innovative enterprises.
Datastax is built to be agile, always-on, and predictably scalable to any
size. With more than 500 customers in 45 countries, DataStax is the
database technology and transactional backbone of choice for the worlds
most innovative companies such as Netflix, Adobe, Intuit, and eBay.

On Fri, Jan 22, 2016 at 1:45 PM, Bhuvan Rawal <bh...@gmail.com> wrote:

> Hi Sebastian,
>
> I had attached nodetool status output in previous mail, pasting it again :
>
> $ nodetool status Datacenter: Analytics =====================
> Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address Load
> Tokens Owns Host ID Rack UN 10.41.55.17 428.5 KB 256 ?
> 39d6d585-e641-4046-9d0b-797356597b5e rack1 UN 10.41.55.19 404.44 KB 256 ?
> 69edf930-efd9-4d74-a798-f3d4ac02e516 rack1 UN 10.41.55.18 423.21 KB 256 ?
> b74bab13-09b2-4760-bce9-c8ef05e50f6d rack1 UN 10.41.55.20 683.23 KB 256 ?
> fb5c4fed-6e1e-4ea8-838d-358106906830 rack1 Datacenter: Cassandra
> ===================== Status=Up/Down |/
> State=Normal/Leaving/Joining/Moving -- Address Load Tokens Owns Host ID
> Rack UN 10.41.55.15 209.4 KB 256 ? ffc3b9a0-5d5c-4a3d-a99e-49d255731278
> rack1 UN 10.41.55.21 227.44 KB 256 ? c68deba4-b9a2-43fc-bb13-6af74c88c210
> rack1 UN 10.41.55.23 222.71 KB 256 ? 8229aa87-af00-48fa-ad6b-3066d3dc0e58
> rack1 UN 10.41.55.22 218.72 KB 256 ? c7ba84fd-7992-41de-8c88-11574a72db99
> rack1
>
> Regards,
> Bhuvan Rawal
>
> On Sat, Jan 23, 2016 at 12:11 AM, Sebastian Estevez <
> sebastian.estevez@datastax.com> wrote:
>
>> The output of `nodetool status` would help us diagnose.
>>
>> All the best,
>>
>>
>> [image: datastax_logo.png] <http://www.datastax.com/>
>>
>> Sebastián Estévez
>>
>> Solutions Architect | 954 905 8615 | sebastian.estevez@datastax.com
>>
>> [image: linkedin.png] <https://www.linkedin.com/company/datastax> [image:
>> facebook.png] <https://www.facebook.com/datastax> [image: twitter.png]
>> <https://twitter.com/datastax> [image: g+.png]
>> <https://plus.google.com/+Datastax/about>
>> <http://feeds.feedburner.com/datastax>
>> <http://goog_410786983>
>>
>>
>> <http://www.datastax.com/gartner-magic-quadrant-odbms>
>>
>> DataStax is the fastest, most scalable distributed database technology,
>> delivering Apache Cassandra to the world’s most innovative enterprises.
>> Datastax is built to be agile, always-on, and predictably scalable to any
>> size. With more than 500 customers in 45 countries, DataStax is the
>> database technology and transactional backbone of choice for the worlds
>> most innovative companies such as Netflix, Adobe, Intuit, and eBay.
>>
>> On Fri, Jan 22, 2016 at 1:39 PM, Bhuvan Rawal <bh...@gmail.com>
>> wrote:
>>
>>> Thanks for the response Alain,
>>>
>>> cqlsh> create keyspace mykeyspace WITH replication =
>>> {'class':'NetworkTopologyStrategy', 'Analytics':2, 'Cassandra':3}
>>> cqlsh> use mykeyspace;
>>> cqlsh:mykeyspace>create table mytable (id int primary key, name text,
>>> address text, phone text);
>>> cqlsh:mykeyspace> insert into mytable (id, name, address, phone) values
>>> (1, 'Kiyu','Texas', '555-1212'); # and other similar statement
>>> I then issued the below command from every node and found consistent
>>> results.
>>> cqlsh:mykeyspace> select * from mytable;
>>>
>>> // Then i repeated the above steps for NetworkTopologyStrategy and found
>>> same results
>>>
>>> I ran basic cassandra stress
>>> seed1 - seed of datacenter 1
>>>  $ cassandra-stress write n=50000 -rate threads=4 -node any_random_ip
>>>  $ cassandra-stress write n=50000 -rate threads=4 -node seed1
>>>  $ cassandra-stress write n=50000 -rate threads=4 -node seed1,seed2
>>>  $ cassandra-stress write n=50000 -rate threads=4 -node
>>> all_8_ip_comma_seperated
>>>  $ cassandra-stress write n=1000000 cl=one -mode native cql3 -schema
>>> keyspace="keyspace1" -pop seq=1..1000000 -node ip1,ip2,ip3,ip4
>>>
>>> All of them threw the exception
>>> *com.datastax.driver.core.exceptions.UnavailableException: Not enough
>>> replica available for query at consistency LOCAL_ONE (1 required but only 0
>>> alive)*
>>>
>>>
>>> I have a feeling that the issue is with datacenter name for some reason,
>>> because in some config files I found DC name to be like DC1/DC2/DC3 in some
>>> it is like Cassandra/Analytics (The ones I had specified while
>>> installation). Im unsure which yaml/property file to look for correct
>>> inconsistency.
>>>
>>> (C*heers :) - im so tempted to copy that)
>>>
>>> Regards,
>>> Bhuvan
>>>
>>> On Fri, Jan 22, 2016 at 8:47 PM, Alain RODRIGUEZ <ar...@gmail.com>
>>> wrote:
>>>
>>>> Hi,
>>>>
>>>> The the exact command you ran (stress-tool with options) could be
>>>> useful to help you on that.
>>>>
>>>> However, Im able to create keyspace, tables and insert data using cqlsh
>>>>> and it is replicating fine to all the nodes.
>>>>
>>>>
>>>> Having the schema might be useful too.
>>>>
>>>> Did you ran the cqlsh and the stress-tool from the same server ? If
>>>> not, you might want to check the port you use (9042/9160/...) are open.
>>>> Also, cqlsh uses local_one by default too. If both commands were run
>>>> against the same DC, from the same machine they should behave the same way.
>>>> Are they ?
>>>>
>>>> C*heers,
>>>>
>>>> -----------------
>>>> Alain
>>>>
>>>> The Last Pickle
>>>> http://www.thelastpickle.com
>>>>
>>>>
>>>> 2016-01-22 9:57 GMT+01:00 Bhuvan Rawal <bh...@gmail.com>:
>>>>
>>>>> Hi,
>>>>>
>>>>> i have created a POC cluster with 2 DC , each having 4 nodes with DSE
>>>>> 4.8.1 installed.
>>>>>
>>>>> On issuing cassandra stress im getting an error  and data is not being
>>>>> inserted:
>>>>> *com.datastax.driver.core.exceptions.UnavailableException: Not enough
>>>>> replica available for query at consistency LOCAL_ONE (1 required but only 0
>>>>> alive)*
>>>>>
>>>>> However, Im able to create keyspace, tables and insert data using
>>>>> cqlsh and it is replicating fine to all the nodes.
>>>>>
>>>>> Details of the cluster can be found below (all the nodes seem to be
>>>>> alive and kicking):
>>>>>
>>>>> $ nodetool status Datacenter: Analytics =====================
>>>>> Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address Load
>>>>> Tokens Owns Host ID Rack UN 10.41.55.17 428.5 KB 256 ?
>>>>> 39d6d585-e641-4046-9d0b-797356597b5e rack1 UN 10.41.55.19 404.44 KB 256 ?
>>>>> 69edf930-efd9-4d74-a798-f3d4ac02e516 rack1 UN 10.41.55.18 423.21 KB 256 ?
>>>>> b74bab13-09b2-4760-bce9-c8ef05e50f6d rack1 UN 10.41.55.20 683.23 KB 256 ?
>>>>> fb5c4fed-6e1e-4ea8-838d-358106906830 rack1 Datacenter: Cassandra
>>>>> ===================== Status=Up/Down |/
>>>>> State=Normal/Leaving/Joining/Moving -- Address Load Tokens Owns Host
>>>>> ID Rack UN 10.41.55.15 209.4 KB 256 ? ffc3b9a0-5d5c-4a3d-a99e-49d255731278
>>>>> rack1 UN 10.41.55.21 227.44 KB 256 ? c68deba4-b9a2-43fc-bb13-6af74c88c210
>>>>> rack1 UN 10.41.55.23 222.71 KB 256 ? 8229aa87-af00-48fa-ad6b-3066d3dc0e58
>>>>> rack1 UN 10.41.55.22 218.72 KB 256 ? c7ba84fd-7992-41de-8c88-11574a72db99
>>>>> rack1
>>>>>
>>>>> Regards,
>>>>> Bhuvan Rawal
>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>

Re: Getting error while issuing Cassandra stress

Posted by Bhuvan Rawal <bh...@gmail.com>.

Hi Sebastian,

I had attached nodetool status output in previous mail, pasting it again :

$ nodetool status Datacenter: Analytics =====================
Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address Load
Tokens Owns Host ID Rack UN 10.41.55.17 428.5 KB 256 ?
39d6d585-e641-4046-9d0b-797356597b5e rack1 UN 10.41.55.19 404.44 KB 256 ?
69edf930-efd9-4d74-a798-f3d4ac02e516 rack1 UN 10.41.55.18 423.21 KB 256 ?
b74bab13-09b2-4760-bce9-c8ef05e50f6d rack1 UN 10.41.55.20 683.23 KB 256 ?
fb5c4fed-6e1e-4ea8-838d-358106906830 rack1 Datacenter: Cassandra
===================== Status=Up/Down |/ State=Normal/Leaving/Joining/Moving --
Address Load Tokens Owns Host ID Rack UN 10.41.55.15 209.4 KB 256 ?
ffc3b9a0-5d5c-4a3d-a99e-49d255731278 rack1 UN 10.41.55.21 227.44 KB 256 ?
c68deba4-b9a2-43fc-bb13-6af74c88c210 rack1 UN 10.41.55.23 222.71 KB 256 ?
8229aa87-af00-48fa-ad6b-3066d3dc0e58 rack1 UN 10.41.55.22 218.72 KB 256 ?
c7ba84fd-7992-41de-8c88-11574a72db99 rack1

Regards,
Bhuvan Rawal

On Sat, Jan 23, 2016 at 12:11 AM, Sebastian Estevez <
sebastian.estevez@datastax.com> wrote:

> The output of `nodetool status` would help us diagnose.
>
> All the best,
>
>
> [image: datastax_logo.png] <http://www.datastax.com/>
>
> Sebastián Estévez
>
> Solutions Architect | 954 905 8615 | sebastian.estevez@datastax.com
>
> [image: linkedin.png] <https://www.linkedin.com/company/datastax> [image:
> facebook.png] <https://www.facebook.com/datastax> [image: twitter.png]
> <https://twitter.com/datastax> [image: g+.png]
> <https://plus.google.com/+Datastax/about>
> <http://feeds.feedburner.com/datastax>
> <http://goog_410786983>
>
>
> <http://www.datastax.com/gartner-magic-quadrant-odbms>
>
> DataStax is the fastest, most scalable distributed database technology,
> delivering Apache Cassandra to the world’s most innovative enterprises.
> Datastax is built to be agile, always-on, and predictably scalable to any
> size. With more than 500 customers in 45 countries, DataStax is the
> database technology and transactional backbone of choice for the worlds
> most innovative companies such as Netflix, Adobe, Intuit, and eBay.
>
> On Fri, Jan 22, 2016 at 1:39 PM, Bhuvan Rawal <bh...@gmail.com> wrote:
>
>> Thanks for the response Alain,
>>
>> cqlsh> create keyspace mykeyspace WITH replication =
>> {'class':'NetworkTopologyStrategy', 'Analytics':2, 'Cassandra':3}
>> cqlsh> use mykeyspace;
>> cqlsh:mykeyspace>create table mytable (id int primary key, name text,
>> address text, phone text);
>> cqlsh:mykeyspace> insert into mytable (id, name, address, phone) values
>> (1, 'Kiyu','Texas', '555-1212'); # and other similar statement
>> I then issued the below command from every node and found consistent
>> results.
>> cqlsh:mykeyspace> select * from mytable;
>>
>> // Then i repeated the above steps for NetworkTopologyStrategy and found
>> same results
>>
>> I ran basic cassandra stress
>> seed1 - seed of datacenter 1
>>  $ cassandra-stress write n=50000 -rate threads=4 -node any_random_ip
>>  $ cassandra-stress write n=50000 -rate threads=4 -node seed1
>>  $ cassandra-stress write n=50000 -rate threads=4 -node seed1,seed2
>>  $ cassandra-stress write n=50000 -rate threads=4 -node
>> all_8_ip_comma_seperated
>>  $ cassandra-stress write n=1000000 cl=one -mode native cql3 -schema
>> keyspace="keyspace1" -pop seq=1..1000000 -node ip1,ip2,ip3,ip4
>>
>> All of them threw the exception
>> *com.datastax.driver.core.exceptions.UnavailableException: Not enough
>> replica available for query at consistency LOCAL_ONE (1 required but only 0
>> alive)*
>>
>>
>> I have a feeling that the issue is with datacenter name for some reason,
>> because in some config files I found DC name to be like DC1/DC2/DC3 in some
>> it is like Cassandra/Analytics (The ones I had specified while
>> installation). Im unsure which yaml/property file to look for correct
>> inconsistency.
>>
>> (C*heers :) - im so tempted to copy that)
>>
>> Regards,
>> Bhuvan
>>
>> On Fri, Jan 22, 2016 at 8:47 PM, Alain RODRIGUEZ <ar...@gmail.com>
>> wrote:
>>
>>> Hi,
>>>
>>> The the exact command you ran (stress-tool with options) could be useful
>>> to help you on that.
>>>
>>> However, Im able to create keyspace, tables and insert data using cqlsh
>>>> and it is replicating fine to all the nodes.
>>>
>>>
>>> Having the schema might be useful too.
>>>
>>> Did you ran the cqlsh and the stress-tool from the same server ? If
>>> not, you might want to check the port you use (9042/9160/...) are open.
>>> Also, cqlsh uses local_one by default too. If both commands were run
>>> against the same DC, from the same machine they should behave the same way.
>>> Are they ?
>>>
>>> C*heers,
>>>
>>> -----------------
>>> Alain
>>>
>>> The Last Pickle
>>> http://www.thelastpickle.com
>>>
>>>
>>> 2016-01-22 9:57 GMT+01:00 Bhuvan Rawal <bh...@gmail.com>:
>>>
>>>> Hi,
>>>>
>>>> i have created a POC cluster with 2 DC , each having 4 nodes with DSE
>>>> 4.8.1 installed.
>>>>
>>>> On issuing cassandra stress im getting an error  and data is not being
>>>> inserted:
>>>> *com.datastax.driver.core.exceptions.UnavailableException: Not enough
>>>> replica available for query at consistency LOCAL_ONE (1 required but only 0
>>>> alive)*
>>>>
>>>> However, Im able to create keyspace, tables and insert data using cqlsh
>>>> and it is replicating fine to all the nodes.
>>>>
>>>> Details of the cluster can be found below (all the nodes seem to be
>>>> alive and kicking):
>>>>
>>>> $ nodetool status Datacenter: Analytics =====================
>>>> Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address Load
>>>> Tokens Owns Host ID Rack UN 10.41.55.17 428.5 KB 256 ?
>>>> 39d6d585-e641-4046-9d0b-797356597b5e rack1 UN 10.41.55.19 404.44 KB 256 ?
>>>> 69edf930-efd9-4d74-a798-f3d4ac02e516 rack1 UN 10.41.55.18 423.21 KB 256 ?
>>>> b74bab13-09b2-4760-bce9-c8ef05e50f6d rack1 UN 10.41.55.20 683.23 KB 256 ?
>>>> fb5c4fed-6e1e-4ea8-838d-358106906830 rack1 Datacenter: Cassandra
>>>> ===================== Status=Up/Down |/
>>>> State=Normal/Leaving/Joining/Moving -- Address Load Tokens Owns Host
>>>> ID Rack UN 10.41.55.15 209.4 KB 256 ? ffc3b9a0-5d5c-4a3d-a99e-49d255731278
>>>> rack1 UN 10.41.55.21 227.44 KB 256 ? c68deba4-b9a2-43fc-bb13-6af74c88c210
>>>> rack1 UN 10.41.55.23 222.71 KB 256 ? 8229aa87-af00-48fa-ad6b-3066d3dc0e58
>>>> rack1 UN 10.41.55.22 218.72 KB 256 ? c7ba84fd-7992-41de-8c88-11574a72db99
>>>> rack1
>>>>
>>>> Regards,
>>>> Bhuvan Rawal
>>>>
>>>>
>>>>
>>>
>>
>

Re: Getting error while issuing Cassandra stress

Posted by Sebastian Estevez <se...@datastax.com>.

The output of `nodetool status` would help us diagnose.

All the best,


[image: datastax_logo.png] <http://www.datastax.com/>

Sebastián Estévez

Solutions Architect | 954 905 8615 | sebastian.estevez@datastax.com

[image: linkedin.png] <https://www.linkedin.com/company/datastax> [image:
facebook.png] <https://www.facebook.com/datastax> [image: twitter.png]
<https://twitter.com/datastax> [image: g+.png]
<https://plus.google.com/+Datastax/about>
<http://feeds.feedburner.com/datastax>
<http://goog_410786983>


<http://www.datastax.com/gartner-magic-quadrant-odbms>

DataStax is the fastest, most scalable distributed database technology,
delivering Apache Cassandra to the world’s most innovative enterprises.
Datastax is built to be agile, always-on, and predictably scalable to any
size. With more than 500 customers in 45 countries, DataStax is the
database technology and transactional backbone of choice for the worlds
most innovative companies such as Netflix, Adobe, Intuit, and eBay.

On Fri, Jan 22, 2016 at 1:39 PM, Bhuvan Rawal <bh...@gmail.com> wrote:

> Thanks for the response Alain,
>
> cqlsh> create keyspace mykeyspace WITH replication =
> {'class':'NetworkTopologyStrategy', 'Analytics':2, 'Cassandra':3}
> cqlsh> use mykeyspace;
> cqlsh:mykeyspace>create table mytable (id int primary key, name text,
> address text, phone text);
> cqlsh:mykeyspace> insert into mytable (id, name, address, phone) values
> (1, 'Kiyu','Texas', '555-1212'); # and other similar statement
> I then issued the below command from every node and found consistent
> results.
> cqlsh:mykeyspace> select * from mytable;
>
> // Then i repeated the above steps for NetworkTopologyStrategy and found
> same results
>
> I ran basic cassandra stress
> seed1 - seed of datacenter 1
>  $ cassandra-stress write n=50000 -rate threads=4 -node any_random_ip
>  $ cassandra-stress write n=50000 -rate threads=4 -node seed1
>  $ cassandra-stress write n=50000 -rate threads=4 -node seed1,seed2
>  $ cassandra-stress write n=50000 -rate threads=4 -node
> all_8_ip_comma_seperated
>  $ cassandra-stress write n=1000000 cl=one -mode native cql3 -schema
> keyspace="keyspace1" -pop seq=1..1000000 -node ip1,ip2,ip3,ip4
>
> All of them threw the exception
> *com.datastax.driver.core.exceptions.UnavailableException: Not enough
> replica available for query at consistency LOCAL_ONE (1 required but only 0
> alive)*
>
>
> I have a feeling that the issue is with datacenter name for some reason,
> because in some config files I found DC name to be like DC1/DC2/DC3 in some
> it is like Cassandra/Analytics (The ones I had specified while
> installation). Im unsure which yaml/property file to look for correct
> inconsistency.
>
> (C*heers :) - im so tempted to copy that)
>
> Regards,
> Bhuvan
>
> On Fri, Jan 22, 2016 at 8:47 PM, Alain RODRIGUEZ <ar...@gmail.com>
> wrote:
>
>> Hi,
>>
>> The the exact command you ran (stress-tool with options) could be useful
>> to help you on that.
>>
>> However, Im able to create keyspace, tables and insert data using cqlsh
>>> and it is replicating fine to all the nodes.
>>
>>
>> Having the schema might be useful too.
>>
>> Did you ran the cqlsh and the stress-tool from the same server ? If not,
>> you might want to check the port you use (9042/9160/...) are open.
>> Also, cqlsh uses local_one by default too. If both commands were run
>> against the same DC, from the same machine they should behave the same way.
>> Are they ?
>>
>> C*heers,
>>
>> -----------------
>> Alain
>>
>> The Last Pickle
>> http://www.thelastpickle.com
>>
>>
>> 2016-01-22 9:57 GMT+01:00 Bhuvan Rawal <bh...@gmail.com>:
>>
>>> Hi,
>>>
>>> i have created a POC cluster with 2 DC , each having 4 nodes with DSE
>>> 4.8.1 installed.
>>>
>>> On issuing cassandra stress im getting an error  and data is not being
>>> inserted:
>>> *com.datastax.driver.core.exceptions.UnavailableException: Not enough
>>> replica available for query at consistency LOCAL_ONE (1 required but only 0
>>> alive)*
>>>
>>> However, Im able to create keyspace, tables and insert data using cqlsh
>>> and it is replicating fine to all the nodes.
>>>
>>> Details of the cluster can be found below (all the nodes seem to be
>>> alive and kicking):
>>>
>>> $ nodetool status Datacenter: Analytics =====================
>>> Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address Load
>>> Tokens Owns Host ID Rack UN 10.41.55.17 428.5 KB 256 ?
>>> 39d6d585-e641-4046-9d0b-797356597b5e rack1 UN 10.41.55.19 404.44 KB 256 ?
>>> 69edf930-efd9-4d74-a798-f3d4ac02e516 rack1 UN 10.41.55.18 423.21 KB 256 ?
>>> b74bab13-09b2-4760-bce9-c8ef05e50f6d rack1 UN 10.41.55.20 683.23 KB 256 ?
>>> fb5c4fed-6e1e-4ea8-838d-358106906830 rack1 Datacenter: Cassandra
>>> ===================== Status=Up/Down |/
>>> State=Normal/Leaving/Joining/Moving -- Address Load Tokens Owns Host ID
>>> Rack UN 10.41.55.15 209.4 KB 256 ? ffc3b9a0-5d5c-4a3d-a99e-49d255731278
>>> rack1 UN 10.41.55.21 227.44 KB 256 ? c68deba4-b9a2-43fc-bb13-6af74c88c210
>>> rack1 UN 10.41.55.23 222.71 KB 256 ? 8229aa87-af00-48fa-ad6b-3066d3dc0e58
>>> rack1 UN 10.41.55.22 218.72 KB 256 ? c7ba84fd-7992-41de-8c88-11574a72db99
>>> rack1
>>>
>>> Regards,
>>> Bhuvan Rawal
>>>
>>>
>>>
>>
>

Re: Getting error while issuing Cassandra stress

Posted by Bhuvan Rawal <bh...@gmail.com>.

Thanks for the response Alain,

cqlsh> create keyspace mykeyspace WITH replication =
{'class':'NetworkTopologyStrategy', 'Analytics':2, 'Cassandra':3}
cqlsh> use mykeyspace;
cqlsh:mykeyspace>create table mytable (id int primary key, name text,
address text, phone text);
cqlsh:mykeyspace> insert into mytable (id, name, address, phone) values (1,
'Kiyu','Texas', '555-1212'); # and other similar statement
I then issued the below command from every node and found consistent
results.
cqlsh:mykeyspace> select * from mytable;

// Then i repeated the above steps for NetworkTopologyStrategy and found
same results

I ran basic cassandra stress
seed1 - seed of datacenter 1
 $ cassandra-stress write n=50000 -rate threads=4 -node any_random_ip
 $ cassandra-stress write n=50000 -rate threads=4 -node seed1
 $ cassandra-stress write n=50000 -rate threads=4 -node seed1,seed2
 $ cassandra-stress write n=50000 -rate threads=4 -node
all_8_ip_comma_seperated
 $ cassandra-stress write n=1000000 cl=one -mode native cql3 -schema
keyspace="keyspace1" -pop seq=1..1000000 -node ip1,ip2,ip3,ip4

All of them threw the exception
*com.datastax.driver.core.exceptions.UnavailableException: Not enough
replica available for query at consistency LOCAL_ONE (1 required but only 0
alive)*


I have a feeling that the issue is with datacenter name for some reason,
because in some config files I found DC name to be like DC1/DC2/DC3 in some
it is like Cassandra/Analytics (The ones I had specified while
installation). Im unsure which yaml/property file to look for correct
inconsistency.

(C*heers :) - im so tempted to copy that)

Regards,
Bhuvan

On Fri, Jan 22, 2016 at 8:47 PM, Alain RODRIGUEZ <ar...@gmail.com> wrote:

> Hi,
>
> The the exact command you ran (stress-tool with options) could be useful
> to help you on that.
>
> However, Im able to create keyspace, tables and insert data using cqlsh
>> and it is replicating fine to all the nodes.
>
>
> Having the schema might be useful too.
>
> Did you ran the cqlsh and the stress-tool from the same server ? If not,
> you might want to check the port you use (9042/9160/...) are open.
> Also, cqlsh uses local_one by default too. If both commands were run
> against the same DC, from the same machine they should behave the same way.
> Are they ?
>
> C*heers,
>
> -----------------
> Alain
>
> The Last Pickle
> http://www.thelastpickle.com
>
>
> 2016-01-22 9:57 GMT+01:00 Bhuvan Rawal <bh...@gmail.com>:
>
>> Hi,
>>
>> i have created a POC cluster with 2 DC , each having 4 nodes with DSE
>> 4.8.1 installed.
>>
>> On issuing cassandra stress im getting an error  and data is not being
>> inserted:
>> *com.datastax.driver.core.exceptions.UnavailableException: Not enough
>> replica available for query at consistency LOCAL_ONE (1 required but only 0
>> alive)*
>>
>> However, Im able to create keyspace, tables and insert data using cqlsh
>> and it is replicating fine to all the nodes.
>>
>> Details of the cluster can be found below (all the nodes seem to be alive
>> and kicking):
>>
>> $ nodetool status Datacenter: Analytics =====================
>> Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address Load
>> Tokens Owns Host ID Rack UN 10.41.55.17 428.5 KB 256 ?
>> 39d6d585-e641-4046-9d0b-797356597b5e rack1 UN 10.41.55.19 404.44 KB 256 ?
>> 69edf930-efd9-4d74-a798-f3d4ac02e516 rack1 UN 10.41.55.18 423.21 KB 256 ?
>> b74bab13-09b2-4760-bce9-c8ef05e50f6d rack1 UN 10.41.55.20 683.23 KB 256 ?
>> fb5c4fed-6e1e-4ea8-838d-358106906830 rack1 Datacenter: Cassandra
>> ===================== Status=Up/Down |/
>> State=Normal/Leaving/Joining/Moving -- Address Load Tokens Owns Host ID
>> Rack UN 10.41.55.15 209.4 KB 256 ? ffc3b9a0-5d5c-4a3d-a99e-49d255731278
>> rack1 UN 10.41.55.21 227.44 KB 256 ? c68deba4-b9a2-43fc-bb13-6af74c88c210
>> rack1 UN 10.41.55.23 222.71 KB 256 ? 8229aa87-af00-48fa-ad6b-3066d3dc0e58
>> rack1 UN 10.41.55.22 218.72 KB 256 ? c7ba84fd-7992-41de-8c88-11574a72db99
>> rack1
>>
>> Regards,
>> Bhuvan Rawal
>>
>>
>>
>

Re: Getting error while issuing Cassandra stress

Posted by Alain RODRIGUEZ <ar...@gmail.com>.

Hi,

The the exact command you ran (stress-tool with options) could be useful to
help you on that.

However, Im able to create keyspace, tables and insert data using cqlsh and
> it is replicating fine to all the nodes.


Having the schema might be useful too.

Did you ran the cqlsh and the stress-tool from the same server ? If not,
you might want to check the port you use (9042/9160/...) are open.
Also, cqlsh uses local_one by default too. If both commands were run
against the same DC, from the same machine they should behave the same way.
Are they ?

C*heers,

-----------------
Alain

The Last Pickle
http://www.thelastpickle.com


2016-01-22 9:57 GMT+01:00 Bhuvan Rawal <bh...@gmail.com>:

> Hi,
>
> i have created a POC cluster with 2 DC , each having 4 nodes with DSE
> 4.8.1 installed.
>
> On issuing cassandra stress im getting an error  and data is not being
> inserted:
> *com.datastax.driver.core.exceptions.UnavailableException: Not enough
> replica available for query at consistency LOCAL_ONE (1 required but only 0
> alive)*
>
> However, Im able to create keyspace, tables and insert data using cqlsh
> and it is replicating fine to all the nodes.
>
> Details of the cluster can be found below (all the nodes seem to be alive
> and kicking):
>
> $ nodetool status Datacenter: Analytics =====================
> Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address Load
> Tokens Owns Host ID Rack UN 10.41.55.17 428.5 KB 256 ?
> 39d6d585-e641-4046-9d0b-797356597b5e rack1 UN 10.41.55.19 404.44 KB 256 ?
> 69edf930-efd9-4d74-a798-f3d4ac02e516 rack1 UN 10.41.55.18 423.21 KB 256 ?
> b74bab13-09b2-4760-bce9-c8ef05e50f6d rack1 UN 10.41.55.20 683.23 KB 256 ?
> fb5c4fed-6e1e-4ea8-838d-358106906830 rack1 Datacenter: Cassandra
> ===================== Status=Up/Down |/
> State=Normal/Leaving/Joining/Moving -- Address Load Tokens Owns Host ID
> Rack UN 10.41.55.15 209.4 KB 256 ? ffc3b9a0-5d5c-4a3d-a99e-49d255731278
> rack1 UN 10.41.55.21 227.44 KB 256 ? c68deba4-b9a2-43fc-bb13-6af74c88c210
> rack1 UN 10.41.55.23 222.71 KB 256 ? 8229aa87-af00-48fa-ad6b-3066d3dc0e58
> rack1 UN 10.41.55.22 218.72 KB 256 ? c7ba84fd-7992-41de-8c88-11574a72db99
> rack1
>
> Regards,
> Bhuvan Rawal
>
>
>