You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Jonathan Baynes <Jo...@tradeweb.com> on 2017/08/23 08:46:44 UTC
C* 3 node issue -Urgent
Hi Everyone.
I need the communities help here.
I have attempted this morning to turn on JMX authentication for Nodetool. I've gone into the Cassandra-env.sh file and updated the following:
LOCAL_JMX=No
JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.authenticate=true"
JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.password.file=/opt/cassandra/application/jmxremote.password"
Then I restarted each node, I have 3 of them in a single DC, Single Rack.
115.63 (seed)
115.64
115.65 (seed)
Nodetool status shows this
[cid:image004.png@01D31BF4.B7FB3460]
In the Yaml I have Authentication turned ON.
Cassandra has been working flawlessly up until the moment I restarted the nodes.
as it stands right now..
115.63 is up, but I cannot connect to CQLSH
I get the following error
[cid:image005.png@01D31BF4.B7FB3460]
When I check the logs I on 115.64 and 115.65 I can see this
WARN [OptionalTasks:1] 2017-08-23 08:37:56,508 CassandraRoleManager.java:355 - CassandraRoleManager skipped default role setup: some nodes were not ready
INFO [OptionalTasks:1] 2017-08-23 08:37:56,508 CassandraRoleManager.java:394 - Setup task failed with error, rescheduling
Caused by: java.lang.RuntimeException: org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE
at org.apache.cassandra.auth.CassandraRoleManager.getRole(CassandraRoleManager.java:512) ~[apache-cassandra-3.0.11.jar:3.0.11]
at org.apache.cassandra.auth.CassandraRoleManager.collectRoles(CassandraRoleManager.java:480) ~[apache-cassandra-3.0.11.jar:3.0.11]
at org.apache.cassandra.auth.CassandraRoleManager.getRoles(CassandraRoleManager.java:284) ~[apache-cassandra-3.0.11.jar:3.0.11]
at org.apache.cassandra.auth.RolesCache$1.load(RolesCache.java:122) ~[apache-cassandra-3.0.11.jar:3.0.11]
at org.apache.cassandra.auth.RolesCache$1.load(RolesCache.java:119) ~[apache-cassandra-3.0.11.jar:3.0.11]
at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527) ~[guava-18.0.jar:na]
at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319) ~[guava-18.0.jar:na]
at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282) ~[guava-18.0.jar:na]
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2197) ~[guava-18.0.jar:na]
... 38 common frames omitted
Caused by: org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE
I have no idea what to do, so any help would be amazing right now..
Thanks
J
Jonathan Baynes
DBA
Tradeweb Europe Limited
Moor Place * 1 Fore Street Avenue * London EC2Y 9DT
P +44 (0)20 77760988 * F +44 (0)20 7776 3201 * M +44 (0)xxxx xxxxxx
Jonathan.Baynes@tradeweb.com
[cid:image001.jpg@01CD26AD.4165F110]<http://www.tradeweb.com/> follow us: [cid:image002.jpg@01CD26AD.4165F110] <https://www.linkedin.com/company/tradeweb?trk=top_nav_home> [cid:image003.jpg@01CD26AD.4165F110] <http://www.twitter.com/Tradeweb>
-----------------------------------------
A leading marketplace<http://www.tradeweb.com/About-Us/Awards/> for electronic fixed income, derivatives and ETF trading
________________________________________________________________________
This e-mail may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and destroy it. Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden. Tradeweb reserves the right to monitor all e-mail communications through its networks. If you do not wish to receive marketing emails about our products / services, please let us know by contacting us, either by email at contactus@tradeweb.com or by writing to us at the registered office of Tradeweb in the UK, which is: Tradeweb Europe Limited (company number 3912826), 1 Fore Street Avenue London EC2Y 9DT. To see our privacy policy, visit our website @ www.tradeweb.com.
Re: C* 3 node issue -Urgent
Posted by kurt greaves <ku...@instaclustr.com>.
Common trap. It's an unfortunate default that is not so easy to change.
RE: C* 3 node issue -Urgent
Posted by Jonathan Baynes <Jo...@tradeweb.com>.
@Kurt, You have hit the nail on the head.
Whilst it was an issue with the cassandra-env.sh file, and I fixed this by overwriting it with the default file, essentially rolling it back, which got me back in, the larger issue is with the system_auth db it is a REPL factor of 1 and thus as stated in this doc http://docs.datastax.com/en/datastax_enterprise/4.8/datastax_enterprise/sec/secConfSysAuthKeyspRepl.html
Attention: To prevent a potential problem logging into a secure cluster, set the replication factor of the system_auth and dse_security keyspaces to a value that is greater than 1. In a multi-node cluster, using the default of 1 prevents logging into any node when the node that stores the user data is down.
Thank you ALL for coming back to me and assisting, Kurt, good shout and thanks for getting me where I need to be, which is back up and running.
Keep up the great work Community!!
From: kurt greaves [mailto:kurt@instaclustr.com]
Sent: 23 August 2017 11:14
To: User
Subject: Re: C* 3 node issue -Urgent
The cassandra user requires QUORUM consistency to be achieved for authentication. Normal users only require ONE. I suspect your system_auth keyspace has an RF of 1, and the node that owns the cassandra users data is down.
Steps to recover:
1. Turn off authentication on all the nodes
2. Restart the nodes and make sure they are UN
3. Alter system_auth to have a higher RF than 1 (3 is probably appropriate)
4. Turn auth back on and restart
5. Create a new user and use that from now on.
________________________________________________________________________
This e-mail may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and destroy it. Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden. Tradeweb reserves the right to monitor all e-mail communications through its networks. If you do not wish to receive marketing emails about our products / services, please let us know by contacting us, either by email at contactus@tradeweb.com or by writing to us at the registered office of Tradeweb in the UK, which is: Tradeweb Europe Limited (company number 3912826), 1 Fore Street Avenue London EC2Y 9DT. To see our privacy policy, visit our website @ www.tradeweb.com.
Re: C* 3 node issue -Urgent
Posted by Ben Bromhead <be...@instaclustr.com>.
Just to clarify that behaviour. QUORUM only applies to the default
superuser, subsequent superusers you create later on are still only queried
at LOCAL_ONE. E.g.
protected static ConsistencyLevel consistencyForRole(String role)
{
if (role.equals(DEFAULT_SUPERUSER_NAME))
return ConsistencyLevel.QUORUM;
else
return ConsistencyLevel.LOCAL_ONE;
}
Despite the fact it suggests consistency for a given role... the function
actually gets passed the username not the role (role lookup happens after
authentication iirc).
Best practices would suggest you change the default superuser password to
some long random password and throw it away and use other superuser
accounts. The Cassandra user is only their to bootstrap auth and nothing
else.
If your RF for the system auth table is very high it will not make it
difficult to login, just to change your password :)
On Wed, 6 Sep 2017 at 11:43 Jeff Jirsa <jj...@gmail.com> wrote:
> More explicitly - if you have 60 nodes, setting rf=60 will likely make it
> very difficult for you to log in as a superuser.
>
> --
> Jeff Jirsa
>
>
> > On Sep 6, 2017, at 11:40 AM, Jon Haddad <jo...@gmail.com>
> wrote:
> >
> > I wouldn’t worry about being meticulous about keeping RF = N as the
> cluster grows. If you had 60 nodes and your auth data was only on 9 you’d
> be completely fine.
> >
> >> On Sep 6, 2017, at 11:36 AM, Cogumelos Maravilha <
> cogumelosmaravilha@sapo.pt> wrote:
> >>
> >> After insert a new node we should:
> >>
> >> ALTER KEYSPACE system_auth WITH REPLICATION = { 'class' : ...
> >> 'replication_factor' : x };
> >>
> >> x = number of nodes in dc
> >>
> >> The default user and password should work:
> >> -u cassandra -p cassandra
> >>
> >> Cheers.
> >>
> >>> On 23-08-2017 11:14, kurt greaves wrote:
> >>> The cassandra user requires QUORUM consistency to be achieved for
> >>> authentication. Normal users only require ONE. I suspect your
> >>> system_auth keyspace has an RF of 1, and the node that owns the
> >>> cassandra users data is down.
> >>>
> >>> Steps to recover:
> >>> 1. Turn off authentication on all the nodes
> >>> 2. Restart the nodes and make sure they are UN
> >>> 3. Alter system_auth to have a higher RF than 1 (3 is probably
> >>> appropriate)
> >>> 4. Turn auth back on and restart
> >>> 5. Create a new user and use that from now on.
> >>>
> >>>
> >>
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
> >> For additional commands, e-mail: user-help@cassandra.apache.org
> >>
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
> > For additional commands, e-mail: user-help@cassandra.apache.org
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
> For additional commands, e-mail: user-help@cassandra.apache.org
>
> --
Ben Bromhead
CTO | Instaclustr <https://www.instaclustr.com/>
+1 650 284 9692
Managed Cassandra / Spark on AWS, Azure and Softlayer
Re: C* 3 node issue -Urgent
Posted by Jeff Jirsa <jj...@gmail.com>.
More explicitly - if you have 60 nodes, setting rf=60 will likely make it very difficult for you to log in as a superuser.
--
Jeff Jirsa
> On Sep 6, 2017, at 11:40 AM, Jon Haddad <jo...@gmail.com> wrote:
>
> I wouldn’t worry about being meticulous about keeping RF = N as the cluster grows. If you had 60 nodes and your auth data was only on 9 you’d be completely fine.
>
>> On Sep 6, 2017, at 11:36 AM, Cogumelos Maravilha <co...@sapo.pt> wrote:
>>
>> After insert a new node we should:
>>
>> ALTER KEYSPACE system_auth WITH REPLICATION = { 'class' : ...
>> 'replication_factor' : x };
>>
>> x = number of nodes in dc
>>
>> The default user and password should work:
>> -u cassandra -p cassandra
>>
>> Cheers.
>>
>>> On 23-08-2017 11:14, kurt greaves wrote:
>>> The cassandra user requires QUORUM consistency to be achieved for
>>> authentication. Normal users only require ONE. I suspect your
>>> system_auth keyspace has an RF of 1, and the node that owns the
>>> cassandra users data is down.
>>>
>>> Steps to recover:
>>> 1. Turn off authentication on all the nodes
>>> 2. Restart the nodes and make sure they are UN
>>> 3. Alter system_auth to have a higher RF than 1 (3 is probably
>>> appropriate)
>>> 4. Turn auth back on and restart
>>> 5. Create a new user and use that from now on.
>>>
>>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
>> For additional commands, e-mail: user-help@cassandra.apache.org
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
> For additional commands, e-mail: user-help@cassandra.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
For additional commands, e-mail: user-help@cassandra.apache.org
Re: C* 3 node issue -Urgent
Posted by Jon Haddad <jo...@gmail.com>.
I wouldn’t worry about being meticulous about keeping RF = N as the cluster grows. If you had 60 nodes and your auth data was only on 9 you’d be completely fine.
> On Sep 6, 2017, at 11:36 AM, Cogumelos Maravilha <co...@sapo.pt> wrote:
>
> After insert a new node we should:
>
> ALTER KEYSPACE system_auth WITH REPLICATION = { 'class' : ...
> 'replication_factor' : x };
>
> x = number of nodes in dc
>
> The default user and password should work:
> -u cassandra -p cassandra
>
> Cheers.
>
> On 23-08-2017 11:14, kurt greaves wrote:
>> The cassandra user requires QUORUM consistency to be achieved for
>> authentication. Normal users only require ONE. I suspect your
>> system_auth keyspace has an RF of 1, and the node that owns the
>> cassandra users data is down.
>>
>> Steps to recover:
>> 1. Turn off authentication on all the nodes
>> 2. Restart the nodes and make sure they are UN
>> 3. Alter system_auth to have a higher RF than 1 (3 is probably
>> appropriate)
>> 4. Turn auth back on and restart
>> 5. Create a new user and use that from now on.
>>
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
> For additional commands, e-mail: user-help@cassandra.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
For additional commands, e-mail: user-help@cassandra.apache.org
Re: C* 3 node issue -Urgent
Posted by Cogumelos Maravilha <co...@sapo.pt>.
After insert a new node we should:
ALTER KEYSPACE system_auth WITH REPLICATION = { 'class' : ...
'replication_factor' : x };
x = number of nodes in dc
The default user and password should work:
-u cassandra -p cassandra
Cheers.
On 23-08-2017 11:14, kurt greaves wrote:
> The cassandra user requires QUORUM consistency to be achieved for
> authentication. Normal users only require ONE. I suspect your
> system_auth keyspace has an RF of 1, and the node that owns the
> cassandra users data is down.
>
> Steps to recover:
> 1. Turn off authentication on all the nodes
> 2. Restart the nodes and make sure they are UN
> 3. Alter system_auth to have a higher RF than 1 (3 is probably
> appropriate)
> 4. Turn auth back on and restart
> 5. Create a new user and use that from now on.
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
For additional commands, e-mail: user-help@cassandra.apache.org
Re: C* 3 node issue -Urgent
Posted by kurt greaves <ku...@instaclustr.com>.
The cassandra user requires QUORUM consistency to be achieved for
authentication. Normal users only require ONE. I suspect your system_auth
keyspace has an RF of 1, and the node that owns the cassandra users data is
down.
Steps to recover:
1. Turn off authentication on all the nodes
2. Restart the nodes and make sure they are UN
3. Alter system_auth to have a higher RF than 1 (3 is probably appropriate)
4. Turn auth back on and restart
5. Create a new user and use that from now on.
Re: C* 3 node issue -Urgent
Posted by Akhil Mehra <ak...@gmail.com>.
I am assuming the following guide or similar was followed to add JMX authentication:
http://docs.datastax.com/en/cassandra/3.0/cassandra/configuration/secureJmxAuthentication.html <http://docs.datastax.com/en/cassandra/3.0/cassandra/configuration/secureJmxAuthentication.html>
> On 23/08/2017, at 9:14 PM, Jonathan Baynes <Jo...@tradeweb.com> wrote:
>
> When trying to connect to CQLSH I get this
>
> “Cannot achieve consistency level QUORUM”
>
> 2 of my 3 nodes are down. So this error is correct. But how do I sign into CQLSH to change this? Or better, how do I get the other 2 nodes back up?
>
> From: Akhil Mehra [mailto:akhilmehra@gmail.com <ma...@gmail.com>]
> Sent: 23 August 2017 10:05
> To: user@cassandra.apache.org <ma...@cassandra.apache.org>
> Subject: Re: C* 3 node issue -Urgent
>
> The cqlsh image say bad credentials. Just confirming that you have the correct username/password when logging on.
>
> By turing on authentication I am assuming you mean using the PasswordAuthenticator instead of the AllowAllAuthenticator in the yaml.
>
> Cheers,
> Akhil
>
> On 23/08/2017, at 8:59 PM, Jonathan Baynes <Jonathan.Baynes@tradeweb.com <ma...@tradeweb.com>> wrote:
>
> I will also mention I am on:
>
> C* 3.0.11
> Linux Oracle red hat 7.1
> Java 1.8.0.31
> Python 2.7
>
> From: Jonathan Baynes
> Sent: 23 August 2017 09:47
> To: 'user@cassandra.apache.org <ma...@cassandra.apache.org>'
> Cc: Stewart Allman
> Subject: C* 3 node issue -Urgent
>
> Hi Everyone.
>
> I need the communities help here.
>
> I have attempted this morning to turn on JMX authentication for Nodetool. I’ve gone into the Cassandra-env.sh file and updated the following:
>
> LOCAL_JMX=No
> JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.authenticate=true"
> JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.password.file=/opt/cassandra/application/jmxremote.password"
>
> Then I restarted each node, I have 3 of them in a single DC, Single Rack.
> 115.63 (seed)
> 115.64
> 115.65 (seed)
>
> Nodetool status shows this
>
> <image004.png>
>
> In the Yaml I have Authentication turned ON.
> Cassandra has been working flawlessly up until the moment I restarted the nodes.
>
> as it stands right now..
>
> 115.63 is up, but I cannot connect to CQLSH
> I get the following error
> <image005.png>
>
> When I check the logs I on 115.64 and 115.65 I can see this
>
> WARN [OptionalTasks:1] 2017-08-23 08:37:56,508 CassandraRoleManager.java:355 - CassandraRoleManager skipped default role setup: some nodes were not ready
> INFO [OptionalTasks:1] 2017-08-23 08:37:56,508 CassandraRoleManager.java:394 - Setup task failed with error, rescheduling
> Caused by: java.lang.RuntimeException: org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE
> at org.apache.cassandra.auth.CassandraRoleManager.getRole(CassandraRoleManager.java:512) ~[apache-cassandra-3.0.11.jar:3.0.11]
> at org.apache.cassandra.auth.CassandraRoleManager.collectRoles(CassandraRoleManager.java:480) ~[apache-cassandra-3.0.11.jar:3.0.11]
> at org.apache.cassandra.auth.CassandraRoleManager.getRoles(CassandraRoleManager.java:284) ~[apache-cassandra-3.0.11.jar:3.0.11]
> at org.apache.cassandra.auth.RolesCache$1.load(RolesCache.java:122) ~[apache-cassandra-3.0.11.jar:3.0.11]
> at org.apache.cassandra.auth.RolesCache$1.load(RolesCache.java:119) ~[apache-cassandra-3.0.11.jar:3.0.11]
> at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527) ~[guava-18.0.jar:na]
> at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319) ~[guava-18.0.jar:na]
> at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282) ~[guava-18.0.jar:na]
> at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2197) ~[guava-18.0.jar:na]
> ... 38 common frames omitted
> Caused by: org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE
>
>
> I have no idea what to do, so any help would be amazing right now..
>
> Thanks
> J
>
> Jonathan Baynes
> DBA
> Tradeweb Europe Limited
> Moor Place • 1 Fore Street Avenue • London EC2Y 9DT
> P +44 (0)20 77760988 • F +44 (0)20 7776 3201 • M +44 (0)xxxx xxxxxx
> Jonathan.Baynes@tradeweb.com <ma...@tradeweb.com>
>
> <image006.jpg> <http://www.tradeweb.com/> follow us: <image007.jpg> <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.linkedin.com_company_tradeweb-3Ftrk-3Dtop-5Fnav-5Fhome&d=DwMFaQ&c=sA0VaJZJFLZREu2pbPeqjXHJ-Wd9NNzgHW3gpUOLSSk&r=CNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-diQ8&m=r8Fr_0dx60l48XrOggPz4OpGpu2K7tEPAmCl14KpBvQ&s=h5M2yR_HjCB_ghUa2rzYYKcwH5VjWUPf_ttbCNUrdqA&e=> <image008.jpg> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.twitter.com_Tradeweb&d=DwMFaQ&c=sA0VaJZJFLZREu2pbPeqjXHJ-Wd9NNzgHW3gpUOLSSk&r=CNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-diQ8&m=r8Fr_0dx60l48XrOggPz4OpGpu2K7tEPAmCl14KpBvQ&s=yzqb445p2cV3fJfSIGPOcGNEkP7zIfxMWNo3jSw7TmM&e=>
> —————————————————————————————————————————
> A leading marketplace <http://www.tradeweb.com/About-Us/Awards/> for electronic fixed income, derivatives and ETF trading
>
> ________________________________________________________________________
> This e-mail may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and destroy it. Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden. Tradeweb reserves the right to monitor all e-mail communications through its networks. If you do not wish to receive marketing emails about our products / services, please let us know by contacting us, either by email at contactus@tradeweb.com <ma...@tradeweb.com> or by writing to us at the registered office of Tradeweb in the UK, which is: Tradeweb Europe Limited (company number 3912826), 1 Fore Street Avenue London EC2Y 9DT. To see our privacy policy, visit our website @ www.tradeweb.com <http://www.tradeweb.com/>.
>
> ________________________________________________________________________
>
> This e-mail may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and destroy it. Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden. Tradeweb reserves the right to monitor all e-mail communications through its networks. If you do not wish to receive marketing emails about our products / services, please let us know by contacting us, either by email at contactus@tradeweb.com <ma...@tradeweb.com> or by writing to us at the registered office of Tradeweb in the UK, which is: Tradeweb Europe Limited (company number 3912826), 1 Fore Street Avenue London EC2Y 9DT. To see our privacy policy, visit our website @ www.tradeweb.com <http://www.tradeweb.com/>.
>
Re: C* 3 node issue -Urgent
Posted by Akhil Mehra <ak...@gmail.com>.
You could try reverting your JMX authentication changes for the time being if getting you nodes up is a priority.
At least you will be able to isolate the problem i.e. is it the Password authenticator or the jmx changes causing the problem.
Cheers,
Akhil
PS Sorry for the silly questions just trying to eliminate the obvious.
> On 23/08/2017, at 9:14 PM, Jonathan Baynes <Jo...@tradeweb.com> wrote:
>
> When trying to connect to CQLSH I get this
>
> “Cannot achieve consistency level QUORUM”
>
> 2 of my 3 nodes are down. So this error is correct. But how do I sign into CQLSH to change this? Or better, how do I get the other 2 nodes back up?
>
> From: Akhil Mehra [mailto:akhilmehra@gmail.com <ma...@gmail.com>]
> Sent: 23 August 2017 10:05
> To: user@cassandra.apache.org <ma...@cassandra.apache.org>
> Subject: Re: C* 3 node issue -Urgent
>
> The cqlsh image say bad credentials. Just confirming that you have the correct username/password when logging on.
>
> By turing on authentication I am assuming you mean using the PasswordAuthenticator instead of the AllowAllAuthenticator in the yaml.
>
> Cheers,
> Akhil
>
> On 23/08/2017, at 8:59 PM, Jonathan Baynes <Jonathan.Baynes@tradeweb.com <ma...@tradeweb.com>> wrote:
>
> I will also mention I am on:
>
> C* 3.0.11
> Linux Oracle red hat 7.1
> Java 1.8.0.31
> Python 2.7
>
> From: Jonathan Baynes
> Sent: 23 August 2017 09:47
> To: 'user@cassandra.apache.org <ma...@cassandra.apache.org>'
> Cc: Stewart Allman
> Subject: C* 3 node issue -Urgent
>
> Hi Everyone.
>
> I need the communities help here.
>
> I have attempted this morning to turn on JMX authentication for Nodetool. I’ve gone into the Cassandra-env.sh file and updated the following:
>
> LOCAL_JMX=No
> JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.authenticate=true"
> JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.password.file=/opt/cassandra/application/jmxremote.password"
>
> Then I restarted each node, I have 3 of them in a single DC, Single Rack.
> 115.63 (seed)
> 115.64
> 115.65 (seed)
>
> Nodetool status shows this
>
> <image004.png>
>
> In the Yaml I have Authentication turned ON.
> Cassandra has been working flawlessly up until the moment I restarted the nodes.
>
> as it stands right now..
>
> 115.63 is up, but I cannot connect to CQLSH
> I get the following error
> <image005.png>
>
> When I check the logs I on 115.64 and 115.65 I can see this
>
> WARN [OptionalTasks:1] 2017-08-23 08:37:56,508 CassandraRoleManager.java:355 - CassandraRoleManager skipped default role setup: some nodes were not ready
> INFO [OptionalTasks:1] 2017-08-23 08:37:56,508 CassandraRoleManager.java:394 - Setup task failed with error, rescheduling
> Caused by: java.lang.RuntimeException: org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE
> at org.apache.cassandra.auth.CassandraRoleManager.getRole(CassandraRoleManager.java:512) ~[apache-cassandra-3.0.11.jar:3.0.11]
> at org.apache.cassandra.auth.CassandraRoleManager.collectRoles(CassandraRoleManager.java:480) ~[apache-cassandra-3.0.11.jar:3.0.11]
> at org.apache.cassandra.auth.CassandraRoleManager.getRoles(CassandraRoleManager.java:284) ~[apache-cassandra-3.0.11.jar:3.0.11]
> at org.apache.cassandra.auth.RolesCache$1.load(RolesCache.java:122) ~[apache-cassandra-3.0.11.jar:3.0.11]
> at org.apache.cassandra.auth.RolesCache$1.load(RolesCache.java:119) ~[apache-cassandra-3.0.11.jar:3.0.11]
> at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527) ~[guava-18.0.jar:na]
> at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319) ~[guava-18.0.jar:na]
> at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282) ~[guava-18.0.jar:na]
> at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2197) ~[guava-18.0.jar:na]
> ... 38 common frames omitted
> Caused by: org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE
>
>
> I have no idea what to do, so any help would be amazing right now..
>
> Thanks
> J
>
> Jonathan Baynes
> DBA
> Tradeweb Europe Limited
> Moor Place • 1 Fore Street Avenue • London EC2Y 9DT
> P +44 (0)20 77760988 • F +44 (0)20 7776 3201 • M +44 (0)xxxx xxxxxx
> Jonathan.Baynes@tradeweb.com <ma...@tradeweb.com>
>
> <image006.jpg> <http://www.tradeweb.com/> follow us: <image007.jpg> <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.linkedin.com_company_tradeweb-3Ftrk-3Dtop-5Fnav-5Fhome&d=DwMFaQ&c=sA0VaJZJFLZREu2pbPeqjXHJ-Wd9NNzgHW3gpUOLSSk&r=CNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-diQ8&m=r8Fr_0dx60l48XrOggPz4OpGpu2K7tEPAmCl14KpBvQ&s=h5M2yR_HjCB_ghUa2rzYYKcwH5VjWUPf_ttbCNUrdqA&e=> <image008.jpg> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.twitter.com_Tradeweb&d=DwMFaQ&c=sA0VaJZJFLZREu2pbPeqjXHJ-Wd9NNzgHW3gpUOLSSk&r=CNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-diQ8&m=r8Fr_0dx60l48XrOggPz4OpGpu2K7tEPAmCl14KpBvQ&s=yzqb445p2cV3fJfSIGPOcGNEkP7zIfxMWNo3jSw7TmM&e=>
> —————————————————————————————————————————
> A leading marketplace <http://www.tradeweb.com/About-Us/Awards/> for electronic fixed income, derivatives and ETF trading
>
> ________________________________________________________________________
> This e-mail may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and destroy it. Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden. Tradeweb reserves the right to monitor all e-mail communications through its networks. If you do not wish to receive marketing emails about our products / services, please let us know by contacting us, either by email at contactus@tradeweb.com <ma...@tradeweb.com> or by writing to us at the registered office of Tradeweb in the UK, which is: Tradeweb Europe Limited (company number 3912826), 1 Fore Street Avenue London EC2Y 9DT. To see our privacy policy, visit our website @ www.tradeweb.com <http://www.tradeweb.com/>.
>
> ________________________________________________________________________
>
> This e-mail may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and destroy it. Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden. Tradeweb reserves the right to monitor all e-mail communications through its networks. If you do not wish to receive marketing emails about our products / services, please let us know by contacting us, either by email at contactus@tradeweb.com <ma...@tradeweb.com> or by writing to us at the registered office of Tradeweb in the UK, which is: Tradeweb Europe Limited (company number 3912826), 1 Fore Street Avenue London EC2Y 9DT. To see our privacy policy, visit our website @ www.tradeweb.com <http://www.tradeweb.com/>.
>
RE: C* 3 node issue -Urgent
Posted by Jonathan Baynes <Jo...@tradeweb.com>.
When trying to connect to CQLSH I get this
“Cannot achieve consistency level QUORUM”
2 of my 3 nodes are down. So this error is correct. But how do I sign into CQLSH to change this? Or better, how do I get the other 2 nodes back up?
From: Akhil Mehra [mailto:akhilmehra@gmail.com]
Sent: 23 August 2017 10:05
To: user@cassandra.apache.org
Subject: Re: C* 3 node issue -Urgent
The cqlsh image say bad credentials. Just confirming that you have the correct username/password when logging on.
By turing on authentication I am assuming you mean using the PasswordAuthenticator instead of the AllowAllAuthenticator in the yaml.
Cheers,
Akhil
On 23/08/2017, at 8:59 PM, Jonathan Baynes <Jo...@tradeweb.com>> wrote:
I will also mention I am on:
C* 3.0.11
Linux Oracle red hat 7.1
Java 1.8.0.31
Python 2.7
From: Jonathan Baynes
Sent: 23 August 2017 09:47
To: 'user@cassandra.apache.org<ma...@cassandra.apache.org>'
Cc: Stewart Allman
Subject: C* 3 node issue -Urgent
Hi Everyone.
I need the communities help here.
I have attempted this morning to turn on JMX authentication for Nodetool. I’ve gone into the Cassandra-env.sh file and updated the following:
LOCAL_JMX=No
JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.authenticate=true"
JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.password.file=/opt/cassandra/application/jmxremote.password"
Then I restarted each node, I have 3 of them in a single DC, Single Rack.
115.63 (seed)
115.64
115.65 (seed)
Nodetool status shows this
<image004.png>
In the Yaml I have Authentication turned ON.
Cassandra has been working flawlessly up until the moment I restarted the nodes.
as it stands right now..
115.63 is up, but I cannot connect to CQLSH
I get the following error
<image005.png>
When I check the logs I on 115.64 and 115.65 I can see this
WARN [OptionalTasks:1] 2017-08-23 08:37:56,508 CassandraRoleManager.java:355 - CassandraRoleManager skipped default role setup: some nodes were not ready
INFO [OptionalTasks:1] 2017-08-23 08:37:56,508 CassandraRoleManager.java:394 - Setup task failed with error, rescheduling
Caused by: java.lang.RuntimeException: org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE
at org.apache.cassandra.auth.CassandraRoleManager.getRole(CassandraRoleManager.java:512) ~[apache-cassandra-3.0.11.jar:3.0.11]
at org.apache.cassandra.auth.CassandraRoleManager.collectRoles(CassandraRoleManager.java:480) ~[apache-cassandra-3.0.11.jar:3.0.11]
at org.apache.cassandra.auth.CassandraRoleManager.getRoles(CassandraRoleManager.java:284) ~[apache-cassandra-3.0.11.jar:3.0.11]
at org.apache.cassandra.auth.RolesCache$1.load(RolesCache.java:122) ~[apache-cassandra-3.0.11.jar:3.0.11]
at org.apache.cassandra.auth.RolesCache$1.load(RolesCache.java:119) ~[apache-cassandra-3.0.11.jar:3.0.11]
at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527) ~[guava-18.0.jar:na]
at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319) ~[guava-18.0.jar:na]
at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282) ~[guava-18.0.jar:na]
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2197) ~[guava-18.0.jar:na]
... 38 common frames omitted
Caused by: org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE
I have no idea what to do, so any help would be amazing right now..
Thanks
J
Jonathan Baynes
DBA
Tradeweb Europe Limited
Moor Place • 1 Fore Street Avenue • London EC2Y 9DT
P +44 (0)20 77760988 • F +44 (0)20 7776 3201 • M +44 (0)xxxx xxxxxx
Jonathan.Baynes@tradeweb.com<ma...@tradeweb.com>
<image006.jpg><http://www.tradeweb.com/> follow us: <image007.jpg><https://urldefense.proofpoint.com/v2/url?u=https-3A__www.linkedin.com_company_tradeweb-3Ftrk-3Dtop-5Fnav-5Fhome&d=DwMFaQ&c=sA0VaJZJFLZREu2pbPeqjXHJ-Wd9NNzgHW3gpUOLSSk&r=CNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-diQ8&m=r8Fr_0dx60l48XrOggPz4OpGpu2K7tEPAmCl14KpBvQ&s=h5M2yR_HjCB_ghUa2rzYYKcwH5VjWUPf_ttbCNUrdqA&e=> <image008.jpg><https://urldefense.proofpoint.com/v2/url?u=http-3A__www.twitter.com_Tradeweb&d=DwMFaQ&c=sA0VaJZJFLZREu2pbPeqjXHJ-Wd9NNzgHW3gpUOLSSk&r=CNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-diQ8&m=r8Fr_0dx60l48XrOggPz4OpGpu2K7tEPAmCl14KpBvQ&s=yzqb445p2cV3fJfSIGPOcGNEkP7zIfxMWNo3jSw7TmM&e=>
—————————————————————————————————————————
A leading marketplace<http://www.tradeweb.com/About-Us/Awards/> for electronic fixed income, derivatives and ETF trading
________________________________________________________________________
This e-mail may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and destroy it. Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden. Tradeweb reserves the right to monitor all e-mail communications through its networks. If you do not wish to receive marketing emails about our products / services, please let us know by contacting us, either by email at contactus@tradeweb.com<ma...@tradeweb.com> or by writing to us at the registered office of Tradeweb in the UK, which is: Tradeweb Europe Limited (company number 3912826), 1 Fore Street Avenue London EC2Y 9DT. To see our privacy policy, visit our website @ www.tradeweb.com<http://www.tradeweb.com/>.
________________________________________________________________________
This e-mail may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and destroy it. Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden. Tradeweb reserves the right to monitor all e-mail communications through its networks. If you do not wish to receive marketing emails about our products / services, please let us know by contacting us, either by email at contactus@tradeweb.com or by writing to us at the registered office of Tradeweb in the UK, which is: Tradeweb Europe Limited (company number 3912826), 1 Fore Street Avenue London EC2Y 9DT. To see our privacy policy, visit our website @ www.tradeweb.com.
RE: C* 3 node issue -Urgent
Posted by Jonathan Baynes <Jo...@tradeweb.com>.
Yes I have the correct credentials I’m using cassandra/cassandra (superuser)
To test that theory I tried a different user and got this
Connection error: ('Unable to connect to any servers', {'10.172.115.63': AuthenticationFailed('Failed to authenticate to 10.172.115.63: Error from server: code=0100 [Bad credentials] message="Username and/or password are incorrect"',)})
Yes you are correct I do have PasswordAuthenticator turned on .
From: Akhil Mehra [mailto:akhilmehra@gmail.com]
Sent: 23 August 2017 10:05
To: user@cassandra.apache.org
Subject: Re: C* 3 node issue -Urgent
The cqlsh image say bad credentials. Just confirming that you have the correct username/password when logging on.
By turing on authentication I am assuming you mean using the PasswordAuthenticator instead of the AllowAllAuthenticator in the yaml.
Cheers,
Akhil
On 23/08/2017, at 8:59 PM, Jonathan Baynes <Jo...@tradeweb.com>> wrote:
I will also mention I am on:
C* 3.0.11
Linux Oracle red hat 7.1
Java 1.8.0.31
Python 2.7
From: Jonathan Baynes
Sent: 23 August 2017 09:47
To: 'user@cassandra.apache.org<ma...@cassandra.apache.org>'
Cc: Stewart Allman
Subject: C* 3 node issue -Urgent
Hi Everyone.
I need the communities help here.
I have attempted this morning to turn on JMX authentication for Nodetool. I’ve gone into the Cassandra-env.sh file and updated the following:
LOCAL_JMX=No
JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.authenticate=true"
JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.password.file=/opt/cassandra/application/jmxremote.password"
Then I restarted each node, I have 3 of them in a single DC, Single Rack.
115.63 (seed)
115.64
115.65 (seed)
Nodetool status shows this
<image004.png>
In the Yaml I have Authentication turned ON.
Cassandra has been working flawlessly up until the moment I restarted the nodes.
as it stands right now..
115.63 is up, but I cannot connect to CQLSH
I get the following error
<image005.png>
When I check the logs I on 115.64 and 115.65 I can see this
WARN [OptionalTasks:1] 2017-08-23 08:37:56,508 CassandraRoleManager.java:355 - CassandraRoleManager skipped default role setup: some nodes were not ready
INFO [OptionalTasks:1] 2017-08-23 08:37:56,508 CassandraRoleManager.java:394 - Setup task failed with error, rescheduling
Caused by: java.lang.RuntimeException: org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE
at org.apache.cassandra.auth.CassandraRoleManager.getRole(CassandraRoleManager.java:512) ~[apache-cassandra-3.0.11.jar:3.0.11]
at org.apache.cassandra.auth.CassandraRoleManager.collectRoles(CassandraRoleManager.java:480) ~[apache-cassandra-3.0.11.jar:3.0.11]
at org.apache.cassandra.auth.CassandraRoleManager.getRoles(CassandraRoleManager.java:284) ~[apache-cassandra-3.0.11.jar:3.0.11]
at org.apache.cassandra.auth.RolesCache$1.load(RolesCache.java:122) ~[apache-cassandra-3.0.11.jar:3.0.11]
at org.apache.cassandra.auth.RolesCache$1.load(RolesCache.java:119) ~[apache-cassandra-3.0.11.jar:3.0.11]
at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527) ~[guava-18.0.jar:na]
at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319) ~[guava-18.0.jar:na]
at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282) ~[guava-18.0.jar:na]
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2197) ~[guava-18.0.jar:na]
... 38 common frames omitted
Caused by: org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE
I have no idea what to do, so any help would be amazing right now..
Thanks
J
Jonathan Baynes
DBA
Tradeweb Europe Limited
Moor Place • 1 Fore Street Avenue • London EC2Y 9DT
P +44 (0)20 77760988 • F +44 (0)20 7776 3201 • M +44 (0)xxxx xxxxxx
Jonathan.Baynes@tradeweb.com<ma...@tradeweb.com>
<image006.jpg><http://www.tradeweb.com/> follow us: <image007.jpg><https://urldefense.proofpoint.com/v2/url?u=https-3A__www.linkedin.com_company_tradeweb-3Ftrk-3Dtop-5Fnav-5Fhome&d=DwMFaQ&c=sA0VaJZJFLZREu2pbPeqjXHJ-Wd9NNzgHW3gpUOLSSk&r=CNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-diQ8&m=r8Fr_0dx60l48XrOggPz4OpGpu2K7tEPAmCl14KpBvQ&s=h5M2yR_HjCB_ghUa2rzYYKcwH5VjWUPf_ttbCNUrdqA&e=> <image008.jpg><https://urldefense.proofpoint.com/v2/url?u=http-3A__www.twitter.com_Tradeweb&d=DwMFaQ&c=sA0VaJZJFLZREu2pbPeqjXHJ-Wd9NNzgHW3gpUOLSSk&r=CNKccIKIKCVbYTu1VxR8dIOP6NLpf4fYjidpNm-diQ8&m=r8Fr_0dx60l48XrOggPz4OpGpu2K7tEPAmCl14KpBvQ&s=yzqb445p2cV3fJfSIGPOcGNEkP7zIfxMWNo3jSw7TmM&e=>
—————————————————————————————————————————
A leading marketplace<http://www.tradeweb.com/About-Us/Awards/> for electronic fixed income, derivatives and ETF trading
________________________________________________________________________
This e-mail may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and destroy it. Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden. Tradeweb reserves the right to monitor all e-mail communications through its networks. If you do not wish to receive marketing emails about our products / services, please let us know by contacting us, either by email at contactus@tradeweb.com<ma...@tradeweb.com> or by writing to us at the registered office of Tradeweb in the UK, which is: Tradeweb Europe Limited (company number 3912826), 1 Fore Street Avenue London EC2Y 9DT. To see our privacy policy, visit our website @ www.tradeweb.com<http://www.tradeweb.com/>.
________________________________________________________________________
This e-mail may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and destroy it. Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden. Tradeweb reserves the right to monitor all e-mail communications through its networks. If you do not wish to receive marketing emails about our products / services, please let us know by contacting us, either by email at contactus@tradeweb.com or by writing to us at the registered office of Tradeweb in the UK, which is: Tradeweb Europe Limited (company number 3912826), 1 Fore Street Avenue London EC2Y 9DT. To see our privacy policy, visit our website @ www.tradeweb.com.
Re: C* 3 node issue -Urgent
Posted by Akhil Mehra <ak...@gmail.com>.
The cqlsh image say bad credentials. Just confirming that you have the correct username/password when logging on.
By turing on authentication I am assuming you mean using the PasswordAuthenticator instead of the AllowAllAuthenticator in the yaml.
Cheers,
Akhil
> On 23/08/2017, at 8:59 PM, Jonathan Baynes <Jo...@tradeweb.com> wrote:
>
> I will also mention I am on:
>
> C* 3.0.11
> Linux Oracle red hat 7.1
> Java 1.8.0.31
> Python 2.7
>
> From: Jonathan Baynes
> Sent: 23 August 2017 09:47
> To: 'user@cassandra.apache.org <ma...@cassandra.apache.org>'
> Cc: Stewart Allman
> Subject: C* 3 node issue -Urgent
>
> Hi Everyone.
>
> I need the communities help here.
>
> I have attempted this morning to turn on JMX authentication for Nodetool. I’ve gone into the Cassandra-env.sh file and updated the following:
>
> LOCAL_JMX=No
> JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.authenticate=true"
> JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.password.file=/opt/cassandra/application/jmxremote.password"
>
> Then I restarted each node, I have 3 of them in a single DC, Single Rack.
> 115.63 (seed)
> 115.64
> 115.65 (seed)
>
> Nodetool status shows this
>
> <image004.png>
>
> In the Yaml I have Authentication turned ON.
> Cassandra has been working flawlessly up until the moment I restarted the nodes.
>
> as it stands right now..
>
> 115.63 is up, but I cannot connect to CQLSH
> I get the following error
> <image005.png>
>
> When I check the logs I on 115.64 and 115.65 I can see this
>
> WARN [OptionalTasks:1] 2017-08-23 08:37:56,508 CassandraRoleManager.java:355 - CassandraRoleManager skipped default role setup: some nodes were not ready
> INFO [OptionalTasks:1] 2017-08-23 08:37:56,508 CassandraRoleManager.java:394 - Setup task failed with error, rescheduling
> Caused by: java.lang.RuntimeException: org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE
> at org.apache.cassandra.auth.CassandraRoleManager.getRole(CassandraRoleManager.java:512) ~[apache-cassandra-3.0.11.jar:3.0.11]
> at org.apache.cassandra.auth.CassandraRoleManager.collectRoles(CassandraRoleManager.java:480) ~[apache-cassandra-3.0.11.jar:3.0.11]
> at org.apache.cassandra.auth.CassandraRoleManager.getRoles(CassandraRoleManager.java:284) ~[apache-cassandra-3.0.11.jar:3.0.11]
> at org.apache.cassandra.auth.RolesCache$1.load(RolesCache.java:122) ~[apache-cassandra-3.0.11.jar:3.0.11]
> at org.apache.cassandra.auth.RolesCache$1.load(RolesCache.java:119) ~[apache-cassandra-3.0.11.jar:3.0.11]
> at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527) ~[guava-18.0.jar:na]
> at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319) ~[guava-18.0.jar:na]
> at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282) ~[guava-18.0.jar:na]
> at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2197) ~[guava-18.0.jar:na]
> ... 38 common frames omitted
> Caused by: org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE
>
>
> I have no idea what to do, so any help would be amazing right now..
>
> Thanks
> J
>
> Jonathan Baynes
> DBA
> Tradeweb Europe Limited
> Moor Place • 1 Fore Street Avenue • London EC2Y 9DT
> P +44 (0)20 77760988 • F +44 (0)20 7776 3201 • M +44 (0)xxxx xxxxxx
> Jonathan.Baynes@tradeweb.com <ma...@tradeweb.com>
>
> <image006.jpg> <http://www.tradeweb.com/> follow us: <image007.jpg> <https://www.linkedin.com/company/tradeweb?trk=top_nav_home> <image008.jpg> <http://www.twitter.com/Tradeweb>
> —————————————————————————————————————————
> A leading marketplace <http://www.tradeweb.com/About-Us/Awards/> for electronic fixed income, derivatives and ETF trading
>
> ________________________________________________________________________
>
> This e-mail may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and destroy it. Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden. Tradeweb reserves the right to monitor all e-mail communications through its networks. If you do not wish to receive marketing emails about our products / services, please let us know by contacting us, either by email at contactus@tradeweb.com <ma...@tradeweb.com> or by writing to us at the registered office of Tradeweb in the UK, which is: Tradeweb Europe Limited (company number 3912826), 1 Fore Street Avenue London EC2Y 9DT. To see our privacy policy, visit our website @ www.tradeweb.com <http://www.tradeweb.com/>.
>
RE: C* 3 node issue -Urgent
Posted by Jonathan Baynes <Jo...@tradeweb.com>.
I will also mention I am on:
C* 3.0.11
Linux Oracle red hat 7.1
Java 1.8.0.31
Python 2.7
From: Jonathan Baynes
Sent: 23 August 2017 09:47
To: 'user@cassandra.apache.org'
Cc: Stewart Allman
Subject: C* 3 node issue -Urgent
Hi Everyone.
I need the communities help here.
I have attempted this morning to turn on JMX authentication for Nodetool. I've gone into the Cassandra-env.sh file and updated the following:
LOCAL_JMX=No
JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.authenticate=true"
JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.password.file=/opt/cassandra/application/jmxremote.password"
Then I restarted each node, I have 3 of them in a single DC, Single Rack.
115.63 (seed)
115.64
115.65 (seed)
Nodetool status shows this
[cid:image004.png@01D31BF6.93C48130]
In the Yaml I have Authentication turned ON.
Cassandra has been working flawlessly up until the moment I restarted the nodes.
as it stands right now..
115.63 is up, but I cannot connect to CQLSH
I get the following error
[cid:image005.png@01D31BF6.93C48130]
When I check the logs I on 115.64 and 115.65 I can see this
WARN [OptionalTasks:1] 2017-08-23 08:37:56,508 CassandraRoleManager.java:355 - CassandraRoleManager skipped default role setup: some nodes were not ready
INFO [OptionalTasks:1] 2017-08-23 08:37:56,508 CassandraRoleManager.java:394 - Setup task failed with error, rescheduling
Caused by: java.lang.RuntimeException: org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE
at org.apache.cassandra.auth.CassandraRoleManager.getRole(CassandraRoleManager.java:512) ~[apache-cassandra-3.0.11.jar:3.0.11]
at org.apache.cassandra.auth.CassandraRoleManager.collectRoles(CassandraRoleManager.java:480) ~[apache-cassandra-3.0.11.jar:3.0.11]
at org.apache.cassandra.auth.CassandraRoleManager.getRoles(CassandraRoleManager.java:284) ~[apache-cassandra-3.0.11.jar:3.0.11]
at org.apache.cassandra.auth.RolesCache$1.load(RolesCache.java:122) ~[apache-cassandra-3.0.11.jar:3.0.11]
at org.apache.cassandra.auth.RolesCache$1.load(RolesCache.java:119) ~[apache-cassandra-3.0.11.jar:3.0.11]
at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527) ~[guava-18.0.jar:na]
at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319) ~[guava-18.0.jar:na]
at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282) ~[guava-18.0.jar:na]
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2197) ~[guava-18.0.jar:na]
... 38 common frames omitted
Caused by: org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE
I have no idea what to do, so any help would be amazing right now..
Thanks
J
Jonathan Baynes
DBA
Tradeweb Europe Limited
Moor Place * 1 Fore Street Avenue * London EC2Y 9DT
P +44 (0)20 77760988 * F +44 (0)20 7776 3201 * M +44 (0)xxxx xxxxxx
Jonathan.Baynes@tradeweb.com<ma...@tradeweb.com>
[cid:image001.jpg@01CD26AD.4165F110]<http://www.tradeweb.com/> follow us: [cid:image002.jpg@01CD26AD.4165F110] <https://www.linkedin.com/company/tradeweb?trk=top_nav_home> [cid:image003.jpg@01CD26AD.4165F110] <http://www.twitter.com/Tradeweb>
-----------------------------------------
A leading marketplace<http://www.tradeweb.com/About-Us/Awards/> for electronic fixed income, derivatives and ETF trading
________________________________________________________________________
This e-mail may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and destroy it. Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden. Tradeweb reserves the right to monitor all e-mail communications through its networks. If you do not wish to receive marketing emails about our products / services, please let us know by contacting us, either by email at contactus@tradeweb.com or by writing to us at the registered office of Tradeweb in the UK, which is: Tradeweb Europe Limited (company number 3912826), 1 Fore Street Avenue London EC2Y 9DT. To see our privacy policy, visit our website @ www.tradeweb.com.