You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by K F <kf...@yahoo.com> on 2015/10/02 04:16:37 UTC
addition of nodes with auth enabled on a datacenter causes existing
nodes to loose their permissions
Hi,
I have 3 DCs out of which in one of the DC, I added 20 nodes. All of the DCs had auth enabled, it was functioning fine. But after addition of 20 nodes in one of the DC, the permissions just got messed-up on the existing nodes. My application started getting errors while querying using the user it normally did.
Finally, I logged onto one of the existing nodes that was operating fine and issued the following cql query and it gave me the following error.
cqlsh:system_auth> select * from users;TSocket read 0 bytes
Upon investigation in system log I found the following exception, what does this mean? Thanks.
2015-10-02 02:03:10,229 [RPC-Thread:3] ERROR Message Unexpected throwable while invoking!java.lang.IllegalArgumentException: Comparison method violates its general contract! at java.util.TimSort.mergeHi(TimSort.java:868) at java.util.TimSort.mergeAt(TimSort.java:485) at java.util.TimSort.mergeForceCollapse(TimSort.java:426) at java.util.TimSort.sort(TimSort.java:223) at java.util.TimSort.sort(TimSort.java:173) at java.util.Arrays.sort(Arrays.java:659) at java.util.Collections.sort(Collections.java:217) at org.apache.cassandra.locator.AbstractEndpointSnitch.sortByProximity(AbstractEndpointSnitch.java:49) at org.apache.cassandra.locator.DynamicEndpointSnitch.sortByProximityWithScore(DynamicEndpointSnitch.java:157) at org.apache.cassandra.locator.DynamicEndpointSnitch.sortByProximityWithBadness(DynamicEndpointSnitch.java:186) at org.apache.cassandra.locator.DynamicEndpointSnitch.sortByProximity(DynamicEndpointSnitch.java:151) at org.apache.cassandra.service.StorageProxy.getLiveSortedEndpoints(StorageProxy.java:1483) at org.apache.cassandra.service.StorageProxy.getRangeSlice(StorageProxy.java:1545) at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:258) at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:228) at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:64) at org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:158) at com.datastax.bdp.cassandra.cql3.DseQueryHandler$StatementExecution.execute(DseQueryHandler.java:448) at com.datastax.bdp.cassandra.cql3.DseQueryHandler.executeOperationWithTiming(DseQueryHandler.java:190) at com.datastax.bdp.cassandra.cql3.DseQueryHandler.executeOperationWithAuditLogging(DseQueryHandler.java:223) at com.datastax.bdp.cassandra.cql3.DseQueryHandler.process(DseQueryHandler.java:103) at org.apache.cassandra.thrift.CassandraServer.execute_cql3_query(CassandraServer.java:1958) at com.datastax.bdp.server.DseServer.execute_cql3_query(DseServer.java:543) at org.apache.cassandra.thrift.Cassandra$Processor$execute_cql3_query.getResult(Cassandra.java:4486) at org.apache.cassandra.thrift.Cassandra$Processor$execute_cql3_query.getResult(Cassandra.java:4470) at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) at com.thinkaurelius.thrift.Message.invoke(Message.java:314) at com.thinkaurelius.thrift.Message$Invocation.execute(Message.java:90) at com.thinkaurelius.thrift.TDisruptorServer$InvocationHandler.onEvent(TDisruptorServer.java:695) at com.thinkaurelius.thrift.TDisruptorServer$InvocationHandler.onEvent(TDisruptorServer.java:689) at com.lmax.disruptor.WorkProcessor.run(WorkProcessor.java:112) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745)
Re: addition of nodes with auth enabled on a datacenter causes existing nodes to loose their permissions
Posted by Graham Sanderson <gr...@vast.com>.
You are seeing
https://issues.apache.org/jira/browse/CASSANDRA-9519 <https://issues.apache.org/jira/browse/CASSANDRA-9519>
> On Oct 1, 2015, at 9:16 PM, K F <kf...@yahoo.com> wrote:
>
> Hi,
>
> I have 3 DCs out of which in one of the DC, I added 20 nodes. All of the DCs had auth enabled, it was functioning fine. But after addition of 20 nodes in one of the DC, the permissions just got messed-up on the existing nodes. My application started getting errors while querying using the user it normally did.
>
> Finally, I logged onto one of the existing nodes that was operating fine and issued the following cql query and it gave me the following error.
>
> cqlsh:system_auth> select * from users;
> TSocket read 0 bytes
>
> Upon investigation in system log I found the following exception, what does this mean? Thanks.
>
> 2015-10-02 02:03:10,229 [RPC-Thread:3] ERROR Message Unexpected throwable while invoking!
> java.lang.IllegalArgumentException: Comparison method violates its general contract!
> at java.util.TimSort.mergeHi(TimSort.java:868)
> at java.util.TimSort.mergeAt(TimSort.java:485)
> at java.util.TimSort.mergeForceCollapse(TimSort.java:426)
> at java.util.TimSort.sort(TimSort.java:223)
> at java.util.TimSort.sort(TimSort.java:173)
> at java.util.Arrays.sort(Arrays.java:659)
> at java.util.Collections.sort(Collections.java:217)
> at org.apache.cassandra.locator.AbstractEndpointSnitch.sortByProximity(AbstractEndpointSnitch.java:49)
> at org.apache.cassandra.locator.DynamicEndpointSnitch.sortByProximityWithScore(DynamicEndpointSnitch.java:157)
> at org.apache.cassandra.locator.DynamicEndpointSnitch.sortByProximityWithBadness(DynamicEndpointSnitch.java:186)
> at org.apache.cassandra.locator.DynamicEndpointSnitch.sortByProximity(DynamicEndpointSnitch.java:151)
> at org.apache.cassandra.service.StorageProxy.getLiveSortedEndpoints(StorageProxy.java:1483)
> at org.apache.cassandra.service.StorageProxy.getRangeSlice(StorageProxy.java:1545)
> at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:258)
> at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:228)
> at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:64)
> at org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:158)
> at com.datastax.bdp.cassandra.cql3.DseQueryHandler$StatementExecution.execute(DseQueryHandler.java:448)
> at com.datastax.bdp.cassandra.cql3.DseQueryHandler.executeOperationWithTiming(DseQueryHandler.java:190)
> at com.datastax.bdp.cassandra.cql3.DseQueryHandler.executeOperationWithAuditLogging(DseQueryHandler.java:223)
> at com.datastax.bdp.cassandra.cql3.DseQueryHandler.process(DseQueryHandler.java:103)
> at org.apache.cassandra.thrift.CassandraServer.execute_cql3_query(CassandraServer.java:1958)
> at com.datastax.bdp.server.DseServer.execute_cql3_query(DseServer.java:543)
> at org.apache.cassandra.thrift.Cassandra$Processor$execute_cql3_query.getResult(Cassandra.java:4486)
> at org.apache.cassandra.thrift.Cassandra$Processor$execute_cql3_query.getResult(Cassandra.java:4470)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> at com.thinkaurelius.thrift.Message.invoke(Message.java:314)
> at com.thinkaurelius.thrift.Message$Invocation.execute(Message.java:90)
> at com.thinkaurelius.thrift.TDisruptorServer$InvocationHandler.onEvent(TDisruptorServer.java:695)
> at com.thinkaurelius.thrift.TDisruptorServer$InvocationHandler.onEvent(TDisruptorServer.java:689)
> at com.lmax.disruptor.WorkProcessor.run(WorkProcessor.java:112)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
>
>