You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Jason Lewis <jl...@packetnexus.com> on 2015/08/25 21:19:42 UTC
'no such object in table'
I'm trying to run nodetool from one node, connecting to another. I
can successfully connect to the majority of nodes in my ring, but two
nodes throw the following error.
nodetool: Failed to connect to '<IP>:7199' NoSuchObjectException: 'no
such object in table'.
Any idea why this is happening? Misconfiguration?
jas
Re: 'no such object in table'
Posted by Jason Lewis <jl...@packetnexus.com>.
I should probably add.. /etc/hosts had the hostname set to 127.0.1.1.
On Thu, Sep 3, 2015 at 11:00 PM, Jason Lewis <jl...@packetnexus.com> wrote:
> I figured this one out. As it turns out, the nodes that I couldn't
> connect to, had the hostname set to 127.0.1.1. The listen IP is *not*
> that IP.
>
> Thanks for the logging tip, it helped track it down.
>
> On Thu, Sep 3, 2015 at 10:43 PM, Jason Lewis <jl...@packetnexus.com> wrote:
>> After enabling that option, I'm seeing errors like this on the node I
>> can't connect to.
>>
>>
>> Sep 04, 2015 2:35:48 AM sun.rmi.server.UnicastServerRef logCallException
>> FINE: RMI TCP Connection(4)-127.0.0.1: [127.0.0.1] exception:
>> javax.management.InstanceNotFoundException:
>> org.apache.cassandra.metrics:type=ColumnFamily,keyspace=ks,scope=cf,name=PendingTasks
>> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getMBean(DefaultMBeanServerInterceptor.java:1095)
>> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:643)
>> at com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.java:678)
>> at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1448)
>> at javax.management.remote.rmi.RMIConnectionImpl.access$300(RMIConnectionImpl.java:76)
>> at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1312)
>> at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1404)
>> at javax.management.remote.rmi.RMIConnectionImpl.getAttribute(RMIConnectionImpl.java:641)
>> at sun.reflect.GeneratedMethodAccessor9.invoke(Unknown Source)
>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:497)
>> at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:323)
>> at sun.rmi.transport.Transport$1.run(Transport.java:200)
>> at sun.rmi.transport.Transport$1.run(Transport.java:197)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at sun.rmi.transport.Transport.serviceCall(Transport.java:196)
>> at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:568)
>> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:826)
>> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$79(TCPTransport.java:683)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:682)
>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> On Wed, Aug 26, 2015 at 2:05 PM, Nate McCall <na...@thelastpickle.com> wrote:
>>>> LOCAL_JMX=no
>>>>
>>>> if [ "$LOCAL_JMX" = "yes" ]; then
>>>> JVM_OPTS="$JVM_OPTS -Dcassandra.jmx.local.port=$JMX_PORT
>>>> -XX:+DisableExplicitGC"
>>>> else
>>>> JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.port=$JMX_PORT"
>>>> JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.rmi.port=$JMX_PORT"
>>>> JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.ssl=false"
>>>> JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.authenticate=false"
>>>> JVM_OPTS="$JVM_OPTS
>>>>
>>>> -Dcom.sun.management.jmxremote.password.file=/etc/cassandra/jmxremote.password"
>>>> fi
>>>>
>>>
>>> Retry with the following option added to your JVM_OPTS:
>>> java.rmi.server.logCalls=true
>>>
>>> This should produce some more information about what is going on.
>>>
>>>
>>>
>>>
>>> --
>>> -----------------
>>> Nate McCall
>>> Austin, TX
>>> @zznate
>>>
>>> Co-Founder & Sr. Technical Consultant
>>> Apache Cassandra Consulting
>>> http://www.thelastpickle.com
Re: 'no such object in table'
Posted by Jason Lewis <jl...@packetnexus.com>.
I figured this one out. As it turns out, the nodes that I couldn't
connect to, had the hostname set to 127.0.1.1. The listen IP is *not*
that IP.
Thanks for the logging tip, it helped track it down.
On Thu, Sep 3, 2015 at 10:43 PM, Jason Lewis <jl...@packetnexus.com> wrote:
> After enabling that option, I'm seeing errors like this on the node I
> can't connect to.
>
>
> Sep 04, 2015 2:35:48 AM sun.rmi.server.UnicastServerRef logCallException
> FINE: RMI TCP Connection(4)-127.0.0.1: [127.0.0.1] exception:
> javax.management.InstanceNotFoundException:
> org.apache.cassandra.metrics:type=ColumnFamily,keyspace=ks,scope=cf,name=PendingTasks
> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getMBean(DefaultMBeanServerInterceptor.java:1095)
> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:643)
> at com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.java:678)
> at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1448)
> at javax.management.remote.rmi.RMIConnectionImpl.access$300(RMIConnectionImpl.java:76)
> at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1312)
> at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1404)
> at javax.management.remote.rmi.RMIConnectionImpl.getAttribute(RMIConnectionImpl.java:641)
> at sun.reflect.GeneratedMethodAccessor9.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:323)
> at sun.rmi.transport.Transport$1.run(Transport.java:200)
> at sun.rmi.transport.Transport$1.run(Transport.java:197)
> at java.security.AccessController.doPrivileged(Native Method)
> at sun.rmi.transport.Transport.serviceCall(Transport.java:196)
> at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:568)
> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:826)
> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$79(TCPTransport.java:683)
> at java.security.AccessController.doPrivileged(Native Method)
> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:682)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> On Wed, Aug 26, 2015 at 2:05 PM, Nate McCall <na...@thelastpickle.com> wrote:
>>> LOCAL_JMX=no
>>>
>>> if [ "$LOCAL_JMX" = "yes" ]; then
>>> JVM_OPTS="$JVM_OPTS -Dcassandra.jmx.local.port=$JMX_PORT
>>> -XX:+DisableExplicitGC"
>>> else
>>> JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.port=$JMX_PORT"
>>> JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.rmi.port=$JMX_PORT"
>>> JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.ssl=false"
>>> JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.authenticate=false"
>>> JVM_OPTS="$JVM_OPTS
>>>
>>> -Dcom.sun.management.jmxremote.password.file=/etc/cassandra/jmxremote.password"
>>> fi
>>>
>>
>> Retry with the following option added to your JVM_OPTS:
>> java.rmi.server.logCalls=true
>>
>> This should produce some more information about what is going on.
>>
>>
>>
>>
>> --
>> -----------------
>> Nate McCall
>> Austin, TX
>> @zznate
>>
>> Co-Founder & Sr. Technical Consultant
>> Apache Cassandra Consulting
>> http://www.thelastpickle.com
Re: 'no such object in table'
Posted by Jason Lewis <jl...@packetnexus.com>.
After enabling that option, I'm seeing errors like this on the node I
can't connect to.
Sep 04, 2015 2:35:48 AM sun.rmi.server.UnicastServerRef logCallException
FINE: RMI TCP Connection(4)-127.0.0.1: [127.0.0.1] exception:
javax.management.InstanceNotFoundException:
org.apache.cassandra.metrics:type=ColumnFamily,keyspace=ks,scope=cf,name=PendingTasks
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getMBean(DefaultMBeanServerInterceptor.java:1095)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:643)
at com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.java:678)
at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1448)
at javax.management.remote.rmi.RMIConnectionImpl.access$300(RMIConnectionImpl.java:76)
at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1312)
at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1404)
at javax.management.remote.rmi.RMIConnectionImpl.getAttribute(RMIConnectionImpl.java:641)
at sun.reflect.GeneratedMethodAccessor9.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:323)
at sun.rmi.transport.Transport$1.run(Transport.java:200)
at sun.rmi.transport.Transport$1.run(Transport.java:197)
at java.security.AccessController.doPrivileged(Native Method)
at sun.rmi.transport.Transport.serviceCall(Transport.java:196)
at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:568)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:826)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$79(TCPTransport.java:683)
at java.security.AccessController.doPrivileged(Native Method)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:682)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
On Wed, Aug 26, 2015 at 2:05 PM, Nate McCall <na...@thelastpickle.com> wrote:
>> LOCAL_JMX=no
>>
>> if [ "$LOCAL_JMX" = "yes" ]; then
>> JVM_OPTS="$JVM_OPTS -Dcassandra.jmx.local.port=$JMX_PORT
>> -XX:+DisableExplicitGC"
>> else
>> JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.port=$JMX_PORT"
>> JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.rmi.port=$JMX_PORT"
>> JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.ssl=false"
>> JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.authenticate=false"
>> JVM_OPTS="$JVM_OPTS
>>
>> -Dcom.sun.management.jmxremote.password.file=/etc/cassandra/jmxremote.password"
>> fi
>>
>
> Retry with the following option added to your JVM_OPTS:
> java.rmi.server.logCalls=true
>
> This should produce some more information about what is going on.
>
>
>
>
> --
> -----------------
> Nate McCall
> Austin, TX
> @zznate
>
> Co-Founder & Sr. Technical Consultant
> Apache Cassandra Consulting
> http://www.thelastpickle.com
Re: 'no such object in table'
Posted by Nate McCall <na...@thelastpickle.com>.
> LOCAL_JMX=no
>
> if [ "$LOCAL_JMX" = "yes" ]; then
> JVM_OPTS="$JVM_OPTS -Dcassandra.jmx.local.port=$JMX_PORT
> -XX:+DisableExplicitGC"
> else
> JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.port=$JMX_PORT"
> JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.rmi.port=$JMX_PORT"
> JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.ssl=false"
> JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.authenticate=false"
> JVM_OPTS="$JVM_OPTS
>
-Dcom.sun.management.jmxremote.password.file=/etc/cassandra/jmxremote.password"
> fi
>
Retry with the following option added to your JVM_OPTS:
java.rmi.server.logCalls=true
This should produce some more information about what is going on.
--
-----------------
Nate McCall
Austin, TX
@zznate
Co-Founder & Sr. Technical Consultant
Apache Cassandra Consulting
http://www.thelastpickle.com
Re: 'no such object in table'
Posted by Jason Lewis <jl...@packetnexus.com>.
All of my nodes have the same cassandra-env.sh. Only a few of them complain.
LOCAL_JMX=no
if [ "$LOCAL_JMX" = "yes" ]; then
JVM_OPTS="$JVM_OPTS -Dcassandra.jmx.local.port=$JMX_PORT
-XX:+DisableExplicitGC"
else
JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.port=$JMX_PORT"
JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.rmi.port=$JMX_PORT"
JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.ssl=false"
JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.authenticate=false"
JVM_OPTS="$JVM_OPTS
-Dcom.sun.management.jmxremote.password.file=/etc/cassandra/jmxremote.password"
fi
On Tue, Aug 25, 2015 at 4:35 PM, Michael Shuler <mi...@pbandjelly.org> wrote:
> On 08/25/2015 02:19 PM, Jason Lewis wrote:
>>
>> I'm trying to run nodetool from one node, connecting to another. I
>> can successfully connect to the majority of nodes in my ring, but two
>> nodes throw the following error.
>>
>> nodetool: Failed to connect to '<IP>:7199' NoSuchObjectException: 'no
>> such object in table'.
>>
>> Any idea why this is happening? Misconfiguration?
>
>
> Possibly. Check those nodes to see if 7199 is listening to only localhost or
> some private IP your client node cannot reach (failed to connect). The
> default is to only listen on localhost, as seen on my machine:
>
> $ netstat -ln | grep 7199
> tcp 0 0 127.0.0.1:7199 0.0.0.0:* LISTEN
>
> JMX configuration is set in conf/cassandra-env.sh - please, configure JMX
> security as documented in that file and/or firewall JMX. Check all your
> nodes JMX security configs! :)
>
> --
> Kind regards,
> Michael
>
Re: 'no such object in table'
Posted by Michael Shuler <mi...@pbandjelly.org>.
On 08/25/2015 02:19 PM, Jason Lewis wrote:
> I'm trying to run nodetool from one node, connecting to another. I
> can successfully connect to the majority of nodes in my ring, but two
> nodes throw the following error.
>
> nodetool: Failed to connect to '<IP>:7199' NoSuchObjectException: 'no
> such object in table'.
>
> Any idea why this is happening? Misconfiguration?
Possibly. Check those nodes to see if 7199 is listening to only
localhost or some private IP your client node cannot reach (failed to
connect). The default is to only listen on localhost, as seen on my machine:
$ netstat -ln | grep 7199
tcp 0 0 127.0.0.1:7199 0.0.0.0:* LISTEN
JMX configuration is set in conf/cassandra-env.sh - please, configure
JMX security as documented in that file and/or firewall JMX. Check all
your nodes JMX security configs! :)
--
Kind regards,
Michael