You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Jason Lewis <jl...@packetnexus.com> on 2015/09/04 04:43:05 UTC
Re: 'no such object in table'
After enabling that option, I'm seeing errors like this on the node I
can't connect to.
Sep 04, 2015 2:35:48 AM sun.rmi.server.UnicastServerRef logCallException
FINE: RMI TCP Connection(4)-127.0.0.1: [127.0.0.1] exception:
javax.management.InstanceNotFoundException:
org.apache.cassandra.metrics:type=ColumnFamily,keyspace=ks,scope=cf,name=PendingTasks
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getMBean(DefaultMBeanServerInterceptor.java:1095)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:643)
at com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.java:678)
at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1448)
at javax.management.remote.rmi.RMIConnectionImpl.access$300(RMIConnectionImpl.java:76)
at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1312)
at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1404)
at javax.management.remote.rmi.RMIConnectionImpl.getAttribute(RMIConnectionImpl.java:641)
at sun.reflect.GeneratedMethodAccessor9.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:323)
at sun.rmi.transport.Transport$1.run(Transport.java:200)
at sun.rmi.transport.Transport$1.run(Transport.java:197)
at java.security.AccessController.doPrivileged(Native Method)
at sun.rmi.transport.Transport.serviceCall(Transport.java:196)
at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:568)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:826)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$79(TCPTransport.java:683)
at java.security.AccessController.doPrivileged(Native Method)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:682)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
On Wed, Aug 26, 2015 at 2:05 PM, Nate McCall <na...@thelastpickle.com> wrote:
>> LOCAL_JMX=no
>>
>> if [ "$LOCAL_JMX" = "yes" ]; then
>> JVM_OPTS="$JVM_OPTS -Dcassandra.jmx.local.port=$JMX_PORT
>> -XX:+DisableExplicitGC"
>> else
>> JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.port=$JMX_PORT"
>> JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.rmi.port=$JMX_PORT"
>> JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.ssl=false"
>> JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.authenticate=false"
>> JVM_OPTS="$JVM_OPTS
>>
>> -Dcom.sun.management.jmxremote.password.file=/etc/cassandra/jmxremote.password"
>> fi
>>
>
> Retry with the following option added to your JVM_OPTS:
> java.rmi.server.logCalls=true
>
> This should produce some more information about what is going on.
>
>
>
>
> --
> -----------------
> Nate McCall
> Austin, TX
> @zznate
>
> Co-Founder & Sr. Technical Consultant
> Apache Cassandra Consulting
> http://www.thelastpickle.com
Re: 'no such object in table'
Posted by Jason Lewis <jl...@packetnexus.com>.
I should probably add.. /etc/hosts had the hostname set to 127.0.1.1.
On Thu, Sep 3, 2015 at 11:00 PM, Jason Lewis <jl...@packetnexus.com> wrote:
> I figured this one out. As it turns out, the nodes that I couldn't
> connect to, had the hostname set to 127.0.1.1. The listen IP is *not*
> that IP.
>
> Thanks for the logging tip, it helped track it down.
>
> On Thu, Sep 3, 2015 at 10:43 PM, Jason Lewis <jl...@packetnexus.com> wrote:
>> After enabling that option, I'm seeing errors like this on the node I
>> can't connect to.
>>
>>
>> Sep 04, 2015 2:35:48 AM sun.rmi.server.UnicastServerRef logCallException
>> FINE: RMI TCP Connection(4)-127.0.0.1: [127.0.0.1] exception:
>> javax.management.InstanceNotFoundException:
>> org.apache.cassandra.metrics:type=ColumnFamily,keyspace=ks,scope=cf,name=PendingTasks
>> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getMBean(DefaultMBeanServerInterceptor.java:1095)
>> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:643)
>> at com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.java:678)
>> at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1448)
>> at javax.management.remote.rmi.RMIConnectionImpl.access$300(RMIConnectionImpl.java:76)
>> at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1312)
>> at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1404)
>> at javax.management.remote.rmi.RMIConnectionImpl.getAttribute(RMIConnectionImpl.java:641)
>> at sun.reflect.GeneratedMethodAccessor9.invoke(Unknown Source)
>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:497)
>> at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:323)
>> at sun.rmi.transport.Transport$1.run(Transport.java:200)
>> at sun.rmi.transport.Transport$1.run(Transport.java:197)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at sun.rmi.transport.Transport.serviceCall(Transport.java:196)
>> at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:568)
>> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:826)
>> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$79(TCPTransport.java:683)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:682)
>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> On Wed, Aug 26, 2015 at 2:05 PM, Nate McCall <na...@thelastpickle.com> wrote:
>>>> LOCAL_JMX=no
>>>>
>>>> if [ "$LOCAL_JMX" = "yes" ]; then
>>>> JVM_OPTS="$JVM_OPTS -Dcassandra.jmx.local.port=$JMX_PORT
>>>> -XX:+DisableExplicitGC"
>>>> else
>>>> JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.port=$JMX_PORT"
>>>> JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.rmi.port=$JMX_PORT"
>>>> JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.ssl=false"
>>>> JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.authenticate=false"
>>>> JVM_OPTS="$JVM_OPTS
>>>>
>>>> -Dcom.sun.management.jmxremote.password.file=/etc/cassandra/jmxremote.password"
>>>> fi
>>>>
>>>
>>> Retry with the following option added to your JVM_OPTS:
>>> java.rmi.server.logCalls=true
>>>
>>> This should produce some more information about what is going on.
>>>
>>>
>>>
>>>
>>> --
>>> -----------------
>>> Nate McCall
>>> Austin, TX
>>> @zznate
>>>
>>> Co-Founder & Sr. Technical Consultant
>>> Apache Cassandra Consulting
>>> http://www.thelastpickle.com
Re: 'no such object in table'
Posted by Jason Lewis <jl...@packetnexus.com>.
I figured this one out. As it turns out, the nodes that I couldn't
connect to, had the hostname set to 127.0.1.1. The listen IP is *not*
that IP.
Thanks for the logging tip, it helped track it down.
On Thu, Sep 3, 2015 at 10:43 PM, Jason Lewis <jl...@packetnexus.com> wrote:
> After enabling that option, I'm seeing errors like this on the node I
> can't connect to.
>
>
> Sep 04, 2015 2:35:48 AM sun.rmi.server.UnicastServerRef logCallException
> FINE: RMI TCP Connection(4)-127.0.0.1: [127.0.0.1] exception:
> javax.management.InstanceNotFoundException:
> org.apache.cassandra.metrics:type=ColumnFamily,keyspace=ks,scope=cf,name=PendingTasks
> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getMBean(DefaultMBeanServerInterceptor.java:1095)
> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:643)
> at com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.java:678)
> at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1448)
> at javax.management.remote.rmi.RMIConnectionImpl.access$300(RMIConnectionImpl.java:76)
> at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1312)
> at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1404)
> at javax.management.remote.rmi.RMIConnectionImpl.getAttribute(RMIConnectionImpl.java:641)
> at sun.reflect.GeneratedMethodAccessor9.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:323)
> at sun.rmi.transport.Transport$1.run(Transport.java:200)
> at sun.rmi.transport.Transport$1.run(Transport.java:197)
> at java.security.AccessController.doPrivileged(Native Method)
> at sun.rmi.transport.Transport.serviceCall(Transport.java:196)
> at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:568)
> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:826)
> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$79(TCPTransport.java:683)
> at java.security.AccessController.doPrivileged(Native Method)
> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:682)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> On Wed, Aug 26, 2015 at 2:05 PM, Nate McCall <na...@thelastpickle.com> wrote:
>>> LOCAL_JMX=no
>>>
>>> if [ "$LOCAL_JMX" = "yes" ]; then
>>> JVM_OPTS="$JVM_OPTS -Dcassandra.jmx.local.port=$JMX_PORT
>>> -XX:+DisableExplicitGC"
>>> else
>>> JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.port=$JMX_PORT"
>>> JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.rmi.port=$JMX_PORT"
>>> JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.ssl=false"
>>> JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.authenticate=false"
>>> JVM_OPTS="$JVM_OPTS
>>>
>>> -Dcom.sun.management.jmxremote.password.file=/etc/cassandra/jmxremote.password"
>>> fi
>>>
>>
>> Retry with the following option added to your JVM_OPTS:
>> java.rmi.server.logCalls=true
>>
>> This should produce some more information about what is going on.
>>
>>
>>
>>
>> --
>> -----------------
>> Nate McCall
>> Austin, TX
>> @zznate
>>
>> Co-Founder & Sr. Technical Consultant
>> Apache Cassandra Consulting
>> http://www.thelastpickle.com