You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cassandra.apache.org by shirish <sh...@gmail.com> on 2010/03/11 08:20:03 UTC
Out of Memory error.
Hello,
I managed to run two nodes on machines with IP address 192.168.0.6(cass01)
and 192.168.0.17(cass02). After a while when i did *nodeprobe --host cass01
--port 9160 I get this messages on cass01 *
INFO 12:35:58,809 Node /192.168.0.17 state jump to normal
java.lang.OutOfMemoryError: Java heap space
Dumping heap to java_pid3496.hprof ...
Heap dump file created [7632952 bytes in 0.233 secs]
ERROR 12:43:19,288 Fatal exception in thread Thread[pool-1-thread-3,5,main]
java.lang.OutOfMemoryError: Java heap space
at
org.apache.thrift.protocol.TBinaryProtocol.readStringBody(TBinaryProtocol.java:296)
at
org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:203)
at
org.apache.cassandra.thrift.Cassandra$Processor.process(Cassandra.java:1103)
at
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:253)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)
and t*his one on cass02 *
Error connecting to remote JMX agent!
java.io.IOException: Failed to retrieve RMIServer stub:
javax.naming.CommunicationException [Root exception is
java.rmi.ConnectIOException: error during JRMP connection establishment;
nested exception is:
java.io.EOFException]
at
javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:342)
at
javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:267)
at org.apache.cassandra.tools.NodeProbe.connect(NodeProbe.java:105)
at org.apache.cassandra.tools.NodeProbe.<init>(NodeProbe.java:81)
at org.apache.cassandra.tools.NodeCmd.main(NodeCmd.java:400)
Caused by: javax.naming.CommunicationException [Root exception is
java.rmi.ConnectIOException: error during JRMP connection establishment;
nested exception is:
java.io.EOFException]
at
com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:118)
at
com.sun.jndi.toolkit.url.GenericURLContext.lookup(GenericURLContext.java:203)
at javax.naming.InitialContext.lookup(InitialContext.java:409)
at
javax.management.remote.rmi.RMIConnector.findRMIServerJNDI(RMIConnector.java:1902)
at
javax.management.remote.rmi.RMIConnector.findRMIServer(RMIConnector.java:1871)
at
javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:276)
... 4 more
Caused by: java.rmi.ConnectIOException: error during JRMP connection
establishment; nested exception is:
java.io.EOFException
at
sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:304)
at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:202)
at sun.rmi.server.UnicastRef.newCall(UnicastRef.java:340)
at sun.rmi.registry.RegistryImpl_Stub.lookup(Unknown Source)
at
com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:114)
... 9 more
Caused by: java.io.EOFException
at java.io.DataInputStream.readByte(DataInputStream.java:267)
at
sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:246)
... 13 more
where can I get to read more about such Exceptions in cassandra ? and what
might be the reason.
Re: Out of Memory error.
Posted by Peter Schüller <sc...@spotify.com>.
> I managed to run two nodes on machines with IP address 192.168.0.6(cass01)
> and 192.168.0.17(cass02). After a while when i did *nodeprobe --host cass01
> --port 9160 I get this messages on cass01 *
[snip]
> java.lang.OutOfMemoryError: Java heap space
> at
> org.apache.thrift.protocol.TBinaryProtocol.readStringBody(TBinaryProtocol.java:296)
> at
This happens when you send non-thrift data to the thrift service port
(unless you're lucky) because your non-thrift bytes on the connection
is interpreted as a length to allocate. There is a bug filed for it.
Port 9160 is probably the wrong port for nodeprobe.
--
/ Peter Schuller aka scode
Re: Out of Memory error.
Posted by Santal Li <sa...@gmail.com>.
This is a issue of Thrift, please refer to
https://issues.apache.org/jira/browse/THRIFT-602
when thrift server recive bogus input, it maybe using too many buffer space
for recive data. Seems it still not got fix.
-Santal
2010/3/11 Eric Yu <su...@gmail.com>
> try this if you use default cassandra.in.sh
> $ nodeprobe --host cass01 --port 8080
>
> On Thu, Mar 11, 2010 at 3:20 PM, shirish <sh...@gmail.com> wrote:
>
> > Hello,
> >
> > I managed to run two nodes on machines with IP address
> 192.168.0.6(cass01)
> > and 192.168.0.17(cass02). After a while when i did *nodeprobe --host
> cass01
> > --port 9160 I get this messages on cass01 *
> >
> > INFO 12:35:58,809 Node /192.168.0.17 state jump to normal
> > java.lang.OutOfMemoryError: Java heap space
> > Dumping heap to java_pid3496.hprof ...
> > Heap dump file created [7632952 bytes in 0.233 secs]
> > ERROR 12:43:19,288 Fatal exception in thread
> Thread[pool-1-thread-3,5,main]
> > java.lang.OutOfMemoryError: Java heap space
> > at
> >
> >
> org.apache.thrift.protocol.TBinaryProtocol.readStringBody(TBinaryProtocol.java:296)
> > at
> >
> >
> org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:203)
> > at
> >
> >
> org.apache.cassandra.thrift.Cassandra$Processor.process(Cassandra.java:1103)
> > at
> >
> >
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:253)
> > at
> >
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> > at
> >
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> > at java.lang.Thread.run(Thread.java:636)
> >
> > and t*his one on cass02 *
> >
> > Error connecting to remote JMX agent!
> > java.io.IOException: Failed to retrieve RMIServer stub:
> > javax.naming.CommunicationException [Root exception is
> > java.rmi.ConnectIOException: error during JRMP connection establishment;
> > nested exception is:
> > java.io.EOFException]
> > at
> > javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:342)
> > at
> >
> >
> javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:267)
> > at org.apache.cassandra.tools.NodeProbe.connect(NodeProbe.java:105)
> > at org.apache.cassandra.tools.NodeProbe.<init>(NodeProbe.java:81)
> > at org.apache.cassandra.tools.NodeCmd.main(NodeCmd.java:400)
> > Caused by: javax.naming.CommunicationException [Root exception is
> > java.rmi.ConnectIOException: error during JRMP connection establishment;
> > nested exception is:
> > java.io.EOFException]
> > at
> >
> com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:118)
> > at
> >
> >
> com.sun.jndi.toolkit.url.GenericURLContext.lookup(GenericURLContext.java:203)
> > at javax.naming.InitialContext.lookup(InitialContext.java:409)
> > at
> >
> >
> javax.management.remote.rmi.RMIConnector.findRMIServerJNDI(RMIConnector.java:1902)
> > at
> >
> >
> javax.management.remote.rmi.RMIConnector.findRMIServer(RMIConnector.java:1871)
> > at
> > javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:276)
> > ... 4 more
> > Caused by: java.rmi.ConnectIOException: error during JRMP connection
> > establishment; nested exception is:
> > java.io.EOFException
> > at
> > sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:304)
> > at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:202)
> > at sun.rmi.server.UnicastRef.newCall(UnicastRef.java:340)
> > at sun.rmi.registry.RegistryImpl_Stub.lookup(Unknown Source)
> > at
> >
> com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:114)
> > ... 9 more
> > Caused by: java.io.EOFException
> > at java.io.DataInputStream.readByte(DataInputStream.java:267)
> > at
> > sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:246)
> > ... 13 more
> >
> >
> >
> >
> > where can I get to read more about such Exceptions in cassandra ? and
> what
> > might be the reason.
> >
>
Re: Out of Memory error.
Posted by Eric Yu <su...@gmail.com>.
try this if you use default cassandra.in.sh
$ nodeprobe --host cass01 --port 8080
On Thu, Mar 11, 2010 at 3:20 PM, shirish <sh...@gmail.com> wrote:
> Hello,
>
> I managed to run two nodes on machines with IP address 192.168.0.6(cass01)
> and 192.168.0.17(cass02). After a while when i did *nodeprobe --host cass01
> --port 9160 I get this messages on cass01 *
>
> INFO 12:35:58,809 Node /192.168.0.17 state jump to normal
> java.lang.OutOfMemoryError: Java heap space
> Dumping heap to java_pid3496.hprof ...
> Heap dump file created [7632952 bytes in 0.233 secs]
> ERROR 12:43:19,288 Fatal exception in thread Thread[pool-1-thread-3,5,main]
> java.lang.OutOfMemoryError: Java heap space
> at
>
> org.apache.thrift.protocol.TBinaryProtocol.readStringBody(TBinaryProtocol.java:296)
> at
>
> org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:203)
> at
>
> org.apache.cassandra.thrift.Cassandra$Processor.process(Cassandra.java:1103)
> at
>
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:253)
> at
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:636)
>
> and t*his one on cass02 *
>
> Error connecting to remote JMX agent!
> java.io.IOException: Failed to retrieve RMIServer stub:
> javax.naming.CommunicationException [Root exception is
> java.rmi.ConnectIOException: error during JRMP connection establishment;
> nested exception is:
> java.io.EOFException]
> at
> javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:342)
> at
>
> javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:267)
> at org.apache.cassandra.tools.NodeProbe.connect(NodeProbe.java:105)
> at org.apache.cassandra.tools.NodeProbe.<init>(NodeProbe.java:81)
> at org.apache.cassandra.tools.NodeCmd.main(NodeCmd.java:400)
> Caused by: javax.naming.CommunicationException [Root exception is
> java.rmi.ConnectIOException: error during JRMP connection establishment;
> nested exception is:
> java.io.EOFException]
> at
> com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:118)
> at
>
> com.sun.jndi.toolkit.url.GenericURLContext.lookup(GenericURLContext.java:203)
> at javax.naming.InitialContext.lookup(InitialContext.java:409)
> at
>
> javax.management.remote.rmi.RMIConnector.findRMIServerJNDI(RMIConnector.java:1902)
> at
>
> javax.management.remote.rmi.RMIConnector.findRMIServer(RMIConnector.java:1871)
> at
> javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:276)
> ... 4 more
> Caused by: java.rmi.ConnectIOException: error during JRMP connection
> establishment; nested exception is:
> java.io.EOFException
> at
> sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:304)
> at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:202)
> at sun.rmi.server.UnicastRef.newCall(UnicastRef.java:340)
> at sun.rmi.registry.RegistryImpl_Stub.lookup(Unknown Source)
> at
> com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:114)
> ... 9 more
> Caused by: java.io.EOFException
> at java.io.DataInputStream.readByte(DataInputStream.java:267)
> at
> sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:246)
> ... 13 more
>
>
>
>
> where can I get to read more about such Exceptions in cassandra ? and what
> might be the reason.
>