You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Dan Larsen <da...@techba.se> on 2009/10/07 19:38:13 UTC
Help! Serious crash!
Hi guys,
I have a small setup with 3 macs running os x snow leopard and the
native java 1.6.0.something.something ;-)
I am doing a lot of inserts, without any problems and I can read the
data again.
But....
At some point, around 1G Commitlog size, cassandra starts a rise in
ram usage, untill it just stops responding - if I restart the server,
with or without -b, it will just usae 100% cpu for 10-15 mins and then
stop with this message:
Another problem, is that it seems to kill other servers too, when I
try to restart :-(
java.lang.OutOfMemoryError: Java heap space
Dumping heap to java_pid53353.hprof ...
Heap dump file created [1164687070 bytes in 89.912 secs]
ERROR - Fatal exception in thread Thread[RMI TCP Connection(idle),
5,RMI Runtime]
java.lang.OutOfMemoryError: Java heap space
at java.io.BufferedInputStream.<init>(BufferedInputStream.java:178)
at java.io.BufferedInputStream.<init>(BufferedInputStream.java:158)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0
(TCPTransport.java:679)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run
(TCPTransport.java:649)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask
(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run
(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:637)
ERROR - Fatal exception in thread Thread[main,5,main]
java.lang.OutOfMemoryError: Java heap space
ERROR - Fatal exception in thread Thread[RMI TCP Connection(idle),
5,RMI Runtime]
java.lang.OutOfMemoryError: Java heap space
at java.lang.Class.getDeclaredMethods0(Native Method)
at java.lang.Class.privateGetDeclaredMethods(Class.java:2427)
at java.lang.Class.getDeclaredMethod(Class.java:1935)
at java.io.ObjectStreamClass.getPrivateMethod(ObjectStreamClass.java:
1382)
at java.io.ObjectStreamClass.access$1700(ObjectStreamClass.java:52)
at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:438)
at java.security.AccessController.doPrivileged(Native Method)
at java.io.ObjectStreamClass.<init>(ObjectStreamClass.java:413)
at java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:310)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:
1106)
at java.io.ObjectOutputStream.defaultWriteFields
(ObjectOutputStream.java:1509)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:
1474)
at java.io.ObjectOutputStream.writeOrdinaryObject
(ObjectOutputStream.java:1392)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:
1150)
at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:326)
at java.util.ArrayList.writeObject(ArrayList.java:570)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke
(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke
(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:
945)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:
1461)
at java.io.ObjectOutputStream.writeOrdinaryObject
(ObjectOutputStream.java:1392)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:
1150)
at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:326)
at sun.rmi.server.UnicastRef.marshalValue(UnicastRef.java:274)
at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:315)
at sun.rmi.transport.Transport$1.run(Transport.java:159)
at java.security.AccessController.doPrivileged(Native Method)
at sun.rmi.transport.Transport.serviceCall(Transport.java:155)
at sun.rmi.transport.tcp.TCPTransport.handleMessages
(TCPTransport.java:535)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0
(TCPTransport.java:790)
ERROR - Fatal exception in thread Thread[RMI TCP Connection(idle),
5,RMI Runtime]
[ABOVE REPEATS A COUPLE OF TIMES]
If I try to connect with nodeprobe, after it halts, I will just get
this:
Error connecting to remote JMX agent!
java.io.IOException: Failed to retrieve RMIServer stub:
javax.naming.ServiceUnavailableException [Root exception is
java.rmi.ConnectException: Connection refused to host:
192.168.100.203; nested exception is:
java.net.ConnectException: Connection refused]
at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:
338)
at javax.management.remote.JMXConnectorFactory.connect
(JMXConnectorFactory.java:248)
at org.apache.cassandra.tools.NodeProbe.connect(NodeProbe.java:152)
at org.apache.cassandra.tools.NodeProbe.<init>(NodeProbe.java:114)
at org.apache.cassandra.tools.NodeProbe.main(NodeProbe.java:545)
Caused by: javax.naming.ServiceUnavailableException [Root exception is
java.rmi.ConnectException: Connection refused to host:
192.168.100.203; nested exception is:
java.net.ConnectException: Connection refused]
at com.sun.jndi.rmi.registry.RegistryContext.lookup
(RegistryContext.java:101)
at com.sun.jndi.toolkit.url.GenericURLContext.lookup
(GenericURLContext.java:185)
at javax.naming.InitialContext.lookup(InitialContext.java:392)
at javax.management.remote.rmi.RMIConnector.findRMIServerJNDI
(RMIConnector.java:1886)
at javax.management.remote.rmi.RMIConnector.findRMIServer
(RMIConnector.java:1856)
at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:
257)
... 4 more
Caused by: java.rmi.ConnectException: Connection refused to host:
192.168.100.203; nested exception is:
java.net.ConnectException: Connection refused
at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:601)
at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:
198)
at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:184)
at sun.rmi.server.UnicastRef.newCall(UnicastRef.java:322)
at sun.rmi.registry.RegistryImpl_Stub.lookup(Unknown Source)
at com.sun.jndi.rmi.registry.RegistryContext.lookup
(RegistryContext.java:97)
... 9 more
Caused by: java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:432)
at java.net.Socket.connect(Socket.java:525)
at java.net.Socket.connect(Socket.java:475)
at java.net.Socket.<init>(Socket.java:372)
at java.net.Socket.<init>(Socket.java:186)
at sun.rmi.transport.proxy.RMIDirectSocketFactory.createSocket
(RMIDirectSocketFactory.java:22)
at sun.rmi.transport.proxy.RMIMasterSocketFactory.createSocket
(RMIMasterSocketFactory.java:128)
at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:595)
... 14 more
I know i responds as supposed, before the crash.
Hope somebody can help! :-)
Best regards
Dan Larsen
Re: Help! Serious crash!
Posted by Dan Larsen <da...@techba.se>.
Thanks for the tip!
I'm using QUORUM - I hate losing data :-P
/Dan
On 07/10/2009, at 22.56, Jonathan Ellis wrote:
> One tip: don't use ConcurrencyLevel.ZERO when bulk inserting. It's
> easy to OOM yourself that way.
>
> -Jonathan
>
Re: Help! Serious crash!
Posted by Jonathan Ellis <jb...@gmail.com>.
One tip: don't use ConcurrencyLevel.ZERO when bulk inserting. It's
easy to OOM yourself that way.
-Jonathan
Re: Help! Serious crash!
Posted by Dan Larsen <da...@techba.se>.
I don't know why... But for some reason, the commitlogs grew bigger
than 128MB - so I suppose a manual compact was needed...?!?!
Anyways... I restarted with the 2GB value in cassandra.in.sh,
everything got compacted, the commitlog rotated - and I could stop,
change the settings back and start everything up, with no probles...
Thanks for your suggestions Igor! :-)
Best regards
Dan
On 07/10/2009, at 20.16, Dan Larsen wrote:
> Sorry about that... running 0.4 taken from trunk.
> I didn't change CommitLogRotationThresholdInMB - it's still 128
>
> Thanks!
>
> On 07/10/2009, at 20.12, Igor Katkov wrote:
>
>> What version are you running? Take 0.4 from trunk.
>> Default config has
>> <CommitLogRotationThresholdInMB>128</
>> CommitLogRotationThresholdInMB>
>> Did you change it allow your commit log chunks be 1Gb?
>>
>> On Wed, Oct 7, 2009 at 1:38 PM, Dan Larsen <da...@techba.se> wrote:
>> Hi guys,
>>
>> I have a small setup with 3 macs running os x snow leopard and the
>> native java 1.6.0.something.something ;-)
>> I am doing a lot of inserts, without any problems and I can read
>> the data again.
>>
>> But....
>> At some point, around 1G Commitlog size, cassandra starts a rise in
>> ram usage, untill it just stops responding - if I restart the
>> server, with or without -b, it will just usae 100% cpu for 10-15
>> mins and then stop with this message:
>> Another problem, is that it seems to kill other servers too, when I
>> try to restart :-(
>>
>> java.lang.OutOfMemoryError: Java heap space
>> Dumping heap to java_pid53353.hprof ...
>> Heap dump file created [1164687070 bytes in 89.912 secs]
>>
>> ERROR - Fatal exception in thread Thread[RMI TCP Connection(idle),
>> 5,RMI Runtime]
>> java.lang.OutOfMemoryError: Java heap space
>> at java.io.BufferedInputStream.<init>
>> (BufferedInputStream.java:178)
>> at java.io.BufferedInputStream.<init>
>> (BufferedInputStream.java:158)
>> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0
>> (TCPTransport.java:679)
>> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run
>> (TCPTransport.java:649)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask
>> (ThreadPoolExecutor.java:886)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run
>> (ThreadPoolExecutor.java:908)
>> at java.lang.Thread.run(Thread.java:637)
>> ERROR - Fatal exception in thread Thread[main,5,main]
>> java.lang.OutOfMemoryError: Java heap space
>> ERROR - Fatal exception in thread Thread[RMI TCP Connection(idle),
>> 5,RMI Runtime]
>> java.lang.OutOfMemoryError: Java heap space
>> at java.lang.Class.getDeclaredMethods0(Native Method)
>> at java.lang.Class.privateGetDeclaredMethods(Class.java:2427)
>> at java.lang.Class.getDeclaredMethod(Class.java:1935)
>> at java.io.ObjectStreamClass.getPrivateMethod
>> (ObjectStreamClass.java:1382)
>> at java.io.ObjectStreamClass.access$1700
>> (ObjectStreamClass.java:52)
>> at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:438)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at java.io.ObjectStreamClass.<init>(ObjectStreamClass.java:
>> 413)
>> at java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:
>> 310)
>> at java.io.ObjectOutputStream.writeObject0
>> (ObjectOutputStream.java:1106)
>> at java.io.ObjectOutputStream.defaultWriteFields
>> (ObjectOutputStream.java:1509)
>> at java.io.ObjectOutputStream.writeSerialData
>> (ObjectOutputStream.java:1474)
>> at java.io.ObjectOutputStream.writeOrdinaryObject
>> (ObjectOutputStream.java:1392)
>> at java.io.ObjectOutputStream.writeObject0
>> (ObjectOutputStream.java:1150)
>> at java.io.ObjectOutputStream.writeObject
>> (ObjectOutputStream.java:326)
>> at java.util.ArrayList.writeObject(ArrayList.java:570)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at sun.reflect.NativeMethodAccessorImpl.invoke
>> (NativeMethodAccessorImpl.java:39)
>> at sun.reflect.DelegatingMethodAccessorImpl.invoke
>> (DelegatingMethodAccessorImpl.java:25)
>> at java.lang.reflect.Method.invoke(Method.java:597)
>> at java.io.ObjectStreamClass.invokeWriteObject
>> (ObjectStreamClass.java:945)
>> at java.io.ObjectOutputStream.writeSerialData
>> (ObjectOutputStream.java:1461)
>> at java.io.ObjectOutputStream.writeOrdinaryObject
>> (ObjectOutputStream.java:1392)
>> at java.io.ObjectOutputStream.writeObject0
>> (ObjectOutputStream.java:1150)
>> at java.io.ObjectOutputStream.writeObject
>> (ObjectOutputStream.java:326)
>> at sun.rmi.server.UnicastRef.marshalValue(UnicastRef.java:274)
>> at sun.rmi.server.UnicastServerRef.dispatch
>> (UnicastServerRef.java:315)
>> at sun.rmi.transport.Transport$1.run(Transport.java:159)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at sun.rmi.transport.Transport.serviceCall(Transport.java:155)
>> at sun.rmi.transport.tcp.TCPTransport.handleMessages
>> (TCPTransport.java:535)
>> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0
>> (TCPTransport.java:790)
>> ERROR - Fatal exception in thread Thread[RMI TCP Connection(idle),
>> 5,RMI Runtime]
>>
>> [ABOVE REPEATS A COUPLE OF TIMES]
>>
>> If I try to connect with nodeprobe, after it halts, I will just get
>> this:
>>
>> Error connecting to remote JMX agent!
>> java.io.IOException: Failed to retrieve RMIServer stub:
>> javax.naming.ServiceUnavailableException [Root exception is
>> java.rmi.ConnectException: Connection refused to host:
>> 192.168.100.203; nested exception is:
>> java.net.ConnectException: Connection refused]
>> at javax.management.remote.rmi.RMIConnector.connect
>> (RMIConnector.java:338)
>> at javax.management.remote.JMXConnectorFactory.connect
>> (JMXConnectorFactory.java:248)
>> at org.apache.cassandra.tools.NodeProbe.connect
>> (NodeProbe.java:152)
>> at org.apache.cassandra.tools.NodeProbe.<init>
>> (NodeProbe.java:114)
>> at org.apache.cassandra.tools.NodeProbe.main(NodeProbe.java:
>> 545)
>> Caused by: javax.naming.ServiceUnavailableException [Root exception
>> is java.rmi.ConnectException: Connection refused to host:
>> 192.168.100.203; nested exception is:
>> java.net.ConnectException: Connection refused]
>> at com.sun.jndi.rmi.registry.RegistryContext.lookup
>> (RegistryContext.java:101)
>> at com.sun.jndi.toolkit.url.GenericURLContext.lookup
>> (GenericURLContext.java:185)
>> at javax.naming.InitialContext.lookup(InitialContext.java:392)
>> at javax.management.remote.rmi.RMIConnector.findRMIServerJNDI
>> (RMIConnector.java:1886)
>> at javax.management.remote.rmi.RMIConnector.findRMIServer
>> (RMIConnector.java:1856)
>> at javax.management.remote.rmi.RMIConnector.connect
>> (RMIConnector.java:257)
>> ... 4 more
>> Caused by: java.rmi.ConnectException: Connection refused to host:
>> 192.168.100.203; nested exception is:
>> java.net.ConnectException: Connection refused
>> at sun.rmi.transport.tcp.TCPEndpoint.newSocket
>> (TCPEndpoint.java:601)
>> at sun.rmi.transport.tcp.TCPChannel.createConnection
>> (TCPChannel.java:198)
>> at sun.rmi.transport.tcp.TCPChannel.newConnection
>> (TCPChannel.java:184)
>> at sun.rmi.server.UnicastRef.newCall(UnicastRef.java:322)
>> at sun.rmi.registry.RegistryImpl_Stub.lookup(Unknown Source)
>> at com.sun.jndi.rmi.registry.RegistryContext.lookup
>> (RegistryContext.java:97)
>> ... 9 more
>> Caused by: java.net.ConnectException: Connection refused
>> at java.net.PlainSocketImpl.socketConnect(Native Method)
>> at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:
>> 333)
>> at java.net.PlainSocketImpl.connectToAddress
>> (PlainSocketImpl.java:195)
>> at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
>> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:432)
>> at java.net.Socket.connect(Socket.java:525)
>> at java.net.Socket.connect(Socket.java:475)
>> at java.net.Socket.<init>(Socket.java:372)
>> at java.net.Socket.<init>(Socket.java:186)
>> at
>> sun.rmi.transport.proxy.RMIDirectSocketFactory.createSocket
>> (RMIDirectSocketFactory.java:22)
>> at
>> sun.rmi.transport.proxy.RMIMasterSocketFactory.createSocket
>> (RMIMasterSocketFactory.java:128)
>> at sun.rmi.transport.tcp.TCPEndpoint.newSocket
>> (TCPEndpoint.java:595)
>> ... 14 more
>>
>>
>> I know i responds as supposed, before the crash.
>>
>> Hope somebody can help! :-)
>>
>> Best regards
>> Dan Larsen
>>
>
Re: Help! Serious crash!
Posted by Dan Larsen <da...@techba.se>.
I will check that thread out! Thanks :-)
Maybe I should try to refresh the code...
I am starting to think, it might be the "native" java of snow leopard,
that is causing some of my problems...
This is what has kept me from coding Java for more than a decade...
Java is never just Java...
I know C is not just C either and C++ is not just C++... But it is so
much more true for Java...
Anyways... Not trying to start a flame, in the middle of an Apache
Incubator project... :-P
So... Thanks for you hints - I'll do some testing and post it here, if
I find out anything interesting.
Good to know, you had that much data! I'll be attempting something
similar... ;-)
On 07/10/2009, at 20.20, Igor Katkov wrote:
> I had similar OutOfMemory issues, see "commit logs are not deleted"
> thread, but it was fixed/commited to trunk.
> What I don't get is how your commit log segment grows beyond
> threshold.
>
> >Is it really nescessary to have as much RAM as you have data?!?!?
> No, I was able to insert 50Gb worth of data with 3Gb cap
>
>
> On Wed, Oct 7, 2009 at 2:16 PM, Dan Larsen <da...@techba.se> wrote:
> Sorry about that... running 0.4 taken from trunk.
> I didn't change CommitLogRotationThresholdInMB - it's still 128
>
> Thanks!
>
> On 07/10/2009, at 20.12, Igor Katkov wrote:
>
>> What version are you running? Take 0.4 from trunk.
>> Default config has
>> <CommitLogRotationThresholdInMB>128</
>> CommitLogRotationThresholdInMB>
>> Did you change it allow your commit log chunks be 1Gb?
>>
>> On Wed, Oct 7, 2009 at 1:38 PM, Dan Larsen <da...@techba.se> wrote:
>> Hi guys,
>>
>> I have a small setup with 3 macs running os x snow leopard and the
>> native java 1.6.0.something.something ;-)
>> I am doing a lot of inserts, without any problems and I can read
>> the data again.
>>
>> But....
>> At some point, around 1G Commitlog size, cassandra starts a rise in
>> ram usage, untill it just stops responding - if I restart the
>> server, with or without -b, it will just usae 100% cpu for 10-15
>> mins and then stop with this message:
>> Another problem, is that it seems to kill other servers too, when I
>> try to restart :-(
>>
>> java.lang.OutOfMemoryError: Java heap space
>> Dumping heap to java_pid53353.hprof ...
>> Heap dump file created [1164687070 bytes in 89.912 secs]
>>
>> ERROR - Fatal exception in thread Thread[RMI TCP Connection(idle),
>> 5,RMI Runtime]
>> java.lang.OutOfMemoryError: Java heap space
>> at java.io.BufferedInputStream.<init>
>> (BufferedInputStream.java:178)
>> at java.io.BufferedInputStream.<init>
>> (BufferedInputStream.java:158)
>> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0
>> (TCPTransport.java:679)
>> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run
>> (TCPTransport.java:649)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask
>> (ThreadPoolExecutor.java:886)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run
>> (ThreadPoolExecutor.java:908)
>> at java.lang.Thread.run(Thread.java:637)
>> ERROR - Fatal exception in thread Thread[main,5,main]
>> java.lang.OutOfMemoryError: Java heap space
>> ERROR - Fatal exception in thread Thread[RMI TCP Connection(idle),
>> 5,RMI Runtime]
>> java.lang.OutOfMemoryError: Java heap space
>> at java.lang.Class.getDeclaredMethods0(Native Method)
>> at java.lang.Class.privateGetDeclaredMethods(Class.java:2427)
>> at java.lang.Class.getDeclaredMethod(Class.java:1935)
>> at java.io.ObjectStreamClass.getPrivateMethod
>> (ObjectStreamClass.java:1382)
>> at java.io.ObjectStreamClass.access$1700
>> (ObjectStreamClass.java:52)
>> at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:438)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at java.io.ObjectStreamClass.<init>(ObjectStreamClass.java:
>> 413)
>> at java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:
>> 310)
>> at java.io.ObjectOutputStream.writeObject0
>> (ObjectOutputStream.java:1106)
>> at java.io.ObjectOutputStream.defaultWriteFields
>> (ObjectOutputStream.java:1509)
>> at java.io.ObjectOutputStream.writeSerialData
>> (ObjectOutputStream.java:1474)
>> at java.io.ObjectOutputStream.writeOrdinaryObject
>> (ObjectOutputStream.java:1392)
>> at java.io.ObjectOutputStream.writeObject0
>> (ObjectOutputStream.java:1150)
>> at java.io.ObjectOutputStream.writeObject
>> (ObjectOutputStream.java:326)
>> at java.util.ArrayList.writeObject(ArrayList.java:570)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at sun.reflect.NativeMethodAccessorImpl.invoke
>> (NativeMethodAccessorImpl.java:39)
>> at sun.reflect.DelegatingMethodAccessorImpl.invoke
>> (DelegatingMethodAccessorImpl.java:25)
>> at java.lang.reflect.Method.invoke(Method.java:597)
>> at java.io.ObjectStreamClass.invokeWriteObject
>> (ObjectStreamClass.java:945)
>> at java.io.ObjectOutputStream.writeSerialData
>> (ObjectOutputStream.java:1461)
>> at java.io.ObjectOutputStream.writeOrdinaryObject
>> (ObjectOutputStream.java:1392)
>> at java.io.ObjectOutputStream.writeObject0
>> (ObjectOutputStream.java:1150)
>> at java.io.ObjectOutputStream.writeObject
>> (ObjectOutputStream.java:326)
>> at sun.rmi.server.UnicastRef.marshalValue(UnicastRef.java:274)
>> at sun.rmi.server.UnicastServerRef.dispatch
>> (UnicastServerRef.java:315)
>> at sun.rmi.transport.Transport$1.run(Transport.java:159)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at sun.rmi.transport.Transport.serviceCall(Transport.java:155)
>> at sun.rmi.transport.tcp.TCPTransport.handleMessages
>> (TCPTransport.java:535)
>> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0
>> (TCPTransport.java:790)
>> ERROR - Fatal exception in thread Thread[RMI TCP Connection(idle),
>> 5,RMI Runtime]
>>
>> [ABOVE REPEATS A COUPLE OF TIMES]
>>
>> If I try to connect with nodeprobe, after it halts, I will just get
>> this:
>>
>> Error connecting to remote JMX agent!
>> java.io.IOException: Failed to retrieve RMIServer stub:
>> javax.naming.ServiceUnavailableException [Root exception is
>> java.rmi.ConnectException: Connection refused to host:
>> 192.168.100.203; nested exception is:
>> java.net.ConnectException: Connection refused]
>> at javax.management.remote.rmi.RMIConnector.connect
>> (RMIConnector.java:338)
>> at javax.management.remote.JMXConnectorFactory.connect
>> (JMXConnectorFactory.java:248)
>> at org.apache.cassandra.tools.NodeProbe.connect
>> (NodeProbe.java:152)
>> at org.apache.cassandra.tools.NodeProbe.<init>
>> (NodeProbe.java:114)
>> at org.apache.cassandra.tools.NodeProbe.main(NodeProbe.java:
>> 545)
>> Caused by: javax.naming.ServiceUnavailableException [Root exception
>> is java.rmi.ConnectException: Connection refused to host:
>> 192.168.100.203; nested exception is:
>> java.net.ConnectException: Connection refused]
>> at com.sun.jndi.rmi.registry.RegistryContext.lookup
>> (RegistryContext.java:101)
>> at com.sun.jndi.toolkit.url.GenericURLContext.lookup
>> (GenericURLContext.java:185)
>> at javax.naming.InitialContext.lookup(InitialContext.java:392)
>> at javax.management.remote.rmi.RMIConnector.findRMIServerJNDI
>> (RMIConnector.java:1886)
>> at javax.management.remote.rmi.RMIConnector.findRMIServer
>> (RMIConnector.java:1856)
>> at javax.management.remote.rmi.RMIConnector.connect
>> (RMIConnector.java:257)
>> ... 4 more
>> Caused by: java.rmi.ConnectException: Connection refused to host:
>> 192.168.100.203; nested exception is:
>> java.net.ConnectException: Connection refused
>> at sun.rmi.transport.tcp.TCPEndpoint.newSocket
>> (TCPEndpoint.java:601)
>> at sun.rmi.transport.tcp.TCPChannel.createConnection
>> (TCPChannel.java:198)
>> at sun.rmi.transport.tcp.TCPChannel.newConnection
>> (TCPChannel.java:184)
>> at sun.rmi.server.UnicastRef.newCall(UnicastRef.java:322)
>> at sun.rmi.registry.RegistryImpl_Stub.lookup(Unknown Source)
>> at com.sun.jndi.rmi.registry.RegistryContext.lookup
>> (RegistryContext.java:97)
>> ... 9 more
>> Caused by: java.net.ConnectException: Connection refused
>> at java.net.PlainSocketImpl.socketConnect(Native Method)
>> at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:
>> 333)
>> at java.net.PlainSocketImpl.connectToAddress
>> (PlainSocketImpl.java:195)
>> at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
>> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:432)
>> at java.net.Socket.connect(Socket.java:525)
>> at java.net.Socket.connect(Socket.java:475)
>> at java.net.Socket.<init>(Socket.java:372)
>> at java.net.Socket.<init>(Socket.java:186)
>> at
>> sun.rmi.transport.proxy.RMIDirectSocketFactory.createSocket
>> (RMIDirectSocketFactory.java:22)
>> at
>> sun.rmi.transport.proxy.RMIMasterSocketFactory.createSocket
>> (RMIMasterSocketFactory.java:128)
>> at sun.rmi.transport.tcp.TCPEndpoint.newSocket
>> (TCPEndpoint.java:595)
>> ... 14 more
>>
>>
>> I know i responds as supposed, before the crash.
>>
>> Hope somebody can help! :-)
>>
>> Best regards
>> Dan Larsen
>>
>
>
Re: Help! Serious crash!
Posted by Igor Katkov <ik...@gmail.com>.
I had similar OutOfMemory issues, see "commit logs are not deleted" thread,
but it was fixed/commited to trunk.
What I don't get is how your commit log segment grows beyond threshold.
>Is it really nescessary to have as much RAM as you have data?!?!?
No, I was able to insert 50Gb worth of data with 3Gb cap
On Wed, Oct 7, 2009 at 2:16 PM, Dan Larsen <da...@techba.se> wrote:
> Sorry about that... running 0.4 taken from trunk.I didn't
> change CommitLogRotationThresholdInMB - it's still 128
>
> Thanks!
>
> On 07/10/2009, at 20.12, Igor Katkov wrote:
>
> What version are you running? Take 0.4 from trunk.
> Default config has
> <CommitLogRotationThresholdInMB>128</CommitLogRotationThresholdInMB>
> Did you change it allow your commit log chunks be 1Gb?
>
> On Wed, Oct 7, 2009 at 1:38 PM, Dan Larsen <da...@techba.se> wrote:
>
>> Hi guys,
>>
>> I have a small setup with 3 macs running os x snow leopard and the native
>> java 1.6.0.something.something ;-)
>> I am doing a lot of inserts, without any problems and I can read the data
>> again.
>>
>> But....
>> At some point, around 1G Commitlog size, cassandra starts a rise in ram
>> usage, untill it just stops responding - if I restart the server, with or
>> without -b, it will just usae 100% cpu for 10-15 mins and then stop with
>> this message:
>> Another problem, is that it seems to kill other servers too, when I try to
>> restart :-(
>>
>> java.lang.OutOfMemoryError: Java heap space
>> Dumping heap to java_pid53353.hprof ...
>> Heap dump file created [1164687070 bytes in 89.912 secs]
>>
>> ERROR - Fatal exception in thread Thread[RMI TCP Connection(idle),5,RMI
>> Runtime]
>> java.lang.OutOfMemoryError: Java heap space
>> at java.io.BufferedInputStream.<init>(BufferedInputStream.java:178)
>> at java.io.BufferedInputStream.<init>(BufferedInputStream.java:158)
>> at
>> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:679)
>> at
>> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>> at java.lang.Thread.run(Thread.java:637)
>> ERROR - Fatal exception in thread Thread[main,5,main]
>> java.lang.OutOfMemoryError: Java heap space
>> ERROR - Fatal exception in thread Thread[RMI TCP Connection(idle),5,RMI
>> Runtime]
>> java.lang.OutOfMemoryError: Java heap space
>> at java.lang.Class.getDeclaredMethods0(Native Method)
>> at java.lang.Class.privateGetDeclaredMethods(Class.java:2427)
>> at java.lang.Class.getDeclaredMethod(Class.java:1935)
>> at
>> java.io.ObjectStreamClass.getPrivateMethod(ObjectStreamClass.java:1382)
>> at java.io.ObjectStreamClass.access$1700(ObjectStreamClass.java:52)
>> at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:438)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at java.io.ObjectStreamClass.<init>(ObjectStreamClass.java:413)
>> at java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:310)
>> at
>> java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1106)
>> at
>> java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1509)
>> at
>> java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1474)
>> at
>> java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1392)
>> at
>> java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1150)
>> at
>> java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:326)
>> at java.util.ArrayList.writeObject(ArrayList.java:570)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> at java.lang.reflect.Method.invoke(Method.java:597)
>> at
>> java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:945)
>> at
>> java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1461)
>> at
>> java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1392)
>> at
>> java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1150)
>> at
>> java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:326)
>> at sun.rmi.server.UnicastRef.marshalValue(UnicastRef.java:274)
>> at
>> sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:315)
>> at sun.rmi.transport.Transport$1.run(Transport.java:159)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at sun.rmi.transport.Transport.serviceCall(Transport.java:155)
>> at
>> sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:535)
>> at
>> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790)
>> ERROR - Fatal exception in thread Thread[RMI TCP Connection(idle),5,RMI
>> Runtime]
>>
>> [ABOVE REPEATS A COUPLE OF TIMES]
>>
>> If I try to connect with nodeprobe, after it halts, I will just get this:
>>
>> Error connecting to remote JMX agent!
>> java.io.IOException: Failed to retrieve RMIServer stub:
>> javax.naming.ServiceUnavailableException [Root exception is
>> java.rmi.ConnectException: Connection refused to host: 192.168.100.203;
>> nested exception is:
>> java.net.ConnectException: Connection refused]
>> at
>> javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:338)
>> at
>> javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:248)
>> at org.apache.cassandra.tools.NodeProbe.connect(NodeProbe.java:152)
>> at org.apache.cassandra.tools.NodeProbe.<init>(NodeProbe.java:114)
>> at org.apache.cassandra.tools.NodeProbe.main(NodeProbe.java:545)
>> Caused by: javax.naming.ServiceUnavailableException [Root exception is
>> java.rmi.ConnectException: Connection refused to host: 192.168.100.203;
>> nested exception is:
>> java.net.ConnectException: Connection refused]
>> at
>> com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:101)
>> at
>> com.sun.jndi.toolkit.url.GenericURLContext.lookup(GenericURLContext.java:185)
>> at javax.naming.InitialContext.lookup(InitialContext.java:392)
>> at
>> javax.management.remote.rmi.RMIConnector.findRMIServerJNDI(RMIConnector.java:1886)
>> at
>> javax.management.remote.rmi.RMIConnector.findRMIServer(RMIConnector.java:1856)
>> at
>> javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:257)
>> ... 4 more
>> Caused by: java.rmi.ConnectException: Connection refused to host:
>> 192.168.100.203; nested exception is:
>> java.net.ConnectException: Connection refused
>> at
>> sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:601)
>> at
>> sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:198)
>> at
>> sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:184)
>> at sun.rmi.server.UnicastRef.newCall(UnicastRef.java:322)
>> at sun.rmi.registry.RegistryImpl_Stub.lookup(Unknown Source)
>> at
>> com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:97)
>> ... 9 more
>> Caused by: java.net.ConnectException: Connection refused
>> at java.net.PlainSocketImpl.socketConnect(Native Method)
>> at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
>> at
>> java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
>> at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
>> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:432)
>> at java.net.Socket.connect(Socket.java:525)
>> at java.net.Socket.connect(Socket.java:475)
>> at java.net.Socket.<init>(Socket.java:372)
>> at java.net.Socket.<init>(Socket.java:186)
>> at
>> sun.rmi.transport.proxy.RMIDirectSocketFactory.createSocket(RMIDirectSocketFactory.java:22)
>> at
>> sun.rmi.transport.proxy.RMIMasterSocketFactory.createSocket(RMIMasterSocketFactory.java:128)
>> at
>> sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:595)
>> ... 14 more
>>
>>
>> I know i responds as supposed, before the crash.
>>
>> Hope somebody can help! :-)
>>
>> Best regards
>> Dan Larsen
>>
>
>
>
Re: Help! Serious crash!
Posted by Dan Larsen <da...@techba.se>.
Sorry about that... running 0.4 taken from trunk.
I didn't change CommitLogRotationThresholdInMB - it's still 128
Thanks!
On 07/10/2009, at 20.12, Igor Katkov wrote:
> What version are you running? Take 0.4 from trunk.
> Default config has
> <CommitLogRotationThresholdInMB>128</CommitLogRotationThresholdInMB>
> Did you change it allow your commit log chunks be 1Gb?
>
> On Wed, Oct 7, 2009 at 1:38 PM, Dan Larsen <da...@techba.se> wrote:
> Hi guys,
>
> I have a small setup with 3 macs running os x snow leopard and the
> native java 1.6.0.something.something ;-)
> I am doing a lot of inserts, without any problems and I can read the
> data again.
>
> But....
> At some point, around 1G Commitlog size, cassandra starts a rise in
> ram usage, untill it just stops responding - if I restart the
> server, with or without -b, it will just usae 100% cpu for 10-15
> mins and then stop with this message:
> Another problem, is that it seems to kill other servers too, when I
> try to restart :-(
>
> java.lang.OutOfMemoryError: Java heap space
> Dumping heap to java_pid53353.hprof ...
> Heap dump file created [1164687070 bytes in 89.912 secs]
>
> ERROR - Fatal exception in thread Thread[RMI TCP Connection(idle),
> 5,RMI Runtime]
> java.lang.OutOfMemoryError: Java heap space
> at java.io.BufferedInputStream.<init>
> (BufferedInputStream.java:178)
> at java.io.BufferedInputStream.<init>
> (BufferedInputStream.java:158)
> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0
> (TCPTransport.java:679)
> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run
> (TCPTransport.java:649)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask
> (ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run
> (ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:637)
> ERROR - Fatal exception in thread Thread[main,5,main]
> java.lang.OutOfMemoryError: Java heap space
> ERROR - Fatal exception in thread Thread[RMI TCP Connection(idle),
> 5,RMI Runtime]
> java.lang.OutOfMemoryError: Java heap space
> at java.lang.Class.getDeclaredMethods0(Native Method)
> at java.lang.Class.privateGetDeclaredMethods(Class.java:2427)
> at java.lang.Class.getDeclaredMethod(Class.java:1935)
> at java.io.ObjectStreamClass.getPrivateMethod
> (ObjectStreamClass.java:1382)
> at java.io.ObjectStreamClass.access$1700
> (ObjectStreamClass.java:52)
> at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:438)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.io.ObjectStreamClass.<init>(ObjectStreamClass.java:413)
> at java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:310)
> at java.io.ObjectOutputStream.writeObject0
> (ObjectOutputStream.java:1106)
> at java.io.ObjectOutputStream.defaultWriteFields
> (ObjectOutputStream.java:1509)
> at java.io.ObjectOutputStream.writeSerialData
> (ObjectOutputStream.java:1474)
> at java.io.ObjectOutputStream.writeOrdinaryObject
> (ObjectOutputStream.java:1392)
> at java.io.ObjectOutputStream.writeObject0
> (ObjectOutputStream.java:1150)
> at java.io.ObjectOutputStream.writeObject
> (ObjectOutputStream.java:326)
> at java.util.ArrayList.writeObject(ArrayList.java:570)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke
> (NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke
> (DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at java.io.ObjectStreamClass.invokeWriteObject
> (ObjectStreamClass.java:945)
> at java.io.ObjectOutputStream.writeSerialData
> (ObjectOutputStream.java:1461)
> at java.io.ObjectOutputStream.writeOrdinaryObject
> (ObjectOutputStream.java:1392)
> at java.io.ObjectOutputStream.writeObject0
> (ObjectOutputStream.java:1150)
> at java.io.ObjectOutputStream.writeObject
> (ObjectOutputStream.java:326)
> at sun.rmi.server.UnicastRef.marshalValue(UnicastRef.java:274)
> at sun.rmi.server.UnicastServerRef.dispatch
> (UnicastServerRef.java:315)
> at sun.rmi.transport.Transport$1.run(Transport.java:159)
> at java.security.AccessController.doPrivileged(Native Method)
> at sun.rmi.transport.Transport.serviceCall(Transport.java:155)
> at sun.rmi.transport.tcp.TCPTransport.handleMessages
> (TCPTransport.java:535)
> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0
> (TCPTransport.java:790)
> ERROR - Fatal exception in thread Thread[RMI TCP Connection(idle),
> 5,RMI Runtime]
>
> [ABOVE REPEATS A COUPLE OF TIMES]
>
> If I try to connect with nodeprobe, after it halts, I will just get
> this:
>
> Error connecting to remote JMX agent!
> java.io.IOException: Failed to retrieve RMIServer stub:
> javax.naming.ServiceUnavailableException [Root exception is
> java.rmi.ConnectException: Connection refused to host:
> 192.168.100.203; nested exception is:
> java.net.ConnectException: Connection refused]
> at javax.management.remote.rmi.RMIConnector.connect
> (RMIConnector.java:338)
> at javax.management.remote.JMXConnectorFactory.connect
> (JMXConnectorFactory.java:248)
> at org.apache.cassandra.tools.NodeProbe.connect
> (NodeProbe.java:152)
> at org.apache.cassandra.tools.NodeProbe.<init>(NodeProbe.java:
> 114)
> at org.apache.cassandra.tools.NodeProbe.main(NodeProbe.java:
> 545)
> Caused by: javax.naming.ServiceUnavailableException [Root exception
> is java.rmi.ConnectException: Connection refused to host:
> 192.168.100.203; nested exception is:
> java.net.ConnectException: Connection refused]
> at com.sun.jndi.rmi.registry.RegistryContext.lookup
> (RegistryContext.java:101)
> at com.sun.jndi.toolkit.url.GenericURLContext.lookup
> (GenericURLContext.java:185)
> at javax.naming.InitialContext.lookup(InitialContext.java:392)
> at javax.management.remote.rmi.RMIConnector.findRMIServerJNDI
> (RMIConnector.java:1886)
> at javax.management.remote.rmi.RMIConnector.findRMIServer
> (RMIConnector.java:1856)
> at javax.management.remote.rmi.RMIConnector.connect
> (RMIConnector.java:257)
> ... 4 more
> Caused by: java.rmi.ConnectException: Connection refused to host:
> 192.168.100.203; nested exception is:
> java.net.ConnectException: Connection refused
> at sun.rmi.transport.tcp.TCPEndpoint.newSocket
> (TCPEndpoint.java:601)
> at sun.rmi.transport.tcp.TCPChannel.createConnection
> (TCPChannel.java:198)
> at sun.rmi.transport.tcp.TCPChannel.newConnection
> (TCPChannel.java:184)
> at sun.rmi.server.UnicastRef.newCall(UnicastRef.java:322)
> at sun.rmi.registry.RegistryImpl_Stub.lookup(Unknown Source)
> at com.sun.jndi.rmi.registry.RegistryContext.lookup
> (RegistryContext.java:97)
> ... 9 more
> Caused by: java.net.ConnectException: Connection refused
> at java.net.PlainSocketImpl.socketConnect(Native Method)
> at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
> at java.net.PlainSocketImpl.connectToAddress
> (PlainSocketImpl.java:195)
> at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:432)
> at java.net.Socket.connect(Socket.java:525)
> at java.net.Socket.connect(Socket.java:475)
> at java.net.Socket.<init>(Socket.java:372)
> at java.net.Socket.<init>(Socket.java:186)
> at sun.rmi.transport.proxy.RMIDirectSocketFactory.createSocket
> (RMIDirectSocketFactory.java:22)
> at sun.rmi.transport.proxy.RMIMasterSocketFactory.createSocket
> (RMIMasterSocketFactory.java:128)
> at sun.rmi.transport.tcp.TCPEndpoint.newSocket
> (TCPEndpoint.java:595)
> ... 14 more
>
>
> I know i responds as supposed, before the crash.
>
> Hope somebody can help! :-)
>
> Best regards
> Dan Larsen
>
Re: Help! Serious crash!
Posted by Igor Katkov <ik...@gmail.com>.
What version are you running? Take 0.4 from trunk.
Default config has
<CommitLogRotationThresholdInMB>128</CommitLogRotationThresholdInMB>
Did you change it allow your commit log chunks be 1Gb?
On Wed, Oct 7, 2009 at 1:38 PM, Dan Larsen <da...@techba.se> wrote:
> Hi guys,
>
> I have a small setup with 3 macs running os x snow leopard and the native
> java 1.6.0.something.something ;-)
> I am doing a lot of inserts, without any problems and I can read the data
> again.
>
> But....
> At some point, around 1G Commitlog size, cassandra starts a rise in ram
> usage, untill it just stops responding - if I restart the server, with or
> without -b, it will just usae 100% cpu for 10-15 mins and then stop with
> this message:
> Another problem, is that it seems to kill other servers too, when I try to
> restart :-(
>
> java.lang.OutOfMemoryError: Java heap space
> Dumping heap to java_pid53353.hprof ...
> Heap dump file created [1164687070 bytes in 89.912 secs]
>
> ERROR - Fatal exception in thread Thread[RMI TCP Connection(idle),5,RMI
> Runtime]
> java.lang.OutOfMemoryError: Java heap space
> at java.io.BufferedInputStream.<init>(BufferedInputStream.java:178)
> at java.io.BufferedInputStream.<init>(BufferedInputStream.java:158)
> at
> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:679)
> at
> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:637)
> ERROR - Fatal exception in thread Thread[main,5,main]
> java.lang.OutOfMemoryError: Java heap space
> ERROR - Fatal exception in thread Thread[RMI TCP Connection(idle),5,RMI
> Runtime]
> java.lang.OutOfMemoryError: Java heap space
> at java.lang.Class.getDeclaredMethods0(Native Method)
> at java.lang.Class.privateGetDeclaredMethods(Class.java:2427)
> at java.lang.Class.getDeclaredMethod(Class.java:1935)
> at
> java.io.ObjectStreamClass.getPrivateMethod(ObjectStreamClass.java:1382)
> at java.io.ObjectStreamClass.access$1700(ObjectStreamClass.java:52)
> at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:438)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.io.ObjectStreamClass.<init>(ObjectStreamClass.java:413)
> at java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:310)
> at
> java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1106)
> at
> java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1509)
> at
> java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1474)
> at
> java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1392)
> at
> java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1150)
> at
> java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:326)
> at java.util.ArrayList.writeObject(ArrayList.java:570)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:945)
> at
> java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1461)
> at
> java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1392)
> at
> java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1150)
> at
> java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:326)
> at sun.rmi.server.UnicastRef.marshalValue(UnicastRef.java:274)
> at
> sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:315)
> at sun.rmi.transport.Transport$1.run(Transport.java:159)
> at java.security.AccessController.doPrivileged(Native Method)
> at sun.rmi.transport.Transport.serviceCall(Transport.java:155)
> at
> sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:535)
> at
> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790)
> ERROR - Fatal exception in thread Thread[RMI TCP Connection(idle),5,RMI
> Runtime]
>
> [ABOVE REPEATS A COUPLE OF TIMES]
>
> If I try to connect with nodeprobe, after it halts, I will just get this:
>
> Error connecting to remote JMX agent!
> java.io.IOException: Failed to retrieve RMIServer stub:
> javax.naming.ServiceUnavailableException [Root exception is
> java.rmi.ConnectException: Connection refused to host: 192.168.100.203;
> nested exception is:
> java.net.ConnectException: Connection refused]
> at
> javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:338)
> at
> javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:248)
> at org.apache.cassandra.tools.NodeProbe.connect(NodeProbe.java:152)
> at org.apache.cassandra.tools.NodeProbe.<init>(NodeProbe.java:114)
> at org.apache.cassandra.tools.NodeProbe.main(NodeProbe.java:545)
> Caused by: javax.naming.ServiceUnavailableException [Root exception is
> java.rmi.ConnectException: Connection refused to host: 192.168.100.203;
> nested exception is:
> java.net.ConnectException: Connection refused]
> at
> com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:101)
> at
> com.sun.jndi.toolkit.url.GenericURLContext.lookup(GenericURLContext.java:185)
> at javax.naming.InitialContext.lookup(InitialContext.java:392)
> at
> javax.management.remote.rmi.RMIConnector.findRMIServerJNDI(RMIConnector.java:1886)
> at
> javax.management.remote.rmi.RMIConnector.findRMIServer(RMIConnector.java:1856)
> at
> javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:257)
> ... 4 more
> Caused by: java.rmi.ConnectException: Connection refused to host:
> 192.168.100.203; nested exception is:
> java.net.ConnectException: Connection refused
> at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:601)
> at
> sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:198)
> at
> sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:184)
> at sun.rmi.server.UnicastRef.newCall(UnicastRef.java:322)
> at sun.rmi.registry.RegistryImpl_Stub.lookup(Unknown Source)
> at
> com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:97)
> ... 9 more
> Caused by: java.net.ConnectException: Connection refused
> at java.net.PlainSocketImpl.socketConnect(Native Method)
> at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
> at
> java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
> at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:432)
> at java.net.Socket.connect(Socket.java:525)
> at java.net.Socket.connect(Socket.java:475)
> at java.net.Socket.<init>(Socket.java:372)
> at java.net.Socket.<init>(Socket.java:186)
> at
> sun.rmi.transport.proxy.RMIDirectSocketFactory.createSocket(RMIDirectSocketFactory.java:22)
> at
> sun.rmi.transport.proxy.RMIMasterSocketFactory.createSocket(RMIMasterSocketFactory.java:128)
> at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:595)
> ... 14 more
>
>
> I know i responds as supposed, before the crash.
>
> Hope somebody can help! :-)
>
> Best regards
> Dan Larsen
>
Re: Help! Serious crash!
Posted by Dan Larsen <da...@techba.se>.
O.k... I experimented a bit: changed "-Xmx1G \" to "-Xmx2G \" in bin/
cassandra.in.sh ...
Is it really nescessary to have as much RAM as you have data?!?!?
Best regards
Dan
On 07/10/2009, at 19.50, Dan Larsen wrote:
> Forgot to say - It will eat up 1.13GB of RAM and 1.32GB VRAM, and
> still throw and OutOfMemory, though 700MB left...
>
> On 07/10/2009, at 19.38, Dan Larsen wrote:
>
>> Hi guys,
>>
>> I have a small setup with 3 macs running os x snow leopard and the
>> native java 1.6.0.something.something ;-)
>> I am doing a lot of inserts, without any problems and I can read
>> the data again.
>>
>> But....
>> At some point, around 1G Commitlog size, cassandra starts a rise in
>> ram usage, untill it just stops responding - if I restart the
>> server, with or without -b, it will just usae 100% cpu for 10-15
>> mins and then stop with this message:
>> Another problem, is that it seems to kill other servers too, when I
>> try to restart :-(
>>
>> java.lang.OutOfMemoryError: Java heap space
>> Dumping heap to java_pid53353.hprof ...
>> Heap dump file created [1164687070 bytes in 89.912 secs]
>>
>> ERROR - Fatal exception in thread Thread[RMI TCP Connection(idle),
>> 5,RMI Runtime]
>> java.lang.OutOfMemoryError: Java heap space
>> at java.io.BufferedInputStream.<init>(BufferedInputStream.java:178)
>> at java.io.BufferedInputStream.<init>(BufferedInputStream.java:158)
>> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0
>> (TCPTransport.java:679)
>> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run
>> (TCPTransport.java:649)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask
>> (ThreadPoolExecutor.java:886)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run
>> (ThreadPoolExecutor.java:908)
>> at java.lang.Thread.run(Thread.java:637)
>> ERROR - Fatal exception in thread Thread[main,5,main]
>> java.lang.OutOfMemoryError: Java heap space
>> ERROR - Fatal exception in thread Thread[RMI TCP Connection(idle),
>> 5,RMI Runtime]
>> java.lang.OutOfMemoryError: Java heap space
>> at java.lang.Class.getDeclaredMethods0(Native Method)
>> at java.lang.Class.privateGetDeclaredMethods(Class.java:2427)
>> at java.lang.Class.getDeclaredMethod(Class.java:1935)
>> at java.io.ObjectStreamClass.getPrivateMethod
>> (ObjectStreamClass.java:1382)
>> at java.io.ObjectStreamClass.access$1700(ObjectStreamClass.java:52)
>> at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:438)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at java.io.ObjectStreamClass.<init>(ObjectStreamClass.java:413)
>> at java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:310)
>> at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:
>> 1106)
>> at java.io.ObjectOutputStream.defaultWriteFields
>> (ObjectOutputStream.java:1509)
>> at java.io.ObjectOutputStream.writeSerialData
>> (ObjectOutputStream.java:1474)
>> at java.io.ObjectOutputStream.writeOrdinaryObject
>> (ObjectOutputStream.java:1392)
>> at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:
>> 1150)
>> at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:
>> 326)
>> at java.util.ArrayList.writeObject(ArrayList.java:570)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at sun.reflect.NativeMethodAccessorImpl.invoke
>> (NativeMethodAccessorImpl.java:39)
>> at sun.reflect.DelegatingMethodAccessorImpl.invoke
>> (DelegatingMethodAccessorImpl.java:25)
>> at java.lang.reflect.Method.invoke(Method.java:597)
>> at java.io.ObjectStreamClass.invokeWriteObject
>> (ObjectStreamClass.java:945)
>> at java.io.ObjectOutputStream.writeSerialData
>> (ObjectOutputStream.java:1461)
>> at java.io.ObjectOutputStream.writeOrdinaryObject
>> (ObjectOutputStream.java:1392)
>> at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:
>> 1150)
>> at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:
>> 326)
>> at sun.rmi.server.UnicastRef.marshalValue(UnicastRef.java:274)
>> at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:
>> 315)
>> at sun.rmi.transport.Transport$1.run(Transport.java:159)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at sun.rmi.transport.Transport.serviceCall(Transport.java:155)
>> at sun.rmi.transport.tcp.TCPTransport.handleMessages
>> (TCPTransport.java:535)
>> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0
>> (TCPTransport.java:790)
>> ERROR - Fatal exception in thread Thread[RMI TCP Connection(idle),
>> 5,RMI Runtime]
>>
>> [ABOVE REPEATS A COUPLE OF TIMES]
>>
>> If I try to connect with nodeprobe, after it halts, I will just get
>> this:
>>
>> Error connecting to remote JMX agent!
>> java.io.IOException: Failed to retrieve RMIServer stub:
>> javax.naming.ServiceUnavailableException [Root exception is
>> java.rmi.ConnectException: Connection refused to host:
>> 192.168.100.203; nested exception is:
>> java.net.ConnectException: Connection refused]
>> at javax.management.remote.rmi.RMIConnector.connect
>> (RMIConnector.java:338)
>> at javax.management.remote.JMXConnectorFactory.connect
>> (JMXConnectorFactory.java:248)
>> at org.apache.cassandra.tools.NodeProbe.connect(NodeProbe.java:152)
>> at org.apache.cassandra.tools.NodeProbe.<init>(NodeProbe.java:114)
>> at org.apache.cassandra.tools.NodeProbe.main(NodeProbe.java:545)
>> Caused by: javax.naming.ServiceUnavailableException [Root exception
>> is java.rmi.ConnectException: Connection refused to host:
>> 192.168.100.203; nested exception is:
>> java.net.ConnectException: Connection refused]
>> at com.sun.jndi.rmi.registry.RegistryContext.lookup
>> (RegistryContext.java:101)
>> at com.sun.jndi.toolkit.url.GenericURLContext.lookup
>> (GenericURLContext.java:185)
>> at javax.naming.InitialContext.lookup(InitialContext.java:392)
>> at javax.management.remote.rmi.RMIConnector.findRMIServerJNDI
>> (RMIConnector.java:1886)
>> at javax.management.remote.rmi.RMIConnector.findRMIServer
>> (RMIConnector.java:1856)
>> at javax.management.remote.rmi.RMIConnector.connect
>> (RMIConnector.java:257)
>> ... 4 more
>> Caused by: java.rmi.ConnectException: Connection refused to host:
>> 192.168.100.203; nested exception is:
>> java.net.ConnectException: Connection refused
>> at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:601)
>> at sun.rmi.transport.tcp.TCPChannel.createConnection
>> (TCPChannel.java:198)
>> at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:
>> 184)
>> at sun.rmi.server.UnicastRef.newCall(UnicastRef.java:322)
>> at sun.rmi.registry.RegistryImpl_Stub.lookup(Unknown Source)
>> at com.sun.jndi.rmi.registry.RegistryContext.lookup
>> (RegistryContext.java:97)
>> ... 9 more
>> Caused by: java.net.ConnectException: Connection refused
>> at java.net.PlainSocketImpl.socketConnect(Native Method)
>> at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
>> at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:
>> 195)
>> at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
>> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:432)
>> at java.net.Socket.connect(Socket.java:525)
>> at java.net.Socket.connect(Socket.java:475)
>> at java.net.Socket.<init>(Socket.java:372)
>> at java.net.Socket.<init>(Socket.java:186)
>> at sun.rmi.transport.proxy.RMIDirectSocketFactory.createSocket
>> (RMIDirectSocketFactory.java:22)
>> at sun.rmi.transport.proxy.RMIMasterSocketFactory.createSocket
>> (RMIMasterSocketFactory.java:128)
>> at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:595)
>> ... 14 more
>>
>>
>> I know i responds as supposed, before the crash.
>>
>> Hope somebody can help! :-)
>>
>> Best regards
>> Dan Larsen
>>
>
>
Re: Help! Serious crash!
Posted by Dan Larsen <da...@techba.se>.
Forgot to say - It will eat up 1.13GB of RAM and 1.32GB VRAM, and
still throw and OutOfMemory, though 700MB left...
On 07/10/2009, at 19.38, Dan Larsen wrote:
> Hi guys,
>
> I have a small setup with 3 macs running os x snow leopard and the
> native java 1.6.0.something.something ;-)
> I am doing a lot of inserts, without any problems and I can read the
> data again.
>
> But....
> At some point, around 1G Commitlog size, cassandra starts a rise in
> ram usage, untill it just stops responding - if I restart the
> server, with or without -b, it will just usae 100% cpu for 10-15
> mins and then stop with this message:
> Another problem, is that it seems to kill other servers too, when I
> try to restart :-(
>
> java.lang.OutOfMemoryError: Java heap space
> Dumping heap to java_pid53353.hprof ...
> Heap dump file created [1164687070 bytes in 89.912 secs]
>
> ERROR - Fatal exception in thread Thread[RMI TCP Connection(idle),
> 5,RMI Runtime]
> java.lang.OutOfMemoryError: Java heap space
> at java.io.BufferedInputStream.<init>(BufferedInputStream.java:178)
> at java.io.BufferedInputStream.<init>(BufferedInputStream.java:158)
> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0
> (TCPTransport.java:679)
> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run
> (TCPTransport.java:649)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask
> (ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run
> (ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:637)
> ERROR - Fatal exception in thread Thread[main,5,main]
> java.lang.OutOfMemoryError: Java heap space
> ERROR - Fatal exception in thread Thread[RMI TCP Connection(idle),
> 5,RMI Runtime]
> java.lang.OutOfMemoryError: Java heap space
> at java.lang.Class.getDeclaredMethods0(Native Method)
> at java.lang.Class.privateGetDeclaredMethods(Class.java:2427)
> at java.lang.Class.getDeclaredMethod(Class.java:1935)
> at java.io.ObjectStreamClass.getPrivateMethod
> (ObjectStreamClass.java:1382)
> at java.io.ObjectStreamClass.access$1700(ObjectStreamClass.java:52)
> at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:438)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.io.ObjectStreamClass.<init>(ObjectStreamClass.java:413)
> at java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:310)
> at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:
> 1106)
> at java.io.ObjectOutputStream.defaultWriteFields
> (ObjectOutputStream.java:1509)
> at java.io.ObjectOutputStream.writeSerialData
> (ObjectOutputStream.java:1474)
> at java.io.ObjectOutputStream.writeOrdinaryObject
> (ObjectOutputStream.java:1392)
> at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:
> 1150)
> at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:
> 326)
> at java.util.ArrayList.writeObject(ArrayList.java:570)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke
> (NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke
> (DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at java.io.ObjectStreamClass.invokeWriteObject
> (ObjectStreamClass.java:945)
> at java.io.ObjectOutputStream.writeSerialData
> (ObjectOutputStream.java:1461)
> at java.io.ObjectOutputStream.writeOrdinaryObject
> (ObjectOutputStream.java:1392)
> at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:
> 1150)
> at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:
> 326)
> at sun.rmi.server.UnicastRef.marshalValue(UnicastRef.java:274)
> at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:
> 315)
> at sun.rmi.transport.Transport$1.run(Transport.java:159)
> at java.security.AccessController.doPrivileged(Native Method)
> at sun.rmi.transport.Transport.serviceCall(Transport.java:155)
> at sun.rmi.transport.tcp.TCPTransport.handleMessages
> (TCPTransport.java:535)
> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0
> (TCPTransport.java:790)
> ERROR - Fatal exception in thread Thread[RMI TCP Connection(idle),
> 5,RMI Runtime]
>
> [ABOVE REPEATS A COUPLE OF TIMES]
>
> If I try to connect with nodeprobe, after it halts, I will just get
> this:
>
> Error connecting to remote JMX agent!
> java.io.IOException: Failed to retrieve RMIServer stub:
> javax.naming.ServiceUnavailableException [Root exception is
> java.rmi.ConnectException: Connection refused to host:
> 192.168.100.203; nested exception is:
> java.net.ConnectException: Connection refused]
> at javax.management.remote.rmi.RMIConnector.connect
> (RMIConnector.java:338)
> at javax.management.remote.JMXConnectorFactory.connect
> (JMXConnectorFactory.java:248)
> at org.apache.cassandra.tools.NodeProbe.connect(NodeProbe.java:152)
> at org.apache.cassandra.tools.NodeProbe.<init>(NodeProbe.java:114)
> at org.apache.cassandra.tools.NodeProbe.main(NodeProbe.java:545)
> Caused by: javax.naming.ServiceUnavailableException [Root exception
> is java.rmi.ConnectException: Connection refused to host:
> 192.168.100.203; nested exception is:
> java.net.ConnectException: Connection refused]
> at com.sun.jndi.rmi.registry.RegistryContext.lookup
> (RegistryContext.java:101)
> at com.sun.jndi.toolkit.url.GenericURLContext.lookup
> (GenericURLContext.java:185)
> at javax.naming.InitialContext.lookup(InitialContext.java:392)
> at javax.management.remote.rmi.RMIConnector.findRMIServerJNDI
> (RMIConnector.java:1886)
> at javax.management.remote.rmi.RMIConnector.findRMIServer
> (RMIConnector.java:1856)
> at javax.management.remote.rmi.RMIConnector.connect
> (RMIConnector.java:257)
> ... 4 more
> Caused by: java.rmi.ConnectException: Connection refused to host:
> 192.168.100.203; nested exception is:
> java.net.ConnectException: Connection refused
> at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:601)
> at sun.rmi.transport.tcp.TCPChannel.createConnection
> (TCPChannel.java:198)
> at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:
> 184)
> at sun.rmi.server.UnicastRef.newCall(UnicastRef.java:322)
> at sun.rmi.registry.RegistryImpl_Stub.lookup(Unknown Source)
> at com.sun.jndi.rmi.registry.RegistryContext.lookup
> (RegistryContext.java:97)
> ... 9 more
> Caused by: java.net.ConnectException: Connection refused
> at java.net.PlainSocketImpl.socketConnect(Native Method)
> at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
> at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:
> 195)
> at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:432)
> at java.net.Socket.connect(Socket.java:525)
> at java.net.Socket.connect(Socket.java:475)
> at java.net.Socket.<init>(Socket.java:372)
> at java.net.Socket.<init>(Socket.java:186)
> at sun.rmi.transport.proxy.RMIDirectSocketFactory.createSocket
> (RMIDirectSocketFactory.java:22)
> at sun.rmi.transport.proxy.RMIMasterSocketFactory.createSocket
> (RMIMasterSocketFactory.java:128)
> at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:595)
> ... 14 more
>
>
> I know i responds as supposed, before the crash.
>
> Hope somebody can help! :-)
>
> Best regards
> Dan Larsen
>