You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Igor Katkov <ik...@gmail.com> on 2009/10/02 00:51:08 UTC
Storage proxy write latency is too high
Hi,
I have the following puzzle:
Storage proxy write latency ~235ms
CF write latency <1 ms
I have 3 nodes in the cluster, Cassandra v.0.4. Tokens evenly distributed.
The client connects to a node and inserts a key with ConsistencyLevel.ONE
If it happen to be a local write operation is fast, same speed as in one
node setup. JMX shows write latency <1 ms
If it happens to be a remote insert StorageProxy sends it to a proper node.
This operation is slow. JMX shows write latency ~ 235ms.
In the same time, on remote node JMX shows same <1ms write latency. So it's
not remote node being sluggish, it's something else.
There are no pending tasks on remote node - JMX counters are always zero,
network is 1Gb, idle. So I can't blame it.
I profiled Cassandra server in JProfiler, could not find a thing. All this
extra time is spent inside QuorumResponseHandler waiting for the condition
to signal. Which should happen as soon as response is received.
There is one pooled TCP connection open to remote host. Hardly a bottleneck,
ThreadPoolExecutors looks OK.
Any ideas why write latency it is so high?
Re: Storage proxy write latency is too high
Posted by Jonathan Ellis <jb...@gmail.com>.
Very interesting. Thanks for figuring that out.
-Jonathan
On Tue, Oct 6, 2009 at 11:14 AM, Igor Katkov <ik...@gmail.com> wrote:
> I think I finally found what. It's implementation of Java NIOon Windows (JVM
> 1.6.0.16, 64b on Windows 2003)
> The very same code, same network but CentOS linux gives almost 4x
> performance. (in Cassandra@linux -> Cassandra@Windows setup)
> I don't have another linux box to test (Cassandra@linux -> Cassandra@linux)
> performance, but expect it to be even better.
>
> A lesson learnt: don't use windows.
>
> P.S.
> Here at Viigo we also learnt the hard way that async IO is also broken in
> .NET (C#). Now I start to wonder if there is some fundamental flaw in async
> IO on windows...
>
> On Mon, Oct 5, 2009 at 3:23 PM, Jonathan Ellis <jb...@gmail.com> wrote:
>>
>> On Mon, Oct 5, 2009 at 2:17 PM, Igor Katkov <ik...@gmail.com> wrote:
>> > measured via JMX console i.e. does not include client-cassandra-client
>> > latency
>> >
>> > 20 client threads 176975b value StorageProxy.WriteLatency ~660ms
>> > 10 client threads 176975b value StorageProxy.WriteLatency ~350ms
>> > 05 client threads 176975b value StorageProxy.WriteLatency ~156ms
>>
>> this is going up basically linearly with amount of (data x clients),
>> so clearly something is getting saturated.
>
>
Re: Help! Serious crash!
Posted by Dan Larsen <da...@techba.se>.
Thanks for the tip!
I'm using QUORUM - I hate losing data :-P
/Dan
On 07/10/2009, at 22.56, Jonathan Ellis wrote:
> One tip: don't use ConcurrencyLevel.ZERO when bulk inserting. It's
> easy to OOM yourself that way.
>
> -Jonathan
>
Re: Help! Serious crash!
Posted by Jonathan Ellis <jb...@gmail.com>.
One tip: don't use ConcurrencyLevel.ZERO when bulk inserting. It's
easy to OOM yourself that way.
-Jonathan
Re: Help! Serious crash!
Posted by Dan Larsen <da...@techba.se>.
I don't know why... But for some reason, the commitlogs grew bigger
than 128MB - so I suppose a manual compact was needed...?!?!
Anyways... I restarted with the 2GB value in cassandra.in.sh,
everything got compacted, the commitlog rotated - and I could stop,
change the settings back and start everything up, with no probles...
Thanks for your suggestions Igor! :-)
Best regards
Dan
On 07/10/2009, at 20.16, Dan Larsen wrote:
> Sorry about that... running 0.4 taken from trunk.
> I didn't change CommitLogRotationThresholdInMB - it's still 128
>
> Thanks!
>
> On 07/10/2009, at 20.12, Igor Katkov wrote:
>
>> What version are you running? Take 0.4 from trunk.
>> Default config has
>> <CommitLogRotationThresholdInMB>128</
>> CommitLogRotationThresholdInMB>
>> Did you change it allow your commit log chunks be 1Gb?
>>
>> On Wed, Oct 7, 2009 at 1:38 PM, Dan Larsen <da...@techba.se> wrote:
>> Hi guys,
>>
>> I have a small setup with 3 macs running os x snow leopard and the
>> native java 1.6.0.something.something ;-)
>> I am doing a lot of inserts, without any problems and I can read
>> the data again.
>>
>> But....
>> At some point, around 1G Commitlog size, cassandra starts a rise in
>> ram usage, untill it just stops responding - if I restart the
>> server, with or without -b, it will just usae 100% cpu for 10-15
>> mins and then stop with this message:
>> Another problem, is that it seems to kill other servers too, when I
>> try to restart :-(
>>
>> java.lang.OutOfMemoryError: Java heap space
>> Dumping heap to java_pid53353.hprof ...
>> Heap dump file created [1164687070 bytes in 89.912 secs]
>>
>> ERROR - Fatal exception in thread Thread[RMI TCP Connection(idle),
>> 5,RMI Runtime]
>> java.lang.OutOfMemoryError: Java heap space
>> at java.io.BufferedInputStream.<init>
>> (BufferedInputStream.java:178)
>> at java.io.BufferedInputStream.<init>
>> (BufferedInputStream.java:158)
>> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0
>> (TCPTransport.java:679)
>> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run
>> (TCPTransport.java:649)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask
>> (ThreadPoolExecutor.java:886)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run
>> (ThreadPoolExecutor.java:908)
>> at java.lang.Thread.run(Thread.java:637)
>> ERROR - Fatal exception in thread Thread[main,5,main]
>> java.lang.OutOfMemoryError: Java heap space
>> ERROR - Fatal exception in thread Thread[RMI TCP Connection(idle),
>> 5,RMI Runtime]
>> java.lang.OutOfMemoryError: Java heap space
>> at java.lang.Class.getDeclaredMethods0(Native Method)
>> at java.lang.Class.privateGetDeclaredMethods(Class.java:2427)
>> at java.lang.Class.getDeclaredMethod(Class.java:1935)
>> at java.io.ObjectStreamClass.getPrivateMethod
>> (ObjectStreamClass.java:1382)
>> at java.io.ObjectStreamClass.access$1700
>> (ObjectStreamClass.java:52)
>> at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:438)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at java.io.ObjectStreamClass.<init>(ObjectStreamClass.java:
>> 413)
>> at java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:
>> 310)
>> at java.io.ObjectOutputStream.writeObject0
>> (ObjectOutputStream.java:1106)
>> at java.io.ObjectOutputStream.defaultWriteFields
>> (ObjectOutputStream.java:1509)
>> at java.io.ObjectOutputStream.writeSerialData
>> (ObjectOutputStream.java:1474)
>> at java.io.ObjectOutputStream.writeOrdinaryObject
>> (ObjectOutputStream.java:1392)
>> at java.io.ObjectOutputStream.writeObject0
>> (ObjectOutputStream.java:1150)
>> at java.io.ObjectOutputStream.writeObject
>> (ObjectOutputStream.java:326)
>> at java.util.ArrayList.writeObject(ArrayList.java:570)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at sun.reflect.NativeMethodAccessorImpl.invoke
>> (NativeMethodAccessorImpl.java:39)
>> at sun.reflect.DelegatingMethodAccessorImpl.invoke
>> (DelegatingMethodAccessorImpl.java:25)
>> at java.lang.reflect.Method.invoke(Method.java:597)
>> at java.io.ObjectStreamClass.invokeWriteObject
>> (ObjectStreamClass.java:945)
>> at java.io.ObjectOutputStream.writeSerialData
>> (ObjectOutputStream.java:1461)
>> at java.io.ObjectOutputStream.writeOrdinaryObject
>> (ObjectOutputStream.java:1392)
>> at java.io.ObjectOutputStream.writeObject0
>> (ObjectOutputStream.java:1150)
>> at java.io.ObjectOutputStream.writeObject
>> (ObjectOutputStream.java:326)
>> at sun.rmi.server.UnicastRef.marshalValue(UnicastRef.java:274)
>> at sun.rmi.server.UnicastServerRef.dispatch
>> (UnicastServerRef.java:315)
>> at sun.rmi.transport.Transport$1.run(Transport.java:159)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at sun.rmi.transport.Transport.serviceCall(Transport.java:155)
>> at sun.rmi.transport.tcp.TCPTransport.handleMessages
>> (TCPTransport.java:535)
>> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0
>> (TCPTransport.java:790)
>> ERROR - Fatal exception in thread Thread[RMI TCP Connection(idle),
>> 5,RMI Runtime]
>>
>> [ABOVE REPEATS A COUPLE OF TIMES]
>>
>> If I try to connect with nodeprobe, after it halts, I will just get
>> this:
>>
>> Error connecting to remote JMX agent!
>> java.io.IOException: Failed to retrieve RMIServer stub:
>> javax.naming.ServiceUnavailableException [Root exception is
>> java.rmi.ConnectException: Connection refused to host:
>> 192.168.100.203; nested exception is:
>> java.net.ConnectException: Connection refused]
>> at javax.management.remote.rmi.RMIConnector.connect
>> (RMIConnector.java:338)
>> at javax.management.remote.JMXConnectorFactory.connect
>> (JMXConnectorFactory.java:248)
>> at org.apache.cassandra.tools.NodeProbe.connect
>> (NodeProbe.java:152)
>> at org.apache.cassandra.tools.NodeProbe.<init>
>> (NodeProbe.java:114)
>> at org.apache.cassandra.tools.NodeProbe.main(NodeProbe.java:
>> 545)
>> Caused by: javax.naming.ServiceUnavailableException [Root exception
>> is java.rmi.ConnectException: Connection refused to host:
>> 192.168.100.203; nested exception is:
>> java.net.ConnectException: Connection refused]
>> at com.sun.jndi.rmi.registry.RegistryContext.lookup
>> (RegistryContext.java:101)
>> at com.sun.jndi.toolkit.url.GenericURLContext.lookup
>> (GenericURLContext.java:185)
>> at javax.naming.InitialContext.lookup(InitialContext.java:392)
>> at javax.management.remote.rmi.RMIConnector.findRMIServerJNDI
>> (RMIConnector.java:1886)
>> at javax.management.remote.rmi.RMIConnector.findRMIServer
>> (RMIConnector.java:1856)
>> at javax.management.remote.rmi.RMIConnector.connect
>> (RMIConnector.java:257)
>> ... 4 more
>> Caused by: java.rmi.ConnectException: Connection refused to host:
>> 192.168.100.203; nested exception is:
>> java.net.ConnectException: Connection refused
>> at sun.rmi.transport.tcp.TCPEndpoint.newSocket
>> (TCPEndpoint.java:601)
>> at sun.rmi.transport.tcp.TCPChannel.createConnection
>> (TCPChannel.java:198)
>> at sun.rmi.transport.tcp.TCPChannel.newConnection
>> (TCPChannel.java:184)
>> at sun.rmi.server.UnicastRef.newCall(UnicastRef.java:322)
>> at sun.rmi.registry.RegistryImpl_Stub.lookup(Unknown Source)
>> at com.sun.jndi.rmi.registry.RegistryContext.lookup
>> (RegistryContext.java:97)
>> ... 9 more
>> Caused by: java.net.ConnectException: Connection refused
>> at java.net.PlainSocketImpl.socketConnect(Native Method)
>> at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:
>> 333)
>> at java.net.PlainSocketImpl.connectToAddress
>> (PlainSocketImpl.java:195)
>> at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
>> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:432)
>> at java.net.Socket.connect(Socket.java:525)
>> at java.net.Socket.connect(Socket.java:475)
>> at java.net.Socket.<init>(Socket.java:372)
>> at java.net.Socket.<init>(Socket.java:186)
>> at
>> sun.rmi.transport.proxy.RMIDirectSocketFactory.createSocket
>> (RMIDirectSocketFactory.java:22)
>> at
>> sun.rmi.transport.proxy.RMIMasterSocketFactory.createSocket
>> (RMIMasterSocketFactory.java:128)
>> at sun.rmi.transport.tcp.TCPEndpoint.newSocket
>> (TCPEndpoint.java:595)
>> ... 14 more
>>
>>
>> I know i responds as supposed, before the crash.
>>
>> Hope somebody can help! :-)
>>
>> Best regards
>> Dan Larsen
>>
>
Re: Help! Serious crash!
Posted by Dan Larsen <da...@techba.se>.
I will check that thread out! Thanks :-)
Maybe I should try to refresh the code...
I am starting to think, it might be the "native" java of snow leopard,
that is causing some of my problems...
This is what has kept me from coding Java for more than a decade...
Java is never just Java...
I know C is not just C either and C++ is not just C++... But it is so
much more true for Java...
Anyways... Not trying to start a flame, in the middle of an Apache
Incubator project... :-P
So... Thanks for you hints - I'll do some testing and post it here, if
I find out anything interesting.
Good to know, you had that much data! I'll be attempting something
similar... ;-)
On 07/10/2009, at 20.20, Igor Katkov wrote:
> I had similar OutOfMemory issues, see "commit logs are not deleted"
> thread, but it was fixed/commited to trunk.
> What I don't get is how your commit log segment grows beyond
> threshold.
>
> >Is it really nescessary to have as much RAM as you have data?!?!?
> No, I was able to insert 50Gb worth of data with 3Gb cap
>
>
> On Wed, Oct 7, 2009 at 2:16 PM, Dan Larsen <da...@techba.se> wrote:
> Sorry about that... running 0.4 taken from trunk.
> I didn't change CommitLogRotationThresholdInMB - it's still 128
>
> Thanks!
>
> On 07/10/2009, at 20.12, Igor Katkov wrote:
>
>> What version are you running? Take 0.4 from trunk.
>> Default config has
>> <CommitLogRotationThresholdInMB>128</
>> CommitLogRotationThresholdInMB>
>> Did you change it allow your commit log chunks be 1Gb?
>>
>> On Wed, Oct 7, 2009 at 1:38 PM, Dan Larsen <da...@techba.se> wrote:
>> Hi guys,
>>
>> I have a small setup with 3 macs running os x snow leopard and the
>> native java 1.6.0.something.something ;-)
>> I am doing a lot of inserts, without any problems and I can read
>> the data again.
>>
>> But....
>> At some point, around 1G Commitlog size, cassandra starts a rise in
>> ram usage, untill it just stops responding - if I restart the
>> server, with or without -b, it will just usae 100% cpu for 10-15
>> mins and then stop with this message:
>> Another problem, is that it seems to kill other servers too, when I
>> try to restart :-(
>>
>> java.lang.OutOfMemoryError: Java heap space
>> Dumping heap to java_pid53353.hprof ...
>> Heap dump file created [1164687070 bytes in 89.912 secs]
>>
>> ERROR - Fatal exception in thread Thread[RMI TCP Connection(idle),
>> 5,RMI Runtime]
>> java.lang.OutOfMemoryError: Java heap space
>> at java.io.BufferedInputStream.<init>
>> (BufferedInputStream.java:178)
>> at java.io.BufferedInputStream.<init>
>> (BufferedInputStream.java:158)
>> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0
>> (TCPTransport.java:679)
>> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run
>> (TCPTransport.java:649)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask
>> (ThreadPoolExecutor.java:886)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run
>> (ThreadPoolExecutor.java:908)
>> at java.lang.Thread.run(Thread.java:637)
>> ERROR - Fatal exception in thread Thread[main,5,main]
>> java.lang.OutOfMemoryError: Java heap space
>> ERROR - Fatal exception in thread Thread[RMI TCP Connection(idle),
>> 5,RMI Runtime]
>> java.lang.OutOfMemoryError: Java heap space
>> at java.lang.Class.getDeclaredMethods0(Native Method)
>> at java.lang.Class.privateGetDeclaredMethods(Class.java:2427)
>> at java.lang.Class.getDeclaredMethod(Class.java:1935)
>> at java.io.ObjectStreamClass.getPrivateMethod
>> (ObjectStreamClass.java:1382)
>> at java.io.ObjectStreamClass.access$1700
>> (ObjectStreamClass.java:52)
>> at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:438)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at java.io.ObjectStreamClass.<init>(ObjectStreamClass.java:
>> 413)
>> at java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:
>> 310)
>> at java.io.ObjectOutputStream.writeObject0
>> (ObjectOutputStream.java:1106)
>> at java.io.ObjectOutputStream.defaultWriteFields
>> (ObjectOutputStream.java:1509)
>> at java.io.ObjectOutputStream.writeSerialData
>> (ObjectOutputStream.java:1474)
>> at java.io.ObjectOutputStream.writeOrdinaryObject
>> (ObjectOutputStream.java:1392)
>> at java.io.ObjectOutputStream.writeObject0
>> (ObjectOutputStream.java:1150)
>> at java.io.ObjectOutputStream.writeObject
>> (ObjectOutputStream.java:326)
>> at java.util.ArrayList.writeObject(ArrayList.java:570)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at sun.reflect.NativeMethodAccessorImpl.invoke
>> (NativeMethodAccessorImpl.java:39)
>> at sun.reflect.DelegatingMethodAccessorImpl.invoke
>> (DelegatingMethodAccessorImpl.java:25)
>> at java.lang.reflect.Method.invoke(Method.java:597)
>> at java.io.ObjectStreamClass.invokeWriteObject
>> (ObjectStreamClass.java:945)
>> at java.io.ObjectOutputStream.writeSerialData
>> (ObjectOutputStream.java:1461)
>> at java.io.ObjectOutputStream.writeOrdinaryObject
>> (ObjectOutputStream.java:1392)
>> at java.io.ObjectOutputStream.writeObject0
>> (ObjectOutputStream.java:1150)
>> at java.io.ObjectOutputStream.writeObject
>> (ObjectOutputStream.java:326)
>> at sun.rmi.server.UnicastRef.marshalValue(UnicastRef.java:274)
>> at sun.rmi.server.UnicastServerRef.dispatch
>> (UnicastServerRef.java:315)
>> at sun.rmi.transport.Transport$1.run(Transport.java:159)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at sun.rmi.transport.Transport.serviceCall(Transport.java:155)
>> at sun.rmi.transport.tcp.TCPTransport.handleMessages
>> (TCPTransport.java:535)
>> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0
>> (TCPTransport.java:790)
>> ERROR - Fatal exception in thread Thread[RMI TCP Connection(idle),
>> 5,RMI Runtime]
>>
>> [ABOVE REPEATS A COUPLE OF TIMES]
>>
>> If I try to connect with nodeprobe, after it halts, I will just get
>> this:
>>
>> Error connecting to remote JMX agent!
>> java.io.IOException: Failed to retrieve RMIServer stub:
>> javax.naming.ServiceUnavailableException [Root exception is
>> java.rmi.ConnectException: Connection refused to host:
>> 192.168.100.203; nested exception is:
>> java.net.ConnectException: Connection refused]
>> at javax.management.remote.rmi.RMIConnector.connect
>> (RMIConnector.java:338)
>> at javax.management.remote.JMXConnectorFactory.connect
>> (JMXConnectorFactory.java:248)
>> at org.apache.cassandra.tools.NodeProbe.connect
>> (NodeProbe.java:152)
>> at org.apache.cassandra.tools.NodeProbe.<init>
>> (NodeProbe.java:114)
>> at org.apache.cassandra.tools.NodeProbe.main(NodeProbe.java:
>> 545)
>> Caused by: javax.naming.ServiceUnavailableException [Root exception
>> is java.rmi.ConnectException: Connection refused to host:
>> 192.168.100.203; nested exception is:
>> java.net.ConnectException: Connection refused]
>> at com.sun.jndi.rmi.registry.RegistryContext.lookup
>> (RegistryContext.java:101)
>> at com.sun.jndi.toolkit.url.GenericURLContext.lookup
>> (GenericURLContext.java:185)
>> at javax.naming.InitialContext.lookup(InitialContext.java:392)
>> at javax.management.remote.rmi.RMIConnector.findRMIServerJNDI
>> (RMIConnector.java:1886)
>> at javax.management.remote.rmi.RMIConnector.findRMIServer
>> (RMIConnector.java:1856)
>> at javax.management.remote.rmi.RMIConnector.connect
>> (RMIConnector.java:257)
>> ... 4 more
>> Caused by: java.rmi.ConnectException: Connection refused to host:
>> 192.168.100.203; nested exception is:
>> java.net.ConnectException: Connection refused
>> at sun.rmi.transport.tcp.TCPEndpoint.newSocket
>> (TCPEndpoint.java:601)
>> at sun.rmi.transport.tcp.TCPChannel.createConnection
>> (TCPChannel.java:198)
>> at sun.rmi.transport.tcp.TCPChannel.newConnection
>> (TCPChannel.java:184)
>> at sun.rmi.server.UnicastRef.newCall(UnicastRef.java:322)
>> at sun.rmi.registry.RegistryImpl_Stub.lookup(Unknown Source)
>> at com.sun.jndi.rmi.registry.RegistryContext.lookup
>> (RegistryContext.java:97)
>> ... 9 more
>> Caused by: java.net.ConnectException: Connection refused
>> at java.net.PlainSocketImpl.socketConnect(Native Method)
>> at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:
>> 333)
>> at java.net.PlainSocketImpl.connectToAddress
>> (PlainSocketImpl.java:195)
>> at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
>> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:432)
>> at java.net.Socket.connect(Socket.java:525)
>> at java.net.Socket.connect(Socket.java:475)
>> at java.net.Socket.<init>(Socket.java:372)
>> at java.net.Socket.<init>(Socket.java:186)
>> at
>> sun.rmi.transport.proxy.RMIDirectSocketFactory.createSocket
>> (RMIDirectSocketFactory.java:22)
>> at
>> sun.rmi.transport.proxy.RMIMasterSocketFactory.createSocket
>> (RMIMasterSocketFactory.java:128)
>> at sun.rmi.transport.tcp.TCPEndpoint.newSocket
>> (TCPEndpoint.java:595)
>> ... 14 more
>>
>>
>> I know i responds as supposed, before the crash.
>>
>> Hope somebody can help! :-)
>>
>> Best regards
>> Dan Larsen
>>
>
>
Re: Help! Serious crash!
Posted by Igor Katkov <ik...@gmail.com>.
I had similar OutOfMemory issues, see "commit logs are not deleted" thread,
but it was fixed/commited to trunk.
What I don't get is how your commit log segment grows beyond threshold.
>Is it really nescessary to have as much RAM as you have data?!?!?
No, I was able to insert 50Gb worth of data with 3Gb cap
On Wed, Oct 7, 2009 at 2:16 PM, Dan Larsen <da...@techba.se> wrote:
> Sorry about that... running 0.4 taken from trunk.I didn't
> change CommitLogRotationThresholdInMB - it's still 128
>
> Thanks!
>
> On 07/10/2009, at 20.12, Igor Katkov wrote:
>
> What version are you running? Take 0.4 from trunk.
> Default config has
> <CommitLogRotationThresholdInMB>128</CommitLogRotationThresholdInMB>
> Did you change it allow your commit log chunks be 1Gb?
>
> On Wed, Oct 7, 2009 at 1:38 PM, Dan Larsen <da...@techba.se> wrote:
>
>> Hi guys,
>>
>> I have a small setup with 3 macs running os x snow leopard and the native
>> java 1.6.0.something.something ;-)
>> I am doing a lot of inserts, without any problems and I can read the data
>> again.
>>
>> But....
>> At some point, around 1G Commitlog size, cassandra starts a rise in ram
>> usage, untill it just stops responding - if I restart the server, with or
>> without -b, it will just usae 100% cpu for 10-15 mins and then stop with
>> this message:
>> Another problem, is that it seems to kill other servers too, when I try to
>> restart :-(
>>
>> java.lang.OutOfMemoryError: Java heap space
>> Dumping heap to java_pid53353.hprof ...
>> Heap dump file created [1164687070 bytes in 89.912 secs]
>>
>> ERROR - Fatal exception in thread Thread[RMI TCP Connection(idle),5,RMI
>> Runtime]
>> java.lang.OutOfMemoryError: Java heap space
>> at java.io.BufferedInputStream.<init>(BufferedInputStream.java:178)
>> at java.io.BufferedInputStream.<init>(BufferedInputStream.java:158)
>> at
>> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:679)
>> at
>> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>> at java.lang.Thread.run(Thread.java:637)
>> ERROR - Fatal exception in thread Thread[main,5,main]
>> java.lang.OutOfMemoryError: Java heap space
>> ERROR - Fatal exception in thread Thread[RMI TCP Connection(idle),5,RMI
>> Runtime]
>> java.lang.OutOfMemoryError: Java heap space
>> at java.lang.Class.getDeclaredMethods0(Native Method)
>> at java.lang.Class.privateGetDeclaredMethods(Class.java:2427)
>> at java.lang.Class.getDeclaredMethod(Class.java:1935)
>> at
>> java.io.ObjectStreamClass.getPrivateMethod(ObjectStreamClass.java:1382)
>> at java.io.ObjectStreamClass.access$1700(ObjectStreamClass.java:52)
>> at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:438)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at java.io.ObjectStreamClass.<init>(ObjectStreamClass.java:413)
>> at java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:310)
>> at
>> java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1106)
>> at
>> java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1509)
>> at
>> java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1474)
>> at
>> java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1392)
>> at
>> java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1150)
>> at
>> java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:326)
>> at java.util.ArrayList.writeObject(ArrayList.java:570)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> at java.lang.reflect.Method.invoke(Method.java:597)
>> at
>> java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:945)
>> at
>> java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1461)
>> at
>> java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1392)
>> at
>> java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1150)
>> at
>> java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:326)
>> at sun.rmi.server.UnicastRef.marshalValue(UnicastRef.java:274)
>> at
>> sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:315)
>> at sun.rmi.transport.Transport$1.run(Transport.java:159)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at sun.rmi.transport.Transport.serviceCall(Transport.java:155)
>> at
>> sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:535)
>> at
>> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790)
>> ERROR - Fatal exception in thread Thread[RMI TCP Connection(idle),5,RMI
>> Runtime]
>>
>> [ABOVE REPEATS A COUPLE OF TIMES]
>>
>> If I try to connect with nodeprobe, after it halts, I will just get this:
>>
>> Error connecting to remote JMX agent!
>> java.io.IOException: Failed to retrieve RMIServer stub:
>> javax.naming.ServiceUnavailableException [Root exception is
>> java.rmi.ConnectException: Connection refused to host: 192.168.100.203;
>> nested exception is:
>> java.net.ConnectException: Connection refused]
>> at
>> javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:338)
>> at
>> javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:248)
>> at org.apache.cassandra.tools.NodeProbe.connect(NodeProbe.java:152)
>> at org.apache.cassandra.tools.NodeProbe.<init>(NodeProbe.java:114)
>> at org.apache.cassandra.tools.NodeProbe.main(NodeProbe.java:545)
>> Caused by: javax.naming.ServiceUnavailableException [Root exception is
>> java.rmi.ConnectException: Connection refused to host: 192.168.100.203;
>> nested exception is:
>> java.net.ConnectException: Connection refused]
>> at
>> com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:101)
>> at
>> com.sun.jndi.toolkit.url.GenericURLContext.lookup(GenericURLContext.java:185)
>> at javax.naming.InitialContext.lookup(InitialContext.java:392)
>> at
>> javax.management.remote.rmi.RMIConnector.findRMIServerJNDI(RMIConnector.java:1886)
>> at
>> javax.management.remote.rmi.RMIConnector.findRMIServer(RMIConnector.java:1856)
>> at
>> javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:257)
>> ... 4 more
>> Caused by: java.rmi.ConnectException: Connection refused to host:
>> 192.168.100.203; nested exception is:
>> java.net.ConnectException: Connection refused
>> at
>> sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:601)
>> at
>> sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:198)
>> at
>> sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:184)
>> at sun.rmi.server.UnicastRef.newCall(UnicastRef.java:322)
>> at sun.rmi.registry.RegistryImpl_Stub.lookup(Unknown Source)
>> at
>> com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:97)
>> ... 9 more
>> Caused by: java.net.ConnectException: Connection refused
>> at java.net.PlainSocketImpl.socketConnect(Native Method)
>> at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
>> at
>> java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
>> at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
>> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:432)
>> at java.net.Socket.connect(Socket.java:525)
>> at java.net.Socket.connect(Socket.java:475)
>> at java.net.Socket.<init>(Socket.java:372)
>> at java.net.Socket.<init>(Socket.java:186)
>> at
>> sun.rmi.transport.proxy.RMIDirectSocketFactory.createSocket(RMIDirectSocketFactory.java:22)
>> at
>> sun.rmi.transport.proxy.RMIMasterSocketFactory.createSocket(RMIMasterSocketFactory.java:128)
>> at
>> sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:595)
>> ... 14 more
>>
>>
>> I know i responds as supposed, before the crash.
>>
>> Hope somebody can help! :-)
>>
>> Best regards
>> Dan Larsen
>>
>
>
>
Re: Help! Serious crash!
Posted by Dan Larsen <da...@techba.se>.
Sorry about that... running 0.4 taken from trunk.
I didn't change CommitLogRotationThresholdInMB - it's still 128
Thanks!
On 07/10/2009, at 20.12, Igor Katkov wrote:
> What version are you running? Take 0.4 from trunk.
> Default config has
> <CommitLogRotationThresholdInMB>128</CommitLogRotationThresholdInMB>
> Did you change it allow your commit log chunks be 1Gb?
>
> On Wed, Oct 7, 2009 at 1:38 PM, Dan Larsen <da...@techba.se> wrote:
> Hi guys,
>
> I have a small setup with 3 macs running os x snow leopard and the
> native java 1.6.0.something.something ;-)
> I am doing a lot of inserts, without any problems and I can read the
> data again.
>
> But....
> At some point, around 1G Commitlog size, cassandra starts a rise in
> ram usage, untill it just stops responding - if I restart the
> server, with or without -b, it will just usae 100% cpu for 10-15
> mins and then stop with this message:
> Another problem, is that it seems to kill other servers too, when I
> try to restart :-(
>
> java.lang.OutOfMemoryError: Java heap space
> Dumping heap to java_pid53353.hprof ...
> Heap dump file created [1164687070 bytes in 89.912 secs]
>
> ERROR - Fatal exception in thread Thread[RMI TCP Connection(idle),
> 5,RMI Runtime]
> java.lang.OutOfMemoryError: Java heap space
> at java.io.BufferedInputStream.<init>
> (BufferedInputStream.java:178)
> at java.io.BufferedInputStream.<init>
> (BufferedInputStream.java:158)
> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0
> (TCPTransport.java:679)
> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run
> (TCPTransport.java:649)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask
> (ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run
> (ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:637)
> ERROR - Fatal exception in thread Thread[main,5,main]
> java.lang.OutOfMemoryError: Java heap space
> ERROR - Fatal exception in thread Thread[RMI TCP Connection(idle),
> 5,RMI Runtime]
> java.lang.OutOfMemoryError: Java heap space
> at java.lang.Class.getDeclaredMethods0(Native Method)
> at java.lang.Class.privateGetDeclaredMethods(Class.java:2427)
> at java.lang.Class.getDeclaredMethod(Class.java:1935)
> at java.io.ObjectStreamClass.getPrivateMethod
> (ObjectStreamClass.java:1382)
> at java.io.ObjectStreamClass.access$1700
> (ObjectStreamClass.java:52)
> at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:438)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.io.ObjectStreamClass.<init>(ObjectStreamClass.java:413)
> at java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:310)
> at java.io.ObjectOutputStream.writeObject0
> (ObjectOutputStream.java:1106)
> at java.io.ObjectOutputStream.defaultWriteFields
> (ObjectOutputStream.java:1509)
> at java.io.ObjectOutputStream.writeSerialData
> (ObjectOutputStream.java:1474)
> at java.io.ObjectOutputStream.writeOrdinaryObject
> (ObjectOutputStream.java:1392)
> at java.io.ObjectOutputStream.writeObject0
> (ObjectOutputStream.java:1150)
> at java.io.ObjectOutputStream.writeObject
> (ObjectOutputStream.java:326)
> at java.util.ArrayList.writeObject(ArrayList.java:570)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke
> (NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke
> (DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at java.io.ObjectStreamClass.invokeWriteObject
> (ObjectStreamClass.java:945)
> at java.io.ObjectOutputStream.writeSerialData
> (ObjectOutputStream.java:1461)
> at java.io.ObjectOutputStream.writeOrdinaryObject
> (ObjectOutputStream.java:1392)
> at java.io.ObjectOutputStream.writeObject0
> (ObjectOutputStream.java:1150)
> at java.io.ObjectOutputStream.writeObject
> (ObjectOutputStream.java:326)
> at sun.rmi.server.UnicastRef.marshalValue(UnicastRef.java:274)
> at sun.rmi.server.UnicastServerRef.dispatch
> (UnicastServerRef.java:315)
> at sun.rmi.transport.Transport$1.run(Transport.java:159)
> at java.security.AccessController.doPrivileged(Native Method)
> at sun.rmi.transport.Transport.serviceCall(Transport.java:155)
> at sun.rmi.transport.tcp.TCPTransport.handleMessages
> (TCPTransport.java:535)
> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0
> (TCPTransport.java:790)
> ERROR - Fatal exception in thread Thread[RMI TCP Connection(idle),
> 5,RMI Runtime]
>
> [ABOVE REPEATS A COUPLE OF TIMES]
>
> If I try to connect with nodeprobe, after it halts, I will just get
> this:
>
> Error connecting to remote JMX agent!
> java.io.IOException: Failed to retrieve RMIServer stub:
> javax.naming.ServiceUnavailableException [Root exception is
> java.rmi.ConnectException: Connection refused to host:
> 192.168.100.203; nested exception is:
> java.net.ConnectException: Connection refused]
> at javax.management.remote.rmi.RMIConnector.connect
> (RMIConnector.java:338)
> at javax.management.remote.JMXConnectorFactory.connect
> (JMXConnectorFactory.java:248)
> at org.apache.cassandra.tools.NodeProbe.connect
> (NodeProbe.java:152)
> at org.apache.cassandra.tools.NodeProbe.<init>(NodeProbe.java:
> 114)
> at org.apache.cassandra.tools.NodeProbe.main(NodeProbe.java:
> 545)
> Caused by: javax.naming.ServiceUnavailableException [Root exception
> is java.rmi.ConnectException: Connection refused to host:
> 192.168.100.203; nested exception is:
> java.net.ConnectException: Connection refused]
> at com.sun.jndi.rmi.registry.RegistryContext.lookup
> (RegistryContext.java:101)
> at com.sun.jndi.toolkit.url.GenericURLContext.lookup
> (GenericURLContext.java:185)
> at javax.naming.InitialContext.lookup(InitialContext.java:392)
> at javax.management.remote.rmi.RMIConnector.findRMIServerJNDI
> (RMIConnector.java:1886)
> at javax.management.remote.rmi.RMIConnector.findRMIServer
> (RMIConnector.java:1856)
> at javax.management.remote.rmi.RMIConnector.connect
> (RMIConnector.java:257)
> ... 4 more
> Caused by: java.rmi.ConnectException: Connection refused to host:
> 192.168.100.203; nested exception is:
> java.net.ConnectException: Connection refused
> at sun.rmi.transport.tcp.TCPEndpoint.newSocket
> (TCPEndpoint.java:601)
> at sun.rmi.transport.tcp.TCPChannel.createConnection
> (TCPChannel.java:198)
> at sun.rmi.transport.tcp.TCPChannel.newConnection
> (TCPChannel.java:184)
> at sun.rmi.server.UnicastRef.newCall(UnicastRef.java:322)
> at sun.rmi.registry.RegistryImpl_Stub.lookup(Unknown Source)
> at com.sun.jndi.rmi.registry.RegistryContext.lookup
> (RegistryContext.java:97)
> ... 9 more
> Caused by: java.net.ConnectException: Connection refused
> at java.net.PlainSocketImpl.socketConnect(Native Method)
> at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
> at java.net.PlainSocketImpl.connectToAddress
> (PlainSocketImpl.java:195)
> at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:432)
> at java.net.Socket.connect(Socket.java:525)
> at java.net.Socket.connect(Socket.java:475)
> at java.net.Socket.<init>(Socket.java:372)
> at java.net.Socket.<init>(Socket.java:186)
> at sun.rmi.transport.proxy.RMIDirectSocketFactory.createSocket
> (RMIDirectSocketFactory.java:22)
> at sun.rmi.transport.proxy.RMIMasterSocketFactory.createSocket
> (RMIMasterSocketFactory.java:128)
> at sun.rmi.transport.tcp.TCPEndpoint.newSocket
> (TCPEndpoint.java:595)
> ... 14 more
>
>
> I know i responds as supposed, before the crash.
>
> Hope somebody can help! :-)
>
> Best regards
> Dan Larsen
>
Re: Help! Serious crash!
Posted by Igor Katkov <ik...@gmail.com>.
What version are you running? Take 0.4 from trunk.
Default config has
<CommitLogRotationThresholdInMB>128</CommitLogRotationThresholdInMB>
Did you change it allow your commit log chunks be 1Gb?
On Wed, Oct 7, 2009 at 1:38 PM, Dan Larsen <da...@techba.se> wrote:
> Hi guys,
>
> I have a small setup with 3 macs running os x snow leopard and the native
> java 1.6.0.something.something ;-)
> I am doing a lot of inserts, without any problems and I can read the data
> again.
>
> But....
> At some point, around 1G Commitlog size, cassandra starts a rise in ram
> usage, untill it just stops responding - if I restart the server, with or
> without -b, it will just usae 100% cpu for 10-15 mins and then stop with
> this message:
> Another problem, is that it seems to kill other servers too, when I try to
> restart :-(
>
> java.lang.OutOfMemoryError: Java heap space
> Dumping heap to java_pid53353.hprof ...
> Heap dump file created [1164687070 bytes in 89.912 secs]
>
> ERROR - Fatal exception in thread Thread[RMI TCP Connection(idle),5,RMI
> Runtime]
> java.lang.OutOfMemoryError: Java heap space
> at java.io.BufferedInputStream.<init>(BufferedInputStream.java:178)
> at java.io.BufferedInputStream.<init>(BufferedInputStream.java:158)
> at
> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:679)
> at
> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:637)
> ERROR - Fatal exception in thread Thread[main,5,main]
> java.lang.OutOfMemoryError: Java heap space
> ERROR - Fatal exception in thread Thread[RMI TCP Connection(idle),5,RMI
> Runtime]
> java.lang.OutOfMemoryError: Java heap space
> at java.lang.Class.getDeclaredMethods0(Native Method)
> at java.lang.Class.privateGetDeclaredMethods(Class.java:2427)
> at java.lang.Class.getDeclaredMethod(Class.java:1935)
> at
> java.io.ObjectStreamClass.getPrivateMethod(ObjectStreamClass.java:1382)
> at java.io.ObjectStreamClass.access$1700(ObjectStreamClass.java:52)
> at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:438)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.io.ObjectStreamClass.<init>(ObjectStreamClass.java:413)
> at java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:310)
> at
> java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1106)
> at
> java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1509)
> at
> java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1474)
> at
> java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1392)
> at
> java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1150)
> at
> java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:326)
> at java.util.ArrayList.writeObject(ArrayList.java:570)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:945)
> at
> java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1461)
> at
> java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1392)
> at
> java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1150)
> at
> java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:326)
> at sun.rmi.server.UnicastRef.marshalValue(UnicastRef.java:274)
> at
> sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:315)
> at sun.rmi.transport.Transport$1.run(Transport.java:159)
> at java.security.AccessController.doPrivileged(Native Method)
> at sun.rmi.transport.Transport.serviceCall(Transport.java:155)
> at
> sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:535)
> at
> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790)
> ERROR - Fatal exception in thread Thread[RMI TCP Connection(idle),5,RMI
> Runtime]
>
> [ABOVE REPEATS A COUPLE OF TIMES]
>
> If I try to connect with nodeprobe, after it halts, I will just get this:
>
> Error connecting to remote JMX agent!
> java.io.IOException: Failed to retrieve RMIServer stub:
> javax.naming.ServiceUnavailableException [Root exception is
> java.rmi.ConnectException: Connection refused to host: 192.168.100.203;
> nested exception is:
> java.net.ConnectException: Connection refused]
> at
> javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:338)
> at
> javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:248)
> at org.apache.cassandra.tools.NodeProbe.connect(NodeProbe.java:152)
> at org.apache.cassandra.tools.NodeProbe.<init>(NodeProbe.java:114)
> at org.apache.cassandra.tools.NodeProbe.main(NodeProbe.java:545)
> Caused by: javax.naming.ServiceUnavailableException [Root exception is
> java.rmi.ConnectException: Connection refused to host: 192.168.100.203;
> nested exception is:
> java.net.ConnectException: Connection refused]
> at
> com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:101)
> at
> com.sun.jndi.toolkit.url.GenericURLContext.lookup(GenericURLContext.java:185)
> at javax.naming.InitialContext.lookup(InitialContext.java:392)
> at
> javax.management.remote.rmi.RMIConnector.findRMIServerJNDI(RMIConnector.java:1886)
> at
> javax.management.remote.rmi.RMIConnector.findRMIServer(RMIConnector.java:1856)
> at
> javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:257)
> ... 4 more
> Caused by: java.rmi.ConnectException: Connection refused to host:
> 192.168.100.203; nested exception is:
> java.net.ConnectException: Connection refused
> at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:601)
> at
> sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:198)
> at
> sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:184)
> at sun.rmi.server.UnicastRef.newCall(UnicastRef.java:322)
> at sun.rmi.registry.RegistryImpl_Stub.lookup(Unknown Source)
> at
> com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:97)
> ... 9 more
> Caused by: java.net.ConnectException: Connection refused
> at java.net.PlainSocketImpl.socketConnect(Native Method)
> at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
> at
> java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
> at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:432)
> at java.net.Socket.connect(Socket.java:525)
> at java.net.Socket.connect(Socket.java:475)
> at java.net.Socket.<init>(Socket.java:372)
> at java.net.Socket.<init>(Socket.java:186)
> at
> sun.rmi.transport.proxy.RMIDirectSocketFactory.createSocket(RMIDirectSocketFactory.java:22)
> at
> sun.rmi.transport.proxy.RMIMasterSocketFactory.createSocket(RMIMasterSocketFactory.java:128)
> at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:595)
> ... 14 more
>
>
> I know i responds as supposed, before the crash.
>
> Hope somebody can help! :-)
>
> Best regards
> Dan Larsen
>
Re: Help! Serious crash!
Posted by Dan Larsen <da...@techba.se>.
O.k... I experimented a bit: changed "-Xmx1G \" to "-Xmx2G \" in bin/
cassandra.in.sh ...
Is it really nescessary to have as much RAM as you have data?!?!?
Best regards
Dan
On 07/10/2009, at 19.50, Dan Larsen wrote:
> Forgot to say - It will eat up 1.13GB of RAM and 1.32GB VRAM, and
> still throw and OutOfMemory, though 700MB left...
>
> On 07/10/2009, at 19.38, Dan Larsen wrote:
>
>> Hi guys,
>>
>> I have a small setup with 3 macs running os x snow leopard and the
>> native java 1.6.0.something.something ;-)
>> I am doing a lot of inserts, without any problems and I can read
>> the data again.
>>
>> But....
>> At some point, around 1G Commitlog size, cassandra starts a rise in
>> ram usage, untill it just stops responding - if I restart the
>> server, with or without -b, it will just usae 100% cpu for 10-15
>> mins and then stop with this message:
>> Another problem, is that it seems to kill other servers too, when I
>> try to restart :-(
>>
>> java.lang.OutOfMemoryError: Java heap space
>> Dumping heap to java_pid53353.hprof ...
>> Heap dump file created [1164687070 bytes in 89.912 secs]
>>
>> ERROR - Fatal exception in thread Thread[RMI TCP Connection(idle),
>> 5,RMI Runtime]
>> java.lang.OutOfMemoryError: Java heap space
>> at java.io.BufferedInputStream.<init>(BufferedInputStream.java:178)
>> at java.io.BufferedInputStream.<init>(BufferedInputStream.java:158)
>> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0
>> (TCPTransport.java:679)
>> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run
>> (TCPTransport.java:649)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask
>> (ThreadPoolExecutor.java:886)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run
>> (ThreadPoolExecutor.java:908)
>> at java.lang.Thread.run(Thread.java:637)
>> ERROR - Fatal exception in thread Thread[main,5,main]
>> java.lang.OutOfMemoryError: Java heap space
>> ERROR - Fatal exception in thread Thread[RMI TCP Connection(idle),
>> 5,RMI Runtime]
>> java.lang.OutOfMemoryError: Java heap space
>> at java.lang.Class.getDeclaredMethods0(Native Method)
>> at java.lang.Class.privateGetDeclaredMethods(Class.java:2427)
>> at java.lang.Class.getDeclaredMethod(Class.java:1935)
>> at java.io.ObjectStreamClass.getPrivateMethod
>> (ObjectStreamClass.java:1382)
>> at java.io.ObjectStreamClass.access$1700(ObjectStreamClass.java:52)
>> at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:438)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at java.io.ObjectStreamClass.<init>(ObjectStreamClass.java:413)
>> at java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:310)
>> at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:
>> 1106)
>> at java.io.ObjectOutputStream.defaultWriteFields
>> (ObjectOutputStream.java:1509)
>> at java.io.ObjectOutputStream.writeSerialData
>> (ObjectOutputStream.java:1474)
>> at java.io.ObjectOutputStream.writeOrdinaryObject
>> (ObjectOutputStream.java:1392)
>> at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:
>> 1150)
>> at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:
>> 326)
>> at java.util.ArrayList.writeObject(ArrayList.java:570)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at sun.reflect.NativeMethodAccessorImpl.invoke
>> (NativeMethodAccessorImpl.java:39)
>> at sun.reflect.DelegatingMethodAccessorImpl.invoke
>> (DelegatingMethodAccessorImpl.java:25)
>> at java.lang.reflect.Method.invoke(Method.java:597)
>> at java.io.ObjectStreamClass.invokeWriteObject
>> (ObjectStreamClass.java:945)
>> at java.io.ObjectOutputStream.writeSerialData
>> (ObjectOutputStream.java:1461)
>> at java.io.ObjectOutputStream.writeOrdinaryObject
>> (ObjectOutputStream.java:1392)
>> at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:
>> 1150)
>> at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:
>> 326)
>> at sun.rmi.server.UnicastRef.marshalValue(UnicastRef.java:274)
>> at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:
>> 315)
>> at sun.rmi.transport.Transport$1.run(Transport.java:159)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at sun.rmi.transport.Transport.serviceCall(Transport.java:155)
>> at sun.rmi.transport.tcp.TCPTransport.handleMessages
>> (TCPTransport.java:535)
>> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0
>> (TCPTransport.java:790)
>> ERROR - Fatal exception in thread Thread[RMI TCP Connection(idle),
>> 5,RMI Runtime]
>>
>> [ABOVE REPEATS A COUPLE OF TIMES]
>>
>> If I try to connect with nodeprobe, after it halts, I will just get
>> this:
>>
>> Error connecting to remote JMX agent!
>> java.io.IOException: Failed to retrieve RMIServer stub:
>> javax.naming.ServiceUnavailableException [Root exception is
>> java.rmi.ConnectException: Connection refused to host:
>> 192.168.100.203; nested exception is:
>> java.net.ConnectException: Connection refused]
>> at javax.management.remote.rmi.RMIConnector.connect
>> (RMIConnector.java:338)
>> at javax.management.remote.JMXConnectorFactory.connect
>> (JMXConnectorFactory.java:248)
>> at org.apache.cassandra.tools.NodeProbe.connect(NodeProbe.java:152)
>> at org.apache.cassandra.tools.NodeProbe.<init>(NodeProbe.java:114)
>> at org.apache.cassandra.tools.NodeProbe.main(NodeProbe.java:545)
>> Caused by: javax.naming.ServiceUnavailableException [Root exception
>> is java.rmi.ConnectException: Connection refused to host:
>> 192.168.100.203; nested exception is:
>> java.net.ConnectException: Connection refused]
>> at com.sun.jndi.rmi.registry.RegistryContext.lookup
>> (RegistryContext.java:101)
>> at com.sun.jndi.toolkit.url.GenericURLContext.lookup
>> (GenericURLContext.java:185)
>> at javax.naming.InitialContext.lookup(InitialContext.java:392)
>> at javax.management.remote.rmi.RMIConnector.findRMIServerJNDI
>> (RMIConnector.java:1886)
>> at javax.management.remote.rmi.RMIConnector.findRMIServer
>> (RMIConnector.java:1856)
>> at javax.management.remote.rmi.RMIConnector.connect
>> (RMIConnector.java:257)
>> ... 4 more
>> Caused by: java.rmi.ConnectException: Connection refused to host:
>> 192.168.100.203; nested exception is:
>> java.net.ConnectException: Connection refused
>> at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:601)
>> at sun.rmi.transport.tcp.TCPChannel.createConnection
>> (TCPChannel.java:198)
>> at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:
>> 184)
>> at sun.rmi.server.UnicastRef.newCall(UnicastRef.java:322)
>> at sun.rmi.registry.RegistryImpl_Stub.lookup(Unknown Source)
>> at com.sun.jndi.rmi.registry.RegistryContext.lookup
>> (RegistryContext.java:97)
>> ... 9 more
>> Caused by: java.net.ConnectException: Connection refused
>> at java.net.PlainSocketImpl.socketConnect(Native Method)
>> at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
>> at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:
>> 195)
>> at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
>> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:432)
>> at java.net.Socket.connect(Socket.java:525)
>> at java.net.Socket.connect(Socket.java:475)
>> at java.net.Socket.<init>(Socket.java:372)
>> at java.net.Socket.<init>(Socket.java:186)
>> at sun.rmi.transport.proxy.RMIDirectSocketFactory.createSocket
>> (RMIDirectSocketFactory.java:22)
>> at sun.rmi.transport.proxy.RMIMasterSocketFactory.createSocket
>> (RMIMasterSocketFactory.java:128)
>> at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:595)
>> ... 14 more
>>
>>
>> I know i responds as supposed, before the crash.
>>
>> Hope somebody can help! :-)
>>
>> Best regards
>> Dan Larsen
>>
>
>
Re: Help! Serious crash!
Posted by Dan Larsen <da...@techba.se>.
Forgot to say - It will eat up 1.13GB of RAM and 1.32GB VRAM, and
still throw and OutOfMemory, though 700MB left...
On 07/10/2009, at 19.38, Dan Larsen wrote:
> Hi guys,
>
> I have a small setup with 3 macs running os x snow leopard and the
> native java 1.6.0.something.something ;-)
> I am doing a lot of inserts, without any problems and I can read the
> data again.
>
> But....
> At some point, around 1G Commitlog size, cassandra starts a rise in
> ram usage, untill it just stops responding - if I restart the
> server, with or without -b, it will just usae 100% cpu for 10-15
> mins and then stop with this message:
> Another problem, is that it seems to kill other servers too, when I
> try to restart :-(
>
> java.lang.OutOfMemoryError: Java heap space
> Dumping heap to java_pid53353.hprof ...
> Heap dump file created [1164687070 bytes in 89.912 secs]
>
> ERROR - Fatal exception in thread Thread[RMI TCP Connection(idle),
> 5,RMI Runtime]
> java.lang.OutOfMemoryError: Java heap space
> at java.io.BufferedInputStream.<init>(BufferedInputStream.java:178)
> at java.io.BufferedInputStream.<init>(BufferedInputStream.java:158)
> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0
> (TCPTransport.java:679)
> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run
> (TCPTransport.java:649)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask
> (ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run
> (ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:637)
> ERROR - Fatal exception in thread Thread[main,5,main]
> java.lang.OutOfMemoryError: Java heap space
> ERROR - Fatal exception in thread Thread[RMI TCP Connection(idle),
> 5,RMI Runtime]
> java.lang.OutOfMemoryError: Java heap space
> at java.lang.Class.getDeclaredMethods0(Native Method)
> at java.lang.Class.privateGetDeclaredMethods(Class.java:2427)
> at java.lang.Class.getDeclaredMethod(Class.java:1935)
> at java.io.ObjectStreamClass.getPrivateMethod
> (ObjectStreamClass.java:1382)
> at java.io.ObjectStreamClass.access$1700(ObjectStreamClass.java:52)
> at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:438)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.io.ObjectStreamClass.<init>(ObjectStreamClass.java:413)
> at java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:310)
> at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:
> 1106)
> at java.io.ObjectOutputStream.defaultWriteFields
> (ObjectOutputStream.java:1509)
> at java.io.ObjectOutputStream.writeSerialData
> (ObjectOutputStream.java:1474)
> at java.io.ObjectOutputStream.writeOrdinaryObject
> (ObjectOutputStream.java:1392)
> at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:
> 1150)
> at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:
> 326)
> at java.util.ArrayList.writeObject(ArrayList.java:570)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke
> (NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke
> (DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at java.io.ObjectStreamClass.invokeWriteObject
> (ObjectStreamClass.java:945)
> at java.io.ObjectOutputStream.writeSerialData
> (ObjectOutputStream.java:1461)
> at java.io.ObjectOutputStream.writeOrdinaryObject
> (ObjectOutputStream.java:1392)
> at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:
> 1150)
> at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:
> 326)
> at sun.rmi.server.UnicastRef.marshalValue(UnicastRef.java:274)
> at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:
> 315)
> at sun.rmi.transport.Transport$1.run(Transport.java:159)
> at java.security.AccessController.doPrivileged(Native Method)
> at sun.rmi.transport.Transport.serviceCall(Transport.java:155)
> at sun.rmi.transport.tcp.TCPTransport.handleMessages
> (TCPTransport.java:535)
> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0
> (TCPTransport.java:790)
> ERROR - Fatal exception in thread Thread[RMI TCP Connection(idle),
> 5,RMI Runtime]
>
> [ABOVE REPEATS A COUPLE OF TIMES]
>
> If I try to connect with nodeprobe, after it halts, I will just get
> this:
>
> Error connecting to remote JMX agent!
> java.io.IOException: Failed to retrieve RMIServer stub:
> javax.naming.ServiceUnavailableException [Root exception is
> java.rmi.ConnectException: Connection refused to host:
> 192.168.100.203; nested exception is:
> java.net.ConnectException: Connection refused]
> at javax.management.remote.rmi.RMIConnector.connect
> (RMIConnector.java:338)
> at javax.management.remote.JMXConnectorFactory.connect
> (JMXConnectorFactory.java:248)
> at org.apache.cassandra.tools.NodeProbe.connect(NodeProbe.java:152)
> at org.apache.cassandra.tools.NodeProbe.<init>(NodeProbe.java:114)
> at org.apache.cassandra.tools.NodeProbe.main(NodeProbe.java:545)
> Caused by: javax.naming.ServiceUnavailableException [Root exception
> is java.rmi.ConnectException: Connection refused to host:
> 192.168.100.203; nested exception is:
> java.net.ConnectException: Connection refused]
> at com.sun.jndi.rmi.registry.RegistryContext.lookup
> (RegistryContext.java:101)
> at com.sun.jndi.toolkit.url.GenericURLContext.lookup
> (GenericURLContext.java:185)
> at javax.naming.InitialContext.lookup(InitialContext.java:392)
> at javax.management.remote.rmi.RMIConnector.findRMIServerJNDI
> (RMIConnector.java:1886)
> at javax.management.remote.rmi.RMIConnector.findRMIServer
> (RMIConnector.java:1856)
> at javax.management.remote.rmi.RMIConnector.connect
> (RMIConnector.java:257)
> ... 4 more
> Caused by: java.rmi.ConnectException: Connection refused to host:
> 192.168.100.203; nested exception is:
> java.net.ConnectException: Connection refused
> at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:601)
> at sun.rmi.transport.tcp.TCPChannel.createConnection
> (TCPChannel.java:198)
> at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:
> 184)
> at sun.rmi.server.UnicastRef.newCall(UnicastRef.java:322)
> at sun.rmi.registry.RegistryImpl_Stub.lookup(Unknown Source)
> at com.sun.jndi.rmi.registry.RegistryContext.lookup
> (RegistryContext.java:97)
> ... 9 more
> Caused by: java.net.ConnectException: Connection refused
> at java.net.PlainSocketImpl.socketConnect(Native Method)
> at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
> at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:
> 195)
> at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:432)
> at java.net.Socket.connect(Socket.java:525)
> at java.net.Socket.connect(Socket.java:475)
> at java.net.Socket.<init>(Socket.java:372)
> at java.net.Socket.<init>(Socket.java:186)
> at sun.rmi.transport.proxy.RMIDirectSocketFactory.createSocket
> (RMIDirectSocketFactory.java:22)
> at sun.rmi.transport.proxy.RMIMasterSocketFactory.createSocket
> (RMIMasterSocketFactory.java:128)
> at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:595)
> ... 14 more
>
>
> I know i responds as supposed, before the crash.
>
> Hope somebody can help! :-)
>
> Best regards
> Dan Larsen
>
Help! Serious crash!
Posted by Dan Larsen <da...@techba.se>.
Hi guys,
I have a small setup with 3 macs running os x snow leopard and the
native java 1.6.0.something.something ;-)
I am doing a lot of inserts, without any problems and I can read the
data again.
But....
At some point, around 1G Commitlog size, cassandra starts a rise in
ram usage, untill it just stops responding - if I restart the server,
with or without -b, it will just usae 100% cpu for 10-15 mins and then
stop with this message:
Another problem, is that it seems to kill other servers too, when I
try to restart :-(
java.lang.OutOfMemoryError: Java heap space
Dumping heap to java_pid53353.hprof ...
Heap dump file created [1164687070 bytes in 89.912 secs]
ERROR - Fatal exception in thread Thread[RMI TCP Connection(idle),
5,RMI Runtime]
java.lang.OutOfMemoryError: Java heap space
at java.io.BufferedInputStream.<init>(BufferedInputStream.java:178)
at java.io.BufferedInputStream.<init>(BufferedInputStream.java:158)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0
(TCPTransport.java:679)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run
(TCPTransport.java:649)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask
(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run
(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:637)
ERROR - Fatal exception in thread Thread[main,5,main]
java.lang.OutOfMemoryError: Java heap space
ERROR - Fatal exception in thread Thread[RMI TCP Connection(idle),
5,RMI Runtime]
java.lang.OutOfMemoryError: Java heap space
at java.lang.Class.getDeclaredMethods0(Native Method)
at java.lang.Class.privateGetDeclaredMethods(Class.java:2427)
at java.lang.Class.getDeclaredMethod(Class.java:1935)
at java.io.ObjectStreamClass.getPrivateMethod(ObjectStreamClass.java:
1382)
at java.io.ObjectStreamClass.access$1700(ObjectStreamClass.java:52)
at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:438)
at java.security.AccessController.doPrivileged(Native Method)
at java.io.ObjectStreamClass.<init>(ObjectStreamClass.java:413)
at java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:310)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:
1106)
at java.io.ObjectOutputStream.defaultWriteFields
(ObjectOutputStream.java:1509)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:
1474)
at java.io.ObjectOutputStream.writeOrdinaryObject
(ObjectOutputStream.java:1392)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:
1150)
at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:326)
at java.util.ArrayList.writeObject(ArrayList.java:570)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke
(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke
(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:
945)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:
1461)
at java.io.ObjectOutputStream.writeOrdinaryObject
(ObjectOutputStream.java:1392)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:
1150)
at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:326)
at sun.rmi.server.UnicastRef.marshalValue(UnicastRef.java:274)
at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:315)
at sun.rmi.transport.Transport$1.run(Transport.java:159)
at java.security.AccessController.doPrivileged(Native Method)
at sun.rmi.transport.Transport.serviceCall(Transport.java:155)
at sun.rmi.transport.tcp.TCPTransport.handleMessages
(TCPTransport.java:535)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0
(TCPTransport.java:790)
ERROR - Fatal exception in thread Thread[RMI TCP Connection(idle),
5,RMI Runtime]
[ABOVE REPEATS A COUPLE OF TIMES]
If I try to connect with nodeprobe, after it halts, I will just get
this:
Error connecting to remote JMX agent!
java.io.IOException: Failed to retrieve RMIServer stub:
javax.naming.ServiceUnavailableException [Root exception is
java.rmi.ConnectException: Connection refused to host:
192.168.100.203; nested exception is:
java.net.ConnectException: Connection refused]
at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:
338)
at javax.management.remote.JMXConnectorFactory.connect
(JMXConnectorFactory.java:248)
at org.apache.cassandra.tools.NodeProbe.connect(NodeProbe.java:152)
at org.apache.cassandra.tools.NodeProbe.<init>(NodeProbe.java:114)
at org.apache.cassandra.tools.NodeProbe.main(NodeProbe.java:545)
Caused by: javax.naming.ServiceUnavailableException [Root exception is
java.rmi.ConnectException: Connection refused to host:
192.168.100.203; nested exception is:
java.net.ConnectException: Connection refused]
at com.sun.jndi.rmi.registry.RegistryContext.lookup
(RegistryContext.java:101)
at com.sun.jndi.toolkit.url.GenericURLContext.lookup
(GenericURLContext.java:185)
at javax.naming.InitialContext.lookup(InitialContext.java:392)
at javax.management.remote.rmi.RMIConnector.findRMIServerJNDI
(RMIConnector.java:1886)
at javax.management.remote.rmi.RMIConnector.findRMIServer
(RMIConnector.java:1856)
at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:
257)
... 4 more
Caused by: java.rmi.ConnectException: Connection refused to host:
192.168.100.203; nested exception is:
java.net.ConnectException: Connection refused
at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:601)
at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:
198)
at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:184)
at sun.rmi.server.UnicastRef.newCall(UnicastRef.java:322)
at sun.rmi.registry.RegistryImpl_Stub.lookup(Unknown Source)
at com.sun.jndi.rmi.registry.RegistryContext.lookup
(RegistryContext.java:97)
... 9 more
Caused by: java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:432)
at java.net.Socket.connect(Socket.java:525)
at java.net.Socket.connect(Socket.java:475)
at java.net.Socket.<init>(Socket.java:372)
at java.net.Socket.<init>(Socket.java:186)
at sun.rmi.transport.proxy.RMIDirectSocketFactory.createSocket
(RMIDirectSocketFactory.java:22)
at sun.rmi.transport.proxy.RMIMasterSocketFactory.createSocket
(RMIMasterSocketFactory.java:128)
at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:595)
... 14 more
I know i responds as supposed, before the crash.
Hope somebody can help! :-)
Best regards
Dan Larsen
Re: Storage proxy write latency is too high
Posted by Jonathan Ellis <jb...@gmail.com>.
It's worth mentioning that I would be in favor of rewriting
MessagingService to use blocking sockets. It would be simpler and
more performant and free of the weird-ass JDK nio bugs that have
bitten us in the past.
On Tue, Oct 6, 2009 at 11:14 AM, Igor Katkov <ik...@gmail.com> wrote:
> I think I finally found what. It's implementation of Java NIOon Windows (JVM
> 1.6.0.16, 64b on Windows 2003)
> The very same code, same network but CentOS linux gives almost 4x
> performance. (in Cassandra@linux -> Cassandra@Windows setup)
> I don't have another linux box to test (Cassandra@linux -> Cassandra@linux)
> performance, but expect it to be even better.
>
> A lesson learnt: don't use windows.
>
> P.S.
> Here at Viigo we also learnt the hard way that async IO is also broken in
> .NET (C#). Now I start to wonder if there is some fundamental flaw in async
> IO on windows...
>
> On Mon, Oct 5, 2009 at 3:23 PM, Jonathan Ellis <jb...@gmail.com> wrote:
>>
>> On Mon, Oct 5, 2009 at 2:17 PM, Igor Katkov <ik...@gmail.com> wrote:
>> > measured via JMX console i.e. does not include client-cassandra-client
>> > latency
>> >
>> > 20 client threads 176975b value StorageProxy.WriteLatency ~660ms
>> > 10 client threads 176975b value StorageProxy.WriteLatency ~350ms
>> > 05 client threads 176975b value StorageProxy.WriteLatency ~156ms
>>
>> this is going up basically linearly with amount of (data x clients),
>> so clearly something is getting saturated.
>
>
Re: Storage proxy write latency is too high
Posted by Igor Katkov <ik...@gmail.com>.
Exact explanation to why Java NIO behave that way on Windows 2003 in Sun VM
1.6.0.16 represent only academical interest.
It would be nice though, if someone could repeat my tests and would confirm
or deny the result.
On Wed, Oct 7, 2009 at 5:52 PM, Jonathan Mischo <jm...@quagility.com>wrote:
> My bad...I guess that makes it even more important to find out if this is
> the problem on Windows? :)
>
> -Jon
>
>
> On Oct 6, 2009, at 6:26 PM, Jonathan Ellis wrote:
>
> Just to clarify, Cassandra only uses UDP for gossip -- passing data
>> between nodes is done over TCP.
>>
>> -Jonathan
>>
>> On Tue, Oct 6, 2009 at 5:31 PM, Jonathan Mischo <jm...@quagility.com>
>> wrote:
>>
>>> Back in the day, I was involved in performance testing JVMs, Solaris on
>>> Intel, and SWS at SunSoft, and our lab actually ran not just our own
>>> numbers, but competitors' numbers as well, for comparison. One thing we
>>> discovered, when analyzing some performance issues we'd seen with Windows
>>> clients, was that Windows' network stacks don't behave as you'd expect.
>>> They very regularly transmitted packets out of sequence.
>>> I'm not sure if this is still the case, as this was the late 90's, but we
>>> discovered this when we were working on SPECweb numbers (we were a SPEC
>>> lab)
>>> for SWS and, in looking at the JVM and other system settings to
>>> understand
>>> why we were seeing unexpectedly large TCP buffers. When we started
>>> sniffing
>>> packets directly from the Windows clients, we discovered the packets were
>>> being emitted out of sequence, which was causing the server to require
>>> larger per-connection buffers and was pushing TCP window size boundaries.
>>> Now, Cassandra uses UDP primarily, but we never tested to find out
>>> whether
>>> this was a TCP stack issue, IP stack issue, or Ethernet stack issue, so
>>> it
>>> may be a similar case. Since reassembly still has to happen, if the
>>> packets
>>> are being transmitted out of order, it makes sense that your write
>>> latency
>>> would be significantly higher.
>>> I'd be interested to see the results if you dig deeper into this.
>>> -Jon
>>>
>>> On Oct 6, 2009, at 11:14 AM, Igor Katkov wrote:
>>>
>>> I think I finally found what. It's implementation of Java NIOon Windows
>>> (JVM
>>> 1.6.0.16, 64b on Windows 2003)
>>> The very same code, same network but CentOS linux gives almost 4x
>>> performance. (in Cassandra@linux -> Cassandra@Windows setup)
>>> I don't have another linux box to test (Cassandra@linux ->
>>> Cassandra@linux)
>>> performance, but expect it to be even better.
>>>
>>> A lesson learnt: don't use windows.
>>>
>>> P.S.
>>> Here at Viigo we also learnt the hard way that async IO is also broken
>>> in
>>> .NET (C#). Now I start to wonder if there is some fundamental flaw in
>>> async
>>> IO on windows...
>>>
>>> On Mon, Oct 5, 2009 at 3:23 PM, Jonathan Ellis <jb...@gmail.com>
>>> wrote:
>>>
>>>>
>>>> On Mon, Oct 5, 2009 at 2:17 PM, Igor Katkov <ik...@gmail.com> wrote:
>>>>
>>>>> measured via JMX console i.e. does not include client-cassandra-client
>>>>> latency
>>>>>
>>>>> 20 client threads 176975b value StorageProxy.WriteLatency ~660ms
>>>>> 10 client threads 176975b value StorageProxy.WriteLatency ~350ms
>>>>> 05 client threads 176975b value StorageProxy.WriteLatency ~156ms
>>>>>
>>>>
>>>> this is going up basically linearly with amount of (data x clients),
>>>> so clearly something is getting saturated.
>>>>
>>>
>>>
>>>
>>>
>>
>
Re: Storage proxy write latency is too high
Posted by Jonathan Mischo <jm...@quagility.com>.
My bad...I guess that makes it even more important to find out if this
is the problem on Windows? :)
-Jon
On Oct 6, 2009, at 6:26 PM, Jonathan Ellis wrote:
> Just to clarify, Cassandra only uses UDP for gossip -- passing data
> between nodes is done over TCP.
>
> -Jonathan
>
> On Tue, Oct 6, 2009 at 5:31 PM, Jonathan Mischo
> <jm...@quagility.com> wrote:
>> Back in the day, I was involved in performance testing JVMs,
>> Solaris on
>> Intel, and SWS at SunSoft, and our lab actually ran not just our own
>> numbers, but competitors' numbers as well, for comparison. One
>> thing we
>> discovered, when analyzing some performance issues we'd seen with
>> Windows
>> clients, was that Windows' network stacks don't behave as you'd
>> expect.
>> They very regularly transmitted packets out of sequence.
>> I'm not sure if this is still the case, as this was the late 90's,
>> but we
>> discovered this when we were working on SPECweb numbers (we were a
>> SPEC lab)
>> for SWS and, in looking at the JVM and other system settings to
>> understand
>> why we were seeing unexpectedly large TCP buffers. When we started
>> sniffing
>> packets directly from the Windows clients, we discovered the
>> packets were
>> being emitted out of sequence, which was causing the server to
>> require
>> larger per-connection buffers and was pushing TCP window size
>> boundaries.
>> Now, Cassandra uses UDP primarily, but we never tested to find out
>> whether
>> this was a TCP stack issue, IP stack issue, or Ethernet stack
>> issue, so it
>> may be a similar case. Since reassembly still has to happen, if
>> the packets
>> are being transmitted out of order, it makes sense that your write
>> latency
>> would be significantly higher.
>> I'd be interested to see the results if you dig deeper into this.
>> -Jon
>>
>> On Oct 6, 2009, at 11:14 AM, Igor Katkov wrote:
>>
>> I think I finally found what. It's implementation of Java NIOon
>> Windows (JVM
>> 1.6.0.16, 64b on Windows 2003)
>> The very same code, same network but CentOS linux gives almost 4x
>> performance. (in Cassandra@linux -> Cassandra@Windows setup)
>> I don't have another linux box to test (Cassandra@linux ->
>> Cassandra@linux)
>> performance, but expect it to be even better.
>>
>> A lesson learnt: don't use windows.
>>
>> P.S.
>> Here at Viigo we also learnt the hard way that async IO is also
>> broken in
>> .NET (C#). Now I start to wonder if there is some fundamental flaw
>> in async
>> IO on windows...
>>
>> On Mon, Oct 5, 2009 at 3:23 PM, Jonathan Ellis <jb...@gmail.com>
>> wrote:
>>>
>>> On Mon, Oct 5, 2009 at 2:17 PM, Igor Katkov <ik...@gmail.com>
>>> wrote:
>>>> measured via JMX console i.e. does not include client-cassandra-
>>>> client
>>>> latency
>>>>
>>>> 20 client threads 176975b value StorageProxy.WriteLatency ~660ms
>>>> 10 client threads 176975b value StorageProxy.WriteLatency ~350ms
>>>> 05 client threads 176975b value StorageProxy.WriteLatency ~156ms
>>>
>>> this is going up basically linearly with amount of (data x clients),
>>> so clearly something is getting saturated.
>>
>>
>>
>
Re: Storage proxy write latency is too high
Posted by Jonathan Ellis <jb...@gmail.com>.
Just to clarify, Cassandra only uses UDP for gossip -- passing data
between nodes is done over TCP.
-Jonathan
On Tue, Oct 6, 2009 at 5:31 PM, Jonathan Mischo <jm...@quagility.com> wrote:
> Back in the day, I was involved in performance testing JVMs, Solaris on
> Intel, and SWS at SunSoft, and our lab actually ran not just our own
> numbers, but competitors' numbers as well, for comparison. One thing we
> discovered, when analyzing some performance issues we'd seen with Windows
> clients, was that Windows' network stacks don't behave as you'd expect.
> They very regularly transmitted packets out of sequence.
> I'm not sure if this is still the case, as this was the late 90's, but we
> discovered this when we were working on SPECweb numbers (we were a SPEC lab)
> for SWS and, in looking at the JVM and other system settings to understand
> why we were seeing unexpectedly large TCP buffers. When we started sniffing
> packets directly from the Windows clients, we discovered the packets were
> being emitted out of sequence, which was causing the server to require
> larger per-connection buffers and was pushing TCP window size boundaries.
> Now, Cassandra uses UDP primarily, but we never tested to find out whether
> this was a TCP stack issue, IP stack issue, or Ethernet stack issue, so it
> may be a similar case. Since reassembly still has to happen, if the packets
> are being transmitted out of order, it makes sense that your write latency
> would be significantly higher.
> I'd be interested to see the results if you dig deeper into this.
> -Jon
>
> On Oct 6, 2009, at 11:14 AM, Igor Katkov wrote:
>
> I think I finally found what. It's implementation of Java NIOon Windows (JVM
> 1.6.0.16, 64b on Windows 2003)
> The very same code, same network but CentOS linux gives almost 4x
> performance. (in Cassandra@linux -> Cassandra@Windows setup)
> I don't have another linux box to test (Cassandra@linux -> Cassandra@linux)
> performance, but expect it to be even better.
>
> A lesson learnt: don't use windows.
>
> P.S.
> Here at Viigo we also learnt the hard way that async IO is also broken in
> .NET (C#). Now I start to wonder if there is some fundamental flaw in async
> IO on windows...
>
> On Mon, Oct 5, 2009 at 3:23 PM, Jonathan Ellis <jb...@gmail.com> wrote:
>>
>> On Mon, Oct 5, 2009 at 2:17 PM, Igor Katkov <ik...@gmail.com> wrote:
>> > measured via JMX console i.e. does not include client-cassandra-client
>> > latency
>> >
>> > 20 client threads 176975b value StorageProxy.WriteLatency ~660ms
>> > 10 client threads 176975b value StorageProxy.WriteLatency ~350ms
>> > 05 client threads 176975b value StorageProxy.WriteLatency ~156ms
>>
>> this is going up basically linearly with amount of (data x clients),
>> so clearly something is getting saturated.
>
>
>
Re: Storage proxy write latency is too high
Posted by Jonathan Mischo <jm...@quagility.com>.
Back in the day, I was involved in performance testing JVMs, Solaris
on Intel, and SWS at SunSoft, and our lab actually ran not just our
own numbers, but competitors' numbers as well, for comparison. One
thing we discovered, when analyzing some performance issues we'd seen
with Windows clients, was that Windows' network stacks don't behave as
you'd expect. They very regularly transmitted packets out of sequence.
I'm not sure if this is still the case, as this was the late 90's, but
we discovered this when we were working on SPECweb numbers (we were a
SPEC lab) for SWS and, in looking at the JVM and other system settings
to understand why we were seeing unexpectedly large TCP buffers. When
we started sniffing packets directly from the Windows clients, we
discovered the packets were being emitted out of sequence, which was
causing the server to require larger per-connection buffers and was
pushing TCP window size boundaries.
Now, Cassandra uses UDP primarily, but we never tested to find out
whether this was a TCP stack issue, IP stack issue, or Ethernet stack
issue, so it may be a similar case. Since reassembly still has to
happen, if the packets are being transmitted out of order, it makes
sense that your write latency would be significantly higher.
I'd be interested to see the results if you dig deeper into this.
-Jon
On Oct 6, 2009, at 11:14 AM, Igor Katkov wrote:
> I think I finally found what. It's implementation of Java NIOon
> Windows (JVM 1.6.0.16, 64b on Windows 2003)
> The very same code, same network but CentOS linux gives almost 4x
> performance. (in Cassandra@linux -> Cassandra@Windows setup)
> I don't have another linux box to test (Cassandra@linux ->
> Cassandra@linux) performance, but expect it to be even better.
>
> A lesson learnt: don't use windows.
>
> P.S.
> Here at Viigo we also learnt the hard way that async IO is also
> broken in .NET (C#). Now I start to wonder if there is some
> fundamental flaw in async IO on windows...
>
> On Mon, Oct 5, 2009 at 3:23 PM, Jonathan Ellis <jb...@gmail.com>
> wrote:
> On Mon, Oct 5, 2009 at 2:17 PM, Igor Katkov <ik...@gmail.com> wrote:
> > measured via JMX console i.e. does not include client-cassandra-
> client
> > latency
> >
> > 20 client threads 176975b value StorageProxy.WriteLatency ~660ms
> > 10 client threads 176975b value StorageProxy.WriteLatency ~350ms
> > 05 client threads 176975b value StorageProxy.WriteLatency ~156ms
>
> this is going up basically linearly with amount of (data x clients),
> so clearly something is getting saturated.
>
Re: Storage proxy write latency is too high
Posted by Igor Katkov <ik...@gmail.com>.
I think I finally found what. It's implementation of Java NIOon Windows (JVM
1.6.0.16, 64b on Windows 2003)
The very same code, same network but CentOS linux gives almost 4x
performance. (in Cassandra@linux -> Cassandra@Windows setup)
I don't have another linux box to test (Cassandra@linux -> Cassandra@linux)
performance, but expect it to be even better.
A lesson learnt: don't use windows.
P.S.
Here at Viigo we also learnt the hard way that async IO is also broken in
.NET (C#). Now I start to wonder if there is some fundamental flaw in async
IO on windows...
On Mon, Oct 5, 2009 at 3:23 PM, Jonathan Ellis <jb...@gmail.com> wrote:
> On Mon, Oct 5, 2009 at 2:17 PM, Igor Katkov <ik...@gmail.com> wrote:
> > measured via JMX console i.e. does not include client-cassandra-client
> > latency
> >
> > 20 client threads 176975b value StorageProxy.WriteLatency ~660ms
> > 10 client threads 176975b value StorageProxy.WriteLatency ~350ms
> > 05 client threads 176975b value StorageProxy.WriteLatency ~156ms
>
> this is going up basically linearly with amount of (data x clients),
> so clearly something is getting saturated.
>
Re: Storage proxy write latency is too high
Posted by Jonathan Ellis <jb...@gmail.com>.
On Mon, Oct 5, 2009 at 2:17 PM, Igor Katkov <ik...@gmail.com> wrote:
> measured via JMX console i.e. does not include client-cassandra-client
> latency
>
> 20 client threads 176975b value StorageProxy.WriteLatency ~660ms
> 10 client threads 176975b value StorageProxy.WriteLatency ~350ms
> 05 client threads 176975b value StorageProxy.WriteLatency ~156ms
this is going up basically linearly with amount of (data x clients),
so clearly something is getting saturated.
Re: Storage proxy write latency is too high
Posted by Igor Katkov <ik...@gmail.com>.
measured via JMX console i.e. does not include client-cassandra-client
latency
20 client threads 176975b value StorageProxy.WriteLatency ~660ms
10 client threads 176975b value StorageProxy.WriteLatency ~350ms
05 client threads 176975b value StorageProxy.WriteLatency ~156ms
20 client threads 4978b value StorageProxy.WriteLatency ~32ms
10 client threads 4978b value StorageProxy.WriteLatency ~32ms
05 client threads 4978b value StorageProxy.WriteLatency ~31ms
20 client threads 1000b value StorageProxy.WriteLatency ~30ms
10 client threads 1000b value StorageProxy.WriteLatency ~30ms
05 client threads 1000b value StorageProxy.WriteLatency ~29ms
On Mon, Oct 5, 2009 at 2:52 PM, Jonathan Ellis <jb...@gmail.com> wrote:
> so for a 200KB value you are seeing >200ms of latency in
> MessagingService? How much for a 1KB value?
>
> -Jonathan
>
> On Mon, Oct 5, 2009 at 1:46 PM, Igor Katkov <ik...@gmail.com> wrote:
> > In case anyone is following this, here is an update:
> >
> > I was able to narrow it down to Cassandra-Cassandra link. Storage proxy
> > latency depends on size of the key. The larger amount of data (per key)
> is
> > transfered the larger latency is. No surprise here.
> > Client connects to a demons "A" and sends key-value, "A" accept thrift
> > message, de-serialize it to an object, sees that key belongs to demons
> "B",
> > serialize it to bytes once again (internal format now) and invoke
> > MessagingService, which in turn writes to a socket. As soon as "B"
> delivers
> > write-acknowledgment over a different connection, the client call is let
> go.
> > Cassandra's MessagingService utilizes java nio to connect to other
> cassandra
> > daemons, all connections are uni-directionals. So in theory it should be
> > very fast. But it's not.
> >
> > What does look suspicious is certain network usage cap, only ~4% of the
> > 1Gbps link is used regardless of "value" size. With smaller value I get a
> > better throughput, with larger (200Kb) - worse.
> >
> > As a temp workaround I see that client might be held responsible to
> > identifying what cassandra instance it should send a key to. On 200kb
> value
> > it's ~10 times faster.
> >
> >
> > On Thu, Oct 1, 2009 at 6:51 PM, Igor Katkov <ik...@gmail.com> wrote:
> >>
> >> Hi,
> >>
> >> I have the following puzzle:
> >> Storage proxy write latency ~235ms
> >> CF write latency <1 ms
> >>
> >> I have 3 nodes in the cluster, Cassandra v.0.4. Tokens evenly
> distributed.
> >> The client connects to a node and inserts a key with
> ConsistencyLevel.ONE
> >> If it happen to be a local write operation is fast, same speed as in one
> >> node setup. JMX shows write latency <1 ms
> >> If it happens to be a remote insert StorageProxy sends it to a proper
> >> node. This operation is slow. JMX shows write latency ~ 235ms.
> >> In the same time, on remote node JMX shows same <1ms write latency. So
> >> it's not remote node being sluggish, it's something else.
> >> There are no pending tasks on remote node - JMX counters are always
> zero,
> >> network is 1Gb, idle. So I can't blame it.
> >>
> >>
> >> I profiled Cassandra server in JProfiler, could not find a thing. All
> this
> >> extra time is spent inside QuorumResponseHandler waiting for the
> condition
> >> to signal. Which should happen as soon as response is received.
> >>
> >> There is one pooled TCP connection open to remote host. Hardly a
> >> bottleneck, ThreadPoolExecutors looks OK.
> >>
> >> Any ideas why write latency it is so high?
> >
> >
>
Re: Storage proxy write latency is too high
Posted by Jonathan Ellis <jb...@gmail.com>.
so for a 200KB value you are seeing >200ms of latency in
MessagingService? How much for a 1KB value?
-Jonathan
On Mon, Oct 5, 2009 at 1:46 PM, Igor Katkov <ik...@gmail.com> wrote:
> In case anyone is following this, here is an update:
>
> I was able to narrow it down to Cassandra-Cassandra link. Storage proxy
> latency depends on size of the key. The larger amount of data (per key) is
> transfered the larger latency is. No surprise here.
> Client connects to a demons "A" and sends key-value, "A" accept thrift
> message, de-serialize it to an object, sees that key belongs to demons "B",
> serialize it to bytes once again (internal format now) and invoke
> MessagingService, which in turn writes to a socket. As soon as "B" delivers
> write-acknowledgment over a different connection, the client call is let go.
> Cassandra's MessagingService utilizes java nio to connect to other cassandra
> daemons, all connections are uni-directionals. So in theory it should be
> very fast. But it's not.
>
> What does look suspicious is certain network usage cap, only ~4% of the
> 1Gbps link is used regardless of "value" size. With smaller value I get a
> better throughput, with larger (200Kb) - worse.
>
> As a temp workaround I see that client might be held responsible to
> identifying what cassandra instance it should send a key to. On 200kb value
> it's ~10 times faster.
>
>
> On Thu, Oct 1, 2009 at 6:51 PM, Igor Katkov <ik...@gmail.com> wrote:
>>
>> Hi,
>>
>> I have the following puzzle:
>> Storage proxy write latency ~235ms
>> CF write latency <1 ms
>>
>> I have 3 nodes in the cluster, Cassandra v.0.4. Tokens evenly distributed.
>> The client connects to a node and inserts a key with ConsistencyLevel.ONE
>> If it happen to be a local write operation is fast, same speed as in one
>> node setup. JMX shows write latency <1 ms
>> If it happens to be a remote insert StorageProxy sends it to a proper
>> node. This operation is slow. JMX shows write latency ~ 235ms.
>> In the same time, on remote node JMX shows same <1ms write latency. So
>> it's not remote node being sluggish, it's something else.
>> There are no pending tasks on remote node - JMX counters are always zero,
>> network is 1Gb, idle. So I can't blame it.
>>
>>
>> I profiled Cassandra server in JProfiler, could not find a thing. All this
>> extra time is spent inside QuorumResponseHandler waiting for the condition
>> to signal. Which should happen as soon as response is received.
>>
>> There is one pooled TCP connection open to remote host. Hardly a
>> bottleneck, ThreadPoolExecutors looks OK.
>>
>> Any ideas why write latency it is so high?
>
>
Re: Storage proxy write latency is too high
Posted by Igor Katkov <ik...@gmail.com>.
In case anyone is following this, here is an update:
I was able to narrow it down to Cassandra-Cassandra link. Storage proxy
latency depends on size of the key. The larger amount of data (per key) is
transfered the larger latency is. No surprise here.
Client connects to a demons "A" and sends key-value, "A" accept thrift
message, de-serialize it to an object, sees that key belongs to demons "B",
serialize it to bytes once again (internal format now) and invoke
MessagingService, which in turn writes to a socket. As soon as "B" delivers
write-acknowledgment over a different connection, the client call is let go.
Cassandra's MessagingService utilizes java nio to connect to other cassandra
daemons, all connections are uni-directionals. So in theory it should be
very fast. But it's not.
What does look suspicious is certain network usage cap, only ~4% of the
1Gbps link is used regardless of "value" size. With smaller value I get a
better throughput, with larger (200Kb) - worse.
As a temp workaround I see that client might be held responsible to
identifying what cassandra instance it should send a key to. On 200kb value
it's ~10 times faster.
On Thu, Oct 1, 2009 at 6:51 PM, Igor Katkov <ik...@gmail.com> wrote:
> Hi,
>
> I have the following puzzle:
> Storage proxy write latency ~235ms
> CF write latency <1 ms
>
> I have 3 nodes in the cluster, Cassandra v.0.4. Tokens evenly distributed.
> The client connects to a node and inserts a key with ConsistencyLevel.ONE
> If it happen to be a local write operation is fast, same speed as in one
> node setup. JMX shows write latency <1 ms
> If it happens to be a remote insert StorageProxy sends it to a proper node.
> This operation is slow. JMX shows write latency ~ 235ms.
> In the same time, on remote node JMX shows same <1ms write latency. So it's
> not remote node being sluggish, it's something else.
> There are no pending tasks on remote node - JMX counters are always zero,
> network is 1Gb, idle. So I can't blame it.
>
>
> I profiled Cassandra server in JProfiler, could not find a thing. All this
> extra time is spent inside QuorumResponseHandler waiting for the condition
> to signal. Which should happen as soon as response is received.
>
> There is one pooled TCP connection open to remote host. Hardly a
> bottleneck, ThreadPoolExecutors looks OK.
>
> Any ideas why write latency it is so high?
>