You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Romain HARDOUIN <ro...@urssaf.fr> on 2012/04/10 16:24:01 UTC
Why so many SSTables?
Hi,
We are surprised by the number of files generated by Cassandra.
Our cluster consists of 9 nodes and each node handles about 35 GB.
We're using Cassandra 1.0.6 with LeveledCompactionStrategy.
We have 30 CF.
We've got roughly 45,000 files under the keyspace directory on each node:
ls -l /var/lib/cassandra/data/OurKeyspace/ | wc -l
44372
The biggest CF is spread over 38,000 files:
ls -l Documents* | wc -l
37870
ls -l Documents*-Data.db | wc -l
7586
Many SSTable are about 4 MB:
19 MB -> 1 SSTable
12 MB -> 2 SSTables
11 MB -> 2 SSTables
9.2 MB -> 1 SSTable
7.0 MB to 7.9 MB -> 6 SSTables
6.0 MB to 6.4 MB -> 6 SSTables
5.0 MB to 5.4 MB -> 4 SSTables
4.0 MB to 4.7 MB -> 7139 SSTables
3.0 MB to 3.9 MB -> 258 SSTables
2.0 MB to 2.9 MB -> 35 SSTables
1.0 MB to 1.9 MB -> 13 SSTables
87 KB to 994 KB -> 87 SSTables
0 KB -> 32 SSTables
FYI here is CF information:
ColumnFamily: Documents
Key Validation Class: org.apache.cassandra.db.marshal.BytesType
Default column value validator:
org.apache.cassandra.db.marshal.BytesType
Columns sorted by: org.apache.cassandra.db.marshal.BytesType
Row cache size / save period in seconds / keys to save : 0.0/0/all
Row Cache Provider: org.apache.cassandra.cache.SerializingCacheProvider
Key cache size / save period in seconds: 200000.0/14400
GC grace seconds: 1728000
Compaction min/max thresholds: 4/32
Read repair chance: 1.0
Replicate on write: true
Column Metadata:
Column Name: refUUID (72656655554944)
Validation Class: org.apache.cassandra.db.marshal.BytesType
Index Name: refUUID_idx
Index Type: KEYS
Compaction Strategy:
org.apache.cassandra.db.compaction.LeveledCompactionStrategy
Compression Options:
sstable_compression: org.apache.cassandra.io.compress.SnappyCompressor
Is it a bug? If not, how can we tune Cassandra to avoid this?
Regards,
Romain
Re: Why so many SSTables?
Posted by Sylvain Lebresne <sy...@datastax.com>.
On Wed, Apr 11, 2012 at 2:43 PM, Romain HARDOUIN
<ro...@urssaf.fr> wrote:
>
> Thank you for your answers.
>
> I originally post this question because we encoutered an OOM Exception on 2
> nodes during repair session.
> Memory analyzing shows an hotspot: an ArrayList of SSTableBoundedScanner
> which contains as many objects there are SSTables on disk (7747 objects at
> the time).
> This ArrayList consumes 47% of the heap space (786 MB).
That's 101KB per element!! I know Java object representation is not
concise but that feels like more than is reasonable. Are you sure of
those numbers?
In any case, we should improve that so as to not create all those
SSTableBoundedScanner upfront. Would you mind opening a ticket on
https://issues.apache.org/jira/browse/CASSANDRA with as many info as
you have on this.
> We want each node to handle 1 TB, so we must dramatically reduce the number
> of SSTables.
>
> Thus, is there any drawback if we set sstable_size_in_mb to 200MB?
> Otherwise shoudl we go back to Tiered Compaction?
>
> Regards,
>
> Romain
>
>
> Maki Watanabe <wa...@gmail.com> a écrit sur 11/04/2012 04:21:47 :
>
>
>> You can configure sstable size by sstable_size_in_mb parameter for LCS.
>> The default value is 5MB.
>> You should better to check you don't have many pending compaction tasks
>> with nodetool tpstats and compactionstats also.
>> If you have enough IO throughput, you can increase
>> compaction_throughput_mb_per_sec
>> in cassandra.yaml to reduce pending compactions.
>>
>> maki
>>
>> 2012/4/10 Romain HARDOUIN <ro...@urssaf.fr>:
>> >
>> > Hi,
>> >
>> > We are surprised by the number of files generated by Cassandra.
>> > Our cluster consists of 9 nodes and each node handles about 35 GB.
>> > We're using Cassandra 1.0.6 with LeveledCompactionStrategy.
>> > We have 30 CF.
>> >
>> > We've got roughly 45,000 files under the keyspace directory on each
>> > node:
>> > ls -l /var/lib/cassandra/data/OurKeyspace/ | wc -l
>> > 44372
>> >
>> > The biggest CF is spread over 38,000 files:
>> > ls -l Documents* | wc -l
>> > 37870
>> >
>> > ls -l Documents*-Data.db | wc -l
>> > 7586
>> >
>> > Many SSTable are about 4 MB:
>> >
>> > 19 MB -> 1 SSTable
>> > 12 MB -> 2 SSTables
>> > 11 MB -> 2 SSTables
>> > 9.2 MB -> 1 SSTable
>> > 7.0 MB to 7.9 MB -> 6 SSTables
>> > 6.0 MB to 6.4 MB -> 6 SSTables
>> > 5.0 MB to 5.4 MB -> 4 SSTables
>> > 4.0 MB to 4.7 MB -> 7139 SSTables
>> > 3.0 MB to 3.9 MB -> 258 SSTables
>> > 2.0 MB to 2.9 MB -> 35 SSTables
>> > 1.0 MB to 1.9 MB -> 13 SSTables
>> > 87 KB to 994 KB -> 87 SSTables
>> > 0 KB -> 32 SSTables
>> >
>> > FYI here is CF information:
>> >
>> > ColumnFamily: Documents
>> > Key Validation Class: org.apache.cassandra.db.marshal.BytesType
>> > Default column value validator:
>> > org.apache.cassandra.db.marshal.BytesType
>> > Columns sorted by: org.apache.cassandra.db.marshal.BytesType
>> > Row cache size / save period in seconds / keys to save : 0.0/0/all
>> > Row Cache Provider:
>> > org.apache.cassandra.cache.SerializingCacheProvider
>> > Key cache size / save period in seconds: 200000.0/14400
>> > GC grace seconds: 1728000
>> > Compaction min/max thresholds: 4/32
>> > Read repair chance: 1.0
>> > Replicate on write: true
>> > Column Metadata:
>> > Column Name: refUUID (72656655554944)
>> > Validation Class: org.apache.cassandra.db.marshal.BytesType
>> > Index Name: refUUID_idx
>> > Index Type: KEYS
>> > Compaction Strategy:
>> > org.apache.cassandra.db.compaction.LeveledCompactionStrategy
>> > Compression Options:
>> > sstable_compression:
>> > org.apache.cassandra.io.compress.SnappyCompressor
>> >
>> > Is it a bug? If not, how can we tune Cassandra to avoid this?
>> >
>> > Regards,
>> >
>> > Romain
Re: RMI/JMX errors, weird
Posted by Maxim Potekhin <po...@bnl.gov>.
Hello Aaron,
it's probably the over-optimistic number of concurrent compactors that
was tripping the system.
I do not entirely understand what's the correlation here, maybe it's
that the compactors were overloading
the neighboring nodes causing time-outs. I tuned the concurrency down
and after a while things seem
to have settled down, thanks for the suggestion.
Maxim
On 4/19/2012 4:13 PM, aaron morton wrote:
>> 1150 pending tasks, and is not
>> making progress.
> Not all pending tasks reported by nodetool compactionstats actually
> run. Once they get a chance to run the files they were going to work
> on may have already been compacted.
>
> Given that repair tests at double the phi threshold, it may not make
> much difference.
>
> Did other nodes notice it was dead ? Was there anything in the log
> that showed it was under duress (GC or dropped message logs) ?
>
> Is the compaction a consequence of repair ? (The streaming stage can
> result in compactions). Or do you think the node is just behind on
> compactions ?
>
> If you feel compaction is hurting the node, consider
> setting concurrent_compactors in the yaml to 2.
>
> You can also isolate the node from updates using nodetool
> disablegossip and disablerthrift , and the turn off the IO limiter
> with nodetool setcompactionthroughput 0.
> Hope that helps.
> -----------------
> Aaron Morton
> Freelance Developer
> @aaronmorton
> http://www.thelastpickle.com
>
> On 20/04/2012, at 12:29 AM, Maxim Potekhin wrote:
>
>> Hello Aaron,
>>
>> how should I go about fixing that? Also, after a repeated attempt to
>> compact
>> it goes again into "building secondary index" with 1150 pending
>> tasks, and is not
>> making progress. I suspected the disk system failure, but this needs
>> to be confirmed.
>>
>> So basically, do I need to tune the phi threshold up? Thing is, there
>> was no heavy load
>> on the cluster at all.
>>
>> Thanks
>>
>> Maxim
>>
>>
>>
>>
>> On 4/19/2012 7:06 AM, aaron morton wrote:
>>> At some point the gossip system on the node this log is from decided
>>> that 130.199.185.195 was DOWN. This was based on how often the node
>>> was gossiping to the cluster.
>>>
>>> The active repair session was informed. And to avoid failing the job
>>> unnecessarily it tested that the errant nodes phi value was twice
>>> the configured phi_convict_threshold. It was and the repair was killed.
>>>
>>> Take a look at the logs on 130.199.185.195 and see if anything was
>>> happening on the node at the same time. Could be GC or an
>>> overloaded node (it would log about dropped messages).
>>>
>>> Perhaps other nodes also saw 130.199.185.195 as down? it only needed
>>> to be down for a few seconds.
>>>
>>> Hope that helps.
>>>
>>> -----------------
>>> Aaron Morton
>>>
Re: RMI/JMX errors, weird
Posted by aaron morton <aa...@thelastpickle.com>.
> 1150 pending tasks, and is not
> making progress.
Not all pending tasks reported by nodetool compactionstats actually run. Once they get a chance to run the files they were going to work on may have already been compacted.
Given that repair tests at double the phi threshold, it may not make much difference.
Did other nodes notice it was dead ? Was there anything in the log that showed it was under duress (GC or dropped message logs) ?
Is the compaction a consequence of repair ? (The streaming stage can result in compactions). Or do you think the node is just behind on compactions ?
If you feel compaction is hurting the node, consider setting concurrent_compactors in the yaml to 2.
You can also isolate the node from updates using nodetool disablegossip and disablerthrift , and the turn off the IO limiter with nodetool setcompactionthroughput 0.
Hope that helps.
-----------------
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com
On 20/04/2012, at 12:29 AM, Maxim Potekhin wrote:
> Hello Aaron,
>
> how should I go about fixing that? Also, after a repeated attempt to compact
> it goes again into "building secondary index" with 1150 pending tasks, and is not
> making progress. I suspected the disk system failure, but this needs to be confirmed.
>
> So basically, do I need to tune the phi threshold up? Thing is, there was no heavy load
> on the cluster at all.
>
> Thanks
>
> Maxim
>
>
>
>
> On 4/19/2012 7:06 AM, aaron morton wrote:
>>
>> At some point the gossip system on the node this log is from decided that 130.199.185.195 was DOWN. This was based on how often the node was gossiping to the cluster.
>>
>> The active repair session was informed. And to avoid failing the job unnecessarily it tested that the errant nodes phi value was twice the configured phi_convict_threshold. It was and the repair was killed.
>>
>> Take a look at the logs on 130.199.185.195 and see if anything was happening on the node at the same time. Could be GC or an overloaded node (it would log about dropped messages).
>>
>> Perhaps other nodes also saw 130.199.185.195 as down? it only needed to be down for a few seconds.
>>
>> Hope that helps.
>>
>> -----------------
>> Aaron Morton
>> Freelance Developer
>> @aaronmorton
>> http://www.thelastpickle.com
>>
>> On 19/04/2012, at 2:25 PM, Maxim Potekhin wrote:
>>
>>> Server log below. Mind you that all the nodes are still up -- even though reported as "dead" in this log.
>>> What's going on here?
>>>
>>> Thanks!
>>>
>>> INFO [GossipTasks:1] 2012-04-18 22:18:26,487 Gossiper.java (line 719) InetAddress /130.199.185.193 is now dead.
>>> INFO [ScheduledTasks:1] 2012-04-18 22:18:26,487 StatusLogger.java (line 50) Pool Name Active Pending Blocked
>>> ERROR [GossipTasks:1] 2012-04-18 22:18:26,488 AntiEntropyService.java (line 722) Problem during repair session manual-repair-1b3453b
>>> 6-28b5-4abd-84ce-0326b5468064, endpoint /130.199.185.193 died
>>> ERROR [RMI TCP Connection(22)-130.199.185.194] 2012-04-18 22:18:26,488 StorageService.java (line 1607) Repair session org.apache.cas
>>> sandra.service.AntiEntropyService$RepairSession@4cc9e2bc failed.
>>> java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.io.IOException: Problem during repair session manual-repai
>>> r-43545b22-ffe8-4243-8a98-509bbfec9872, endpoint /130.199.185.195 died
>>> at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
>>> at java.util.concurrent.FutureTask.get(FutureTask.java:83)
>>> at org.apache.cassandra.service.StorageService.forceTableRepair(StorageService.java:1603)
>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>> at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:93)
>>> at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:27)
>>> at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:208)
>>> at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:120)
>>> at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:262)
>>> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:836)
>>> at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:761)
>>> at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1427)
>>> at javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:72)
>>> at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1265)
>>> at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1360)
>>> at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:788)
>>> at sun.reflect.GeneratedMethodAccessor13.invoke(Unknown Source)
>>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>> at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:303)
>>> at sun.rmi.transport.Transport$1.run(Transport.java:159)
>>> at java.security.AccessController.doPrivileged(Native Method)
>>> at sun.rmi.transport.Transport.serviceCall(Transport.java:155)
>>> at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:535)
>>> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790)
>>> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649)
>>> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>> at java.lang.Thread.run(Thread.java:662)
>>> Caused by: java.lang.RuntimeException: java.io.IOException: Problem during repair session manual-repair-43545b22-ffe8-4243-8a98-509b
>>> bfec9872, endpoint /130.199.185.195 died
>>> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
>>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
>>> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>>> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>>> ... 3 more
>>> Caused by: java.io.IOException: Problem during repair session manual-repair-43545b22-ffe8-4243-8a98-509bbfec9872, endpoint /130.199.
>>> 185.195 died
>>> at org.apache.cassandra.service.AntiEntropyService$RepairSession.failedNode(AntiEntropyService.java:723)
>>> at org.apache.cassandra.service.AntiEntropyService$RepairSession.convict(AntiEntropyService.java:760)
>>> at org.apache.cassandra.gms.FailureDetector.interpret(FailureDetector.java:165)
>>> at org.apache.cassandra.gms.Gossiper.doStatusCheck(Gossiper.java:538)
>>> at org.apache.cassandra.gms.Gossiper.access$700(Gossiper.java:57)
>>> at org.apache.cassandra.gms.Gossiper$GossipTask.run(Gossiper.java:157)
>>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
>>> at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
>>> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
>>> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
>>> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)
>>> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)
>>> ... 3 more
>>> INFO [FlushWriter:4] 2012-04-18 22:18:26,489 Memtable.java (line 237) Writing Memtable-files.pandaid@988533693(1754325/140346000 se
>>> rialized/live bytes, 70173 ops)
>>> ERROR [AntiEntropySessions:8] 2012-04-18 22:18:26,488 AntiEntropyService.java (line 691) Repair session manual-repair-1b3453b6-28b5-
>>> 4abd-84ce-0326b5468064 (on cfs [Ljava.lang.String;@5a44cc06, range (113427455640312821154458202477256070484,0]) failed with the foll
>>> owing error
>>> java.io.IOException: Problem during repair session manual-repair-1b3453b6-28b5-4abd-84ce-0326b5468064, endpoint /130.199.185.193 die
>>> d
>>> at org.apache.cassandra.service.AntiEntropyService$RepairSession.failedNode(AntiEntropyService.java:723)
>>> at org.apache.cassandra.service.AntiEntropyService$RepairSession.convict(AntiEntropyService.java:760)
>>> at org.apache.cassandra.gms.FailureDetector.interpret(FailureDetector.java:165)
>>> at org.apache.cassandra.gms.Gossiper.doStatusCheck(Gossiper.java:538)
>>> at org.apache.cassandra.gms.Gossiper.access$700(Gossiper.java:57)
>>> at org.apache.cassandra.gms.Gossiper$GossipTask.run(Gossiper.java:157)
>>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
>>> at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
>>> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
>>> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
>>> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)
>>> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)
>>> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>> at java.lang.Thread.run(Thread.java:662)
>>> INFO [ScheduledTasks:1] 2012-04-18 22:18:26,488 StatusLogger.java (line 65) ReadStage 0 0 0
>>> ERROR [RMI TCP Connection(22)-130.199.185.194] 2012-04-18 22:18:26,491 StorageService.java (line 1607) Repair session org.apache.cas
>>> sandra.service.AntiEntropyService$RepairSession@41bb342c failed.
>>> java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.io.IOException: Problem during repair session manual-repai
>>> r-1b3453b6-28b5-4abd-84ce-0326b5468064, endpoint /130.199.185.193 died
>>> at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
>>> at java.util.concurrent.FutureTask.get(FutureTask.java:83)
>>> at org.apache.cassandra.service.StorageService.forceTableRepair(StorageService.java:1603)
>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>> at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:93)
>>> at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:27)
>>> at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:208)
>>> at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:120)
>>> at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:262)
>>> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:836)
>>> at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:761)
>>> at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1427)
>>> at javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:72)
>>> at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1265)
>>> at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1360)
>>> at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:788)
>>> at sun.reflect.GeneratedMethodAccessor13.invoke(Unknown Source)
>>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>> at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:303)
>>> at sun.rmi.transport.Transport$1.run(Transport.java:159)
>>> at java.security.AccessController.doPrivileged(Native Method)
>>> at sun.rmi.transport.Transport.serviceCall(Transport.java:155)
>>> at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:535)
>>> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790)
>>> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649)
>>> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>> at java.lang.Thread.run(Thread.java:662)
>>> Caused by: java.lang.RuntimeException: java.io.IOException: Problem during repair session manual-repair-1b3453b6-28b5-4abd-84ce-0326
>>> b5468064, endpoint /130.199.185.193 died
>>> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
>>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
>>> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>>> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>>> ... 3 more
>>> Caused by: java.io.IOException: Problem during repair session manual-repair-1b3453b6-28b5-4abd-84ce-0326b5468064, endpoint /130.199.
>>> 185.193 died
>>> at org.apache.cassandra.service.AntiEntropyService$RepairSession.failedNode(AntiEntropyService.java:723)
>>> at org.apache.cassandra.service.AntiEntropyService$RepairSession.convict(AntiEntropyService.java:760)
>>> at org.apache.cassandra.gms.FailureDetector.interpret(FailureDetector.java:165)
>>> at org.apache.cassandra.gms.Gossiper.doStatusCheck(Gossiper.java:538)
>>> at org.apache.cassandra.gms.Gossiper.access$700(Gossiper.java:57)
>>> at org.apache.cassandra.gms.Gossiper$GossipTask.run(Gossiper.java:157)
>>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
>>> at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
>>>
>>>
>>>
>>> On 4/12/2012 10:03 PM, aaron morton wrote:
>>>>
>>>> Look at the server side logs for errors.
>>>>
>>>> Cheers
>>>>
>>>> -----------------
>>>> Aaron Morton
>>>> Freelance Developer
>>>> @aaronmorton
>>>> http://www.thelastpickle.com
>>>>
>>>> On 13/04/2012, at 11:47 AM, Maxim Potekhin wrote:
>>>>
>>>>> Hello,
>>>>>
>>>>> I'm doing compactions under 0.8.8.
>>>>>
>>>>> Recently, I started seeing a stack trace like one below, and I can't figure out what causes this to appear.
>>>>> The cluster has been in operation for mode than half a year w/o errors like this one.
>>>>>
>>>>> Any help will be appreciated,
>>>>> Thanks
>>>>>
>>>>> Maxim
>>>>>
>>>>>
>>>>> WARNING: Failed to check the connection: java.net.SocketTimeoutException: Read timed out
>>>>> Exception in thread "main" java.io.IOException: Repair command #1: some repair session(s) failed (see log for details).
>>>>> at org.apache.cassandra.service.StorageService.forceTableRepair(StorageService.java:1613)
>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>>>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>>>> at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:93)
>>>>> at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:27)
>>>>> at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:208)
>>>>> at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:120)
>>>>> at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:262)
>>>>> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:836)
>>>>> at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:761)
>>>>> at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1427)
>>>>> at javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:72)
>>>>> at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1265)
>>>>> at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1360)
>>>>> at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:788)
>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>>>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>>>> at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:303)
>>>>> at sun.rmi.transport.Transport$1.run(Transport.java:159)
>>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>>> at sun.rmi.transport.Transport.serviceCall(Transport.java:155)
>>>>> at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:535)
>>>>> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790)
>>>>> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649)
>>>>> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>> at java.lang.Thread.run(Thread.java:662)
>>>>>
>>>>
>>>
>>
>
Re: RMI/JMX errors, weird
Posted by Maxim Potekhin <po...@bnl.gov>.
Hello Aaron,
how should I go about fixing that? Also, after a repeated attempt to compact
it goes again into "building secondary index" with 1150 pending tasks,
and is not
making progress. I suspected the disk system failure, but this needs to
be confirmed.
So basically, do I need to tune the phi threshold up? Thing is, there
was no heavy load
on the cluster at all.
Thanks
Maxim
On 4/19/2012 7:06 AM, aaron morton wrote:
> At some point the gossip system on the node this log is from decided
> that 130.199.185.195 was DOWN. This was based on how often the node
> was gossiping to the cluster.
>
> The active repair session was informed. And to avoid failing the job
> unnecessarily it tested that the errant nodes phi value was twice the
> configured phi_convict_threshold. It was and the repair was killed.
>
> Take a look at the logs on 130.199.185.195 and see if anything was
> happening on the node at the same time. Could be GC or an overloaded
> node (it would log about dropped messages).
>
> Perhaps other nodes also saw 130.199.185.195 as down? it only needed
> to be down for a few seconds.
>
> Hope that helps.
>
> -----------------
> Aaron Morton
> Freelance Developer
> @aaronmorton
> http://www.thelastpickle.com
>
> On 19/04/2012, at 2:25 PM, Maxim Potekhin wrote:
>
>> Server log below. Mind you that all the nodes are still up -- even
>> though reported as "dead" in this log.
>> What's going on here?
>>
>> Thanks!
>>
>> INFO [GossipTasks:1] 2012-04-18 22:18:26,487 Gossiper.java (line
>> 719) InetAddress /130.199.185.193 is now dead.
>> INFO [ScheduledTasks:1] 2012-04-18 22:18:26,487 StatusLogger.java
>> (line 50) Pool Name Active Pending Blocked
>> ERROR [GossipTasks:1] 2012-04-18 22:18:26,488 AntiEntropyService.java
>> (line 722) Problem during repair session manual-repair-1b3453b
>> 6-28b5-4abd-84ce-0326b5468064, endpoint /130.199.185.193 died
>> ERROR [RMI TCP Connection(22)-130.199.185.194] 2012-04-18
>> 22:18:26,488 StorageService.java (line 1607) Repair session
>> org.apache.cas
>> sandra.service.AntiEntropyService$RepairSession@4cc9e2bc failed.
>> java.util.concurrent.ExecutionException: java.lang.RuntimeException:
>> java.io.IOException: Problem during repair session manual-repai
>> r-43545b22-ffe8-4243-8a98-509bbfec9872, endpoint /130.199.185.195 died
>> at
>> java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
>> at java.util.concurrent.FutureTask.get(FutureTask.java:83)
>> at
>> org.apache.cassandra.service.StorageService.forceTableRepair(StorageService.java:1603)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> at java.lang.reflect.Method.invoke(Method.java:597)
>> at
>> com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:93)
>> at
>> com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:27)
>> at
>> com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:208)
>> at
>> com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:120)
>> at
>> com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:262)
>> at
>> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:836)
>> at
>> com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:761)
>> at
>> javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1427)
>> at
>> javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:72)
>> at
>> javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1265)
>> at
>> javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1360)
>> at
>> javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:788)
>> at sun.reflect.GeneratedMethodAccessor13.invoke(Unknown Source)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> at java.lang.reflect.Method.invoke(Method.java:597)
>> at
>> sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:303)
>> at sun.rmi.transport.Transport$1.run(Transport.java:159)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at sun.rmi.transport.Transport.serviceCall(Transport.java:155)
>> at
>> sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:535)
>> at
>> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790)
>> at
>> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>> at java.lang.Thread.run(Thread.java:662)
>> Caused by: java.lang.RuntimeException: java.io.IOException: Problem
>> during repair session manual-repair-43545b22-ffe8-4243-8a98-509b
>> bfec9872, endpoint /130.199.185.195 died
>> at
>> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
>> at
>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
>> at
>> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>> ... 3 more
>> Caused by: java.io.IOException: Problem during repair session
>> manual-repair-43545b22-ffe8-4243-8a98-509bbfec9872, endpoint /130.199.
>> 185.195 died
>> at
>> org.apache.cassandra.service.AntiEntropyService$RepairSession.failedNode(AntiEntropyService.java:723)
>> at
>> org.apache.cassandra.service.AntiEntropyService$RepairSession.convict(AntiEntropyService.java:760)
>> at
>> org.apache.cassandra.gms.FailureDetector.interpret(FailureDetector.java:165)
>> at
>> org.apache.cassandra.gms.Gossiper.doStatusCheck(Gossiper.java:538)
>> at org.apache.cassandra.gms.Gossiper.access$700(Gossiper.java:57)
>> at
>> org.apache.cassandra.gms.Gossiper$GossipTask.run(Gossiper.java:157)
>> at
>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
>> at
>> java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
>> at
>> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)
>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)
>> ... 3 more
>> INFO [FlushWriter:4] 2012-04-18 22:18:26,489 Memtable.java (line
>> 237) Writing Memtable-files.pandaid@988533693(1754325/140346000 se
>> rialized/live bytes, 70173 ops)
>> ERROR [AntiEntropySessions:8] 2012-04-18 22:18:26,488
>> AntiEntropyService.java (line 691) Repair session
>> manual-repair-1b3453b6-28b5-
>> 4abd-84ce-0326b5468064 (on cfs [Ljava.lang.String;@5a44cc06, range
>> (113427455640312821154458202477256070484,0]) failed with the foll
>> owing error
>> java.io.IOException: Problem during repair session
>> manual-repair-1b3453b6-28b5-4abd-84ce-0326b5468064, endpoint
>> /130.199.185.193 die
>> d
>> at
>> org.apache.cassandra.service.AntiEntropyService$RepairSession.failedNode(AntiEntropyService.java:723)
>> at
>> org.apache.cassandra.service.AntiEntropyService$RepairSession.convict(AntiEntropyService.java:760)
>> at
>> org.apache.cassandra.gms.FailureDetector.interpret(FailureDetector.java:165)
>> at
>> org.apache.cassandra.gms.Gossiper.doStatusCheck(Gossiper.java:538)
>> at org.apache.cassandra.gms.Gossiper.access$700(Gossiper.java:57)
>> at
>> org.apache.cassandra.gms.Gossiper$GossipTask.run(Gossiper.java:157)
>> at
>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
>> at
>> java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
>> at
>> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)
>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>> at java.lang.Thread.run(Thread.java:662)
>> INFO [ScheduledTasks:1] 2012-04-18 22:18:26,488 StatusLogger.java
>> (line 65) ReadStage 0 0 0
>> ERROR [RMI TCP Connection(22)-130.199.185.194] 2012-04-18
>> 22:18:26,491 StorageService.java (line 1607) Repair session
>> org.apache.cas
>> sandra.service.AntiEntropyService$RepairSession@41bb342c failed.
>> java.util.concurrent.ExecutionException: java.lang.RuntimeException:
>> java.io.IOException: Problem during repair session manual-repai
>> r-1b3453b6-28b5-4abd-84ce-0326b5468064, endpoint /130.199.185.193 died
>> at
>> java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
>> at java.util.concurrent.FutureTask.get(FutureTask.java:83)
>> at
>> org.apache.cassandra.service.StorageService.forceTableRepair(StorageService.java:1603)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> at java.lang.reflect.Method.invoke(Method.java:597)
>> at
>> com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:93)
>> at
>> com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:27)
>> at
>> com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:208)
>> at
>> com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:120)
>> at
>> com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:262)
>> at
>> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:836)
>> at
>> com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:761)
>> at
>> javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1427)
>> at
>> javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:72)
>> at
>> javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1265)
>> at
>> javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1360)
>> at
>> javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:788)
>> at sun.reflect.GeneratedMethodAccessor13.invoke(Unknown Source)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> at java.lang.reflect.Method.invoke(Method.java:597)
>> at
>> sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:303)
>> at sun.rmi.transport.Transport$1.run(Transport.java:159)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at sun.rmi.transport.Transport.serviceCall(Transport.java:155)
>> at
>> sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:535)
>> at
>> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790)
>> at
>> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>> at java.lang.Thread.run(Thread.java:662)
>> Caused by: java.lang.RuntimeException: java.io.IOException: Problem
>> during repair session manual-repair-1b3453b6-28b5-4abd-84ce-0326
>> b5468064, endpoint /130.199.185.193 died
>> at
>> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
>> at
>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
>> at
>> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>> ... 3 more
>> Caused by: java.io.IOException: Problem during repair session
>> manual-repair-1b3453b6-28b5-4abd-84ce-0326b5468064, endpoint /130.199.
>> 185.193 died
>> at
>> org.apache.cassandra.service.AntiEntropyService$RepairSession.failedNode(AntiEntropyService.java:723)
>> at
>> org.apache.cassandra.service.AntiEntropyService$RepairSession.convict(AntiEntropyService.java:760)
>> at
>> org.apache.cassandra.gms.FailureDetector.interpret(FailureDetector.java:165)
>> at
>> org.apache.cassandra.gms.Gossiper.doStatusCheck(Gossiper.java:538)
>> at org.apache.cassandra.gms.Gossiper.access$700(Gossiper.java:57)
>> at
>> org.apache.cassandra.gms.Gossiper$GossipTask.run(Gossiper.java:157)
>> at
>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
>> at
>> java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
>>
>>
>>
>> On 4/12/2012 10:03 PM, aaron morton wrote:
>>> Look at the server side logs for errors.
>>>
>>> Cheers
>>>
>>> -----------------
>>> Aaron Morton
>>> Freelance Developer
>>> @aaronmorton
>>> http://www.thelastpickle.com <http://www.thelastpickle.com/>
>>>
>>> On 13/04/2012, at 11:47 AM, Maxim Potekhin wrote:
>>>
>>>> Hello,
>>>>
>>>> I'm doing compactions under 0.8.8.
>>>>
>>>> Recently, I started seeing a stack trace like one below, and I
>>>> can't figure out what causes this to appear.
>>>> The cluster has been in operation for mode than half a year w/o
>>>> errors like this one.
>>>>
>>>> Any help will be appreciated,
>>>> Thanks
>>>>
>>>> Maxim
>>>>
>>>>
>>>> WARNING: Failed to check the connection:
>>>> java.net.SocketTimeoutException: Read timed out
>>>> Exception in thread "main" java.io.IOException: Repair command #1:
>>>> some repair session(s) failed (see log for details).
>>>> at
>>>> org.apache.cassandra.service.StorageService.forceTableRepair(StorageService.java:1613)
>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>> at
>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>>> at
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>>> at
>>>> com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:93)
>>>> at
>>>> com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:27)
>>>> at
>>>> com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:208)
>>>> at
>>>> com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:120)
>>>> at
>>>> com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:262)
>>>> at
>>>> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:836)
>>>> at
>>>> com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:761)
>>>> at
>>>> javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1427)
>>>> at
>>>> javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:72)
>>>> at
>>>> javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1265)
>>>> at
>>>> javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1360)
>>>> at
>>>> javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:788)
>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>> at
>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>>> at
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>>> at
>>>> sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:303)
>>>> at sun.rmi.transport.Transport$1.run(Transport.java:159)
>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>> at sun.rmi.transport.Transport.serviceCall(Transport.java:155)
>>>> at
>>>> sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:535)
>>>> at
>>>> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790)
>>>> at
>>>> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>> at java.lang.Thread.run(Thread.java:662)
>>>>
>>>
>>
>
Re: RMI/JMX errors, weird
Posted by aaron morton <aa...@thelastpickle.com>.
At some point the gossip system on the node this log is from decided that 130.199.185.195 was DOWN. This was based on how often the node was gossiping to the cluster.
The active repair session was informed. And to avoid failing the job unnecessarily it tested that the errant nodes phi value was twice the configured phi_convict_threshold. It was and the repair was killed.
Take a look at the logs on 130.199.185.195 and see if anything was happening on the node at the same time. Could be GC or an overloaded node (it would log about dropped messages).
Perhaps other nodes also saw 130.199.185.195 as down? it only needed to be down for a few seconds.
Hope that helps.
-----------------
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com
On 19/04/2012, at 2:25 PM, Maxim Potekhin wrote:
> Server log below. Mind you that all the nodes are still up -- even though reported as "dead" in this log.
> What's going on here?
>
> Thanks!
>
> INFO [GossipTasks:1] 2012-04-18 22:18:26,487 Gossiper.java (line 719) InetAddress /130.199.185.193 is now dead.
> INFO [ScheduledTasks:1] 2012-04-18 22:18:26,487 StatusLogger.java (line 50) Pool Name Active Pending Blocked
> ERROR [GossipTasks:1] 2012-04-18 22:18:26,488 AntiEntropyService.java (line 722) Problem during repair session manual-repair-1b3453b
> 6-28b5-4abd-84ce-0326b5468064, endpoint /130.199.185.193 died
> ERROR [RMI TCP Connection(22)-130.199.185.194] 2012-04-18 22:18:26,488 StorageService.java (line 1607) Repair session org.apache.cas
> sandra.service.AntiEntropyService$RepairSession@4cc9e2bc failed.
> java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.io.IOException: Problem during repair session manual-repai
> r-43545b22-ffe8-4243-8a98-509bbfec9872, endpoint /130.199.185.195 died
> at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
> at java.util.concurrent.FutureTask.get(FutureTask.java:83)
> at org.apache.cassandra.service.StorageService.forceTableRepair(StorageService.java:1603)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:93)
> at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:27)
> at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:208)
> at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:120)
> at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:262)
> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:836)
> at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:761)
> at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1427)
> at javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:72)
> at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1265)
> at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1360)
> at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:788)
> at sun.reflect.GeneratedMethodAccessor13.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:303)
> at sun.rmi.transport.Transport$1.run(Transport.java:159)
> at java.security.AccessController.doPrivileged(Native Method)
> at sun.rmi.transport.Transport.serviceCall(Transport.java:155)
> at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:535)
> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790)
> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> Caused by: java.lang.RuntimeException: java.io.IOException: Problem during repair session manual-repair-43545b22-ffe8-4243-8a98-509b
> bfec9872, endpoint /130.199.185.195 died
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> ... 3 more
> Caused by: java.io.IOException: Problem during repair session manual-repair-43545b22-ffe8-4243-8a98-509bbfec9872, endpoint /130.199.
> 185.195 died
> at org.apache.cassandra.service.AntiEntropyService$RepairSession.failedNode(AntiEntropyService.java:723)
> at org.apache.cassandra.service.AntiEntropyService$RepairSession.convict(AntiEntropyService.java:760)
> at org.apache.cassandra.gms.FailureDetector.interpret(FailureDetector.java:165)
> at org.apache.cassandra.gms.Gossiper.doStatusCheck(Gossiper.java:538)
> at org.apache.cassandra.gms.Gossiper.access$700(Gossiper.java:57)
> at org.apache.cassandra.gms.Gossiper$GossipTask.run(Gossiper.java:157)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
> at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)
> ... 3 more
> INFO [FlushWriter:4] 2012-04-18 22:18:26,489 Memtable.java (line 237) Writing Memtable-files.pandaid@988533693(1754325/140346000 se
> rialized/live bytes, 70173 ops)
> ERROR [AntiEntropySessions:8] 2012-04-18 22:18:26,488 AntiEntropyService.java (line 691) Repair session manual-repair-1b3453b6-28b5-
> 4abd-84ce-0326b5468064 (on cfs [Ljava.lang.String;@5a44cc06, range (113427455640312821154458202477256070484,0]) failed with the foll
> owing error
> java.io.IOException: Problem during repair session manual-repair-1b3453b6-28b5-4abd-84ce-0326b5468064, endpoint /130.199.185.193 die
> d
> at org.apache.cassandra.service.AntiEntropyService$RepairSession.failedNode(AntiEntropyService.java:723)
> at org.apache.cassandra.service.AntiEntropyService$RepairSession.convict(AntiEntropyService.java:760)
> at org.apache.cassandra.gms.FailureDetector.interpret(FailureDetector.java:165)
> at org.apache.cassandra.gms.Gossiper.doStatusCheck(Gossiper.java:538)
> at org.apache.cassandra.gms.Gossiper.access$700(Gossiper.java:57)
> at org.apache.cassandra.gms.Gossiper$GossipTask.run(Gossiper.java:157)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
> at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> INFO [ScheduledTasks:1] 2012-04-18 22:18:26,488 StatusLogger.java (line 65) ReadStage 0 0 0
> ERROR [RMI TCP Connection(22)-130.199.185.194] 2012-04-18 22:18:26,491 StorageService.java (line 1607) Repair session org.apache.cas
> sandra.service.AntiEntropyService$RepairSession@41bb342c failed.
> java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.io.IOException: Problem during repair session manual-repai
> r-1b3453b6-28b5-4abd-84ce-0326b5468064, endpoint /130.199.185.193 died
> at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
> at java.util.concurrent.FutureTask.get(FutureTask.java:83)
> at org.apache.cassandra.service.StorageService.forceTableRepair(StorageService.java:1603)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:93)
> at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:27)
> at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:208)
> at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:120)
> at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:262)
> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:836)
> at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:761)
> at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1427)
> at javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:72)
> at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1265)
> at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1360)
> at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:788)
> at sun.reflect.GeneratedMethodAccessor13.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:303)
> at sun.rmi.transport.Transport$1.run(Transport.java:159)
> at java.security.AccessController.doPrivileged(Native Method)
> at sun.rmi.transport.Transport.serviceCall(Transport.java:155)
> at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:535)
> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790)
> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> Caused by: java.lang.RuntimeException: java.io.IOException: Problem during repair session manual-repair-1b3453b6-28b5-4abd-84ce-0326
> b5468064, endpoint /130.199.185.193 died
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> ... 3 more
> Caused by: java.io.IOException: Problem during repair session manual-repair-1b3453b6-28b5-4abd-84ce-0326b5468064, endpoint /130.199.
> 185.193 died
> at org.apache.cassandra.service.AntiEntropyService$RepairSession.failedNode(AntiEntropyService.java:723)
> at org.apache.cassandra.service.AntiEntropyService$RepairSession.convict(AntiEntropyService.java:760)
> at org.apache.cassandra.gms.FailureDetector.interpret(FailureDetector.java:165)
> at org.apache.cassandra.gms.Gossiper.doStatusCheck(Gossiper.java:538)
> at org.apache.cassandra.gms.Gossiper.access$700(Gossiper.java:57)
> at org.apache.cassandra.gms.Gossiper$GossipTask.run(Gossiper.java:157)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
> at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
>
>
>
> On 4/12/2012 10:03 PM, aaron morton wrote:
>>
>> Look at the server side logs for errors.
>>
>> Cheers
>>
>> -----------------
>> Aaron Morton
>> Freelance Developer
>> @aaronmorton
>> http://www.thelastpickle.com
>>
>> On 13/04/2012, at 11:47 AM, Maxim Potekhin wrote:
>>
>>> Hello,
>>>
>>> I'm doing compactions under 0.8.8.
>>>
>>> Recently, I started seeing a stack trace like one below, and I can't figure out what causes this to appear.
>>> The cluster has been in operation for mode than half a year w/o errors like this one.
>>>
>>> Any help will be appreciated,
>>> Thanks
>>>
>>> Maxim
>>>
>>>
>>> WARNING: Failed to check the connection: java.net.SocketTimeoutException: Read timed out
>>> Exception in thread "main" java.io.IOException: Repair command #1: some repair session(s) failed (see log for details).
>>> at org.apache.cassandra.service.StorageService.forceTableRepair(StorageService.java:1613)
>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>> at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:93)
>>> at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:27)
>>> at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:208)
>>> at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:120)
>>> at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:262)
>>> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:836)
>>> at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:761)
>>> at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1427)
>>> at javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:72)
>>> at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1265)
>>> at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1360)
>>> at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:788)
>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>> at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:303)
>>> at sun.rmi.transport.Transport$1.run(Transport.java:159)
>>> at java.security.AccessController.doPrivileged(Native Method)
>>> at sun.rmi.transport.Transport.serviceCall(Transport.java:155)
>>> at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:535)
>>> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790)
>>> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649)
>>> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>> at java.lang.Thread.run(Thread.java:662)
>>>
>>
>
Re: RMI/JMX errors, weird
Posted by Maxim Potekhin <po...@bnl.gov>.
Server log below. Mind you that all the nodes are still up -- even
though reported as "dead" in this log.
What's going on here?
Thanks!
INFO [GossipTasks:1] 2012-04-18 22:18:26,487 Gossiper.java (line 719)
InetAddress /130.199.185.193 is now dead.
INFO [ScheduledTasks:1] 2012-04-18 22:18:26,487 StatusLogger.java
(line 50) Pool Name Active Pending Blocked
ERROR [GossipTasks:1] 2012-04-18 22:18:26,488 AntiEntropyService.java
(line 722) Problem during repair session manual-repair-1b3453b
6-28b5-4abd-84ce-0326b5468064, endpoint /130.199.185.193 died
ERROR [RMI TCP Connection(22)-130.199.185.194] 2012-04-18 22:18:26,488
StorageService.java (line 1607) Repair session org.apache.cas
sandra.service.AntiEntropyService$RepairSession@4cc9e2bc failed.
java.util.concurrent.ExecutionException: java.lang.RuntimeException:
java.io.IOException: Problem during repair session manual-repai
r-43545b22-ffe8-4243-8a98-509bbfec9872, endpoint /130.199.185.195 died
at
java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
at java.util.concurrent.FutureTask.get(FutureTask.java:83)
at
org.apache.cassandra.service.StorageService.forceTableRepair(StorageService.java:1603)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:93)
at
com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:27)
at
com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:208)
at
com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:120)
at
com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:262)
at
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:836)
at
com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:761)
at
javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1427)
at
javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:72)
at
javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1265)
at
javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1360)
at
javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:788)
at sun.reflect.GeneratedMethodAccessor13.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:303)
at sun.rmi.transport.Transport$1.run(Transport.java:159)
at java.security.AccessController.doPrivileged(Native Method)
at sun.rmi.transport.Transport.serviceCall(Transport.java:155)
at
sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:535)
at
sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790)
at
sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.RuntimeException: java.io.IOException: Problem
during repair session manual-repair-43545b22-ffe8-4243-8a98-509b
bfec9872, endpoint /130.199.185.195 died
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at
java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
... 3 more
Caused by: java.io.IOException: Problem during repair session
manual-repair-43545b22-ffe8-4243-8a98-509bbfec9872, endpoint /130.199.
185.195 died
at
org.apache.cassandra.service.AntiEntropyService$RepairSession.failedNode(AntiEntropyService.java:723)
at
org.apache.cassandra.service.AntiEntropyService$RepairSession.convict(AntiEntropyService.java:760)
at
org.apache.cassandra.gms.FailureDetector.interpret(FailureDetector.java:165)
at
org.apache.cassandra.gms.Gossiper.doStatusCheck(Gossiper.java:538)
at org.apache.cassandra.gms.Gossiper.access$700(Gossiper.java:57)
at
org.apache.cassandra.gms.Gossiper$GossipTask.run(Gossiper.java:157)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)
... 3 more
INFO [FlushWriter:4] 2012-04-18 22:18:26,489 Memtable.java (line 237)
Writing Memtable-files.pandaid@988533693(1754325/140346000 se
rialized/live bytes, 70173 ops)
ERROR [AntiEntropySessions:8] 2012-04-18 22:18:26,488
AntiEntropyService.java (line 691) Repair session
manual-repair-1b3453b6-28b5-
4abd-84ce-0326b5468064 (on cfs [Ljava.lang.String;@5a44cc06, range
(113427455640312821154458202477256070484,0]) failed with the foll
owing error
java.io.IOException: Problem during repair session
manual-repair-1b3453b6-28b5-4abd-84ce-0326b5468064, endpoint
/130.199.185.193 die
d
at
org.apache.cassandra.service.AntiEntropyService$RepairSession.failedNode(AntiEntropyService.java:723)
at
org.apache.cassandra.service.AntiEntropyService$RepairSession.convict(AntiEntropyService.java:760)
at
org.apache.cassandra.gms.FailureDetector.interpret(FailureDetector.java:165)
at
org.apache.cassandra.gms.Gossiper.doStatusCheck(Gossiper.java:538)
at org.apache.cassandra.gms.Gossiper.access$700(Gossiper.java:57)
at
org.apache.cassandra.gms.Gossiper$GossipTask.run(Gossiper.java:157)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
INFO [ScheduledTasks:1] 2012-04-18 22:18:26,488 StatusLogger.java
(line 65) ReadStage 0 0 0
ERROR [RMI TCP Connection(22)-130.199.185.194] 2012-04-18 22:18:26,491
StorageService.java (line 1607) Repair session org.apache.cas
sandra.service.AntiEntropyService$RepairSession@41bb342c failed.
java.util.concurrent.ExecutionException: java.lang.RuntimeException:
java.io.IOException: Problem during repair session manual-repai
r-1b3453b6-28b5-4abd-84ce-0326b5468064, endpoint /130.199.185.193 died
at
java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
at java.util.concurrent.FutureTask.get(FutureTask.java:83)
at
org.apache.cassandra.service.StorageService.forceTableRepair(StorageService.java:1603)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:93)
at
com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:27)
at
com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:208)
at
com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:120)
at
com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:262)
at
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:836)
at
com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:761)
at
javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1427)
at
javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:72)
at
javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1265)
at
javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1360)
at
javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:788)
at sun.reflect.GeneratedMethodAccessor13.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:303)
at sun.rmi.transport.Transport$1.run(Transport.java:159)
at java.security.AccessController.doPrivileged(Native Method)
at sun.rmi.transport.Transport.serviceCall(Transport.java:155)
at
sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:535)
at
sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790)
at
sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.RuntimeException: java.io.IOException: Problem
during repair session manual-repair-1b3453b6-28b5-4abd-84ce-0326
b5468064, endpoint /130.199.185.193 died
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at
java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
... 3 more
Caused by: java.io.IOException: Problem during repair session
manual-repair-1b3453b6-28b5-4abd-84ce-0326b5468064, endpoint /130.199.
185.193 died
at
org.apache.cassandra.service.AntiEntropyService$RepairSession.failedNode(AntiEntropyService.java:723)
at
org.apache.cassandra.service.AntiEntropyService$RepairSession.convict(AntiEntropyService.java:760)
at
org.apache.cassandra.gms.FailureDetector.interpret(FailureDetector.java:165)
at
org.apache.cassandra.gms.Gossiper.doStatusCheck(Gossiper.java:538)
at org.apache.cassandra.gms.Gossiper.access$700(Gossiper.java:57)
at
org.apache.cassandra.gms.Gossiper$GossipTask.run(Gossiper.java:157)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
On 4/12/2012 10:03 PM, aaron morton wrote:
> Look at the server side logs for errors.
>
> Cheers
>
> -----------------
> Aaron Morton
> Freelance Developer
> @aaronmorton
> http://www.thelastpickle.com
>
> On 13/04/2012, at 11:47 AM, Maxim Potekhin wrote:
>
>> Hello,
>>
>> I'm doing compactions under 0.8.8.
>>
>> Recently, I started seeing a stack trace like one below, and I can't
>> figure out what causes this to appear.
>> The cluster has been in operation for mode than half a year w/o
>> errors like this one.
>>
>> Any help will be appreciated,
>> Thanks
>>
>> Maxim
>>
>>
>> WARNING: Failed to check the connection:
>> java.net.SocketTimeoutException: Read timed out
>> Exception in thread "main" java.io.IOException: Repair command #1:
>> some repair session(s) failed (see log for details).
>> at
>> org.apache.cassandra.service.StorageService.forceTableRepair(StorageService.java:1613)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> at java.lang.reflect.Method.invoke(Method.java:597)
>> at
>> com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:93)
>> at
>> com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:27)
>> at
>> com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:208)
>> at
>> com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:120)
>> at
>> com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:262)
>> at
>> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:836)
>> at
>> com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:761)
>> at
>> javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1427)
>> at
>> javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:72)
>> at
>> javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1265)
>> at
>> javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1360)
>> at
>> javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:788)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> at java.lang.reflect.Method.invoke(Method.java:597)
>> at
>> sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:303)
>> at sun.rmi.transport.Transport$1.run(Transport.java:159)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at sun.rmi.transport.Transport.serviceCall(Transport.java:155)
>> at
>> sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:535)
>> at
>> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790)
>> at
>> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>> at java.lang.Thread.run(Thread.java:662)
>>
>
Re: RMI/JMX errors, weird
Posted by aaron morton <aa...@thelastpickle.com>.
Look at the server side logs for errors.
Cheers
-----------------
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com
On 13/04/2012, at 11:47 AM, Maxim Potekhin wrote:
> Hello,
>
> I'm doing compactions under 0.8.8.
>
> Recently, I started seeing a stack trace like one below, and I can't figure out what causes this to appear.
> The cluster has been in operation for mode than half a year w/o errors like this one.
>
> Any help will be appreciated,
> Thanks
>
> Maxim
>
>
> WARNING: Failed to check the connection: java.net.SocketTimeoutException: Read timed out
> Exception in thread "main" java.io.IOException: Repair command #1: some repair session(s) failed (see log for details).
> at org.apache.cassandra.service.StorageService.forceTableRepair(StorageService.java:1613)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:93)
> at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:27)
> at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:208)
> at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:120)
> at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:262)
> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:836)
> at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:761)
> at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1427)
> at javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:72)
> at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1265)
> at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1360)
> at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:788)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:303)
> at sun.rmi.transport.Transport$1.run(Transport.java:159)
> at java.security.AccessController.doPrivileged(Native Method)
> at sun.rmi.transport.Transport.serviceCall(Transport.java:155)
> at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:535)
> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790)
> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
>
RMI/JMX errors, weird
Posted by Maxim Potekhin <po...@bnl.gov>.
Hello,
I'm doing compactions under 0.8.8.
Recently, I started seeing a stack trace like one below, and I can't
figure out what causes this to appear.
The cluster has been in operation for mode than half a year w/o errors
like this one.
Any help will be appreciated,
Thanks
Maxim
WARNING: Failed to check the connection:
java.net.SocketTimeoutException: Read timed out
Exception in thread "main" java.io.IOException: Repair command #1: some
repair session(s) failed (see log for details).
at
org.apache.cassandra.service.StorageService.forceTableRepair(StorageService.java:1613)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:93)
at
com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:27)
at
com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:208)
at
com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:120)
at
com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:262)
at
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:836)
at
com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:761)
at
javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1427)
at
javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:72)
at
javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1265)
at
javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1360)
at
javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:788)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:303)
at sun.rmi.transport.Transport$1.run(Transport.java:159)
at java.security.AccessController.doPrivileged(Native Method)
at sun.rmi.transport.Transport.serviceCall(Transport.java:155)
at
sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:535)
at
sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790)
at
sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Re: Why so many SSTables?
Posted by Thorsten von Eicken <tv...@rightscale.com>.
>From my experience I would strongly advise against leveled compaction
for your use-case. But you should certainly test and see for yourself!
I have ~1TB on a node with ~13GB of heap. I ended up with 30k SSTables.
I raised the SSTable size to 100MB but that didn't prove to be
sufficient and I did it too late. I also had issues along the way where
I ran out of heap space (before going to 13GB) and the flushing that
then happens caused lots of small SSTables to be produced. Loading up
30k SSTables at start-up took 30+ minutes. With 30k SSTables I had 5
levels, if I remember correctly, that means everything goes through 5
compactions in the long run and the resulting cpu load is just
"phenomenal". I was using snappy compressor, maybe that exacerbated the
problem. At some point I overloaded thing with writes and had a
compaction backlog of over 3000. I stopped all data loading and after
several days it was still compacting away and not about to end. I
eventually switched to size tiered compaction and life is back to normal...
Of course, leveled compaction may work wonderfully for others, this is
just what I experienced. If I were to try it again, I'd watch my SSTable
count and abort the experiment if I end up with >10k and I'd watch my
compaction backlog and abort if it goes above 100.
- Thorsten
On 4/12/2012 12:54 AM, Romain HARDOUIN wrote:
>
> I've just opened a new JIRA: CASSANDRA-4142
>
> I've double checked numbers, 7747 seems to be array list object's
> capacity (Eclipse Memory Analyzer displays "java.lang.Object[7747] @
> 0x7d3f3f798").
> Actually there are 5757 browsable entries in EMA therefore each object
> is about 140 KB (size varies between 143088 bits and 143768 bits).
>
> We have no pending compactions tasks. Our cluster is currently
> under-loaded.
>
> Our goal is to handle hundreds of tera bytes which explains 1 TB per
> node. Our need is to archive data so our cluster is not running under
> a read-heavy load.
>
> @Dave Brosius: I made a mistake. To be correct 786 MB is 47% of *leak
> suspects* as reported by Eclipse Memory Analyser.
> Our Cassandra nodes are pretty standard: 10 GB of ram with Xmx set to
> 2,5 GB.
>
> Regards,
>
> Romain
Re: Why so many SSTables?
Posted by Romain HARDOUIN <ro...@urssaf.fr>.
I've just opened a new JIRA: CASSANDRA-4142
I've double checked numbers, 7747 seems to be array list object's capacity
(Eclipse Memory Analyzer displays "java.lang.Object[7747] @ 0x7d3f3f798").
Actually there are 5757 browsable entries in EMA therefore each object is
about 140 KB (size varies between 143088 bits and 143768 bits).
We have no pending compactions tasks. Our cluster is currently
under-loaded.
Our goal is to handle hundreds of tera bytes which explains 1 TB per node.
Our need is to archive data so our cluster is not running under a
read-heavy load.
@Dave Brosius: I made a mistake. To be correct 786 MB is 47% of *leak
suspects* as reported by Eclipse Memory Analyser.
Our Cassandra nodes are pretty standard: 10 GB of ram with Xmx set to 2,5
GB.
Regards,
Romain
Re: Why so many SSTables?
Posted by Watanabe Maki <wa...@gmail.com>.
If you increase sstable_size_in_mb to 200MB, you will need more IO for each compaction. For example, if your memtable will be flushed, and LCS needs to compact it with 10 overwrapped L1 sstables, you will need almost 2GB read and 2GB write for the single compaction.
From iPhone
On 2012/04/11, at 21:43, Romain HARDOUIN <ro...@urssaf.fr> wrote:
>
> Thank you for your answers.
>
> I originally post this question because we encoutered an OOM Exception on 2 nodes during repair session.
> Memory analyzing shows an hotspot: an ArrayList of SSTableBoundedScanner which contains as many objects there are SSTables on disk (7747 objects at the time).
> This ArrayList consumes 47% of the heap space (786 MB).
>
> We want each node to handle 1 TB, so we must dramatically reduce the number of SSTables.
>
> Thus, is there any drawback if we set sstable_size_in_mb to 200MB?
> Otherwise shoudl we go back to Tiered Compaction?
>
> Regards,
>
> Romain
>
>
> Maki Watanabe <wa...@gmail.com> a écrit sur 11/04/2012 04:21:47 :
>
> > You can configure sstable size by sstable_size_in_mb parameter for LCS.
> > The default value is 5MB.
> > You should better to check you don't have many pending compaction tasks
> > with nodetool tpstats and compactionstats also.
> > If you have enough IO throughput, you can increase
> > compaction_throughput_mb_per_sec
> > in cassandra.yaml to reduce pending compactions.
> >
> > maki
> >
> > 2012/4/10 Romain HARDOUIN <ro...@urssaf.fr>:
> > >
> > > Hi,
> > >
> > > We are surprised by the number of files generated by Cassandra.
> > > Our cluster consists of 9 nodes and each node handles about 35 GB.
> > > We're using Cassandra 1.0.6 with LeveledCompactionStrategy.
> > > We have 30 CF.
> > >
> > > We've got roughly 45,000 files under the keyspace directory on each node:
> > > ls -l /var/lib/cassandra/data/OurKeyspace/ | wc -l
> > > 44372
> > >
> > > The biggest CF is spread over 38,000 files:
> > > ls -l Documents* | wc -l
> > > 37870
> > >
> > > ls -l Documents*-Data.db | wc -l
> > > 7586
> > >
> > > Many SSTable are about 4 MB:
> > >
> > > 19 MB -> 1 SSTable
> > > 12 MB -> 2 SSTables
> > > 11 MB -> 2 SSTables
> > > 9.2 MB -> 1 SSTable
> > > 7.0 MB to 7.9 MB -> 6 SSTables
> > > 6.0 MB to 6.4 MB -> 6 SSTables
> > > 5.0 MB to 5.4 MB -> 4 SSTables
> > > 4.0 MB to 4.7 MB -> 7139 SSTables
> > > 3.0 MB to 3.9 MB -> 258 SSTables
> > > 2.0 MB to 2.9 MB -> 35 SSTables
> > > 1.0 MB to 1.9 MB -> 13 SSTables
> > > 87 KB to 994 KB -> 87 SSTables
> > > 0 KB -> 32 SSTables
> > >
> > > FYI here is CF information:
> > >
> > > ColumnFamily: Documents
> > > Key Validation Class: org.apache.cassandra.db.marshal.BytesType
> > > Default column value validator: org.apache.cassandra.db.marshal.BytesType
> > > Columns sorted by: org.apache.cassandra.db.marshal.BytesType
> > > Row cache size / save period in seconds / keys to save : 0.0/0/all
> > > Row Cache Provider: org.apache.cassandra.cache.SerializingCacheProvider
> > > Key cache size / save period in seconds: 200000.0/14400
> > > GC grace seconds: 1728000
> > > Compaction min/max thresholds: 4/32
> > > Read repair chance: 1.0
> > > Replicate on write: true
> > > Column Metadata:
> > > Column Name: refUUID (72656655554944)
> > > Validation Class: org.apache.cassandra.db.marshal.BytesType
> > > Index Name: refUUID_idx
> > > Index Type: KEYS
> > > Compaction Strategy:
> > > org.apache.cassandra.db.compaction.LeveledCompactionStrategy
> > > Compression Options:
> > > sstable_compression: org.apache.cassandra.io.compress.SnappyCompressor
> > >
> > > Is it a bug? If not, how can we tune Cassandra to avoid this?
> > >
> > > Regards,
> > >
> > > Romain
Re: Why so many SSTables?
Posted by Ben Coverston <be...@datastax.com>.
>In general I would limit the data load per node to 300 to 400GB. Otherwise
> things can painful when it comes time to run compaction / repair / move .
+1 on more nodes of moderate size
Re: Why so many SSTables?
Posted by aaron morton <aa...@thelastpickle.com>.
In general I would limit the data load per node to 300 to 400GB. Otherwise things can painful when it comes time to run compaction / repair / move .
Cheers
-----------------
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com
On 12/04/2012, at 1:00 AM, Dave Brosius wrote:
> It's easy to spend other people's money, but handling 1TB of data with 1.5 g heap? Memory is cheap, and just a little more will solve many problems.
>
>
> On 04/11/2012 08:43 AM, Romain HARDOUIN wrote:
>>
>>
>> Thank you for your answers.
>>
>> I originally post this question because we encoutered an OOM Exception on 2 nodes during repair session.
>> Memory analyzing shows an hotspot: an ArrayList of SSTableBoundedScanner which contains as many objects there are SSTables on disk (7747 objects at the time).
>> This ArrayList consumes 47% of the heap space (786 MB).
>>
>> We want each node to handle 1 TB, so we must dramatically reduce the number of SSTables.
>>
>> Thus, is there any drawback if we set sstable_size_in_mb to 200MB?
>> Otherwise shoudl we go back to Tiered Compaction?
>>
>> Regards,
>>
>> Romain
>>
>>
>> Maki Watanabe <wa...@gmail.com> a écrit sur 11/04/2012 04:21:47 :
>>
>> > You can configure sstable size by sstable_size_in_mb parameter for LCS.
>> > The default value is 5MB.
>> > You should better to check you don't have many pending compaction tasks
>> > with nodetool tpstats and compactionstats also.
>> > If you have enough IO throughput, you can increase
>> > compaction_throughput_mb_per_sec
>> > in cassandra.yaml to reduce pending compactions.
>> >
>> > maki
>> >
>> > 2012/4/10 Romain HARDOUIN <ro...@urssaf.fr>:
>> > >
>> > > Hi,
>> > >
>> > > We are surprised by the number of files generated by Cassandra.
>> > > Our cluster consists of 9 nodes and each node handles about 35 GB.
>> > > We're using Cassandra 1.0.6 with LeveledCompactionStrategy.
>> > > We have 30 CF.
>> > >
>> > > We've got roughly 45,000 files under the keyspace directory on each node:
>> > > ls -l /var/lib/cassandra/data/OurKeyspace/ | wc -l
>> > > 44372
>> > >
>> > > The biggest CF is spread over 38,000 files:
>> > > ls -l Documents* | wc -l
>> > > 37870
>> > >
>> > > ls -l Documents*-Data.db | wc -l
>> > > 7586
>> > >
>> > > Many SSTable are about 4 MB:
>> > >
>> > > 19 MB -> 1 SSTable
>> > > 12 MB -> 2 SSTables
>> > > 11 MB -> 2 SSTables
>> > > 9.2 MB -> 1 SSTable
>> > > 7.0 MB to 7.9 MB -> 6 SSTables
>> > > 6.0 MB to 6.4 MB -> 6 SSTables
>> > > 5.0 MB to 5.4 MB -> 4 SSTables
>> > > 4.0 MB to 4.7 MB -> 7139 SSTables
>> > > 3.0 MB to 3.9 MB -> 258 SSTables
>> > > 2.0 MB to 2.9 MB -> 35 SSTables
>> > > 1.0 MB to 1.9 MB -> 13 SSTables
>> > > 87 KB to 994 KB -> 87 SSTables
>> > > 0 KB -> 32 SSTables
>> > >
>> > > FYI here is CF information:
>> > >
>> > > ColumnFamily: Documents
>> > > Key Validation Class: org.apache.cassandra.db.marshal.BytesType
>> > > Default column value validator: org.apache.cassandra.db.marshal.BytesType
>> > > Columns sorted by: org.apache.cassandra.db.marshal.BytesType
>> > > Row cache size / save period in seconds / keys to save : 0.0/0/all
>> > > Row Cache Provider: org.apache.cassandra.cache.SerializingCacheProvider
>> > > Key cache size / save period in seconds: 200000.0/14400
>> > > GC grace seconds: 1728000
>> > > Compaction min/max thresholds: 4/32
>> > > Read repair chance: 1.0
>> > > Replicate on write: true
>> > > Column Metadata:
>> > > Column Name: refUUID (72656655554944)
>> > > Validation Class: org.apache.cassandra.db.marshal.BytesType
>> > > Index Name: refUUID_idx
>> > > Index Type: KEYS
>> > > Compaction Strategy:
>> > > org.apache.cassandra.db.compaction.LeveledCompactionStrategy
>> > > Compression Options:
>> > > sstable_compression: org.apache.cassandra.io.compress.SnappyCompressor
>> > >
>> > > Is it a bug? If not, how can we tune Cassandra to avoid this?
>> > >
>> > > Regards,
>> > >
>> > > Romain
>
Re: Why so many SSTables?
Posted by Dave Brosius <db...@mebigfatguy.com>.
It's easy to spend other people's money, but handling 1TB of data with
1.5 g heap? Memory is cheap, and just a little more will solve many
problems.
On 04/11/2012 08:43 AM, Romain HARDOUIN wrote:
>
> Thank you for your answers.
>
> I originally post this question because we encoutered an OOM Exception
> on 2 nodes during repair session.
> Memory analyzing shows an hotspot: an ArrayList of
> SSTableBoundedScanner which contains as many objects there are
> SSTables on disk (7747 objects at the time).
> This ArrayList consumes 47% of the heap space (786 MB).
>
> We want each node to handle 1 TB, so we must dramatically reduce the
> number of SSTables.
>
> Thus, is there any drawback if we set sstable_size_in_mb to 200MB?
> Otherwise shoudl we go back to Tiered Compaction?
>
> Regards,
>
> Romain
>
>
> Maki Watanabe <wa...@gmail.com> a écrit sur 11/04/2012 04:21:47 :
>
> > You can configure sstable size by sstable_size_in_mb parameter for LCS.
> > The default value is 5MB.
> > You should better to check you don't have many pending compaction tasks
> > with nodetool tpstats and compactionstats also.
> > If you have enough IO throughput, you can increase
> > compaction_throughput_mb_per_sec
> > in cassandra.yaml to reduce pending compactions.
> >
> > maki
> >
> > 2012/4/10 Romain HARDOUIN <ro...@urssaf.fr>:
> > >
> > > Hi,
> > >
> > > We are surprised by the number of files generated by Cassandra.
> > > Our cluster consists of 9 nodes and each node handles about 35 GB.
> > > We're using Cassandra 1.0.6 with LeveledCompactionStrategy.
> > > We have 30 CF.
> > >
> > > We've got roughly 45,000 files under the keyspace directory on
> each node:
> > > ls -l /var/lib/cassandra/data/OurKeyspace/ | wc -l
> > > 44372
> > >
> > > The biggest CF is spread over 38,000 files:
> > > ls -l Documents* | wc -l
> > > 37870
> > >
> > > ls -l Documents*-Data.db | wc -l
> > > 7586
> > >
> > > Many SSTable are about 4 MB:
> > >
> > > 19 MB -> 1 SSTable
> > > 12 MB -> 2 SSTables
> > > 11 MB -> 2 SSTables
> > > 9.2 MB -> 1 SSTable
> > > 7.0 MB to 7.9 MB -> 6 SSTables
> > > 6.0 MB to 6.4 MB -> 6 SSTables
> > > 5.0 MB to 5.4 MB -> 4 SSTables
> > > 4.0 MB to 4.7 MB -> 7139 SSTables
> > > 3.0 MB to 3.9 MB -> 258 SSTables
> > > 2.0 MB to 2.9 MB -> 35 SSTables
> > > 1.0 MB to 1.9 MB -> 13 SSTables
> > > 87 KB to 994 KB -> 87 SSTables
> > > 0 KB -> 32 SSTables
> > >
> > > FYI here is CF information:
> > >
> > > ColumnFamily: Documents
> > > Key Validation Class: org.apache.cassandra.db.marshal.BytesType
> > > Default column value validator:
> org.apache.cassandra.db.marshal.BytesType
> > > Columns sorted by: org.apache.cassandra.db.marshal.BytesType
> > > Row cache size / save period in seconds / keys to save : 0.0/0/all
> > > Row Cache Provider:
> org.apache.cassandra.cache.SerializingCacheProvider
> > > Key cache size / save period in seconds: 200000.0/14400
> > > GC grace seconds: 1728000
> > > Compaction min/max thresholds: 4/32
> > > Read repair chance: 1.0
> > > Replicate on write: true
> > > Column Metadata:
> > > Column Name: refUUID (72656655554944)
> > > Validation Class: org.apache.cassandra.db.marshal.BytesType
> > > Index Name: refUUID_idx
> > > Index Type: KEYS
> > > Compaction Strategy:
> > > org.apache.cassandra.db.compaction.LeveledCompactionStrategy
> > > Compression Options:
> > > sstable_compression:
> org.apache.cassandra.io.compress.SnappyCompressor
> > >
> > > Is it a bug? If not, how can we tune Cassandra to avoid this?
> > >
> > > Regards,
> > >
> > > Romain
Re: Why so many SSTables?
Posted by Romain HARDOUIN <ro...@urssaf.fr>.
Thank you for your answers.
I originally post this question because we encoutered an OOM Exception on
2 nodes during repair session.
Memory analyzing shows an hotspot: an ArrayList of SSTableBoundedScanner
which contains as many objects there are SSTables on disk (7747 objects at
the time).
This ArrayList consumes 47% of the heap space (786 MB).
We want each node to handle 1 TB, so we must dramatically reduce the
number of SSTables.
Thus, is there any drawback if we set sstable_size_in_mb to 200MB?
Otherwise shoudl we go back to Tiered Compaction?
Regards,
Romain
Maki Watanabe <wa...@gmail.com> a écrit sur 11/04/2012 04:21:47 :
> You can configure sstable size by sstable_size_in_mb parameter for LCS.
> The default value is 5MB.
> You should better to check you don't have many pending compaction tasks
> with nodetool tpstats and compactionstats also.
> If you have enough IO throughput, you can increase
> compaction_throughput_mb_per_sec
> in cassandra.yaml to reduce pending compactions.
>
> maki
>
> 2012/4/10 Romain HARDOUIN <ro...@urssaf.fr>:
> >
> > Hi,
> >
> > We are surprised by the number of files generated by Cassandra.
> > Our cluster consists of 9 nodes and each node handles about 35 GB.
> > We're using Cassandra 1.0.6 with LeveledCompactionStrategy.
> > We have 30 CF.
> >
> > We've got roughly 45,000 files under the keyspace directory on each
node:
> > ls -l /var/lib/cassandra/data/OurKeyspace/ | wc -l
> > 44372
> >
> > The biggest CF is spread over 38,000 files:
> > ls -l Documents* | wc -l
> > 37870
> >
> > ls -l Documents*-Data.db | wc -l
> > 7586
> >
> > Many SSTable are about 4 MB:
> >
> > 19 MB -> 1 SSTable
> > 12 MB -> 2 SSTables
> > 11 MB -> 2 SSTables
> > 9.2 MB -> 1 SSTable
> > 7.0 MB to 7.9 MB -> 6 SSTables
> > 6.0 MB to 6.4 MB -> 6 SSTables
> > 5.0 MB to 5.4 MB -> 4 SSTables
> > 4.0 MB to 4.7 MB -> 7139 SSTables
> > 3.0 MB to 3.9 MB -> 258 SSTables
> > 2.0 MB to 2.9 MB -> 35 SSTables
> > 1.0 MB to 1.9 MB -> 13 SSTables
> > 87 KB to 994 KB -> 87 SSTables
> > 0 KB -> 32 SSTables
> >
> > FYI here is CF information:
> >
> > ColumnFamily: Documents
> > Key Validation Class: org.apache.cassandra.db.marshal.BytesType
> > Default column value validator:
org.apache.cassandra.db.marshal.BytesType
> > Columns sorted by: org.apache.cassandra.db.marshal.BytesType
> > Row cache size / save period in seconds / keys to save : 0.0/0/all
> > Row Cache Provider:
org.apache.cassandra.cache.SerializingCacheProvider
> > Key cache size / save period in seconds: 200000.0/14400
> > GC grace seconds: 1728000
> > Compaction min/max thresholds: 4/32
> > Read repair chance: 1.0
> > Replicate on write: true
> > Column Metadata:
> > Column Name: refUUID (72656655554944)
> > Validation Class: org.apache.cassandra.db.marshal.BytesType
> > Index Name: refUUID_idx
> > Index Type: KEYS
> > Compaction Strategy:
> > org.apache.cassandra.db.compaction.LeveledCompactionStrategy
> > Compression Options:
> > sstable_compression:
org.apache.cassandra.io.compress.SnappyCompressor
> >
> > Is it a bug? If not, how can we tune Cassandra to avoid this?
> >
> > Regards,
> >
> > Romain
Re: Why so many SSTables?
Posted by Maki Watanabe <wa...@gmail.com>.
You can configure sstable size by sstable_size_in_mb parameter for LCS.
The default value is 5MB.
You should better to check you don't have many pending compaction tasks
with nodetool tpstats and compactionstats also.
If you have enough IO throughput, you can increase
compaction_throughput_mb_per_sec
in cassandra.yaml to reduce pending compactions.
maki
2012/4/10 Romain HARDOUIN <ro...@urssaf.fr>:
>
> Hi,
>
> We are surprised by the number of files generated by Cassandra.
> Our cluster consists of 9 nodes and each node handles about 35 GB.
> We're using Cassandra 1.0.6 with LeveledCompactionStrategy.
> We have 30 CF.
>
> We've got roughly 45,000 files under the keyspace directory on each node:
> ls -l /var/lib/cassandra/data/OurKeyspace/ | wc -l
> 44372
>
> The biggest CF is spread over 38,000 files:
> ls -l Documents* | wc -l
> 37870
>
> ls -l Documents*-Data.db | wc -l
> 7586
>
> Many SSTable are about 4 MB:
>
> 19 MB -> 1 SSTable
> 12 MB -> 2 SSTables
> 11 MB -> 2 SSTables
> 9.2 MB -> 1 SSTable
> 7.0 MB to 7.9 MB -> 6 SSTables
> 6.0 MB to 6.4 MB -> 6 SSTables
> 5.0 MB to 5.4 MB -> 4 SSTables
> 4.0 MB to 4.7 MB -> 7139 SSTables
> 3.0 MB to 3.9 MB -> 258 SSTables
> 2.0 MB to 2.9 MB -> 35 SSTables
> 1.0 MB to 1.9 MB -> 13 SSTables
> 87 KB to 994 KB -> 87 SSTables
> 0 KB -> 32 SSTables
>
> FYI here is CF information:
>
> ColumnFamily: Documents
> Key Validation Class: org.apache.cassandra.db.marshal.BytesType
> Default column value validator: org.apache.cassandra.db.marshal.BytesType
> Columns sorted by: org.apache.cassandra.db.marshal.BytesType
> Row cache size / save period in seconds / keys to save : 0.0/0/all
> Row Cache Provider: org.apache.cassandra.cache.SerializingCacheProvider
> Key cache size / save period in seconds: 200000.0/14400
> GC grace seconds: 1728000
> Compaction min/max thresholds: 4/32
> Read repair chance: 1.0
> Replicate on write: true
> Column Metadata:
> Column Name: refUUID (72656655554944)
> Validation Class: org.apache.cassandra.db.marshal.BytesType
> Index Name: refUUID_idx
> Index Type: KEYS
> Compaction Strategy:
> org.apache.cassandra.db.compaction.LeveledCompactionStrategy
> Compression Options:
> sstable_compression: org.apache.cassandra.io.compress.SnappyCompressor
>
> Is it a bug? If not, how can we tune Cassandra to avoid this?
>
> Regards,
>
> Romain
Re: Why so many SSTables?
Posted by Jonathan Ellis <jb...@gmail.com>.
LCS explicitly tries to keep sstables under 5MB to minimize extra work
done by compacting data that didn't really overlap across different
levels.
On Tue, Apr 10, 2012 at 9:24 AM, Romain HARDOUIN
<ro...@urssaf.fr> wrote:
>
> Hi,
>
> We are surprised by the number of files generated by Cassandra.
> Our cluster consists of 9 nodes and each node handles about 35 GB.
> We're using Cassandra 1.0.6 with LeveledCompactionStrategy.
> We have 30 CF.
>
> We've got roughly 45,000 files under the keyspace directory on each node:
> ls -l /var/lib/cassandra/data/OurKeyspace/ | wc -l
> 44372
>
> The biggest CF is spread over 38,000 files:
> ls -l Documents* | wc -l
> 37870
>
> ls -l Documents*-Data.db | wc -l
> 7586
>
> Many SSTable are about 4 MB:
>
> 19 MB -> 1 SSTable
> 12 MB -> 2 SSTables
> 11 MB -> 2 SSTables
> 9.2 MB -> 1 SSTable
> 7.0 MB to 7.9 MB -> 6 SSTables
> 6.0 MB to 6.4 MB -> 6 SSTables
> 5.0 MB to 5.4 MB -> 4 SSTables
> 4.0 MB to 4.7 MB -> 7139 SSTables
> 3.0 MB to 3.9 MB -> 258 SSTables
> 2.0 MB to 2.9 MB -> 35 SSTables
> 1.0 MB to 1.9 MB -> 13 SSTables
> 87 KB to 994 KB -> 87 SSTables
> 0 KB -> 32 SSTables
>
> FYI here is CF information:
>
> ColumnFamily: Documents
> Key Validation Class: org.apache.cassandra.db.marshal.BytesType
> Default column value validator: org.apache.cassandra.db.marshal.BytesType
> Columns sorted by: org.apache.cassandra.db.marshal.BytesType
> Row cache size / save period in seconds / keys to save : 0.0/0/all
> Row Cache Provider: org.apache.cassandra.cache.SerializingCacheProvider
> Key cache size / save period in seconds: 200000.0/14400
> GC grace seconds: 1728000
> Compaction min/max thresholds: 4/32
> Read repair chance: 1.0
> Replicate on write: true
> Column Metadata:
> Column Name: refUUID (72656655554944)
> Validation Class: org.apache.cassandra.db.marshal.BytesType
> Index Name: refUUID_idx
> Index Type: KEYS
> Compaction Strategy:
> org.apache.cassandra.db.compaction.LeveledCompactionStrategy
> Compression Options:
> sstable_compression: org.apache.cassandra.io.compress.SnappyCompressor
>
> Is it a bug? If not, how can we tune Cassandra to avoid this?
>
> Regards,
>
> Romain
--
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of DataStax, the source for professional Cassandra support
http://www.datastax.com