You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Nish garg <pi...@gmail.com> on 2014/02/27 20:09:18 UTC

OOM while performing major compaction

I am having OOM during major compaction on one of the column family where
there are lot of SStables (33000) to be compacted. Is there any other way
for them to be compacted? Any help will be really appreciated.

Here are the details

 /opt/cassandra/current/bin/nodetool -h us1emscsm-01  compact tomcat
sessions
Error occurred during compaction
java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java
heap space
 at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
 at java.util.concurrent.FutureTask.get(FutureTask.java:83)
 at
org.apache.cassandra.db.compaction.CompactionManager.performMaximal(CompactionManager.java:334)
 at
org.apache.cassandra.db.ColumnFamilyStore.forceMajorCompaction(ColumnFamilyStore.java:1691)
 at
org.apache.cassandra.service.StorageService.forceTableCompaction(StorageService.java:2168)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
 at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
 at java.lang.reflect.Method.invoke(Method.java:597)
 at
com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:93)
 at
com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:27)
 at
com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:208)
 at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:120)
 at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:262)
 at
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:836)
 at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:761)
 at
javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1427)
 at
javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:72)
 at
javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1265)
 at
javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1360)
 at
javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:788)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
 at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
 at java.lang.reflect.Method.invoke(Method.java:597)
 at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:305)
 at sun.rmi.transport.Transport$1.run(Transport.java:159)
 at java.security.AccessController.doPrivileged(Native Method)
 at sun.rmi.transport.Transport.serviceCall(Transport.java:155)
 at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:535)
 at
sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790)
 at
sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649)
 at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
 at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
 at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.OutOfMemoryError: Java heap space
 at
org.apache.cassandra.io.util.RandomAccessReader.<init>(RandomAccessReader.java:77)
 at
org.apache.cassandra.io.compress.CompressedRandomAccessReader.<init>(CompressedRandomAccessReader.java:75)
 at
org.apache.cassandra.io.compress.CompressedThrottledReader.<init>(CompressedThrottledReader.java:38)
 at
org.apache.cassandra.io.compress.CompressedThrottledReader.open(CompressedThrottledReader.java:52)
 at
org.apache.cassandra.io.sstable.SSTableReader.openDataReader(SSTableReader.java:1212)
 at
org.apache.cassandra.io.sstable.SSTableScanner.<init>(SSTableScanner.java:54)
 at
org.apache.cassandra.io.sstable.SSTableReader.getDirectScanner(SSTableReader.java:1032)
 at
org.apache.cassandra.io.sstable.SSTableReader.getDirectScanner(SSTableReader.java:1044)
 at
org.apache.cassandra.db.compaction.AbstractCompactionStrategy.getScanners(AbstractCompactionStrategy.java:157)
 at
org.apache.cassandra.db.compaction.AbstractCompactionStrategy.getScanners(AbstractCompactionStrategy.java:163)
 at
org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:117)
 at
org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48)
 at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
 at
org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:58)
 at
org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:60)
 at
org.apache.cassandra.db.compaction.CompactionManager$6.runMayThrow(CompactionManager.java:355)
 at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
 at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
 at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
 at java.util.concurrent.FutureTask.run(FutureTask.java:138)
 ... 3 more


Also


/opt/cassandra/current/bin/nodetool -h us1emscsm-01  getcompactionthreshold
tomcat sessions
Current compaction thresholds for tomcat/sessions:
 min = 4,  max = 32


 /opt/cassandra/current/bin/nodetool -h us1emscsm-01
getcompactionthroughput tomcat sessions
Current compaction throughput: 16 MB/s



/opt/cassandra/current/bin/nodetool -h us1emscsm-01  compactionstats
pending tasks: 167
Active compaction remaining time :        n/a

Re: OOM while performing major compaction

Posted by Tupshin Harper <tu...@tupshin.com>.
You are right that modifying your code to access two CFs is a hack, and not
an ideal solution, but I think it should be pretty easy to implement, and
would help you get out of this jam pretty quickly. Not saying you should go
down that path, but if you lack better options, that would probably be my
choice.

-Tupshin


On Thu, Feb 27, 2014 at 6:03 PM, Nish garg <pi...@gmail.com> wrote:

> Hello Tupshin,
>
> Yes all the data needs to be kept for just last 6 hours. Yes changing to
> new CF every 6 hours solves the compaction issue, but between the change we
> will have less than 6 hours of data. We can use CF1 and CF2 and truncate
> them one at a time every 6 hours in loop but we need some kind of view that
> does (CF1 union CF2) to get final data. Unfortunately views are not
> supported in Cassandra. May be we can change our code to see 2 CFs all the
> time...it is kind of hack and does not seem to be perfect solution.
>
>
> On Thu, Feb 27, 2014 at 4:49 PM, Tupshin Harper <tu...@tupshin.com>wrote:
>
>> If you can programmatically roll over onto a new column family every 6
>> hours (or every day or other reasonable increment), and then just drop your
>> existing column family after all the columns would have been expired, you
>> could skip your compaction entirely. It was not clear to me from your
>> description whether *all* of the data only needs to be retained for 6
>> hours. If that is true, rolling over to a new cf will be your simplest
>> option.
>>
>> -Tupshin
>>
>>
>> On Thu, Feb 27, 2014 at 5:31 PM, Nish garg <pi...@gmail.com> wrote:
>>
>>> Thanks for replying.
>>>
>>> We are  on Cassandra 1.2.9.
>>>
>>> We have time series like data structure where we need to keep only last
>>> 6 hours of data. So we expire data using  expireddatetime column on column
>>> family and then we run expire script via cron to create tombstones. We
>>> don't use ttl yet and planning to use it in our future release. Hope that
>>> will fix some of the issues caused by expire script as it needs to read the
>>> data first before creating tombstones.
>>>
>>> So to answer your question, we have almost 80% of tombstones in those
>>> sstables. (There is no easy way to confirm this unless I convert all those
>>> 33000 sstables to JSON file and query them for tombstones).
>>> The reason of 33000 of them may be due to machine load too high for
>>> minor compaction and it was falling behind or some thing happened to minor
>>> compaction thread on this node. Other two nodes in this cluster are fine.
>>> Yes, we are using sized compaction strategy.
>>>
>>> I am inclined towards 'decommission and bootstrap' this node as it seems
>>> like performing major compaction on this node is impossible.
>>>
>>> However still looking for other solutions...
>>>
>>>
>>>
>>>
>>> On Thu, Feb 27, 2014 at 4:03 PM, Robert Coli <rc...@eventbrite.com>wrote:
>>>
>>>> On Thu, Feb 27, 2014 at 11:09 AM, Nish garg <pi...@gmail.com>wrote:
>>>>
>>>>> I am having OOM during major compaction on one of the column family
>>>>> where there are lot of SStables (33000) to be compacted. Is there any other
>>>>> way for them to be compacted? Any help will be really appreciated.
>>>>>
>>>>
>>>> You can use user defined compaction to reduce the working set, but only
>>>> a major compaction is capable of purging 100% of tombstones.
>>>>
>>>> How much garbage is actually in the files? Why do you have 33,000 of
>>>> them? You mention a major compaction so you are likely not using LCS with
>>>> the bad 5mb default... how did you end up with so many SSTables?
>>>>
>>>> Have you removed the throttle from compaction, generally?
>>>>
>>>> What version of Cassandra?
>>>>
>>>> =Rob
>>>>
>>>>
>>>
>>
>

Re: OOM while performing major compaction

Posted by Nish garg <pi...@gmail.com>.
Hello Tupshin,

Yes all the data needs to be kept for just last 6 hours. Yes changing to
new CF every 6 hours solves the compaction issue, but between the change we
will have less than 6 hours of data. We can use CF1 and CF2 and truncate
them one at a time every 6 hours in loop but we need some kind of view that
does (CF1 union CF2) to get final data. Unfortunately views are not
supported in Cassandra. May be we can change our code to see 2 CFs all the
time...it is kind of hack and does not seem to be perfect solution.


On Thu, Feb 27, 2014 at 4:49 PM, Tupshin Harper <tu...@tupshin.com> wrote:

> If you can programmatically roll over onto a new column family every 6
> hours (or every day or other reasonable increment), and then just drop your
> existing column family after all the columns would have been expired, you
> could skip your compaction entirely. It was not clear to me from your
> description whether *all* of the data only needs to be retained for 6
> hours. If that is true, rolling over to a new cf will be your simplest
> option.
>
> -Tupshin
>
>
> On Thu, Feb 27, 2014 at 5:31 PM, Nish garg <pi...@gmail.com> wrote:
>
>> Thanks for replying.
>>
>> We are  on Cassandra 1.2.9.
>>
>> We have time series like data structure where we need to keep only last 6
>> hours of data. So we expire data using  expireddatetime column on column
>> family and then we run expire script via cron to create tombstones. We
>> don't use ttl yet and planning to use it in our future release. Hope that
>> will fix some of the issues caused by expire script as it needs to read the
>> data first before creating tombstones.
>>
>> So to answer your question, we have almost 80% of tombstones in those
>> sstables. (There is no easy way to confirm this unless I convert all those
>> 33000 sstables to JSON file and query them for tombstones).
>> The reason of 33000 of them may be due to machine load too high for minor
>> compaction and it was falling behind or some thing happened to minor
>> compaction thread on this node. Other two nodes in this cluster are fine.
>> Yes, we are using sized compaction strategy.
>>
>> I am inclined towards 'decommission and bootstrap' this node as it seems
>> like performing major compaction on this node is impossible.
>>
>> However still looking for other solutions...
>>
>>
>>
>>
>> On Thu, Feb 27, 2014 at 4:03 PM, Robert Coli <rc...@eventbrite.com>wrote:
>>
>>> On Thu, Feb 27, 2014 at 11:09 AM, Nish garg <pi...@gmail.com> wrote:
>>>
>>>> I am having OOM during major compaction on one of the column family
>>>> where there are lot of SStables (33000) to be compacted. Is there any other
>>>> way for them to be compacted? Any help will be really appreciated.
>>>>
>>>
>>> You can use user defined compaction to reduce the working set, but only
>>> a major compaction is capable of purging 100% of tombstones.
>>>
>>> How much garbage is actually in the files? Why do you have 33,000 of
>>> them? You mention a major compaction so you are likely not using LCS with
>>> the bad 5mb default... how did you end up with so many SSTables?
>>>
>>> Have you removed the throttle from compaction, generally?
>>>
>>> What version of Cassandra?
>>>
>>> =Rob
>>>
>>>
>>
>

Re: OOM while performing major compaction

Posted by Tupshin Harper <tu...@tupshin.com>.
If you can programmatically roll over onto a new column family every 6
hours (or every day or other reasonable increment), and then just drop your
existing column family after all the columns would have been expired, you
could skip your compaction entirely. It was not clear to me from your
description whether *all* of the data only needs to be retained for 6
hours. If that is true, rolling over to a new cf will be your simplest
option.

-Tupshin


On Thu, Feb 27, 2014 at 5:31 PM, Nish garg <pi...@gmail.com> wrote:

> Thanks for replying.
>
> We are  on Cassandra 1.2.9.
>
> We have time series like data structure where we need to keep only last 6
> hours of data. So we expire data using  expireddatetime column on column
> family and then we run expire script via cron to create tombstones. We
> don't use ttl yet and planning to use it in our future release. Hope that
> will fix some of the issues caused by expire script as it needs to read the
> data first before creating tombstones.
>
> So to answer your question, we have almost 80% of tombstones in those
> sstables. (There is no easy way to confirm this unless I convert all those
> 33000 sstables to JSON file and query them for tombstones).
> The reason of 33000 of them may be due to machine load too high for minor
> compaction and it was falling behind or some thing happened to minor
> compaction thread on this node. Other two nodes in this cluster are fine.
> Yes, we are using sized compaction strategy.
>
> I am inclined towards 'decommission and bootstrap' this node as it seems
> like performing major compaction on this node is impossible.
>
> However still looking for other solutions...
>
>
>
>
> On Thu, Feb 27, 2014 at 4:03 PM, Robert Coli <rc...@eventbrite.com> wrote:
>
>> On Thu, Feb 27, 2014 at 11:09 AM, Nish garg <pi...@gmail.com> wrote:
>>
>>> I am having OOM during major compaction on one of the column family
>>> where there are lot of SStables (33000) to be compacted. Is there any other
>>> way for them to be compacted? Any help will be really appreciated.
>>>
>>
>> You can use user defined compaction to reduce the working set, but only a
>> major compaction is capable of purging 100% of tombstones.
>>
>> How much garbage is actually in the files? Why do you have 33,000 of
>> them? You mention a major compaction so you are likely not using LCS with
>> the bad 5mb default... how did you end up with so many SSTables?
>>
>> Have you removed the throttle from compaction, generally?
>>
>> What version of Cassandra?
>>
>> =Rob
>>
>>
>

Re: OOM while performing major compaction

Posted by Nish garg <pi...@gmail.com>.
Thanks for replying.

We are  on Cassandra 1.2.9.

We have time series like data structure where we need to keep only last 6
hours of data. So we expire data using  expireddatetime column on column
family and then we run expire script via cron to create tombstones. We
don't use ttl yet and planning to use it in our future release. Hope that
will fix some of the issues caused by expire script as it needs to read the
data first before creating tombstones.

So to answer your question, we have almost 80% of tombstones in those
sstables. (There is no easy way to confirm this unless I convert all those
33000 sstables to JSON file and query them for tombstones).
The reason of 33000 of them may be due to machine load too high for minor
compaction and it was falling behind or some thing happened to minor
compaction thread on this node. Other two nodes in this cluster are fine.
Yes, we are using sized compaction strategy.

I am inclined towards 'decommission and bootstrap' this node as it seems
like performing major compaction on this node is impossible.

However still looking for other solutions...




On Thu, Feb 27, 2014 at 4:03 PM, Robert Coli <rc...@eventbrite.com> wrote:

> On Thu, Feb 27, 2014 at 11:09 AM, Nish garg <pi...@gmail.com> wrote:
>
>> I am having OOM during major compaction on one of the column family where
>> there are lot of SStables (33000) to be compacted. Is there any other way
>> for them to be compacted? Any help will be really appreciated.
>>
>
> You can use user defined compaction to reduce the working set, but only a
> major compaction is capable of purging 100% of tombstones.
>
> How much garbage is actually in the files? Why do you have 33,000 of them?
> You mention a major compaction so you are likely not using LCS with the bad
> 5mb default... how did you end up with so many SSTables?
>
> Have you removed the throttle from compaction, generally?
>
> What version of Cassandra?
>
> =Rob
>
>

Re: OOM while performing major compaction

Posted by Robert Coli <rc...@eventbrite.com>.
On Thu, Feb 27, 2014 at 11:09 AM, Nish garg <pi...@gmail.com> wrote:

> I am having OOM during major compaction on one of the column family where
> there are lot of SStables (33000) to be compacted. Is there any other way
> for them to be compacted? Any help will be really appreciated.
>

You can use user defined compaction to reduce the working set, but only a
major compaction is capable of purging 100% of tombstones.

How much garbage is actually in the files? Why do you have 33,000 of them?
You mention a major compaction so you are likely not using LCS with the bad
5mb default... how did you end up with so many SSTables?

Have you removed the throttle from compaction, generally?

What version of Cassandra?

=Rob

Re: OOM while performing major compaction

Posted by Edward Capriolo <ed...@gmail.com>.
One big downside about major compaction is that (depending on your
cassandra version) the bloom filters size is pre-calculated. Thus cassandra
needs enough heap for your existing 33 k+ sstables and the new large
compacted one. In the past this happened to us when the compaction thread
got hung up, and the sstables grew. When this happens I delete the data
directory and jump in a fresh node. The time to recover from that sstable
build up can be huge even with multi-threaded compaction.


On Thu, Feb 27, 2014 at 2:09 PM, Nish garg <pi...@gmail.com> wrote:

> I am having OOM during major compaction on one of the column family where
> there are lot of SStables (33000) to be compacted. Is there any other way
> for them to be compacted? Any help will be really appreciated.
>
> Here are the details
>
>  /opt/cassandra/current/bin/nodetool -h us1emscsm-01  compact tomcat
> sessions
> Error occurred during compaction
> java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java
> heap space
>  at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
>  at java.util.concurrent.FutureTask.get(FutureTask.java:83)
>  at
> org.apache.cassandra.db.compaction.CompactionManager.performMaximal(CompactionManager.java:334)
>  at
> org.apache.cassandra.db.ColumnFamilyStore.forceMajorCompaction(ColumnFamilyStore.java:1691)
>  at
> org.apache.cassandra.service.StorageService.forceTableCompaction(StorageService.java:2168)
>  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>  at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>  at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>  at java.lang.reflect.Method.invoke(Method.java:597)
>  at
> com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:93)
>  at
> com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:27)
>  at
> com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:208)
>  at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:120)
>  at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:262)
>  at
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:836)
>  at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:761)
>  at
> javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1427)
>  at
> javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:72)
>  at
> javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1265)
>  at
> javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1360)
>  at
> javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:788)
>  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>  at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>  at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>  at java.lang.reflect.Method.invoke(Method.java:597)
>  at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:305)
>  at sun.rmi.transport.Transport$1.run(Transport.java:159)
>  at java.security.AccessController.doPrivileged(Native Method)
>  at sun.rmi.transport.Transport.serviceCall(Transport.java:155)
>  at
> sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:535)
>  at
> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790)
>  at
> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
> Caused by: java.lang.OutOfMemoryError: Java heap space
>  at
> org.apache.cassandra.io.util.RandomAccessReader.<init>(RandomAccessReader.java:77)
>  at
> org.apache.cassandra.io.compress.CompressedRandomAccessReader.<init>(CompressedRandomAccessReader.java:75)
>  at
> org.apache.cassandra.io.compress.CompressedThrottledReader.<init>(CompressedThrottledReader.java:38)
>  at
> org.apache.cassandra.io.compress.CompressedThrottledReader.open(CompressedThrottledReader.java:52)
>  at
> org.apache.cassandra.io.sstable.SSTableReader.openDataReader(SSTableReader.java:1212)
>  at
> org.apache.cassandra.io.sstable.SSTableScanner.<init>(SSTableScanner.java:54)
>  at
> org.apache.cassandra.io.sstable.SSTableReader.getDirectScanner(SSTableReader.java:1032)
>  at
> org.apache.cassandra.io.sstable.SSTableReader.getDirectScanner(SSTableReader.java:1044)
>  at
> org.apache.cassandra.db.compaction.AbstractCompactionStrategy.getScanners(AbstractCompactionStrategy.java:157)
>  at
> org.apache.cassandra.db.compaction.AbstractCompactionStrategy.getScanners(AbstractCompactionStrategy.java:163)
>  at
> org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:117)
>  at
> org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48)
>  at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
>  at
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:58)
>  at
> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:60)
>  at
> org.apache.cassandra.db.compaction.CompactionManager$6.runMayThrow(CompactionManager.java:355)
>  at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
>  at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
>  at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>  at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>  ... 3 more
>
>
> Also
>
>
> /opt/cassandra/current/bin/nodetool -h us1emscsm-01
> getcompactionthreshold tomcat sessions
> Current compaction thresholds for tomcat/sessions:
>  min = 4,  max = 32
>
>
>  /opt/cassandra/current/bin/nodetool -h us1emscsm-01
> getcompactionthroughput tomcat sessions
> Current compaction throughput: 16 MB/s
>
>
>
> /opt/cassandra/current/bin/nodetool -h us1emscsm-01  compactionstats
> pending tasks: 167
> Active compaction remaining time :        n/a
>
>
>