You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Bryon Spahn <bs...@kitedesk.com> on 2014/07/07 14:20:53 UTC
Compaction causing listeners to stall
Greetings all,
I am experiencing a strange issue where we run a compaction job weekly and
as a result, the listeners stall. This is a single node cluster running on
an i2.2xl instance in AWS. We are getting the message:
*[StorageServiceShutdownHook]*
and then:
*MessagingService has terminated the accept() thread*
The result is that the cluster will not accept any requests and must be
re-started to function again properly. Based on my research, this appears
to be the result of hitting a memory limit in the jvm. We have not
customized the memory settings from default and are using the calculated
results as defined in the environment script. We have disabled the weekly
compaction for now but would prefer to work out the cause.
Any help would be greatly appreciated!
Re: Compaction causing listeners to stall
Posted by Robert Coli <rc...@eventbrite.com>.
On Tue, Jul 8, 2014 at 5:31 AM, Bryon Spahn <bs...@kitedesk.com> wrote:
> FSReadError in /mnt/data/cassandra/COMPANY/crmFieldInfo/COMPANYFieldInfo-
> jb-710-Data.db
>
>
FSReadError seems relatively likely to be a problem on the underlying
filesystem, though I don't have a sec right now to grep and check.
> 08 05:00:09,126 StorageService.java (line 364) Stopping gossiper
>
> 08 05:00:09,126 StorageService.java (line 278) Stopping gossip by operator
> request
>
>
This sequence almost certainly means that the gossiper was in fact shut
down by the JBOD functionality.
> After this the listeners are no longer available but the DB does not
> officially die just hangs and needs a restart.
>
Other than during clean shutdown, there is no circumstance under which
Cassandra the application terminates itself within the JVM.
tl;dr - fix your broken disk. :)
=Rob
RE: Compaction causing listeners to stall
Posted by Bryon Spahn <bs...@kitedesk.com>.
Robert,
New development today:
FSReadError in
/mnt/data/cassandra/COMPANY/crmFieldInfo/COMPANYFieldInfo-jb-710-Data.db
at
org.apache.cassandra.io.compress.CompressedRandomAccessReader.reBuffer(CompressedRandomAccessReader.java:95)
at
org.apache.cassandra.io.compress.CompressedThrottledReader.reBuffer(CompressedThrottledReader.java:41)
at
org.apache.cassandra.io.util.RandomAccessReader.read(RandomAccessReader.java:323)
at java.io.RandomAccessFile.readFully(RandomAccessFile.java:444)
at java.io.RandomAccessFile.readFully(RandomAccessFile.java:424)
at
org.apache.cassandra.io.util.RandomAccessReader.readBytes(RandomAccessReader.java:348)
at
org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:392)
at
org.apache.cassandra.utils.ByteBufferUtil.readWithLength(ByteBufferUtil.java:355)
at
org.apache.cassandra.db.ColumnSerializer.deserializeColumnBody(ColumnSerializer.java:124)
at
org.apache.cassandra.db.OnDiskAtom$Serializer.deserializeFromSSTable(OnDiskAtom.java:85)
at org.apache.cassandra.db.Column$1.computeNext(Column.java:75)
at org.apache.cassandra.db.Column$1.computeNext(Column.java:64)
at
com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
at
com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
at
org.apache.cassandra.io.sstable.SSTableIdentityIterator.getColumnFamilyWithColumns(SSTableIdentityIterator.java:185)
at
org.apache.cassandra.db.compaction.ParallelCompactionIterable$Deserializer$1.runMayThrow(ParallelCompactionIterable.java:271)
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
at java.lang.Thread.run(Thread.java:744)
Caused by: java.nio.channels.ClosedChannelException
at sun.nio.ch.FileChannelImpl.ensureOpen(FileChannelImpl.java:99)
at sun.nio.ch.FileChannelImpl.position(FileChannelImpl.java:250)
at
org.apache.cassandra.io.compress.CompressedRandomAccessReader.decompressChunk(CompressedRandomAccessReader.java:101)
at
org.apache.cassandra.io.compress.CompressedRandomAccessReader.reBuffer(CompressedRandomAccessReader.java:87)
... 17 more
Followed by:
ERROR [Deserialize
SSTableReader(path='/mnt/data/cassandra/COMPANY/crmFieldInfo/COMPANYFieldInfo-jb-710-Data.db')]
2014-07-08 05:00:09,126 StorageService.java (line 364) Stopping gossiper
WARN [Deserialize
SSTableReader(path='/mnt/data/cassandra/COMPANY/crmFieldInfo/COMPANYFieldInfo-jb-710-Data.db')]
2014-07-08 05:00:09,126 StorageService.java (line 278) Stopping gossip by
operator request
INFO [Deserialize
SSTableReader(path='/mnt/data/cassandra/COMPANY/crmFieldInfo/COMPANYFieldInfo-jb-710-Data.db')]
2014-07-08 05:00:09,126 Gossiper.java (line 1251) Announcing shutdown
After this the listeners are no longer available but the DB does not
officially die just hangs and needs a restart.
*From:* Robert Coli [mailto:rcoli@eventbrite.com]
*Sent:* Monday, July 7, 2014 6:55 PM
*To:* user@cassandra.apache.org
*Subject:* Re: Compaction causing listeners to stall
On Mon, Jul 7, 2014 at 5:20 AM, Bryon Spahn <bs...@kitedesk.com> wrote:
I am experiencing a strange issue where we run a compaction job weekly and
as a result, the listeners stall. This is a single node cluster running on
an i2.2xl instance in AWS. We are getting the message:
There are almost no cases where it makes sense to run a single node of
Cassandra, especially in production.
*[StorageServiceShutdownHook]*
I bet you a donut that you're OOMing the JVM. Stop doing that, and your
Cassandra node will stop crashing.
https://issues.apache.org/jira/browse/CASSANDRA-7507
Is probably the case you have just hit.
Basically, in some pathological circumstances, the JVM will send Cassandra
a signal that it handles as if you were an operator attempting a clean
shutdown. This probably usually does not succeed, but may be worth a shot.
=Rob
Re: Compaction causing listeners to stall
Posted by Robert Coli <rc...@eventbrite.com>.
On Mon, Jul 7, 2014 at 5:20 AM, Bryon Spahn <bs...@kitedesk.com> wrote:
> I am experiencing a strange issue where we run a compaction job weekly and
> as a result, the listeners stall. This is a single node cluster running on
> an i2.2xl instance in AWS. We are getting the message:
>
There are almost no cases where it makes sense to run a single node of
Cassandra, especially in production.
> *[StorageServiceShutdownHook]*
>
I bet you a donut that you're OOMing the JVM. Stop doing that, and your
Cassandra node will stop crashing.
https://issues.apache.org/jira/browse/CASSANDRA-7507
Is probably the case you have just hit.
Basically, in some pathological circumstances, the JVM will send Cassandra
a signal that it handles as if you were an operator attempting a clean
shutdown. This probably usually does not succeed, but may be worth a shot.
=Rob