You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by John Sanda <jo...@gmail.com> on 2016/11/01 18:10:10 UTC

commit log on NFS volume

I know that using NFS is discouraged, particularly for the commit log. Can
anyone shed some light into what kinds of problems I might encounter aside
from performance? The reason for my inquiry is because I have some
deployments with Cassandra 2.2.1 that use NFS and are experiencing some
problems like reoccurring corrupted commit log segments on start up:

    ERROR 19:38:42 Exiting due to error while processing commit log
during initialization.
    org.apache.cassandra.db.commitlog.CommitLogReplayer$CommitLogReplayException:
Mutation checksum failure at 33296351 in CommitLog-5-1474325237114.log
            at org.apache.cassandra.db.commitlog.CommitLogReplayer.handleReplayError(CommitLogReplayer.java:622)
[apache-cassandra-2.2.1.jar]
            at org.apache.cassandra.db.commitlog.CommitLogReplayer.replaySyncSection(CommitLogReplayer.java:492)
[apache-cassandra-2.2.1.jar]
            at org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:388)
[apache-cassandra-2.2.1.jar]
            at org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:147)
[apache-cassandra-2.2.1.jar]
            at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:189)
[apache-cassandra-2.2.1.jar]
            at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:169)
[apache-cassandra-2.2.1.jar]
            at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:266)
[apache-cassandra-2.2.1.jar]
            at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:488)
[apache-cassandra-2.2.1.jar]
            at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:595)
[apache-cassandra-2.2.1.jar]


In one deployment after removing all of corrupted commit log segments I got
a different error:

Exception (java.lang.RuntimeException) encountered during startup:
java.nio.file.NoSuchFileException:
/cassandra_data/data/hawkular_metrics/metrics_idx-1ea881506ee311e6890b131c1bc89929/la-86-big-Statistics.db
java.lang.RuntimeException: java.nio.file.NoSuchFileException:
/cassandra_data/data/hawkular_metrics/metrics_idx-1ea881506ee311e6890b131c1bc89929/la-86-big-Statistics.db
	at org.apache.cassandra.io.util.ChannelProxy.openChannel(ChannelProxy.java:55)
	at org.apache.cassandra.io.util.ChannelProxy.<init>(ChannelProxy.java:66)
	at org.apache.cassandra.io.util.RandomAccessReader.open(RandomAccessReader.java:78)
	at org.apache.cassandra.io.sstable.metadata.MetadataSerializer.deserialize(MetadataSerializer.java:91)
	at org.apache.cassandra.io.sstable.metadata.MetadataSerializer.deserialize(MetadataSerializer.java:101)
	at org.apache.cassandra.db.ColumnFamilyStore.removeUnfinishedCompactionLeftovers(ColumnFamilyStore.java:672)
	at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:216)
	at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:488)
	at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:595)
Caused by: java.nio.file.NoSuchFileException:
/cassandra_data/data/hawkular_metrics/metrics_idx-1ea881506ee311e6890b131c1bc89929/la-86-big-Statistics.db
	at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
	at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
	at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
	at sun.nio.fs.UnixFileSystemProvider.newFileChannel(UnixFileSystemProvider.java:177)
	at java.nio.channels.FileChannel.open(FileChannel.java:287)
	at java.nio.channels.FileChannel.open(FileChannel.java:335)
	at org.apache.cassandra.io.util.ChannelProxy.openChannel(ChannelProxy.java:51)
	... 8 more


The latter error looks like it involves compaction and might be unrelated.
I don't know if it matters, but I have commit log compression enabled in
these environments.

-- 

- John

Re: commit log on NFS volume

Posted by Benjamin Roth <be...@jaumo.com>.
The Second exception states there is a File of an sstable Missing. Is it
possible you didnt only delete commit logs or nfs Mount is stale or Not
mounted?

Am 01.11.2016 19:52 schrieb "John Sanda" <jo...@gmail.com>:

> Using nfs for a distribited System like Cassandra is like putting a
>> Ferrari on a Truck and going for a Race with the Truck. It is simply
>> nonsense.
>
>
> As I mentioned in my original post, I am aware that using NFS is
> considered bad and even documented as an anti-pattern. Your analogy,
> interesting as it may be, is not helpful. It simply restating what has
> already been said. I don't even know that NFS is to blame for the
> CommitLogReplayException that I cited.
>
> On Tue, Nov 1, 2016 at 2:43 PM, Benjamin Roth <be...@jaumo.com>
> wrote:
>
>> Using nfs for a distribited System like Cassandra is like putting a
>> Ferrari on a Truck and going for a Race with the Truck. It is simply
>> nonsense.
>>
>> Am 01.11.2016 19:39 schrieb "Vladimir Yudovin" <vl...@winguzone.com>:
>>
>>> Hi,
>>>
>>> it's not only performance issue. In case of network problem writer tread
>>> can be blocked, also in case of failure loss of data can occur.
>>>
>>> Best regards, Vladimir Yudovin,
>>>
>>> *Winguzone <https://winguzone.com?from=list> - Hosted Cloud
>>> CassandraLaunch your cluster in minutes.*
>>>
>>>
>>> ---- On Tue, 01 Nov 2016 14:10:10 -0400*John Sanda
>>> <john.sanda@gmail.com <jo...@gmail.com>>* wrote ----
>>>
>>> I know that using NFS is discouraged, particularly for the commit log.
>>> Can anyone shed some light into what kinds of problems I might encounter
>>> aside from performance? The reason for my inquiry is because I have some
>>> deployments with Cassandra 2.2.1 that use NFS and are experiencing some
>>> problems like reoccurring corrupted commit log segments on start up:
>>>
>>>     ERROR 19:38:42 Exiting due to error while processing commit log during initialization.
>>>     org.apache.cassandra.db.commitlog.CommitLogReplayer$CommitLogReplayException: Mutation checksum failure at 33296351 in CommitLog-5-1474325237114.log
>>>             at org.apache.cassandra.db.commitlog.CommitLogReplayer.handleReplayError(CommitLogReplayer.java:622) [apache-cassandra-2.2.1.jar]
>>>             at org.apache.cassandra.db.commitlog.CommitLogReplayer.replaySyncSection(CommitLogReplayer.java:492) [apache-cassandra-2.2.1.jar]
>>>             at org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:388) [apache-cassandra-2.2.1.jar]
>>>             at org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:147) [apache-cassandra-2.2.1.jar]
>>>             at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:189) [apache-cassandra-2.2.1.jar]
>>>             at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:169) [apache-cassandra-2.2.1.jar]
>>>             at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:266) [apache-cassandra-2.2.1.jar]
>>>             at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:488) [apache-cassandra-2.2.1.jar]
>>>             at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:595) [apache-cassandra-2.2.1.jar]
>>>
>>>
>>> In one deployment after removing all of corrupted commit log segments I
>>> got a different error:
>>>
>>> Exception (java.lang.RuntimeException) encountered during startup: java.nio.file.NoSuchFileException: /cassandra_data/data/hawkular_metrics/metrics_idx-1ea881506ee311e6890b131c1bc89929/la-86-big-Statistics.db
>>> java.lang.RuntimeException: java.nio.file.NoSuchFileException: /cassandra_data/data/hawkular_metrics/metrics_idx-1ea881506ee311e6890b131c1bc89929/la-86-big-Statistics.db
>>> 	at org.apache.cassandra.io.util.ChannelProxy.openChannel(ChannelProxy.java:55)
>>> 	at org.apache.cassandra.io.util.ChannelProxy.<init>(ChannelProxy.java:66)
>>> 	at org.apache.cassandra.io.util.RandomAccessReader.open(RandomAccessReader.java:78)
>>> 	at org.apache.cassandra.io.sstable.metadata.MetadataSerializer.deserialize(MetadataSerializer.java:91)
>>> 	at org.apache.cassandra.io.sstable.metadata.MetadataSerializer.deserialize(MetadataSerializer.java:101)
>>> 	at org.apache.cassandra.db.ColumnFamilyStore.removeUnfinishedCompactionLeftovers(ColumnFamilyStore.java:672)
>>> 	at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:216)
>>> 	at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:488)
>>> 	at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:595)
>>> Caused by: java.nio.file.NoSuchFileException: /cassandra_data/data/hawkular_metrics/metrics_idx-1ea881506ee311e6890b131c1bc89929/la-86-big-Statistics.db
>>> 	at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
>>> 	at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
>>> 	at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
>>> 	at sun.nio.fs.UnixFileSystemProvider.newFileChannel(UnixFileSystemProvider.java:177)
>>> 	at java.nio.channels.FileChannel.open(FileChannel.java:287)
>>> 	at java.nio.channels.FileChannel.open(FileChannel.java:335)
>>> 	at org.apache.cassandra.io.util.ChannelProxy.openChannel(ChannelProxy.java:51)
>>> 	... 8 more
>>>
>>>
>>> The latter error looks like it involves compaction and might be
>>> unrelated. I don't know if it matters, but I have commit log compression
>>> enabled in these environments.
>>>
>>> --
>>>
>>> - John
>>>
>>>
>>>
>
>
> --
>
> - John
>

Re: commit log on NFS volume

Posted by John Sanda <jo...@gmail.com>.
>
> Using nfs for a distribited System like Cassandra is like putting a
> Ferrari on a Truck and going for a Race with the Truck. It is simply
> nonsense.


As I mentioned in my original post, I am aware that using NFS is considered
bad and even documented as an anti-pattern. Your analogy, interesting as it
may be, is not helpful. It simply restating what has already been said. I
don't even know that NFS is to blame for the CommitLogReplayException that
I cited.

On Tue, Nov 1, 2016 at 2:43 PM, Benjamin Roth <be...@jaumo.com>
wrote:

> Using nfs for a distribited System like Cassandra is like putting a
> Ferrari on a Truck and going for a Race with the Truck. It is simply
> nonsense.
>
> Am 01.11.2016 19:39 schrieb "Vladimir Yudovin" <vl...@winguzone.com>:
>
>> Hi,
>>
>> it's not only performance issue. In case of network problem writer tread
>> can be blocked, also in case of failure loss of data can occur.
>>
>> Best regards, Vladimir Yudovin,
>>
>> *Winguzone <https://winguzone.com?from=list> - Hosted Cloud
>> CassandraLaunch your cluster in minutes.*
>>
>>
>> ---- On Tue, 01 Nov 2016 14:10:10 -0400*John Sanda <john.sanda@gmail.com
>> <jo...@gmail.com>>* wrote ----
>>
>> I know that using NFS is discouraged, particularly for the commit log.
>> Can anyone shed some light into what kinds of problems I might encounter
>> aside from performance? The reason for my inquiry is because I have some
>> deployments with Cassandra 2.2.1 that use NFS and are experiencing some
>> problems like reoccurring corrupted commit log segments on start up:
>>
>>     ERROR 19:38:42 Exiting due to error while processing commit log during initialization.
>>     org.apache.cassandra.db.commitlog.CommitLogReplayer$CommitLogReplayException: Mutation checksum failure at 33296351 in CommitLog-5-1474325237114.log
>>             at org.apache.cassandra.db.commitlog.CommitLogReplayer.handleReplayError(CommitLogReplayer.java:622) [apache-cassandra-2.2.1.jar]
>>             at org.apache.cassandra.db.commitlog.CommitLogReplayer.replaySyncSection(CommitLogReplayer.java:492) [apache-cassandra-2.2.1.jar]
>>             at org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:388) [apache-cassandra-2.2.1.jar]
>>             at org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:147) [apache-cassandra-2.2.1.jar]
>>             at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:189) [apache-cassandra-2.2.1.jar]
>>             at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:169) [apache-cassandra-2.2.1.jar]
>>             at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:266) [apache-cassandra-2.2.1.jar]
>>             at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:488) [apache-cassandra-2.2.1.jar]
>>             at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:595) [apache-cassandra-2.2.1.jar]
>>
>>
>> In one deployment after removing all of corrupted commit log segments I
>> got a different error:
>>
>> Exception (java.lang.RuntimeException) encountered during startup: java.nio.file.NoSuchFileException: /cassandra_data/data/hawkular_metrics/metrics_idx-1ea881506ee311e6890b131c1bc89929/la-86-big-Statistics.db
>> java.lang.RuntimeException: java.nio.file.NoSuchFileException: /cassandra_data/data/hawkular_metrics/metrics_idx-1ea881506ee311e6890b131c1bc89929/la-86-big-Statistics.db
>> 	at org.apache.cassandra.io.util.ChannelProxy.openChannel(ChannelProxy.java:55)
>> 	at org.apache.cassandra.io.util.ChannelProxy.<init>(ChannelProxy.java:66)
>> 	at org.apache.cassandra.io.util.RandomAccessReader.open(RandomAccessReader.java:78)
>> 	at org.apache.cassandra.io.sstable.metadata.MetadataSerializer.deserialize(MetadataSerializer.java:91)
>> 	at org.apache.cassandra.io.sstable.metadata.MetadataSerializer.deserialize(MetadataSerializer.java:101)
>> 	at org.apache.cassandra.db.ColumnFamilyStore.removeUnfinishedCompactionLeftovers(ColumnFamilyStore.java:672)
>> 	at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:216)
>> 	at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:488)
>> 	at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:595)
>> Caused by: java.nio.file.NoSuchFileException: /cassandra_data/data/hawkular_metrics/metrics_idx-1ea881506ee311e6890b131c1bc89929/la-86-big-Statistics.db
>> 	at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
>> 	at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
>> 	at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
>> 	at sun.nio.fs.UnixFileSystemProvider.newFileChannel(UnixFileSystemProvider.java:177)
>> 	at java.nio.channels.FileChannel.open(FileChannel.java:287)
>> 	at java.nio.channels.FileChannel.open(FileChannel.java:335)
>> 	at org.apache.cassandra.io.util.ChannelProxy.openChannel(ChannelProxy.java:51)
>> 	... 8 more
>>
>>
>> The latter error looks like it involves compaction and might be
>> unrelated. I don't know if it matters, but I have commit log compression
>> enabled in these environments.
>>
>> --
>>
>> - John
>>
>>
>>


-- 

- John

Re: commit log on NFS volume

Posted by Benjamin Roth <be...@jaumo.com>.
Using nfs for a distribited System like Cassandra is like putting a Ferrari
on a Truck and going for a Race with the Truck. It is simply nonsense.

Am 01.11.2016 19:39 schrieb "Vladimir Yudovin" <vl...@winguzone.com>:

> Hi,
>
> it's not only performance issue. In case of network problem writer tread
> can be blocked, also in case of failure loss of data can occur.
>
> Best regards, Vladimir Yudovin,
>
> *Winguzone <https://winguzone.com?from=list> - Hosted Cloud
> CassandraLaunch your cluster in minutes.*
>
>
> ---- On Tue, 01 Nov 2016 14:10:10 -0400*John Sanda <john.sanda@gmail.com
> <jo...@gmail.com>>* wrote ----
>
> I know that using NFS is discouraged, particularly for the commit log. Can
> anyone shed some light into what kinds of problems I might encounter aside
> from performance? The reason for my inquiry is because I have some
> deployments with Cassandra 2.2.1 that use NFS and are experiencing some
> problems like reoccurring corrupted commit log segments on start up:
>
>     ERROR 19:38:42 Exiting due to error while processing commit log during initialization.
>     org.apache.cassandra.db.commitlog.CommitLogReplayer$CommitLogReplayException: Mutation checksum failure at 33296351 in CommitLog-5-1474325237114.log
>             at org.apache.cassandra.db.commitlog.CommitLogReplayer.handleReplayError(CommitLogReplayer.java:622) [apache-cassandra-2.2.1.jar]
>             at org.apache.cassandra.db.commitlog.CommitLogReplayer.replaySyncSection(CommitLogReplayer.java:492) [apache-cassandra-2.2.1.jar]
>             at org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:388) [apache-cassandra-2.2.1.jar]
>             at org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:147) [apache-cassandra-2.2.1.jar]
>             at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:189) [apache-cassandra-2.2.1.jar]
>             at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:169) [apache-cassandra-2.2.1.jar]
>             at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:266) [apache-cassandra-2.2.1.jar]
>             at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:488) [apache-cassandra-2.2.1.jar]
>             at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:595) [apache-cassandra-2.2.1.jar]
>
>
> In one deployment after removing all of corrupted commit log segments I
> got a different error:
>
> Exception (java.lang.RuntimeException) encountered during startup: java.nio.file.NoSuchFileException: /cassandra_data/data/hawkular_metrics/metrics_idx-1ea881506ee311e6890b131c1bc89929/la-86-big-Statistics.db
> java.lang.RuntimeException: java.nio.file.NoSuchFileException: /cassandra_data/data/hawkular_metrics/metrics_idx-1ea881506ee311e6890b131c1bc89929/la-86-big-Statistics.db
> 	at org.apache.cassandra.io.util.ChannelProxy.openChannel(ChannelProxy.java:55)
> 	at org.apache.cassandra.io.util.ChannelProxy.<init>(ChannelProxy.java:66)
> 	at org.apache.cassandra.io.util.RandomAccessReader.open(RandomAccessReader.java:78)
> 	at org.apache.cassandra.io.sstable.metadata.MetadataSerializer.deserialize(MetadataSerializer.java:91)
> 	at org.apache.cassandra.io.sstable.metadata.MetadataSerializer.deserialize(MetadataSerializer.java:101)
> 	at org.apache.cassandra.db.ColumnFamilyStore.removeUnfinishedCompactionLeftovers(ColumnFamilyStore.java:672)
> 	at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:216)
> 	at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:488)
> 	at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:595)
> Caused by: java.nio.file.NoSuchFileException: /cassandra_data/data/hawkular_metrics/metrics_idx-1ea881506ee311e6890b131c1bc89929/la-86-big-Statistics.db
> 	at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
> 	at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
> 	at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
> 	at sun.nio.fs.UnixFileSystemProvider.newFileChannel(UnixFileSystemProvider.java:177)
> 	at java.nio.channels.FileChannel.open(FileChannel.java:287)
> 	at java.nio.channels.FileChannel.open(FileChannel.java:335)
> 	at org.apache.cassandra.io.util.ChannelProxy.openChannel(ChannelProxy.java:51)
> 	... 8 more
>
>
> The latter error looks like it involves compaction and might be unrelated.
> I don't know if it matters, but I have commit log compression enabled in
> these environments.
>
> --
>
> - John
>
>
>

Re: commit log on NFS volume

Posted by Vladimir Yudovin <vl...@winguzone.com>.
Hi,



it's not only performance issue. In case of network problem writer tread can be blocked, also in case of failure loss of data can occur.



Best regards, Vladimir Yudovin, 

Winguzone - Hosted Cloud Cassandra
Launch your cluster in minutes.





---- On Tue, 01 Nov 2016 14:10:10 -0400John Sanda &lt;john.sanda@gmail.com&gt; wrote ----




I know that using NFS is discouraged, particularly for the commit log. Can anyone shed some light into what kinds of problems I might encounter aside from performance? The reason for my inquiry is because I have some deployments with Cassandra 2.2.1 that use NFS and are experiencing some problems like reoccurring corrupted commit log segments on start up:



 ERROR 19:38:42 Exiting due to error while processing commit log during initialization. org.apache.cassandra.db.commitlog.CommitLogReplayer$CommitLogReplayException: Mutation checksum failure at 33296351 in CommitLog-5-1474325237114.log at org.apache.cassandra.db.commitlog.CommitLogReplayer.handleReplayError(CommitLogReplayer.java:622) [apache-cassandra-2.2.1.jar] at org.apache.cassandra.db.commitlog.CommitLogReplayer.replaySyncSection(CommitLogReplayer.java:492) [apache-cassandra-2.2.1.jar] at org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:388) [apache-cassandra-2.2.1.jar] at org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:147) [apache-cassandra-2.2.1.jar] at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:189) [apache-cassandra-2.2.1.jar] at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:169) [apache-cassandra-2.2.1.jar] at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:266) [apache-cassandra-2.2.1.jar] at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:488) [apache-cassandra-2.2.1.jar] at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:595) [apache-cassandra-2.2.1.jar]



In one deployment after removing all of corrupted commit log segments I got a different error:



Exception (java.lang.RuntimeException) encountered during startup: java.nio.file.NoSuchFileException: /cassandra_data/data/hawkular_metrics/metrics_idx-1ea881506ee311e6890b131c1bc89929/la-86-big-Statistics.db java.lang.RuntimeException: java.nio.file.NoSuchFileException: /cassandra_data/data/hawkular_metrics/metrics_idx-1ea881506ee311e6890b131c1bc89929/la-86-big-Statistics.db at org.apache.cassandra.io.util.ChannelProxy.openChannel(ChannelProxy.java:55) at org.apache.cassandra.io.util.ChannelProxy.&lt;init&gt;(ChannelProxy.java:66) at org.apache.cassandra.io.util.RandomAccessReader.open(RandomAccessReader.java:78) at org.apache.cassandra.io.sstable.metadata.MetadataSerializer.deserialize(MetadataSerializer.java:91) at org.apache.cassandra.io.sstable.metadata.MetadataSerializer.deserialize(MetadataSerializer.java:101) at org.apache.cassandra.db.ColumnFamilyStore.removeUnfinishedCompactionLeftovers(ColumnFamilyStore.java:672) at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:216) at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:488) at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:595) Caused by: java.nio.file.NoSuchFileException: /cassandra_data/data/hawkular_metrics/metrics_idx-1ea881506ee311e6890b131c1bc89929/la-86-big-Statistics.db at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86) at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) at sun.nio.fs.UnixFileSystemProvider.newFileChannel(UnixFileSystemProvider.java:177) at java.nio.channels.FileChannel.open(FileChannel.java:287) at java.nio.channels.FileChannel.open(FileChannel.java:335) at org.apache.cassandra.io.util.ChannelProxy.openChannel(ChannelProxy.java:51) ... 8 more



The latter error looks like it involves compaction and might be unrelated. I don't know if it matters, but I have commit log compression enabled in these environments.



-- 



- John