You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Kenneth Failbus (JIRA)" <ji...@apache.org> on 2016/07/27 17:47:20 UTC
[jira] [Updated] (CASSANDRA-8898) sstableloader utility should
allow loading of data from mounted filesystem
[ https://issues.apache.org/jira/browse/CASSANDRA-8898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Kenneth Failbus updated CASSANDRA-8898:
---------------------------------------
Description:
When trying to load data from a mounted filesystem onto a new cluster, following exceptions is observed intermittently, and at some point the sstableloader process gets hung without completing the loading process.
Please note that in my case the scenario was loading the existing sstables from an existing cluster to a brand new cluster.
Finally, it was found that there were some hard assumptions been made by sstableloader utility w.r.t response from the filesystem, which was not working with mounted filesystem.
The work-around was to copy each existing nodes sstable data files locally and then point sstableloader to that local filesystem to then load data onto new cluster.
In case of restoring during disaster recovery from backups the data using sstableloader, this copying to local filesystem of data files and then loading would take a long time.
It would be a good enhancement of the sstableloader utility to enable use of mounted filesystem as copying data locally and then loading is time consuming.
Below is the exception seen during the use of mounted filesystem.
{code}
java.lang.AssertionError: Reference counter -1 for Min-System-jb-5449-Data.db
at org.apache.cassandra.io.sstable.SSTableReader.releaseReference(SSTableReader.java:1146)
at org.apache.cassandra.streaming.StreamTransferTask.complete(StreamTransferTask.java:74)
at org.apache.cassandra.streaming.StreamSession.received(StreamSession.java:542)
at org.apache.cassandra.streaming.StreamSession.messageReceived(StreamSession.java:424)
at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:245)
at java.lang.Thread.run(Thread.java:744)
WARN 21:07:16,853 [Stream #3e5a5ba0-bdef-11e4-a975-5777dbff0945] Stream failed
at org.apache.cassandra.io.compress.CompressedRandomAccessReader.open(CompressedRandomAccessReader.java:59)
at org.apache.cassandra.io.sstable.SSTableReader.openDataReader(SSTableReader.java:1406)
at org.apache.cassandra.streaming.compress.CompressedStreamWriter.write(CompressedStreamWriter.java:55)
at org.apache.cassandra.streaming.messages.OutgoingFileMessage$1.serialize(OutgoingFileMessage.java:59)
at org.apache.cassandra.streaming.messages.OutgoingFileMessage$1.serialize(OutgoingFileMessage.java:42)
at org.apache.cassandra.streaming.messages.StreamMessage.serialize(StreamMessage.java:45)
at org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.sendMessage(ConnectionHandler.java:339)
at org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.run(ConnectionHandler.java:311)
at java.lang.Thread.run(Thread.java:744)
Caused by: java.io.FileNotFoundException: /opt/tmp/casapp-c1-c00055-g.ch.tvx.comcast.com/MinDataService/System/MinDataService-System-jb-5997-Data.db (No such file or directory)
at java.io.RandomAccessFile.open(Native Method)
at java.io.RandomAccessFile.<init>(RandomAccessFile.java:241)
at org.apache.cassandra.io.util.RandomAccessReader.<init>(RandomAccessReader.java:58)
at org.apache.cassandra.io.compress.CompressedRandomAccessReader.<init>(CompressedRandomAccessReader.java:76)
at org.apache.cassandra.io.compress.CompressedRandomAccessReader.open(CompressedRandomAccessReader.java:55)
... 8 more
Exception in thread "STREAM-OUT-/96.115.88.196" java.lang.NullPointerException
at org.apache.cassandra.streaming.ConnectionHandler$MessageHandler.signalCloseDone(ConnectionHandler.java:205)
at org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.run(ConnectionHandler.java:331)
at java.lang.Thread.run(Thread.java:744)
ERROR 20:49:35,646 [Stream #d9fce650-bdf3-11e4-b6c0-252cb9b3e9f3] Streaming error occurred
java.lang.AssertionError: Reference counter -3 for /opt/tmp/casapp-c1-c00055-g.ch.tvx.comcast.com/MinDataService/System/MinDataService-System-jb-4897-Data.db
at org.apache.cassandra.io.sstable.SSTableReader.releaseReference(SSTableReader.java:1146)
at org.apache.cassandra.streaming.StreamTransferTask.complete(StreamTransferTask.java:74)
at org.apache.cassandra.streaming.StreamSession.received(StreamSession.java:542)
at org.apache.cassandra.streaming.StreamSession.messageReceived(StreamSession.java:424)
at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:245)
at java.lang.Thread.run(Thread.java:744)
Exception in thread "STREAM-IN-/96.115.88.196" java.lang.NullPointerException
at org.apache.cassandra.streaming.ConnectionHandler$MessageHandler.signalCloseDone(ConnectionHandler.java:205)
at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:260)
at java.lang.Thread.run(Thread.java:744)
{code}
was:
When trying to load data from a mounted filesystem onto a new cluster, following exceptions is observed intermittently, and at some point the sstableloader process gets hung without completing the loading process.
Please note that in my case the scenario was loading the existing sstables from an existing cluster to a brand new cluster.
Finally, it was found that there were some hard assumptions been made by sstableloader utility w.r.t response from the filesystem, which was not working with mounted filesystem.
The work-around was to copy each existing nodes sstable data files locally and then point sstableloader to that local filesystem to then load data onto new cluster.
In case of restoring during disaster recovery from backups the data using sstableloader, this copying to local filesystem of data files and then loading would take a long time.
It would be a good enhancement of the sstableloader utility to enable use of mounted filesystem as copying data locally and then loading is time consuming.
Below is the exception seen during the use of mounted filesystem.
{code}
java.lang.AssertionError: Reference counter -1 for /opt/tmp/casapp-c1-c00053-g.ch.tvx.comcast.com/MinDataService/System/MinDataService-System-jb-5449-Data.db
at org.apache.cassandra.io.sstable.SSTableReader.releaseReference(SSTableReader.java:1146)
at org.apache.cassandra.streaming.StreamTransferTask.complete(StreamTransferTask.java:74)
at org.apache.cassandra.streaming.StreamSession.received(StreamSession.java:542)
at org.apache.cassandra.streaming.StreamSession.messageReceived(StreamSession.java:424)
at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:245)
at java.lang.Thread.run(Thread.java:744)
WARN 21:07:16,853 [Stream #3e5a5ba0-bdef-11e4-a975-5777dbff0945] Stream failed
at org.apache.cassandra.io.compress.CompressedRandomAccessReader.open(CompressedRandomAccessReader.java:59)
at org.apache.cassandra.io.sstable.SSTableReader.openDataReader(SSTableReader.java:1406)
at org.apache.cassandra.streaming.compress.CompressedStreamWriter.write(CompressedStreamWriter.java:55)
at org.apache.cassandra.streaming.messages.OutgoingFileMessage$1.serialize(OutgoingFileMessage.java:59)
at org.apache.cassandra.streaming.messages.OutgoingFileMessage$1.serialize(OutgoingFileMessage.java:42)
at org.apache.cassandra.streaming.messages.StreamMessage.serialize(StreamMessage.java:45)
at org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.sendMessage(ConnectionHandler.java:339)
at org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.run(ConnectionHandler.java:311)
at java.lang.Thread.run(Thread.java:744)
Caused by: java.io.FileNotFoundException: /opt/tmp/casapp-c1-c00055-g.ch.tvx.comcast.com/MinDataService/System/MinDataService-System-jb-5997-Data.db (No such file or directory)
at java.io.RandomAccessFile.open(Native Method)
at java.io.RandomAccessFile.<init>(RandomAccessFile.java:241)
at org.apache.cassandra.io.util.RandomAccessReader.<init>(RandomAccessReader.java:58)
at org.apache.cassandra.io.compress.CompressedRandomAccessReader.<init>(CompressedRandomAccessReader.java:76)
at org.apache.cassandra.io.compress.CompressedRandomAccessReader.open(CompressedRandomAccessReader.java:55)
... 8 more
Exception in thread "STREAM-OUT-/96.115.88.196" java.lang.NullPointerException
at org.apache.cassandra.streaming.ConnectionHandler$MessageHandler.signalCloseDone(ConnectionHandler.java:205)
at org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.run(ConnectionHandler.java:331)
at java.lang.Thread.run(Thread.java:744)
ERROR 20:49:35,646 [Stream #d9fce650-bdf3-11e4-b6c0-252cb9b3e9f3] Streaming error occurred
java.lang.AssertionError: Reference counter -3 for /opt/tmp/casapp-c1-c00055-g.ch.tvx.comcast.com/MinDataService/System/MinDataService-System-jb-4897-Data.db
at org.apache.cassandra.io.sstable.SSTableReader.releaseReference(SSTableReader.java:1146)
at org.apache.cassandra.streaming.StreamTransferTask.complete(StreamTransferTask.java:74)
at org.apache.cassandra.streaming.StreamSession.received(StreamSession.java:542)
at org.apache.cassandra.streaming.StreamSession.messageReceived(StreamSession.java:424)
at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:245)
at java.lang.Thread.run(Thread.java:744)
Exception in thread "STREAM-IN-/96.115.88.196" java.lang.NullPointerException
at org.apache.cassandra.streaming.ConnectionHandler$MessageHandler.signalCloseDone(ConnectionHandler.java:205)
at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:260)
at java.lang.Thread.run(Thread.java:744)
{code}
> sstableloader utility should allow loading of data from mounted filesystem
> --------------------------------------------------------------------------
>
> Key: CASSANDRA-8898
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8898
> Project: Cassandra
> Issue Type: Improvement
> Components: Tools
> Environment: 2.0.12
> Reporter: Kenneth Failbus
>
> When trying to load data from a mounted filesystem onto a new cluster, following exceptions is observed intermittently, and at some point the sstableloader process gets hung without completing the loading process.
> Please note that in my case the scenario was loading the existing sstables from an existing cluster to a brand new cluster.
> Finally, it was found that there were some hard assumptions been made by sstableloader utility w.r.t response from the filesystem, which was not working with mounted filesystem.
> The work-around was to copy each existing nodes sstable data files locally and then point sstableloader to that local filesystem to then load data onto new cluster.
> In case of restoring during disaster recovery from backups the data using sstableloader, this copying to local filesystem of data files and then loading would take a long time.
> It would be a good enhancement of the sstableloader utility to enable use of mounted filesystem as copying data locally and then loading is time consuming.
> Below is the exception seen during the use of mounted filesystem.
> {code}
> java.lang.AssertionError: Reference counter -1 for Min-System-jb-5449-Data.db
> at org.apache.cassandra.io.sstable.SSTableReader.releaseReference(SSTableReader.java:1146)
> at org.apache.cassandra.streaming.StreamTransferTask.complete(StreamTransferTask.java:74)
> at org.apache.cassandra.streaming.StreamSession.received(StreamSession.java:542)
> at org.apache.cassandra.streaming.StreamSession.messageReceived(StreamSession.java:424)
> at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:245)
> at java.lang.Thread.run(Thread.java:744)
> WARN 21:07:16,853 [Stream #3e5a5ba0-bdef-11e4-a975-5777dbff0945] Stream failed
> at org.apache.cassandra.io.compress.CompressedRandomAccessReader.open(CompressedRandomAccessReader.java:59)
> at org.apache.cassandra.io.sstable.SSTableReader.openDataReader(SSTableReader.java:1406)
> at org.apache.cassandra.streaming.compress.CompressedStreamWriter.write(CompressedStreamWriter.java:55)
> at org.apache.cassandra.streaming.messages.OutgoingFileMessage$1.serialize(OutgoingFileMessage.java:59)
> at org.apache.cassandra.streaming.messages.OutgoingFileMessage$1.serialize(OutgoingFileMessage.java:42)
> at org.apache.cassandra.streaming.messages.StreamMessage.serialize(StreamMessage.java:45)
> at org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.sendMessage(ConnectionHandler.java:339)
> at org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.run(ConnectionHandler.java:311)
> at java.lang.Thread.run(Thread.java:744)
> Caused by: java.io.FileNotFoundException: /opt/tmp/casapp-c1-c00055-g.ch.tvx.comcast.com/MinDataService/System/MinDataService-System-jb-5997-Data.db (No such file or directory)
> at java.io.RandomAccessFile.open(Native Method)
> at java.io.RandomAccessFile.<init>(RandomAccessFile.java:241)
> at org.apache.cassandra.io.util.RandomAccessReader.<init>(RandomAccessReader.java:58)
> at org.apache.cassandra.io.compress.CompressedRandomAccessReader.<init>(CompressedRandomAccessReader.java:76)
> at org.apache.cassandra.io.compress.CompressedRandomAccessReader.open(CompressedRandomAccessReader.java:55)
> ... 8 more
> Exception in thread "STREAM-OUT-/96.115.88.196" java.lang.NullPointerException
> at org.apache.cassandra.streaming.ConnectionHandler$MessageHandler.signalCloseDone(ConnectionHandler.java:205)
> at org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.run(ConnectionHandler.java:331)
> at java.lang.Thread.run(Thread.java:744)
> ERROR 20:49:35,646 [Stream #d9fce650-bdf3-11e4-b6c0-252cb9b3e9f3] Streaming error occurred
> java.lang.AssertionError: Reference counter -3 for /opt/tmp/casapp-c1-c00055-g.ch.tvx.comcast.com/MinDataService/System/MinDataService-System-jb-4897-Data.db
> at org.apache.cassandra.io.sstable.SSTableReader.releaseReference(SSTableReader.java:1146)
> at org.apache.cassandra.streaming.StreamTransferTask.complete(StreamTransferTask.java:74)
> at org.apache.cassandra.streaming.StreamSession.received(StreamSession.java:542)
> at org.apache.cassandra.streaming.StreamSession.messageReceived(StreamSession.java:424)
> at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:245)
> at java.lang.Thread.run(Thread.java:744)
> Exception in thread "STREAM-IN-/96.115.88.196" java.lang.NullPointerException
> at org.apache.cassandra.streaming.ConnectionHandler$MessageHandler.signalCloseDone(ConnectionHandler.java:205)
> at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:260)
> at java.lang.Thread.run(Thread.java:744)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)