You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Ondřej Černoš <ce...@gmail.com> on 2013/03/19 17:51:02 UTC
java.io.IOException: FAILED_TO_UNCOMPRESS(5) exception when running
nodetool rebuild
Hi all,
I am running into strange error when bootstrapping Cassandra cluster
in multiple datacenter setup.
The setup is as follows: 3 nodes in AWS east, 3 nodes somewhere on
Rackspace/Openstack. I use my own snitch based on EC2MultiRegionSnitch
(it just adds some ec2 avalability zone parsing capabilities). Nodes
in the cluster connect to each other and all seems ok.
When I start the Rackspace cluster first, populate it with data and
then let the AWS cluster bootstrap from it, it works great. However
the other way round it just breaks.
The breakage demonstrates as follows:
- nodetool rebuild us-east command hangs
- cassandra's log contains the following:
2013-03-19 12:42:15.796+0100 [Thread-14] [DEBUG]
IncomingTcpConnection.java(63)
org.apache.cassandra.net.IncomingTcpConnection: Connection version 6
from ec2-xxx-xxx-xxx-xxx.compute-1.amazonaws.com/xxx.xxx.xxx.xxx
2013-03-19 12:42:15.803+0100 [Thread-14] [DEBUG]
StreamInSession.java(104)
org.apache.cassandra.streaming.StreamInSession: Adding file
/path/to/cassandra/data/key_space/column_family/key_space-column_family-ib-2-Data.db
to Stream Request queue
2013-03-19 12:42:15.803+0100 [Thread-14] [DEBUG]
StreamInSession.java(104)
org.apache.cassandra.streaming.StreamInSession: Adding file
/path/to/cassandra/data/key_space/column_family/key_space-column_family-ib-1-Data.db
to Stream Request queue
2013-03-19 12:42:15.806+0100 [Thread-14] [DEBUG]
IncomingStreamReader.java(112)
org.apache.cassandra.streaming.IncomingStreamReader: Receiving stream
2013-03-19 12:42:15.807+0100 [Thread-14] [DEBUG]
IncomingStreamReader.java(113)
org.apache.cassandra.streaming.IncomingStreamReader: Creating file for
/path/to/cassandra/data/key_space/column_family/key_space-column_family-tmp-ib-2-Data.db
with 7808 estimat
ed keys
2013-03-19 12:42:15.808+0100 [Thread-14] [DEBUG]
ColumnFamilyStore.java(863) org.apache.cassandra.db.ColumnFamilyStore:
component=key_space Checking for sstables overlapping []
2013-03-19 12:42:15.962+0100 [Thread-14] [DEBUG] FileUtils.java(110)
org.apache.cassandra.io.util.FileUtils: Deleting
key_space-column_family-tmp-ib-2-Data.db
2013-03-19 12:42:15.962+0100 [Thread-14] [DEBUG] FileUtils.java(110)
org.apache.cassandra.io.util.FileUtils: Deleting
key_space-column_family-tmp-ib-2-CompressionInfo.db
2013-03-19 12:42:15.962+0100 [Thread-14] [DEBUG] FileUtils.java(110)
org.apache.cassandra.io.util.FileUtils: Deleting
key_space-column_family-tmp-ib-2-TOC.txt
2013-03-19 12:42:15.962+0100 [Thread-14] [DEBUG] FileUtils.java(110)
org.apache.cassandra.io.util.FileUtils: Deleting
key_space-column_family-tmp-ib-2-Filter.db
2013-03-19 12:42:15.963+0100 [Thread-14] [DEBUG] FileUtils.java(110)
org.apache.cassandra.io.util.FileUtils: Deleting
key_space-column_family-tmp-ib-2-Index.db
2013-03-19 12:42:15.963+0100 [Thread-14] [DEBUG] SSTable.java(154)
org.apache.cassandra.io.sstable.SSTable: Deleted
/path/to/cassandra/data/key_space/column_family/key_space-column_family-tmp-ib-2
2013-03-19 12:42:15.963+0100 [Thread-14] [INFO]
StreamInSession.java(136)
org.apache.cassandra.streaming.StreamInSession: Streaming of file
/path/to/cassandra/data/key_space/column_family/key_space-column_family-ib-2-Data.db
sections=127 progress=81048/2444
2013-03-19 12:42:16.059+0100 [Thread-13] [DEBUG]
IncomingTcpConnection.java(79)
org.apache.cassandra.net.IncomingTcpConnection: IOException reading
from socket; closing
java.io.IOException: FAILED_TO_UNCOMPRESS(5)
at org.xerial.snappy.SnappyNative.throw_error(SnappyNative.java:78)
at org.xerial.snappy.SnappyNative.rawUncompress(Native Method)
at org.xerial.snappy.Snappy.rawUncompress(Snappy.java:391)
at org.apache.cassandra.io.compress.SnappyCompressor.uncompress(SnappyCompressor.java:93)
at org.apache.cassandra.streaming.compress.CompressedInputStream.decompress(CompressedInputStream.java:101)
at org.apache.cassandra.streaming.compress.CompressedInputStream.read(CompressedInputStream.java:79)
at java.io.DataInputStream.readUnsignedShort(DataInputStream.java:337)
at org.apache.cassandra.utils.BytesReadTracker.readUnsignedShort(BytesReadTracker.java:140)
at org.apache.cassandra.utils.ByteBufferUtil.readShortLength(ByteBufferUtil.java:361)
at org.apache.cassandra.utils.ByteBufferUtil.readWithShortLength(ByteBufferUtil.java:371)
at org.apache.cassandra.streaming.IncomingStreamReader.streamIn(IncomingStreamReader.java:160)
at org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:122)
at org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:226)
at org.apache.cassandra.net.IncomingTcpConnection.handleStream(IncomingTcpConnection.java:166)
at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:66)
2013-03-19 12:42:15.971+0100 [Thread-16] [ERROR]
CassandraDaemon.java(133)
org.apache.cassandra.service.CassandraDaemon: Exception in thread
Thread[Thread-16,5,main]
java.lang.RuntimeException: java.net.SocketException: Socket closed
at com.google.common.base.Throwables.propagate(Throwables.java:160)
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:32)
at java.lang.Thread.run(Thread.java:679)
Caused by: java.net.SocketException: Socket closed
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:146)
at sun.security.ssl.InputRecord.readFully(InputRecord.java:442)
at sun.security.ssl.InputRecord.readV3Record(InputRecord.java:554)
at sun.security.ssl.InputRecord.read(InputRecord.java:509)
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:850)
at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:807)
at sun.security.ssl.AppInputStream.read(AppInputStream.java:94)
at org.apache.cassandra.streaming.compress.CompressedInputStream$Reader.runMayThrow(CompressedInputStream.java:151)
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
... 1 more
- the SSTable file is not downloaded.
To make it even more confusing, when I tried just one node in AWS and
one on Rackspace, the rebuild ran well. Also for two nodes. Only with
three on AWS side the process failed.
It's cassandra 1.2.1 running on
java version "1.6.0_24"
OpenJDK Runtime Environment (IcedTea6 1.11.8) (rhel-1.56.1.11.8.el6_3-x86_64)
OpenJDK 64-Bit Server VM (build 20.0-b12, mixed mode)
SSL is configured for inter-DC communication.
Any hints?
regards,
Ondrej Cernos
Re: java.io.IOException: FAILED_TO_UNCOMPRESS(5) exception when
running nodetool rebuild
Posted by Ondřej Černoš <ce...@gmail.com>.
Hi Aaron,
I switched to 1.2.3 with no luck. I created
https://issues.apache.org/jira/browse/CASSANDRA-5391 describing the
problem. Maybe it's related to the EOFException problem, but I am not
sure - I don't know Cassandra internals well and I have never seen the
EOFException.
regards,
ondrej
On Tue, Mar 26, 2013 at 9:26 PM, aaron morton <aa...@thelastpickle.com> wrote:
> If you are still on 1.2.1 may be this
>
> https://issues.apache.org/jira/browse/CASSANDRA-5105
>
> Fixed in 1.2.2
>
> If you are on 1.2.3 there is also
> https://issues.apache.org/jira/browse/CASSANDRA-5381
>
> Cheers
>
> -----------------
> Aaron Morton
> Freelance Cassandra Consultant
> New Zealand
>
> @aaronmorton
> http://www.thelastpickle.com
>
> On 26/03/2013, at 5:10 AM, Ondřej Černoš <ce...@gmail.com> wrote:
>
> Hi all,
>
> I am still unable to move forward with this issue.
>
> - when I switch SSL off in inter-DC communication, nodetool rebuild works
> well
> - when I switch internode_compression off, I still get
> java.io.IOException: FAILED_TO_UNCOMPRESS exception. Does
> internode_compression: none really switch off the snappy compression
> of the internode communication? The stacktrace - see the previous mail
> - clearly demonstrates some compression is involved
> - I managed to trigger another exception:
>
> java.lang.RuntimeException: javax.net.ssl.SSLException: bad record MAC
> at com.google.common.base.Throwables.propagate(Throwables.java:160)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:32)
> at java.lang.Thread.run(Thread.java:662)
> Caused by: javax.net.ssl.SSLException: bad record MAC
> at com.sun.net.ssl.internal.ssl.Alerts.getSSLException(Alerts.java:190)
> at com.sun.net.ssl.internal.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1649)
> at com.sun.net.ssl.internal.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1607)
> at
> com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:859)
> at
> com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:755)
> at com.sun.net.ssl.internal.ssl.AppInputStream.read(AppInputStream.java:75)
> at
> org.apache.cassandra.streaming.compress.CompressedInputStream$Reader.runMayThrow(CompressedInputStream.java:151)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> ... 1 more
>
> I managed to trigger this exception only once however.
>
> The fact the transfer works when SSL is off and fails with SSL is
> another strange thing with this issue.
>
> Any ideas or hints?
>
> regards,
>
> Ondrej Cernos
>
> On Tue, Mar 19, 2013 at 5:51 PM, Ondřej Černoš <ce...@gmail.com> wrote:
>
> Hi all,
>
> I am running into strange error when bootstrapping Cassandra cluster
> in multiple datacenter setup.
>
> The setup is as follows: 3 nodes in AWS east, 3 nodes somewhere on
> Rackspace/Openstack. I use my own snitch based on EC2MultiRegionSnitch
> (it just adds some ec2 avalability zone parsing capabilities). Nodes
> in the cluster connect to each other and all seems ok.
>
> When I start the Rackspace cluster first, populate it with data and
> then let the AWS cluster bootstrap from it, it works great. However
> the other way round it just breaks.
>
> The breakage demonstrates as follows:
>
> - nodetool rebuild us-east command hangs
> - cassandra's log contains the following:
>
> 2013-03-19 12:42:15.796+0100 [Thread-14] [DEBUG]
> IncomingTcpConnection.java(63)
> org.apache.cassandra.net.IncomingTcpConnection: Connection version 6
> from ec2-xxx-xxx-xxx-xxx.compute-1.amazonaws.com/xxx.xxx.xxx.xxx
> 2013-03-19 12:42:15.803+0100 [Thread-14] [DEBUG]
> StreamInSession.java(104)
> org.apache.cassandra.streaming.StreamInSession: Adding file
> /path/to/cassandra/data/key_space/column_family/key_space-column_family-ib-2-Data.db
> to Stream Request queue
> 2013-03-19 12:42:15.803+0100 [Thread-14] [DEBUG]
> StreamInSession.java(104)
> org.apache.cassandra.streaming.StreamInSession: Adding file
> /path/to/cassandra/data/key_space/column_family/key_space-column_family-ib-1-Data.db
> to Stream Request queue
> 2013-03-19 12:42:15.806+0100 [Thread-14] [DEBUG]
> IncomingStreamReader.java(112)
> org.apache.cassandra.streaming.IncomingStreamReader: Receiving stream
> 2013-03-19 12:42:15.807+0100 [Thread-14] [DEBUG]
> IncomingStreamReader.java(113)
> org.apache.cassandra.streaming.IncomingStreamReader: Creating file for
> /path/to/cassandra/data/key_space/column_family/key_space-column_family-tmp-ib-2-Data.db
> with 7808 estimat
> ed keys
> 2013-03-19 12:42:15.808+0100 [Thread-14] [DEBUG]
> ColumnFamilyStore.java(863) org.apache.cassandra.db.ColumnFamilyStore:
> component=key_space Checking for sstables overlapping []
> 2013-03-19 12:42:15.962+0100 [Thread-14] [DEBUG] FileUtils.java(110)
> org.apache.cassandra.io.util.FileUtils: Deleting
> key_space-column_family-tmp-ib-2-Data.db
> 2013-03-19 12:42:15.962+0100 [Thread-14] [DEBUG] FileUtils.java(110)
> org.apache.cassandra.io.util.FileUtils: Deleting
> key_space-column_family-tmp-ib-2-CompressionInfo.db
> 2013-03-19 12:42:15.962+0100 [Thread-14] [DEBUG] FileUtils.java(110)
> org.apache.cassandra.io.util.FileUtils: Deleting
> key_space-column_family-tmp-ib-2-TOC.txt
> 2013-03-19 12:42:15.962+0100 [Thread-14] [DEBUG] FileUtils.java(110)
> org.apache.cassandra.io.util.FileUtils: Deleting
> key_space-column_family-tmp-ib-2-Filter.db
> 2013-03-19 12:42:15.963+0100 [Thread-14] [DEBUG] FileUtils.java(110)
> org.apache.cassandra.io.util.FileUtils: Deleting
> key_space-column_family-tmp-ib-2-Index.db
> 2013-03-19 12:42:15.963+0100 [Thread-14] [DEBUG] SSTable.java(154)
> org.apache.cassandra.io.sstable.SSTable: Deleted
> /path/to/cassandra/data/key_space/column_family/key_space-column_family-tmp-ib-2
> 2013-03-19 12:42:15.963+0100 [Thread-14] [INFO]
> StreamInSession.java(136)
> org.apache.cassandra.streaming.StreamInSession: Streaming of file
> /path/to/cassandra/data/key_space/column_family/key_space-column_family-ib-2-Data.db
> sections=127 progress=81048/2444
> 2013-03-19 12:42:16.059+0100 [Thread-13] [DEBUG]
> IncomingTcpConnection.java(79)
> org.apache.cassandra.net.IncomingTcpConnection: IOException reading
> from socket; closing
> java.io.IOException: FAILED_TO_UNCOMPRESS(5)
> at org.xerial.snappy.SnappyNative.throw_error(SnappyNative.java:78)
> at org.xerial.snappy.SnappyNative.rawUncompress(Native Method)
> at org.xerial.snappy.Snappy.rawUncompress(Snappy.java:391)
> at
> org.apache.cassandra.io.compress.SnappyCompressor.uncompress(SnappyCompressor.java:93)
> at
> org.apache.cassandra.streaming.compress.CompressedInputStream.decompress(CompressedInputStream.java:101)
> at
> org.apache.cassandra.streaming.compress.CompressedInputStream.read(CompressedInputStream.java:79)
> at
> java.io.DataInputStream.readUnsignedShort(DataInputStream.java:337)
> at
> org.apache.cassandra.utils.BytesReadTracker.readUnsignedShort(BytesReadTracker.java:140)
> at
> org.apache.cassandra.utils.ByteBufferUtil.readShortLength(ByteBufferUtil.java:361)
> at
> org.apache.cassandra.utils.ByteBufferUtil.readWithShortLength(ByteBufferUtil.java:371)
> at
> org.apache.cassandra.streaming.IncomingStreamReader.streamIn(IncomingStreamReader.java:160)
> at
> org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:122)
> at
> org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:226)
> at
> org.apache.cassandra.net.IncomingTcpConnection.handleStream(IncomingTcpConnection.java:166)
> at
> org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:66)
> 2013-03-19 12:42:15.971+0100 [Thread-16] [ERROR]
> CassandraDaemon.java(133)
> org.apache.cassandra.service.CassandraDaemon: Exception in thread
> Thread[Thread-16,5,main]
> java.lang.RuntimeException: java.net.SocketException: Socket closed
> at com.google.common.base.Throwables.propagate(Throwables.java:160)
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:32)
> at java.lang.Thread.run(Thread.java:679)
> Caused by: java.net.SocketException: Socket closed
> at java.net.SocketInputStream.socketRead0(Native Method)
> at java.net.SocketInputStream.read(SocketInputStream.java:146)
> at sun.security.ssl.InputRecord.readFully(InputRecord.java:442)
> at sun.security.ssl.InputRecord.readV3Record(InputRecord.java:554)
> at sun.security.ssl.InputRecord.read(InputRecord.java:509)
> at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:850)
> at
> sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:807)
> at sun.security.ssl.AppInputStream.read(AppInputStream.java:94)
> at
> org.apache.cassandra.streaming.compress.CompressedInputStream$Reader.runMayThrow(CompressedInputStream.java:151)
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> ... 1 more
>
> - the SSTable file is not downloaded.
>
> To make it even more confusing, when I tried just one node in AWS and
> one on Rackspace, the rebuild ran well. Also for two nodes. Only with
> three on AWS side the process failed.
>
> It's cassandra 1.2.1 running on
>
> java version "1.6.0_24"
> OpenJDK Runtime Environment (IcedTea6 1.11.8)
> (rhel-1.56.1.11.8.el6_3-x86_64)
> OpenJDK 64-Bit Server VM (build 20.0-b12, mixed mode)
>
> SSL is configured for inter-DC communication.
>
> Any hints?
>
> regards,
>
> Ondrej Cernos
>
>
Re: java.io.IOException: FAILED_TO_UNCOMPRESS(5) exception when running nodetool rebuild
Posted by aaron morton <aa...@thelastpickle.com>.
If you are still on 1.2.1 may be this
https://issues.apache.org/jira/browse/CASSANDRA-5105
Fixed in 1.2.2
If you are on 1.2.3 there is also https://issues.apache.org/jira/browse/CASSANDRA-5381
Cheers
-----------------
Aaron Morton
Freelance Cassandra Consultant
New Zealand
@aaronmorton
http://www.thelastpickle.com
On 26/03/2013, at 5:10 AM, Ondřej Černoš <ce...@gmail.com> wrote:
> Hi all,
>
> I am still unable to move forward with this issue.
>
> - when I switch SSL off in inter-DC communication, nodetool rebuild works well
> - when I switch internode_compression off, I still get
> java.io.IOException: FAILED_TO_UNCOMPRESS exception. Does
> internode_compression: none really switch off the snappy compression
> of the internode communication? The stacktrace - see the previous mail
> - clearly demonstrates some compression is involved
> - I managed to trigger another exception:
>
> java.lang.RuntimeException: javax.net.ssl.SSLException: bad record MAC
> at com.google.common.base.Throwables.propagate(Throwables.java:160)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:32)
> at java.lang.Thread.run(Thread.java:662)
> Caused by: javax.net.ssl.SSLException: bad record MAC
> at com.sun.net.ssl.internal.ssl.Alerts.getSSLException(Alerts.java:190)
> at com.sun.net.ssl.internal.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1649)
> at com.sun.net.ssl.internal.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1607)
> at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:859)
> at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:755)
> at com.sun.net.ssl.internal.ssl.AppInputStream.read(AppInputStream.java:75)
> at org.apache.cassandra.streaming.compress.CompressedInputStream$Reader.runMayThrow(CompressedInputStream.java:151)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> ... 1 more
>
> I managed to trigger this exception only once however.
>
> The fact the transfer works when SSL is off and fails with SSL is
> another strange thing with this issue.
>
> Any ideas or hints?
>
> regards,
>
> Ondrej Cernos
>
> On Tue, Mar 19, 2013 at 5:51 PM, Ondřej Černoš <ce...@gmail.com> wrote:
>> Hi all,
>>
>> I am running into strange error when bootstrapping Cassandra cluster
>> in multiple datacenter setup.
>>
>> The setup is as follows: 3 nodes in AWS east, 3 nodes somewhere on
>> Rackspace/Openstack. I use my own snitch based on EC2MultiRegionSnitch
>> (it just adds some ec2 avalability zone parsing capabilities). Nodes
>> in the cluster connect to each other and all seems ok.
>>
>> When I start the Rackspace cluster first, populate it with data and
>> then let the AWS cluster bootstrap from it, it works great. However
>> the other way round it just breaks.
>>
>> The breakage demonstrates as follows:
>>
>> - nodetool rebuild us-east command hangs
>> - cassandra's log contains the following:
>>
>> 2013-03-19 12:42:15.796+0100 [Thread-14] [DEBUG]
>> IncomingTcpConnection.java(63)
>> org.apache.cassandra.net.IncomingTcpConnection: Connection version 6
>> from ec2-xxx-xxx-xxx-xxx.compute-1.amazonaws.com/xxx.xxx.xxx.xxx
>> 2013-03-19 12:42:15.803+0100 [Thread-14] [DEBUG]
>> StreamInSession.java(104)
>> org.apache.cassandra.streaming.StreamInSession: Adding file
>> /path/to/cassandra/data/key_space/column_family/key_space-column_family-ib-2-Data.db
>> to Stream Request queue
>> 2013-03-19 12:42:15.803+0100 [Thread-14] [DEBUG]
>> StreamInSession.java(104)
>> org.apache.cassandra.streaming.StreamInSession: Adding file
>> /path/to/cassandra/data/key_space/column_family/key_space-column_family-ib-1-Data.db
>> to Stream Request queue
>> 2013-03-19 12:42:15.806+0100 [Thread-14] [DEBUG]
>> IncomingStreamReader.java(112)
>> org.apache.cassandra.streaming.IncomingStreamReader: Receiving stream
>> 2013-03-19 12:42:15.807+0100 [Thread-14] [DEBUG]
>> IncomingStreamReader.java(113)
>> org.apache.cassandra.streaming.IncomingStreamReader: Creating file for
>> /path/to/cassandra/data/key_space/column_family/key_space-column_family-tmp-ib-2-Data.db
>> with 7808 estimat
>> ed keys
>> 2013-03-19 12:42:15.808+0100 [Thread-14] [DEBUG]
>> ColumnFamilyStore.java(863) org.apache.cassandra.db.ColumnFamilyStore:
>> component=key_space Checking for sstables overlapping []
>> 2013-03-19 12:42:15.962+0100 [Thread-14] [DEBUG] FileUtils.java(110)
>> org.apache.cassandra.io.util.FileUtils: Deleting
>> key_space-column_family-tmp-ib-2-Data.db
>> 2013-03-19 12:42:15.962+0100 [Thread-14] [DEBUG] FileUtils.java(110)
>> org.apache.cassandra.io.util.FileUtils: Deleting
>> key_space-column_family-tmp-ib-2-CompressionInfo.db
>> 2013-03-19 12:42:15.962+0100 [Thread-14] [DEBUG] FileUtils.java(110)
>> org.apache.cassandra.io.util.FileUtils: Deleting
>> key_space-column_family-tmp-ib-2-TOC.txt
>> 2013-03-19 12:42:15.962+0100 [Thread-14] [DEBUG] FileUtils.java(110)
>> org.apache.cassandra.io.util.FileUtils: Deleting
>> key_space-column_family-tmp-ib-2-Filter.db
>> 2013-03-19 12:42:15.963+0100 [Thread-14] [DEBUG] FileUtils.java(110)
>> org.apache.cassandra.io.util.FileUtils: Deleting
>> key_space-column_family-tmp-ib-2-Index.db
>> 2013-03-19 12:42:15.963+0100 [Thread-14] [DEBUG] SSTable.java(154)
>> org.apache.cassandra.io.sstable.SSTable: Deleted
>> /path/to/cassandra/data/key_space/column_family/key_space-column_family-tmp-ib-2
>> 2013-03-19 12:42:15.963+0100 [Thread-14] [INFO]
>> StreamInSession.java(136)
>> org.apache.cassandra.streaming.StreamInSession: Streaming of file
>> /path/to/cassandra/data/key_space/column_family/key_space-column_family-ib-2-Data.db
>> sections=127 progress=81048/2444
>> 2013-03-19 12:42:16.059+0100 [Thread-13] [DEBUG]
>> IncomingTcpConnection.java(79)
>> org.apache.cassandra.net.IncomingTcpConnection: IOException reading
>> from socket; closing
>> java.io.IOException: FAILED_TO_UNCOMPRESS(5)
>> at org.xerial.snappy.SnappyNative.throw_error(SnappyNative.java:78)
>> at org.xerial.snappy.SnappyNative.rawUncompress(Native Method)
>> at org.xerial.snappy.Snappy.rawUncompress(Snappy.java:391)
>> at org.apache.cassandra.io.compress.SnappyCompressor.uncompress(SnappyCompressor.java:93)
>> at org.apache.cassandra.streaming.compress.CompressedInputStream.decompress(CompressedInputStream.java:101)
>> at org.apache.cassandra.streaming.compress.CompressedInputStream.read(CompressedInputStream.java:79)
>> at java.io.DataInputStream.readUnsignedShort(DataInputStream.java:337)
>> at org.apache.cassandra.utils.BytesReadTracker.readUnsignedShort(BytesReadTracker.java:140)
>> at org.apache.cassandra.utils.ByteBufferUtil.readShortLength(ByteBufferUtil.java:361)
>> at org.apache.cassandra.utils.ByteBufferUtil.readWithShortLength(ByteBufferUtil.java:371)
>> at org.apache.cassandra.streaming.IncomingStreamReader.streamIn(IncomingStreamReader.java:160)
>> at org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:122)
>> at org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:226)
>> at org.apache.cassandra.net.IncomingTcpConnection.handleStream(IncomingTcpConnection.java:166)
>> at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:66)
>> 2013-03-19 12:42:15.971+0100 [Thread-16] [ERROR]
>> CassandraDaemon.java(133)
>> org.apache.cassandra.service.CassandraDaemon: Exception in thread
>> Thread[Thread-16,5,main]
>> java.lang.RuntimeException: java.net.SocketException: Socket closed
>> at com.google.common.base.Throwables.propagate(Throwables.java:160)
>> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:32)
>> at java.lang.Thread.run(Thread.java:679)
>> Caused by: java.net.SocketException: Socket closed
>> at java.net.SocketInputStream.socketRead0(Native Method)
>> at java.net.SocketInputStream.read(SocketInputStream.java:146)
>> at sun.security.ssl.InputRecord.readFully(InputRecord.java:442)
>> at sun.security.ssl.InputRecord.readV3Record(InputRecord.java:554)
>> at sun.security.ssl.InputRecord.read(InputRecord.java:509)
>> at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:850)
>> at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:807)
>> at sun.security.ssl.AppInputStream.read(AppInputStream.java:94)
>> at org.apache.cassandra.streaming.compress.CompressedInputStream$Reader.runMayThrow(CompressedInputStream.java:151)
>> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
>> ... 1 more
>>
>> - the SSTable file is not downloaded.
>>
>> To make it even more confusing, when I tried just one node in AWS and
>> one on Rackspace, the rebuild ran well. Also for two nodes. Only with
>> three on AWS side the process failed.
>>
>> It's cassandra 1.2.1 running on
>>
>> java version "1.6.0_24"
>> OpenJDK Runtime Environment (IcedTea6 1.11.8) (rhel-1.56.1.11.8.el6_3-x86_64)
>> OpenJDK 64-Bit Server VM (build 20.0-b12, mixed mode)
>>
>> SSL is configured for inter-DC communication.
>>
>> Any hints?
>>
>> regards,
>>
>> Ondrej Cernos
Re: java.io.IOException: FAILED_TO_UNCOMPRESS(5) exception when
running nodetool rebuild
Posted by Ondřej Černoš <ce...@gmail.com>.
Hi all,
I am still unable to move forward with this issue.
- when I switch SSL off in inter-DC communication, nodetool rebuild works well
- when I switch internode_compression off, I still get
java.io.IOException: FAILED_TO_UNCOMPRESS exception. Does
internode_compression: none really switch off the snappy compression
of the internode communication? The stacktrace - see the previous mail
- clearly demonstrates some compression is involved
- I managed to trigger another exception:
java.lang.RuntimeException: javax.net.ssl.SSLException: bad record MAC
at com.google.common.base.Throwables.propagate(Throwables.java:160)
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:32)
at java.lang.Thread.run(Thread.java:662)
Caused by: javax.net.ssl.SSLException: bad record MAC
at com.sun.net.ssl.internal.ssl.Alerts.getSSLException(Alerts.java:190)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1649)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1607)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:859)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:755)
at com.sun.net.ssl.internal.ssl.AppInputStream.read(AppInputStream.java:75)
at org.apache.cassandra.streaming.compress.CompressedInputStream$Reader.runMayThrow(CompressedInputStream.java:151)
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
... 1 more
I managed to trigger this exception only once however.
The fact the transfer works when SSL is off and fails with SSL is
another strange thing with this issue.
Any ideas or hints?
regards,
Ondrej Cernos
On Tue, Mar 19, 2013 at 5:51 PM, Ondřej Černoš <ce...@gmail.com> wrote:
> Hi all,
>
> I am running into strange error when bootstrapping Cassandra cluster
> in multiple datacenter setup.
>
> The setup is as follows: 3 nodes in AWS east, 3 nodes somewhere on
> Rackspace/Openstack. I use my own snitch based on EC2MultiRegionSnitch
> (it just adds some ec2 avalability zone parsing capabilities). Nodes
> in the cluster connect to each other and all seems ok.
>
> When I start the Rackspace cluster first, populate it with data and
> then let the AWS cluster bootstrap from it, it works great. However
> the other way round it just breaks.
>
> The breakage demonstrates as follows:
>
> - nodetool rebuild us-east command hangs
> - cassandra's log contains the following:
>
> 2013-03-19 12:42:15.796+0100 [Thread-14] [DEBUG]
> IncomingTcpConnection.java(63)
> org.apache.cassandra.net.IncomingTcpConnection: Connection version 6
> from ec2-xxx-xxx-xxx-xxx.compute-1.amazonaws.com/xxx.xxx.xxx.xxx
> 2013-03-19 12:42:15.803+0100 [Thread-14] [DEBUG]
> StreamInSession.java(104)
> org.apache.cassandra.streaming.StreamInSession: Adding file
> /path/to/cassandra/data/key_space/column_family/key_space-column_family-ib-2-Data.db
> to Stream Request queue
> 2013-03-19 12:42:15.803+0100 [Thread-14] [DEBUG]
> StreamInSession.java(104)
> org.apache.cassandra.streaming.StreamInSession: Adding file
> /path/to/cassandra/data/key_space/column_family/key_space-column_family-ib-1-Data.db
> to Stream Request queue
> 2013-03-19 12:42:15.806+0100 [Thread-14] [DEBUG]
> IncomingStreamReader.java(112)
> org.apache.cassandra.streaming.IncomingStreamReader: Receiving stream
> 2013-03-19 12:42:15.807+0100 [Thread-14] [DEBUG]
> IncomingStreamReader.java(113)
> org.apache.cassandra.streaming.IncomingStreamReader: Creating file for
> /path/to/cassandra/data/key_space/column_family/key_space-column_family-tmp-ib-2-Data.db
> with 7808 estimat
> ed keys
> 2013-03-19 12:42:15.808+0100 [Thread-14] [DEBUG]
> ColumnFamilyStore.java(863) org.apache.cassandra.db.ColumnFamilyStore:
> component=key_space Checking for sstables overlapping []
> 2013-03-19 12:42:15.962+0100 [Thread-14] [DEBUG] FileUtils.java(110)
> org.apache.cassandra.io.util.FileUtils: Deleting
> key_space-column_family-tmp-ib-2-Data.db
> 2013-03-19 12:42:15.962+0100 [Thread-14] [DEBUG] FileUtils.java(110)
> org.apache.cassandra.io.util.FileUtils: Deleting
> key_space-column_family-tmp-ib-2-CompressionInfo.db
> 2013-03-19 12:42:15.962+0100 [Thread-14] [DEBUG] FileUtils.java(110)
> org.apache.cassandra.io.util.FileUtils: Deleting
> key_space-column_family-tmp-ib-2-TOC.txt
> 2013-03-19 12:42:15.962+0100 [Thread-14] [DEBUG] FileUtils.java(110)
> org.apache.cassandra.io.util.FileUtils: Deleting
> key_space-column_family-tmp-ib-2-Filter.db
> 2013-03-19 12:42:15.963+0100 [Thread-14] [DEBUG] FileUtils.java(110)
> org.apache.cassandra.io.util.FileUtils: Deleting
> key_space-column_family-tmp-ib-2-Index.db
> 2013-03-19 12:42:15.963+0100 [Thread-14] [DEBUG] SSTable.java(154)
> org.apache.cassandra.io.sstable.SSTable: Deleted
> /path/to/cassandra/data/key_space/column_family/key_space-column_family-tmp-ib-2
> 2013-03-19 12:42:15.963+0100 [Thread-14] [INFO]
> StreamInSession.java(136)
> org.apache.cassandra.streaming.StreamInSession: Streaming of file
> /path/to/cassandra/data/key_space/column_family/key_space-column_family-ib-2-Data.db
> sections=127 progress=81048/2444
> 2013-03-19 12:42:16.059+0100 [Thread-13] [DEBUG]
> IncomingTcpConnection.java(79)
> org.apache.cassandra.net.IncomingTcpConnection: IOException reading
> from socket; closing
> java.io.IOException: FAILED_TO_UNCOMPRESS(5)
> at org.xerial.snappy.SnappyNative.throw_error(SnappyNative.java:78)
> at org.xerial.snappy.SnappyNative.rawUncompress(Native Method)
> at org.xerial.snappy.Snappy.rawUncompress(Snappy.java:391)
> at org.apache.cassandra.io.compress.SnappyCompressor.uncompress(SnappyCompressor.java:93)
> at org.apache.cassandra.streaming.compress.CompressedInputStream.decompress(CompressedInputStream.java:101)
> at org.apache.cassandra.streaming.compress.CompressedInputStream.read(CompressedInputStream.java:79)
> at java.io.DataInputStream.readUnsignedShort(DataInputStream.java:337)
> at org.apache.cassandra.utils.BytesReadTracker.readUnsignedShort(BytesReadTracker.java:140)
> at org.apache.cassandra.utils.ByteBufferUtil.readShortLength(ByteBufferUtil.java:361)
> at org.apache.cassandra.utils.ByteBufferUtil.readWithShortLength(ByteBufferUtil.java:371)
> at org.apache.cassandra.streaming.IncomingStreamReader.streamIn(IncomingStreamReader.java:160)
> at org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:122)
> at org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:226)
> at org.apache.cassandra.net.IncomingTcpConnection.handleStream(IncomingTcpConnection.java:166)
> at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:66)
> 2013-03-19 12:42:15.971+0100 [Thread-16] [ERROR]
> CassandraDaemon.java(133)
> org.apache.cassandra.service.CassandraDaemon: Exception in thread
> Thread[Thread-16,5,main]
> java.lang.RuntimeException: java.net.SocketException: Socket closed
> at com.google.common.base.Throwables.propagate(Throwables.java:160)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:32)
> at java.lang.Thread.run(Thread.java:679)
> Caused by: java.net.SocketException: Socket closed
> at java.net.SocketInputStream.socketRead0(Native Method)
> at java.net.SocketInputStream.read(SocketInputStream.java:146)
> at sun.security.ssl.InputRecord.readFully(InputRecord.java:442)
> at sun.security.ssl.InputRecord.readV3Record(InputRecord.java:554)
> at sun.security.ssl.InputRecord.read(InputRecord.java:509)
> at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:850)
> at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:807)
> at sun.security.ssl.AppInputStream.read(AppInputStream.java:94)
> at org.apache.cassandra.streaming.compress.CompressedInputStream$Reader.runMayThrow(CompressedInputStream.java:151)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> ... 1 more
>
> - the SSTable file is not downloaded.
>
> To make it even more confusing, when I tried just one node in AWS and
> one on Rackspace, the rebuild ran well. Also for two nodes. Only with
> three on AWS side the process failed.
>
> It's cassandra 1.2.1 running on
>
> java version "1.6.0_24"
> OpenJDK Runtime Environment (IcedTea6 1.11.8) (rhel-1.56.1.11.8.el6_3-x86_64)
> OpenJDK 64-Bit Server VM (build 20.0-b12, mixed mode)
>
> SSL is configured for inter-DC communication.
>
> Any hints?
>
> regards,
>
> Ondrej Cernos