You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Christian Spriegel (Created) (JIRA)" <ji...@apache.org> on 2011/10/17 01:04:11 UTC
[jira] [Created] (CASSANDRA-3370) Deflate Compression corrupts
SSTables
Deflate Compression corrupts SSTables
-------------------------------------
Key: CASSANDRA-3370
URL: https://issues.apache.org/jira/browse/CASSANDRA-3370
Project: Cassandra
Issue Type: Bug
Components: Core
Affects Versions: 1.0.0
Environment: Ubuntu Linux, amd64, Cassandra 1.0.0-rc2
Reporter: Christian Spriegel
Hi,
it seems that the Deflate Compressor corrupts the SSTables. 3 out of 3 Installations were corrupt. Snappy works fine.
Here is what I did:
1. Start a single cassandra node (I was using ByteOrderedPartitioner)
2. Write data into cf that uses deflate compression - I think it has to be enough data so that the data folder contains some files.
3. When I now try to read (I did a range scan) from my application, it fails and the logs show corruptions:
Caused by: org.apache.cassandra.io.compress.CorruptedBlockException: (/home/cspriegel/Development/cassandra1/data/Test/Response-h-2-Data.db): corruption detected, chunk at 0 of length 65536.
regards,
Christian
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3370) Deflate Compression corrupts
SSTables
Posted by "Christian Spriegel (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Christian Spriegel updated CASSANDRA-3370:
------------------------------------------
Attachment: Test.zip
Attached my client that causes the crash.
> Deflate Compression corrupts SSTables
> -------------------------------------
>
> Key: CASSANDRA-3370
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3370
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.0.0
> Environment: Ubuntu Linux, amd64, Cassandra 1.0.0-rc2
> Reporter: Christian Spriegel
> Assignee: Sylvain Lebresne
> Attachments: Test.zip, system.log
>
>
> Hi,
> it seems that the Deflate Compressor corrupts the SSTables. 3 out of 3 Installations were corrupt. Snappy works fine.
> Here is what I did:
> 1. Start a single cassandra node (I was using ByteOrderedPartitioner)
> 2. Write data into cf that uses deflate compression - I think it has to be enough data so that the data folder contains some files.
> 3. When I now try to read (I did a range scan) from my application, it fails and the logs show corruptions:
> Caused by: org.apache.cassandra.io.compress.CorruptedBlockException: (/home/cspriegel/Development/cassandra1/data/Test/Response-h-2-Data.db): corruption detected, chunk at 0 of length 65536.
> regards,
> Christian
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3370) Deflate Compression corrupts
SSTables
Posted by "Christian Spriegel (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13130962#comment-13130962 ]
Christian Spriegel commented on CASSANDRA-3370:
-----------------------------------------------
I tried using stress from 1.0.0 and I got the same results as you. Stress for some reason works fine.
One thing is strange about stress:
I let stress run for quite some time, but there is only 12 MB in the datafolder.
I let my tool run for 10 seconds, but there are 97MB in the data folder.
Is stress maybe not generating random data, so that it compresses really well? Might that be the difference?
Can I maybe share my application with you? Its a single source file with a pom.xml.
If you have any idea what I can do, please let me know.
> Deflate Compression corrupts SSTables
> -------------------------------------
>
> Key: CASSANDRA-3370
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3370
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.0.0
> Environment: Ubuntu Linux, amd64, Cassandra 1.0.0-rc2
> Reporter: Christian Spriegel
> Assignee: Sylvain Lebresne
> Attachments: system.log
>
>
> Hi,
> it seems that the Deflate Compressor corrupts the SSTables. 3 out of 3 Installations were corrupt. Snappy works fine.
> Here is what I did:
> 1. Start a single cassandra node (I was using ByteOrderedPartitioner)
> 2. Write data into cf that uses deflate compression - I think it has to be enough data so that the data folder contains some files.
> 3. When I now try to read (I did a range scan) from my application, it fails and the logs show corruptions:
> Caused by: org.apache.cassandra.io.compress.CorruptedBlockException: (/home/cspriegel/Development/cassandra1/data/Test/Response-h-2-Data.db): corruption detected, chunk at 0 of length 65536.
> regards,
> Christian
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3370) Deflate Compression corrupts
SSTables
Posted by "Jonathan Ellis (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-3370:
--------------------------------------
Reviewer: xedin
> Deflate Compression corrupts SSTables
> -------------------------------------
>
> Key: CASSANDRA-3370
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3370
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.0.0
> Environment: Ubuntu Linux, amd64, Cassandra 1.0.0-rc2
> Reporter: Christian Spriegel
> Assignee: Sylvain Lebresne
> Labels: compression
> Fix For: 1.0.1
>
> Attachments: 3370.patch, Test.zip, system.log
>
>
> Hi,
> it seems that the Deflate Compressor corrupts the SSTables. 3 out of 3 Installations were corrupt. Snappy works fine.
> Here is what I did:
> 1. Start a single cassandra node (I was using ByteOrderedPartitioner)
> 2. Write data into cf that uses deflate compression - I think it has to be enough data so that the data folder contains some files.
> 3. When I now try to read (I did a range scan) from my application, it fails and the logs show corruptions:
> Caused by: org.apache.cassandra.io.compress.CorruptedBlockException: (/home/cspriegel/Development/cassandra1/data/Test/Response-h-2-Data.db): corruption detected, chunk at 0 of length 65536.
> regards,
> Christian
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (CASSANDRA-3370) Deflate Compression corrupts
SSTables
Posted by "Pavel Yaskevich (Assigned) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Pavel Yaskevich reassigned CASSANDRA-3370:
------------------------------------------
Assignee: Sylvain Lebresne (was: Pavel Yaskevich)
> Deflate Compression corrupts SSTables
> -------------------------------------
>
> Key: CASSANDRA-3370
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3370
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.0.0
> Environment: Ubuntu Linux, amd64, Cassandra 1.0.0-rc2
> Reporter: Christian Spriegel
> Assignee: Sylvain Lebresne
> Attachments: system.log
>
>
> Hi,
> it seems that the Deflate Compressor corrupts the SSTables. 3 out of 3 Installations were corrupt. Snappy works fine.
> Here is what I did:
> 1. Start a single cassandra node (I was using ByteOrderedPartitioner)
> 2. Write data into cf that uses deflate compression - I think it has to be enough data so that the data folder contains some files.
> 3. When I now try to read (I did a range scan) from my application, it fails and the logs show corruptions:
> Caused by: org.apache.cassandra.io.compress.CorruptedBlockException: (/home/cspriegel/Development/cassandra1/data/Test/Response-h-2-Data.db): corruption detected, chunk at 0 of length 65536.
> regards,
> Christian
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3370) Deflate Compression corrupts
SSTables
Posted by "Christian Spriegel (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Christian Spriegel updated CASSANDRA-3370:
------------------------------------------
Attachment: system.log
Attached system.log of broken installation
> Deflate Compression corrupts SSTables
> -------------------------------------
>
> Key: CASSANDRA-3370
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3370
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.0.0
> Environment: Ubuntu Linux, amd64, Cassandra 1.0.0-rc2
> Reporter: Christian Spriegel
> Attachments: system.log
>
>
> Hi,
> it seems that the Deflate Compressor corrupts the SSTables. 3 out of 3 Installations were corrupt. Snappy works fine.
> Here is what I did:
> 1. Start a single cassandra node (I was using ByteOrderedPartitioner)
> 2. Write data into cf that uses deflate compression - I think it has to be enough data so that the data folder contains some files.
> 3. When I now try to read (I did a range scan) from my application, it fails and the logs show corruptions:
> Caused by: org.apache.cassandra.io.compress.CorruptedBlockException: (/home/cspriegel/Development/cassandra1/data/Test/Response-h-2-Data.db): corruption detected, chunk at 0 of length 65536.
> regards,
> Christian
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3370) Deflate Compression corrupts
SSTables
Posted by "Sylvain Lebresne (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13129788#comment-13129788 ]
Sylvain Lebresne commented on CASSANDRA-3370:
---------------------------------------------
The exception is raised because there is a digest mismatch for the initial block of one of the sstable.
Haven't been able to reproduce so far using stress with deflate with the default 1M keys (which create a bunch of sstables, at least on my machine) using row slices and range scans (using both Random and ByteOrdered partitioners).
Would you be able to 1) try with 1.0.0 and 2) try with the stress tool that comes with Cassandra (it's in tools/stress of the source distribution, and you'll want to insert values with 'stress -I DeflateCompressor' and read with 'stress -I DeflateCompressor -o RANGE_SLICE' ) and see if you can reproduce? Another question is, did you used openJDK or Sun JDK?
> Deflate Compression corrupts SSTables
> -------------------------------------
>
> Key: CASSANDRA-3370
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3370
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.0.0
> Environment: Ubuntu Linux, amd64, Cassandra 1.0.0-rc2
> Reporter: Christian Spriegel
> Assignee: Sylvain Lebresne
> Attachments: system.log
>
>
> Hi,
> it seems that the Deflate Compressor corrupts the SSTables. 3 out of 3 Installations were corrupt. Snappy works fine.
> Here is what I did:
> 1. Start a single cassandra node (I was using ByteOrderedPartitioner)
> 2. Write data into cf that uses deflate compression - I think it has to be enough data so that the data folder contains some files.
> 3. When I now try to read (I did a range scan) from my application, it fails and the logs show corruptions:
> Caused by: org.apache.cassandra.io.compress.CorruptedBlockException: (/home/cspriegel/Development/cassandra1/data/Test/Response-h-2-Data.db): corruption detected, chunk at 0 of length 65536.
> regards,
> Christian
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3370) Deflate Compression corrupts
SSTables
Posted by "Sylvain Lebresne (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sylvain Lebresne updated CASSANDRA-3370:
----------------------------------------
Attachment: 3370.patch
You test did help. Turns out that's because you're inserting random and thus basically uncompressible data, and the compressed data was bigger than the uncompressed one. The code is supposed to handle that but there is a bug in that part.
Patch attached to fix.
> Deflate Compression corrupts SSTables
> -------------------------------------
>
> Key: CASSANDRA-3370
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3370
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.0.0
> Environment: Ubuntu Linux, amd64, Cassandra 1.0.0-rc2
> Reporter: Christian Spriegel
> Assignee: Sylvain Lebresne
> Attachments: 3370.patch, Test.zip, system.log
>
>
> Hi,
> it seems that the Deflate Compressor corrupts the SSTables. 3 out of 3 Installations were corrupt. Snappy works fine.
> Here is what I did:
> 1. Start a single cassandra node (I was using ByteOrderedPartitioner)
> 2. Write data into cf that uses deflate compression - I think it has to be enough data so that the data folder contains some files.
> 3. When I now try to read (I did a range scan) from my application, it fails and the logs show corruptions:
> Caused by: org.apache.cassandra.io.compress.CorruptedBlockException: (/home/cspriegel/Development/cassandra1/data/Test/Response-h-2-Data.db): corruption detected, chunk at 0 of length 65536.
> regards,
> Christian
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3370) Deflate Compression corrupts
SSTables
Posted by "Christian Spriegel (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13130001#comment-13130001 ]
Christian Spriegel commented on CASSANDRA-3370:
-----------------------------------------------
I tested again with 1.0.0. Unfortunetaly the problem still exists.
But I think I was able to narrow it down: It seems that the problem only occurs when I insert large byte-arrays.
It seems to work fine with 10kb arrays, no problem there. I was able to repeatedly insert and read.
With 100kb or 200kb arrays it crashes after about 1000-2000 insertions. (Insertions work, but range scan afterwards crashes)
> Deflate Compression corrupts SSTables
> -------------------------------------
>
> Key: CASSANDRA-3370
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3370
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.0.0
> Environment: Ubuntu Linux, amd64, Cassandra 1.0.0-rc2
> Reporter: Christian Spriegel
> Assignee: Sylvain Lebresne
> Attachments: system.log
>
>
> Hi,
> it seems that the Deflate Compressor corrupts the SSTables. 3 out of 3 Installations were corrupt. Snappy works fine.
> Here is what I did:
> 1. Start a single cassandra node (I was using ByteOrderedPartitioner)
> 2. Write data into cf that uses deflate compression - I think it has to be enough data so that the data folder contains some files.
> 3. When I now try to read (I did a range scan) from my application, it fails and the logs show corruptions:
> Caused by: org.apache.cassandra.io.compress.CorruptedBlockException: (/home/cspriegel/Development/cassandra1/data/Test/Response-h-2-Data.db): corruption detected, chunk at 0 of length 65536.
> regards,
> Christian
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3370) Deflate Compression corrupts
SSTables
Posted by "Christian Spriegel (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13130007#comment-13130007 ]
Christian Spriegel commented on CASSANDRA-3370:
-----------------------------------------------
btw: I just tested the DeflaterOutputStream/DeflaterInputStream classes in a small testcase and there it works fine. I thought maybe Deflate in my jvm is broken.
> Deflate Compression corrupts SSTables
> -------------------------------------
>
> Key: CASSANDRA-3370
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3370
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.0.0
> Environment: Ubuntu Linux, amd64, Cassandra 1.0.0-rc2
> Reporter: Christian Spriegel
> Assignee: Sylvain Lebresne
> Attachments: system.log
>
>
> Hi,
> it seems that the Deflate Compressor corrupts the SSTables. 3 out of 3 Installations were corrupt. Snappy works fine.
> Here is what I did:
> 1. Start a single cassandra node (I was using ByteOrderedPartitioner)
> 2. Write data into cf that uses deflate compression - I think it has to be enough data so that the data folder contains some files.
> 3. When I now try to read (I did a range scan) from my application, it fails and the logs show corruptions:
> Caused by: org.apache.cassandra.io.compress.CorruptedBlockException: (/home/cspriegel/Development/cassandra1/data/Test/Response-h-2-Data.db): corruption detected, chunk at 0 of length 65536.
> regards,
> Christian
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3370) Deflate Compression corrupts
SSTables
Posted by "Sylvain Lebresne (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13130553#comment-13130553 ]
Sylvain Lebresne commented on CASSANDRA-3370:
---------------------------------------------
Still cannot reproduce. I've tried multiple times inserting 5000 keys using the stress tool using values of 10KB, 20KB, 100KB and 200KB (using 'stress -I DeflateCompressor -S 200000 -n 5000'). I then try to reading both with 'stress -o RANGE_SLICE -n 5000' and by simply fetching the 100 first keys using the CLI (with a simple 'list Standard1;') and got no exceptions (the actual listing in the CLI took a while to be printed on screen because the columns are big but outside of that, no errors).
Would you mind trying the same experiment (with the same tools) or providing the test script you're using so we can check if it has to do with the specific insertions or with something in your environment.
> Deflate Compression corrupts SSTables
> -------------------------------------
>
> Key: CASSANDRA-3370
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3370
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.0.0
> Environment: Ubuntu Linux, amd64, Cassandra 1.0.0-rc2
> Reporter: Christian Spriegel
> Assignee: Sylvain Lebresne
> Attachments: system.log
>
>
> Hi,
> it seems that the Deflate Compressor corrupts the SSTables. 3 out of 3 Installations were corrupt. Snappy works fine.
> Here is what I did:
> 1. Start a single cassandra node (I was using ByteOrderedPartitioner)
> 2. Write data into cf that uses deflate compression - I think it has to be enough data so that the data folder contains some files.
> 3. When I now try to read (I did a range scan) from my application, it fails and the logs show corruptions:
> Caused by: org.apache.cassandra.io.compress.CorruptedBlockException: (/home/cspriegel/Development/cassandra1/data/Test/Response-h-2-Data.db): corruption detected, chunk at 0 of length 65536.
> regards,
> Christian
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3370) Deflate Compression corrupts
SSTables
Posted by "Christian Spriegel (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13131519#comment-13131519 ]
Christian Spriegel commented on CASSANDRA-3370:
-----------------------------------------------
Great! This would also apply if some app would insert already compressed data.
> Deflate Compression corrupts SSTables
> -------------------------------------
>
> Key: CASSANDRA-3370
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3370
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.0.0
> Environment: Ubuntu Linux, amd64, Cassandra 1.0.0-rc2
> Reporter: Christian Spriegel
> Assignee: Sylvain Lebresne
> Labels: compression
> Fix For: 1.0.1
>
> Attachments: 3370.patch, Test.zip, system.log
>
>
> Hi,
> it seems that the Deflate Compressor corrupts the SSTables. 3 out of 3 Installations were corrupt. Snappy works fine.
> Here is what I did:
> 1. Start a single cassandra node (I was using ByteOrderedPartitioner)
> 2. Write data into cf that uses deflate compression - I think it has to be enough data so that the data folder contains some files.
> 3. When I now try to read (I did a range scan) from my application, it fails and the logs show corruptions:
> Caused by: org.apache.cassandra.io.compress.CorruptedBlockException: (/home/cspriegel/Development/cassandra1/data/Test/Response-h-2-Data.db): corruption detected, chunk at 0 of length 65536.
> regards,
> Christian
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3370) Deflate Compression corrupts
SSTables
Posted by "Christian Spriegel (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13130695#comment-13130695 ]
Christian Spriegel commented on CASSANDRA-3370:
-----------------------------------------------
I see! I got the wrong version. Sorry, I did not know that it was included in the cassandra source. I thought I had to download it some place else.
I will try with that and let you know about the results...
> Deflate Compression corrupts SSTables
> -------------------------------------
>
> Key: CASSANDRA-3370
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3370
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.0.0
> Environment: Ubuntu Linux, amd64, Cassandra 1.0.0-rc2
> Reporter: Christian Spriegel
> Assignee: Sylvain Lebresne
> Attachments: system.log
>
>
> Hi,
> it seems that the Deflate Compressor corrupts the SSTables. 3 out of 3 Installations were corrupt. Snappy works fine.
> Here is what I did:
> 1. Start a single cassandra node (I was using ByteOrderedPartitioner)
> 2. Write data into cf that uses deflate compression - I think it has to be enough data so that the data folder contains some files.
> 3. When I now try to read (I did a range scan) from my application, it fails and the logs show corruptions:
> Caused by: org.apache.cassandra.io.compress.CorruptedBlockException: (/home/cspriegel/Development/cassandra1/data/Test/Response-h-2-Data.db): corruption detected, chunk at 0 of length 65536.
> regards,
> Christian
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (CASSANDRA-3370) Deflate Compression corrupts
SSTables
Posted by "Brandon Williams (Assigned) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Brandon Williams reassigned CASSANDRA-3370:
-------------------------------------------
Assignee: Pavel Yaskevich
> Deflate Compression corrupts SSTables
> -------------------------------------
>
> Key: CASSANDRA-3370
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3370
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.0.0
> Environment: Ubuntu Linux, amd64, Cassandra 1.0.0-rc2
> Reporter: Christian Spriegel
> Assignee: Pavel Yaskevich
> Attachments: system.log
>
>
> Hi,
> it seems that the Deflate Compressor corrupts the SSTables. 3 out of 3 Installations were corrupt. Snappy works fine.
> Here is what I did:
> 1. Start a single cassandra node (I was using ByteOrderedPartitioner)
> 2. Write data into cf that uses deflate compression - I think it has to be enough data so that the data folder contains some files.
> 3. When I now try to read (I did a range scan) from my application, it fails and the logs show corruptions:
> Caused by: org.apache.cassandra.io.compress.CorruptedBlockException: (/home/cspriegel/Development/cassandra1/data/Test/Response-h-2-Data.db): corruption detected, chunk at 0 of length 65536.
> regards,
> Christian
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3370) Deflate Compression corrupts
SSTables
Posted by "Sylvain Lebresne (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13130691#comment-13130691 ]
Sylvain Lebresne commented on CASSANDRA-3370:
---------------------------------------------
No, I'm talking of the java one. The python one is old and won't support compression for instance. The java one in the 1.0.0 source does support compression through the -I parameter. Are you sure you're looking at the right version ?
> Deflate Compression corrupts SSTables
> -------------------------------------
>
> Key: CASSANDRA-3370
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3370
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.0.0
> Environment: Ubuntu Linux, amd64, Cassandra 1.0.0-rc2
> Reporter: Christian Spriegel
> Assignee: Sylvain Lebresne
> Attachments: system.log
>
>
> Hi,
> it seems that the Deflate Compressor corrupts the SSTables. 3 out of 3 Installations were corrupt. Snappy works fine.
> Here is what I did:
> 1. Start a single cassandra node (I was using ByteOrderedPartitioner)
> 2. Write data into cf that uses deflate compression - I think it has to be enough data so that the data folder contains some files.
> 3. When I now try to read (I did a range scan) from my application, it fails and the logs show corruptions:
> Caused by: org.apache.cassandra.io.compress.CorruptedBlockException: (/home/cspriegel/Development/cassandra1/data/Test/Response-h-2-Data.db): corruption detected, chunk at 0 of length 65536.
> regards,
> Christian
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3370) Deflate Compression corrupts
SSTables
Posted by "Christian Spriegel (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13130685#comment-13130685 ]
Christian Spriegel commented on CASSANDRA-3370:
-----------------------------------------------
Ok, I will try the stress tool.
Just to be sure: You are talking about stress.py and not the java-based stress? Because I was trying the java stress and it did not accept the -I parameter.
> Deflate Compression corrupts SSTables
> -------------------------------------
>
> Key: CASSANDRA-3370
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3370
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.0.0
> Environment: Ubuntu Linux, amd64, Cassandra 1.0.0-rc2
> Reporter: Christian Spriegel
> Assignee: Sylvain Lebresne
> Attachments: system.log
>
>
> Hi,
> it seems that the Deflate Compressor corrupts the SSTables. 3 out of 3 Installations were corrupt. Snappy works fine.
> Here is what I did:
> 1. Start a single cassandra node (I was using ByteOrderedPartitioner)
> 2. Write data into cf that uses deflate compression - I think it has to be enough data so that the data folder contains some files.
> 3. When I now try to read (I did a range scan) from my application, it fails and the logs show corruptions:
> Caused by: org.apache.cassandra.io.compress.CorruptedBlockException: (/home/cspriegel/Development/cassandra1/data/Test/Response-h-2-Data.db): corruption detected, chunk at 0 of length 65536.
> regards,
> Christian
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3370) Deflate Compression corrupts
SSTables
Posted by "Christian Spriegel (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13129794#comment-13129794 ]
Christian Spriegel commented on CASSANDRA-3370:
-----------------------------------------------
My Java version is:
java version "1.6.0_26"
Java(TM) SE Runtime Environment (build 1.6.0_26-b03)
Java HotSpot(TM) 64-Bit Server VM (build 20.1-b02, mixed mode)
Cassandra report during startup:
INFO 17:20:39,113 JVM vendor/version: Java HotSpot(TM) 64-Bit Server VM/1.6.0_26
I will test tonight ...
> Deflate Compression corrupts SSTables
> -------------------------------------
>
> Key: CASSANDRA-3370
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3370
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.0.0
> Environment: Ubuntu Linux, amd64, Cassandra 1.0.0-rc2
> Reporter: Christian Spriegel
> Assignee: Sylvain Lebresne
> Attachments: system.log
>
>
> Hi,
> it seems that the Deflate Compressor corrupts the SSTables. 3 out of 3 Installations were corrupt. Snappy works fine.
> Here is what I did:
> 1. Start a single cassandra node (I was using ByteOrderedPartitioner)
> 2. Write data into cf that uses deflate compression - I think it has to be enough data so that the data folder contains some files.
> 3. When I now try to read (I did a range scan) from my application, it fails and the logs show corruptions:
> Caused by: org.apache.cassandra.io.compress.CorruptedBlockException: (/home/cspriegel/Development/cassandra1/data/Test/Response-h-2-Data.db): corruption detected, chunk at 0 of length 65536.
> regards,
> Christian
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3370) Deflate Compression corrupts
SSTables
Posted by "Pavel Yaskevich (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13131603#comment-13131603 ]
Pavel Yaskevich commented on CASSANDRA-3370:
--------------------------------------------
+1
> Deflate Compression corrupts SSTables
> -------------------------------------
>
> Key: CASSANDRA-3370
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3370
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.0.0
> Environment: Ubuntu Linux, amd64, Cassandra 1.0.0-rc2
> Reporter: Christian Spriegel
> Assignee: Sylvain Lebresne
> Labels: compression
> Fix For: 1.0.1
>
> Attachments: 3370.patch, Test.zip, system.log
>
>
> Hi,
> it seems that the Deflate Compressor corrupts the SSTables. 3 out of 3 Installations were corrupt. Snappy works fine.
> Here is what I did:
> 1. Start a single cassandra node (I was using ByteOrderedPartitioner)
> 2. Write data into cf that uses deflate compression - I think it has to be enough data so that the data folder contains some files.
> 3. When I now try to read (I did a range scan) from my application, it fails and the logs show corruptions:
> Caused by: org.apache.cassandra.io.compress.CorruptedBlockException: (/home/cspriegel/Development/cassandra1/data/Test/Response-h-2-Data.db): corruption detected, chunk at 0 of length 65536.
> regards,
> Christian
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira