[jira] [Commented] (CASSANDRA-4297) Use java NIO as much as possible when streaming compressed SSTables

["Sylvain Lebresne (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/CASSANDRA-4297?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13287370#comment-13287370 ] 

Sylvain Lebresne commented on CASSANDRA-4297:
---------------------------------------------

bq. It looks like the only reason to decompress is to compare crc32... is that right?

No. We decompress because we need to build secondary indexes, compute the sstable stats, clean counters delta, etc...
                
> Use java NIO as much as possible when streaming compressed SSTables
> -------------------------------------------------------------------
>
>                 Key: CASSANDRA-4297
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-4297
>             Project: Cassandra
>          Issue Type: Improvement
>            Reporter: Yuki Morishita
>            Assignee: Yuki Morishita
>            Priority: Minor
>              Labels: streaming
>             Fix For: 1.2
>
>
> Back in 0.8, streaming uses java NIO (FileChannel#transferTo/transferFrom) to perform zero copy file transfer between nodes. Since 1.0, in order to add new features like sstable compression and internode encryption we had to switch to java IO Input/OutputStreams. What we currently do to transfer compressed SSTable is, in source node, 1) decompress chunk in SSTable, 2) compress using LZF for network, and in destination node, 3) decompress using LZF as reading from socket, 4) compress for SSTable on disk.
> Now, 1.1 comes out with SSTable compression turned on by default. It is reasonable to transfer compressed file as is using NIO instead of decompress/compress in source node.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira