[jira] [Comment Edited] (CASSANDRA-10990) Support streaming of older version sstables in 3.0

["Paulo Motta (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/CASSANDRA-10990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15181464#comment-15181464 ] 

Paulo Motta edited comment on CASSANDRA-10990 at 3/5/16 2:52 AM:
-----------------------------------------------------------------

Updated branch with reworked {{RewindableDataInputStreamPlus}}, now treating the spill file as circular buffer with max size. Now required space for spill buffer file of legacy {{StreamDeserializer}} is {{max(sstableSize, MAX_INT)}}, capping the max required space (and max partition size) for streaming legacy/compact sstables on ~2GB. I also updated documentation and tests. WDYT [~yukim]?

Besides that, I added a new [dtest|https://github.com/pauloricardomg/cassandra-dtest/commit/b59ea329a5b9302372c57956f7015111fd47a4ae] for testing repair of older version sstables on 3.0+, and it seems to work. Now we have the following dtests:
* [bootstrap dtests|https://github.com/pauloricardomg/cassandra-dtest/blob/10990/upgrade_8099_test.py#L340]
* [sstable loader dtests|https://github.com/pauloricardomg/cassandra-dtest/blob/10990/sstable_generation_loading_test.py#L181]
* [repair dtests|https://github.com/pauloricardomg/cassandra-dtest/commit/b59ea329a5b9302372c57956f7015111fd47a4ae]

For simplicity, I opted for writing the old streamed sstables in the new format without {{EncodingStats}} (since there is no {{SerializationHeader}} available), what may make these sstables less optimized in terms of storage space from what I understood. Do you think we should construct these stats when receiving the sstable or not bother [~slebresne] ? Also, do you recall of any other edge case we should watch here?

Thanks!


was (Author: pauloricardomg):
Updated branch with reworked {{RewindableDataInputStreamPlus}}, now treating the spill file as circular buffer with max size. Now required space for spill buffer file of legacy {{StreamDeserializer}} is {{max(sstableSize, MAX_INT)}}, capping the max required space for streaming older version sstables on ~2GB. I also updated documentation and tests. WDYT [~yukim]?

Besides that, I added a new [dtest|https://github.com/pauloricardomg/cassandra-dtest/commit/b59ea329a5b9302372c57956f7015111fd47a4ae] for testing repair of older version sstables on 3.0+, and it seems to work. Now we have the following dtests:
* [bootstrap dtests|https://github.com/pauloricardomg/cassandra-dtest/blob/10990/upgrade_8099_test.py#L340]
* [sstable loader dtests|https://github.com/pauloricardomg/cassandra-dtest/blob/10990/sstable_generation_loading_test.py#L181]
* [repair dtests|https://github.com/pauloricardomg/cassandra-dtest/commit/b59ea329a5b9302372c57956f7015111fd47a4ae]

For simplicity, I opted for writing the old streamed sstables in the new format without {{EncodingStats}} (since there is no {{SerializationHeader}} available), what may make these sstables less optimized in terms of storage space from what I understood. Do you think we should construct these stats when receiving the sstable or not bother [~slebresne] ? Also, do you recall of any other edge case we should watch here?

Thanks!

> Support streaming of older version sstables in 3.0
> --------------------------------------------------
>
>                 Key: CASSANDRA-10990
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-10990
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Streaming and Messaging
>            Reporter: Jeremy Hanna
>            Assignee: Paulo Motta
>
> In 2.0 we introduced support for streaming older versioned sstables (CASSANDRA-5772).  In 3.0, because of the rewrite of the storage layer, this became no longer supported.  So currently, while 3.0 can read sstables in the 2.1/2.2 format, it cannot stream the older versioned sstables.  We should do some work to make this still possible to be consistent with what CASSANDRA-5772 provided.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)