[jira] [Commented] (CASSANDRA-7779) Add option to sstableloader to only stream to the local dc

["Thom Valley (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/CASSANDRA-7779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15216190#comment-15216190 ] 

Thom Valley commented on CASSANDRA-7779:
----------------------------------------

This grows to an even bigger scale when you move up to 5 DCs.

SSTABLELOADER is very inefficient when trying to load multiple DCs.  Most global multi-DC implementations are bandwidth sensitive and streaming RF copies of the data to each target DC is very expensive / can be impacted by bandwidth limitations.

Being able to LOAD data to a single DC and have Cassandra do the replication to the additional DCs would be much more efficient, as Cassandra does a great job of limiting resource consumption.  

I realize that's not really SSTABLELOADER as it exists today, but didn't want to file yet another ticket for something so closely related.

> Add option to sstableloader to only stream to the local dc
> ----------------------------------------------------------
>
>                 Key: CASSANDRA-7779
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-7779
>             Project: Cassandra
>          Issue Type: Improvement
>          Components: Tools
>            Reporter: Nick Bailey
>             Fix For: 2.1.x
>
>
> This is meant to be a potential workaround for CASSANDRA-4756. Due to that ticket, trying to load a cluster wide snapshot via sstableloader will potentially stream an enormous amount of data. In a 3 datacenter cluster with rf=3 in each datacenter, 81 copies of the data would be streamed. Once we have per range sstables we can optimize sstableloader to merge data and only stream one copy, but until then we need a workaround. By only streaming to the local datacenter we can load the data locally in each datacenter and only have 9 copies of the data rather than 81.
> This could potentially be achieved by the option to ignore certain nodes that already exists in sstableloader, but in the case of vnodes and topology changes in the cluster, this could require specifying every node in the cluster as 'ignored' on the command line which could be problematic. This is just a shortcut to avoid that.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)