[jira] [Commented] (CASSANDRA-12245) initial view build can be parallel

["Sylvain Lebresne (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/CASSANDRA-12245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15385657#comment-15385657 ] 

Sylvain Lebresne commented on CASSANDRA-12245:
----------------------------------------------

bq. just iterate through sstables, not worrying about duplicates, and include the timestamp of the original write in the MV mutation.

I could misunderstand this but if you talking about dealing with each sstable completely individually, I don't think we can do that. It's not that we have to merge sstables to avoid dealing with duplicates, it's we need to current full version of a row to create the proper MV entry. For instance, say you have a first insert to a row in one sstable that contains the following columns/values {{c1=2, c2=3, c3='foo'}}, and a later update to that row in another sstable containing {{c1=4}}. And further say that {{c1}} is part of your view PK. If we deal with both sstable individually, in the first case we'll insert an entry for {{c1=2}}, with the other value, while for the 2nd sstable we'll create a *different* entry for {{c1=4}} with no other values. Which is doubly wrong as the {{c1=2}} entry shouldn't exist, and the {{c1=4}} entry should have the other values for {{c2}} and {{c3}}.

With that said, several weeks to build a view over 3TB is indeed not ideal, and we can indeed at least do vnodes in parallel. 

> initial view build can be parallel
> ----------------------------------
>
>                 Key: CASSANDRA-12245
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-12245
>             Project: Cassandra
>          Issue Type: Improvement
>            Reporter: Tom van der Woerdt
>
> On a node with lots of data (~3TB) building a materialized view takes several weeks, which is not ideal. It's doing this in a single thread.
> There are several potential ways this can be optimized :
>  * do vnodes in parallel, instead of going through the entire range in one thread
>  * just iterate through sstables, not worrying about duplicates, and include the timestamp of the original write in the MV mutation. since this doesn't exclude duplicates it does increase the amount of work and could temporarily surface ghost rows (yikes) but I guess that's why they call it eventual consistency. doing it this way can avoid holding references to all tables on disk, allows parallelization, and removes the need to check other sstables for existing data. this is essentially the 'do a full repair' path



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)