Re: Challenge with initial data load with TWCS

[DuyHai Doan <do...@gmail.com>]

Thanks Alex for confirming

Le 30 sept. 2019 09:17, "Oleksandr Shulgin" <ol...@zalando.de>
a écrit :

> On Sun, Sep 29, 2019 at 9:42 AM DuyHai Doan <do...@gmail.com> wrote:
>
>> Thanks Jeff for sharing the ideas. I have some question though:
>>
>> - CQLSSTableWriter and explicitly break between windows --> Even if
>> you break between windows, If we have data worth of 1 years it would
>> requires us to use CQLSSTableWriter during 1 year (365 days) because
>> the write time taken into account when flushing to SSTable is the
>> current clock timestamp :
>>
>> https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/io/sstable/CQLSSTableWriter.java#L252-L259
>>
>
> Well, it's not obvious from that block of code alone if one cannot
> override the write time.  But if that's the case, maybe you want to
> subclass it and extend for that possibility.
>
> What we're looking for is a way to load 1 year of data and forcing
>> write timestamp to the past so that the initial loading operation is
>> seen by TWCS as if we have loaded the data normally day by day during
>> 1 year
>>
>> - Use the normal write path for a single window at a time, explicitly
>> calling flush between windows. --> I don't understand how calling
>> flush will trigger windowing in TWCS, as far as I know, it is based on
>> write time. And by the way, can we load data using normal CQL and just
>> forcing the write time to be in the past so that TWCS will trigger
>> compaction properly ?
>>
>
> Exactly.  If you go for this option, you should specify the write time
> explicitly by adding `USING TIMESTAMP :xxx` to your CQL statement.
>
> --
> Alex
>
>