Difference between drop and truncate

[changan zuo <zc...@163.com>]

Hi,guys
   We have a very big table. when I excute "truncate table" it takes such a long time at last show "request timeout ". 
   well, if I execute drop table which complete very quickly.
   I cannot see the big difference since they both delete the data.
   
   Anyone can explain it to me ? thanks a lot

Re: Difference between drop and truncate

[zuochangan <zc...@163.com>]

Detailed explanation, thanks 

> 在 2017年12月21日，上午11:49，Jeff Jirsa <jj...@gmail.com> 写道：
> 
> Assume you’re running 3.0 or 3.x - there’s a patch that’ll be in the next releases that speed up truncate significantly - there’s some slowish code in adding the sstables to the transaction log before deleting them, but it’ll be much faster. 
> 
> Truncate marks all the data as removed, and then removes it. It requires all hosts be online. It removes the risk of conflicting CFIDs. It reduces races.
> 
> Dropping the table isn’t guaranteed to hit all hosts at the same time, and recreating it won’t either. It’s far more likely you could have data lost if you’re writing as you drop and recreate vs using truncate. 
> 
> 
> 
> -- 
> Jeff Jirsa
> 
> 
> On Dec 20, 2017, at 7:41 PM, changan zuo <zcauniverse@163.com <ma...@163.com>> wrote:
> 
>> Hi,guys
>>    We have a very big table. when I excute "truncate table" it takes such a long time at last show "request timeout ". 
>>    well, if I execute drop table which complete very quickly.
>>    I cannot see the big difference since they both delete the data.
>>    
>>    Anyone can explain it to me ? thanks a lot
>> 
>> 
>> 
>>  

Re: Difference between drop and truncate

[Jeff Jirsa <jj...@gmail.com>]

Assume you’re running 3.0 or 3.x - there’s a patch that’ll be in the next releases that speed up truncate significantly - there’s some slowish code in adding the sstables to the transaction log before deleting them, but it’ll be much faster. 

Truncate marks all the data as removed, and then removes it. It requires all hosts be online. It removes the risk of conflicting CFIDs. It reduces races.

Dropping the table isn’t guaranteed to hit all hosts at the same time, and recreating it won’t either. It’s far more likely you could have data lost if you’re writing as you drop and recreate vs using truncate. 



-- 
Jeff Jirsa


> On Dec 20, 2017, at 7:41 PM, changan zuo <zc...@163.com> wrote:
> 
> Hi,guys
>    We have a very big table. when I excute "truncate table" it takes such a long time at last show "request timeout ". 
>    well, if I execute drop table which complete very quickly.
>    I cannot see the big difference since they both delete the data.
>    
>    Anyone can explain it to me ? thanks a lot
> 
> 
> 
>  

Re: Why does SASI index consume such a huge disk space?

[Mick Semb Wever <mi...@thelastpickle.com>]

> I use zipkin (https://github.com/openzipkin/zipkin) to trace my system.
>
> When I upgraded to the latest version ,3.23 be specific. I met a problem
which our monitor keep alerting that there is not enough disk space for
cassandra.


You're right. CONTAINS SASI indexes do indeed use a lot of disk space.

If you don't search by annotations in Zipkin feel free to drop that index.

`DROP INDEX zipkin2.span (annotation_query)`

We're looking into a more effective approach with this.
The CONTAINS SASI was used here because SASI on collections are not yet
supported: CASSANDRA-11182
<https://issues.apache.org/jira/browse/CASSANDRA-11182>
An obvious improvement would be to manually denormalise the collection
first.

ref: https://github.com/openzipkin/zipkin/issues/1861#issuecomment-353510778

regards,
Mick

Why does SASI index consume such a huge disk space?

[zuochangan <zc...@163.com>]

Hi,all

I use zipkin (https://github.com/openzipkin/zipkin <https://github.com/openzipkin/zipkin>) to trace my system. 

When I upgraded to the latest version ,3.23 be specific. I met a problem which our monitor keep alerting that there is not enough disk space for cassandra.

After some investigation,I found the biggest file is the index file.  And also I have googled some blogs like (http://www.doanduyhai.com/blog/?p=2058 <http://www.doanduyhai.com/blog/?p=2058>).

Which said as belows:


As we can see, using CONTAINS mode can increase the disk usage by x4 - x6. Since album titles tends to be a long text, the inflation rate is x6. It will be more if we chose the NonTokenizingAnalyzer because the StandardAnalyzer splits the text into tokens, remove stop words and perform stemming. All this help reducing the total size of the term.

As a conclusion, use CONTAINS mode wisely and be ready to pay the price in term of disk space. There is no way to avoid it. Even with efficient search engines like ElasticSearch or Solr, it is officially recommended to avoid substring search (LIKE %substring%) for the sake of performance.

 
Zipkin2 create index as follows  :



CREATE CUSTOM INDEX IF NOT EXISTS ON zipkin2.span (annotation_query) USING 'org.apache.cassandra.index.sasi.SASIIndex'
   WITH OPTIONS = {
    'mode': 'CONTAINS',
    'analyzed': 'true',
    'analyzer_class':'org.apache.cassandra.index.sasi.analyzer.NonTokenizingAnalyzer',
    'case_sensitive': 'false'
   };


I cannot understand why it will use more disk space when we choose NonTokenizingAnalyzer rather than StandardAnalyzer as analyzer_class.

As I debug the code , there is only one term returned when use NonTokenizingAnalyzer



Need Some Help! Thanks a lot