secondary index creation causes C* oom

[Peng Xiao <25...@qq.com>]

Dear All,


We met some C* nodes oom during secondary index creation with C* 2.1.18.
As per https://issues.apache.org/jira/browse/CASSANDRA-12796,the flush writer will be blocked by index rebuild.but we still have some confusions:
1.not sure if secondary index creation is the same as index rebuild
2.we noticed that the memory table flush looks still working,not the same as CASSANDRA-12796 mentioned,but the compactionExecutor pending is increasing.
3.I'm wondering if the block only blocks the specified table which is creating secondary index?


Could anyone please explain?





Thanks

回复： secondary index creation causes C* oom

[Peng Xiao <25...@qq.com>]

Thanks Kurt.




------------------ 原始邮件 ------------------
发件人: "kurt";<ku...@instaclustr.com>;
发送时间: 2018年1月11日(星期四) 中午11:46
收件人: "User"<us...@cassandra.apache.org>;

主题: Re: secondary index creation causes C* oom




1.not sure if secondary index creation is the same as index rebuild

Fairly sure they are the same. 
2.we noticed that the memory table flush looks still working,not the same as CASSANDRA-12796 mentioned,but the compactionExecutor pending is increasing.

Do you by chance have concurrent_compactors only set to 2? Seems like your index builds are blocking other compactions from taking place.
Seems that maybe postflush is backed up because it's blocked on writes generated from the rebuild? Maybe, anyway.
3.I'm wondering if the block only blocks the specified table which is creating secondary index?

If all your flush writers/post flushers are blocked I assume no other flushes will be able to take place, regardless of table. 


Seems like CASSANDRA-12796 is related but not sure why it didn't get fixed in 2.1.

Re: secondary index creation causes C* oom

[kurt greaves <ku...@instaclustr.com>]

> 1.not sure if secondary index creation is the same as index rebuild
>
Fairly sure they are the same.

> 2.we noticed that the memory table flush looks still working,not the same
> as CASSANDRA-12796 mentioned,but the compactionExecutor pending is
> increasing.
>
Do you by chance have concurrent_compactors only set to 2? Seems like your
index builds are blocking other compactions from taking place.
Seems that maybe postflush is backed up because it's blocked on writes
generated from the rebuild? Maybe, anyway.

> 3.I'm wondering if the block only blocks the specified table which is
> creating secondary index?
>
If all your flush writers/post flushers are blocked I assume no other
flushes will be able to take place, regardless of table.

Seems like CASSANDRA-12796
<https://issues.apache.org/jira/browse/CASSANDRA-12796> is related but not
sure why it didn't get fixed in 2.1.