How to make Cassandra flush CommitLog files more frequently?

[Bingqin Zhou <bi...@wepay.com.INVALID>]

Hi,

We're working with the CDC feature to develop an agent to stream changes in
Cassandra DB into Kafka. However, the CDC feature doesn't work well for us
so far because CommitLog files are rarely flushed into cdc_raw directory,
and the frequency can be as low as a few months.

Is there any suggested and feasible way to increase the frequency for
Cassandra to flush CommitLog files please?

We're thinking about decreasing *commitlog_segment_size_in_mb* from 32 to
16, and decreasing *commitlog_total_space_in_mb* from 8192 to 160. Does
this sound like a reasonable approach? Is there any concern or anything we
need to be warned about trying this please?

Thank you!

Bingqin Zhou

Re: How to make Cassandra flush CommitLog files more frequently?

[Kane Wilson <k...@raft.so>]

>
> Actually, the design and initiation of our service is based on the fact
> that the CDC feature in Cassandra is used for streaming data changes in
> Cassandra with low latency. If this is not the case, may I understand
> what's the purpose and the intended use case for the CDC feature in
> Cassandra please?
>

I'm sure that was the purpose, but as I said it's a recent addition and I
don't think it's been well utilised in the wild. But in general I feel
you're better off having some form of queue based system prior to your
database that supports this functionality, especially as C* isn't much used
for transforming data, nor was it ever designed as a streaming system.

raft.so - Cassandra consulting, support, and managed services


On Wed, May 5, 2021 at 5:58 AM Bingqin Zhou <bi...@wepay.com> wrote:

> Hi Kane,
>
> Thank you for the insights!
>
> Reducing the total space on its own will help, however definitely test
>> this as such a large drop could result in a massive increase in SSTables
>> and thus compaction overhead. You'll in general want to look into any
>> property that makes memtables flush more frequently (which is based on heap
>> size and some tuning properties in cassandra.yaml).
>
>
> If we decrease *memtable_heap_sapce_in_mb* and *memtable_off_space_in_mb*,
> is it going to cause more compaction activities potentially as well?
>
> I'm typically not a fan of using a database as a streaming/workflow
>> service, so I have to ask have you considered managing this from your
>> clients rather than using CDC in C*?
>
>
> Actually, the design and initiation of our service is based on the fact
> that the CDC feature in Cassandra is used for streaming data changes in
> Cassandra with low latency. If this is not the case, may I understand
> what's the purpose and the intended use case for the CDC feature in
> Cassandra please?
>
> Thank you so much!
> Bingqin Zhou
>
> On Mon, May 3, 2021 at 5:00 PM Kane Wilson <k...@raft.so> wrote:
>
>> (removing dev)
>>
>> commitlog_segment_size_in_mb isn't going to help, in fact you probably
>> don't want to modify this as it'll reduce the maximum size of your
>> mutations.
>> Reducing the total space on its own will help, however definitely test
>> this as such a large drop could result in a massive increase in SSTables
>> and thus compaction overhead. You'll in general want to look into any
>> property that makes memtables flush more frequently (which is based on heap
>> size and some tuning properties in cassandra.yaml).
>>
>> I'm typically not a fan of using a database as a streaming/workflow
>> service, so I have to ask have you considered managing this from your
>> clients rather than using CDC in C*?
>>
>> raft.so - Cassandra consulting, support, and managed services
>>
>>
>> On Tue, May 4, 2021 at 4:16 AM Bingqin Zhou <bi...@wepay.com> wrote:
>>
>>> Hi,
>>>
>>> We're working with the CDC feature to develop an agent to stream changes
>>> in Cassandra DB into Kafka. However, the CDC feature doesn't work well for
>>> us so far because CommitLog files are rarely flushed into cdc_raw
>>> directory, and the frequency can be as low as a few months.
>>>
>>> Is there any suggested and feasible way to increase the frequency for
>>> Cassandra to flush CommitLog files please?
>>>
>>> We're thinking about decreasing *commitlog_segment_size_in_mb* from 32
>>> to 16, and decreasing *commitlog_total_space_in_mb* from 8192 to 160.
>>> Does this sound like a reasonable approach? Is there any concern or
>>> anything we need to be warned about trying this please?
>>>
>>> Thank you!
>>>
>>> Bingqin Zhou
>>>
>>

Re: How to make Cassandra flush CommitLog files more frequently?

[vytenis silgalis <vs...@gmail.com>]

I believe you could set your tables to flush to disk at specific intervals
(memtable_flush_period_in_ms), note that you'd have to set this for all
tables (not just the CDC enabled tables) to ensure that commitlog files are
flushed to the cdc_raw directory. Or as Dhanunjaya noted you could just
periodically call the `nodetool flush` endpoint to flush all the tables at
one go.

Vytenis





On Tue, May 4, 2021 at 11:04 PM Dhanunjaya Tokala <
dhanunjayatokala@gmail.com> wrote:

> One way to flush commitlog is nodetool flush
> On Cassandra nodes .
>
> On Tue, May 4, 2021 at 3:58 PM Bingqin Zhou <bi...@wepay.com> wrote:
>
>> Hi Kane,
>>
>> Thank you for the insights!
>>
>> Reducing the total space on its own will help, however definitely test
>>> this as such a large drop could result in a massive increase in SSTables
>>> and thus compaction overhead. You'll in general want to look into any
>>> property that makes memtables flush more frequently (which is based on heap
>>> size and some tuning properties in cassandra.yaml).
>>
>>
>> If we decrease *memtable_heap_sapce_in_mb* and *memtable_off_space_in_mb*,
>> is it going to cause more compaction activities potentially as well?
>>
>> I'm typically not a fan of using a database as a streaming/workflow
>>> service, so I have to ask have you considered managing this from your
>>> clients rather than using CDC in C*?
>>
>>
>> Actually, the design and initiation of our service is based on the fact
>> that the CDC feature in Cassandra is used for streaming data changes in
>> Cassandra with low latency. If this is not the case, may I understand
>> what's the purpose and the intended use case for the CDC feature in
>> Cassandra please?
>>
>> Thank you so much!
>> Bingqin Zhou
>>
>> On Mon, May 3, 2021 at 5:00 PM Kane Wilson <k...@raft.so> wrote:
>>
>>> (removing dev)
>>>
>>> commitlog_segment_size_in_mb isn't going to help, in fact you probably
>>> don't want to modify this as it'll reduce the maximum size of your
>>> mutations.
>>> Reducing the total space on its own will help, however definitely test
>>> this as such a large drop could result in a massive increase in SSTables
>>> and thus compaction overhead. You'll in general want to look into any
>>> property that makes memtables flush more frequently (which is based on heap
>>> size and some tuning properties in cassandra.yaml).
>>>
>>> I'm typically not a fan of using a database as a streaming/workflow
>>> service, so I have to ask have you considered managing this from your
>>> clients rather than using CDC in C*?
>>>
>>> raft.so - Cassandra consulting, support, and managed services
>>>
>>>
>>> On Tue, May 4, 2021 at 4:16 AM Bingqin Zhou <bi...@wepay.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> We're working with the CDC feature to develop an agent to stream
>>>> changes in Cassandra DB into Kafka. However, the CDC feature doesn't work
>>>> well for us so far because CommitLog files are rarely flushed into cdc_raw
>>>> directory, and the frequency can be as low as a few months.
>>>>
>>>> Is there any suggested and feasible way to increase the frequency for
>>>> Cassandra to flush CommitLog files please?
>>>>
>>>> We're thinking about decreasing *commitlog_segment_size_in_mb* from 32
>>>> to 16, and decreasing *commitlog_total_space_in_mb* from 8192 to 160.
>>>> Does this sound like a reasonable approach? Is there any concern or
>>>> anything we need to be warned about trying this please?
>>>>
>>>> Thank you!
>>>>
>>>> Bingqin Zhou
>>>>
>>>

Re: How to make Cassandra flush CommitLog files more frequently?

[Dhanunjaya Tokala <dh...@gmail.com>]

One way to flush commitlog is nodetool flush
On Cassandra nodes .

On Tue, May 4, 2021 at 3:58 PM Bingqin Zhou <bi...@wepay.com> wrote:

> Hi Kane,
>
> Thank you for the insights!
>
> Reducing the total space on its own will help, however definitely test
>> this as such a large drop could result in a massive increase in SSTables
>> and thus compaction overhead. You'll in general want to look into any
>> property that makes memtables flush more frequently (which is based on heap
>> size and some tuning properties in cassandra.yaml).
>
>
> If we decrease *memtable_heap_sapce_in_mb* and *memtable_off_space_in_mb*,
> is it going to cause more compaction activities potentially as well?
>
> I'm typically not a fan of using a database as a streaming/workflow
>> service, so I have to ask have you considered managing this from your
>> clients rather than using CDC in C*?
>
>
> Actually, the design and initiation of our service is based on the fact
> that the CDC feature in Cassandra is used for streaming data changes in
> Cassandra with low latency. If this is not the case, may I understand
> what's the purpose and the intended use case for the CDC feature in
> Cassandra please?
>
> Thank you so much!
> Bingqin Zhou
>
> On Mon, May 3, 2021 at 5:00 PM Kane Wilson <k...@raft.so> wrote:
>
>> (removing dev)
>>
>> commitlog_segment_size_in_mb isn't going to help, in fact you probably
>> don't want to modify this as it'll reduce the maximum size of your
>> mutations.
>> Reducing the total space on its own will help, however definitely test
>> this as such a large drop could result in a massive increase in SSTables
>> and thus compaction overhead. You'll in general want to look into any
>> property that makes memtables flush more frequently (which is based on heap
>> size and some tuning properties in cassandra.yaml).
>>
>> I'm typically not a fan of using a database as a streaming/workflow
>> service, so I have to ask have you considered managing this from your
>> clients rather than using CDC in C*?
>>
>> raft.so - Cassandra consulting, support, and managed services
>>
>>
>> On Tue, May 4, 2021 at 4:16 AM Bingqin Zhou <bi...@wepay.com> wrote:
>>
>>> Hi,
>>>
>>> We're working with the CDC feature to develop an agent to stream changes
>>> in Cassandra DB into Kafka. However, the CDC feature doesn't work well for
>>> us so far because CommitLog files are rarely flushed into cdc_raw
>>> directory, and the frequency can be as low as a few months.
>>>
>>> Is there any suggested and feasible way to increase the frequency for
>>> Cassandra to flush CommitLog files please?
>>>
>>> We're thinking about decreasing *commitlog_segment_size_in_mb* from 32
>>> to 16, and decreasing *commitlog_total_space_in_mb* from 8192 to 160.
>>> Does this sound like a reasonable approach? Is there any concern or
>>> anything we need to be warned about trying this please?
>>>
>>> Thank you!
>>>
>>> Bingqin Zhou
>>>
>>

Re: How to make Cassandra flush CommitLog files more frequently?

[Bingqin Zhou <bi...@wepay.com>]

Hi Kane,

Thank you for the insights!

Reducing the total space on its own will help, however definitely test this
> as such a large drop could result in a massive increase in SSTables and
> thus compaction overhead. You'll in general want to look into any property
> that makes memtables flush more frequently (which is based on heap size and
> some tuning properties in cassandra.yaml).


If we decrease *memtable_heap_sapce_in_mb* and *memtable_off_space_in_mb*,
is it going to cause more compaction activities potentially as well?

I'm typically not a fan of using a database as a streaming/workflow
> service, so I have to ask have you considered managing this from your
> clients rather than using CDC in C*?


Actually, the design and initiation of our service is based on the fact
that the CDC feature in Cassandra is used for streaming data changes in
Cassandra with low latency. If this is not the case, may I understand
what's the purpose and the intended use case for the CDC feature in
Cassandra please?

Thank you so much!
Bingqin Zhou

On Mon, May 3, 2021 at 5:00 PM Kane Wilson <k...@raft.so> wrote:

> (removing dev)
>
> commitlog_segment_size_in_mb isn't going to help, in fact you probably
> don't want to modify this as it'll reduce the maximum size of your
> mutations.
> Reducing the total space on its own will help, however definitely test
> this as such a large drop could result in a massive increase in SSTables
> and thus compaction overhead. You'll in general want to look into any
> property that makes memtables flush more frequently (which is based on heap
> size and some tuning properties in cassandra.yaml).
>
> I'm typically not a fan of using a database as a streaming/workflow
> service, so I have to ask have you considered managing this from your
> clients rather than using CDC in C*?
>
> raft.so - Cassandra consulting, support, and managed services
>
>
> On Tue, May 4, 2021 at 4:16 AM Bingqin Zhou <bi...@wepay.com> wrote:
>
>> Hi,
>>
>> We're working with the CDC feature to develop an agent to stream changes
>> in Cassandra DB into Kafka. However, the CDC feature doesn't work well for
>> us so far because CommitLog files are rarely flushed into cdc_raw
>> directory, and the frequency can be as low as a few months.
>>
>> Is there any suggested and feasible way to increase the frequency for
>> Cassandra to flush CommitLog files please?
>>
>> We're thinking about decreasing *commitlog_segment_size_in_mb* from 32
>> to 16, and decreasing *commitlog_total_space_in_mb* from 8192 to 160.
>> Does this sound like a reasonable approach? Is there any concern or
>> anything we need to be warned about trying this please?
>>
>> Thank you!
>>
>> Bingqin Zhou
>>
>

Re: How to make Cassandra flush CommitLog files more frequently?

[Kane Wilson <k...@raft.so>]

(removing dev)

commitlog_segment_size_in_mb isn't going to help, in fact you probably
don't want to modify this as it'll reduce the maximum size of your
mutations.
Reducing the total space on its own will help, however definitely test this
as such a large drop could result in a massive increase in SSTables and
thus compaction overhead. You'll in general want to look into any property
that makes memtables flush more frequently (which is based on heap size and
some tuning properties in cassandra.yaml).

I'm typically not a fan of using a database as a streaming/workflow
service, so I have to ask have you considered managing this from your
clients rather than using CDC in C*?

raft.so - Cassandra consulting, support, and managed services


On Tue, May 4, 2021 at 4:16 AM Bingqin Zhou <bi...@wepay.com> wrote:

> Hi,
>
> We're working with the CDC feature to develop an agent to stream changes
> in Cassandra DB into Kafka. However, the CDC feature doesn't work well for
> us so far because CommitLog files are rarely flushed into cdc_raw
> directory, and the frequency can be as low as a few months.
>
> Is there any suggested and feasible way to increase the frequency for
> Cassandra to flush CommitLog files please?
>
> We're thinking about decreasing *commitlog_segment_size_in_mb* from 32 to
> 16, and decreasing *commitlog_total_space_in_mb* from 8192 to 160. Does
> this sound like a reasonable approach? Is there any concern or anything we
> need to be warned about trying this please?
>
> Thank you!
>
> Bingqin Zhou
>