Cassandra event notification on INSERT/DELETE of records

[Aaditya Vadnere <sk...@gmail.com>]

Hi experts,

We are evaluating Cassandra as messaging infrastructure for a project.

In our workflow Cassandra database will be synchronized across two nodes, a
component will INSERT/UPDATE records on one node and another component (who
has registered for the specific table) on second node will get notified of
record change.

The second component will then try to read the database to find out the
specific message.

Is it possible for Cassandra to support such workflow? Basically, is there
a way for Cassandra to generate a notification anytime schema changes (so
we can set processes to listen for schema changes). As I understand,
polling the database periodically or database triggers might work but they
are costly operations.

-- 
Aaditya Vadnere

Re: Cassandra event notification on INSERT/DELETE of records

[Eric Stevens <mi...@gmail.com>]

Unfortunately, read immediately after write is another antipattern in
*all* eventually
consistent databases (though reading and writing both at quorum should
effectively produce immediate consistency).

The upcoming Change Data Capture feature that Michael Laing linked might be
a useful feature to support your need, but that feature is still in
development.

Is your hope that Thread 2 only knows about an update after it has come to
rest?  Or is it ok if sometimes Thread 2 sometimes knows about an update
after Thread 1 does?  If these are in the same process or address space,
they might just share data, or use an in-process messaging architecture
like Akka if you want to avoid the race conditions that come from sharing
memory.

If it's ok that Thread 2 sometimes knows about an update before Thread 1,
then possibly you can have both threads listen to the same event stream and
update themselves accordingly.

In any event it strikes me as a fragile and under-performing design to put
the database as a hop in the middle of a realtime message sharing system,
particularly if that database is eventually consistent.

On Wed, May 25, 2016 at 10:11 AM Aaditya Vadnere <sk...@gmail.com> wrote:

> Thanks Eric and Mark, we were thinking along similar lines. But we already
> need Cassandra for regular database purpose, so instead of having both
> Kafka and Cassandra, the possibility of using Cassandra alone was explored.
>
> Another usecase where update notification can be useful is when we want to
> synchronize two or more instances of same component. Say two threads of
> component 'A' can share the same database. When a record is updated in
> database by thread 1, a notification is sent to thread 2. After that thread
> 2, performs a read.
>
> I think this also is an anti-pattern.
>
> Regards,
> Aaditya
>
> On Tue, May 24, 2016 at 12:45 PM, Mark Reddy <ma...@gmail.com>
> wrote:
>
>> +1 to what Eric said, a queue is a classic C* anti-pattern. Something
>> like Kafka or RabbitMQ might fit your use case better.
>>
>>
>> Mark
>>
>> On 24 May 2016 at 18:03, Eric Stevens <mi...@gmail.com> wrote:
>>
>>> It sounds like you're trying to build a queue in Cassandra, which is one
>>> of the classic anti-pattern use cases for Cassandra.
>>>
>>> You may be able to do something clever with triggers, but I highly
>>> recommend you look at purpose-built queuing software such as Kafka to solve
>>> this instead.
>>>
>>> On Tue, May 24, 2016 at 9:49 AM Aaditya Vadnere <sk...@gmail.com>
>>> wrote:
>>>
>>>> Hi experts,
>>>>
>>>> We are evaluating Cassandra as messaging infrastructure for a project.
>>>>
>>>> In our workflow Cassandra database will be synchronized across two
>>>> nodes, a component will INSERT/UPDATE records on one node and another
>>>> component (who has registered for the specific table) on second node will
>>>> get notified of record change.
>>>>
>>>> The second component will then try to read the database to find out the
>>>> specific message.
>>>>
>>>> Is it possible for Cassandra to support such workflow? Basically, is
>>>> there a way for Cassandra to generate a notification anytime schema changes
>>>> (so we can set processes to listen for schema changes). As I understand,
>>>> polling the database periodically or database triggers might work but they
>>>> are costly operations.
>>>>
>>>>
>>>> --
>>>> Aaditya Vadnere
>>>>
>>>
>>
>
>
> --
> Aaditya Vadnere
>

Re: Cassandra event notification on INSERT/DELETE of records

["Laing, Michael" <mi...@nytimes.com>]

You could also follow this related issue:
https://issues.apache.org/jira/browse/CASSANDRA-8844

On Wed, May 25, 2016 at 12:04 PM, Aaditya Vadnere <sk...@gmail.com> wrote:

> Thanks Eric and Mark, we were thinking along similar lines. But we already
> need Cassandra for regular database purpose, so instead of having both
> Kafka and Cassandra, the possibility of using Cassandra alone was explored.
>
> Another usecase where update notification can be useful is when we want to
> synchronize two or more instances of same component. Say two threads of
> component 'A' can share the same database. When a record is updated in
> database by thread 1, a notification is sent to thread 2. After that thread
> 2, performs a read.
>
> I think this also is an anti-pattern.
>
> Regards,
> Aaditya
>
> On Tue, May 24, 2016 at 12:45 PM, Mark Reddy <ma...@gmail.com>
> wrote:
>
>> +1 to what Eric said, a queue is a classic C* anti-pattern. Something
>> like Kafka or RabbitMQ might fit your use case better.
>>
>>
>> Mark
>>
>> On 24 May 2016 at 18:03, Eric Stevens <mi...@gmail.com> wrote:
>>
>>> It sounds like you're trying to build a queue in Cassandra, which is one
>>> of the classic anti-pattern use cases for Cassandra.
>>>
>>> You may be able to do something clever with triggers, but I highly
>>> recommend you look at purpose-built queuing software such as Kafka to solve
>>> this instead.
>>>
>>> On Tue, May 24, 2016 at 9:49 AM Aaditya Vadnere <sk...@gmail.com>
>>> wrote:
>>>
>>>> Hi experts,
>>>>
>>>> We are evaluating Cassandra as messaging infrastructure for a project.
>>>>
>>>> In our workflow Cassandra database will be synchronized across two
>>>> nodes, a component will INSERT/UPDATE records on one node and another
>>>> component (who has registered for the specific table) on second node will
>>>> get notified of record change.
>>>>
>>>> The second component will then try to read the database to find out the
>>>> specific message.
>>>>
>>>> Is it possible for Cassandra to support such workflow? Basically, is
>>>> there a way for Cassandra to generate a notification anytime schema changes
>>>> (so we can set processes to listen for schema changes). As I understand,
>>>> polling the database periodically or database triggers might work but they
>>>> are costly operations.
>>>>
>>>>
>>>> --
>>>> Aaditya Vadnere
>>>>
>>>
>>
>
>
> --
> Aaditya Vadnere
>

Re: Cassandra event notification on INSERT/DELETE of records

[Aaditya Vadnere <sk...@gmail.com>]

Thanks Eric and Mark, we were thinking along similar lines. But we already
need Cassandra for regular database purpose, so instead of having both
Kafka and Cassandra, the possibility of using Cassandra alone was explored.

Another usecase where update notification can be useful is when we want to
synchronize two or more instances of same component. Say two threads of
component 'A' can share the same database. When a record is updated in
database by thread 1, a notification is sent to thread 2. After that thread
2, performs a read.

I think this also is an anti-pattern.

Regards,
Aaditya

On Tue, May 24, 2016 at 12:45 PM, Mark Reddy <ma...@gmail.com> wrote:

> +1 to what Eric said, a queue is a classic C* anti-pattern. Something like
> Kafka or RabbitMQ might fit your use case better.
>
>
> Mark
>
> On 24 May 2016 at 18:03, Eric Stevens <mi...@gmail.com> wrote:
>
>> It sounds like you're trying to build a queue in Cassandra, which is one
>> of the classic anti-pattern use cases for Cassandra.
>>
>> You may be able to do something clever with triggers, but I highly
>> recommend you look at purpose-built queuing software such as Kafka to solve
>> this instead.
>>
>> On Tue, May 24, 2016 at 9:49 AM Aaditya Vadnere <sk...@gmail.com> wrote:
>>
>>> Hi experts,
>>>
>>> We are evaluating Cassandra as messaging infrastructure for a project.
>>>
>>> In our workflow Cassandra database will be synchronized across two
>>> nodes, a component will INSERT/UPDATE records on one node and another
>>> component (who has registered for the specific table) on second node will
>>> get notified of record change.
>>>
>>> The second component will then try to read the database to find out the
>>> specific message.
>>>
>>> Is it possible for Cassandra to support such workflow? Basically, is
>>> there a way for Cassandra to generate a notification anytime schema changes
>>> (so we can set processes to listen for schema changes). As I understand,
>>> polling the database periodically or database triggers might work but they
>>> are costly operations.
>>>
>>>
>>> --
>>> Aaditya Vadnere
>>>
>>
>


-- 
Aaditya Vadnere

Re: Cassandra event notification on INSERT/DELETE of records

[Mark Reddy <ma...@gmail.com>]

+1 to what Eric said, a queue is a classic C* anti-pattern. Something like
Kafka or RabbitMQ might fit your use case better.


Mark

On 24 May 2016 at 18:03, Eric Stevens <mi...@gmail.com> wrote:

> It sounds like you're trying to build a queue in Cassandra, which is one
> of the classic anti-pattern use cases for Cassandra.
>
> You may be able to do something clever with triggers, but I highly
> recommend you look at purpose-built queuing software such as Kafka to solve
> this instead.
>
> On Tue, May 24, 2016 at 9:49 AM Aaditya Vadnere <sk...@gmail.com> wrote:
>
>> Hi experts,
>>
>> We are evaluating Cassandra as messaging infrastructure for a project.
>>
>> In our workflow Cassandra database will be synchronized across two nodes,
>> a component will INSERT/UPDATE records on one node and another component
>> (who has registered for the specific table) on second node will get
>> notified of record change.
>>
>> The second component will then try to read the database to find out the
>> specific message.
>>
>> Is it possible for Cassandra to support such workflow? Basically, is
>> there a way for Cassandra to generate a notification anytime schema changes
>> (so we can set processes to listen for schema changes). As I understand,
>> polling the database periodically or database triggers might work but they
>> are costly operations.
>>
>>
>> --
>> Aaditya Vadnere
>>
>

Re: Cassandra event notification on INSERT/DELETE of records

[Eric Stevens <mi...@gmail.com>]

It sounds like you're trying to build a queue in Cassandra, which is one of
the classic anti-pattern use cases for Cassandra.

You may be able to do something clever with triggers, but I highly
recommend you look at purpose-built queuing software such as Kafka to solve
this instead.

On Tue, May 24, 2016 at 9:49 AM Aaditya Vadnere <sk...@gmail.com> wrote:

> Hi experts,
>
> We are evaluating Cassandra as messaging infrastructure for a project.
>
> In our workflow Cassandra database will be synchronized across two nodes,
> a component will INSERT/UPDATE records on one node and another component
> (who has registered for the specific table) on second node will get
> notified of record change.
>
> The second component will then try to read the database to find out the
> specific message.
>
> Is it possible for Cassandra to support such workflow? Basically, is there
> a way for Cassandra to generate a notification anytime schema changes (so
> we can set processes to listen for schema changes). As I understand,
> polling the database periodically or database triggers might work but they
> are costly operations.
>
>
> --
> Aaditya Vadnere
>