You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@cassandra.apache.org by Jagan Ranganathan <ja...@zohocorp.com> on 2014/02/22 16:03:34 UTC

Queuing System

Hi,

I need to decouple some of the work being processed from the user thread to provide better user experience. For that I need a queuing system with the following needs,
High Availability
No Data Loss
Better Performance.

Following are some libraries that were considered along with the limitation I see,
Redis - Data Loss
ZooKeeper - Not advised for Queue system.
TokyoCabinet/SQLite/LevelDB - of this Level DB seem to be performing better. With replication requirement, I probably have to look at Apache ActiveMQ+LevelDB.

After checking on the third option above, I kind of wonder if Cassandra with Leveled Compaction offer a similar system. Do you see any issues in such a usage or is there other better solutions available.


Will be great to get insights on this.


Regards,
Jagan

Re: Queuing System

Posted by Jagan Ranganathan <ja...@zohocorp.com>.

Thanks Joe. That's a nice pointer. Will explore the possibility. I am just concerned about the Leader swap time window, but may be thats the tradeoff b/n data consistency Vs availability.

Regards,
Jagan

---- On Sat, 22 Feb 2014 23:08:00 +0530 Joe Stein &lt;cryptcom@gmail.com&gt; wrote ----

Without them you have no durability.

With them you have guarantees... More than any other system with messaging features. It is a durable CP commit log. Works very well for data pipelines with AP systems like Cassandra which is a different system solving different problems. When a Kafka leader fails you right might block and wait for 10ms while a new leader is elected but writes can be guaranteed.

The consumers then read and process data and write to Cassandra. And then have your app read from Cassandra for what what was processed.

These are very typical type architectures at scale https://cwiki.apache.org/confluence/display/KAFKA/Kafka+papers+and+presentations

/*******************************************
Joe Stein
Founder, Principal Consultant
Big Data Open Source Security LLC
http://www.stealth.ly
Twitter: @allthingshadoop
********************************************/

On Feb 22, 2014, at 11:49 AM, Jagan Ranganathan &lt;jagan@zohocorp.com&gt; wrote:

Hi Joe,

If my understanding is right, Kafka does not satisfy the high availability/replication part well because of the need for leader and In-Sync replicas.

Regards,
Jagan

---- On Sat, 22 Feb 2014 22:02:27 +0530 Joe Stein&lt;cryptcom@gmail.com&gt; wrote ----

If performance and availability for messaging is a requirement then use Apache Kafka http://kafka.apache.org/

You can pass the same thrift/avro objects through the Kafka commit log or strings or whatever you want.

On Feb 22, 2014, at 11:13 AM, Jagan Ranganathan &lt;jagan@zohocorp.com&gt; wrote:

Hi Michael,

Yes I am planning to use RabbitMQ for my messaging system. But I wonder which will give better performance if writing directly into Rabbit with Ack support Vs a temporary Queue in Cassandra first and then dequeue and publish in Rabbit.

Complexities involving - Handling scenarios like Rabbit Connection failure etc Vs Cassandra write performance and replication with hinted handoff support etc, makes me wonder if this is a better path.

Regards,
Jagan

---- On Sat, 22 Feb 2014 21:01:14 +0530 Michael Laing &lt;michael.laing@nytimes.com&gt; wrote ----

We use RabbitMQ for queuing and Cassandra for persistence.

RabbitMQ with clustering and/or federation should meet your high availability needs.

Michael

On Sat, Feb 22, 2014 at 10:25 AM, DuyHai Doan &lt;doanduyhai@gmail.com&gt; wrote:
Jagan

Queue-like data structures are known to be one of the worst anti patterns for Cassandra: http://www.datastax.com/dev/blog/cassandra-anti-patterns-queues-and-queue-like-datasets

On Sat, Feb 22, 2014 at 4:03 PM, Jagan Ranganathan &lt;jagan@zohocorp.com&gt; wrote:
Hi,

I need to decouple some of the work being processed from the user thread to provide better user experience. For that I need a queuing system with the following needs,
High Availability
No Data Loss
Better Performance.

Following are some libraries that were considered along with the limitation I see,
Redis - Data Loss
ZooKeeper - Not advised for Queue system.
TokyoCabinet/SQLite/LevelDB - of this Level DB seem to be performing better. With replication requirement, I probably have to look at Apache ActiveMQ+LevelDB.

After checking on the third option above, I kind of wonder if Cassandra with Leveled Compaction offer a similar system. Do you see any issues in such a usage or is there other better solutions available.

Will be great to get insights on this.

Regards,
Jagan

Re: Queuing System

Posted by Joe Stein <cr...@gmail.com>.

Without them you have no durability.  

With them you have guarantees... More than any other system with messaging features.  It is a durable CP commit log.  Works very well for data pipelines with AP systems like Cassandra which is a different system solving different problems.  When a Kafka leader fails you right might block and wait for 10ms while a new leader is elected but writes can be guaranteed.

The consumers then read and process data and write to Cassandra. And then have your app read from Cassandra for what what was processed.

These are very typical type architectures at scale https://cwiki.apache.org/confluence/display/KAFKA/Kafka+papers+and+presentations

/*******************************************
 Joe Stein
 Founder, Principal Consultant
 Big Data Open Source Security LLC
 http://www.stealth.ly
 Twitter: @allthingshadoop
********************************************/


On Feb 22, 2014, at 11:49 AM, Jagan Ranganathan <ja...@zohocorp.com> wrote:

> Hi Joe,
> 
> If my understanding is right, Kafka does not satisfy the high availability/replication part well because of the need for leader and In-Sync replicas. 
> 
> Regards,
> Jagan
> 
> ---- On Sat, 22 Feb 2014 22:02:27 +0530 Joe Stein<cr...@gmail.com> wrote ---- 
> 
> If performance and availability for messaging is a requirement then use Apache Kafka http://kafka.apache.org/
> 
> You can pass the same thrift/avro objects through the Kafka commit log or strings or whatever you want.
> 
> /*******************************************
>  Joe Stein
>  Founder, Principal Consultant
>  Big Data Open Source Security LLC
>  http://www.stealth.ly
>  Twitter: @allthingshadoop
> ********************************************/
> 
> 
> On Feb 22, 2014, at 11:13 AM, Jagan Ranganathan <ja...@zohocorp.com> wrote:
> 
> Hi Michael,
> 
> Yes I am planning to use RabbitMQ for my messaging system. But I wonder which will give better performance if writing directly into Rabbit with Ack support Vs a temporary Queue in Cassandra first and then dequeue and publish in Rabbit.
> 
> Complexities involving - Handling scenarios like Rabbit Connection failure etc Vs Cassandra write performance and replication with hinted handoff support etc, makes me wonder if this is a better path.
> 
> Regards,
> Jagan
> 
> ---- On Sat, 22 Feb 2014 21:01:14 +0530 Michael Laing <mi...@nytimes.com> wrote ---- 
> 
> We use RabbitMQ for queuing and Cassandra for persistence.
> 
> RabbitMQ with clustering and/or federation should meet your high availability needs.
> 
> Michael
> 
> 
> On Sat, Feb 22, 2014 at 10:25 AM, DuyHai Doan <do...@gmail.com> wrote:
> Jagan 
> 
>  Queue-like data structures are known to be one of the worst anti patterns for Cassandra:  http://www.datastax.com/dev/blog/cassandra-anti-patterns-queues-and-queue-like-datasets
> 
> 
> 
> On Sat, Feb 22, 2014 at 4:03 PM, Jagan Ranganathan <ja...@zohocorp.com> wrote:
> Hi,
> 
> I need to decouple some of the work being processed from the user thread to provide better user experience. For that I need a                                     queuing system with the following needs,
> High Availability
> No Data Loss
> Better Performance.
> Following are some libraries that were considered along with the limitation I see,
> Redis - Data Loss
> ZooKeeper - Not advised for Queue system.
> TokyoCabinet/SQLite/LevelDB - of this Level DB seem to be performing better. With replication requirement, I probably have to look at Apache ActiveMQ+LevelDB.
> After checking on the third option above, I kind of wonder if Cassandra with Leveled Compaction offer a similar system. Do you see any issues in such a usage or is there other better solutions available.
> 
> Will be great to get insights on this.
> 
> Regards,
> Jagan
> 
> 
> 
>

Re: Queuing System

Posted by Jagan Ranganathan <ja...@zohocorp.com>.

Hi Joe,

If my understanding is right, Kafka does not satisfy the high availability/replication part well because of the need for leader and In-Sync replicas.

Regards,
Jagan

---- On Sat, 22 Feb 2014 22:02:27 +0530 Joe Stein&lt;cryptcom@gmail.com&gt; wrote ----

If performance and availability for messaging is a requirement then use Apache Kafka http://kafka.apache.org/

You can pass the same thrift/avro objects through the Kafka commit log or strings or whatever you want.

On Feb 22, 2014, at 11:13 AM, Jagan Ranganathan &lt;jagan@zohocorp.com&gt; wrote:

Hi Michael,

Regards,
Jagan

---- On Sat, 22 Feb 2014 21:01:14 +0530 Michael Laing &lt;michael.laing@nytimes.com&gt; wrote ----

We use RabbitMQ for queuing and Cassandra for persistence.

RabbitMQ with clustering and/or federation should meet your high availability needs.

Michael

On Sat, Feb 22, 2014 at 10:25 AM, DuyHai Doan &lt;doanduyhai@gmail.com&gt; wrote:
Jagan

Queue-like data structures are known to be one of the worst anti patterns for Cassandra: http://www.datastax.com/dev/blog/cassandra-anti-patterns-queues-and-queue-like-datasets

On Sat, Feb 22, 2014 at 4:03 PM, Jagan Ranganathan &lt;jagan@zohocorp.com&gt; wrote:
Hi,

Will be great to get insights on this.

Regards,
Jagan

Re: Queuing System

Posted by Joe Stein <cr...@gmail.com>.

If performance and availability for messaging is a requirement then use Apache Kafka http://kafka.apache.org/

You can pass the same thrift/avro objects through the Kafka commit log or strings or whatever you want.

/*******************************************
 Joe Stein
 Founder, Principal Consultant
 Big Data Open Source Security LLC
 http://www.stealth.ly
 Twitter: @allthingshadoop
********************************************/


On Feb 22, 2014, at 11:13 AM, Jagan Ranganathan <ja...@zohocorp.com> wrote:

> Hi Michael,
> 
> Yes I am planning to use RabbitMQ for my messaging system. But I wonder which will give better performance if writing directly into Rabbit with Ack support Vs a temporary Queue in Cassandra first and then dequeue and publish in Rabbit.
> 
> Complexities involving - Handling scenarios like Rabbit Connection failure etc Vs Cassandra write performance and replication with hinted handoff support etc, makes me wonder if this is a better path.
> 
> Regards,
> Jagan
> 
> ---- On Sat, 22 Feb 2014 21:01:14 +0530 Michael Laing <mi...@nytimes.com> wrote ---- 
> 
> We use RabbitMQ for queuing and Cassandra for persistence.
> 
> RabbitMQ with clustering and/or federation should meet your high availability needs.
> 
> Michael
> 
> 
> On Sat, Feb 22, 2014 at 10:25 AM, DuyHai Doan <do...@gmail.com> wrote:
> Jagan 
> 
>  Queue-like data structures are known to be one of the worst anti               patterns for Cassandra:  http://www.datastax.com/dev/blog/cassandra-anti-patterns-queues-and-queue-like-datasets
> 
> 
> 
> On Sat, Feb 22, 2014 at 4:03 PM, Jagan Ranganathan <ja...@zohocorp.com> wrote:
> Hi,
> 
> I need to decouple some of the work being processed from the user thread to provide better user experience. For that I need a queuing system with the following needs,
> High Availability
> No Data Loss
> Better Performance.
> Following are some libraries that were considered along with the limitation I see,
> Redis - Data Loss
> ZooKeeper - Not advised for Queue system.
> TokyoCabinet/SQLite/LevelDB - of this Level DB seem to be performing better. With replication requirement, I probably have to look at Apache ActiveMQ+LevelDB.
> After checking on the third option above, I kind of wonder if Cassandra with Leveled Compaction offer a similar system. Do you see any issues in such a usage or is there other                           better solutions available.
> 
> Will be great to get insights on this.
> 
> Regards,
> Jagan
> 
> 
>

Re: Queuing System

Posted by Jagan Ranganathan <ja...@zohocorp.com>.

Hi Michael,

Regards,
Jagan

---- On Sat, 22 Feb 2014 21:01:14 +0530 Michael Laing &lt;michael.laing@nytimes.com&gt; wrote ----

We use RabbitMQ for queuing and Cassandra for persistence.

RabbitMQ with clustering and/or federation should meet your high availability needs.

Michael

On Sat, Feb 22, 2014 at 10:25 AM, DuyHai Doan &lt;doanduyhai@gmail.com&gt; wrote:
Jagan

Queue-like data structures are known to be one of the worst anti patterns for Cassandra: http://www.datastax.com/dev/blog/cassandra-anti-patterns-queues-and-queue-like-datasets

On Sat, Feb 22, 2014 at 4:03 PM, Jagan Ranganathan &lt;jagan@zohocorp.com&gt; wrote:
Hi,

Will be great to get insights on this.

Regards,
Jagan

Re: Queuing System

Posted by "Laing, Michael" <mi...@nytimes.com>.

We use RabbitMQ for queuing and Cassandra for persistence.

RabbitMQ with clustering and/or federation should meet your high
availability needs.

Michael


On Sat, Feb 22, 2014 at 10:25 AM, DuyHai Doan <do...@gmail.com> wrote:

> Jagan
>
>  Queue-like data structures are known to be one of the worst anti patterns
> for Cassandra:
> http://www.datastax.com/dev/blog/cassandra-anti-patterns-queues-and-queue-like-datasets
>
>
>
> On Sat, Feb 22, 2014 at 4:03 PM, Jagan Ranganathan <ja...@zohocorp.com>wrote:
>
>> Hi,
>>
>> I need to decouple some of the work being processed from the user thread
>> to provide better user experience. For that I need a queuing system with
>> the following needs,
>>
>>    - High Availability
>>    - No Data Loss
>>    - Better Performance.
>>
>> Following are some libraries that were considered along with the
>> limitation I see,
>>
>>    - Redis - Data Loss
>>    - ZooKeeper - Not advised for Queue system.
>>    - TokyoCabinet/SQLite/LevelDB - of this Level DB seem to be
>>    performing better. With replication requirement, I probably have to look at
>>    Apache ActiveMQ+LevelDB.
>>
>> After checking on the third option above, I kind of wonder if Cassandra
>> with Leveled Compaction offer a similar system. Do you see any issues in
>> such a usage or is there other better solutions available.
>>
>> Will be great to get insights on this.
>>
>> Regards,
>> Jagan
>>
>
>

Re: Queuing System

Posted by Edward Capriolo <ed...@gmail.com>.

Once schema I was thinking of was this:

max.row.age: The producer will not write to this row after max row age,
even if not full (e.g. 10 minutes)
max.number.of.messages: Most messages that will ever be written to a row
max.expire.time: TTL every column is written with

set Queue[timeuuid][max.row.age] = date()
set Queue[timeuuid][max.number.of.messages] = 50000
set Queue[timeuuid][messages.written] = 0
set Queue[timeuuid][consumer.id] = ''
set Queue[timeuuid][messages.consumed] = 0
set Queue[timeuuid][uuid-of-message1] = value-of-message1
set Queue[timeuuid][uuid-of-message2] = value-of-message2

So a producer writes these messages.

Fields like messages.consumed and consumer.id are only accesses by
consumers with CAS to ensure that messages are not double counted. You want
to have either TTL or a very low gc_grace_next. This way N consumers can
get_range_slice to find rather large rows and can use CAS to register
without double consuming. The consumer can update messages.consumed
atomically as it is readying, chunks of slice (1000 sized messages).When
the consumer is done reading all the data one tombstone destroys the entire
row.


On Sat, Feb 22, 2014 at 9:27 PM, Jagan Ranganathan <ja...@zohocorp.com>wrote:

> Thanks Duy Hai for sharing the details. I have a doubt. If for some reason
> there is a Network Partition or more than 2 Node failure serving the same
> partition/load and you ended up writing hinted hand-off.
>
> Is there a possibility of a data loss? If yes, how do we avoid that?
>
> Regards,
> Jagan
>
> ---- On Sat, 22 Feb 2014 22:48:19 +0530 *DuyHai Doan
> <doanduyhai@gmail.com <do...@gmail.com>>* wrote ----
>
>     Jagan
>
> Few time ago I dealed with a similar queuing design for one customer.
>
> *If you never delete messages in the queue*, then it is possible to use
> wide rows with bucketing and increasing monotonic column name to store
> messages.
>
> CREATE TABLE *read_only_queue *(
>    bucket_number int,
>    insertion_time timeuuid,
>    message text,
>    PRIMARY KEY(bucket_number,insertion_time)
>   );
>
>  Let's say that you allow only 100 000 messages per partition (physical
> row) to avoid too wide rows, then inserting/reading from the table *read_only_queue
> *is easy;
>
>  For message producer :
>
>    1) Start at bucket_number = 1
>    2) Insert messages with column name = generated timeUUID with
> micro-second precision (depending on whether the insertion rate is high or
> not)
>     3) If message count = 100 000, increment bucket_number by one and go
> to 2)
>
> For message reader:
>
>    1) Start at bucket_number = 1
>    2) Read messages by slice of * N, *save the *insertion_time *of the
> last read message
>     3) Use the saved *insertion_time *to perform next slice query
>    4) If read messages count = 100 000, increment bucket_number and go to
> 2). Keep the *insertion_time, *do not reset it since his value is
> increasing monotonically
>
> For multiple and concurrent producers & writers, there is a trick. Let's
> assume you have *P* concurrent producers and *C* concurrent consumers.
>
>   Assign a numerical ID for each producer and consumer. First producer ID
> = 1... last producer ID = *P*. Same for consumers.
>
>   - re-use the above algorithm
>   - each producer/consumer start at *bucket_number *= his ID
>   - at the end of the row,
>         - next bucket_number = current bucker_number + *P* for producers
>         - next bucket_number = current bucker_number + *C* for consumers
>
>
> The last thing to take care of is compaction configuration to reduce the
> number of SSTables on disk.
>
> If you achieve to get rid of accumulation effects, e.g reading rate is
> faster than writing rate,  the message are likely to be consumed while it's
> still in memory (in memtable) at server side. In this particular case, you
> can optimize further by deactivating compaction for the table.
>
> Regards
>
>  Duy Hai
>
>
>
>
>
>
>
>
> On Sat, Feb 22, 2014 at 5:56 PM, Jagan Ranganathan <ja...@zohocorp.com>wrote:
>
>  Hi,
>
> Thanks for the pointer.
>
> Following are some options given there,
>
>    - If you know where your live data begins, hint Cassandra with a start
>    column, to reduce the scan times and the amount of tombstones to collect.
>    -  A broker will usually have some notion of what's next in the
>    sequence and thus be able to do much more targeted queries, down to a
>    single record if the storage strategy were to choose monotonic sequence
>    numbers.
>
>  We need to do is have some intelligence in using the system and avoid
> tombstones either use the pointed Column Name or use proper start column if
> slice query is used.
>
>  Is that right or I am missing something here?
>
>  Regards,
>  Jagan
>
> ---- On Sat, 22 Feb 2014 20:55:39 +0530 *DuyHai Doan<doanduyhai@gmail.com
> <do...@gmail.com>>* wrote ----
>
>   Jagan
>
>   Queue-like data structures are known to be one of the worst anti
> patterns for Cassandra:
> http://www.datastax.com/dev/blog/cassandra-anti-patterns-queues-and-queue-like-datasets
>
>
>
> On Sat, Feb 22, 2014 at 4:03 PM, Jagan Ranganathan <ja...@zohocorp.com>wrote:
>
>  Hi,
>
>  I need to decouple some of the work being processed from the user thread
> to provide better user experience. For that I need a queuing system with
> the following needs,
>
>    - High Availability
>    - No Data Loss
>    - Better Performance.
>
> Following are some libraries that were considered along with the
> limitation I see,
>
>    - Redis - Data Loss
>    - ZooKeeper - Not advised for Queue system.
>    - TokyoCabinet/SQLite/LevelDB - of this Level DB seem to be performing
>    better. With replication requirement, I probably have to look at Apache
>    ActiveMQ+LevelDB.
>
> After checking on the third option above, I kind of wonder if Cassandra
> with Leveled Compaction offer a similar system. Do you see any issues in
> such a usage or is there other better solutions available.
>
> Will be great to get insights on this.
>
> Regards,
> Jagan
>
>
>
>
>
>

Re: Queuing System

Posted by Jagan Ranganathan <ja...@zohocorp.com>.

Thanks Duy Hai for sharing the details. I have a doubt. If for some reason there is a Network Partition or more than 2 Node failure serving the same partition/load and you ended up writing hinted hand-off.

Is there a possibility of a data loss? If yes, how do we avoid that?

Regards,
Jagan

---- On Sat, 22 Feb 2014 22:48:19 +0530 DuyHai Doan &lt;doanduyhai@gmail.com&gt; wrote ----

Jagan

Few time ago I dealed with a similar queuing design for one customer.

If you never delete messages in the queue, then it is possible to use wide rows with bucketing and increasing monotonic column name to store messages.

CREATE TABLE read_only_queue (

bucket_number int,

insertion_time timeuuid,

message text,

PRIMARY KEY(bucket_number,insertion_time)
);

Let's say that you allow only 100 000 messages per partition (physical row) to avoid too wide rows, then inserting/reading from the table read_only_queue is easy;

For message producer :

1) Start at bucket_number = 1

2) Insert messages with column name = generated timeUUID with micro-second precision (depending on whether the insertion rate is high or not)

3) If message count = 100 000, increment bucket_number by one and go to 2)

For message reader:

1) Start at bucket_number = 1
2) Read messages by slice of N, save the insertion_time of the last read message
3) Use the saved insertion_time to perform next slice query
4) If read messages count = 100 000, increment bucket_number and go to 2). Keep the insertion_time, do not reset it since his value is increasing monotonically

For multiple and concurrent producers &amp; writers, there is a trick. Let's assume you have P concurrent producers and C concurrent consumers.

Assign a numerical ID for each producer and consumer. First producer ID = 1... last producer ID = P. Same for consumers.

- re-use the above algorithm

- each producer/consumer start at bucket_number = his ID

- at the end of the row,
- next bucket_number = current bucker_number + P for producers
- next bucket_number = current bucker_number + C for consumers

The last thing to take care of is compaction configuration to reduce the number of SSTables on disk.

If you achieve to get rid of accumulation effects, e.g reading rate is faster than writing rate, the message are likely to be consumed while it's still in memory (in memtable) at server side. In this particular case, you can optimize further by deactivating compaction for the table.

Regards

Duy Hai

On Sat, Feb 22, 2014 at 5:56 PM, Jagan Ranganathan &lt;jagan@zohocorp.com&gt; wrote:
Hi,

Thanks for the pointer.

Following are some options given there,
If you know where your live data begins, hint Cassandra with a start column, to reduce the scan times and the amount of tombstones to collect.
A broker will usually have some notion of what’s next in the sequence and thus be able to do much more targeted queries, down to a single record if the storage strategy were to choose monotonic sequence numbers.

We need to do is have some intelligence in using the system and avoid tombstones either use the pointed Column Name or use proper start column if slice query is used.

Is that right or I am missing something here?

Regards,
Jagan

---- On Sat, 22 Feb 2014 20:55:39 +0530 DuyHai Doan&lt;doanduyhai@gmail.com&gt; wrote ----

Jagan

Queue-like data structures are known to be one of the worst anti patterns for Cassandra: http://www.datastax.com/dev/blog/cassandra-anti-patterns-queues-and-queue-like-datasets

On Sat, Feb 22, 2014 at 4:03 PM, Jagan Ranganathan &lt;jagan@zohocorp.com&gt; wrote:
Hi,

Will be great to get insights on this.

Regards,
Jagan

Re: Queuing System

Posted by DuyHai Doan <do...@gmail.com>.

Jagan

Few time ago I dealed with a similar queuing design for one customer.

*If you never delete messages in the queue*, then it is possible to use
wide rows with bucketing and increasing monotonic column name to store
messages.

CREATE TABLE *read_only_queue *(
   bucket_number int,
   insertion_time timeuuid,
   message text,
   PRIMARY KEY(bucket_number,insertion_time)
);

 Let's say that you allow only 100 000 messages per partition (physical
row) to avoid too wide rows, then inserting/reading from the table
*read_only_queue
*is easy;

 For message producer :

   1) Start at bucket_number = 1
   2) Insert messages with column name = generated timeUUID with
micro-second precision (depending on whether the insertion rate is high or
not)
   3) If message count = 100 000, increment bucket_number by one and go to
2)

For message reader:

   1) Start at bucket_number = 1
   2) Read messages by slice of *N, *save the *insertion_time *of the last
read message
   3) Use the saved *insertion_time *to perform next slice query
   4) If read messages count = 100 000, increment bucket_number and go to
2). Keep the *insertion_time, *do not reset it since his value is
increasing monotonically

For multiple and concurrent producers & writers, there is a trick. Let's
assume you have *P* concurrent producers and *C* concurrent consumers.

  Assign a numerical ID for each producer and consumer. First producer ID =
1... last producer ID = *P*. Same for consumers.

  - re-use the above algorithm
  - each producer/consumer start at *bucket_number *= his ID
  - at the end of the row,
        - next bucket_number = current bucker_number + *P* for producers
        - next bucket_number = current bucker_number + *C* for consumers

The last thing to take care of is compaction configuration to reduce the
number of SSTables on disk.

If you achieve to get rid of accumulation effects, e.g reading rate is
faster than writing rate,  the message are likely to be consumed while it's
still in memory (in memtable) at server side. In this particular case, you
can optimize further by deactivating compaction for the table.

Regards

 Duy Hai

On Sat, Feb 22, 2014 at 5:56 PM, Jagan Ranganathan <ja...@zohocorp.com>wrote:

> Hi,
>
> Thanks for the pointer.
>
> Following are some options given there,
>
>    - If you know where your live data begins, hint Cassandra with a start
>    column, to reduce the scan times and the amount of tombstones to collect.
>    -  A broker will usually have some notion of what's next in the
>    sequence and thus be able to do much more targeted queries, down to a
>    single record if the storage strategy were to choose monotonic sequence
>    numbers.
>
> We need to do is have some intelligence in using the system and avoid
> tombstones either use the pointed Column Name or use proper start column if
> slice query is used.
>
> Is that right or I am missing something here?
>
> Regards,
> Jagan
>
> ---- On Sat, 22 Feb 2014 20:55:39 +0530 *DuyHai Doan<doanduyhai@gmail.com
> <do...@gmail.com>>* wrote ----
>
>  Jagan
>
>   Queue-like data structures are known to be one of the worst anti
> patterns for Cassandra:
> http://www.datastax.com/dev/blog/cassandra-anti-patterns-queues-and-queue-like-datasets
>
>
>
> On Sat, Feb 22, 2014 at 4:03 PM, Jagan Ranganathan <ja...@zohocorp.com>wrote:
>
>  Hi,
>
>  I need to decouple some of the work being processed from the user thread
> to provide better user experience. For that I need a queuing system with
> the following needs,
>
>    - High Availability
>    - No Data Loss
>    - Better Performance.
>
> Following are some libraries that were considered along with the
> limitation I see,
>
>    - Redis - Data Loss
>    - ZooKeeper - Not advised for Queue system.
>    - TokyoCabinet/SQLite/LevelDB - of this Level DB seem to be performing
>    better. With replication requirement, I probably have to look at Apache
>    ActiveMQ+LevelDB.
>
> After checking on the third option above, I kind of wonder if Cassandra
> with Leveled Compaction offer a similar system. Do you see any issues in
> such a usage or is there other better solutions available.
>
> Will be great to get insights on this.
>
> Regards,
> Jagan
>
>
>
>

Re: Queuing System

Posted by Jagan Ranganathan <ja...@zohocorp.com>.

Hi,

Thanks for the pointer.

We need to do is have some intelligence in using the system and avoid tombstones either use the pointed Column Name or use proper start column if slice query is used.

Is that right or I am missing something here?

Regards,
Jagan

---- On Sat, 22 Feb 2014 20:55:39 +0530 DuyHai Doan&lt;doanduyhai@gmail.com&gt; wrote ----

Jagan

Queue-like data structures are known to be one of the worst anti patterns for Cassandra: http://www.datastax.com/dev/blog/cassandra-anti-patterns-queues-and-queue-like-datasets

On Sat, Feb 22, 2014 at 4:03 PM, Jagan Ranganathan &lt;jagan@zohocorp.com&gt; wrote:
Hi,

Will be great to get insights on this.

Regards,
Jagan

Re: Queuing System

Posted by DuyHai Doan <do...@gmail.com>.

Jagan

 Queue-like data structures are known to be one of the worst anti patterns
for Cassandra:
http://www.datastax.com/dev/blog/cassandra-anti-patterns-queues-and-queue-like-datasets



On Sat, Feb 22, 2014 at 4:03 PM, Jagan Ranganathan <ja...@zohocorp.com>wrote:

> Hi,
>
> I need to decouple some of the work being processed from the user thread
> to provide better user experience. For that I need a queuing system with
> the following needs,
>
>    - High Availability
>    - No Data Loss
>    - Better Performance.
>
> Following are some libraries that were considered along with the
> limitation I see,
>
>    - Redis - Data Loss
>    - ZooKeeper - Not advised for Queue system.
>    - TokyoCabinet/SQLite/LevelDB - of this Level DB seem to be performing
>    better. With replication requirement, I probably have to look at Apache
>    ActiveMQ+LevelDB.
>
> After checking on the third option above, I kind of wonder if Cassandra
> with Leveled Compaction offer a similar system. Do you see any issues in
> such a usage or is there other better solutions available.
>
> Will be great to get insights on this.
>
> Regards,
> Jagan
>

Re: Queuing System

Posted by Jagan Ranganathan <ja...@zohocorp.com>.

Thanks Tupshin for your assistance. As I mentioned in the other mail, Yes I am planning to use RabbitMQ for my messaging system. But I wonder which will give better performance if writing directly into Rabbit with Ack support Vs a temporary Queue in Cassandra first and then dequeue and publish in Rabbit.I use Rabbit for Messaging because of the Routing and Push model communication etc. So I am thinking of using Cassandra as a temporary Queue which will give fast write performance with no data loss Vs waiting for Rabbit Ack @ application level or handling Rabbit re-connection Vs Cassandra hinted handoff writes.

So Cassandra might aggregate all my msg queue temporarily before I publish them to Rabbit. Is this fine? If so, please share your insight on which model &amp; access pattern will be a better fit for this usage. Throughput requirements may be around say 100 ops/sec.

Regards,
Jagan

---- On Sat, 22 Feb 2014 21:10:36 +0530 Tupshin Harper&lt;tupshin@tupshin.com&gt; wrote ----

While, historically, it has been true that queuing in Cassandra has been an anti-pattern, it is also true that Leveled Compaction addresses the worst aspect of frequent deletes in Cassandra, and that overall, queuing in Cassandra is nowhere near the anti-pattern that it used to be. This is something that I've been meaning to write about more extensively.

If your requirements are more around availability (particularly multi-dc) and relability with moderate (not extreme) performance, it is quite possible to build a pretty decent system on top of Cassandra. You don't mention your throughput requirements, nor additional semantics that might be necessary (e.g. deliver at-least-once vs deliver exactly once), but Cassandra 2.0's lightweight transactions provide a CAS primitive that can be used to ensure deliver-once if that is a requirement.

I'd be happy to continue discussing appropriate data-models and access patterns if you decide to go down this path.

-Tupshin

On Sat, Feb 22, 2014 at 10:03 AM, Jagan Ranganathan &lt;jagan@zohocorp.com&gt; wrote:
Hi,

Will be great to get insights on this.

Regards,
Jagan

Re: Queuing System

Posted by Tupshin Harper <tu...@tupshin.com>.

While, historically, it has been true that queuing in Cassandra has been an
anti-pattern, it is also true that Leveled Compaction addresses the worst
aspect of frequent deletes in Cassandra, and that overall, queuing in
Cassandra is nowhere near the anti-pattern that it used to be. This is
something that I've been meaning to write about more extensively.

 If your requirements are more around availability (particularly multi-dc)
and relability with moderate (not extreme) performance, it is quite
possible to build a pretty decent system on top of Cassandra. You don't
mention your throughput requirements, nor additional semantics that might
be necessary (e.g. deliver at-least-once vs deliver exactly once), but
Cassandra 2.0's lightweight transactions provide a CAS primitive that can
be used to ensure deliver-once if that is a requirement.

I'd be happy to continue discussing appropriate data-models and access
patterns if you decide to go down this path.

-Tupshin

On Sat, Feb 22, 2014 at 10:03 AM, Jagan Ranganathan <ja...@zohocorp.com>wrote:

> Hi,
>
> I need to decouple some of the work being processed from the user thread
> to provide better user experience. For that I need a queuing system with
> the following needs,
>
>    - High Availability
>    - No Data Loss
>    - Better Performance.
>
> Following are some libraries that were considered along with the
> limitation I see,
>
>    - Redis - Data Loss
>    - ZooKeeper - Not advised for Queue system.
>    - TokyoCabinet/SQLite/LevelDB - of this Level DB seem to be performing
>    better. With replication requirement, I probably have to look at Apache
>    ActiveMQ+LevelDB.
>
> After checking on the third option above, I kind of wonder if Cassandra
> with Leveled Compaction offer a similar system. Do you see any issues in
> such a usage or is there other better solutions available.
>
> Will be great to get insights on this.
>
> Regards,
> Jagan
>