Cassandra table as Queue

[Radoslav Smilyanov <ra...@novarto.com>]

Hello,

I need to have a table that is acting as a queue for specific data. Data
that has to be stored in this table are some unique ids that have to be
predefined and whenever it is requested one id has to be obtained by the
"queue" and new one has to be added. This queue table will have fixed size
of 50 000 entries.

I see that it is not recommended at all to use cassandra table for a queue,
but I need to find a design for my data that will not cause performance
issues caused by tombstones.

I am using cassandra 2.1.6 with java driver and I am afraid that at some
point of time I will start experiencing performance issues caused by many
tombstones.
Current design of my table with one column is not good enough for querying
the data since now I am using:
1. "select * from <table> limit 1" which returns me first id in the table
2. "delete from <table> where id = <id_from step 1>"

Did someone try to implement a queue with cassandra table that is working
productively now without any performance issues? I will appreciate some
hints how can I achieve good performance in cassandra for a queue table.

Thanks,
Rado

Re: Cassandra table as Queue

[Narendra Sharma <na...@gmail.com>]

I would suggest looking at Comcast Message Bus schema definition.

https://github.com/Comcast/cmb

-Naren

On Thu, Aug 20, 2015 at 10:30 AM, algermissen1971 <
algermissen1971@icloud.com> wrote:

> Hi Rado,
>
> On 20 Aug 2015, at 15:05, Radoslav Smilyanov <
> radoslav.smilyanov@novarto.com> wrote:
>
> > Hello,
> >
> > I need to have a table that is acting as a queue for specific data. Data
> that has to be stored in this table are some unique ids that have to be
> predefined and whenever it is requested one id has to be obtained by the
> "queue" and new one has to be added. This queue table will have fixed size
> of 50 000 entries.
> >
> > I see that it is not recommended at all to use cassandra table for a
> queue, but I need to find a design for my data that will not cause
> performance issues caused by tombstones.
> >
> > I am using cassandra 2.1.6 with java driver and I am afraid that at some
> point of time I will start experiencing performance issues caused by many
> tombstones.
> > Current design of my table with one column is not good enough for
> querying the data since now I am using:
> > 1. "select * from <table> limit 1" which returns me first id in the table
> > 2. "delete from <table> where id = <id_from step 1>"
> >
> > Did someone try to implement a queue with cassandra table that is
> working productively now without any performance issues? I will appreciate
> some hints how can I achieve good performance in cassandra for a queue
> table.
> >
>
> I came up with a design last year that I am using without problems with a
> java-driver -based implementation in production since several months.
>
> Two caveats:
>
> - Our environment is not high-volume or high-frequency. Message counts per
> minute come in dozens, at most. So the design is not tested in heavy
> scenarios. We merely needed something based on the existing tech-stack.
> - The Ruby version has a logical bug, mentioned in the README.
>
> https://github.com/algermissen/cassandra-ruby-sharded-workers
>
> Given the tombstone problem I what I know by now, I'd rather not use a TTL
> on the messages but remove outdated time shards completely after e.g. a
> week. But since reads never really go to an outdated shard, the tombstones
> do not slow down the reads.
>
> Hope that helps.
>
> Jan
>
>
>
>
>
>
> > Thanks,
> > Rado
>
>


-- 
Narendra Sharma
Software Engineer
*http://www.aeris.com <http://www.aeris.com>*
*http://narendrasharma.blogspot.com/ <http://narendrasharma.blogspot.com/>*

Re: Cassandra table as Queue

[algermissen1971 <al...@icloud.com>]

Hi Rado,

On 20 Aug 2015, at 15:05, Radoslav Smilyanov <ra...@novarto.com> wrote:

> Hello,
> 
> I need to have a table that is acting as a queue for specific data. Data that has to be stored in this table are some unique ids that have to be predefined and whenever it is requested one id has to be obtained by the "queue" and new one has to be added. This queue table will have fixed size of 50 000 entries.
> 
> I see that it is not recommended at all to use cassandra table for a queue, but I need to find a design for my data that will not cause performance issues caused by tombstones.
> 
> I am using cassandra 2.1.6 with java driver and I am afraid that at some point of time I will start experiencing performance issues caused by many tombstones. 
> Current design of my table with one column is not good enough for querying the data since now I am using:
> 1. "select * from <table> limit 1" which returns me first id in the table
> 2. "delete from <table> where id = <id_from step 1>"
> 
> Did someone try to implement a queue with cassandra table that is working productively now without any performance issues? I will appreciate some hints how can I achieve good performance in cassandra for a queue table.
> 

I came up with a design last year that I am using without problems with a java-driver -based implementation in production since several months.

Two caveats:

- Our environment is not high-volume or high-frequency. Message counts per minute come in dozens, at most. So the design is not tested in heavy scenarios. We merely needed something based on the existing tech-stack.
- The Ruby version has a logical bug, mentioned in the README.

https://github.com/algermissen/cassandra-ruby-sharded-workers

Given the tombstone problem I what I know by now, I'd rather not use a TTL on the messages but remove outdated time shards completely after e.g. a week. But since reads never really go to an outdated shard, the tombstones do not slow down the reads.

Hope that helps.

Jan






> Thanks,
> Rado