Queue suggestion in Cassandra

[Daning Wang <da...@netseer.com>]

We try to implement an ordered queue system in Cassandra(ver 0.8.5). In
initial design  we use a row as queue,  a column for each item in queue.
that means creating new column when inserting  item and delete column when
top item is popped. Since columns are sorted in Cassandra we got the ordered
queue.

It works fine until queue size reaches 50K, then we got high CPU usage and
constant GC, that makes the whole Cassandra server very slow and not
responsive, we have to do full compaction to fix this problem.

Due to this performance issue that this queue is not useful for us. We are
looking for other designs. I want to know if anybody has implemented a large
ordered queue successfully.

Let me know if you have suggestion,

Thank you in advance.

Daning

RE: Queue suggestion in Cassandra

[Scott Fines <Sc...@nisc.coop>]

ZooKeeper + Menagerie might work, but it's unlikely to have the throughput you require. I'd have a look at Kafka (http://incubator.apache.org/<http://incubator.apache.org/kafka/>), Kestrel(https://github.com/robey/kestrel/), or one of the AMQP-compliant systems like ActiveMQ (activemq.apache.org)  or RabbitMQ(http://www.rabbitmq.com/). I'm a fan of Kafka myself.

Thanks for the menagerie shoutout though...:)

Scott Fines

________________________________
From: Milind Parikh [milindparikh@gmail.com]
Sent: Friday, September 16, 2011 6:12 PM
To: user@cassandra.apache.org
Subject: Re: Queue suggestion in Cassandra

use zookeeper. Scott Fines has a great library on top of zk.

On Fri, Sep 16, 2011 at 7:08 PM, Daning Wang <da...@netseer.com>> wrote:
We try to implement an ordered queue system in Cassandra(ver 0.8.5). In initial design  we use a row as queue,  a column for each item in queue. that means creating new column when inserting  item and delete column when top item is popped. Since columns are sorted in Cassandra we got the ordered queue.

It works fine until queue size reaches 50K, then we got high CPU usage and constant GC, that makes the whole Cassandra server very slow and not responsive, we have to do full compaction to fix this problem.

Due to this performance issue that this queue is not useful for us. We are looking for other designs. I want to know if anybody has implemented a large ordered queue successfully.

Let me know if you have suggestion,

Thank you in advance.

Daning

Re: Queue suggestion in Cassandra

[Milind Parikh <mi...@gmail.com>]

use zookeeper. Scott Fines has a great library on top of zk.

On Fri, Sep 16, 2011 at 7:08 PM, Daning Wang <da...@netseer.com> wrote:

> We try to implement an ordered queue system in Cassandra(ver 0.8.5). In
> initial design  we use a row as queue,  a column for each item in queue.
> that means creating new column when inserting  item and delete column when
> top item is popped. Since columns are sorted in Cassandra we got the ordered
> queue.
>
> It works fine until queue size reaches 50K, then we got high CPU usage and
> constant GC, that makes the whole Cassandra server very slow and not
> responsive, we have to do full compaction to fix this problem.
>
> Due to this performance issue that this queue is not useful for us. We are
> looking for other designs. I want to know if anybody has implemented a large
> ordered queue successfully.
>
> Let me know if you have suggestion,
>
> Thank you in advance.
>
> Daning
>
>
>

Re: Queue suggestion in Cassandra

[David Allsopp <dn...@gmail.com>]

I have had similar experiences. On the advice of the Cassandra team, I now
maintain the queue itself in-memory, but persist the data items in Cassandra
(one per row). When re-starting the system I pull data from Cassandra to
re-construct the ordered queue in-memory. In some cases I write columns to
the rows indicate the progress of an item (unprocessed, pending, complete,
error, etc).  However, my approach may not work well with transient items
that don't need storing once processed. My items need persisting even after
their initial processing.

On 17 September 2011 00:08, Daning Wang <da...@netseer.com> wrote:

> We try to implement an ordered queue system in Cassandra(ver 0.8.5). In
> initial design  we use a row as queue,  a column for each item in queue.
> that means creating new column when inserting  item and delete column when
> top item is popped. Since columns are sorted in Cassandra we got the ordered
> queue.
>
> It works fine until queue size reaches 50K, then we got high CPU usage and
> constant GC, that makes the whole Cassandra server very slow and not
> responsive, we have to do full compaction to fix this problem.
>
> Due to this performance issue that this queue is not useful for us. We are
> looking for other designs. I want to know if anybody has implemented a large
> ordered queue successfully.
>
> Let me know if you have suggestion,
>
> Thank you in advance.
>
> Daning
>
>
>