How many BATCH inserts in to many?

[Alan Ristić <al...@gmail.com>]

Hi,
I'm implementing Facebook style notifications/activities (e.g "your friend
liked this article") in our app. I considered message queue for this task
before exploring how BATCH insert in C* could perform. Could I leave Queue
altogether and just use batch insert? Is like...5000 insers too much
overload for this kind of work? And what is limiting the number of inserts?

oh..the data model in C* is straightforward - "give me all notifications
for user A"

Lp,
*Alan Ristić*

*m*: 040 423 688

Re: How many BATCH inserts in to many?

[aaron morton <aa...@thelastpickle.com>]

With regard to a large number of records in a batch mutation there are some potential issues. 

Each row becomes a task in the write thread pool on each replica. If a single client sends 1,000 rows in a mutation it will take time for the (default) 32 threads in the write pool to work through the mutations. While they are doing this other clients / requests will appear to be starved / stalled. 

There are also issues with the max message size in thrift and cql over thrift. 

IMHO as a rule of thumb dont go over a few hundred if you have a high number of concurrent writers. 

Cheers

-----------------
Aaron Morton
Freelance Cassandra Developer
New Zealand

@aaronmorton
http://www.thelastpickle.com

On 14/01/2013, at 12:56 AM, Radim Kolar <hs...@filez.com> wrote:

> do not use cassandra for implementing queueing system with high throughput. It does not scale because of tombstone management. Use hornetQ, its amazingly fast broker but it has quite slow persistence if you want to create queues significantly larger then your memory and use selectors for searching for specific messages in them.
> 
> My point is for implementing queue message broker is what you want.

Re: How many BATCH inserts in to many?

[Radim Kolar <hs...@filez.com>]

do not use cassandra for implementing queueing system with high 
throughput. It does not scale because of tombstone management. Use 
hornetQ, its amazingly fast broker but it has quite slow persistence if 
you want to create queues significantly larger then your memory and use 
selectors for searching for specific messages in them.

My point is for implementing queue message broker is what you want.

Re: How many BATCH inserts in to many?

[Alan Ristić <al...@gmail.com>]

wauu! huh...em...wau - can't say anything else. Tnx for insight, it
certanly gives me some "frame" how batch is performing. Tnx a lot.


2013/1/12 Carlos Rolo <c....@ocom.com>

> 63k rec




Lp,
*Alan Ristić*

*m*: 040 423 688

Re: How many BATCH inserts in to many?

[Carlos Rolo <c....@ocom.com>]

Hello,

I have managed to insert up to 63k records without any problem. In certain workloads I found that massive batch inserts perform way better than lots of not-so-massive inserts. I guess it also depends on your setup. Just try it.

Alan Ristić <al...@gmail.com> escreveu:



Hi,
I'm implementing Facebook style notifications/activities (e.g "your friend liked this article") in our app. I considered message queue for this task before exploring how BATCH insert in C* could perform. Could I leave Queue altogether and just use batch insert? Is like...5000 insers too much overload for this kind of work? And what is limiting the number of inserts?

oh..the data model in C* is straightforward - "give me all notifications for user A"

Lp,
Alan Ristić

m: 040 423 688