You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Christian Rolf (JIRA)" <ji...@apache.org> on 2014/02/06 14:46:09 UTC
[jira] [Updated] (CASSANDRA-6665) Batching in CqlRecordWriter
[ https://issues.apache.org/jira/browse/CASSANDRA-6665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Christian Rolf updated CASSANDRA-6665:
--------------------------------------
Attachment: batchWrite.txt
Never submitted a patch here before, let me know if the format's bad.
> Batching in CqlRecordWriter
> ---------------------------
>
> Key: CASSANDRA-6665
> URL: https://issues.apache.org/jira/browse/CASSANDRA-6665
> Project: Cassandra
> Issue Type: Improvement
> Components: Hadoop
> Environment: Cluster of 12 nodes, each node with 256-384 vnodes. RPC threads capped at 2048.
> Reporter: Christian Rolf
> Priority: Minor
> Attachments: batchWrite.txt
>
>
> We're writing from Pig map tasks, about 20 million records of one integer each.
> For the case of 12 nodes, with 256-384 vnodes per node, we get around 4000 threads per mapper. This obviously overloads the nodes, since the number of RPC threads are capped, and the write fails.
> Also, each transfer is only in the order of a few bytes of payload. Clearly batching is a good solution.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)