read huge data from CSV and write into Cassandra

[Akshay Ballarpure <ak...@tcs.com>]

How to read data from large CSV file which is having 100+ columns and 
millions of rows and inserting into Cassandra every 1 minute.

Thanks & Regards
Akshay Ghanshyam Ballarpure
Tata Consultancy Services
Cell:- 9985084075
Mailto: akshay.ballarpure@tcs.com
Website: http://www.tcs.com
____________________________________________
Experience certainty.   IT Services
                        Business Solutions
                        Consulting
____________________________________________
=====-----=====-----=====
Notice: The information contained in this e-mail
message and/or attachments to it may contain 
confidential or privileged information. If you are 
not the intended recipient, any dissemination, use, 
review, distribution, printing or copying of the 
information contained in this e-mail message 
and/or attachments to it are strictly prohibited. If 
you have received this communication in error, 
please notify us by reply e-mail or telephone and 
immediately and permanently delete the message 
and any attachments. Thank you

Re: read huge data from CSV and write into Cassandra

[Jack Krupansky <ja...@basetechnology.com>]

Read the csv file using a Java app and then index the rows using the Cassandra Java driver with multiple, parallel input streams.

Oh, and make sure to provision your cluster with enough nodes to handle your desired ingestion and query rates. Do a proof of concept with a six node cluster with RF=2 to see what ingestion and query rates you can get for a fraction of your data and then scale from there. Although a 12-node cluster with RF=3 would be more realistic. RF=2 is not for production – doesn’t permit any failures, while RF=3 permits quorum operations with a single node failure. But RF=2 at least lets you test with a more realistic scenario of coordinator nodes and inter-node traffic.

And if your total row count does manage to fit on one machine (or three nodes with RF=3), at least make sure you have enough CPU cores and I/O bandwidth to handle your desired ingestion and query rate.

-- Jack Krupansky

From: Akshay Ballarpure 
Sent: Friday, July 25, 2014 5:26 AM
To: user@cassandra.apache.org 
Subject: read huge data from CSV and write into Cassandra

How to read data from large CSV file which is having 100+ columns and millions of rows and inserting into Cassandra every 1 minute. 

Thanks & Regards
Akshay Ghanshyam Ballarpure
Tata Consultancy Services
Cell:- 9985084075
Mailto: akshay.ballarpure@tcs.com
Website: http://www.tcs.com
____________________________________________
Experience certainty.        IT Services
                       Business Solutions
                       Consulting
____________________________________________ 
=====-----=====-----=====
Notice: The information contained in this e-mail
message and/or attachments to it may contain 
confidential or privileged information. If you are 
not the intended recipient, any dissemination, use, 
review, distribution, printing or copying of the 
information contained in this e-mail message 
and/or attachments to it are strictly prohibited. If 
you have received this communication in error, 
please notify us by reply e-mail or telephone and 
immediately and permanently delete the message 
and any attachments. Thank you