You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by ravikumar visweswara <ta...@gmail.com> on 2011/12/23 16:12:33 UTC
cassandra data to hadoop.
Hello All,
I have a situation to dump cassandra data to hadoop cluster for further
analytics. Lot of other relevant data which is not present in cassandra is
already available in hdfs for analysis. Both are independent clusters right
now.
Is there a suggested way to get the data periodically or continuously to
HDFS from cassandra? Any ideas or references will be very helpful for me.
Thanks and Regards
R
Re: cassandra data to hadoop.
Posted by Brian O'Neill <bo...@alumni.brown.edu>.
I'm not sure this is much help, but we actually run Hadoop jobs to load and
extract data to and from HDFS. You can use ColumnFamilyInputFormat to race
over the data in Cassandra and output it to a file. That doesn't solve the
continuous problem, but should give you a batch mechanism to refresh the
data in HDFS. I presume its even speedier if you are running enterprise,
because the Hadoop process is collocated with Cassandra.
-brian
On Fri, Dec 23, 2011 at 10:12 AM, ravikumar visweswara <
talk2hadoop@gmail.com> wrote:
> Hello All,
>
> I have a situation to dump cassandra data to hadoop cluster for further
> analytics. Lot of other relevant data which is not present in cassandra is
> already available in hdfs for analysis. Both are independent clusters right
> now.
> Is there a suggested way to get the data periodically or continuously to
> HDFS from cassandra? Any ideas or references will be very helpful for me.
>
> Thanks and Regards
> R
>
--
Brian ONeill
Lead Architect, Health Market Science (http://healthmarketscience.com)
mobile:215.588.6024
blog: http://weblogs.java.net/blog/boneill42/
blog: http://brianoneill.blogspot.com/
Re: cassandra data to hadoop.
Posted by Jeremy Hanna <je...@gmail.com>.
We currently have cassandra nodes co-located with hadoop nodes and do a lot of data analytics with it. We've looked at brisk - brisk still open-source and available but datastax is putting its resources in a closed version of brisk as part of datastax enterprise. We'll likely be moving to that over the next month. That said, there's nothing wrong with just using vanilla hadoop with cassandra either - it's worked for us for almost a year now. Brisk/Datastax enterprise is simpler though, especially for doing a lot with the two together.
On Dec 23, 2011, at 11:33 AM, Praveen Sadhu wrote:
> Have you tried Brisk?
>
>
>
> On Dec 23, 2011, at 9:30 AM, "Jeremy Hanna" <je...@gmail.com> wrote:
>
>> We do this all the time. Take a look at http://wiki.apache.org/cassandra/HadoopSupport for some details - you can use mapreduce or pig to get data out of cassandra. If it's going to a separate hadoop cluster, I don't think you'd need to co-locate task trackers or data nodes on your cassandra nodes - it would just need to copy over the network though. We also use oozie for job scheduling, fwiw.
>>
>> On Dec 23, 2011, at 9:12 AM, ravikumar visweswara wrote:
>>
>>> Hello All,
>>>
>>> I have a situation to dump cassandra data to hadoop cluster for further analytics. Lot of other relevant data which is not present in cassandra is already available in hdfs for analysis. Both are independent clusters right now.
>>> Is there a suggested way to get the data periodically or continuously to HDFS from cassandra? Any ideas or references will be very helpful for me.
>>>
>>> Thanks and Regards
>>> R
>>
>>
Re: cassandra data to hadoop.
Posted by Praveen Sadhu <ps...@netflix.com>.
Have you tried Brisk?
On Dec 23, 2011, at 9:30 AM, "Jeremy Hanna" <je...@gmail.com> wrote:
> We do this all the time. Take a look at http://wiki.apache.org/cassandra/HadoopSupport for some details - you can use mapreduce or pig to get data out of cassandra. If it's going to a separate hadoop cluster, I don't think you'd need to co-locate task trackers or data nodes on your cassandra nodes - it would just need to copy over the network though. We also use oozie for job scheduling, fwiw.
>
> On Dec 23, 2011, at 9:12 AM, ravikumar visweswara wrote:
>
>> Hello All,
>>
>> I have a situation to dump cassandra data to hadoop cluster for further analytics. Lot of other relevant data which is not present in cassandra is already available in hdfs for analysis. Both are independent clusters right now.
>> Is there a suggested way to get the data periodically or continuously to HDFS from cassandra? Any ideas or references will be very helpful for me.
>>
>> Thanks and Regards
>> R
>
>
Re: cassandra data to hadoop.
Posted by Jeremy Hanna <je...@gmail.com>.
We're using cdh3 right now and we're in the process of testing out brisk/datastax enterprise right now. I know some people who copy data from Cassandra to a completely separate Hadoop cluster in EC2 at least. It just takes a bit longer because of the intermachine copy. That's better to do in the case that you want to keep them completely separate. You can, as described in the wiki link, just have task trackers and data nodes on your cassandra nodes, but you'll need to group your cassandra nodes by analytic nodes for one virtual DC and another virtual DC for realtime nodes - as described in the wiki link as well. We actually use cdh3/cassandra 0.8 in that setup right now and it's working for us.
I don't know that flume/hue would need much tweaking to work with brisk/datastax enterprise. We found one patch we needed to have applied to make the oozie configuration better, but you'd really have to try it out before you know for sure.
Hope that helps - if you have any other specific questions, ask away.
Jeremy
On Dec 24, 2011, at 1:20 AM, ravikumar visweswara wrote:
> Jeremy,
>
> We use cloudera distribution for our hadoop cluster and may not be possible to migrate to brisk quickly because of flume/hue dependencies. Did you successfully pull the data from independent cassandra cluster and dump into completely disconnected hadoop cluster? It will be really helpful if you elaborate on how to achieve this.
>
> -R
>
> On Fri, Dec 23, 2011 at 9:28 AM, Jeremy Hanna <je...@gmail.com> wrote:
> We do this all the time. Take a look at http://wiki.apache.org/cassandra/HadoopSupport for some details - you can use mapreduce or pig to get data out of cassandra. If it's going to a separate hadoop cluster, I don't think you'd need to co-locate task trackers or data nodes on your cassandra nodes - it would just need to copy over the network though. We also use oozie for job scheduling, fwiw.
>
> On Dec 23, 2011, at 9:12 AM, ravikumar visweswara wrote:
>
> > Hello All,
> >
> > I have a situation to dump cassandra data to hadoop cluster for further analytics. Lot of other relevant data which is not present in cassandra is already available in hdfs for analysis. Both are independent clusters right now.
> > Is there a suggested way to get the data periodically or continuously to HDFS from cassandra? Any ideas or references will be very helpful for me.
> >
> > Thanks and Regards
> > R
>
>
Re: cassandra data to hadoop.
Posted by Mohit Anchlia <mo...@gmail.com>.
You could read using Cassandra client and write to HDFS using Hadoop FS Api.
On Fri, Dec 23, 2011 at 11:20 PM, ravikumar visweswara
<ta...@gmail.com> wrote:
> Jeremy,
>
> We use cloudera distribution for our hadoop cluster and may not be possible
> to migrate to brisk quickly because of flume/hue dependencies. Did you
> successfully pull the data from independent cassandra cluster and dump into
> completely disconnected hadoop cluster? It will be really helpful if you
> elaborate on how to achieve this.
>
> -R
>
>
> On Fri, Dec 23, 2011 at 9:28 AM, Jeremy Hanna <je...@gmail.com>
> wrote:
>>
>> We do this all the time. Take a look at
>> http://wiki.apache.org/cassandra/HadoopSupport for some details - you can
>> use mapreduce or pig to get data out of cassandra. If it's going to a
>> separate hadoop cluster, I don't think you'd need to co-locate task trackers
>> or data nodes on your cassandra nodes - it would just need to copy over the
>> network though. We also use oozie for job scheduling, fwiw.
>>
>> On Dec 23, 2011, at 9:12 AM, ravikumar visweswara wrote:
>>
>> > Hello All,
>> >
>> > I have a situation to dump cassandra data to hadoop cluster for further
>> > analytics. Lot of other relevant data which is not present in cassandra is
>> > already available in hdfs for analysis. Both are independent clusters right
>> > now.
>> > Is there a suggested way to get the data periodically or continuously to
>> > HDFS from cassandra? Any ideas or references will be very helpful for me.
>> >
>> > Thanks and Regards
>> > R
>>
>
Re: cassandra data to hadoop.
Posted by ravikumar visweswara <ta...@gmail.com>.
Jeremy,
We use cloudera distribution for our hadoop cluster and may not be possible
to migrate to brisk quickly because of flume/hue dependencies. Did you
successfully pull the data from independent cassandra cluster and dump into
completely disconnected hadoop cluster? It will be really helpful if you
elaborate on how to achieve this.
-R
On Fri, Dec 23, 2011 at 9:28 AM, Jeremy Hanna <je...@gmail.com>wrote:
> We do this all the time. Take a look at
> http://wiki.apache.org/cassandra/HadoopSupport for some details - you can
> use mapreduce or pig to get data out of cassandra. If it's going to a
> separate hadoop cluster, I don't think you'd need to co-locate task
> trackers or data nodes on your cassandra nodes - it would just need to copy
> over the network though. We also use oozie for job scheduling, fwiw.
>
> On Dec 23, 2011, at 9:12 AM, ravikumar visweswara wrote:
>
> > Hello All,
> >
> > I have a situation to dump cassandra data to hadoop cluster for further
> analytics. Lot of other relevant data which is not present in cassandra is
> already available in hdfs for analysis. Both are independent clusters right
> now.
> > Is there a suggested way to get the data periodically or continuously to
> HDFS from cassandra? Any ideas or references will be very helpful for me.
> >
> > Thanks and Regards
> > R
>
>
Re: cassandra data to hadoop.
Posted by Jeremy Hanna <je...@gmail.com>.
We do this all the time. Take a look at http://wiki.apache.org/cassandra/HadoopSupport for some details - you can use mapreduce or pig to get data out of cassandra. If it's going to a separate hadoop cluster, I don't think you'd need to co-locate task trackers or data nodes on your cassandra nodes - it would just need to copy over the network though. We also use oozie for job scheduling, fwiw.
On Dec 23, 2011, at 9:12 AM, ravikumar visweswara wrote:
> Hello All,
>
> I have a situation to dump cassandra data to hadoop cluster for further analytics. Lot of other relevant data which is not present in cassandra is already available in hdfs for analysis. Both are independent clusters right now.
> Is there a suggested way to get the data periodically or continuously to HDFS from cassandra? Any ideas or references will be very helpful for me.
>
> Thanks and Regards
> R