You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by "Himanshu Bafna (HB)" <hb...@yahoo.com> on 2013/10/24 07:36:40 UTC

Loading leveldb into Spark..

Hi 


I have use case where in our data is stored in Leveldb with data size close to 100TB for 20 days.
Is it possible to integrate or load leveldb data into spark.

OR should I first load the data from leveldb into HDFS for spark consumption.

Regards
Himanshu

Re: Loading leveldb into Spark..

Posted by Patrick Wendell <pw...@gmail.com>.

load it from HDFS...


On Wed, Oct 23, 2013 at 10:40 PM, Patrick Wendell <pw...@gmail.com>wrote:

> You could try reading it directly in Spark via JDBC, but for this size
> data you may want to do a higher throughput transfer into HDFS first and
> then load it from Spark.
>
>
> On Wed, Oct 23, 2013 at 10:36 PM, Himanshu Bafna (HB) <hb...@yahoo.com>wrote:
>
>> Hi
>>
>> I have use case where in our data is stored in Leveldb with data size
>> close to 100TB for 20 days.
>> Is it possible to integrate or load leveldb data into spark.
>>
>> OR should I first load the data from leveldb into HDFS for spark
>> consumption.
>>
>> Regards
>> Himanshu
>>
>>
>

Re: Loading leveldb into Spark..

Posted by Patrick Wendell <pw...@gmail.com>.

You could try reading it directly in Spark via JDBC, but for this size data
you may want to do a higher throughput transfer into HDFS first and then
load it from Spark.

On Wed, Oct 23, 2013 at 10:36 PM, Himanshu Bafna (HB) <hb...@yahoo.com>wrote:

> Hi
>
> I have use case where in our data is stored in Leveldb with data size
> close to 100TB for 20 days.
> Is it possible to integrate or load leveldb data into spark.
>
> OR should I first load the data from leveldb into HDFS for spark
> consumption.
>
> Regards
> Himanshu
>
>