You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Mohanraj Ragupathiraj <mo...@gmail.com> on 2016/05/18 04:14:34 UTC
SPARK - DataFrame for BulkLoad
I have 100 million records to be inserted to a HBase table (PHOENIX) as a
result of a Spark Job. I would like to know if i convert it to a Dataframe
and save it, will it do Bulk load (or) it is not the efficient way to write
data to a HBase table
--
Thanks and Regards
Mohan
Re: SPARK - DataFrame for BulkLoad
Posted by Michael Segel <ms...@hotmail.com>.
Yes, but he’s using phoenix which may not work cleanly with your HBase spark module.
They key issue here may be Phoenix which is separate from HBase.
> On May 18, 2016, at 5:36 AM, Ted Yu <yu...@gmail.com> wrote:
>
> Please see HBASE-14150
>
> The hbase-spark module would be available in the upcoming hbase 2.0 release.
>
> On Tue, May 17, 2016 at 11:48 PM, Takeshi Yamamuro <linguin.m.s@gmail.com <ma...@gmail.com>> wrote:
> Hi,
>
> Have you checked this?
> http://mail-archives.apache.org/mod_mbox/spark-user/201311.mbox/%3CCACyZca3ASKwD-tuJHQi1805BN7ScTguAoRuHd5xTxCSUL1aNvQ@mail.gmail.com%3E <http://mail-archives.apache.org/mod_mbox/spark-user/201311.mbox/%3CCACyZca3ASKwD-tuJHQi1805BN7ScTguAoRuHd5xTxCSUL1aNvQ@mail.gmail.com%3E>
>
> // maropu
>
> On Wed, May 18, 2016 at 1:14 PM, Mohanraj Ragupathiraj <mohanaugust@gmail.com <ma...@gmail.com>> wrote:
> I have 100 million records to be inserted to a HBase table (PHOENIX) as a result of a Spark Job. I would like to know if i convert it to a Dataframe and save it, will it do Bulk load (or) it is not the efficient way to write data to a HBase table
>
> --
> Thanks and Regards
> Mohan
>
>
>
> --
> ---
> Takeshi Yamamuro
>
Re: SPARK - DataFrame for BulkLoad
Posted by Ted Yu <yu...@gmail.com>.
Please see HBASE-14150
The hbase-spark module would be available in the upcoming hbase 2.0 release.
On Tue, May 17, 2016 at 11:48 PM, Takeshi Yamamuro <li...@gmail.com>
wrote:
> Hi,
>
> Have you checked this?
>
> http://mail-archives.apache.org/mod_mbox/spark-user/201311.mbox/%3CCACyZca3ASKwD-tuJHQi1805BN7ScTguAoRuHd5xTxCSUL1aNvQ@mail.gmail.com%3E
>
> // maropu
>
> On Wed, May 18, 2016 at 1:14 PM, Mohanraj Ragupathiraj <
> mohanaugust@gmail.com> wrote:
>
>> I have 100 million records to be inserted to a HBase table (PHOENIX) as a
>> result of a Spark Job. I would like to know if i convert it to a Dataframe
>> and save it, will it do Bulk load (or) it is not the efficient way to write
>> data to a HBase table
>>
>> --
>> Thanks and Regards
>> Mohan
>>
>
>
>
> --
> ---
> Takeshi Yamamuro
>
Re: SPARK - DataFrame for BulkLoad
Posted by Takeshi Yamamuro <li...@gmail.com>.
Hi,
Have you checked this?
http://mail-archives.apache.org/mod_mbox/spark-user/201311.mbox/%3CCACyZca3ASKwD-tuJHQi1805BN7ScTguAoRuHd5xTxCSUL1aNvQ@mail.gmail.com%3E
// maropu
On Wed, May 18, 2016 at 1:14 PM, Mohanraj Ragupathiraj <
mohanaugust@gmail.com> wrote:
> I have 100 million records to be inserted to a HBase table (PHOENIX) as a
> result of a Spark Job. I would like to know if i convert it to a Dataframe
> and save it, will it do Bulk load (or) it is not the efficient way to write
> data to a HBase table
>
> --
> Thanks and Regards
> Mohan
>
--
---
Takeshi Yamamuro