You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Benjamin Kim <bb...@gmail.com> on 2016/04/21 15:52:35 UTC

Save DataFrame to HBase

Has anyone found an easy way to save a DataFrame into HBase?

Thanks,
Ben


---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org


Re: Save DataFrame to HBase

Posted by Zhan Zhang <zz...@hortonworks.com>.
You can try this

https://github.com/hortonworks/shc.git

or here
http://spark-packages.org/package/zhzhan/shc

Currently it is in the process of merging into HBase.

Thanks.

Zhan Zhang

On Apr 21, 2016, at 8:44 AM, Benjamin Kim <bb...@gmail.com>> wrote:

Hi Ted,

Can this module be used with an older version of HBase, such as 1.0 or 1.1? Where can I get the module from?

Thanks,
Ben

On Apr 21, 2016, at 6:56 AM, Ted Yu <yu...@gmail.com>> wrote:

The hbase-spark module in Apache HBase (coming with hbase 2.0 release) can do this.

On Thu, Apr 21, 2016 at 6:52 AM, Benjamin Kim <bb...@gmail.com>> wrote:
Has anyone found an easy way to save a DataFrame into HBase?

Thanks,
Ben


---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org<ma...@spark.apache.org>
For additional commands, e-mail: user-help@spark.apache.org<ma...@spark.apache.org>





Re: Save DataFrame to HBase

Posted by Benjamin Kim <bb...@gmail.com>.
Hi Ted,

Can this module be used with an older version of HBase, such as 1.0 or 1.1? Where can I get the module from?

Thanks,
Ben

> On Apr 21, 2016, at 6:56 AM, Ted Yu <yu...@gmail.com> wrote:
> 
> The hbase-spark module in Apache HBase (coming with hbase 2.0 release) can do this.
> 
> On Thu, Apr 21, 2016 at 6:52 AM, Benjamin Kim <bbuild11@gmail.com <ma...@gmail.com>> wrote:
> Has anyone found an easy way to save a DataFrame into HBase?
> 
> Thanks,
> Ben
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org <ma...@spark.apache.org>
> For additional commands, e-mail: user-help@spark.apache.org <ma...@spark.apache.org>
> 
> 


Re: Save DataFrame to HBase

Posted by Ted Yu <yu...@gmail.com>.
Please note:
The name of hbase table is specified in:

  def writeCatalog = s"""{
                    |"table":{"namespace":"default", "name":"table1"},

not by the:

HBaseTableCatalog.newTable -> "5"

FYI

On Tue, May 10, 2016 at 3:11 PM, Ted Yu <yu...@gmail.com> wrote:

> I think so.
>
> Please refer to the table population tests in (master branch):
>
> hbase-spark/src/test/scala/org/apache/hadoop/hbase/spark/DefaultSourceSuite.scala
>
> Cheers
>
> On Tue, May 10, 2016 at 2:53 PM, Benjamin Kim <bb...@gmail.com> wrote:
>
>> Ted,
>>
>> Will the hbase-spark module allow for creating tables in Spark SQL that
>> reference the hbase tables underneath? In this way, users can query using
>> just SQL.
>>
>> Thanks,
>> Ben
>>
>> On Apr 28, 2016, at 3:09 AM, Ted Yu <yu...@gmail.com> wrote:
>>
>> Hbase 2.0 release likely would come after Spark 2.0 release.
>>
>> There're other features being developed in hbase 2.0
>> I am not sure when hbase 2.0 would be released.
>>
>> The refguide is incomplete.
>> Zhan has assigned the doc JIRA to himself. The documentation would be
>> done after fixing bugs in hbase-spark module.
>>
>> Cheers
>>
>> On Apr 27, 2016, at 10:31 PM, Benjamin Kim <bb...@gmail.com> wrote:
>>
>> Hi Ted,
>>
>> Do you know when the release will be? I also see some documentation for
>> usage of the hbase-spark module at the hbase website. But, I don’t see an
>> example on how to save data. There is only one for reading/querying data.
>> Will this be added when the final version does get released?
>>
>> Thanks,
>> Ben
>>
>> On Apr 21, 2016, at 6:56 AM, Ted Yu <yu...@gmail.com> wrote:
>>
>> The hbase-spark module in Apache HBase (coming with hbase 2.0 release)
>> can do this.
>>
>> On Thu, Apr 21, 2016 at 6:52 AM, Benjamin Kim <bb...@gmail.com> wrote:
>>
>>> Has anyone found an easy way to save a DataFrame into HBase?
>>>
>>> Thanks,
>>> Ben
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>>> For additional commands, e-mail: user-help@spark.apache.org
>>>
>>>
>>
>>
>>
>

Re: Save DataFrame to HBase

Posted by Ted Yu <yu...@gmail.com>.
I think so.

Please refer to the table population tests in (master branch):
hbase-spark/src/test/scala/org/apache/hadoop/hbase/spark/DefaultSourceSuite.scala

Cheers

On Tue, May 10, 2016 at 2:53 PM, Benjamin Kim <bb...@gmail.com> wrote:

> Ted,
>
> Will the hbase-spark module allow for creating tables in Spark SQL that
> reference the hbase tables underneath? In this way, users can query using
> just SQL.
>
> Thanks,
> Ben
>
> On Apr 28, 2016, at 3:09 AM, Ted Yu <yu...@gmail.com> wrote:
>
> Hbase 2.0 release likely would come after Spark 2.0 release.
>
> There're other features being developed in hbase 2.0
> I am not sure when hbase 2.0 would be released.
>
> The refguide is incomplete.
> Zhan has assigned the doc JIRA to himself. The documentation would be done
> after fixing bugs in hbase-spark module.
>
> Cheers
>
> On Apr 27, 2016, at 10:31 PM, Benjamin Kim <bb...@gmail.com> wrote:
>
> Hi Ted,
>
> Do you know when the release will be? I also see some documentation for
> usage of the hbase-spark module at the hbase website. But, I don’t see an
> example on how to save data. There is only one for reading/querying data.
> Will this be added when the final version does get released?
>
> Thanks,
> Ben
>
> On Apr 21, 2016, at 6:56 AM, Ted Yu <yu...@gmail.com> wrote:
>
> The hbase-spark module in Apache HBase (coming with hbase 2.0 release) can
> do this.
>
> On Thu, Apr 21, 2016 at 6:52 AM, Benjamin Kim <bb...@gmail.com> wrote:
>
>> Has anyone found an easy way to save a DataFrame into HBase?
>>
>> Thanks,
>> Ben
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>> For additional commands, e-mail: user-help@spark.apache.org
>>
>>
>
>
>

Re: Save DataFrame to HBase

Posted by Benjamin Kim <bb...@gmail.com>.
Ted,

Will the hbase-spark module allow for creating tables in Spark SQL that reference the hbase tables underneath? In this way, users can query using just SQL.

Thanks,
Ben

> On Apr 28, 2016, at 3:09 AM, Ted Yu <yu...@gmail.com> wrote:
> 
> Hbase 2.0 release likely would come after Spark 2.0 release. 
> 
> There're other features being developed in hbase 2.0
> I am not sure when hbase 2.0 would be released. 
> 
> The refguide is incomplete. 
> Zhan has assigned the doc JIRA to himself. The documentation would be done after fixing bugs in hbase-spark module. 
> 
> Cheers
> 
> On Apr 27, 2016, at 10:31 PM, Benjamin Kim <bbuild11@gmail.com <ma...@gmail.com>> wrote:
> 
>> Hi Ted,
>> 
>> Do you know when the release will be? I also see some documentation for usage of the hbase-spark module at the hbase website. But, I don’t see an example on how to save data. There is only one for reading/querying data. Will this be added when the final version does get released?
>> 
>> Thanks,
>> Ben
>> 
>>> On Apr 21, 2016, at 6:56 AM, Ted Yu <yuzhihong@gmail.com <ma...@gmail.com>> wrote:
>>> 
>>> The hbase-spark module in Apache HBase (coming with hbase 2.0 release) can do this.
>>> 
>>> On Thu, Apr 21, 2016 at 6:52 AM, Benjamin Kim <bbuild11@gmail.com <ma...@gmail.com>> wrote:
>>> Has anyone found an easy way to save a DataFrame into HBase?
>>> 
>>> Thanks,
>>> Ben
>>> 
>>> 
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org <ma...@spark.apache.org>
>>> For additional commands, e-mail: user-help@spark.apache.org <ma...@spark.apache.org>
>>> 
>>> 
>> 


Re: Save DataFrame to HBase

Posted by Ted Yu <yu...@gmail.com>.
Hbase 2.0 release likely would come after Spark 2.0 release. 

There're other features being developed in hbase 2.0
I am not sure when hbase 2.0 would be released. 

The refguide is incomplete. 
Zhan has assigned the doc JIRA to himself. The documentation would be done after fixing bugs in hbase-spark module. 

Cheers

> On Apr 27, 2016, at 10:31 PM, Benjamin Kim <bb...@gmail.com> wrote:
> 
> Hi Ted,
> 
> Do you know when the release will be? I also see some documentation for usage of the hbase-spark module at the hbase website. But, I don’t see an example on how to save data. There is only one for reading/querying data. Will this be added when the final version does get released?
> 
> Thanks,
> Ben
> 
>> On Apr 21, 2016, at 6:56 AM, Ted Yu <yu...@gmail.com> wrote:
>> 
>> The hbase-spark module in Apache HBase (coming with hbase 2.0 release) can do this.
>> 
>>> On Thu, Apr 21, 2016 at 6:52 AM, Benjamin Kim <bb...@gmail.com> wrote:
>>> Has anyone found an easy way to save a DataFrame into HBase?
>>> 
>>> Thanks,
>>> Ben
>>> 
>>> 
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>>> For additional commands, e-mail: user-help@spark.apache.org
> 

Re: Save DataFrame to HBase

Posted by Benjamin Kim <bb...@gmail.com>.
Hi Ted,

Do you know when the release will be? I also see some documentation for usage of the hbase-spark module at the hbase website. But, I don’t see an example on how to save data. There is only one for reading/querying data. Will this be added when the final version does get released?

Thanks,
Ben

> On Apr 21, 2016, at 6:56 AM, Ted Yu <yu...@gmail.com> wrote:
> 
> The hbase-spark module in Apache HBase (coming with hbase 2.0 release) can do this.
> 
> On Thu, Apr 21, 2016 at 6:52 AM, Benjamin Kim <bbuild11@gmail.com <ma...@gmail.com>> wrote:
> Has anyone found an easy way to save a DataFrame into HBase?
> 
> Thanks,
> Ben
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org <ma...@spark.apache.org>
> For additional commands, e-mail: user-help@spark.apache.org <ma...@spark.apache.org>
> 
> 


Re: Save DataFrame to HBase

Posted by Ted Yu <yu...@gmail.com>.
The hbase-spark module in Apache HBase (coming with hbase 2.0 release) can
do this.

On Thu, Apr 21, 2016 at 6:52 AM, Benjamin Kim <bb...@gmail.com> wrote:

> Has anyone found an easy way to save a DataFrame into HBase?
>
> Thanks,
> Ben
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>

Re: Save DataFrame to HBase

Posted by Benjamin Kim <bb...@gmail.com>.
Daniel,

If you can get the code snippet, that would be great! I’ve been trying to get it to work for me as well. The examples on the Phoenix website do not work for me. If you are willing to also, can you include your setup to make Phoenix work with Spark?

Thanks,
Ben

> On Apr 27, 2016, at 11:46 AM, Paras sachdeva <pa...@gmail.com> wrote:
> 
> Hi Daniel,
> 
> Would you possibly be able to share the snipped to code you have used ?
> 
> Thank you.
> 
> On Wed, Apr 27, 2016 at 3:13 PM, Daniel Haviv <daniel.haviv@veracity-group.com <ma...@veracity-group.com>> wrote:
> Hi Benjamin,
> Yes it should work.
> 
> Let me know if you need further assistance I might be able to get the code I've used for that project.
> 
> Thank you.
> Daniel
> 
> On 24 Apr 2016, at 17:35, Benjamin Kim <bbuild11@gmail.com <ma...@gmail.com>> wrote:
> 
>> Hi Daniel,
>> 
>> How did you get the Phoenix plugin to work? I have CDH 5.5.2 installed which comes with HBase 1.0.0 and Phoenix 4.5.2. Do you think this will work?
>> 
>> Thanks,
>> Ben
>> 
>>> On Apr 24, 2016, at 1:43 AM, Daniel Haviv <daniel.haviv@veracity-group.com <ma...@veracity-group.com>> wrote:
>>> 
>>> Hi,
>>> I tried saving DF to HBase using a hive table with hbase storage handler and hiveContext but it failed due to a bug.
>>> 
>>> I was able to persist the DF to hbase using Apache Pheonix which was pretty simple.
>>> 
>>> Thank you.
>>> Daniel
>>> 
>>> On 21 Apr 2016, at 16:52, Benjamin Kim <bbuild11@gmail.com <ma...@gmail.com>> wrote:
>>> 
>>>> Has anyone found an easy way to save a DataFrame into HBase?
>>>> 
>>>> Thanks,
>>>> Ben
>>>> 
>>>> 
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org <ma...@spark.apache.org>
>>>> For additional commands, e-mail: user-help@spark.apache.org <ma...@spark.apache.org>
>>>> 
>> 
> 


Re: Save DataFrame to HBase

Posted by Paras sachdeva <pa...@gmail.com>.
Hi Daniel,

Would you possibly be able to share the snipped to code you have used ?

Thank you.

On Wed, Apr 27, 2016 at 3:13 PM, Daniel Haviv <
daniel.haviv@veracity-group.com> wrote:

> Hi Benjamin,
> Yes it should work.
>
> Let me know if you need further assistance I might be able to get the code
> I've used for that project.
>
> Thank you.
> Daniel
>
> On 24 Apr 2016, at 17:35, Benjamin Kim <bb...@gmail.com> wrote:
>
> Hi Daniel,
>
> How did you get the Phoenix plugin to work? I have CDH 5.5.2 installed
> which comes with HBase 1.0.0 and Phoenix 4.5.2. Do you think this will work?
>
> Thanks,
> Ben
>
> On Apr 24, 2016, at 1:43 AM, Daniel Haviv <da...@veracity-group.com>
> wrote:
>
> Hi,
> I tried saving DF to HBase using a hive table with hbase storage handler
> and hiveContext but it failed due to a bug.
>
> I was able to persist the DF to hbase using Apache Pheonix which was
> pretty simple.
>
> Thank you.
> Daniel
>
> On 21 Apr 2016, at 16:52, Benjamin Kim <bb...@gmail.com> wrote:
>
> Has anyone found an easy way to save a DataFrame into HBase?
>
> Thanks,
> Ben
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>
>

Re: Save DataFrame to HBase

Posted by Daniel Haviv <da...@veracity-group.com>.
Hi Benjamin,
Yes it should work.

Let me know if you need further assistance I might be able to get the code I've used for that project.

Thank you.
Daniel

> On 24 Apr 2016, at 17:35, Benjamin Kim <bb...@gmail.com> wrote:
> 
> Hi Daniel,
> 
> How did you get the Phoenix plugin to work? I have CDH 5.5.2 installed which comes with HBase 1.0.0 and Phoenix 4.5.2. Do you think this will work?
> 
> Thanks,
> Ben
> 
>> On Apr 24, 2016, at 1:43 AM, Daniel Haviv <da...@veracity-group.com> wrote:
>> 
>> Hi,
>> I tried saving DF to HBase using a hive table with hbase storage handler and hiveContext but it failed due to a bug.
>> 
>> I was able to persist the DF to hbase using Apache Pheonix which was pretty simple.
>> 
>> Thank you.
>> Daniel
>> 
>>> On 21 Apr 2016, at 16:52, Benjamin Kim <bb...@gmail.com> wrote:
>>> 
>>> Has anyone found an easy way to save a DataFrame into HBase?
>>> 
>>> Thanks,
>>> Ben
>>> 
>>> 
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>>> For additional commands, e-mail: user-help@spark.apache.org
> 

Re: Save DataFrame to HBase

Posted by Benjamin Kim <bb...@gmail.com>.
Hi Daniel,

How did you get the Phoenix plugin to work? I have CDH 5.5.2 installed which comes with HBase 1.0.0 and Phoenix 4.5.2. Do you think this will work?

Thanks,
Ben

> On Apr 24, 2016, at 1:43 AM, Daniel Haviv <da...@veracity-group.com> wrote:
> 
> Hi,
> I tried saving DF to HBase using a hive table with hbase storage handler and hiveContext but it failed due to a bug.
> 
> I was able to persist the DF to hbase using Apache Pheonix which was pretty simple.
> 
> Thank you.
> Daniel
> 
> On 21 Apr 2016, at 16:52, Benjamin Kim <bbuild11@gmail.com <ma...@gmail.com>> wrote:
> 
>> Has anyone found an easy way to save a DataFrame into HBase?
>> 
>> Thanks,
>> Ben
>> 
>> 
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org <ma...@spark.apache.org>
>> For additional commands, e-mail: user-help@spark.apache.org <ma...@spark.apache.org>
>> 


Re: Save DataFrame to HBase

Posted by Daniel Haviv <da...@veracity-group.com>.
Hi,
I tried saving DF to HBase using a hive table with hbase storage handler and hiveContext but it failed due to a bug.

I was able to persist the DF to hbase using Apache Pheonix which was pretty simple.

Thank you.
Daniel

> On 21 Apr 2016, at 16:52, Benjamin Kim <bb...@gmail.com> wrote:
> 
> Has anyone found an easy way to save a DataFrame into HBase?
> 
> Thanks,
> Ben
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>