You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hadoop.apache.org by Divya Gehlot <di...@gmail.com> on 2016/04/08 07:22:02 UTC
[HELP:]Save Spark Dataframe in Phoenix Table
Hi,
I hava a Hortonworks Hadoop cluster having below Configurations :
Spark 1.5.2
HBASE 1.1.x
Phoenix 4.4
I am able to connect to Phoenix through JDBC connection and able to read
the Phoenix tables .
But while writing the data back to Phoenix table
I am getting below error :
org.apache.spark.sql.AnalysisException:
org.apache.phoenix.spark.DefaultSource does not allow user-specified
schemas.;
Can any body help in resolving the above errors or any other solution of
saving Spark Dataframes to Phoenix.
Would really appareciate the help.
Thanks,
Divya
Fwd: [HELP:]Save Spark Dataframe in Phoenix Table
Posted by Divya Gehlot <di...@gmail.com>.
Reposting for other user benefits
---------- Forwarded message ----------
From: Divya Gehlot <di...@gmail.com>
Date: 8 April 2016 at 19:54
Subject: Re: [HELP:]Save Spark Dataframe in Phoenix Table
To: Josh Mahonin <jm...@gmail.com>
Hi Josh,
I am doing in the same manner as mentioned in Phoenix Spark manner.
Using the latest version of HDP 2.3.4 .
In case of version mismatch/lack of spark Phoenix support it's should have
thrown the error at read also.
Which is working fine as expected .
Will surely pass on the code snippets once I log on to my System.
In the mean while I would like to know the zkURL parameter.If I build it
with HbaseConfiguration and passing zk quorom ,znode and port .
It throws error for example localhost :2181/hbase-unsecure
This localhost gets replaced by all the quorom
Like quorum1,quorum2:2181/hbase-unsecure
I am just providing the IP address of my HBase master.
I feel like I am not on right track so asked for the help .
How to connect to Phoenix through Spark on hadoop cluster .
Thanks for the help.
Cheers,
Divya
On Apr 8, 2016 7:06 PM, "Josh Mahonin" <jm...@gmail.com> wrote:
> Hi Divya,
>
> That's strange. Are you able to post a snippet of your code to look at?
> And are you sure that you're saving the dataframes as per the docs (
> https://phoenix.apache.org/phoenix_spark.html)?
>
> Depending on your HDP version, it may or may not actually have
> phoenix-spark support. Double-check that your Spark configuration is setup
> with the right worker/driver classpath settings. and that the phoenix JARs
> contain the necessary phoenix-spark classes
> (e.g. org.apache.phoenix.spark.PhoenixRelation). If not, I suggest
> following up with Hortonworks.
>
> Josh
>
>
>
> On Fri, Apr 8, 2016 at 1:22 AM, Divya Gehlot <di...@gmail.com>
> wrote:
>
>> Hi,
>> I hava a Hortonworks Hadoop cluster having below Configurations :
>> Spark 1.5.2
>> HBASE 1.1.x
>> Phoenix 4.4
>>
>> I am able to connect to Phoenix through JDBC connection and able to read
>> the Phoenix tables .
>> But while writing the data back to Phoenix table
>> I am getting below error :
>>
>> org.apache.spark.sql.AnalysisException:
>> org.apache.phoenix.spark.DefaultSource does not allow user-specified
>> schemas.;
>>
>> Can any body help in resolving the above errors or any other solution of
>> saving Spark Dataframes to Phoenix.
>>
>> Would really appareciate the help.
>>
>> Thanks,
>> Divya
>>
>
>
Re: [HELP:]Save Spark Dataframe in Phoenix Table
Posted by Divya Gehlot <di...@gmail.com>.
Hi Josh,
I downloaded the Apache Phoenix v4.4.0-HBase-1.1
<https://github.com/apache/phoenix/releases/tag/v4.4.0-HBase-1.1> and
tried packaging it through *apache-maven-3.3.9*
When I try to build it using maven I am getting following error
maven command I used to build Phoenix mvn package -DskipTests
-Dhadoop.profile=2
[INFO]
------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Apache Phoenix ..................................... SUCCESS [
2.550 s]
[INFO] Phoenix Core ....................................... SUCCESS [
36.224 s]
[INFO] Phoenix - Flume .................................... SUCCESS [
2.575 s]
[INFO] Phoenix - Pig ...................................... SUCCESS [
1.809 s]
[INFO] Phoenix Query Server Client ........................ SUCCESS [
0.734 s]
[INFO] Phoenix Query Server ............................... SUCCESS [
12.427 s]
[INFO] Phoenix - Pherf .................................... SUCCESS [
13.394 s]
[INFO] Phoenix - Spark .................................... FAILURE [
18.693 s]
[INFO] Phoenix Assembly ................................... SKIPPED
[INFO]
------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO]
------------------------------------------------------------------------
[INFO] Total time: 01:28 min
[INFO] Finished at: 2016-04-19T12:44:23+08:00
[INFO] Final Memory: 73M/732M
[INFO]
------------------------------------------------------------------------
[ERROR] Failed to execute goal
net.alchim31.maven:scala-maven-plugin:3.2.0:testCompile
(scala-test-compile) on project phoenix-spark: wrap:
org.apache.commons.exec.ExecuteException: Process exited with an error:
-10000 (Exit value: -10000) -> [Help 1]
[ERROR]
Could please help me finding the the issue .
Would really appreciate your help .
Thanks,
Divya
On 11 April 2016 at 02:45, Josh Mahonin <jm...@gmail.com> wrote:
> Hi Divya,
>
> No, there is a separate JAR that would look like
> 'phoenix-4.4.0.XXX-client-spark.jar'. If you download a binary release of
> Phoenix, or compile the latest version yourself, you will be able to see
> and use it. It does not come with the HDP 2.3.4 platform, at least last I
> checked.
>
> Regards,
>
> Josh
>
> On Sat, Apr 9, 2016 at 2:24 PM, Divya Gehlot <di...@gmail.com>
> wrote:
>
>> Hi Josh,
>> Thank you very much for your help.
>> I could see there is phoenix-spark-4.4.0.2.3.4.0-3485.jar in my
>> phoenix/lib.
>> Please confirm is the above jar you are talking about?
>>
>> Thanks,
>> Divya
>>
>> Josh Mahonin <jmahonin@
>>
>> On 9 April 2016 at 23:01, Josh Mahonin <jm...@gmail.com> wrote:
>>
>>> Hi Divya,
>>>
>>> You don't have the phoenix client-spark JAR in your classpath, which is
>>> required for the phoenix-spark integration to work (as per the
>>> documentation).
>>>
>>> As well, you aren't using the vanilla Apache project that this mailing
>>> list supports, but are using a vendor packaged platform (Hortonworks).
>>> Since they maintain their own patches and forks to the upstream Apache
>>> versions, in general you should opt for filing support tickets with them
>>> first. In this particular case, HDP 2.3.4 doesn't actually provide the
>>> necessary phoenix client-spark JAR by default, so your options are limited
>>> here. Again, I recommend filing a support ticket with Hortonworks.
>>>
>>> Regards,
>>>
>>> Josh
>>>
>>> On Sat, Apr 9, 2016 at 9:11 AM, Divya Gehlot <di...@gmail.com>
>>> wrote:
>>>
>>>> Hi,
>>>> The code which I using to connect to Phoenix for writing
>>>> def writeToTable(df: DataFrame,dbtable: String) = {
>>>> val phx_properties = collection.immutable.Map[String, String](
>>>> "zkUrl" -> "localhost:2181:/hbase-unsecure",
>>>> "table" -> dbtable)
>>>>
>>>> df.write.format("org.apache.phoenix.spark").mode(SaveMode.Overwrite).options(phx_properties).saveAsTable(dbtable)
>>>> }
>>>>
>>>> While Submitting Spark job
>>>> * spark-shell --properties-file /TestDivya/Spark/Phoenix.properties
>>>> --jars
>>>> /usr/hdp/2.3.4.0-3485/hive/lib/hive-hbase-handler-1.2.1.2.3.4.0-3485.jar,/usr/hdp/2.3.4.0-3485/phoenix/lib/zookeeper.jar,/usr/hdp/2.3.4.0-3485/phoenix/lib/hbase-client.jar,/usr/hdp/2.3.4.0-3485/phoenix/lib/hbase-common.jar,/usr/hdp/2.3.4.0-3485/phoenix/lib/hbase-protocol.jar,/usr/hdp/2.3.4.0-3485/hbase/lib/phoenix-server.jar,/usr/hdp/2.3.4.0-3485/hbase/lib/phoenix-client-4.4.0.jar
>>>> --driver-class-path
>>>> /usr/hdp/2.3.4.0-3485/hbase/lib/phoenix-server.jar,/usr/hdp/2.3.4.0-3485/hbase/lib/phoenix-client-4.4.0.jar
>>>> --packages com.databricks:spark-csv_2.10:1.4.0 --master yarn-client -i
>>>> /TestDivya/Spark/WriteToPheonix.scala*
>>>>
>>>>
>>>> Getting the below error :
>>>>
>>>> org.apache.spark.sql.AnalysisException: org.apache.phoenix.spark.DefaultSource
>>>> does not allow user-specified schemas.;
>>>>
>>>> Am I on the right track or missing any properties ?
>>>>
>>>> Because of this I am unable to proceed with Phoenix and have to find
>>>> alternate options.
>>>> Would really appreciate the help
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> ---------- Forwarded message ----------
>>>> From: Divya Gehlot <di...@gmail.com>
>>>> Date: 8 April 2016 at 19:54
>>>> Subject: Re: [HELP:]Save Spark Dataframe in Phoenix Table
>>>> To: Josh Mahonin <jm...@gmail.com>
>>>>
>>>>
>>>> Hi Josh,
>>>> I am doing in the same manner as mentioned in Phoenix Spark manner.
>>>> Using the latest version of HDP 2.3.4 .
>>>> In case of version mismatch/lack of spark Phoenix support it's should
>>>> have thrown the error at read also.
>>>> Which is working fine as expected .
>>>> Will surely pass on the code snippets once I log on to my System.
>>>> In the mean while I would like to know the zkURL parameter.If I build
>>>> it with HbaseConfiguration and passing zk quorom ,znode and port .
>>>> It throws error for example localhost :2181/hbase-unsecure
>>>> This localhost gets replaced by all the quorom
>>>> Like quorum1,quorum2:2181/hbase-unsecure
>>>>
>>>> I am just providing the IP address of my HBase master.
>>>>
>>>> I feel like I am not on right track so asked for the help .
>>>> How to connect to Phoenix through Spark on hadoop cluster .
>>>> Thanks for the help.
>>>> Cheers,
>>>> Divya
>>>> On Apr 8, 2016 7:06 PM, "Josh Mahonin" <jm...@gmail.com> wrote:
>>>>
>>>>> Hi Divya,
>>>>>
>>>>> That's strange. Are you able to post a snippet of your code to look
>>>>> at? And are you sure that you're saving the dataframes as per the docs (
>>>>> https://phoenix.apache.org/phoenix_spark.html)?
>>>>>
>>>>> Depending on your HDP version, it may or may not actually have
>>>>> phoenix-spark support. Double-check that your Spark configuration is setup
>>>>> with the right worker/driver classpath settings. and that the phoenix JARs
>>>>> contain the necessary phoenix-spark classes
>>>>> (e.g. org.apache.phoenix.spark.PhoenixRelation). If not, I suggest
>>>>> following up with Hortonworks.
>>>>>
>>>>> Josh
>>>>>
>>>>>
>>>>>
>>>>> On Fri, Apr 8, 2016 at 1:22 AM, Divya Gehlot <di...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hi,
>>>>>> I hava a Hortonworks Hadoop cluster having below Configurations :
>>>>>> Spark 1.5.2
>>>>>> HBASE 1.1.x
>>>>>> Phoenix 4.4
>>>>>>
>>>>>> I am able to connect to Phoenix through JDBC connection and able to
>>>>>> read the Phoenix tables .
>>>>>> But while writing the data back to Phoenix table
>>>>>> I am getting below error :
>>>>>>
>>>>>> org.apache.spark.sql.AnalysisException:
>>>>>> org.apache.phoenix.spark.DefaultSource does not allow user-specified
>>>>>> schemas.;
>>>>>>
>>>>>> Can any body help in resolving the above errors or any other solution
>>>>>> of saving Spark Dataframes to Phoenix.
>>>>>>
>>>>>> Would really appareciate the help.
>>>>>>
>>>>>> Thanks,
>>>>>> Divya
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>
Re: [HELP:]Save Spark Dataframe in Phoenix Table
Posted by Josh Mahonin <jm...@gmail.com>.
Hi Divya,
No, there is a separate JAR that would look like
'phoenix-4.4.0.XXX-client-spark.jar'. If you download a binary release of
Phoenix, or compile the latest version yourself, you will be able to see
and use it. It does not come with the HDP 2.3.4 platform, at least last I
checked.
Regards,
Josh
On Sat, Apr 9, 2016 at 2:24 PM, Divya Gehlot <di...@gmail.com>
wrote:
> Hi Josh,
> Thank you very much for your help.
> I could see there is phoenix-spark-4.4.0.2.3.4.0-3485.jar in my
> phoenix/lib.
> Please confirm is the above jar you are talking about?
>
> Thanks,
> Divya
>
> Josh Mahonin <jmahonin@
>
> On 9 April 2016 at 23:01, Josh Mahonin <jm...@gmail.com> wrote:
>
>> Hi Divya,
>>
>> You don't have the phoenix client-spark JAR in your classpath, which is
>> required for the phoenix-spark integration to work (as per the
>> documentation).
>>
>> As well, you aren't using the vanilla Apache project that this mailing
>> list supports, but are using a vendor packaged platform (Hortonworks).
>> Since they maintain their own patches and forks to the upstream Apache
>> versions, in general you should opt for filing support tickets with them
>> first. In this particular case, HDP 2.3.4 doesn't actually provide the
>> necessary phoenix client-spark JAR by default, so your options are limited
>> here. Again, I recommend filing a support ticket with Hortonworks.
>>
>> Regards,
>>
>> Josh
>>
>> On Sat, Apr 9, 2016 at 9:11 AM, Divya Gehlot <di...@gmail.com>
>> wrote:
>>
>>> Hi,
>>> The code which I using to connect to Phoenix for writing
>>> def writeToTable(df: DataFrame,dbtable: String) = {
>>> val phx_properties = collection.immutable.Map[String, String](
>>> "zkUrl" -> "localhost:2181:/hbase-unsecure",
>>> "table" -> dbtable)
>>>
>>> df.write.format("org.apache.phoenix.spark").mode(SaveMode.Overwrite).options(phx_properties).saveAsTable(dbtable)
>>> }
>>>
>>> While Submitting Spark job
>>> * spark-shell --properties-file /TestDivya/Spark/Phoenix.properties
>>> --jars
>>> /usr/hdp/2.3.4.0-3485/hive/lib/hive-hbase-handler-1.2.1.2.3.4.0-3485.jar,/usr/hdp/2.3.4.0-3485/phoenix/lib/zookeeper.jar,/usr/hdp/2.3.4.0-3485/phoenix/lib/hbase-client.jar,/usr/hdp/2.3.4.0-3485/phoenix/lib/hbase-common.jar,/usr/hdp/2.3.4.0-3485/phoenix/lib/hbase-protocol.jar,/usr/hdp/2.3.4.0-3485/hbase/lib/phoenix-server.jar,/usr/hdp/2.3.4.0-3485/hbase/lib/phoenix-client-4.4.0.jar
>>> --driver-class-path
>>> /usr/hdp/2.3.4.0-3485/hbase/lib/phoenix-server.jar,/usr/hdp/2.3.4.0-3485/hbase/lib/phoenix-client-4.4.0.jar
>>> --packages com.databricks:spark-csv_2.10:1.4.0 --master yarn-client -i
>>> /TestDivya/Spark/WriteToPheonix.scala*
>>>
>>>
>>> Getting the below error :
>>>
>>> org.apache.spark.sql.AnalysisException: org.apache.phoenix.spark.DefaultSource
>>> does not allow user-specified schemas.;
>>>
>>> Am I on the right track or missing any properties ?
>>>
>>> Because of this I am unable to proceed with Phoenix and have to find
>>> alternate options.
>>> Would really appreciate the help
>>>
>>>
>>>
>>>
>>>
>>> ---------- Forwarded message ----------
>>> From: Divya Gehlot <di...@gmail.com>
>>> Date: 8 April 2016 at 19:54
>>> Subject: Re: [HELP:]Save Spark Dataframe in Phoenix Table
>>> To: Josh Mahonin <jm...@gmail.com>
>>>
>>>
>>> Hi Josh,
>>> I am doing in the same manner as mentioned in Phoenix Spark manner.
>>> Using the latest version of HDP 2.3.4 .
>>> In case of version mismatch/lack of spark Phoenix support it's should
>>> have thrown the error at read also.
>>> Which is working fine as expected .
>>> Will surely pass on the code snippets once I log on to my System.
>>> In the mean while I would like to know the zkURL parameter.If I build it
>>> with HbaseConfiguration and passing zk quorom ,znode and port .
>>> It throws error for example localhost :2181/hbase-unsecure
>>> This localhost gets replaced by all the quorom
>>> Like quorum1,quorum2:2181/hbase-unsecure
>>>
>>> I am just providing the IP address of my HBase master.
>>>
>>> I feel like I am not on right track so asked for the help .
>>> How to connect to Phoenix through Spark on hadoop cluster .
>>> Thanks for the help.
>>> Cheers,
>>> Divya
>>> On Apr 8, 2016 7:06 PM, "Josh Mahonin" <jm...@gmail.com> wrote:
>>>
>>>> Hi Divya,
>>>>
>>>> That's strange. Are you able to post a snippet of your code to look at?
>>>> And are you sure that you're saving the dataframes as per the docs (
>>>> https://phoenix.apache.org/phoenix_spark.html)?
>>>>
>>>> Depending on your HDP version, it may or may not actually have
>>>> phoenix-spark support. Double-check that your Spark configuration is setup
>>>> with the right worker/driver classpath settings. and that the phoenix JARs
>>>> contain the necessary phoenix-spark classes
>>>> (e.g. org.apache.phoenix.spark.PhoenixRelation). If not, I suggest
>>>> following up with Hortonworks.
>>>>
>>>> Josh
>>>>
>>>>
>>>>
>>>> On Fri, Apr 8, 2016 at 1:22 AM, Divya Gehlot <di...@gmail.com>
>>>> wrote:
>>>>
>>>>> Hi,
>>>>> I hava a Hortonworks Hadoop cluster having below Configurations :
>>>>> Spark 1.5.2
>>>>> HBASE 1.1.x
>>>>> Phoenix 4.4
>>>>>
>>>>> I am able to connect to Phoenix through JDBC connection and able to
>>>>> read the Phoenix tables .
>>>>> But while writing the data back to Phoenix table
>>>>> I am getting below error :
>>>>>
>>>>> org.apache.spark.sql.AnalysisException:
>>>>> org.apache.phoenix.spark.DefaultSource does not allow user-specified
>>>>> schemas.;
>>>>>
>>>>> Can any body help in resolving the above errors or any other solution
>>>>> of saving Spark Dataframes to Phoenix.
>>>>>
>>>>> Would really appareciate the help.
>>>>>
>>>>> Thanks,
>>>>> Divya
>>>>>
>>>>
>>>>
>>>
>>
>
Re: [HELP:]Save Spark Dataframe in Phoenix Table
Posted by Divya Gehlot <di...@gmail.com>.
Hi Josh,
Thank you very much for your help.
I could see there is phoenix-spark-4.4.0.2.3.4.0-3485.jar in my
phoenix/lib.
Please confirm is the above jar you are talking about?
Thanks,
Divya
Josh Mahonin <jmahonin@
On 9 April 2016 at 23:01, Josh Mahonin <jm...@gmail.com> wrote:
> Hi Divya,
>
> You don't have the phoenix client-spark JAR in your classpath, which is
> required for the phoenix-spark integration to work (as per the
> documentation).
>
> As well, you aren't using the vanilla Apache project that this mailing
> list supports, but are using a vendor packaged platform (Hortonworks).
> Since they maintain their own patches and forks to the upstream Apache
> versions, in general you should opt for filing support tickets with them
> first. In this particular case, HDP 2.3.4 doesn't actually provide the
> necessary phoenix client-spark JAR by default, so your options are limited
> here. Again, I recommend filing a support ticket with Hortonworks.
>
> Regards,
>
> Josh
>
> On Sat, Apr 9, 2016 at 9:11 AM, Divya Gehlot <di...@gmail.com>
> wrote:
>
>> Hi,
>> The code which I using to connect to Phoenix for writing
>> def writeToTable(df: DataFrame,dbtable: String) = {
>> val phx_properties = collection.immutable.Map[String, String](
>> "zkUrl" -> "localhost:2181:/hbase-unsecure",
>> "table" -> dbtable)
>>
>> df.write.format("org.apache.phoenix.spark").mode(SaveMode.Overwrite).options(phx_properties).saveAsTable(dbtable)
>> }
>>
>> While Submitting Spark job
>> * spark-shell --properties-file /TestDivya/Spark/Phoenix.properties
>> --jars
>> /usr/hdp/2.3.4.0-3485/hive/lib/hive-hbase-handler-1.2.1.2.3.4.0-3485.jar,/usr/hdp/2.3.4.0-3485/phoenix/lib/zookeeper.jar,/usr/hdp/2.3.4.0-3485/phoenix/lib/hbase-client.jar,/usr/hdp/2.3.4.0-3485/phoenix/lib/hbase-common.jar,/usr/hdp/2.3.4.0-3485/phoenix/lib/hbase-protocol.jar,/usr/hdp/2.3.4.0-3485/hbase/lib/phoenix-server.jar,/usr/hdp/2.3.4.0-3485/hbase/lib/phoenix-client-4.4.0.jar
>> --driver-class-path
>> /usr/hdp/2.3.4.0-3485/hbase/lib/phoenix-server.jar,/usr/hdp/2.3.4.0-3485/hbase/lib/phoenix-client-4.4.0.jar
>> --packages com.databricks:spark-csv_2.10:1.4.0 --master yarn-client -i
>> /TestDivya/Spark/WriteToPheonix.scala*
>>
>>
>> Getting the below error :
>>
>> org.apache.spark.sql.AnalysisException: org.apache.phoenix.spark.DefaultSource
>> does not allow user-specified schemas.;
>>
>> Am I on the right track or missing any properties ?
>>
>> Because of this I am unable to proceed with Phoenix and have to find
>> alternate options.
>> Would really appreciate the help
>>
>>
>>
>>
>>
>> ---------- Forwarded message ----------
>> From: Divya Gehlot <di...@gmail.com>
>> Date: 8 April 2016 at 19:54
>> Subject: Re: [HELP:]Save Spark Dataframe in Phoenix Table
>> To: Josh Mahonin <jm...@gmail.com>
>>
>>
>> Hi Josh,
>> I am doing in the same manner as mentioned in Phoenix Spark manner.
>> Using the latest version of HDP 2.3.4 .
>> In case of version mismatch/lack of spark Phoenix support it's should
>> have thrown the error at read also.
>> Which is working fine as expected .
>> Will surely pass on the code snippets once I log on to my System.
>> In the mean while I would like to know the zkURL parameter.If I build it
>> with HbaseConfiguration and passing zk quorom ,znode and port .
>> It throws error for example localhost :2181/hbase-unsecure
>> This localhost gets replaced by all the quorom
>> Like quorum1,quorum2:2181/hbase-unsecure
>>
>> I am just providing the IP address of my HBase master.
>>
>> I feel like I am not on right track so asked for the help .
>> How to connect to Phoenix through Spark on hadoop cluster .
>> Thanks for the help.
>> Cheers,
>> Divya
>> On Apr 8, 2016 7:06 PM, "Josh Mahonin" <jm...@gmail.com> wrote:
>>
>>> Hi Divya,
>>>
>>> That's strange. Are you able to post a snippet of your code to look at?
>>> And are you sure that you're saving the dataframes as per the docs (
>>> https://phoenix.apache.org/phoenix_spark.html)?
>>>
>>> Depending on your HDP version, it may or may not actually have
>>> phoenix-spark support. Double-check that your Spark configuration is setup
>>> with the right worker/driver classpath settings. and that the phoenix JARs
>>> contain the necessary phoenix-spark classes
>>> (e.g. org.apache.phoenix.spark.PhoenixRelation). If not, I suggest
>>> following up with Hortonworks.
>>>
>>> Josh
>>>
>>>
>>>
>>> On Fri, Apr 8, 2016 at 1:22 AM, Divya Gehlot <di...@gmail.com>
>>> wrote:
>>>
>>>> Hi,
>>>> I hava a Hortonworks Hadoop cluster having below Configurations :
>>>> Spark 1.5.2
>>>> HBASE 1.1.x
>>>> Phoenix 4.4
>>>>
>>>> I am able to connect to Phoenix through JDBC connection and able to
>>>> read the Phoenix tables .
>>>> But while writing the data back to Phoenix table
>>>> I am getting below error :
>>>>
>>>> org.apache.spark.sql.AnalysisException:
>>>> org.apache.phoenix.spark.DefaultSource does not allow user-specified
>>>> schemas.;
>>>>
>>>> Can any body help in resolving the above errors or any other solution
>>>> of saving Spark Dataframes to Phoenix.
>>>>
>>>> Would really appareciate the help.
>>>>
>>>> Thanks,
>>>> Divya
>>>>
>>>
>>>
>>
>
Re: [HELP:]Save Spark Dataframe in Phoenix Table
Posted by Divya Gehlot <di...@gmail.com>.
Hi Josh,
You mean because of missing Jar I am not unable to write it to DB and
getting this exception "org.apache.spark.sql.AnalysisException:
org.apache.phoenix.spark.DefaultSource does not allow user-specified schemas
"
If I add the jar you mentioned I will be able to write it DB ?
Thanks,
Divya
On 9 April 2016 at 23:01, Josh Mahonin <jm...@gmail.com> wrote:
> Hi Divya,
>
> You don't have the phoenix client-spark JAR in your classpath, which is
> required for the phoenix-spark integration to work (as per the
> documentation).
>
> As well, you aren't using the vanilla Apache project that this mailing
> list supports, but are using a vendor packaged platform (Hortonworks).
> Since they maintain their own patches and forks to the upstream Apache
> versions, in general you should opt for filing support tickets with them
> first. In this particular case, HDP 2.3.4 doesn't actually provide the
> necessary phoenix client-spark JAR by default, so your options are limited
> here. Again, I recommend filing a support ticket with Hortonworks.
>
> Regards,
>
> Josh
>
> On Sat, Apr 9, 2016 at 9:11 AM, Divya Gehlot <di...@gmail.com>
> wrote:
>
>> Hi,
>> The code which I using to connect to Phoenix for writing
>> def writeToTable(df: DataFrame,dbtable: String) = {
>> val phx_properties = collection.immutable.Map[String, String](
>> "zkUrl" -> "localhost:2181:/hbase-unsecure",
>> "table" -> dbtable)
>>
>> df.write.format("org.apache.phoenix.spark").mode(SaveMode.Overwrite).options(phx_properties).saveAsTable(dbtable)
>> }
>>
>> While Submitting Spark job
>> * spark-shell --properties-file /TestDivya/Spark/Phoenix.properties
>> --jars
>> /usr/hdp/2.3.4.0-3485/hive/lib/hive-hbase-handler-1.2.1.2.3.4.0-3485.jar,/usr/hdp/2.3.4.0-3485/phoenix/lib/zookeeper.jar,/usr/hdp/2.3.4.0-3485/phoenix/lib/hbase-client.jar,/usr/hdp/2.3.4.0-3485/phoenix/lib/hbase-common.jar,/usr/hdp/2.3.4.0-3485/phoenix/lib/hbase-protocol.jar,/usr/hdp/2.3.4.0-3485/hbase/lib/phoenix-server.jar,/usr/hdp/2.3.4.0-3485/hbase/lib/phoenix-client-4.4.0.jar
>> --driver-class-path
>> /usr/hdp/2.3.4.0-3485/hbase/lib/phoenix-server.jar,/usr/hdp/2.3.4.0-3485/hbase/lib/phoenix-client-4.4.0.jar
>> --packages com.databricks:spark-csv_2.10:1.4.0 --master yarn-client -i
>> /TestDivya/Spark/WriteToPheonix.scala*
>>
>>
>> Getting the below error :
>>
>> org.apache.spark.sql.AnalysisException: org.apache.phoenix.spark.DefaultSource
>> does not allow user-specified schemas.;
>>
>> Am I on the right track or missing any properties ?
>>
>> Because of this I am unable to proceed with Phoenix and have to find
>> alternate options.
>> Would really appreciate the help
>>
>>
>>
>>
>>
>> ---------- Forwarded message ----------
>> From: Divya Gehlot <di...@gmail.com>
>> Date: 8 April 2016 at 19:54
>> Subject: Re: [HELP:]Save Spark Dataframe in Phoenix Table
>> To: Josh Mahonin <jm...@gmail.com>
>>
>>
>> Hi Josh,
>> I am doing in the same manner as mentioned in Phoenix Spark manner.
>> Using the latest version of HDP 2.3.4 .
>> In case of version mismatch/lack of spark Phoenix support it's should
>> have thrown the error at read also.
>> Which is working fine as expected .
>> Will surely pass on the code snippets once I log on to my System.
>> In the mean while I would like to know the zkURL parameter.If I build it
>> with HbaseConfiguration and passing zk quorom ,znode and port .
>> It throws error for example localhost :2181/hbase-unsecure
>> This localhost gets replaced by all the quorom
>> Like quorum1,quorum2:2181/hbase-unsecure
>>
>> I am just providing the IP address of my HBase master.
>>
>> I feel like I am not on right track so asked for the help .
>> How to connect to Phoenix through Spark on hadoop cluster .
>> Thanks for the help.
>> Cheers,
>> Divya
>> On Apr 8, 2016 7:06 PM, "Josh Mahonin" <jm...@gmail.com> wrote:
>>
>>> Hi Divya,
>>>
>>> That's strange. Are you able to post a snippet of your code to look at?
>>> And are you sure that you're saving the dataframes as per the docs (
>>> https://phoenix.apache.org/phoenix_spark.html)?
>>>
>>> Depending on your HDP version, it may or may not actually have
>>> phoenix-spark support. Double-check that your Spark configuration is setup
>>> with the right worker/driver classpath settings. and that the phoenix JARs
>>> contain the necessary phoenix-spark classes
>>> (e.g. org.apache.phoenix.spark.PhoenixRelation). If not, I suggest
>>> following up with Hortonworks.
>>>
>>> Josh
>>>
>>>
>>>
>>> On Fri, Apr 8, 2016 at 1:22 AM, Divya Gehlot <di...@gmail.com>
>>> wrote:
>>>
>>>> Hi,
>>>> I hava a Hortonworks Hadoop cluster having below Configurations :
>>>> Spark 1.5.2
>>>> HBASE 1.1.x
>>>> Phoenix 4.4
>>>>
>>>> I am able to connect to Phoenix through JDBC connection and able to
>>>> read the Phoenix tables .
>>>> But while writing the data back to Phoenix table
>>>> I am getting below error :
>>>>
>>>> org.apache.spark.sql.AnalysisException:
>>>> org.apache.phoenix.spark.DefaultSource does not allow user-specified
>>>> schemas.;
>>>>
>>>> Can any body help in resolving the above errors or any other solution
>>>> of saving Spark Dataframes to Phoenix.
>>>>
>>>> Would really appareciate the help.
>>>>
>>>> Thanks,
>>>> Divya
>>>>
>>>
>>>
>>
>
Re: [HELP:]Save Spark Dataframe in Phoenix Table
Posted by Josh Mahonin <jm...@gmail.com>.
Hi Divya,
You don't have the phoenix client-spark JAR in your classpath, which is
required for the phoenix-spark integration to work (as per the
documentation).
As well, you aren't using the vanilla Apache project that this mailing list
supports, but are using a vendor packaged platform (Hortonworks). Since
they maintain their own patches and forks to the upstream Apache versions,
in general you should opt for filing support tickets with them first. In
this particular case, HDP 2.3.4 doesn't actually provide the necessary
phoenix client-spark JAR by default, so your options are limited here.
Again, I recommend filing a support ticket with Hortonworks.
Regards,
Josh
On Sat, Apr 9, 2016 at 9:11 AM, Divya Gehlot <di...@gmail.com>
wrote:
> Hi,
> The code which I using to connect to Phoenix for writing
> def writeToTable(df: DataFrame,dbtable: String) = {
> val phx_properties = collection.immutable.Map[String, String](
> "zkUrl" -> "localhost:2181:/hbase-unsecure",
> "table" -> dbtable)
>
> df.write.format("org.apache.phoenix.spark").mode(SaveMode.Overwrite).options(phx_properties).saveAsTable(dbtable)
> }
>
> While Submitting Spark job
> * spark-shell --properties-file /TestDivya/Spark/Phoenix.properties
> --jars
> /usr/hdp/2.3.4.0-3485/hive/lib/hive-hbase-handler-1.2.1.2.3.4.0-3485.jar,/usr/hdp/2.3.4.0-3485/phoenix/lib/zookeeper.jar,/usr/hdp/2.3.4.0-3485/phoenix/lib/hbase-client.jar,/usr/hdp/2.3.4.0-3485/phoenix/lib/hbase-common.jar,/usr/hdp/2.3.4.0-3485/phoenix/lib/hbase-protocol.jar,/usr/hdp/2.3.4.0-3485/hbase/lib/phoenix-server.jar,/usr/hdp/2.3.4.0-3485/hbase/lib/phoenix-client-4.4.0.jar
> --driver-class-path
> /usr/hdp/2.3.4.0-3485/hbase/lib/phoenix-server.jar,/usr/hdp/2.3.4.0-3485/hbase/lib/phoenix-client-4.4.0.jar
> --packages com.databricks:spark-csv_2.10:1.4.0 --master yarn-client -i
> /TestDivya/Spark/WriteToPheonix.scala*
>
>
> Getting the below error :
>
> org.apache.spark.sql.AnalysisException: org.apache.phoenix.spark.DefaultSource
> does not allow user-specified schemas.;
>
> Am I on the right track or missing any properties ?
>
> Because of this I am unable to proceed with Phoenix and have to find
> alternate options.
> Would really appreciate the help
>
>
>
>
>
> ---------- Forwarded message ----------
> From: Divya Gehlot <di...@gmail.com>
> Date: 8 April 2016 at 19:54
> Subject: Re: [HELP:]Save Spark Dataframe in Phoenix Table
> To: Josh Mahonin <jm...@gmail.com>
>
>
> Hi Josh,
> I am doing in the same manner as mentioned in Phoenix Spark manner.
> Using the latest version of HDP 2.3.4 .
> In case of version mismatch/lack of spark Phoenix support it's should have
> thrown the error at read also.
> Which is working fine as expected .
> Will surely pass on the code snippets once I log on to my System.
> In the mean while I would like to know the zkURL parameter.If I build it
> with HbaseConfiguration and passing zk quorom ,znode and port .
> It throws error for example localhost :2181/hbase-unsecure
> This localhost gets replaced by all the quorom
> Like quorum1,quorum2:2181/hbase-unsecure
>
> I am just providing the IP address of my HBase master.
>
> I feel like I am not on right track so asked for the help .
> How to connect to Phoenix through Spark on hadoop cluster .
> Thanks for the help.
> Cheers,
> Divya
> On Apr 8, 2016 7:06 PM, "Josh Mahonin" <jm...@gmail.com> wrote:
>
>> Hi Divya,
>>
>> That's strange. Are you able to post a snippet of your code to look at?
>> And are you sure that you're saving the dataframes as per the docs (
>> https://phoenix.apache.org/phoenix_spark.html)?
>>
>> Depending on your HDP version, it may or may not actually have
>> phoenix-spark support. Double-check that your Spark configuration is setup
>> with the right worker/driver classpath settings. and that the phoenix JARs
>> contain the necessary phoenix-spark classes
>> (e.g. org.apache.phoenix.spark.PhoenixRelation). If not, I suggest
>> following up with Hortonworks.
>>
>> Josh
>>
>>
>>
>> On Fri, Apr 8, 2016 at 1:22 AM, Divya Gehlot <di...@gmail.com>
>> wrote:
>>
>>> Hi,
>>> I hava a Hortonworks Hadoop cluster having below Configurations :
>>> Spark 1.5.2
>>> HBASE 1.1.x
>>> Phoenix 4.4
>>>
>>> I am able to connect to Phoenix through JDBC connection and able to read
>>> the Phoenix tables .
>>> But while writing the data back to Phoenix table
>>> I am getting below error :
>>>
>>> org.apache.spark.sql.AnalysisException:
>>> org.apache.phoenix.spark.DefaultSource does not allow user-specified
>>> schemas.;
>>>
>>> Can any body help in resolving the above errors or any other solution of
>>> saving Spark Dataframes to Phoenix.
>>>
>>> Would really appareciate the help.
>>>
>>> Thanks,
>>> Divya
>>>
>>
>>
>
Fwd: [HELP:]Save Spark Dataframe in Phoenix Table
Posted by Divya Gehlot <di...@gmail.com>.
Hi,
The code which I using to connect to Phoenix for writing
def writeToTable(df: DataFrame,dbtable: String) = {
val phx_properties = collection.immutable.Map[String, String](
"zkUrl" -> "localhost:2181:/hbase-unsecure",
"table" -> dbtable)
df.write.format("org.apache.phoenix.spark").mode(SaveMode.Overwrite).options(phx_properties).saveAsTable(dbtable)
}
While Submitting Spark job
* spark-shell --properties-file /TestDivya/Spark/Phoenix.properties
--jars
/usr/hdp/2.3.4.0-3485/hive/lib/hive-hbase-handler-1.2.1.2.3.4.0-3485.jar,/usr/hdp/2.3.4.0-3485/phoenix/lib/zookeeper.jar,/usr/hdp/2.3.4.0-3485/phoenix/lib/hbase-client.jar,/usr/hdp/2.3.4.0-3485/phoenix/lib/hbase-common.jar,/usr/hdp/2.3.4.0-3485/phoenix/lib/hbase-protocol.jar,/usr/hdp/2.3.4.0-3485/hbase/lib/phoenix-server.jar,/usr/hdp/2.3.4.0-3485/hbase/lib/phoenix-client-4.4.0.jar
--driver-class-path
/usr/hdp/2.3.4.0-3485/hbase/lib/phoenix-server.jar,/usr/hdp/2.3.4.0-3485/hbase/lib/phoenix-client-4.4.0.jar
--packages com.databricks:spark-csv_2.10:1.4.0 --master yarn-client -i
/TestDivya/Spark/WriteToPheonix.scala*
Getting the below error :
org.apache.spark.sql.AnalysisException: org.apache.phoenix.spark.DefaultSource
does not allow user-specified schemas.;
Am I on the right track or missing any properties ?
Because of this I am unable to proceed with Phoenix and have to find
alternate options.
Would really appreciate the help
---------- Forwarded message ----------
From: Divya Gehlot <di...@gmail.com>
Date: 8 April 2016 at 19:54
Subject: Re: [HELP:]Save Spark Dataframe in Phoenix Table
To: Josh Mahonin <jm...@gmail.com>
Hi Josh,
I am doing in the same manner as mentioned in Phoenix Spark manner.
Using the latest version of HDP 2.3.4 .
In case of version mismatch/lack of spark Phoenix support it's should have
thrown the error at read also.
Which is working fine as expected .
Will surely pass on the code snippets once I log on to my System.
In the mean while I would like to know the zkURL parameter.If I build it
with HbaseConfiguration and passing zk quorom ,znode and port .
It throws error for example localhost :2181/hbase-unsecure
This localhost gets replaced by all the quorom
Like quorum1,quorum2:2181/hbase-unsecure
I am just providing the IP address of my HBase master.
I feel like I am not on right track so asked for the help .
How to connect to Phoenix through Spark on hadoop cluster .
Thanks for the help.
Cheers,
Divya
On Apr 8, 2016 7:06 PM, "Josh Mahonin" <jm...@gmail.com> wrote:
> Hi Divya,
>
> That's strange. Are you able to post a snippet of your code to look at?
> And are you sure that you're saving the dataframes as per the docs (
> https://phoenix.apache.org/phoenix_spark.html)?
>
> Depending on your HDP version, it may or may not actually have
> phoenix-spark support. Double-check that your Spark configuration is setup
> with the right worker/driver classpath settings. and that the phoenix JARs
> contain the necessary phoenix-spark classes
> (e.g. org.apache.phoenix.spark.PhoenixRelation). If not, I suggest
> following up with Hortonworks.
>
> Josh
>
>
>
> On Fri, Apr 8, 2016 at 1:22 AM, Divya Gehlot <di...@gmail.com>
> wrote:
>
>> Hi,
>> I hava a Hortonworks Hadoop cluster having below Configurations :
>> Spark 1.5.2
>> HBASE 1.1.x
>> Phoenix 4.4
>>
>> I am able to connect to Phoenix through JDBC connection and able to read
>> the Phoenix tables .
>> But while writing the data back to Phoenix table
>> I am getting below error :
>>
>> org.apache.spark.sql.AnalysisException:
>> org.apache.phoenix.spark.DefaultSource does not allow user-specified
>> schemas.;
>>
>> Can any body help in resolving the above errors or any other solution of
>> saving Spark Dataframes to Phoenix.
>>
>> Would really appareciate the help.
>>
>> Thanks,
>> Divya
>>
>
>
Re: [HELP:]Save Spark Dataframe in Phoenix Table
Posted by Josh Mahonin <jm...@gmail.com>.
Hi Divya,
That's strange. Are you able to post a snippet of your code to look at? And
are you sure that you're saving the dataframes as per the docs (
https://phoenix.apache.org/phoenix_spark.html)?
Depending on your HDP version, it may or may not actually have
phoenix-spark support. Double-check that your Spark configuration is setup
with the right worker/driver classpath settings. and that the phoenix JARs
contain the necessary phoenix-spark classes
(e.g. org.apache.phoenix.spark.PhoenixRelation). If not, I suggest
following up with Hortonworks.
Josh
On Fri, Apr 8, 2016 at 1:22 AM, Divya Gehlot <di...@gmail.com>
wrote:
> Hi,
> I hava a Hortonworks Hadoop cluster having below Configurations :
> Spark 1.5.2
> HBASE 1.1.x
> Phoenix 4.4
>
> I am able to connect to Phoenix through JDBC connection and able to read
> the Phoenix tables .
> But while writing the data back to Phoenix table
> I am getting below error :
>
> org.apache.spark.sql.AnalysisException:
> org.apache.phoenix.spark.DefaultSource does not allow user-specified
> schemas.;
>
> Can any body help in resolving the above errors or any other solution of
> saving Spark Dataframes to Phoenix.
>
> Would really appareciate the help.
>
> Thanks,
> Divya
>
Re: [HELP:]Save Spark Dataframe in Phoenix Table
Posted by Josh Mahonin <jm...@gmail.com>.
Hi Divya,
That's strange. Are you able to post a snippet of your code to look at? And
are you sure that you're saving the dataframes as per the docs (
https://phoenix.apache.org/phoenix_spark.html)?
Depending on your HDP version, it may or may not actually have
phoenix-spark support. Double-check that your Spark configuration is setup
with the right worker/driver classpath settings. and that the phoenix JARs
contain the necessary phoenix-spark classes
(e.g. org.apache.phoenix.spark.PhoenixRelation). If not, I suggest
following up with Hortonworks.
Josh
On Fri, Apr 8, 2016 at 1:22 AM, Divya Gehlot <di...@gmail.com>
wrote:
> Hi,
> I hava a Hortonworks Hadoop cluster having below Configurations :
> Spark 1.5.2
> HBASE 1.1.x
> Phoenix 4.4
>
> I am able to connect to Phoenix through JDBC connection and able to read
> the Phoenix tables .
> But while writing the data back to Phoenix table
> I am getting below error :
>
> org.apache.spark.sql.AnalysisException:
> org.apache.phoenix.spark.DefaultSource does not allow user-specified
> schemas.;
>
> Can any body help in resolving the above errors or any other solution of
> saving Spark Dataframes to Phoenix.
>
> Would really appareciate the help.
>
> Thanks,
> Divya
>
Re: [HELP:]Save Spark Dataframe in Phoenix Table
Posted by Josh Mahonin <jm...@gmail.com>.
Hi Divya,
That's strange. Are you able to post a snippet of your code to look at? And
are you sure that you're saving the dataframes as per the docs (
https://phoenix.apache.org/phoenix_spark.html)?
Depending on your HDP version, it may or may not actually have
phoenix-spark support. Double-check that your Spark configuration is setup
with the right worker/driver classpath settings. and that the phoenix JARs
contain the necessary phoenix-spark classes
(e.g. org.apache.phoenix.spark.PhoenixRelation). If not, I suggest
following up with Hortonworks.
Josh
On Fri, Apr 8, 2016 at 1:22 AM, Divya Gehlot <di...@gmail.com>
wrote:
> Hi,
> I hava a Hortonworks Hadoop cluster having below Configurations :
> Spark 1.5.2
> HBASE 1.1.x
> Phoenix 4.4
>
> I am able to connect to Phoenix through JDBC connection and able to read
> the Phoenix tables .
> But while writing the data back to Phoenix table
> I am getting below error :
>
> org.apache.spark.sql.AnalysisException:
> org.apache.phoenix.spark.DefaultSource does not allow user-specified
> schemas.;
>
> Can any body help in resolving the above errors or any other solution of
> saving Spark Dataframes to Phoenix.
>
> Would really appareciate the help.
>
> Thanks,
> Divya
>
Re: [HELP:]Save Spark Dataframe in Phoenix Table
Posted by Josh Mahonin <jm...@gmail.com>.
Hi Divya,
That's strange. Are you able to post a snippet of your code to look at? And
are you sure that you're saving the dataframes as per the docs (
https://phoenix.apache.org/phoenix_spark.html)?
Depending on your HDP version, it may or may not actually have
phoenix-spark support. Double-check that your Spark configuration is setup
with the right worker/driver classpath settings. and that the phoenix JARs
contain the necessary phoenix-spark classes
(e.g. org.apache.phoenix.spark.PhoenixRelation). If not, I suggest
following up with Hortonworks.
Josh
On Fri, Apr 8, 2016 at 1:22 AM, Divya Gehlot <di...@gmail.com>
wrote:
> Hi,
> I hava a Hortonworks Hadoop cluster having below Configurations :
> Spark 1.5.2
> HBASE 1.1.x
> Phoenix 4.4
>
> I am able to connect to Phoenix through JDBC connection and able to read
> the Phoenix tables .
> But while writing the data back to Phoenix table
> I am getting below error :
>
> org.apache.spark.sql.AnalysisException:
> org.apache.phoenix.spark.DefaultSource does not allow user-specified
> schemas.;
>
> Can any body help in resolving the above errors or any other solution of
> saving Spark Dataframes to Phoenix.
>
> Would really appareciate the help.
>
> Thanks,
> Divya
>
Re: [HELP:]Save Spark Dataframe in Phoenix Table
Posted by Josh Mahonin <jm...@gmail.com>.
Hi Divya,
That's strange. Are you able to post a snippet of your code to look at? And
are you sure that you're saving the dataframes as per the docs (
https://phoenix.apache.org/phoenix_spark.html)?
Depending on your HDP version, it may or may not actually have
phoenix-spark support. Double-check that your Spark configuration is setup
with the right worker/driver classpath settings. and that the phoenix JARs
contain the necessary phoenix-spark classes
(e.g. org.apache.phoenix.spark.PhoenixRelation). If not, I suggest
following up with Hortonworks.
Josh
On Fri, Apr 8, 2016 at 1:22 AM, Divya Gehlot <di...@gmail.com>
wrote:
> Hi,
> I hava a Hortonworks Hadoop cluster having below Configurations :
> Spark 1.5.2
> HBASE 1.1.x
> Phoenix 4.4
>
> I am able to connect to Phoenix through JDBC connection and able to read
> the Phoenix tables .
> But while writing the data back to Phoenix table
> I am getting below error :
>
> org.apache.spark.sql.AnalysisException:
> org.apache.phoenix.spark.DefaultSource does not allow user-specified
> schemas.;
>
> Can any body help in resolving the above errors or any other solution of
> saving Spark Dataframes to Phoenix.
>
> Would really appareciate the help.
>
> Thanks,
> Divya
>
Re: [HELP:]Save Spark Dataframe in Phoenix Table
Posted by Josh Mahonin <jm...@gmail.com>.
Hi Divya,
That's strange. Are you able to post a snippet of your code to look at? And
are you sure that you're saving the dataframes as per the docs (
https://phoenix.apache.org/phoenix_spark.html)?
Depending on your HDP version, it may or may not actually have
phoenix-spark support. Double-check that your Spark configuration is setup
with the right worker/driver classpath settings. and that the phoenix JARs
contain the necessary phoenix-spark classes
(e.g. org.apache.phoenix.spark.PhoenixRelation). If not, I suggest
following up with Hortonworks.
Josh
On Fri, Apr 8, 2016 at 1:22 AM, Divya Gehlot <di...@gmail.com>
wrote:
> Hi,
> I hava a Hortonworks Hadoop cluster having below Configurations :
> Spark 1.5.2
> HBASE 1.1.x
> Phoenix 4.4
>
> I am able to connect to Phoenix through JDBC connection and able to read
> the Phoenix tables .
> But while writing the data back to Phoenix table
> I am getting below error :
>
> org.apache.spark.sql.AnalysisException:
> org.apache.phoenix.spark.DefaultSource does not allow user-specified
> schemas.;
>
> Can any body help in resolving the above errors or any other solution of
> saving Spark Dataframes to Phoenix.
>
> Would really appareciate the help.
>
> Thanks,
> Divya
>
Re: [HELP:]Save Spark Dataframe in Phoenix Table
Posted by Josh Mahonin <jm...@gmail.com>.
Hi Divya,
That's strange. Are you able to post a snippet of your code to look at? And
are you sure that you're saving the dataframes as per the docs (
https://phoenix.apache.org/phoenix_spark.html)?
Depending on your HDP version, it may or may not actually have
phoenix-spark support. Double-check that your Spark configuration is setup
with the right worker/driver classpath settings. and that the phoenix JARs
contain the necessary phoenix-spark classes
(e.g. org.apache.phoenix.spark.PhoenixRelation). If not, I suggest
following up with Hortonworks.
Josh
On Fri, Apr 8, 2016 at 1:22 AM, Divya Gehlot <di...@gmail.com>
wrote:
> Hi,
> I hava a Hortonworks Hadoop cluster having below Configurations :
> Spark 1.5.2
> HBASE 1.1.x
> Phoenix 4.4
>
> I am able to connect to Phoenix through JDBC connection and able to read
> the Phoenix tables .
> But while writing the data back to Phoenix table
> I am getting below error :
>
> org.apache.spark.sql.AnalysisException:
> org.apache.phoenix.spark.DefaultSource does not allow user-specified
> schemas.;
>
> Can any body help in resolving the above errors or any other solution of
> saving Spark Dataframes to Phoenix.
>
> Would really appareciate the help.
>
> Thanks,
> Divya
>