You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@phoenix.apache.org by Nimrod Oren <ni...@veracity-group.com> on 2017/02/15 15:29:49 UTC
FW: Failing on writing Dataframe to Phoenix
Hi,
I'm trying to write a simple dataframe to Phoenix:
df.save("org.apache.phoenix.spark", SaveMode.Overwrite,
Map("table" -> "TEST_SAVE", "zkUrl" -> "zk.internal:2181"))
I have the following in my pom.xml:
<dependency>
<groupId>org.apache.phoenix</groupId>
<artifactId>phoenix-spark</artifactId>
<version>${phoenix-version}</version>
<scope>provided</scope>
</dependency>
and phoenix-spark is in spark-defaults.conf on all servers. However I'm
getting the following error:
Exception in thread "main" java.lang.NoClassDefFoundError:
org/apache/phoenix/util/SchemaUtil
at
org.apache.phoenix.spark.DataFrameFunctions$$anonfun$1.apply(DataFrameFunctions.scala:33)
at
org.apache.phoenix.spark.DataFrameFunctions$$anonfun$1.apply(DataFrameFunctions.scala:33)
at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
at
scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
at
scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:108)
at
scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
at scala.collection.mutable.ArrayOps$ofRef.map(ArrayOps.scala:108)
at
org.apache.phoenix.spark.DataFrameFunctions.saveToPhoenix(DataFrameFunctions.scala:33)
at
org.apache.phoenix.spark.DefaultSource.createRelation(DefaultSource.scala:47)
at
org.apache.spark.sql.execution.datasources.ResolvedDataSource$.apply(ResolvedDataSource.scala:222)
at
org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:148)
at org.apache.spark.sql.DataFrame.save(DataFrame.scala:2045)
at com.pelephone.TrueCallLoader$.main(TrueCallLoader.scala:184)
at com.pelephone.TrueCallLoader.main(TrueCallLoader.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:731)
at
org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181)
at
org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.ClassNotFoundException:
org.apache.phoenix.util.SchemaUtil
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
Am I missing something?
Nimrod
Re: FW: Failing on writing Dataframe to Phoenix
Posted by Dequn Zhang <d....@gmail.com>.
Hello, I used Phoenix–4.9-HBase–1.2, the pom file is like
<dependency>
<groupId>org.apache.phoenix</groupId>
<artifactId>phoenix-client</artifactId>
<version>${phoenix.version}</version>
</dependency>
and the save() method is unaccessible in Phoenix–4.9, so use the following
code instead, while “BIGJOY.TARJS” is my destination table ,
imoDF.sparkSession.sparkContext.hadoopConfiguration is for getting config
needed, traces is a dataset of class *Trajectory* which correspond to
schema of table “BIGJOY.TRAJS”.
traces.toDF().saveToPhoenix("BIGJOY.TRAJS",
imoDF.sparkSession.sparkContext.hadoopConfiguration,
Some("hadoop-master:2181"))
Hope this can help you.
On 15 February 2017 at 23:59:57, Josh Mahonin (jmahonin@gmail.com) wrote:
Hi,
Spark is unable to load the Phoenix classes it needs. If you're using a
recent version of Phoenix, please ensure the "fat" *client* JAR (or for
older versions of Phoenix, the Phoenix *client*-spark JAR) is on your Spark
driver and executor classpath [1]. The 'phoenix-spark' JAR is insufficient
to provide Spark all of the classes necessary.
[1] https://phoenix.apache.org/phoenix_spark.html
On Wed, Feb 15, 2017 at 10:29 AM, Nimrod Oren <
nimrod.oren@veracity-group.com> wrote:
> Hi,
>
>
>
> I'm trying to write a simple dataframe to Phoenix:
>
> df.save("org.apache.phoenix.spark", SaveMode.Overwrite,
>
> Map("table" -> "TEST_SAVE", "zkUrl" -> "zk.internal:2181"))
>
>
>
> I have the following in my pom.xml:
>
> <dependency>
>
> <groupId>org.apache.phoenix</groupId>
>
> <artifactId>phoenix-spark</artifactId>
>
> <version>${phoenix-version}</version>
>
> <scope>provided</scope>
>
> </dependency>
>
>
>
> and phoenix-spark is in spark-defaults.conf on all servers. However I'm
> getting the following error:
>
>
>
> Exception in thread "main" java.lang.NoClassDefFoundError:
> org/apache/phoenix/util/SchemaUtil
>
> at org.apache.phoenix.spark.DataFrameFunctions$$anonfun$1.
> apply(DataFrameFunctions.scala:33)
>
> at org.apache.phoenix.spark.DataFrameFunctions$$anonfun$1.
> apply(DataFrameFunctions.scala:33)
>
> at scala.collection.TraversableLike$$anonfun$map$
> 1.apply(TraversableLike.scala:244)
>
> at scala.collection.TraversableLike$$anonfun$map$
> 1.apply(TraversableLike.scala:244)
>
> at scala.collection.IndexedSeqOptimized$class.
> foreach(IndexedSeqOptimized.scala:33)
>
> at scala.collection.mutable.ArrayOps$ofRef.foreach(
> ArrayOps.scala:108)
>
> at scala.collection.TraversableLike$class.map(
> TraversableLike.scala:244)
>
> at scala.collection.mutable.ArrayOps$ofRef.map(ArrayOps.scala:108)
>
> at org.apache.phoenix.spark.DataFrameFunctions.saveToPhoenix(
> DataFrameFunctions.scala:33)
>
> at org.apache.phoenix.spark.DefaultSource.createRelation(
> DefaultSource.scala:47)
>
> at org.apache.spark.sql.execution.datasources.
> ResolvedDataSource$.apply(ResolvedDataSource.scala:222)
>
> at org.apache.spark.sql.DataFrameWriter.save(
> DataFrameWriter.scala:148)
>
> at org.apache.spark.sql.DataFrame.save(DataFrame.scala:2045)
>
> at com.pelephone.TrueCallLoader$.main(TrueCallLoader.scala:184)
>
> at com.pelephone.TrueCallLoader.main(TrueCallLoader.scala)
>
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>
> at sun.reflect.NativeMethodAccessorImpl.invoke(
> NativeMethodAccessorImpl.java:57)
>
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(
> DelegatingMethodAccessorImpl.java:43)
>
> at java.lang.reflect.Method.invoke(Method.java:606)
>
> at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$
> deploy$SparkSubmit$$runMain(SparkSubmit.scala:731)
>
> at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(
> SparkSubmit.scala:181)
>
> at org.apache.spark.deploy.SparkSubmit$.submit(
> SparkSubmit.scala:206)
>
> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.
> scala:121)
>
> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
>
> Caused by: java.lang.ClassNotFoundException: org.apache.phoenix.util.
> SchemaUtil
>
> at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
>
> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
>
> at java.security.AccessController.doPrivileged(Native Method)
>
> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
>
> at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
>
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
>
> at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
>
>
>
> Am I missing something?
>
>
>
> Nimrod
>
>
>
>
>
>
>
Re: Still having issues
Posted by Andrew Purtell <ap...@apache.org>.
If you are using CDH, Phoenix has to be patched to work with Cloudera's
HBase derivative. You might try https://github.com/
chiastic-security/phoenix-for-cloudera/tree/4.9-HBase-1.2-cdh5.9
<https://github.com/chiastic-security/phoenix-for-cloudera/tree/4.9-HBase-1.2-cdh5.9?files=1>
Check out this tree and build. The procedure for building is the same (
https://phoenix.apache.org/building.html) This will pick up Cloudera's
artifacts and produce a binary compatible result.
On Thu, Feb 16, 2017 at 7:53 AM, Josh Mahonin <jm...@gmail.com> wrote:
> It still seems that Spark is unable to find all of the Phoenix/HBase
> classes that are necessary.
>
> As a reference, I've got a Docker image that might help:
>
> https://github.com/jmahonin/docker-phoenix/tree/phoenix_spark
>
> The versions of Phoenix and Spark it uses are a bit out of date, but it
> shows the necessary classes and settings to make Spark happy.
>
> Good luck!
>
> Josh
>
> On Thu, Feb 16, 2017 at 4:10 AM, Dequn Zhang <d....@gmail.com>
> wrote:
>
>> Please check whether your table is created by Phoenix( means this table
>> is not a *Mapping* ) , you can follow the sample on phoenix official
>> site, only need *change the version to the latest*, use *phoenix-client*
>> instead, and promise *Schema Corresponding*. Create a new table to test,
>> use simple data, *be independent to your business*.
>>
>> We don’t know your background and code, so that’s all I can help you.
>>
>>
>> On 16 February 2017 at 16:57:26, Nimrod Oren (
>> nimrod.oren@veracity-group.com) wrote:
>>
>> org.apache.hadoop.hbase.HTableDescriptor.setValue
>>
>>
>
--
Best regards,
- Andy
If you are given a choice, you believe you have acted freely. - Raymond
Teller (via Peter Watts)
Re: Still having issues
Posted by Josh Mahonin <jm...@gmail.com>.
It still seems that Spark is unable to find all of the Phoenix/HBase
classes that are necessary.
As a reference, I've got a Docker image that might help:
https://github.com/jmahonin/docker-phoenix/tree/phoenix_spark
The versions of Phoenix and Spark it uses are a bit out of date, but it
shows the necessary classes and settings to make Spark happy.
Good luck!
Josh
On Thu, Feb 16, 2017 at 4:10 AM, Dequn Zhang <d....@gmail.com>
wrote:
> Please check whether your table is created by Phoenix( means this table is
> not a *Mapping* ) , you can follow the sample on phoenix official site,
> only need *change the version to the latest*, use *phoenix-client*
> instead, and promise *Schema Corresponding*. Create a new table to test,
> use simple data, *be independent to your business*.
>
> We don’t know your background and code, so that’s all I can help you.
>
>
> On 16 February 2017 at 16:57:26, Nimrod Oren (nimrod.oren@veracity-group.
> com) wrote:
>
> org.apache.hadoop.hbase.HTableDescriptor.setValue
>
>
Re:
Posted by Dequn Zhang <d....@gmail.com>.
Sorry for my tone, I didn’t mean that.
I use Phoenix4.9 on HBase1.2, and according to your err info, I looked up
HTableDescriptor definition and found this method
public HTableDescriptor setValue(String key, String value) {
if(value == null) {
this.remove(key);
} else {
this.setValue(Bytes.toBytes(key), Bytes.toBytes(value));
}
return this;
}
It looks like a little weird you get this error, if you can, check your
HTableDescriptor definition, add the above code to it and have a try, if
this can’t work either, maybe you can upgrade your HBase version.
On 16 February 2017 at 17:50:00, Nimrod Oren (nimrod.oren@veracity-group.com)
wrote:
I didn't mean to sound rude.
I created another table using phoenix-sqlline according to the
documentation:
CREATE TABLE IF NOT EXISTS TEST_SAVE_NEW (
IMEI VARCHAR NOT NULL PRIMARY KEY,
END_LATITUDE VARCHAR);
Still getting the same issue.
Re:
Posted by Andrew Purtell <an...@gmail.com>.
Phoenix has to be patched to work with Cloudera's HBase derivative. You might try https://github.com/chiastic-security/phoenix-for-cloudera/tree/4.9-HBase-1.2-cdh5.9
Check out this tree and build. The procedure for building is the same (https://phoenix.apache.org/building.html) This will pick up Cloudera's artifacts and produce a binary compatible result.
> On Feb 16, 2017, at 8:07 AM, Marcin Januszkiewicz <ka...@gmail.com> wrote:
>
> Are you maybe using a HBase distribution from a vendor? I had similar errors when trying to use Phoenix 4.9 on the Cloudera build of HBase.
>
>> On 2017-02-16 10:46 (+0100), "Nimrod Oren"<ni...@veracity-group.com> wrote:
>> I didn't mean to sound rude.
>>
>> I created another table using phoenix-sqlline according to the documentation:
>>
>> CREATE TABLE IF NOT EXISTS TEST_SAVE_NEW (
>> IMEI VARCHAR NOT NULL PRIMARY KEY,
>> END_LATITUDE VARCHAR);
>>
>> Still getting the same issue.
>>
Re:
Posted by Marcin Januszkiewicz <ka...@gmail.com>.
Are you maybe using a HBase distribution from a vendor? I had similar errors when trying to use Phoenix 4.9 on the Cloudera build of HBase.
On 2017-02-16 10:46 (+0100), "Nimrod Oren"<ni...@veracity-group.com> wrote:
> I didn't mean to sound rude.
>
> I created another table using phoenix-sqlline according to the documentation:
>
> CREATE TABLE IF NOT EXISTS TEST_SAVE_NEW (
> IMEI VARCHAR NOT NULL PRIMARY KEY,
> END_LATITUDE VARCHAR);
>
> Still getting the same issue.
>
Posted by Nimrod Oren <ni...@veracity-group.com>.
I didn't mean to sound rude.
I created another table using phoenix-sqlline according to the documentation:
CREATE TABLE IF NOT EXISTS TEST_SAVE_NEW (
IMEI VARCHAR NOT NULL PRIMARY KEY,
END_LATITUDE VARCHAR);
Still getting the same issue.
Re: Still having issues
Posted by Dequn Zhang <d....@gmail.com>.
Please check whether your table is created by Phoenix( means this table is
not a *Mapping* ) , you can follow the sample on phoenix official site,
only need *change the version to the latest*, use *phoenix-client* instead,
and promise *Schema Corresponding*. Create a new table to test, use simple
data, *be independent to your business*.
We don’t know your background and code, so that’s all I can help you.
On 16 February 2017 at 16:57:26, Nimrod Oren (nimrod.oren@veracity-group.com)
wrote:
org.apache.hadoop.hbase.HTableDescriptor.setValue
Still having issues
Posted by Nimrod Oren <ni...@veracity-group.com>.
Hi,
Even after adding phoenix-spark and phoenix-client I keep missing other methods. Now it's HBase methods:
java.lang.NoSuchMethodError: org.apache.hadoop.hbase.HTableDescriptor.setValue(Ljava/lang/String;Ljava/lang/String;)Lorg/apache/hadoop/hbase/HTableDescriptor
Isn't there a way for a beginner like me to find all the needed dependencies in a single place?
NImrod
Re: FW: Failing on writing Dataframe to Phoenix
Posted by Josh Mahonin <jm...@gmail.com>.
Hi,
Spark is unable to load the Phoenix classes it needs. If you're using a
recent version of Phoenix, please ensure the "fat" *client* JAR (or for
older versions of Phoenix, the Phoenix *client*-spark JAR) is on your Spark
driver and executor classpath [1]. The 'phoenix-spark' JAR is insufficient
to provide Spark all of the classes necessary.
[1] https://phoenix.apache.org/phoenix_spark.html
On Wed, Feb 15, 2017 at 10:29 AM, Nimrod Oren <
nimrod.oren@veracity-group.com> wrote:
> Hi,
>
>
>
> I'm trying to write a simple dataframe to Phoenix:
>
> df.save("org.apache.phoenix.spark", SaveMode.Overwrite,
>
> Map("table" -> "TEST_SAVE", "zkUrl" -> "zk.internal:2181"))
>
>
>
> I have the following in my pom.xml:
>
> <dependency>
>
> <groupId>org.apache.phoenix</groupId>
>
> <artifactId>phoenix-spark</artifactId>
>
> <version>${phoenix-version}</version>
>
> <scope>provided</scope>
>
> </dependency>
>
>
>
> and phoenix-spark is in spark-defaults.conf on all servers. However I'm
> getting the following error:
>
>
>
> Exception in thread "main" java.lang.NoClassDefFoundError:
> org/apache/phoenix/util/SchemaUtil
>
> at org.apache.phoenix.spark.DataFrameFunctions$$anonfun$1.
> apply(DataFrameFunctions.scala:33)
>
> at org.apache.phoenix.spark.DataFrameFunctions$$anonfun$1.
> apply(DataFrameFunctions.scala:33)
>
> at scala.collection.TraversableLike$$anonfun$map$
> 1.apply(TraversableLike.scala:244)
>
> at scala.collection.TraversableLike$$anonfun$map$
> 1.apply(TraversableLike.scala:244)
>
> at scala.collection.IndexedSeqOptimized$class.
> foreach(IndexedSeqOptimized.scala:33)
>
> at scala.collection.mutable.ArrayOps$ofRef.foreach(
> ArrayOps.scala:108)
>
> at scala.collection.TraversableLike$class.map(
> TraversableLike.scala:244)
>
> at scala.collection.mutable.ArrayOps$ofRef.map(ArrayOps.scala:108)
>
> at org.apache.phoenix.spark.DataFrameFunctions.saveToPhoenix(
> DataFrameFunctions.scala:33)
>
> at org.apache.phoenix.spark.DefaultSource.createRelation(
> DefaultSource.scala:47)
>
> at org.apache.spark.sql.execution.datasources.
> ResolvedDataSource$.apply(ResolvedDataSource.scala:222)
>
> at org.apache.spark.sql.DataFrameWriter.save(
> DataFrameWriter.scala:148)
>
> at org.apache.spark.sql.DataFrame.save(DataFrame.scala:2045)
>
> at com.pelephone.TrueCallLoader$.main(TrueCallLoader.scala:184)
>
> at com.pelephone.TrueCallLoader.main(TrueCallLoader.scala)
>
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>
> at sun.reflect.NativeMethodAccessorImpl.invoke(
> NativeMethodAccessorImpl.java:57)
>
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(
> DelegatingMethodAccessorImpl.java:43)
>
> at java.lang.reflect.Method.invoke(Method.java:606)
>
> at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$
> deploy$SparkSubmit$$runMain(SparkSubmit.scala:731)
>
> at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(
> SparkSubmit.scala:181)
>
> at org.apache.spark.deploy.SparkSubmit$.submit(
> SparkSubmit.scala:206)
>
> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.
> scala:121)
>
> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
>
> Caused by: java.lang.ClassNotFoundException: org.apache.phoenix.util.
> SchemaUtil
>
> at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
>
> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
>
> at java.security.AccessController.doPrivileged(Native Method)
>
> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
>
> at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
>
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
>
> at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
>
>
>
> Am I missing something?
>
>
>
> Nimrod
>
>
>
>
>
>
>