You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Hao Ren <in...@gmail.com> on 2014/11/17 15:02:32 UTC
Building Spark with hive does not work
Hi,
I am building spark on the most recent master branch.
I checked this page:
https://github.com/apache/spark/blob/master/docs/sql-programming-guide.md
The cmd *./sbt/sbt -Phive -Phive-thirftserver clean assembly/assembly* works
fine. A fat jar is created.
However, when I started the SQL-CLI, I encountered an exception:
Spark assembly has been built with Hive, including Datanucleus jars on
classpath
java.lang.ClassNotFoundException:
org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:270)
at org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:337)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:75)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Failed to load main class
org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.
*You need to build Spark with -Phive and -Phive-thriftserver.*
Using Spark's default log4j profile:
org/apache/spark/log4j-defaults.properties
It's suggested to do with -Phive and -Phive-thriftserver, which is actually
what I have done.
Any idea ?
Hao
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Building-Spark-with-hive-does-not-work-tp19072.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Building Spark with hive does not work
Posted by Hao Ren <in...@gmail.com>.
Hi Cheng,
Actually, I am just using the commit 64c6b9b
Here is HEAD of the master branch:
$ git log
commit 64c6b9bad559c21f25cd9fbe37c8813cdab939f2
Author: Michael Armbrust <mi...@databricks.com>
Date: Sun Nov 16 21:55:57 2014 -0800
[SPARK-4410][SQL] Add support for external sort
Adds a new operator that uses Spark's `ExternalSort` class. It is off
by default now, but we might consider makin
Author: Michael Armbrust <mi...@databricks.com>
Closes #3268 from marmbrus/externalSort and squashes the following
commits:
48b9726 [Michael Armbrust] comments
b98799d [Michael Armbrust] Add test
afd7562 [Michael Armbrust] Add support for external sort.
I will also try to build spark on other PCs in order to isolated the
problem, cuz I have multiple versions of Spark on the current PC. No idea if
it is caused by that.
Hao
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Building-Spark-with-hive-does-not-work-tp19072p19085.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Building Spark with hive does not work
Posted by Hao Ren <in...@gmail.com>.
nvm, it would be better if correctness of flags could be checked by sbt
during building.
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Building-Spark-with-hive-does-not-work-tp19072p19183.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Building Spark with hive does not work
Posted by Cheng Lian <li...@gmail.com>.
Ah... Thanks Ted! And Hao, sorry for being the original trouble maker :)
On 11/18/14 1:50 AM, Ted Yu wrote:
> Looks like this was where you got that commandline:
>
> http://search-hadoop.com/m/JW1q5RlPrl
>
> Cheers
>
> On Mon, Nov 17, 2014 at 9:44 AM, Hao Ren <invkrh@gmail.com
> <ma...@gmail.com>> wrote:
>
> Sry for spamming,
>
> Just after my previous post, I noticed that the command used is:
>
> ./sbt/sbt -Phive -Phive-thirftserver clean assembly/assembly
>
> thriftserver*
>
> the typo error is the evil. Stupid, me.
>
> I believe I just copy-pasted from somewhere else, but no even
> checked it,
> meanwhile no error msg, such as "no such option", is displayed,
> which makes
> me consider the flags are correct.
>
> Sry for the carelessness.
>
> Hao
>
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/Building-Spark-with-hive-does-not-work-tp19072p19087.html
> Sent from the Apache Spark User List mailing list archive at
> Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> <ma...@spark.apache.org>
> For additional commands, e-mail: user-help@spark.apache.org
> <ma...@spark.apache.org>
>
>
Re: Building Spark with hive does not work
Posted by Ted Yu <yu...@gmail.com>.
Looks like this was where you got that commandline:
http://search-hadoop.com/m/JW1q5RlPrl
Cheers
On Mon, Nov 17, 2014 at 9:44 AM, Hao Ren <in...@gmail.com> wrote:
> Sry for spamming,
>
> Just after my previous post, I noticed that the command used is:
>
> ./sbt/sbt -Phive -Phive-thirftserver clean assembly/assembly
>
> thriftserver*
>
> the typo error is the evil. Stupid, me.
>
> I believe I just copy-pasted from somewhere else, but no even checked it,
> meanwhile no error msg, such as "no such option", is displayed, which makes
> me consider the flags are correct.
>
> Sry for the carelessness.
>
> Hao
>
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/Building-Spark-with-hive-does-not-work-tp19072p19087.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>
Re: Building Spark with hive does not work
Posted by Hao Ren <in...@gmail.com>.
Sry for spamming,
Just after my previous post, I noticed that the command used is:
./sbt/sbt -Phive -Phive-thirftserver clean assembly/assembly
thriftserver*
the typo error is the evil. Stupid, me.
I believe I just copy-pasted from somewhere else, but no even checked it,
meanwhile no error msg, such as "no such option", is displayed, which makes
me consider the flags are correct.
Sry for the carelessness.
Hao
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Building-Spark-with-hive-does-not-work-tp19072p19087.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Building Spark with hive does not work
Posted by Cheng Lian <li...@gmail.com>.
Hey Hao,
Which commit are you using? Just tried 64c6b9b with exactly the same
command line flags, couldn't reproduce this issue.
Cheng
On 11/17/14 10:02 PM, Hao Ren wrote:
> Hi,
>
> I am building spark on the most recent master branch.
>
> I checked this page:
> https://github.com/apache/spark/blob/master/docs/sql-programming-guide.md
>
> The cmd *./sbt/sbt -Phive -Phive-thirftserver clean assembly/assembly* works
> fine. A fat jar is created.
>
> However, when I started the SQL-CLI, I encountered an exception:
>
> Spark assembly has been built with Hive, including Datanucleus jars on
> classpath
> java.lang.ClassNotFoundException:
> org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver
> at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:270)
> at org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:337)
> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:75)
> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
> Failed to load main class
> org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.
> *You need to build Spark with -Phive and -Phive-thriftserver.*
> Using Spark's default log4j profile:
> org/apache/spark/log4j-defaults.properties
>
> It's suggested to do with -Phive and -Phive-thriftserver, which is actually
> what I have done.
>
> Any idea ?
>
> Hao
>
>
>
>
> --
> View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Building-Spark-with-hive-does-not-work-tp19072.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org