You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by "vdiwakar.malladi" <vd...@gmail.com> on 2014/11/26 17:53:35 UTC
Unable to generate assembly jar which includes jdbc-thrift server
Hi,
When I'm trying to build spark assembly to include the dependencies related
to thrift server, build is getting failed by throwing the following error.
Could any one help me on this.
[ERROR] Failed to execute goal on project spark-assembly_2.10: Could not
resolve dependencies for project
org.apache.spark:spark-assembly_2.10:pom:1.1.0: Couldnot find artifact
org.apache.spark:spark-hive-thriftserver_2.10:jar:1.1.0 in central
(https://repo1.maven.org/maven2) -> [Help 1]
Thanks in advance.
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Unable-to-generate-assembly-jar-which-includes-jdbc-thrift-server-tp19887.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Unable to generate assembly jar which includes jdbc-thrift server
Posted by Cheng Lian <li...@gmail.com>.
Actually all components are merged into the assembly jar under
assembly/target/scala-2.10. You don’t need to configure the classpath
unless you need to include some customized jars (e.g. customized Hive
SerDes). Existing scripts can compute the classpath correctly (if they
can’t, it should be a bug). You may refer to |bin/compute-classpath.sh|
for details.
On 11/27/14 4:27 PM, vdiwakar.malladi wrote:
> Hi,
>
> I setup maven environment on a Linux machine and able to build the pom file
> in spark home directory. Each module refreshed with corresponding target
> directory with jar files.
>
> In order to include all the libraries to classpath, what I need to do?
> earlier, I used single assembly jar file to include the same in classpath
> (without having hive profile by running pom file available in assembly
> folder). But now, I could see jar files generated in the individual module
> folders. Could you please advice.
>
> Thanks in advance.
>
>
>
> --
> View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Unable-to-generate-assembly-jar-which-includes-jdbc-thrift-server-tp19887p19963.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>
Re: Unable to generate assembly jar which includes jdbc-thrift
server
Posted by "vdiwakar.malladi" <vd...@gmail.com>.
Hi,
I setup maven environment on a Linux machine and able to build the pom file
in spark home directory. Each module refreshed with corresponding target
directory with jar files.
In order to include all the libraries to classpath, what I need to do?
earlier, I used single assembly jar file to include the same in classpath
(without having hive profile by running pom file available in assembly
folder). But now, I could see jar files generated in the individual module
folders. Could you please advice.
Thanks in advance.
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Unable-to-generate-assembly-jar-which-includes-jdbc-thrift-server-tp19887p19963.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Unable to generate assembly jar which includes jdbc-thrift server
Posted by Cheng Lian <li...@gmail.com>.
I see. As what the exception stated, Maven can’t find |unzip| to help
building PySpark. So you need a Windows version of |unzip| (probably
from MinGW or Cygwin?)
On 11/27/14 2:10 PM, vdiwakar.malladi wrote:
> Thanks for your prompt responses.
>
> I'm generating assembly jar file from windows 7 machine. When i try to build
> it from parent directory, I'm getting the following exception. Can I do it
> from windows machine?
>
> Note: Earlier, I could able to generate the assembly jar from windows box
> (with out hive profile from the pom file in assembly directory not from
> parent directory)
>
> [ERROR] Failed to execute goal
> org.codehaus.mojo:exec-maven-plugin:1.2.1:exec (default) on project
> spark-core_2.10: Command execution failed. Cannot run program "unzip" (in
> directory "C:\spark-1.1.0\python"): CreateProcess error=2, The system cannot
> find the file specified -> [Help 1]
>
> Thanks in advance.
>
>
>
> --
> View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Unable-to-generate-assembly-jar-which-includes-jdbc-thrift-server-tp19887p19945.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>
Re: Unable to generate assembly jar which includes jdbc-thrift
server
Posted by "vdiwakar.malladi" <vd...@gmail.com>.
Thanks for your prompt responses.
I'm generating assembly jar file from windows 7 machine. When i try to build
it from parent directory, I'm getting the following exception. Can I do it
from windows machine?
Note: Earlier, I could able to generate the assembly jar from windows box
(with out hive profile from the pom file in assembly directory not from
parent directory)
[ERROR] Failed to execute goal
org.codehaus.mojo:exec-maven-plugin:1.2.1:exec (default) on project
spark-core_2.10: Command execution failed. Cannot run program "unzip" (in
directory "C:\spark-1.1.0\python"): CreateProcess error=2, The system cannot
find the file specified -> [Help 1]
Thanks in advance.
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Unable-to-generate-assembly-jar-which-includes-jdbc-thrift-server-tp19887p19945.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Unable to generate assembly jar which includes jdbc-thrift server
Posted by Cheng Lian <li...@gmail.com>.
Hm, then the command line you used should be fine. Actually just tried
it locally and it’s fine. Make sure to run it in the root directory of
Spark source tree (don’t |cd| into assembly).
On 11/27/14 1:35 PM, vdiwakar.malladi wrote:
> Yes, I'm building it from Spark 1.1.0
>
> Thanks in advance.
>
>
>
> --
> View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Unable-to-generate-assembly-jar-which-includes-jdbc-thrift-server-tp19887p19937.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>
Re: Unable to generate assembly jar which includes jdbc-thrift
server
Posted by "vdiwakar.malladi" <vd...@gmail.com>.
Yes, I'm building it from Spark 1.1.0
Thanks in advance.
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Unable-to-generate-assembly-jar-which-includes-jdbc-thrift-server-tp19887p19937.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Unable to generate assembly jar which includes jdbc-thrift server
Posted by Cheng Lian <li...@gmail.com>.
What version are you trying to build? I was at first assuming you're
using the most recent master, but from your first mail it seems that you
were trying to build Spark v1.1.0?
On 11/27/14 12:57 PM, vdiwakar.malladi wrote:
> Thanks for your response.
>
> I'm using the following command.
>
> mvn -Pyarn -Phadoop-2.4 -Dhadoop.version=2.4.0 -Phive -DskipTests clean
> package
>
> Regards.
>
>
>
> --
> View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Unable-to-generate-assembly-jar-which-includes-jdbc-thrift-server-tp19887p19933.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Unable to generate assembly jar which includes jdbc-thrift
server
Posted by "vdiwakar.malladi" <vd...@gmail.com>.
Thanks for your response.
I'm using the following command.
mvn -Pyarn -Phadoop-2.4 -Dhadoop.version=2.4.0 -Phive -DskipTests clean
package
Regards.
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Unable-to-generate-assembly-jar-which-includes-jdbc-thrift-server-tp19887p19933.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Unable to generate assembly jar which includes jdbc-thrift server
Posted by Cheng Lian <li...@gmail.com>.
What’s the command line you used to build Spark? Notice that you need to
add |-Phive-thriftserver| to build the JDBC Thrift server. This profile
was once removed in in v1.1.0, but added back in v1.2.0 because of
dependency issue introduced by Scala 2.11 support.
On 11/27/14 12:53 AM, vdiwakar.malladi wrote:
> Hi,
>
> When I'm trying to build spark assembly to include the dependencies related
> to thrift server, build is getting failed by throwing the following error.
> Could any one help me on this.
>
> [ERROR] Failed to execute goal on project spark-assembly_2.10: Could not
> resolve dependencies for project
> org.apache.spark:spark-assembly_2.10:pom:1.1.0: Couldnot find artifact
> org.apache.spark:spark-hive-thriftserver_2.10:jar:1.1.0 in central
> (https://repo1.maven.org/maven2) -> [Help 1]
>
> Thanks in advance.
>
>
>
>
>
> --
> View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Unable-to-generate-assembly-jar-which-includes-jdbc-thrift-server-tp19887.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>