You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Ishwardeep Singh <is...@impetus.co.in> on 2014/11/28 08:30:49 UTC
Unable to compile spark 1.1.0 on windows 8.1
Hi,
I am trying to compile spark 1.1.0 on windows 8.1 but I get the following
exception.
[info] Compiling 3 Scala sources to
D:\myworkplace\software\spark-1.1.0\project\target\scala-2.10\sbt0.13\classes...
[error] D:\myworkplace\software\spark-1.1.0\project\SparkBuild.scala:26:
object sbt is not a member of package com.typesafe
[error] import com.typesafe.sbt.pom.{PomBuild, SbtPomKeys}
[error] ^
[error] D:\myworkplace\software\spark-1.1.0\project\SparkBuild.scala:53: not
found: type PomBuild
[error] object SparkBuild extends PomBuild {
[error] ^
[error] D:\myworkplace\software\spark-1.1.0\project\SparkBuild.scala:121:
not found: value SbtPomKeys
[error] otherResolvers <<= SbtPomKeys.mvnLocalRepository(dotM2 =>
Seq(Resolver.file("dotM2", dotM2))),
[error] ^
[error] D:\myworkplace\software\spark-1.1.0\project\SparkBuild.scala:165:
value projectDefinitions is not a member of AnyRef
[error] super.projectDefinitions(baseDirectory).map { x =>
[error] ^
[error] four errors found
[error] (plugins/compile:compile) Compilation failed
I have also setup scala 2.10.
Need help to resolve this issue.
Regards,
Ishwardeep
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Unable-to-compile-spark-1-1-0-on-windows-8-1-tp19996.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
RE: Unable to compile spark 1.1.0 on windows 8.1
Posted by Judy Nash <ju...@exchange.microsoft.com>.
Have you checked out the wiki here?
http://spark.apache.org/docs/latest/building-with-maven.html
A couple things I did differently from you:
1) I got the bits directly from github (https://github.com/apache/spark/). Use branch 1.1 for spark 1.1
2) execute maven command on cmd (powershell misses libraries sometimes)
3) Increase maven memory per suggested by building with maven wiki
Hope this helps.
-----Original Message-----
From: Ishwardeep Singh [mailto:ishwardeep.singh@impetus.co.in]
Sent: Monday, December 1, 2014 1:50 AM
To: user@spark.incubator.apache.org
Subject: RE: Unable to compile spark 1.1.0 on windows 8.1
Hi Judy,
Thank you for your response.
When I try to compile using maven "mvn -Dhadoop.version=1.2.1 -DskipTests clean package" I get an error "Error: Could not find or load main class" .
I have maven 3.0.4.
And when I run command "sbt package" I get the same exception as earlier.
I have done the following steps:
1. Download spark-1.1.0.tgz from the spark site and unzip the compressed zip to a folder "d:\myworkplace\software\spark-1.1.0"
2. Then I downloaded sbt-0.13.7.zip and extract it to folder "d:\myworkplace\software\sbt"
3. Update the PATH environment variable to include "d:\myworkplace\software\sbt\bin" in the PATH.
4. Navigate to spark folder d:\myworkplace\software\spark-1.1.0
5. Run the command "sbt assembly"
6. As a side effect of this command a number of libraries are downloaded and I get an initial error that path C:\Users\ishwardeep.singh\.sbt\0.13\staging\ec3aa8f39111944cc5f2\sbt-pom-reader
does not exist.
7. I manually create this subfolder "ec3aa8f39111944cc5f2\sbt-pom-reader"
and retry to get the next error as described in my initial error.
Is this the correct procedure to compile spark 1.1.0? Please let me know.
Hoping to hear from you soon.
Regards,
ishwardeep
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Unable-to-compile-spark-1-1-0-on-windows-8-1-tp19996p20075.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org For additional commands, e-mail: user-help@spark.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
RE: Unable to compile spark 1.1.0 on windows 8.1
Posted by Ishwardeep Singh <is...@impetus.co.in>.
Hi Judy,
Thank you for your response.
When I try to compile using maven "mvn -Dhadoop.version=1.2.1 -DskipTests
clean package" I get an error "Error: Could not find or load main class" .
I have maven 3.0.4.
And when I run command "sbt package" I get the same exception as earlier.
I have done the following steps:
1. Download spark-1.1.0.tgz from the spark site and unzip the compressed zip
to a folder "d:\myworkplace\software\spark-1.1.0"
2. Then I downloaded sbt-0.13.7.zip and extract it to folder
"d:\myworkplace\software\sbt"
3. Update the PATH environment variable to include
"d:\myworkplace\software\sbt\bin" in the PATH.
4. Navigate to spark folder d:\myworkplace\software\spark-1.1.0
5. Run the command "sbt assembly"
6. As a side effect of this command a number of libraries are downloaded and
I get an initial error that path
C:\Users\ishwardeep.singh\.sbt\0.13\staging\ec3aa8f39111944cc5f2\sbt-pom-reader
does not exist.
7. I manually create this subfolder "ec3aa8f39111944cc5f2\sbt-pom-reader"
and retry to get the next error as described in my initial error.
Is this the correct procedure to compile spark 1.1.0? Please let me know.
Hoping to hear from you soon.
Regards,
ishwardeep
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Unable-to-compile-spark-1-1-0-on-windows-8-1-tp19996p20075.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
RE: Unable to compile spark 1.1.0 on windows 8.1
Posted by Judy Nash <ju...@exchange.microsoft.com>.
I have found the following to work for me on win 8.1:
1) run sbt assembly
2) Use Maven. You can find the maven commands for your build at : docs\building-spark.md
-----Original Message-----
From: Ishwardeep Singh [mailto:ishwardeep.singh@impetus.co.in]
Sent: Thursday, November 27, 2014 11:31 PM
To: user@spark.incubator.apache.org
Subject: Unable to compile spark 1.1.0 on windows 8.1
Hi,
I am trying to compile spark 1.1.0 on windows 8.1 but I get the following exception.
[info] Compiling 3 Scala sources to
D:\myworkplace\software\spark-1.1.0\project\target\scala-2.10\sbt0.13\classes...
[error] D:\myworkplace\software\spark-1.1.0\project\SparkBuild.scala:26:
object sbt is not a member of package com.typesafe [error] import com.typesafe.sbt.pom.{PomBuild, SbtPomKeys}
[error] ^
[error] D:\myworkplace\software\spark-1.1.0\project\SparkBuild.scala:53: not
found: type PomBuild
[error] object SparkBuild extends PomBuild {
[error] ^
[error] D:\myworkplace\software\spark-1.1.0\project\SparkBuild.scala:121:
not found: value SbtPomKeys
[error] otherResolvers <<= SbtPomKeys.mvnLocalRepository(dotM2 =>
Seq(Resolver.file("dotM2", dotM2))),
[error] ^
[error] D:\myworkplace\software\spark-1.1.0\project\SparkBuild.scala:165:
value projectDefinitions is not a member of AnyRef
[error] super.projectDefinitions(baseDirectory).map { x =>
[error] ^
[error] four errors found
[error] (plugins/compile:compile) Compilation failed
I have also setup scala 2.10.
Need help to resolve this issue.
Regards,
Ishwardeep
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Unable-to-compile-spark-1-1-0-on-windows-8-1-tp19996.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org For additional commands, e-mail: user-help@spark.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org