You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Thorsten Bergler <sp...@tbonline.de> on 2014/09/10 12:46:01 UTC
Dependency Problem with Spark / ScalaTest / SBT
Hello,
I am writing a Spark App which is already working so far.
Now I started to build also some UnitTests, but I am running into some
dependecy problems and I cannot find a solution right now. Perhaps
someone could help me.
I build my Spark Project with SBT and it seems to be configured well,
because compiling, assembling and running the built jar with
spark-submit are working well.
Now I started with the UnitTests, which I located under /src/test/scala.
When I call "test" in sbt, I get the following:
14/09/10 12:22:06 INFO storage.BlockManagerMaster: Registered BlockManager
14/09/10 12:22:06 INFO spark.HttpServer: Starting HTTP Server
[trace] Stack trace suppressed: run last test:test for the full output.
[error] Could not run test test.scala.SetSuite:
java.lang.NoClassDefFoundError: javax/servlet/http/HttpServletResponse
[info] Run completed in 626 milliseconds.
[info] Total number of tests run: 0
[info] Suites: completed 0, aborted 0
[info] Tests: succeeded 0, failed 0, canceled 0, ignored 0, pending 0
[info] All tests passed.
[error] Error during tests:
[error] test.scala.SetSuite
[error] (test:test) sbt.TestsFailedException: Tests unsuccessful
[error] Total time: 3 s, completed 10.09.2014 12:22:06
last test:test gives me the following:
> last test:test
[debug] Running TaskDef(test.scala.SetSuite,
org.scalatest.tools.Framework$$anon$1@6e5626c8, false, [SuiteSelector])
java.lang.NoClassDefFoundError: javax/servlet/http/HttpServletResponse
at org.apache.spark.HttpServer.start(HttpServer.scala:54)
at
org.apache.spark.broadcast.HttpBroadcast$.createServer(HttpBroadcast.scala:156)
at
org.apache.spark.broadcast.HttpBroadcast$.initialize(HttpBroadcast.scala:127)
at
org.apache.spark.broadcast.HttpBroadcastFactory.initialize(HttpBroadcastFactory.scala:31)
at
org.apache.spark.broadcast.BroadcastManager.initialize(BroadcastManager.scala:48)
at
org.apache.spark.broadcast.BroadcastManager.<init>(BroadcastManager.scala:35)
at org.apache.spark.SparkEnv$.create(SparkEnv.scala:218)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:202)
at test.scala.SetSuite.<init>(SparkTest.scala:16)
I also noticed right now, that sbt run is also not working:
14/09/10 12:44:46 INFO spark.HttpServer: Starting HTTP Server
[error] (run-main-2) java.lang.NoClassDefFoundError:
javax/servlet/http/HttpServletResponse
java.lang.NoClassDefFoundError: javax/servlet/http/HttpServletResponse
at org.apache.spark.HttpServer.start(HttpServer.scala:54)
at
org.apache.spark.broadcast.HttpBroadcast$.createServer(HttpBroadcast.scala:156)
at
org.apache.spark.broadcast.HttpBroadcast$.initialize(HttpBroadcast.scala:127)
at
org.apache.spark.broadcast.HttpBroadcastFactory.initialize(HttpBroadcastFactory.scala:31)
at
org.apache.spark.broadcast.BroadcastManager.initialize(BroadcastManager.scala:48)
at
org.apache.spark.broadcast.BroadcastManager.<init>(BroadcastManager.scala:35)
at org.apache.spark.SparkEnv$.create(SparkEnv.scala:218)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:202)
at
main.scala.PartialDuplicateScanner$.main(PartialDuplicateScanner.scala:29)
at
main.scala.PartialDuplicateScanner.main(PartialDuplicateScanner.scala)
Here is my Testprojekt.sbt file:
name := "Testprojekt"
version := "1.0"
scalaVersion := "2.10.4"
libraryDependencies ++= {
Seq(
"org.apache.lucene" % "lucene-core" % "4.9.0",
"org.apache.lucene" % "lucene-analyzers-common" % "4.9.0",
"org.apache.lucene" % "lucene-queryparser" % "4.9.0",
("org.apache.spark" %% "spark-core" % "1.0.2").
exclude("org.mortbay.jetty", "servlet-api").
exclude("commons-beanutils", "commons-beanutils-core").
exclude("commons-collections", "commons-collections").
exclude("commons-collections", "commons-collections").
exclude("com.esotericsoftware.minlog", "minlog").
exclude("org.eclipse.jetty.orbit", "javax.mail.glassfish").
exclude("org.eclipse.jetty.orbit", "javax.transaction").
exclude("org.eclipse.jetty.orbit", "javax.servlet")
)
}
resolvers += "Akka Repository" at "http://repo.akka.io/releases/"
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Dependency Problem with Spark / ScalaTest / SBT
Posted by Thorsten Bergler <sp...@tbonline.de>.
Hello,
When I remove the line and try to execute "sbt run", I end up with the
following lines:
14/09/15 10:11:35 INFO ui.SparkUI: Stopped Spark web UI at http://base:4040
[...]
14/09/15 10:11:05 WARN scheduler.TaskSchedulerImpl: Initial job has not
accepted any resources; check your cluster UI to ensure that workers are
registered and have sufficient memory
14/09/15 10:11:15 INFO client.AppClient$ClientActor: Connecting to
master spark://base:7077...
It seems that the configuration within sbt doesn't use my original
Spark, because my original Spark web UI is running under
http://base:8080. Seems like sbt is starting another spark instance??
Best regards
Thorsten
Am 14.09.2014 um 18:56 schrieb Dean Wampler:
> Sorry, I meant any *other* SBT files.
>
> However, what happens if you remove the line:
>
> exclude("org.eclipse.jetty.orbit", "javax.servlet")
>
>
> dean
>
>
> Dean Wampler, Ph.D.
> Author: Programming Scala, 2nd Edition
> <http://shop.oreilly.com/product/0636920033073.do> (O'Reilly)
> Typesafe <http://typesafe.com>
> @deanwampler <http://twitter.com/deanwampler>
> http://polyglotprogramming.com
>
> On Sun, Sep 14, 2014 at 11:53 AM, Dean Wampler <deanwampler@gmail.com
> <ma...@gmail.com>> wrote:
>
> Can you post your whole SBT build file(s)?
>
> Dean Wampler, Ph.D.
> Author: Programming Scala, 2nd Edition
> <http://shop.oreilly.com/product/0636920033073.do> (O'Reilly)
> Typesafe <http://typesafe.com>
> @deanwampler <http://twitter.com/deanwampler>
> http://polyglotprogramming.com
>
> On Wed, Sep 10, 2014 at 6:48 AM, Thorsten Bergler
> <spark@tbonline.de <ma...@tbonline.de>> wrote:
>
> Hi,
>
> I just called:
>
> > test
>
> or
>
> > run
>
> Thorsten
>
>
> Am 10.09.2014 um 13:38 schrieb Arthur.hk.chan@gmail.com
> <ma...@gmail.com>:
>
> Hi,
>
> What is your SBT command and the parameters?
>
> Arthur
>
>
> On 10 Sep, 2014, at 6:46 pm, Thorsten Bergler
> <spark@tbonline.de <ma...@tbonline.de>> wrote:
>
> Hello,
>
> I am writing a Spark App which is already working so far.
> Now I started to build also some UnitTests, but I am
> running into some dependecy problems and I cannot find
> a solution right now. Perhaps someone could help me.
>
> I build my Spark Project with SBT and it seems to be
> configured well, because compiling, assembling and
> running the built jar with spark-submit are working well.
>
> Now I started with the UnitTests, which I located
> under /src/test/scala.
>
> When I call "test" in sbt, I get the following:
>
> 14/09/10 12:22:06 INFO storage.BlockManagerMaster:
> Registered BlockManager
> 14/09/10 12:22:06 INFO spark.HttpServer: Starting HTTP
> Server
> [trace] Stack trace suppressed: run last test:test for
> the full output.
> [error] Could not run test test.scala.SetSuite:
> java.lang.NoClassDefFoundError:
> javax/servlet/http/HttpServletResponse
> [info] Run completed in 626 milliseconds.
> [info] Total number of tests run: 0
> [info] Suites: completed 0, aborted 0
> [info] Tests: succeeded 0, failed 0, canceled 0,
> ignored 0, pending 0
> [info] All tests passed.
> [error] Error during tests:
> [error] test.scala.SetSuite
> [error] (test:test) sbt.TestsFailedException: Tests
> unsuccessful
> [error] Total time: 3 s, completed 10.09.2014 12:22:06
>
> last test:test gives me the following:
>
> last test:test
>
> [debug] Running TaskDef(test.scala.SetSuite,
> org.scalatest.tools.Framework$$anon$1@6e5626c8, false,
> [SuiteSelector])
> java.lang.NoClassDefFoundError:
> javax/servlet/http/HttpServletResponse
> at
> org.apache.spark.HttpServer.start(HttpServer.scala:54)
> at
> org.apache.spark.broadcast.HttpBroadcast$.createServer(HttpBroadcast.scala:156)
> at
> org.apache.spark.broadcast.HttpBroadcast$.initialize(HttpBroadcast.scala:127)
> at
> org.apache.spark.broadcast.HttpBroadcastFactory.initialize(HttpBroadcastFactory.scala:31)
> at
> org.apache.spark.broadcast.BroadcastManager.initialize(BroadcastManager.scala:48)
> at
> org.apache.spark.broadcast.BroadcastManager.<init>(BroadcastManager.scala:35)
> at
> org.apache.spark.SparkEnv$.create(SparkEnv.scala:218)
> at
> org.apache.spark.SparkContext.<init>(SparkContext.scala:202)
> at test.scala.SetSuite.<init>(SparkTest.scala:16)
>
> I also noticed right now, that sbt run is also not
> working:
>
> 14/09/10 12:44:46 INFO spark.HttpServer: Starting HTTP
> Server
> [error] (run-main-2) java.lang.NoClassDefFoundError:
> javax/servlet/http/HttpServletResponse
> java.lang.NoClassDefFoundError:
> javax/servlet/http/HttpServletResponse
> at
> org.apache.spark.HttpServer.start(HttpServer.scala:54)
> at
> org.apache.spark.broadcast.HttpBroadcast$.createServer(HttpBroadcast.scala:156)
> at
> org.apache.spark.broadcast.HttpBroadcast$.initialize(HttpBroadcast.scala:127)
> at
> org.apache.spark.broadcast.HttpBroadcastFactory.initialize(HttpBroadcastFactory.scala:31)
> at
> org.apache.spark.broadcast.BroadcastManager.initialize(BroadcastManager.scala:48)
> at
> org.apache.spark.broadcast.BroadcastManager.<init>(BroadcastManager.scala:35)
> at
> org.apache.spark.SparkEnv$.create(SparkEnv.scala:218)
> at
> org.apache.spark.SparkContext.<init>(SparkContext.scala:202)
> at
> main.scala.PartialDuplicateScanner$.main(PartialDuplicateScanner.scala:29)
> at
> main.scala.PartialDuplicateScanner.main(PartialDuplicateScanner.scala)
>
> Here is my Testprojekt.sbt file:
>
> name := "Testprojekt"
>
> version := "1.0"
>
> scalaVersion := "2.10.4"
>
> libraryDependencies ++= {
> Seq(
> "org.apache.lucene" % "lucene-core" % "4.9.0",
> "org.apache.lucene" % "lucene-analyzers-common" %
> "4.9.0",
> "org.apache.lucene" % "lucene-queryparser" % "4.9.0",
> ("org.apache.spark" %% "spark-core" % "1.0.2").
> exclude("org.mortbay.jetty", "servlet-api").
> exclude("commons-beanutils",
> "commons-beanutils-core").
> exclude("commons-collections",
> "commons-collections").
> exclude("commons-collections",
> "commons-collections").
> exclude("com.esotericsoftware.minlog", "minlog").
> exclude("org.eclipse.jetty.orbit",
> "javax.mail.glassfish").
> exclude("org.eclipse.jetty.orbit",
> "javax.transaction").
> exclude("org.eclipse.jetty.orbit",
> "javax.servlet")
> )
> }
>
> resolvers += "Akka Repository" at
> "http://repo.akka.io/releases/"
>
>
>
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail:
> user-unsubscribe@spark.apache.org
> <ma...@spark.apache.org>
> For additional commands, e-mail:
> user-help@spark.apache.org
> <ma...@spark.apache.org>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> <ma...@spark.apache.org>
> For additional commands, e-mail:
> user-help@spark.apache.org <ma...@spark.apache.org>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> <ma...@spark.apache.org>
> For additional commands, e-mail: user-help@spark.apache.org
> <ma...@spark.apache.org>
>
>
>
Re: Dependency Problem with Spark / ScalaTest / SBT
Posted by Dean Wampler <de...@gmail.com>.
Sorry, I meant any *other* SBT files.
However, what happens if you remove the line:
exclude("org.eclipse.jetty.orbit", "javax.servlet")
dean
Dean Wampler, Ph.D.
Author: Programming Scala, 2nd Edition
<http://shop.oreilly.com/product/0636920033073.do> (O'Reilly)
Typesafe <http://typesafe.com>
@deanwampler <http://twitter.com/deanwampler>
http://polyglotprogramming.com
On Sun, Sep 14, 2014 at 11:53 AM, Dean Wampler <de...@gmail.com>
wrote:
> Can you post your whole SBT build file(s)?
>
> Dean Wampler, Ph.D.
> Author: Programming Scala, 2nd Edition
> <http://shop.oreilly.com/product/0636920033073.do> (O'Reilly)
> Typesafe <http://typesafe.com>
> @deanwampler <http://twitter.com/deanwampler>
> http://polyglotprogramming.com
>
> On Wed, Sep 10, 2014 at 6:48 AM, Thorsten Bergler <sp...@tbonline.de>
> wrote:
>
>> Hi,
>>
>> I just called:
>>
>> > test
>>
>> or
>>
>> > run
>>
>> Thorsten
>>
>>
>> Am 10.09.2014 um 13:38 schrieb Arthur.hk.chan@gmail.com:
>>
>> Hi,
>>>
>>> What is your SBT command and the parameters?
>>>
>>> Arthur
>>>
>>>
>>> On 10 Sep, 2014, at 6:46 pm, Thorsten Bergler <sp...@tbonline.de> wrote:
>>>
>>> Hello,
>>>>
>>>> I am writing a Spark App which is already working so far.
>>>> Now I started to build also some UnitTests, but I am running into some
>>>> dependecy problems and I cannot find a solution right now. Perhaps someone
>>>> could help me.
>>>>
>>>> I build my Spark Project with SBT and it seems to be configured well,
>>>> because compiling, assembling and running the built jar with spark-submit
>>>> are working well.
>>>>
>>>> Now I started with the UnitTests, which I located under /src/test/scala.
>>>>
>>>> When I call "test" in sbt, I get the following:
>>>>
>>>> 14/09/10 12:22:06 INFO storage.BlockManagerMaster: Registered
>>>> BlockManager
>>>> 14/09/10 12:22:06 INFO spark.HttpServer: Starting HTTP Server
>>>> [trace] Stack trace suppressed: run last test:test for the full output.
>>>> [error] Could not run test test.scala.SetSuite: java.lang.NoClassDefFoundError:
>>>> javax/servlet/http/HttpServletResponse
>>>> [info] Run completed in 626 milliseconds.
>>>> [info] Total number of tests run: 0
>>>> [info] Suites: completed 0, aborted 0
>>>> [info] Tests: succeeded 0, failed 0, canceled 0, ignored 0, pending 0
>>>> [info] All tests passed.
>>>> [error] Error during tests:
>>>> [error] test.scala.SetSuite
>>>> [error] (test:test) sbt.TestsFailedException: Tests unsuccessful
>>>> [error] Total time: 3 s, completed 10.09.2014 12:22:06
>>>>
>>>> last test:test gives me the following:
>>>>
>>>> last test:test
>>>>>
>>>> [debug] Running TaskDef(test.scala.SetSuite,
>>>> org.scalatest.tools.Framework$$anon$1@6e5626c8, false, [SuiteSelector])
>>>> java.lang.NoClassDefFoundError: javax/servlet/http/HttpServletResponse
>>>> at org.apache.spark.HttpServer.start(HttpServer.scala:54)
>>>> at org.apache.spark.broadcast.HttpBroadcast$.createServer(
>>>> HttpBroadcast.scala:156)
>>>> at org.apache.spark.broadcast.HttpBroadcast$.initialize(
>>>> HttpBroadcast.scala:127)
>>>> at org.apache.spark.broadcast.HttpBroadcastFactory.initialize(
>>>> HttpBroadcastFactory.scala:31)
>>>> at org.apache.spark.broadcast.BroadcastManager.initialize(
>>>> BroadcastManager.scala:48)
>>>> at org.apache.spark.broadcast.BroadcastManager.<init>(
>>>> BroadcastManager.scala:35)
>>>> at org.apache.spark.SparkEnv$.create(SparkEnv.scala:218)
>>>> at org.apache.spark.SparkContext.<init>(SparkContext.scala:202)
>>>> at test.scala.SetSuite.<init>(SparkTest.scala:16)
>>>>
>>>> I also noticed right now, that sbt run is also not working:
>>>>
>>>> 14/09/10 12:44:46 INFO spark.HttpServer: Starting HTTP Server
>>>> [error] (run-main-2) java.lang.NoClassDefFoundError:
>>>> javax/servlet/http/HttpServletResponse
>>>> java.lang.NoClassDefFoundError: javax/servlet/http/HttpServletResponse
>>>> at org.apache.spark.HttpServer.start(HttpServer.scala:54)
>>>> at org.apache.spark.broadcast.HttpBroadcast$.createServer(
>>>> HttpBroadcast.scala:156)
>>>> at org.apache.spark.broadcast.HttpBroadcast$.initialize(
>>>> HttpBroadcast.scala:127)
>>>> at org.apache.spark.broadcast.HttpBroadcastFactory.initialize(
>>>> HttpBroadcastFactory.scala:31)
>>>> at org.apache.spark.broadcast.BroadcastManager.initialize(
>>>> BroadcastManager.scala:48)
>>>> at org.apache.spark.broadcast.BroadcastManager.<init>(
>>>> BroadcastManager.scala:35)
>>>> at org.apache.spark.SparkEnv$.create(SparkEnv.scala:218)
>>>> at org.apache.spark.SparkContext.<init>(SparkContext.scala:202)
>>>> at main.scala.PartialDuplicateScanner$.main(
>>>> PartialDuplicateScanner.scala:29)
>>>> at main.scala.PartialDuplicateScanner.main(
>>>> PartialDuplicateScanner.scala)
>>>>
>>>> Here is my Testprojekt.sbt file:
>>>>
>>>> name := "Testprojekt"
>>>>
>>>> version := "1.0"
>>>>
>>>> scalaVersion := "2.10.4"
>>>>
>>>> libraryDependencies ++= {
>>>> Seq(
>>>> "org.apache.lucene" % "lucene-core" % "4.9.0",
>>>> "org.apache.lucene" % "lucene-analyzers-common" % "4.9.0",
>>>> "org.apache.lucene" % "lucene-queryparser" % "4.9.0",
>>>> ("org.apache.spark" %% "spark-core" % "1.0.2").
>>>> exclude("org.mortbay.jetty", "servlet-api").
>>>> exclude("commons-beanutils", "commons-beanutils-core").
>>>> exclude("commons-collections", "commons-collections").
>>>> exclude("commons-collections", "commons-collections").
>>>> exclude("com.esotericsoftware.minlog", "minlog").
>>>> exclude("org.eclipse.jetty.orbit", "javax.mail.glassfish").
>>>> exclude("org.eclipse.jetty.orbit", "javax.transaction").
>>>> exclude("org.eclipse.jetty.orbit", "javax.servlet")
>>>> )
>>>> }
>>>>
>>>> resolvers += "Akka Repository" at "http://repo.akka.io/releases/"
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>>>> For additional commands, e-mail: user-help@spark.apache.org
>>>>
>>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>>> For additional commands, e-mail: user-help@spark.apache.org
>>>
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>> For additional commands, e-mail: user-help@spark.apache.org
>>
>>
>
Re: Dependency Problem with Spark / ScalaTest / SBT
Posted by Dean Wampler <de...@gmail.com>.
Can you post your whole SBT build file(s)?
Dean Wampler, Ph.D.
Author: Programming Scala, 2nd Edition
<http://shop.oreilly.com/product/0636920033073.do> (O'Reilly)
Typesafe <http://typesafe.com>
@deanwampler <http://twitter.com/deanwampler>
http://polyglotprogramming.com
On Wed, Sep 10, 2014 at 6:48 AM, Thorsten Bergler <sp...@tbonline.de> wrote:
> Hi,
>
> I just called:
>
> > test
>
> or
>
> > run
>
> Thorsten
>
>
> Am 10.09.2014 um 13:38 schrieb Arthur.hk.chan@gmail.com:
>
> Hi,
>>
>> What is your SBT command and the parameters?
>>
>> Arthur
>>
>>
>> On 10 Sep, 2014, at 6:46 pm, Thorsten Bergler <sp...@tbonline.de> wrote:
>>
>> Hello,
>>>
>>> I am writing a Spark App which is already working so far.
>>> Now I started to build also some UnitTests, but I am running into some
>>> dependecy problems and I cannot find a solution right now. Perhaps someone
>>> could help me.
>>>
>>> I build my Spark Project with SBT and it seems to be configured well,
>>> because compiling, assembling and running the built jar with spark-submit
>>> are working well.
>>>
>>> Now I started with the UnitTests, which I located under /src/test/scala.
>>>
>>> When I call "test" in sbt, I get the following:
>>>
>>> 14/09/10 12:22:06 INFO storage.BlockManagerMaster: Registered
>>> BlockManager
>>> 14/09/10 12:22:06 INFO spark.HttpServer: Starting HTTP Server
>>> [trace] Stack trace suppressed: run last test:test for the full output.
>>> [error] Could not run test test.scala.SetSuite: java.lang.NoClassDefFoundError:
>>> javax/servlet/http/HttpServletResponse
>>> [info] Run completed in 626 milliseconds.
>>> [info] Total number of tests run: 0
>>> [info] Suites: completed 0, aborted 0
>>> [info] Tests: succeeded 0, failed 0, canceled 0, ignored 0, pending 0
>>> [info] All tests passed.
>>> [error] Error during tests:
>>> [error] test.scala.SetSuite
>>> [error] (test:test) sbt.TestsFailedException: Tests unsuccessful
>>> [error] Total time: 3 s, completed 10.09.2014 12:22:06
>>>
>>> last test:test gives me the following:
>>>
>>> last test:test
>>>>
>>> [debug] Running TaskDef(test.scala.SetSuite,
>>> org.scalatest.tools.Framework$$anon$1@6e5626c8, false, [SuiteSelector])
>>> java.lang.NoClassDefFoundError: javax/servlet/http/HttpServletResponse
>>> at org.apache.spark.HttpServer.start(HttpServer.scala:54)
>>> at org.apache.spark.broadcast.HttpBroadcast$.createServer(
>>> HttpBroadcast.scala:156)
>>> at org.apache.spark.broadcast.HttpBroadcast$.initialize(
>>> HttpBroadcast.scala:127)
>>> at org.apache.spark.broadcast.HttpBroadcastFactory.initialize(
>>> HttpBroadcastFactory.scala:31)
>>> at org.apache.spark.broadcast.BroadcastManager.initialize(
>>> BroadcastManager.scala:48)
>>> at org.apache.spark.broadcast.BroadcastManager.<init>(
>>> BroadcastManager.scala:35)
>>> at org.apache.spark.SparkEnv$.create(SparkEnv.scala:218)
>>> at org.apache.spark.SparkContext.<init>(SparkContext.scala:202)
>>> at test.scala.SetSuite.<init>(SparkTest.scala:16)
>>>
>>> I also noticed right now, that sbt run is also not working:
>>>
>>> 14/09/10 12:44:46 INFO spark.HttpServer: Starting HTTP Server
>>> [error] (run-main-2) java.lang.NoClassDefFoundError: javax/servlet/http/
>>> HttpServletResponse
>>> java.lang.NoClassDefFoundError: javax/servlet/http/HttpServletResponse
>>> at org.apache.spark.HttpServer.start(HttpServer.scala:54)
>>> at org.apache.spark.broadcast.HttpBroadcast$.createServer(
>>> HttpBroadcast.scala:156)
>>> at org.apache.spark.broadcast.HttpBroadcast$.initialize(
>>> HttpBroadcast.scala:127)
>>> at org.apache.spark.broadcast.HttpBroadcastFactory.initialize(
>>> HttpBroadcastFactory.scala:31)
>>> at org.apache.spark.broadcast.BroadcastManager.initialize(
>>> BroadcastManager.scala:48)
>>> at org.apache.spark.broadcast.BroadcastManager.<init>(
>>> BroadcastManager.scala:35)
>>> at org.apache.spark.SparkEnv$.create(SparkEnv.scala:218)
>>> at org.apache.spark.SparkContext.<init>(SparkContext.scala:202)
>>> at main.scala.PartialDuplicateScanner$.main(
>>> PartialDuplicateScanner.scala:29)
>>> at main.scala.PartialDuplicateScanner.main(
>>> PartialDuplicateScanner.scala)
>>>
>>> Here is my Testprojekt.sbt file:
>>>
>>> name := "Testprojekt"
>>>
>>> version := "1.0"
>>>
>>> scalaVersion := "2.10.4"
>>>
>>> libraryDependencies ++= {
>>> Seq(
>>> "org.apache.lucene" % "lucene-core" % "4.9.0",
>>> "org.apache.lucene" % "lucene-analyzers-common" % "4.9.0",
>>> "org.apache.lucene" % "lucene-queryparser" % "4.9.0",
>>> ("org.apache.spark" %% "spark-core" % "1.0.2").
>>> exclude("org.mortbay.jetty", "servlet-api").
>>> exclude("commons-beanutils", "commons-beanutils-core").
>>> exclude("commons-collections", "commons-collections").
>>> exclude("commons-collections", "commons-collections").
>>> exclude("com.esotericsoftware.minlog", "minlog").
>>> exclude("org.eclipse.jetty.orbit", "javax.mail.glassfish").
>>> exclude("org.eclipse.jetty.orbit", "javax.transaction").
>>> exclude("org.eclipse.jetty.orbit", "javax.servlet")
>>> )
>>> }
>>>
>>> resolvers += "Akka Repository" at "http://repo.akka.io/releases/"
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>>> For additional commands, e-mail: user-help@spark.apache.org
>>>
>>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>> For additional commands, e-mail: user-help@spark.apache.org
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>
Re: Dependency Problem with Spark / ScalaTest / SBT
Posted by Thorsten Bergler <sp...@tbonline.de>.
Hi,
I just called:
> test
or
> run
Thorsten
Am 10.09.2014 um 13:38 schrieb Arthur.hk.chan@gmail.com:
> Hi,
>
> What is your SBT command and the parameters?
>
> Arthur
>
>
> On 10 Sep, 2014, at 6:46 pm, Thorsten Bergler <sp...@tbonline.de> wrote:
>
>> Hello,
>>
>> I am writing a Spark App which is already working so far.
>> Now I started to build also some UnitTests, but I am running into some dependecy problems and I cannot find a solution right now. Perhaps someone could help me.
>>
>> I build my Spark Project with SBT and it seems to be configured well, because compiling, assembling and running the built jar with spark-submit are working well.
>>
>> Now I started with the UnitTests, which I located under /src/test/scala.
>>
>> When I call "test" in sbt, I get the following:
>>
>> 14/09/10 12:22:06 INFO storage.BlockManagerMaster: Registered BlockManager
>> 14/09/10 12:22:06 INFO spark.HttpServer: Starting HTTP Server
>> [trace] Stack trace suppressed: run last test:test for the full output.
>> [error] Could not run test test.scala.SetSuite: java.lang.NoClassDefFoundError: javax/servlet/http/HttpServletResponse
>> [info] Run completed in 626 milliseconds.
>> [info] Total number of tests run: 0
>> [info] Suites: completed 0, aborted 0
>> [info] Tests: succeeded 0, failed 0, canceled 0, ignored 0, pending 0
>> [info] All tests passed.
>> [error] Error during tests:
>> [error] test.scala.SetSuite
>> [error] (test:test) sbt.TestsFailedException: Tests unsuccessful
>> [error] Total time: 3 s, completed 10.09.2014 12:22:06
>>
>> last test:test gives me the following:
>>
>>> last test:test
>> [debug] Running TaskDef(test.scala.SetSuite, org.scalatest.tools.Framework$$anon$1@6e5626c8, false, [SuiteSelector])
>> java.lang.NoClassDefFoundError: javax/servlet/http/HttpServletResponse
>> at org.apache.spark.HttpServer.start(HttpServer.scala:54)
>> at org.apache.spark.broadcast.HttpBroadcast$.createServer(HttpBroadcast.scala:156)
>> at org.apache.spark.broadcast.HttpBroadcast$.initialize(HttpBroadcast.scala:127)
>> at org.apache.spark.broadcast.HttpBroadcastFactory.initialize(HttpBroadcastFactory.scala:31)
>> at org.apache.spark.broadcast.BroadcastManager.initialize(BroadcastManager.scala:48)
>> at org.apache.spark.broadcast.BroadcastManager.<init>(BroadcastManager.scala:35)
>> at org.apache.spark.SparkEnv$.create(SparkEnv.scala:218)
>> at org.apache.spark.SparkContext.<init>(SparkContext.scala:202)
>> at test.scala.SetSuite.<init>(SparkTest.scala:16)
>>
>> I also noticed right now, that sbt run is also not working:
>>
>> 14/09/10 12:44:46 INFO spark.HttpServer: Starting HTTP Server
>> [error] (run-main-2) java.lang.NoClassDefFoundError: javax/servlet/http/HttpServletResponse
>> java.lang.NoClassDefFoundError: javax/servlet/http/HttpServletResponse
>> at org.apache.spark.HttpServer.start(HttpServer.scala:54)
>> at org.apache.spark.broadcast.HttpBroadcast$.createServer(HttpBroadcast.scala:156)
>> at org.apache.spark.broadcast.HttpBroadcast$.initialize(HttpBroadcast.scala:127)
>> at org.apache.spark.broadcast.HttpBroadcastFactory.initialize(HttpBroadcastFactory.scala:31)
>> at org.apache.spark.broadcast.BroadcastManager.initialize(BroadcastManager.scala:48)
>> at org.apache.spark.broadcast.BroadcastManager.<init>(BroadcastManager.scala:35)
>> at org.apache.spark.SparkEnv$.create(SparkEnv.scala:218)
>> at org.apache.spark.SparkContext.<init>(SparkContext.scala:202)
>> at main.scala.PartialDuplicateScanner$.main(PartialDuplicateScanner.scala:29)
>> at main.scala.PartialDuplicateScanner.main(PartialDuplicateScanner.scala)
>>
>> Here is my Testprojekt.sbt file:
>>
>> name := "Testprojekt"
>>
>> version := "1.0"
>>
>> scalaVersion := "2.10.4"
>>
>> libraryDependencies ++= {
>> Seq(
>> "org.apache.lucene" % "lucene-core" % "4.9.0",
>> "org.apache.lucene" % "lucene-analyzers-common" % "4.9.0",
>> "org.apache.lucene" % "lucene-queryparser" % "4.9.0",
>> ("org.apache.spark" %% "spark-core" % "1.0.2").
>> exclude("org.mortbay.jetty", "servlet-api").
>> exclude("commons-beanutils", "commons-beanutils-core").
>> exclude("commons-collections", "commons-collections").
>> exclude("commons-collections", "commons-collections").
>> exclude("com.esotericsoftware.minlog", "minlog").
>> exclude("org.eclipse.jetty.orbit", "javax.mail.glassfish").
>> exclude("org.eclipse.jetty.orbit", "javax.transaction").
>> exclude("org.eclipse.jetty.orbit", "javax.servlet")
>> )
>> }
>>
>> resolvers += "Akka Repository" at "http://repo.akka.io/releases/"
>>
>>
>>
>>
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>> For additional commands, e-mail: user-help@spark.apache.org
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Dependency Problem with Spark / ScalaTest / SBT
Posted by "Arthur.hk.chan@gmail.com" <ar...@gmail.com>.
Hi,
What is your SBT command and the parameters?
Arthur
On 10 Sep, 2014, at 6:46 pm, Thorsten Bergler <sp...@tbonline.de> wrote:
> Hello,
>
> I am writing a Spark App which is already working so far.
> Now I started to build also some UnitTests, but I am running into some dependecy problems and I cannot find a solution right now. Perhaps someone could help me.
>
> I build my Spark Project with SBT and it seems to be configured well, because compiling, assembling and running the built jar with spark-submit are working well.
>
> Now I started with the UnitTests, which I located under /src/test/scala.
>
> When I call "test" in sbt, I get the following:
>
> 14/09/10 12:22:06 INFO storage.BlockManagerMaster: Registered BlockManager
> 14/09/10 12:22:06 INFO spark.HttpServer: Starting HTTP Server
> [trace] Stack trace suppressed: run last test:test for the full output.
> [error] Could not run test test.scala.SetSuite: java.lang.NoClassDefFoundError: javax/servlet/http/HttpServletResponse
> [info] Run completed in 626 milliseconds.
> [info] Total number of tests run: 0
> [info] Suites: completed 0, aborted 0
> [info] Tests: succeeded 0, failed 0, canceled 0, ignored 0, pending 0
> [info] All tests passed.
> [error] Error during tests:
> [error] test.scala.SetSuite
> [error] (test:test) sbt.TestsFailedException: Tests unsuccessful
> [error] Total time: 3 s, completed 10.09.2014 12:22:06
>
> last test:test gives me the following:
>
> > last test:test
> [debug] Running TaskDef(test.scala.SetSuite, org.scalatest.tools.Framework$$anon$1@6e5626c8, false, [SuiteSelector])
> java.lang.NoClassDefFoundError: javax/servlet/http/HttpServletResponse
> at org.apache.spark.HttpServer.start(HttpServer.scala:54)
> at org.apache.spark.broadcast.HttpBroadcast$.createServer(HttpBroadcast.scala:156)
> at org.apache.spark.broadcast.HttpBroadcast$.initialize(HttpBroadcast.scala:127)
> at org.apache.spark.broadcast.HttpBroadcastFactory.initialize(HttpBroadcastFactory.scala:31)
> at org.apache.spark.broadcast.BroadcastManager.initialize(BroadcastManager.scala:48)
> at org.apache.spark.broadcast.BroadcastManager.<init>(BroadcastManager.scala:35)
> at org.apache.spark.SparkEnv$.create(SparkEnv.scala:218)
> at org.apache.spark.SparkContext.<init>(SparkContext.scala:202)
> at test.scala.SetSuite.<init>(SparkTest.scala:16)
>
> I also noticed right now, that sbt run is also not working:
>
> 14/09/10 12:44:46 INFO spark.HttpServer: Starting HTTP Server
> [error] (run-main-2) java.lang.NoClassDefFoundError: javax/servlet/http/HttpServletResponse
> java.lang.NoClassDefFoundError: javax/servlet/http/HttpServletResponse
> at org.apache.spark.HttpServer.start(HttpServer.scala:54)
> at org.apache.spark.broadcast.HttpBroadcast$.createServer(HttpBroadcast.scala:156)
> at org.apache.spark.broadcast.HttpBroadcast$.initialize(HttpBroadcast.scala:127)
> at org.apache.spark.broadcast.HttpBroadcastFactory.initialize(HttpBroadcastFactory.scala:31)
> at org.apache.spark.broadcast.BroadcastManager.initialize(BroadcastManager.scala:48)
> at org.apache.spark.broadcast.BroadcastManager.<init>(BroadcastManager.scala:35)
> at org.apache.spark.SparkEnv$.create(SparkEnv.scala:218)
> at org.apache.spark.SparkContext.<init>(SparkContext.scala:202)
> at main.scala.PartialDuplicateScanner$.main(PartialDuplicateScanner.scala:29)
> at main.scala.PartialDuplicateScanner.main(PartialDuplicateScanner.scala)
>
> Here is my Testprojekt.sbt file:
>
> name := "Testprojekt"
>
> version := "1.0"
>
> scalaVersion := "2.10.4"
>
> libraryDependencies ++= {
> Seq(
> "org.apache.lucene" % "lucene-core" % "4.9.0",
> "org.apache.lucene" % "lucene-analyzers-common" % "4.9.0",
> "org.apache.lucene" % "lucene-queryparser" % "4.9.0",
> ("org.apache.spark" %% "spark-core" % "1.0.2").
> exclude("org.mortbay.jetty", "servlet-api").
> exclude("commons-beanutils", "commons-beanutils-core").
> exclude("commons-collections", "commons-collections").
> exclude("commons-collections", "commons-collections").
> exclude("com.esotericsoftware.minlog", "minlog").
> exclude("org.eclipse.jetty.orbit", "javax.mail.glassfish").
> exclude("org.eclipse.jetty.orbit", "javax.transaction").
> exclude("org.eclipse.jetty.orbit", "javax.servlet")
> )
> }
>
> resolvers += "Akka Repository" at "http://repo.akka.io/releases/"
>
>
>
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org