You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Koert Kuipers <ko...@tresata.com> on 2014/12/07 02:43:31 UTC
run JavaAPISuite with mavem
when i run "mvn test -pl core", i dont see JavaAPISuite being run. or if it
is, its being very very quiet about it. is this by design?
Re: run JavaAPISuite with mavem
Posted by Ted Yu <yu...@gmail.com>.
Running JavaAPISuite (master branch) on Linux, I got:
testGuavaOptional(org.apache.spark.JavaAPISuite) Time elapsed: 32.945 sec
<<< ERROR!
org.apache.spark.SparkException: Job aborted due to stage failure: Master
removed our application: FAILED
at org.apache.spark.scheduler.DAGScheduler.org
$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1214)
at
org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1203)
at
org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1202)
at
scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
at
org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1202)
at
org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:696)
at
org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:696)
at scala.Option.foreach(Option.scala:236)
at
org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:696)
at
org.apache.spark.scheduler.DAGSchedulerEventProcessActor$$anonfun$receive$2.applyOrElse(DAGScheduler.scala:1420)
at akka.actor.Actor$class.aroundReceive(Actor.scala:465)
at
org.apache.spark.scheduler.DAGSchedulerEventProcessActor.aroundReceive(DAGScheduler.scala:1375)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)
at akka.actor.ActorCell.invoke(ActorCell.scala:487)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:238)
at akka.dispatch.Mailbox.run(Mailbox.scala:220)
at
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:393)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at
scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
FYI
On Sun, Dec 7, 2014 at 3:25 PM, Sean Owen <so...@cloudera.com> wrote:
> I think it's a known issue:
>
> https://issues.apache.org/jira/browse/SPARK-4159
> https://issues.apache.org/jira/browse/SPARK-661
>
> I got bit by this too recently and meant to look into it.
>
> On Sun, Dec 7, 2014 at 4:50 PM, Koert Kuipers <ko...@tresata.com> wrote:
> > so as part of the official build the java api does not get tested then?
> > i am sure there is a good reason for it, but thats surprising to me.
> >
> > On Sun, Dec 7, 2014 at 12:19 PM, Ted Yu <yu...@gmail.com> wrote:
> >>
> >> Looking at the pom.xml, I think I found the reason - scalatest is used.
> >> With the following diff:
> >>
> >> diff --git a/pom.xml b/pom.xml
> >> index b7df53d..b0da893 100644
> >> --- a/pom.xml
> >> +++ b/pom.xml
> >> @@ -947,7 +947,7 @@
> >> <version>2.17</version>
> >> <configuration>
> >> <!-- Uses scalatest instead -->
> >> - <skipTests>true</skipTests>
> >> + <skipTests>false</skipTests>
> >> </configuration>
> >> </plugin>
> >> <plugin>
> >>
> >> I was able to run JavaAPISuite using:
> >>
> >> mvn test -pl core -Dtest=JavaAPISuite
> >>
> >> But it takes a long time ...
> >>
> >> Cheers
> >>
> >> On Sun, Dec 7, 2014 at 8:56 AM, Koert Kuipers <ko...@tresata.com>
> wrote:
> >>>
> >>> hey guys,
> >>> i was able to run the test just fine with:
> >>> $ sbt
> >>> > project core
> >>> > testOnly org.apache.spark.JavaAPISuite
> >>>
> >>> however i found it strange that it didnt run when i do "mvn test -pl
> >>> core", or at least didnt seem like it ran to me. this would mean that
> when
> >>> someone tests/publishes with maven the japa api does not get tested at
> all.
> >>>
> >>> On Sun, Dec 7, 2014 at 1:52 AM, Michael Armbrust <
> michael@databricks.com>
> >>> wrote:
> >>>>
> >>>> Not sure about maven, but you can run that test with sbt:
> >>>>
> >>>> sbt/sbt "sql/test-only org.apache.spark.sql.api.java.JavaAPISuite"
> >>>>
> >>>> On Sat, Dec 6, 2014 at 9:59 PM, Ted Yu <yu...@gmail.com> wrote:
> >>>>>
> >>>>> I tried to run tests for core but there were failures. e.g. :
> >>>>>
> >>>>> ^[[32mExternalAppendOnlyMapSuite:^[[0m
> >>>>> ^[[32m- simple insert^[[0m
> >>>>> ^[[32m- insert with collision^[[0m
> >>>>> ^[[32m- ordering^[[0m
> >>>>> ^[[32m- null keys and values^[[0m
> >>>>> ^[[32m- simple aggregator^[[0m
> >>>>> ^[[32m- simple cogroup^[[0m
> >>>>> Spark assembly has been built with Hive, including Datanucleus jars
> on
> >>>>> classpath
> >>>>> ^[[31m- spilling *** FAILED ***^[[0m
> >>>>> ^[[31m org.apache.spark.SparkException: Job aborted due to stage
> >>>>> failure: Task 0 in stage 0.0 failed 4 times, most recent failure:
> Lost task
> >>>>> 0.3 in stage 0.0 (TID 6, localhost):
> java.lang.ClassNotFoundException:
> >>>>> org.apache.spark.rdd.RDD$$anonfun$map$1^[[0m
> >>>>> ^[[31m at
> java.net.URLClassLoader$1.run(URLClassLoader.java:366)^[[0m
> >>>>> ^[[31m at
> java.net.URLClassLoader$1.run(URLClassLoader.java:355)^[[0m
> >>>>> ^[[31m at java.security.AccessController.doPrivileged(Native
> >>>>> Method)^[[0m
> >>>>> ^[[31m at
> >>>>> java.net.URLClassLoader.findClass(URLClassLoader.java:354)^[[0m
> >>>>> ^[[31m at java.lang.ClassLoader.loadClass(ClassLoader.java:424)^[[0m
> >>>>> ^[[31m at java.lang.ClassLoader.loadClass(ClassLoader.java:357)^[[0m
> >>>>> ^[[31m at java.lang.Class.forName0(Native Method)^[[0m
> >>>>> ^[[31m at java.lang.Class.forName(Class.java:270)^[[0m
> >>>>>
> >>>>> BTW I didn't find JavaAPISuite in test output either.
> >>>>>
> >>>>> Cheers
> >>>>>
> >>>>> On Sat, Dec 6, 2014 at 9:12 PM, Koert Kuipers <ko...@tresata.com>
> >>>>> wrote:
> >>>>>>
> >>>>>> Ted,
> >>>>>> i mean
> >>>>>> core/src/test/java/org/apache/spark/JavaAPISuite.java
> >>>>>>
> >>>>>> On Sat, Dec 6, 2014 at 9:27 PM, Ted Yu <yu...@gmail.com> wrote:
> >>>>>>>
> >>>>>>> Pardon me, the test is here:
> >>>>>>>
> >>>>>>>
> >>>>>>>
> sql/core/src/test/java/org/apache/spark/sql/api/java/JavaAPISuite.java
> >>>>>>>
> >>>>>>> You can run 'mvn test' under sql/core
> >>>>>>>
> >>>>>>> Cheers
> >>>>>>>
> >>>>>>> On Sat, Dec 6, 2014 at 5:55 PM, Ted Yu <yu...@gmail.com>
> wrote:
> >>>>>>>>
> >>>>>>>> In master branch, I only found JavaAPISuite in comment:
> >>>>>>>>
> >>>>>>>> spark tyu$ find . -name '*.scala' -exec grep JavaAPISuite {} \;
> >>>>>>>> -print
> >>>>>>>> * For usage example, see test case
> JavaAPISuite.testJavaJdbcRDD.
> >>>>>>>> * converted into a `Object` array. For usage example, see test
> >>>>>>>> case JavaAPISuite.testJavaJdbcRDD.
> >>>>>>>> ./core/src/main/scala/org/apache/spark/rdd/JdbcRDD.scala
> >>>>>>>>
> >>>>>>>> FYI
> >>>>>>>>
> >>>>>>>> On Sat, Dec 6, 2014 at 5:43 PM, Koert Kuipers <ko...@tresata.com>
> >>>>>>>> wrote:
> >>>>>>>>>
> >>>>>>>>> when i run "mvn test -pl core", i dont see JavaAPISuite being
> run.
> >>>>>>>>> or if it is, its being very very quiet about it. is this by
> design?
> >>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>
> >
>
Re: run JavaAPISuite with mavem
Posted by Koert Kuipers <ko...@tresata.com>.
thanks. that makes sense. i searched the mailing list but couldnt find any
mention of it. i should have searched jira instead...
On Sun, Dec 7, 2014 at 6:25 PM, Sean Owen <so...@cloudera.com> wrote:
> I think it's a known issue:
>
> https://issues.apache.org/jira/browse/SPARK-4159
> https://issues.apache.org/jira/browse/SPARK-661
>
> I got bit by this too recently and meant to look into it.
>
> On Sun, Dec 7, 2014 at 4:50 PM, Koert Kuipers <ko...@tresata.com> wrote:
> > so as part of the official build the java api does not get tested then?
> > i am sure there is a good reason for it, but thats surprising to me.
> >
> > On Sun, Dec 7, 2014 at 12:19 PM, Ted Yu <yu...@gmail.com> wrote:
> >>
> >> Looking at the pom.xml, I think I found the reason - scalatest is used.
> >> With the following diff:
> >>
> >> diff --git a/pom.xml b/pom.xml
> >> index b7df53d..b0da893 100644
> >> --- a/pom.xml
> >> +++ b/pom.xml
> >> @@ -947,7 +947,7 @@
> >> <version>2.17</version>
> >> <configuration>
> >> <!-- Uses scalatest instead -->
> >> - <skipTests>true</skipTests>
> >> + <skipTests>false</skipTests>
> >> </configuration>
> >> </plugin>
> >> <plugin>
> >>
> >> I was able to run JavaAPISuite using:
> >>
> >> mvn test -pl core -Dtest=JavaAPISuite
> >>
> >> But it takes a long time ...
> >>
> >> Cheers
> >>
> >> On Sun, Dec 7, 2014 at 8:56 AM, Koert Kuipers <ko...@tresata.com>
> wrote:
> >>>
> >>> hey guys,
> >>> i was able to run the test just fine with:
> >>> $ sbt
> >>> > project core
> >>> > testOnly org.apache.spark.JavaAPISuite
> >>>
> >>> however i found it strange that it didnt run when i do "mvn test -pl
> >>> core", or at least didnt seem like it ran to me. this would mean that
> when
> >>> someone tests/publishes with maven the japa api does not get tested at
> all.
> >>>
> >>> On Sun, Dec 7, 2014 at 1:52 AM, Michael Armbrust <
> michael@databricks.com>
> >>> wrote:
> >>>>
> >>>> Not sure about maven, but you can run that test with sbt:
> >>>>
> >>>> sbt/sbt "sql/test-only org.apache.spark.sql.api.java.JavaAPISuite"
> >>>>
> >>>> On Sat, Dec 6, 2014 at 9:59 PM, Ted Yu <yu...@gmail.com> wrote:
> >>>>>
> >>>>> I tried to run tests for core but there were failures. e.g. :
> >>>>>
> >>>>> ^[[32mExternalAppendOnlyMapSuite:^[[0m
> >>>>> ^[[32m- simple insert^[[0m
> >>>>> ^[[32m- insert with collision^[[0m
> >>>>> ^[[32m- ordering^[[0m
> >>>>> ^[[32m- null keys and values^[[0m
> >>>>> ^[[32m- simple aggregator^[[0m
> >>>>> ^[[32m- simple cogroup^[[0m
> >>>>> Spark assembly has been built with Hive, including Datanucleus jars
> on
> >>>>> classpath
> >>>>> ^[[31m- spilling *** FAILED ***^[[0m
> >>>>> ^[[31m org.apache.spark.SparkException: Job aborted due to stage
> >>>>> failure: Task 0 in stage 0.0 failed 4 times, most recent failure:
> Lost task
> >>>>> 0.3 in stage 0.0 (TID 6, localhost):
> java.lang.ClassNotFoundException:
> >>>>> org.apache.spark.rdd.RDD$$anonfun$map$1^[[0m
> >>>>> ^[[31m at
> java.net.URLClassLoader$1.run(URLClassLoader.java:366)^[[0m
> >>>>> ^[[31m at
> java.net.URLClassLoader$1.run(URLClassLoader.java:355)^[[0m
> >>>>> ^[[31m at java.security.AccessController.doPrivileged(Native
> >>>>> Method)^[[0m
> >>>>> ^[[31m at
> >>>>> java.net.URLClassLoader.findClass(URLClassLoader.java:354)^[[0m
> >>>>> ^[[31m at java.lang.ClassLoader.loadClass(ClassLoader.java:424)^[[0m
> >>>>> ^[[31m at java.lang.ClassLoader.loadClass(ClassLoader.java:357)^[[0m
> >>>>> ^[[31m at java.lang.Class.forName0(Native Method)^[[0m
> >>>>> ^[[31m at java.lang.Class.forName(Class.java:270)^[[0m
> >>>>>
> >>>>> BTW I didn't find JavaAPISuite in test output either.
> >>>>>
> >>>>> Cheers
> >>>>>
> >>>>> On Sat, Dec 6, 2014 at 9:12 PM, Koert Kuipers <ko...@tresata.com>
> >>>>> wrote:
> >>>>>>
> >>>>>> Ted,
> >>>>>> i mean
> >>>>>> core/src/test/java/org/apache/spark/JavaAPISuite.java
> >>>>>>
> >>>>>> On Sat, Dec 6, 2014 at 9:27 PM, Ted Yu <yu...@gmail.com> wrote:
> >>>>>>>
> >>>>>>> Pardon me, the test is here:
> >>>>>>>
> >>>>>>>
> >>>>>>>
> sql/core/src/test/java/org/apache/spark/sql/api/java/JavaAPISuite.java
> >>>>>>>
> >>>>>>> You can run 'mvn test' under sql/core
> >>>>>>>
> >>>>>>> Cheers
> >>>>>>>
> >>>>>>> On Sat, Dec 6, 2014 at 5:55 PM, Ted Yu <yu...@gmail.com>
> wrote:
> >>>>>>>>
> >>>>>>>> In master branch, I only found JavaAPISuite in comment:
> >>>>>>>>
> >>>>>>>> spark tyu$ find . -name '*.scala' -exec grep JavaAPISuite {} \;
> >>>>>>>> -print
> >>>>>>>> * For usage example, see test case
> JavaAPISuite.testJavaJdbcRDD.
> >>>>>>>> * converted into a `Object` array. For usage example, see test
> >>>>>>>> case JavaAPISuite.testJavaJdbcRDD.
> >>>>>>>> ./core/src/main/scala/org/apache/spark/rdd/JdbcRDD.scala
> >>>>>>>>
> >>>>>>>> FYI
> >>>>>>>>
> >>>>>>>> On Sat, Dec 6, 2014 at 5:43 PM, Koert Kuipers <ko...@tresata.com>
> >>>>>>>> wrote:
> >>>>>>>>>
> >>>>>>>>> when i run "mvn test -pl core", i dont see JavaAPISuite being
> run.
> >>>>>>>>> or if it is, its being very very quiet about it. is this by
> design?
> >>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>
> >
>
Re: run JavaAPISuite with mavem
Posted by Sean Owen <so...@cloudera.com>.
I think it's a known issue:
https://issues.apache.org/jira/browse/SPARK-4159
https://issues.apache.org/jira/browse/SPARK-661
I got bit by this too recently and meant to look into it.
On Sun, Dec 7, 2014 at 4:50 PM, Koert Kuipers <ko...@tresata.com> wrote:
> so as part of the official build the java api does not get tested then?
> i am sure there is a good reason for it, but thats surprising to me.
>
> On Sun, Dec 7, 2014 at 12:19 PM, Ted Yu <yu...@gmail.com> wrote:
>>
>> Looking at the pom.xml, I think I found the reason - scalatest is used.
>> With the following diff:
>>
>> diff --git a/pom.xml b/pom.xml
>> index b7df53d..b0da893 100644
>> --- a/pom.xml
>> +++ b/pom.xml
>> @@ -947,7 +947,7 @@
>> <version>2.17</version>
>> <configuration>
>> <!-- Uses scalatest instead -->
>> - <skipTests>true</skipTests>
>> + <skipTests>false</skipTests>
>> </configuration>
>> </plugin>
>> <plugin>
>>
>> I was able to run JavaAPISuite using:
>>
>> mvn test -pl core -Dtest=JavaAPISuite
>>
>> But it takes a long time ...
>>
>> Cheers
>>
>> On Sun, Dec 7, 2014 at 8:56 AM, Koert Kuipers <ko...@tresata.com> wrote:
>>>
>>> hey guys,
>>> i was able to run the test just fine with:
>>> $ sbt
>>> > project core
>>> > testOnly org.apache.spark.JavaAPISuite
>>>
>>> however i found it strange that it didnt run when i do "mvn test -pl
>>> core", or at least didnt seem like it ran to me. this would mean that when
>>> someone tests/publishes with maven the japa api does not get tested at all.
>>>
>>> On Sun, Dec 7, 2014 at 1:52 AM, Michael Armbrust <mi...@databricks.com>
>>> wrote:
>>>>
>>>> Not sure about maven, but you can run that test with sbt:
>>>>
>>>> sbt/sbt "sql/test-only org.apache.spark.sql.api.java.JavaAPISuite"
>>>>
>>>> On Sat, Dec 6, 2014 at 9:59 PM, Ted Yu <yu...@gmail.com> wrote:
>>>>>
>>>>> I tried to run tests for core but there were failures. e.g. :
>>>>>
>>>>> ^[[32mExternalAppendOnlyMapSuite:^[[0m
>>>>> ^[[32m- simple insert^[[0m
>>>>> ^[[32m- insert with collision^[[0m
>>>>> ^[[32m- ordering^[[0m
>>>>> ^[[32m- null keys and values^[[0m
>>>>> ^[[32m- simple aggregator^[[0m
>>>>> ^[[32m- simple cogroup^[[0m
>>>>> Spark assembly has been built with Hive, including Datanucleus jars on
>>>>> classpath
>>>>> ^[[31m- spilling *** FAILED ***^[[0m
>>>>> ^[[31m org.apache.spark.SparkException: Job aborted due to stage
>>>>> failure: Task 0 in stage 0.0 failed 4 times, most recent failure: Lost task
>>>>> 0.3 in stage 0.0 (TID 6, localhost): java.lang.ClassNotFoundException:
>>>>> org.apache.spark.rdd.RDD$$anonfun$map$1^[[0m
>>>>> ^[[31m at java.net.URLClassLoader$1.run(URLClassLoader.java:366)^[[0m
>>>>> ^[[31m at java.net.URLClassLoader$1.run(URLClassLoader.java:355)^[[0m
>>>>> ^[[31m at java.security.AccessController.doPrivileged(Native
>>>>> Method)^[[0m
>>>>> ^[[31m at
>>>>> java.net.URLClassLoader.findClass(URLClassLoader.java:354)^[[0m
>>>>> ^[[31m at java.lang.ClassLoader.loadClass(ClassLoader.java:424)^[[0m
>>>>> ^[[31m at java.lang.ClassLoader.loadClass(ClassLoader.java:357)^[[0m
>>>>> ^[[31m at java.lang.Class.forName0(Native Method)^[[0m
>>>>> ^[[31m at java.lang.Class.forName(Class.java:270)^[[0m
>>>>>
>>>>> BTW I didn't find JavaAPISuite in test output either.
>>>>>
>>>>> Cheers
>>>>>
>>>>> On Sat, Dec 6, 2014 at 9:12 PM, Koert Kuipers <ko...@tresata.com>
>>>>> wrote:
>>>>>>
>>>>>> Ted,
>>>>>> i mean
>>>>>> core/src/test/java/org/apache/spark/JavaAPISuite.java
>>>>>>
>>>>>> On Sat, Dec 6, 2014 at 9:27 PM, Ted Yu <yu...@gmail.com> wrote:
>>>>>>>
>>>>>>> Pardon me, the test is here:
>>>>>>>
>>>>>>>
>>>>>>> sql/core/src/test/java/org/apache/spark/sql/api/java/JavaAPISuite.java
>>>>>>>
>>>>>>> You can run 'mvn test' under sql/core
>>>>>>>
>>>>>>> Cheers
>>>>>>>
>>>>>>> On Sat, Dec 6, 2014 at 5:55 PM, Ted Yu <yu...@gmail.com> wrote:
>>>>>>>>
>>>>>>>> In master branch, I only found JavaAPISuite in comment:
>>>>>>>>
>>>>>>>> spark tyu$ find . -name '*.scala' -exec grep JavaAPISuite {} \;
>>>>>>>> -print
>>>>>>>> * For usage example, see test case JavaAPISuite.testJavaJdbcRDD.
>>>>>>>> * converted into a `Object` array. For usage example, see test
>>>>>>>> case JavaAPISuite.testJavaJdbcRDD.
>>>>>>>> ./core/src/main/scala/org/apache/spark/rdd/JdbcRDD.scala
>>>>>>>>
>>>>>>>> FYI
>>>>>>>>
>>>>>>>> On Sat, Dec 6, 2014 at 5:43 PM, Koert Kuipers <ko...@tresata.com>
>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>> when i run "mvn test -pl core", i dont see JavaAPISuite being run.
>>>>>>>>> or if it is, its being very very quiet about it. is this by design?
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: run JavaAPISuite with mavem
Posted by Koert Kuipers <ko...@tresata.com>.
so as part of the official build the java api does not get tested then?
i am sure there is a good reason for it, but thats surprising to me.
On Sun, Dec 7, 2014 at 12:19 PM, Ted Yu <yu...@gmail.com> wrote:
> Looking at the pom.xml, I think I found the reason - scalatest is used.
> With the following diff:
>
> diff --git a/pom.xml b/pom.xml
> index b7df53d..b0da893 100644
> --- a/pom.xml
> +++ b/pom.xml
> @@ -947,7 +947,7 @@
> <version>2.17</version>
> <configuration>
> <!-- Uses scalatest instead -->
> - <skipTests>true</skipTests>
> + <skipTests>false</skipTests>
> </configuration>
> </plugin>
> <plugin>
>
> I was able to run JavaAPISuite using:
>
> mvn test -pl core -Dtest=JavaAPISuite
>
> But it takes a long time ...
>
> Cheers
>
> On Sun, Dec 7, 2014 at 8:56 AM, Koert Kuipers <ko...@tresata.com> wrote:
>
>> hey guys,
>> i was able to run the test just fine with:
>> $ sbt
>> > project core
>> > testOnly org.apache.spark.JavaAPISuite
>>
>> however i found it strange that it didnt run when i do "mvn test -pl
>> core", or at least didnt seem like it ran to me. this would mean that when
>> someone tests/publishes with maven the japa api does not get tested at all.
>>
>> On Sun, Dec 7, 2014 at 1:52 AM, Michael Armbrust <mi...@databricks.com>
>> wrote:
>>
>>> Not sure about maven, but you can run that test with sbt:
>>>
>>> sbt/sbt "sql/test-only org.apache.spark.sql.api.java.JavaAPISuite"
>>>
>>> On Sat, Dec 6, 2014 at 9:59 PM, Ted Yu <yu...@gmail.com> wrote:
>>>
>>>> I tried to run tests for core but there were failures. e.g. :
>>>>
>>>> ^[[32mExternalAppendOnlyMapSuite:^[[0m
>>>> ^[[32m- simple insert^[[0m
>>>> ^[[32m- insert with collision^[[0m
>>>> ^[[32m- ordering^[[0m
>>>> ^[[32m- null keys and values^[[0m
>>>> ^[[32m- simple aggregator^[[0m
>>>> ^[[32m- simple cogroup^[[0m
>>>> Spark assembly has been built with Hive, including Datanucleus jars on
>>>> classpath
>>>> ^[[31m- spilling *** FAILED ***^[[0m
>>>> ^[[31m org.apache.spark.SparkException: Job aborted due to stage
>>>> failure: Task 0 in stage 0.0 failed 4 times, most recent failure: Lost task
>>>> 0.3 in stage 0.0 (TID 6, localhost): java.lang.ClassNotFoundException:
>>>> org.apache.spark.rdd.RDD$$anonfun$map$1^[[0m
>>>> ^[[31m at java.net.URLClassLoader$1.run(URLClassLoader.java:366)^[[0m
>>>> ^[[31m at java.net.URLClassLoader$1.run(URLClassLoader.java:355)^[[0m
>>>> ^[[31m at java.security.AccessController.doPrivileged(Native
>>>> Method)^[[0m
>>>> ^[[31m at
>>>> java.net.URLClassLoader.findClass(URLClassLoader.java:354)^[[0m
>>>> ^[[31m at java.lang.ClassLoader.loadClass(ClassLoader.java:424)^[[0m
>>>> ^[[31m at java.lang.ClassLoader.loadClass(ClassLoader.java:357)^[[0m
>>>> ^[[31m at java.lang.Class.forName0(Native Method)^[[0m
>>>> ^[[31m at java.lang.Class.forName(Class.java:270)^[[0m
>>>>
>>>> BTW I didn't find JavaAPISuite in test output either.
>>>>
>>>> Cheers
>>>>
>>>> On Sat, Dec 6, 2014 at 9:12 PM, Koert Kuipers <ko...@tresata.com>
>>>> wrote:
>>>>
>>>>> Ted,
>>>>> i mean
>>>>> core/src/test/java/org/apache/spark/JavaAPISuite.java
>>>>>
>>>>> On Sat, Dec 6, 2014 at 9:27 PM, Ted Yu <yu...@gmail.com> wrote:
>>>>>
>>>>>> Pardon me, the test is here:
>>>>>>
>>>>>> sql/core/src/test/java/org/apache/spark/sql/api/java/JavaAPISuite.java
>>>>>>
>>>>>> You can run 'mvn test' under sql/core
>>>>>>
>>>>>> Cheers
>>>>>>
>>>>>> On Sat, Dec 6, 2014 at 5:55 PM, Ted Yu <yu...@gmail.com> wrote:
>>>>>>
>>>>>>> In master branch, I only found JavaAPISuite in comment:
>>>>>>>
>>>>>>> spark tyu$ find . -name '*.scala' -exec grep JavaAPISuite {} \;
>>>>>>> -print
>>>>>>> * For usage example, see test case JavaAPISuite.testJavaJdbcRDD.
>>>>>>> * converted into a `Object` array. For usage example, see test
>>>>>>> case JavaAPISuite.testJavaJdbcRDD.
>>>>>>> ./core/src/main/scala/org/apache/spark/rdd/JdbcRDD.scala
>>>>>>>
>>>>>>> FYI
>>>>>>>
>>>>>>> On Sat, Dec 6, 2014 at 5:43 PM, Koert Kuipers <ko...@tresata.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> when i run "mvn test -pl core", i dont see JavaAPISuite being run.
>>>>>>>> or if it is, its being very very quiet about it. is this by design?
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: run JavaAPISuite with mavem
Posted by Ted Yu <yu...@gmail.com>.
Looking at the pom.xml, I think I found the reason - scalatest is used.
With the following diff:
diff --git a/pom.xml b/pom.xml
index b7df53d..b0da893 100644
--- a/pom.xml
+++ b/pom.xml
@@ -947,7 +947,7 @@
<version>2.17</version>
<configuration>
<!-- Uses scalatest instead -->
- <skipTests>true</skipTests>
+ <skipTests>false</skipTests>
</configuration>
</plugin>
<plugin>
I was able to run JavaAPISuite using:
mvn test -pl core -Dtest=JavaAPISuite
But it takes a long time ...
Cheers
On Sun, Dec 7, 2014 at 8:56 AM, Koert Kuipers <ko...@tresata.com> wrote:
> hey guys,
> i was able to run the test just fine with:
> $ sbt
> > project core
> > testOnly org.apache.spark.JavaAPISuite
>
> however i found it strange that it didnt run when i do "mvn test -pl
> core", or at least didnt seem like it ran to me. this would mean that when
> someone tests/publishes with maven the japa api does not get tested at all.
>
> On Sun, Dec 7, 2014 at 1:52 AM, Michael Armbrust <mi...@databricks.com>
> wrote:
>
>> Not sure about maven, but you can run that test with sbt:
>>
>> sbt/sbt "sql/test-only org.apache.spark.sql.api.java.JavaAPISuite"
>>
>> On Sat, Dec 6, 2014 at 9:59 PM, Ted Yu <yu...@gmail.com> wrote:
>>
>>> I tried to run tests for core but there were failures. e.g. :
>>>
>>> ^[[32mExternalAppendOnlyMapSuite:^[[0m
>>> ^[[32m- simple insert^[[0m
>>> ^[[32m- insert with collision^[[0m
>>> ^[[32m- ordering^[[0m
>>> ^[[32m- null keys and values^[[0m
>>> ^[[32m- simple aggregator^[[0m
>>> ^[[32m- simple cogroup^[[0m
>>> Spark assembly has been built with Hive, including Datanucleus jars on
>>> classpath
>>> ^[[31m- spilling *** FAILED ***^[[0m
>>> ^[[31m org.apache.spark.SparkException: Job aborted due to stage
>>> failure: Task 0 in stage 0.0 failed 4 times, most recent failure: Lost task
>>> 0.3 in stage 0.0 (TID 6, localhost): java.lang.ClassNotFoundException:
>>> org.apache.spark.rdd.RDD$$anonfun$map$1^[[0m
>>> ^[[31m at java.net.URLClassLoader$1.run(URLClassLoader.java:366)^[[0m
>>> ^[[31m at java.net.URLClassLoader$1.run(URLClassLoader.java:355)^[[0m
>>> ^[[31m at java.security.AccessController.doPrivileged(Native
>>> Method)^[[0m
>>> ^[[31m at
>>> java.net.URLClassLoader.findClass(URLClassLoader.java:354)^[[0m
>>> ^[[31m at java.lang.ClassLoader.loadClass(ClassLoader.java:424)^[[0m
>>> ^[[31m at java.lang.ClassLoader.loadClass(ClassLoader.java:357)^[[0m
>>> ^[[31m at java.lang.Class.forName0(Native Method)^[[0m
>>> ^[[31m at java.lang.Class.forName(Class.java:270)^[[0m
>>>
>>> BTW I didn't find JavaAPISuite in test output either.
>>>
>>> Cheers
>>>
>>> On Sat, Dec 6, 2014 at 9:12 PM, Koert Kuipers <ko...@tresata.com> wrote:
>>>
>>>> Ted,
>>>> i mean
>>>> core/src/test/java/org/apache/spark/JavaAPISuite.java
>>>>
>>>> On Sat, Dec 6, 2014 at 9:27 PM, Ted Yu <yu...@gmail.com> wrote:
>>>>
>>>>> Pardon me, the test is here:
>>>>>
>>>>> sql/core/src/test/java/org/apache/spark/sql/api/java/JavaAPISuite.java
>>>>>
>>>>> You can run 'mvn test' under sql/core
>>>>>
>>>>> Cheers
>>>>>
>>>>> On Sat, Dec 6, 2014 at 5:55 PM, Ted Yu <yu...@gmail.com> wrote:
>>>>>
>>>>>> In master branch, I only found JavaAPISuite in comment:
>>>>>>
>>>>>> spark tyu$ find . -name '*.scala' -exec grep JavaAPISuite {} \; -print
>>>>>> * For usage example, see test case JavaAPISuite.testJavaJdbcRDD.
>>>>>> * converted into a `Object` array. For usage example, see test
>>>>>> case JavaAPISuite.testJavaJdbcRDD.
>>>>>> ./core/src/main/scala/org/apache/spark/rdd/JdbcRDD.scala
>>>>>>
>>>>>> FYI
>>>>>>
>>>>>> On Sat, Dec 6, 2014 at 5:43 PM, Koert Kuipers <ko...@tresata.com>
>>>>>> wrote:
>>>>>>
>>>>>>> when i run "mvn test -pl core", i dont see JavaAPISuite being run.
>>>>>>> or if it is, its being very very quiet about it. is this by design?
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: run JavaAPISuite with mavem
Posted by Koert Kuipers <ko...@tresata.com>.
hey guys,
i was able to run the test just fine with:
$ sbt
> project core
> testOnly org.apache.spark.JavaAPISuite
however i found it strange that it didnt run when i do "mvn test -pl core",
or at least didnt seem like it ran to me. this would mean that when someone
tests/publishes with maven the japa api does not get tested at all.
On Sun, Dec 7, 2014 at 1:52 AM, Michael Armbrust <mi...@databricks.com>
wrote:
> Not sure about maven, but you can run that test with sbt:
>
> sbt/sbt "sql/test-only org.apache.spark.sql.api.java.JavaAPISuite"
>
> On Sat, Dec 6, 2014 at 9:59 PM, Ted Yu <yu...@gmail.com> wrote:
>
>> I tried to run tests for core but there were failures. e.g. :
>>
>> ^[[32mExternalAppendOnlyMapSuite:^[[0m
>> ^[[32m- simple insert^[[0m
>> ^[[32m- insert with collision^[[0m
>> ^[[32m- ordering^[[0m
>> ^[[32m- null keys and values^[[0m
>> ^[[32m- simple aggregator^[[0m
>> ^[[32m- simple cogroup^[[0m
>> Spark assembly has been built with Hive, including Datanucleus jars on
>> classpath
>> ^[[31m- spilling *** FAILED ***^[[0m
>> ^[[31m org.apache.spark.SparkException: Job aborted due to stage
>> failure: Task 0 in stage 0.0 failed 4 times, most recent failure: Lost task
>> 0.3 in stage 0.0 (TID 6, localhost): java.lang.ClassNotFoundException:
>> org.apache.spark.rdd.RDD$$anonfun$map$1^[[0m
>> ^[[31m at java.net.URLClassLoader$1.run(URLClassLoader.java:366)^[[0m
>> ^[[31m at java.net.URLClassLoader$1.run(URLClassLoader.java:355)^[[0m
>> ^[[31m at java.security.AccessController.doPrivileged(Native Method)^[[0m
>> ^[[31m at java.net.URLClassLoader.findClass(URLClassLoader.java:354)^[[0m
>> ^[[31m at java.lang.ClassLoader.loadClass(ClassLoader.java:424)^[[0m
>> ^[[31m at java.lang.ClassLoader.loadClass(ClassLoader.java:357)^[[0m
>> ^[[31m at java.lang.Class.forName0(Native Method)^[[0m
>> ^[[31m at java.lang.Class.forName(Class.java:270)^[[0m
>>
>> BTW I didn't find JavaAPISuite in test output either.
>>
>> Cheers
>>
>> On Sat, Dec 6, 2014 at 9:12 PM, Koert Kuipers <ko...@tresata.com> wrote:
>>
>>> Ted,
>>> i mean
>>> core/src/test/java/org/apache/spark/JavaAPISuite.java
>>>
>>> On Sat, Dec 6, 2014 at 9:27 PM, Ted Yu <yu...@gmail.com> wrote:
>>>
>>>> Pardon me, the test is here:
>>>>
>>>> sql/core/src/test/java/org/apache/spark/sql/api/java/JavaAPISuite.java
>>>>
>>>> You can run 'mvn test' under sql/core
>>>>
>>>> Cheers
>>>>
>>>> On Sat, Dec 6, 2014 at 5:55 PM, Ted Yu <yu...@gmail.com> wrote:
>>>>
>>>>> In master branch, I only found JavaAPISuite in comment:
>>>>>
>>>>> spark tyu$ find . -name '*.scala' -exec grep JavaAPISuite {} \; -print
>>>>> * For usage example, see test case JavaAPISuite.testJavaJdbcRDD.
>>>>> * converted into a `Object` array. For usage example, see test case
>>>>> JavaAPISuite.testJavaJdbcRDD.
>>>>> ./core/src/main/scala/org/apache/spark/rdd/JdbcRDD.scala
>>>>>
>>>>> FYI
>>>>>
>>>>> On Sat, Dec 6, 2014 at 5:43 PM, Koert Kuipers <ko...@tresata.com>
>>>>> wrote:
>>>>>
>>>>>> when i run "mvn test -pl core", i dont see JavaAPISuite being run. or
>>>>>> if it is, its being very very quiet about it. is this by design?
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>
Re: run JavaAPISuite with mavem
Posted by Michael Armbrust <mi...@databricks.com>.
Not sure about maven, but you can run that test with sbt:
sbt/sbt "sql/test-only org.apache.spark.sql.api.java.JavaAPISuite"
On Sat, Dec 6, 2014 at 9:59 PM, Ted Yu <yu...@gmail.com> wrote:
> I tried to run tests for core but there were failures. e.g. :
>
> ^[[32mExternalAppendOnlyMapSuite:^[[0m
> ^[[32m- simple insert^[[0m
> ^[[32m- insert with collision^[[0m
> ^[[32m- ordering^[[0m
> ^[[32m- null keys and values^[[0m
> ^[[32m- simple aggregator^[[0m
> ^[[32m- simple cogroup^[[0m
> Spark assembly has been built with Hive, including Datanucleus jars on
> classpath
> ^[[31m- spilling *** FAILED ***^[[0m
> ^[[31m org.apache.spark.SparkException: Job aborted due to stage failure:
> Task 0 in stage 0.0 failed 4 times, most recent failure: Lost task 0.3 in
> stage 0.0 (TID 6, localhost): java.lang.ClassNotFoundException:
> org.apache.spark.rdd.RDD$$anonfun$map$1^[[0m
> ^[[31m at java.net.URLClassLoader$1.run(URLClassLoader.java:366)^[[0m
> ^[[31m at java.net.URLClassLoader$1.run(URLClassLoader.java:355)^[[0m
> ^[[31m at java.security.AccessController.doPrivileged(Native Method)^[[0m
> ^[[31m at java.net.URLClassLoader.findClass(URLClassLoader.java:354)^[[0m
> ^[[31m at java.lang.ClassLoader.loadClass(ClassLoader.java:424)^[[0m
> ^[[31m at java.lang.ClassLoader.loadClass(ClassLoader.java:357)^[[0m
> ^[[31m at java.lang.Class.forName0(Native Method)^[[0m
> ^[[31m at java.lang.Class.forName(Class.java:270)^[[0m
>
> BTW I didn't find JavaAPISuite in test output either.
>
> Cheers
>
> On Sat, Dec 6, 2014 at 9:12 PM, Koert Kuipers <ko...@tresata.com> wrote:
>
>> Ted,
>> i mean
>> core/src/test/java/org/apache/spark/JavaAPISuite.java
>>
>> On Sat, Dec 6, 2014 at 9:27 PM, Ted Yu <yu...@gmail.com> wrote:
>>
>>> Pardon me, the test is here:
>>>
>>> sql/core/src/test/java/org/apache/spark/sql/api/java/JavaAPISuite.java
>>>
>>> You can run 'mvn test' under sql/core
>>>
>>> Cheers
>>>
>>> On Sat, Dec 6, 2014 at 5:55 PM, Ted Yu <yu...@gmail.com> wrote:
>>>
>>>> In master branch, I only found JavaAPISuite in comment:
>>>>
>>>> spark tyu$ find . -name '*.scala' -exec grep JavaAPISuite {} \; -print
>>>> * For usage example, see test case JavaAPISuite.testJavaJdbcRDD.
>>>> * converted into a `Object` array. For usage example, see test case
>>>> JavaAPISuite.testJavaJdbcRDD.
>>>> ./core/src/main/scala/org/apache/spark/rdd/JdbcRDD.scala
>>>>
>>>> FYI
>>>>
>>>> On Sat, Dec 6, 2014 at 5:43 PM, Koert Kuipers <ko...@tresata.com>
>>>> wrote:
>>>>
>>>>> when i run "mvn test -pl core", i dont see JavaAPISuite being run. or
>>>>> if it is, its being very very quiet about it. is this by design?
>>>>>
>>>>
>>>>
>>>
>>
>
Re: run JavaAPISuite with mavem
Posted by Ted Yu <yu...@gmail.com>.
I tried to run tests for core but there were failures. e.g. :
^[[32mExternalAppendOnlyMapSuite:^[[0m
^[[32m- simple insert^[[0m
^[[32m- insert with collision^[[0m
^[[32m- ordering^[[0m
^[[32m- null keys and values^[[0m
^[[32m- simple aggregator^[[0m
^[[32m- simple cogroup^[[0m
Spark assembly has been built with Hive, including Datanucleus jars on
classpath
^[[31m- spilling *** FAILED ***^[[0m
^[[31m org.apache.spark.SparkException: Job aborted due to stage failure:
Task 0 in stage 0.0 failed 4 times, most recent failure: Lost task 0.3 in
stage 0.0 (TID 6, localhost): java.lang.ClassNotFoundException:
org.apache.spark.rdd.RDD$$anonfun$map$1^[[0m
^[[31m at java.net.URLClassLoader$1.run(URLClassLoader.java:366)^[[0m
^[[31m at java.net.URLClassLoader$1.run(URLClassLoader.java:355)^[[0m
^[[31m at java.security.AccessController.doPrivileged(Native Method)^[[0m
^[[31m at java.net.URLClassLoader.findClass(URLClassLoader.java:354)^[[0m
^[[31m at java.lang.ClassLoader.loadClass(ClassLoader.java:424)^[[0m
^[[31m at java.lang.ClassLoader.loadClass(ClassLoader.java:357)^[[0m
^[[31m at java.lang.Class.forName0(Native Method)^[[0m
^[[31m at java.lang.Class.forName(Class.java:270)^[[0m
BTW I didn't find JavaAPISuite in test output either.
Cheers
On Sat, Dec 6, 2014 at 9:12 PM, Koert Kuipers <ko...@tresata.com> wrote:
> Ted,
> i mean
> core/src/test/java/org/apache/spark/JavaAPISuite.java
>
> On Sat, Dec 6, 2014 at 9:27 PM, Ted Yu <yu...@gmail.com> wrote:
>
>> Pardon me, the test is here:
>>
>> sql/core/src/test/java/org/apache/spark/sql/api/java/JavaAPISuite.java
>>
>> You can run 'mvn test' under sql/core
>>
>> Cheers
>>
>> On Sat, Dec 6, 2014 at 5:55 PM, Ted Yu <yu...@gmail.com> wrote:
>>
>>> In master branch, I only found JavaAPISuite in comment:
>>>
>>> spark tyu$ find . -name '*.scala' -exec grep JavaAPISuite {} \; -print
>>> * For usage example, see test case JavaAPISuite.testJavaJdbcRDD.
>>> * converted into a `Object` array. For usage example, see test case
>>> JavaAPISuite.testJavaJdbcRDD.
>>> ./core/src/main/scala/org/apache/spark/rdd/JdbcRDD.scala
>>>
>>> FYI
>>>
>>> On Sat, Dec 6, 2014 at 5:43 PM, Koert Kuipers <ko...@tresata.com> wrote:
>>>
>>>> when i run "mvn test -pl core", i dont see JavaAPISuite being run. or
>>>> if it is, its being very very quiet about it. is this by design?
>>>>
>>>
>>>
>>
>
Re: run JavaAPISuite with mavem
Posted by Koert Kuipers <ko...@tresata.com>.
Ted,
i mean
core/src/test/java/org/apache/spark/JavaAPISuite.java
On Sat, Dec 6, 2014 at 9:27 PM, Ted Yu <yu...@gmail.com> wrote:
> Pardon me, the test is here:
>
> sql/core/src/test/java/org/apache/spark/sql/api/java/JavaAPISuite.java
>
> You can run 'mvn test' under sql/core
>
> Cheers
>
> On Sat, Dec 6, 2014 at 5:55 PM, Ted Yu <yu...@gmail.com> wrote:
>
>> In master branch, I only found JavaAPISuite in comment:
>>
>> spark tyu$ find . -name '*.scala' -exec grep JavaAPISuite {} \; -print
>> * For usage example, see test case JavaAPISuite.testJavaJdbcRDD.
>> * converted into a `Object` array. For usage example, see test case
>> JavaAPISuite.testJavaJdbcRDD.
>> ./core/src/main/scala/org/apache/spark/rdd/JdbcRDD.scala
>>
>> FYI
>>
>> On Sat, Dec 6, 2014 at 5:43 PM, Koert Kuipers <ko...@tresata.com> wrote:
>>
>>> when i run "mvn test -pl core", i dont see JavaAPISuite being run. or if
>>> it is, its being very very quiet about it. is this by design?
>>>
>>
>>
>
Re: run JavaAPISuite with mavem
Posted by Ted Yu <yu...@gmail.com>.
Pardon me, the test is here:
sql/core/src/test/java/org/apache/spark/sql/api/java/JavaAPISuite.java
You can run 'mvn test' under sql/core
Cheers
On Sat, Dec 6, 2014 at 5:55 PM, Ted Yu <yu...@gmail.com> wrote:
> In master branch, I only found JavaAPISuite in comment:
>
> spark tyu$ find . -name '*.scala' -exec grep JavaAPISuite {} \; -print
> * For usage example, see test case JavaAPISuite.testJavaJdbcRDD.
> * converted into a `Object` array. For usage example, see test case
> JavaAPISuite.testJavaJdbcRDD.
> ./core/src/main/scala/org/apache/spark/rdd/JdbcRDD.scala
>
> FYI
>
> On Sat, Dec 6, 2014 at 5:43 PM, Koert Kuipers <ko...@tresata.com> wrote:
>
>> when i run "mvn test -pl core", i dont see JavaAPISuite being run. or if
>> it is, its being very very quiet about it. is this by design?
>>
>
>
Re: run JavaAPISuite with mavem
Posted by Ted Yu <yu...@gmail.com>.
In master branch, I only found JavaAPISuite in comment:
spark tyu$ find . -name '*.scala' -exec grep JavaAPISuite {} \; -print
* For usage example, see test case JavaAPISuite.testJavaJdbcRDD.
* converted into a `Object` array. For usage example, see test case
JavaAPISuite.testJavaJdbcRDD.
./core/src/main/scala/org/apache/spark/rdd/JdbcRDD.scala
FYI
On Sat, Dec 6, 2014 at 5:43 PM, Koert Kuipers <ko...@tresata.com> wrote:
> when i run "mvn test -pl core", i dont see JavaAPISuite being run. or if
> it is, its being very very quiet about it. is this by design?
>