You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Hrishikesh Subramonian <hr...@flytxt.com> on 2015/04/21 10:30:22 UTC
python/run-tests fails at spark master branch
Hi,
I cloned spark master branch from github and was built successfully
using the mvn -DskipTests clean package command.
But the python/run-tests command fails. Please see the log below:
Running PySpark tests. Output is in python/unit-tests.log.
Testing with Python version:
Python 2.7.3
Run core tests ...
Running test: pyspark/rdd.py
Running test: pyspark/context.py
Running test: pyspark/conf.py
Running test: pyspark/broadcast.py
Running test: pyspark/accumulators.py
Running test: pyspark/serializers.py
Running test: pyspark/profiler.py
Running test: pyspark/shuffle.py
Running test: pyspark/tests.py
Run sql tests ...
Running test: pyspark/sql/_types.py
Running test: pyspark/sql/context.py
Running test: pyspark/sql/dataframe.py
Running test: pyspark/sql/functions.py
Running test: pyspark/sql/tests.py
Run mllib tests ...
Running test: pyspark/mllib/classification.py
Running test: pyspark/mllib/clustering.py
Running test: pyspark/mllib/evaluation.py
Running test: pyspark/mllib/feature.py
Running test: pyspark/mllib/fpm.py
Running test: pyspark/mllib/linalg.py
Running test: pyspark/mllib/rand.py
Running test: pyspark/mllib/recommendation.py
Running test: pyspark/mllib/regression.py
Running test: pyspark/mllib/stat/_statistics.py
Running test: pyspark/mllib/tree.py
Running test: pyspark/mllib/util.py
Running test: pyspark/mllib/tests.py
Run ml tests ...
Running test: pyspark/ml/feature.py
Running test: pyspark/ml/classification.py
Running test: pyspark/ml/tests.py
Run streaming tests ...
Failed to find Spark Streaming Kafka assembly jar in
/home/xyz/spark/external/kafka-assembly
You need to build Spark with 'build/sbt assembly/assembly
streaming-kafka-assembly/assembly' or 'build/mvn package' before running
this program
Is anybody facing the same problem?
Regards,
Hrishikesh
Re: python/run-tests fails at spark master branch
Posted by Saisai Shao <sa...@gmail.com>.
Hi Hrishikesh,
Seems the behavior of Kafka-assembly is a little different when using Maven
to sbt. The assembly jar name and location is different while using `mvn
package`. This is a actually bug, I'm fixing this now.
Thanks
Jerry
2015-04-22 13:37 GMT+08:00 Hrishikesh Subramonian <
hrishikesh.subramonian@flytxt.com>:
> Hi,
>
> The *python/run-tests* executes successfully after I ran *'build/sbt
> assembly*' command. But the tests fail if I run it after *'mvn
> -Dskiptests clean package'* command. Why does it run in *sbt assembly*
> and not in* mvn package*?
>
> --
> Hrishikesh
>
> On Wednesday 22 April 2015 07:38 AM, Saisai Shao wrote:
>
> Hi Hrishikesh,
>
> Now we add Kafka unit test for python which relies on Kafka assembly
> jar, so you need to run `sbt assembly` or mvn `package` at first to get an
> assemble jar.
>
>
>
> 2015-04-22 1:15 GMT+08:00 Marcelo Vanzin <va...@cloudera.com>:
>
>> On Tue, Apr 21, 2015 at 1:30 AM, Hrishikesh Subramonian
>> <hr...@flytxt.com> wrote:
>>
>> > Run streaming tests ...
>> > Failed to find Spark Streaming Kafka assembly jar in
>> > /home/xyz/spark/external/kafka-assembly
>> > You need to build Spark with 'build/sbt assembly/assembly
>> > streaming-kafka-assembly/assembly' or 'build/mvn package' before running
>> > this program
>> >
>> >
>> > Is anybody facing the same problem?
>>
>> Have you built the assemblies before running the tests? (mvn package
>> -DskipTests, or sbt assembly)
>>
>>
>> --
>> Marcelo
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
>> For additional commands, e-mail: dev-help@spark.apache.org
>>
>>
>
>
Re: python/run-tests fails at spark master branch
Posted by Hrishikesh Subramonian <hr...@flytxt.com>.
Hi,
The /python/run-tests/ executes successfully after I ran /'build/sbt
assembly/' command. But the tests fail if I run it after /'mvn
-Dskiptests clean package'/ command. Why does it run in /sbt assembly/
and not in/mvn package/?
--
Hrishikesh
On Wednesday 22 April 2015 07:38 AM, Saisai Shao wrote:
> Hi Hrishikesh,
>
> Now we add Kafka unit test for python which relies on Kafka assembly
> jar, so you need to run `sbt assembly` or mvn `package` at first to
> get an assemble jar.
>
>
>
> 2015-04-22 1:15 GMT+08:00 Marcelo Vanzin <vanzin@cloudera.com
> <ma...@cloudera.com>>:
>
> On Tue, Apr 21, 2015 at 1:30 AM, Hrishikesh Subramonian
> <hrishikesh.subramonian@flytxt.com
> <ma...@flytxt.com>> wrote:
>
> > Run streaming tests ...
> > Failed to find Spark Streaming Kafka assembly jar in
> > /home/xyz/spark/external/kafka-assembly
> > You need to build Spark with 'build/sbt assembly/assembly
> > streaming-kafka-assembly/assembly' or 'build/mvn package' before
> running
> > this program
> >
> >
> > Is anybody facing the same problem?
>
> Have you built the assemblies before running the tests? (mvn package
> -DskipTests, or sbt assembly)
>
>
> --
> Marcelo
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
> <ma...@spark.apache.org>
> For additional commands, e-mail: dev-help@spark.apache.org
> <ma...@spark.apache.org>
>
>
Re: python/run-tests fails at spark master branch
Posted by Saisai Shao <sa...@gmail.com>.
Hi Hrishikesh,
Now we add Kafka unit test for python which relies on Kafka assembly jar,
so you need to run `sbt assembly` or mvn `package` at first to get an
assemble jar.
2015-04-22 1:15 GMT+08:00 Marcelo Vanzin <va...@cloudera.com>:
> On Tue, Apr 21, 2015 at 1:30 AM, Hrishikesh Subramonian
> <hr...@flytxt.com> wrote:
>
> > Run streaming tests ...
> > Failed to find Spark Streaming Kafka assembly jar in
> > /home/xyz/spark/external/kafka-assembly
> > You need to build Spark with 'build/sbt assembly/assembly
> > streaming-kafka-assembly/assembly' or 'build/mvn package' before running
> > this program
> >
> >
> > Is anybody facing the same problem?
>
> Have you built the assemblies before running the tests? (mvn package
> -DskipTests, or sbt assembly)
>
>
> --
> Marcelo
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
> For additional commands, e-mail: dev-help@spark.apache.org
>
>
Re: python/run-tests fails at spark master branch
Posted by Marcelo Vanzin <va...@cloudera.com>.
On Tue, Apr 21, 2015 at 1:30 AM, Hrishikesh Subramonian
<hr...@flytxt.com> wrote:
> Run streaming tests ...
> Failed to find Spark Streaming Kafka assembly jar in
> /home/xyz/spark/external/kafka-assembly
> You need to build Spark with 'build/sbt assembly/assembly
> streaming-kafka-assembly/assembly' or 'build/mvn package' before running
> this program
>
>
> Is anybody facing the same problem?
Have you built the assemblies before running the tests? (mvn package
-DskipTests, or sbt assembly)
--
Marcelo
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org