You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Piotr Nestorow (JIRA)" <ji...@apache.org> on 2017/02/13 13:30:41 UTC
[jira] [Updated] (SPARK-19579) spark-submit fails to run Kafka
Stream python script
[ https://issues.apache.org/jira/browse/SPARK-19579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Piotr Nestorow updated SPARK-19579:
-----------------------------------
Description:
Kafka Stream python script is executed but it fails with:
TypeError: 'JavaPackage' object is not callable
The Spark Kafka streaming jar is provided:
spark-streaming-kafka-0-10_2.11-2.1.0.jar
Kafka version: kafka_2.11-0.10.1.1
In the conf/spark-defaults.conf:
spark.jars.packages org.apache.spark:spark-streaming-kafka-0-10_2.11:2.1.0
Also at runtime:
bin/spark-submit --jars /usr/local/spark/jars/spark-streaming-kafka-0-10_2.11-2.1.0.jar --driver-class-path /usr/local/spark/jars/spark-streaming-kafka-0-10_2.11-2.1.0.jar /home/sysveradmin/work/Programs/ApacheSpark/ex_kafka_stream.py localhost:2181 example_topic
Also the Spark Kafka streaming example:
'examples/src/main/python/streaming/kafka_wordcount.py'
can be used to check this problem.
Nothing helps...
Detailed output:
_________________
Spark Streaming's Kafka libraries not found in class path. Try one of the following.
1. Include the Kafka library and its dependencies with in the
spark-submit command as
$ bin/spark-submit --packages org.apache.spark:spark-streaming-kafka-0-8:2.1.0 ...
2. Download the JAR of the artifact from Maven Central http://search.maven.org/,
Group Id = org.apache.spark, Artifact Id = spark-streaming-kafka-0-8-assembly, Version = 2.1.0.
Then, include the jar in the spark-submit command as
$ bin/spark-submit --jars <spark-streaming-kafka-0-8-assembly.jar> ...
________________________________________________________________________________________________
Traceback (most recent call last):
File "/home/sysveradmin/work/Programs/ApacheSpark/ex_kafka_stream.py", line 18, in <module>
kvs = KafkaUtils.createStream(ssc, zkQuorum, "spark-streaming-consumer", {topic: 1})
File "/usr/local/spark/python/lib/pyspark.zip/pyspark/streaming/kafka.py", line 69, in createStream
File "/usr/local/spark/python/lib/pyspark.zip/pyspark/streaming/kafka.py", line 195, in _get_helper
TypeError: 'JavaPackage' object is not callable
was:
Kafka Stream python script is executed but it fails with:
TypeError: 'JavaPackage' object is not callable
The Spark Kafka streaming jar is provided:
spark-streaming-kafka-0-10_2.11-2.1.0.jar
In the conf/spark-defaults.conf:
spark.jars.packages org.apache.spark:spark-streaming-kafka-0-10_2.11:2.1.0
Also at runtime:
bin/spark-submit --jars /usr/local/spark/jars/spark-streaming-kafka-0-10_2.11-2.1.0.jar --driver-class-path /usr/local/spark/jars/spark-streaming-kafka-0-10_2.11-2.1.0.jar /home/sysveradmin/work/Programs/ApacheSpark/ex_kafka_stream.py localhost:2181 example_topic
Nothing helps...
Detailed output:
_________________
Spark Streaming's Kafka libraries not found in class path. Try one of the following.
1. Include the Kafka library and its dependencies with in the
spark-submit command as
$ bin/spark-submit --packages org.apache.spark:spark-streaming-kafka-0-8:2.1.0 ...
2. Download the JAR of the artifact from Maven Central http://search.maven.org/,
Group Id = org.apache.spark, Artifact Id = spark-streaming-kafka-0-8-assembly, Version = 2.1.0.
Then, include the jar in the spark-submit command as
$ bin/spark-submit --jars <spark-streaming-kafka-0-8-assembly.jar> ...
________________________________________________________________________________________________
Traceback (most recent call last):
File "/home/sysveradmin/work/Programs/ApacheSpark/ex_kafka_stream.py", line 18, in <module>
kvs = KafkaUtils.createStream(ssc, zkQuorum, "spark-streaming-consumer", {topic: 1})
File "/usr/local/spark/python/lib/pyspark.zip/pyspark/streaming/kafka.py", line 69, in createStream
File "/usr/local/spark/python/lib/pyspark.zip/pyspark/streaming/kafka.py", line 195, in _get_helper
TypeError: 'JavaPackage' object is not callable
> spark-submit fails to run Kafka Stream python script
> ----------------------------------------------------
>
> Key: SPARK-19579
> URL: https://issues.apache.org/jira/browse/SPARK-19579
> Project: Spark
> Issue Type: Bug
> Components: Spark Submit
> Affects Versions: 2.1.0
> Environment: Linux Ubuntu 16.10 64bit
> Reporter: Piotr Nestorow
>
> Kafka Stream python script is executed but it fails with:
> TypeError: 'JavaPackage' object is not callable
> The Spark Kafka streaming jar is provided:
> spark-streaming-kafka-0-10_2.11-2.1.0.jar
> Kafka version: kafka_2.11-0.10.1.1
> In the conf/spark-defaults.conf:
> spark.jars.packages org.apache.spark:spark-streaming-kafka-0-10_2.11:2.1.0
> Also at runtime:
> bin/spark-submit --jars /usr/local/spark/jars/spark-streaming-kafka-0-10_2.11-2.1.0.jar --driver-class-path /usr/local/spark/jars/spark-streaming-kafka-0-10_2.11-2.1.0.jar /home/sysveradmin/work/Programs/ApacheSpark/ex_kafka_stream.py localhost:2181 example_topic
> Also the Spark Kafka streaming example:
> 'examples/src/main/python/streaming/kafka_wordcount.py'
> can be used to check this problem.
> Nothing helps...
> Detailed output:
> _________________
> Spark Streaming's Kafka libraries not found in class path. Try one of the following.
> 1. Include the Kafka library and its dependencies with in the
> spark-submit command as
> $ bin/spark-submit --packages org.apache.spark:spark-streaming-kafka-0-8:2.1.0 ...
> 2. Download the JAR of the artifact from Maven Central http://search.maven.org/,
> Group Id = org.apache.spark, Artifact Id = spark-streaming-kafka-0-8-assembly, Version = 2.1.0.
> Then, include the jar in the spark-submit command as
> $ bin/spark-submit --jars <spark-streaming-kafka-0-8-assembly.jar> ...
> ________________________________________________________________________________________________
> Traceback (most recent call last):
> File "/home/sysveradmin/work/Programs/ApacheSpark/ex_kafka_stream.py", line 18, in <module>
> kvs = KafkaUtils.createStream(ssc, zkQuorum, "spark-streaming-consumer", {topic: 1})
> File "/usr/local/spark/python/lib/pyspark.zip/pyspark/streaming/kafka.py", line 69, in createStream
> File "/usr/local/spark/python/lib/pyspark.zip/pyspark/streaming/kafka.py", line 195, in _get_helper
> TypeError: 'JavaPackage' object is not callable
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org