You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Saisai Shao (JIRA)" <ji...@apache.org> on 2017/02/14 08:33:42 UTC
[jira] [Comment Edited] (SPARK-19579) spark-submit fails to run
Kafka Stream python script
[ https://issues.apache.org/jira/browse/SPARK-19579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15865360#comment-15865360 ]
Saisai Shao edited comment on SPARK-19579 at 2/14/17 8:33 AM:
--------------------------------------------------------------
Only this specific part is not supported, all the other Python APIs are still supported.
was (Author: jerryshao):
Only this specific part is not supported, all the that Python APIs are still supported.
> spark-submit fails to run Kafka Stream python script
> ----------------------------------------------------
>
> Key: SPARK-19579
> URL: https://issues.apache.org/jira/browse/SPARK-19579
> Project: Spark
> Issue Type: Bug
> Components: Spark Submit
> Affects Versions: 2.1.0
> Environment: Linux Ubuntu 16.10 64bit
> Reporter: Piotr Nestorow
>
> Kafka Stream python script is executed but it fails with:
> TypeError: 'JavaPackage' object is not callable
> The Spark Kafka streaming jar is provided:
> spark-streaming-kafka-0-10_2.11-2.1.0.jar
> Kafka version: kafka_2.11-0.10.1.1
> In the conf/spark-defaults.conf:
> spark.jars.packages org.apache.spark:spark-streaming-kafka-0-10_2.11:2.1.0
> Also at runtime:
> bin/spark-submit --jars /usr/local/spark/jars/spark-streaming-kafka-0-10_2.11-2.1.0.jar --driver-class-path /usr/local/spark/jars/spark-streaming-kafka-0-10_2.11-2.1.0.jar /home/sysveradmin/work/Programs/ApacheSpark/ex_kafka_stream.py localhost:2181 example_topic
> Also the Spark Kafka streaming example:
> 'examples/src/main/python/streaming/kafka_wordcount.py'
> can be used to check this problem.
> Nothing helps...
> Detailed output:
> _________________
> Spark Streaming's Kafka libraries not found in class path. Try one of the following.
> 1. Include the Kafka library and its dependencies with in the
> spark-submit command as
> $ bin/spark-submit --packages org.apache.spark:spark-streaming-kafka-0-8:2.1.0 ...
> 2. Download the JAR of the artifact from Maven Central http://search.maven.org/,
> Group Id = org.apache.spark, Artifact Id = spark-streaming-kafka-0-8-assembly, Version = 2.1.0.
> Then, include the jar in the spark-submit command as
> $ bin/spark-submit --jars <spark-streaming-kafka-0-8-assembly.jar> ...
> ________________________________________________________________________________________________
> Traceback (most recent call last):
> File "/home/sysveradmin/work/Programs/ApacheSpark/ex_kafka_stream.py", line 18, in <module>
> kvs = KafkaUtils.createStream(ssc, zkQuorum, "spark-streaming-consumer", {topic: 1})
> File "/usr/local/spark/python/lib/pyspark.zip/pyspark/streaming/kafka.py", line 69, in createStream
> File "/usr/local/spark/python/lib/pyspark.zip/pyspark/streaming/kafka.py", line 195, in _get_helper
> TypeError: 'JavaPackage' object is not callable
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org