You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by jerryshao <gi...@git.apache.org> on 2016/07/25 02:53:41 UTC
[GitHub] spark pull request #14340: [SPARK-16534][Streaming][Kafka] Add Python API su...
GitHub user jerryshao opened a pull request:
https://github.com/apache/spark/pull/14340
[SPARK-16534][Streaming][Kafka] Add Python API support for Kafka 0.10 connector
## What changes were proposed in this pull request?
This PR adds the support of Python API for Spark Streaming Kafka 0.10 connector.
1. Provides equal functionality for Python API.
2. Adds unit test to test different location strategies and consumer strategies for python API.
3. Adds a example `kafka010_direct_wordcount.py`.
## How was this patch tested?
Adds the unit tests to cover different APIs. Also did integrated tests in real cluster.
CC @tdas @koeninger please help to review, thanks a lot.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/jerryshao/apache-spark SPARK-16534
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/14340.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #14340
----
commit 569fa9e8089f410f7aea476232b697c299a235ae
Author: jerryshao <ss...@hortonworks.com>
Date: 2016-07-20T07:42:20Z
Add python support for Spark Streaming Kafka 0.10 connector
commit dcca3371da0c73bbcadb4331a2869e0303bd8ae8
Author: jerryshao <ss...@hortonworks.com>
Date: 2016-07-25T02:20:48Z
And unit test for Python Kafka 0.10
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14340: [SPARK-16534][Streaming][Kafka] Add Python API support f...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14340
**[Test build #62870 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62870/consoleFull)** for PR 14340 at commit [`828051a`](https://github.com/apache/spark/commit/828051a02d5051c2e37150167df12660184beb18).
* This patch **fails Python style tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `class Kafka010DStream(KafkaDStream):`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14340: [SPARK-16534][Streaming][Kafka] Add Python API support f...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14340
**[Test build #62871 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62871/consoleFull)** for PR 14340 at commit [`9e6895a`](https://github.com/apache/spark/commit/9e6895a47ef403ef76ef605829bdaa8b90e5a2b7).
* This patch **fails PySpark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14340: [SPARK-16534][Streaming][Kafka] Add Python API su...
Posted by koeninger <gi...@git.apache.org>.
Github user koeninger commented on a diff in the pull request:
https://github.com/apache/spark/pull/14340#discussion_r72120535
--- Diff: external/kafka-0-10/src/main/scala/org/apache/spark/streaming/kafka010/KafkaUtils.scala ---
@@ -177,3 +182,172 @@ object KafkaUtils extends Logging {
}
}
}
+
+private[kafka010] class KafkaUtilsPythonHelper extends Logging {
+ import KafkaUtilsPythonHelper._
+
+ def createDirectStream(
+ jssc: JavaStreamingContext,
+ locationStrategy: LocationStrategy,
+ consumerStrategy: ConsumerStrategy[Array[Byte], Array[Byte]]
+ ): JavaDStream[Array[Byte]] = {
+ fixKafkaParams(consumerStrategy.executorKafkaParams)
+ val stream = KafkaUtils.createDirectStream(jssc.ssc, locationStrategy, consumerStrategy)
+ .map { r =>
+ PythonConsumerRecord(r.topic(), r.partition(), r.offset(), r.timestamp(),
+ r.timestampType().toString, r.checksum(), r.serializedKeySize(), r.serializedValueSize(),
+ r.key(), r.value())
+ }.mapPartitions(picklerIterator)
+ new JavaDStream(stream)
+ }
+
+ def createRDD(
+ jsc: JavaSparkContext,
+ kafkaParams: ju.Map[String, Object],
+ offsetRanges: ju.List[OffsetRange],
+ locationStrategy: LocationStrategy
+ ): JavaRDD[Array[Byte]] = {
+ fixKafkaParams(kafkaParams)
+ val rdd = KafkaUtils.createRDD[Array[Byte], Array[Byte]](
+ jsc.sc,
+ kafkaParams,
+ offsetRanges.toArray(new Array[OffsetRange](offsetRanges.size())),
+ locationStrategy)
+ .map { r =>
+ PythonConsumerRecord(r.topic(), r.partition(), r.offset(), r.timestamp(),
+ r.timestampType().toString, r.checksum(), r.serializedKeySize(),
+ r.serializedValueSize(), r.key(), r.value())
+ }.mapPartitions(picklerIterator)
+ new JavaRDD(rdd)
+ }
+
+ private def fixKafkaParams(kafkaParams: ju.Map[String, Object]): Unit = {
+ val decoder = classOf[ByteArrayDeserializer].getCanonicalName
+
+ val keyDecoder = kafkaParams.get(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG)
+ if (keyDecoder != null && keyDecoder != decoder) {
+ logWarning(s"${ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG} $keyDecoder is not supported " +
+ s"for python Kafka API, overriding with $decoder")
+ }
+ kafkaParams.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, decoder)
+
+ val valueDecoder = kafkaParams.get(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG)
+ if (valueDecoder != null && valueDecoder != decoder) {
+ logWarning(s"${ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG} $valueDecoder is not " +
+ s"supported for python Kafka API, overriding with $decoder")
+ }
+ kafkaParams.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, decoder)
+ }
+
+ // Helper functions to convert Python object to Java object
+ def createOffsetRange(topic: String, partition: JInt, fromOffset: JLong, untilOffset: JLong
+ ): OffsetRange = OffsetRange.create(topic, partition, fromOffset, untilOffset)
+
+ def createPreferBrokers(): LocationStrategy = LocationStrategies.PreferBrokers
+
+ def createPreferConsistent(): LocationStrategy = LocationStrategies.PreferConsistent
+
+ def createPreferFixed(hostMap: ju.Map[TopicPartition, String]): LocationStrategy = {
+ LocationStrategies.PreferFixed(hostMap)
+ }
+
+ def createTopicPartition(topic: String, partition: JInt): TopicPartition =
+ new TopicPartition(topic, partition)
+
+ def createSubscribe(
+ topics: ju.Set[String],
+ kafkaParams: ju.Map[String, Object],
+ offsets: ju.Map[TopicPartition, JLong]): ConsumerStrategy[Array[Byte], Array[Byte]] =
+ ConsumerStrategies.Subscribe(topics, kafkaParams, offsets)
+
+ def createSubscribePattern(
+ pattern: String,
+ kafkaParams: ju.Map[String, Object],
+ offsets: ju.Map[TopicPartition, JLong]): ConsumerStrategy[Array[Byte], Array[Byte]] = {
+ ConsumerStrategies.SubscribePattern(ju.regex.Pattern.compile(pattern), kafkaParams, offsets)
+ }
+
+ def createAssign(
+ topicPartitions: ju.Set[TopicPartition],
+ kafkaParams: ju.Map[String, Object],
+ offsets: ju.Map[TopicPartition, JLong]): ConsumerStrategy[Array[Byte], Array[Byte]] = {
+ ConsumerStrategies.Assign(topicPartitions, kafkaParams, offsets)
+ }
+
+ def offsetRangesOfKafkaRDD(rdd: RDD[_]): ju.List[OffsetRange] = {
--- End diff --
We probably need a similar way to do the cast of the java dstream to CanCommitOffsets so that users can access the commitAsync method
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14340: [SPARK-16534][Streaming][Kafka] Add Python API su...
Posted by koeninger <gi...@git.apache.org>.
Github user koeninger commented on a diff in the pull request:
https://github.com/apache/spark/pull/14340#discussion_r72248712
--- Diff: external/kafka-0-10/src/main/scala/org/apache/spark/streaming/kafka010/KafkaUtils.scala ---
@@ -221,22 +222,20 @@ private[kafka010] class KafkaUtilsPythonHelper extends Logging {
new JavaRDD(rdd)
}
- private def fixKafkaParams(kafkaParams: ju.Map[String, Object]): Unit = {
+ private def validateKafkaParams(kafkaParams: ju.Map[String, Object]): Unit = {
val decoder = classOf[ByteArrayDeserializer].getCanonicalName
val keyDecoder = kafkaParams.get(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG)
- if (keyDecoder != null && keyDecoder != decoder) {
- logWarning(s"${ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG} $keyDecoder is not supported " +
- s"for python Kafka API, overriding with $decoder")
+ if (keyDecoder == null || keyDecoder != decoder) {
+ throw new SparkException(s"${ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG} $keyDecoder " +
+ s"is not supported for python Kafka API, please configured with $decoder")
--- End diff --
"configured" should be "configure"
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14340: [SPARK-16534][Streaming][Kafka] Add Python API support f...
Posted by koeninger <gi...@git.apache.org>.
Github user koeninger commented on the issue:
https://github.com/apache/spark/pull/14340
In terms of feature parity, are we OK with telling python users they are out of luck if they need something not covered by Assign / Subscribe / SubscribePattern? For scala / java users they can just implement that interface themselves
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14340: [SPARK-16534][Streaming][Kafka] Add Python API su...
Posted by koeninger <gi...@git.apache.org>.
Github user koeninger commented on a diff in the pull request:
https://github.com/apache/spark/pull/14340#discussion_r72122440
--- Diff: python/pyspark/streaming/kafka010.py ---
@@ -0,0 +1,370 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+from pyspark.rdd import RDD
+from pyspark.serializers import AutoBatchedSerializer, PickleSerializer
+from pyspark.streaming import DStream
+from pyspark.streaming.kafka import KafkaDStream, KafkaRDD, OffsetRange
+
+__all__ = ['Assign', 'KafkaConsumerRecord', 'KafkaUtils', 'PreferBrokers', 'PreferConsistent',
+ 'PreferFixed', 'Subscribe', 'SubscribePattern', 'TopicPartition', 'utf8_decoder']
+
+
+def utf8_decoder(s):
+ """ Decode the unicode as UTF-8 """
+ if s is None:
+ return None
+ return s.decode('utf-8')
+
+
+class KafkaUtils(object):
+
+ @staticmethod
+ def createDirectStream(ssc, locationStrategy, consumerStrategy,
+ keyDecoder=utf8_decoder, valueDecoder=utf8_decoder):
+ """
+ .. note:: Experimental
+
+ Create an input stream that directly pulls messages from Kafka 0.10 brokers with different
+ location strategy and consumer strategy.
+
+ This does not use Zookeeper to store offsets. The consumed offsets are tracked
+ by the stream itself. For interoperability with Kafka monitoring tools that depend on
+ Zookeeper, you have to update Kafka/Zookeeper yourself from the streaming application.
+ You can access the offsets used in each batch from the generated RDDs (see
+
+ To recover from driver failures, you have to enable checkpointing in the StreamingContext.
+ The information on consumed offset can be recovered from the checkpoint.
+ See the programming guide for details (constraints, etc.).
+
+ :param ssc: StreamingContext object,
+ :param locationStrategy: Strategy to schedule consumers for a given TopicPartition on an
+ executor.
+ :param consumerStrategy: Choices of how to create and configure underlying Kafka
+ Consumers on driver and executors.
+ :param keyDecoder: A function to decode key (default is utf8_decoder).
+ :param valueDecoder: A function to decode value (default is utf8_decoder).
+ :return: A DStream object.
+ """
+
+ helper = KafkaUtils._get_helper(ssc._sc)
+ ser = AutoBatchedSerializer(PickleSerializer())
+
+ jlocationStrategy = locationStrategy._jLocationStrategy(helper)
+ jconsumerStrategy = consumerStrategy._jConsumerStrategy(helper)
+
+ jstream = helper.createDirectStream(ssc._jssc, jlocationStrategy, jconsumerStrategy)
+
+ def func(m):
+ m._set_key_deserializer(keyDecoder)
+ m._set_value_deserializer(valueDecoder)
+ return m
+
+ stream = DStream(jstream, ssc, ser).map(func)
+
+ return KafkaDStream(stream._jdstream, ssc, stream._jrdd_deserializer)
+
+ @staticmethod
+ def createRDD(sc, kafkaParams, offsetRanges, locationStrategy,
+ keyDecoder=utf8_decoder, valueDecoder=utf8_decoder):
+ """
+ .. note:: Experimental
+
+ Create a Kafka RDD using offset ranges and location strategy.
+
+ :param sc: SparkContext object.
+ :param kafkaParams: Additional params for Kafka.
+ :param offsetRanges: list of offsetRange to specify topic:partition:[start, end) to consume.
+ :param locationStrategy: Strategy to schedule consumers for a given TopicPartition on an
+ executor.
+ :param keyDecoder: A function to decode key (default is utf8_decoder).
+ :param valueDecoder: A function to decode value (default is utf8_decoder).
+ :return: A RDD object.
+ """
+
+ if not isinstance(kafkaParams, dict):
+ raise TypeError("kafkaParams should be dict")
+
+ helper = KafkaUtils._get_helper(sc)
+ joffsetRanges = [o._jOffsetRange(helper) for o in offsetRanges]
+ jlocationStrategy = locationStrategy._jLocationStrategy(helper)
+
+ jrdd = helper.createRDD(sc._jsc, kafkaParams, joffsetRanges, jlocationStrategy)
+
+ def func(m):
+ m._set_key_deserializer(keyDecoder)
+ m._set_value_deserializer(valueDecoder)
+ return m
+
+ rdd = RDD(jrdd, sc).map(func)
+
+ return KafkaRDD(rdd._jrdd, sc, rdd._jrdd_deserializer)
+
+ @staticmethod
+ def _get_helper(sc):
+ try:
+ helper = sc._jvm.org.apache.spark.streaming.kafka010.KafkaUtilsPythonHelper()
+ KafkaRDD.set_helper(helper)
+ return helper
+ except TypeError as e:
+ if str(e) == "'JavaPackage' object is not callable":
+ KafkaUtils._printErrorMsg(sc)
+ raise
+
+ @staticmethod
+ def _printErrorMsg(sc):
+ print("""
+________________________________________________________________________________________________
+
+ Spark Streaming's Kafka libraries not found in class path. Try one of the following.
+
+ 1. Include the Kafka library and its dependencies with in the
+ spark-submit command as
+
+ $ bin/spark-submit --packages org.apache.spark:spark-streaming-kafka-0-10:%s ...
+
+ 2. Download the JAR of the artifact from Maven Central http://search.maven.org/,
+ Group Id = org.apache.spark, Artifact Id = spark-streaming-kafka-0-10-assembly, Version = %s.
+ Then, include the jar in the spark-submit command as
+
+ $ bin/spark-submit --jars <spark-streaming-kafka-0-10-assembly.jar> ...
+
+________________________________________________________________________________________________
+
+""" % (sc.version, sc.version))
+
+
+class LocationStrategy(object):
+ """
+ .. note:: Experimental
+
+ A python wrapper of Scala LocationStrategy.
+ """
+
+ def _jLocationStrategy(self, helper):
+ pass
+
+
+class PreferBrokers(LocationStrategy):
+ """
+ .. note:: Experimental
+
+ Use this only if your executors are on the same nodes as your kafka brokers.
+
+ """
+ def _jLocationStrategy(self, helper):
+ return helper.createPreferBrokers()
+
+
+class PreferConsistent(LocationStrategy):
+ """
+ .. note:: Experimental
+
+ Use this in most cases, it will consistently distribute partitions across all executors.
+ """
+
+ def _jLocationStrategy(self, helper):
+ return helper.createPreferConsistent()
+
+
+class PreferFixed(LocationStrategy):
+ """
+ .. note:: Experimental
+
+ Use this to place particular TopicPartitions on particular hosts if your load is uneven. Any
+ TopicPartition not specified in the map will use a consistent location.
+ """
+
+ def __init__(self, hostMap):
+ """
+ Python wrapper of Scala PreferFixed.
+
+ :param hostMap: A dict of TopicPartition to hostname.
+ """
+ self.hostMap = hostMap
+
+ def _jLocationStrategy(self, helper):
+ jhostMap = dict([(k._jTopicPartition(helper), v) for (k, v) in self.hostMap.items()])
+ return helper.createPreferFixed(jhostMap)
+
+
+class ConsumerStrategy(object):
+ """
+ .. note:: Experimental
+
+ A python wrapper of Scala ConsumerStrategy.
+ """
+
+ def _jConsumerStrategy(self, helper):
+ pass
+
+
+class Subscribe(ConsumerStrategy):
+ """
+ .. note:: Experimental
+
+ Subscribe to a collection of topics.
+ """
+
+ def __init__(self, topics, kafkaParams, offsets=None):
+ """
+ Subscribe to a collection of topics.
+
+ :param topics: List of topics to subscribe.
+ :param kafkaParams: Kafka parameters.
+ :param offsets: offsets to begin at on initial startup. If no offset is given for a
+ TopicPartition, the committed offset (if applicable) or kafka param
+ auto.offset.reset will be used.
+ """
+ self.topics = set(topics)
+ self.kafkaParams = kafkaParams
+ self.offsets = dict() if offsets is None else offsets
+
+ def _jConsumerStrategy(self, helper):
+ jOffsets = dict([k._jTopicPartition(helper), v] for (k, v) in self.offsets.items())
+ return helper.createSubscribe(self.topics, self.kafkaParams, jOffsets)
+
+
+class SubscribePattern(ConsumerStrategy):
+ """
+ .. note:: Experimental
+
+ Subscribe to all topics matching specified pattern to get dynamically assigned partitions.
+ """
+
+ def __init__(self, pattern, kafkaParams, offsets=None):
+ """
+ Subscribe to all topics matching specified pattern to get dynamically assigned partitions.
+
+ :param pattern: pattern to subscribe to.
+ :param kafkaParams: Kafka parameters.
+ :param offsets: offsets to begin at on initial startup. If no offset is given for a
+ TopicPartition, the committed offset (if applicable) or kafka param
+ auto.offset.reset will be used.
+ """
+ self.pattern = pattern
+ self.kafkaParams = kafkaParams
+ self.offsets = dict() if offsets is None else offsets
+
+ def _jConsumerStrategy(self, helper):
+ jOffsets = dict([k._jTopicPartition(helper), v] for (k, v) in self.offsets.items())
+ return helper.createSubscribePattern(self.pattern, self.kafkaParams, jOffsets)
+
+
+class Assign(ConsumerStrategy):
+ """
+ .. note:: Experimental
+
+ Assign a fixed collection of TopicPartitions.
+ """
+
+ def __init__(self, topicPartitions, kafkaParams, offsets=None):
+ """
+ Assign a fixed collection of TopicPartitions.
+
+ :param topicPartitions: List of TopicPartitions to assign.
+ :param kafkaParams: kafka parameters.
+ :param offsets: offsets to begin at on initial startup. If no offset is given for a
+ TopicPartition, the committed offset (if applicable) or kafka param
+ auto.offset.reset will be used.
+ """
+ self.topicPartitions = set(topicPartitions)
+ self.kafkaParams = kafkaParams
+ self.offsets = dict() if offsets is None else offsets
+
+ def _jConsumerStrategy(self, helper):
+ jTopicPartitions = [i._jTopicPartition(helper) for i in self.topicPartitions]
+ jOffsets = dict([k._jTopicPartition(helper), v] for (k, v) in self.offsets.items())
+ return helper.createAssign(set(jTopicPartitions), self.kafkaParams, jOffsets)
+
+
+class TopicPartition(object):
+ """
+ Represents a specific top and partition for Kafka.
--- End diff --
top should be topic
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14340: [SPARK-16534][Streaming][Kafka] Add Python API support f...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14340
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62795/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14340: [SPARK-16534][Streaming][Kafka] Add Python API support f...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14340
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14340: [SPARK-16534][Streaming][Kafka] Add Python API su...
Posted by koeninger <gi...@git.apache.org>.
Github user koeninger commented on a diff in the pull request:
https://github.com/apache/spark/pull/14340#discussion_r72177258
--- Diff: external/kafka-0-10/src/main/scala/org/apache/spark/streaming/kafka010/KafkaUtils.scala ---
@@ -177,3 +182,172 @@ object KafkaUtils extends Logging {
}
}
}
+
+private[kafka010] class KafkaUtilsPythonHelper extends Logging {
+ import KafkaUtilsPythonHelper._
+
+ def createDirectStream(
+ jssc: JavaStreamingContext,
+ locationStrategy: LocationStrategy,
+ consumerStrategy: ConsumerStrategy[Array[Byte], Array[Byte]]
+ ): JavaDStream[Array[Byte]] = {
+ fixKafkaParams(consumerStrategy.executorKafkaParams)
--- End diff --
Good point. I think the reasonable choices here are to wrap the passed in consumer strategy with a python-specific class that delegates and does the defensive copy, or to just error if the serializers are not correct. I think I'm more in favor of just throwing an error, having to specify the correct serializers isn't any more work than the scala / java api, and it's more explicit.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14340: [SPARK-16534][Streaming][Kafka] Add Python API support f...
Posted by jerryshao <gi...@git.apache.org>.
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/14340
Thanks a lot @koeninger for your review, I think it is not so flexible for Python API to achieve same functionalities as Java/Scala APIs, especially for things like extended class like as`ConsumerStrategy`. I think we could document this limitation to the user. Also I think current `ConsumerStrategy` could possibly cover most of the scenarios.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14340: [SPARK-16534][Streaming][Kafka] Add Python API support f...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14340
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14340: [SPARK-16534][Streaming][Kafka] Add Python API su...
Posted by jerryshao <gi...@git.apache.org>.
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/14340#discussion_r72176261
--- Diff: external/kafka-0-10/src/main/scala/org/apache/spark/streaming/kafka010/KafkaUtils.scala ---
@@ -177,3 +182,172 @@ object KafkaUtils extends Logging {
}
}
}
+
+private[kafka010] class KafkaUtilsPythonHelper extends Logging {
+ import KafkaUtilsPythonHelper._
+
+ def createDirectStream(
+ jssc: JavaStreamingContext,
+ locationStrategy: LocationStrategy,
+ consumerStrategy: ConsumerStrategy[Array[Byte], Array[Byte]]
+ ): JavaDStream[Array[Byte]] = {
+ fixKafkaParams(consumerStrategy.executorKafkaParams)
--- End diff --
If we did a defensive copy on the map first, how do we set it back to `ConsumerStrategy`, I'm not pretty sure how to achieve this.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14340: [SPARK-16534][Streaming][Kafka] Add Python API support f...
Posted by jerryshao <gi...@git.apache.org>.
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/14340
Jenkins, retest this please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14340: [SPARK-16534][Streaming][Kafka] Add Python API support f...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14340
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62870/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14340: [SPARK-16534][Streaming][Kafka] Add Python API support f...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14340
**[Test build #62795 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62795/consoleFull)** for PR 14340 at commit [`38a89f6`](https://github.com/apache/spark/commit/38a89f68dac408453ad8cdb871d2b64f23d758e1).
* This patch **fails PySpark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14340: [SPARK-16534][Streaming][Kafka] Add Python API support f...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14340
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14340: [SPARK-16534][Streaming][Kafka] Add Python API support f...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14340
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62871/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14340: [SPARK-16534][Streaming][Kafka] Add Python API support f...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14340
**[Test build #62795 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62795/consoleFull)** for PR 14340 at commit [`38a89f6`](https://github.com/apache/spark/commit/38a89f68dac408453ad8cdb871d2b64f23d758e1).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14340: [SPARK-16534][Streaming][Kafka] Add Python API su...
Posted by koeninger <gi...@git.apache.org>.
Github user koeninger commented on a diff in the pull request:
https://github.com/apache/spark/pull/14340#discussion_r72115899
--- Diff: examples/src/main/python/streaming/direct_kafka010_wordcount.py ---
@@ -0,0 +1,59 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""
+ Counts words in UTF8 encoded, '\n' delimited text directly received from Kafka in every 2 seconds.
+ Usage: direct_kafka_wordcount.py <broker_list> <topic>
+
+ To run this on your local machine, you need to setup Kafka and create a producer first, see
+ http://kafka.apache.org/documentation.html#quickstart
+
+ and then run the example
+ `$ bin/spark-submit --jars \
+ external/kafka-0-10-assembly/target/scala-*/spark-streaming-kafka-assembly-*.jar \
+ examples/src/main/python/streaming/direct_kafka_wordcount.py \
+ localhost:9092 test`
+"""
+from __future__ import print_function
+
+import sys
+
+from pyspark import SparkContext
+from pyspark.streaming import StreamingContext
+from pyspark.streaming.kafka010 import KafkaUtils, PreferBrokers, Subscribe
+
+if __name__ == "__main__":
+ if len(sys.argv) != 3:
+ print("Usage: direct_kafka010_wordcount.py <broker_list> <topic>", file=sys.stderr)
+ exit(-1)
+
+ sc = SparkContext(appName="PythonStreamingDirectKafkaWordCount")
+ ssc = StreamingContext(sc, 2)
+
+ brokers, topic = sys.argv[1:]
+ kafkaParams = {"bootstrap.servers": brokers, "group.id": "test"}
+ subscribe = Subscribe([topic], kafkaParams)
+ preferBrokers = PreferBrokers()
--- End diff --
Is it actually necessary to assign the result of subscribe / prefer brokers to local variables, or is this just a style thing?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14340: [SPARK-16534][Streaming][Kafka] Add Python API su...
Posted by jerryshao <gi...@git.apache.org>.
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/14340#discussion_r72175444
--- Diff: examples/src/main/python/streaming/direct_kafka010_wordcount.py ---
@@ -0,0 +1,59 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""
+ Counts words in UTF8 encoded, '\n' delimited text directly received from Kafka in every 2 seconds.
+ Usage: direct_kafka_wordcount.py <broker_list> <topic>
+
+ To run this on your local machine, you need to setup Kafka and create a producer first, see
+ http://kafka.apache.org/documentation.html#quickstart
+
+ and then run the example
+ `$ bin/spark-submit --jars \
+ external/kafka-0-10-assembly/target/scala-*/spark-streaming-kafka-assembly-*.jar \
+ examples/src/main/python/streaming/direct_kafka_wordcount.py \
+ localhost:9092 test`
+"""
+from __future__ import print_function
+
+import sys
+
+from pyspark import SparkContext
+from pyspark.streaming import StreamingContext
+from pyspark.streaming.kafka010 import KafkaUtils, PreferBrokers, Subscribe
+
+if __name__ == "__main__":
+ if len(sys.argv) != 3:
+ print("Usage: direct_kafka010_wordcount.py <broker_list> <topic>", file=sys.stderr)
+ exit(-1)
+
+ sc = SparkContext(appName="PythonStreamingDirectKafkaWordCount")
+ ssc = StreamingContext(sc, 2)
+
+ brokers, topic = sys.argv[1:]
+ kafkaParams = {"bootstrap.servers": brokers, "group.id": "test"}
+ subscribe = Subscribe([topic], kafkaParams)
+ preferBrokers = PreferBrokers()
--- End diff --
Here in python, it is mainly a style thing.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14340: [SPARK-16534][Streaming][Kafka] Add Python API su...
Posted by koeninger <gi...@git.apache.org>.
Github user koeninger commented on a diff in the pull request:
https://github.com/apache/spark/pull/14340#discussion_r72251359
--- Diff: external/kafka-0-10/src/main/scala/org/apache/spark/streaming/kafka010/KafkaUtils.scala ---
@@ -286,6 +285,30 @@ private[kafka010] class KafkaUtilsPythonHelper extends Logging {
val kafkaRDD = kafkaRDDs.head.asInstanceOf[KafkaRDD[_, _]]
kafkaRDD.offsetRanges.toSeq.asJava
}
+
+ def commitAsyncForKafkaDStream(dstream: DStream[_], offsetRanges: ju.List[OffsetRange]): Unit = {
+ val dstreams = new mutable.HashSet[DStream[_]]()
+
+ def visit(parent: DStream[_]): Unit = {
+ val parents = parent.dependencies
+ parents.filterNot(dstreams.contains).foreach { p =>
+ dstreams.add(p)
+ visit(p)
+ }
+ }
+ visit(dstream)
+
+ val kafkaDStreams = dstreams.filter(s => s.isInstanceOf[DirectKafkaInputDStream[_, _]])
+ require(
+ kafkaDStreams.size == 1,
+ "Cannot commit offset ranges to DirectKafkaInputDStream, as there may be multiple Kafka " +
--- End diff --
It's slightly odd to refer to private classes in an error message. But it may make more sense than referring to the CanCommitOffsets / HasOffsets interfaces ... from a python user point of view, which do you think is more likely to make sense?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14340: [SPARK-16534][Streaming][Kafka] Add Python API su...
Posted by jerryshao <gi...@git.apache.org>.
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/14340#discussion_r72192793
--- Diff: python/pyspark/streaming/kafka010.py ---
@@ -0,0 +1,370 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+from pyspark.rdd import RDD
+from pyspark.serializers import AutoBatchedSerializer, PickleSerializer
+from pyspark.streaming import DStream
+from pyspark.streaming.kafka import KafkaDStream, KafkaRDD, OffsetRange
+
+__all__ = ['Assign', 'KafkaConsumerRecord', 'KafkaUtils', 'PreferBrokers', 'PreferConsistent',
+ 'PreferFixed', 'Subscribe', 'SubscribePattern', 'TopicPartition', 'utf8_decoder']
+
+
+def utf8_decoder(s):
+ """ Decode the unicode as UTF-8 """
+ if s is None:
+ return None
+ return s.decode('utf-8')
+
+
+class KafkaUtils(object):
+
+ @staticmethod
+ def createDirectStream(ssc, locationStrategy, consumerStrategy,
+ keyDecoder=utf8_decoder, valueDecoder=utf8_decoder):
+ """
+ .. note:: Experimental
+
+ Create an input stream that directly pulls messages from Kafka 0.10 brokers with different
+ location strategy and consumer strategy.
+
+ This does not use Zookeeper to store offsets. The consumed offsets are tracked
+ by the stream itself. For interoperability with Kafka monitoring tools that depend on
+ Zookeeper, you have to update Kafka/Zookeeper yourself from the streaming application.
+ You can access the offsets used in each batch from the generated RDDs (see
+
+ To recover from driver failures, you have to enable checkpointing in the StreamingContext.
+ The information on consumed offset can be recovered from the checkpoint.
+ See the programming guide for details (constraints, etc.).
+
+ :param ssc: StreamingContext object,
+ :param locationStrategy: Strategy to schedule consumers for a given TopicPartition on an
+ executor.
+ :param consumerStrategy: Choices of how to create and configure underlying Kafka
+ Consumers on driver and executors.
+ :param keyDecoder: A function to decode key (default is utf8_decoder).
+ :param valueDecoder: A function to decode value (default is utf8_decoder).
+ :return: A DStream object.
+ """
+
+ helper = KafkaUtils._get_helper(ssc._sc)
+ ser = AutoBatchedSerializer(PickleSerializer())
+
+ jlocationStrategy = locationStrategy._jLocationStrategy(helper)
+ jconsumerStrategy = consumerStrategy._jConsumerStrategy(helper)
+
+ jstream = helper.createDirectStream(ssc._jssc, jlocationStrategy, jconsumerStrategy)
+
+ def func(m):
+ m._set_key_deserializer(keyDecoder)
+ m._set_value_deserializer(valueDecoder)
+ return m
+
+ stream = DStream(jstream, ssc, ser).map(func)
+
+ return KafkaDStream(stream._jdstream, ssc, stream._jrdd_deserializer)
+
+ @staticmethod
+ def createRDD(sc, kafkaParams, offsetRanges, locationStrategy,
+ keyDecoder=utf8_decoder, valueDecoder=utf8_decoder):
+ """
+ .. note:: Experimental
+
+ Create a Kafka RDD using offset ranges and location strategy.
+
+ :param sc: SparkContext object.
+ :param kafkaParams: Additional params for Kafka.
+ :param offsetRanges: list of offsetRange to specify topic:partition:[start, end) to consume.
+ :param locationStrategy: Strategy to schedule consumers for a given TopicPartition on an
+ executor.
+ :param keyDecoder: A function to decode key (default is utf8_decoder).
+ :param valueDecoder: A function to decode value (default is utf8_decoder).
+ :return: A RDD object.
+ """
+
+ if not isinstance(kafkaParams, dict):
+ raise TypeError("kafkaParams should be dict")
+
+ helper = KafkaUtils._get_helper(sc)
+ joffsetRanges = [o._jOffsetRange(helper) for o in offsetRanges]
+ jlocationStrategy = locationStrategy._jLocationStrategy(helper)
+
+ jrdd = helper.createRDD(sc._jsc, kafkaParams, joffsetRanges, jlocationStrategy)
+
+ def func(m):
+ m._set_key_deserializer(keyDecoder)
+ m._set_value_deserializer(valueDecoder)
+ return m
+
+ rdd = RDD(jrdd, sc).map(func)
+
+ return KafkaRDD(rdd._jrdd, sc, rdd._jrdd_deserializer)
+
+ @staticmethod
+ def _get_helper(sc):
+ try:
+ helper = sc._jvm.org.apache.spark.streaming.kafka010.KafkaUtilsPythonHelper()
+ KafkaRDD.set_helper(helper)
+ return helper
+ except TypeError as e:
+ if str(e) == "'JavaPackage' object is not callable":
+ KafkaUtils._printErrorMsg(sc)
+ raise
+
+ @staticmethod
+ def _printErrorMsg(sc):
+ print("""
+________________________________________________________________________________________________
+
+ Spark Streaming's Kafka libraries not found in class path. Try one of the following.
+
+ 1. Include the Kafka library and its dependencies with in the
+ spark-submit command as
+
+ $ bin/spark-submit --packages org.apache.spark:spark-streaming-kafka-0-10:%s ...
+
+ 2. Download the JAR of the artifact from Maven Central http://search.maven.org/,
+ Group Id = org.apache.spark, Artifact Id = spark-streaming-kafka-0-10-assembly, Version = %s.
+ Then, include the jar in the spark-submit command as
+
+ $ bin/spark-submit --jars <spark-streaming-kafka-0-10-assembly.jar> ...
+
+________________________________________________________________________________________________
+
+""" % (sc.version, sc.version))
+
+
+class LocationStrategy(object):
+ """
+ .. note:: Experimental
+
+ A python wrapper of Scala LocationStrategy.
+ """
+
+ def _jLocationStrategy(self, helper):
+ pass
+
+
+class PreferBrokers(LocationStrategy):
+ """
+ .. note:: Experimental
+
+ Use this only if your executors are on the same nodes as your kafka brokers.
+
+ """
+ def _jLocationStrategy(self, helper):
+ return helper.createPreferBrokers()
+
+
+class PreferConsistent(LocationStrategy):
+ """
+ .. note:: Experimental
+
+ Use this in most cases, it will consistently distribute partitions across all executors.
+ """
+
+ def _jLocationStrategy(self, helper):
+ return helper.createPreferConsistent()
+
+
+class PreferFixed(LocationStrategy):
+ """
+ .. note:: Experimental
+
+ Use this to place particular TopicPartitions on particular hosts if your load is uneven. Any
+ TopicPartition not specified in the map will use a consistent location.
+ """
+
+ def __init__(self, hostMap):
+ """
+ Python wrapper of Scala PreferFixed.
+
+ :param hostMap: A dict of TopicPartition to hostname.
+ """
+ self.hostMap = hostMap
+
+ def _jLocationStrategy(self, helper):
+ jhostMap = dict([(k._jTopicPartition(helper), v) for (k, v) in self.hostMap.items()])
+ return helper.createPreferFixed(jhostMap)
+
+
+class ConsumerStrategy(object):
+ """
+ .. note:: Experimental
+
+ A python wrapper of Scala ConsumerStrategy.
+ """
+
+ def _jConsumerStrategy(self, helper):
+ pass
+
+
+class Subscribe(ConsumerStrategy):
+ """
+ .. note:: Experimental
+
+ Subscribe to a collection of topics.
+ """
+
+ def __init__(self, topics, kafkaParams, offsets=None):
+ """
+ Subscribe to a collection of topics.
+
+ :param topics: List of topics to subscribe.
+ :param kafkaParams: Kafka parameters.
+ :param offsets: offsets to begin at on initial startup. If no offset is given for a
+ TopicPartition, the committed offset (if applicable) or kafka param
+ auto.offset.reset will be used.
+ """
+ self.topics = set(topics)
+ self.kafkaParams = kafkaParams
+ self.offsets = dict() if offsets is None else offsets
+
+ def _jConsumerStrategy(self, helper):
+ jOffsets = dict([k._jTopicPartition(helper), v] for (k, v) in self.offsets.items())
+ return helper.createSubscribe(self.topics, self.kafkaParams, jOffsets)
+
+
+class SubscribePattern(ConsumerStrategy):
+ """
+ .. note:: Experimental
+
+ Subscribe to all topics matching specified pattern to get dynamically assigned partitions.
+ """
+
+ def __init__(self, pattern, kafkaParams, offsets=None):
+ """
+ Subscribe to all topics matching specified pattern to get dynamically assigned partitions.
+
+ :param pattern: pattern to subscribe to.
+ :param kafkaParams: Kafka parameters.
+ :param offsets: offsets to begin at on initial startup. If no offset is given for a
+ TopicPartition, the committed offset (if applicable) or kafka param
+ auto.offset.reset will be used.
+ """
+ self.pattern = pattern
+ self.kafkaParams = kafkaParams
+ self.offsets = dict() if offsets is None else offsets
+
+ def _jConsumerStrategy(self, helper):
+ jOffsets = dict([k._jTopicPartition(helper), v] for (k, v) in self.offsets.items())
+ return helper.createSubscribePattern(self.pattern, self.kafkaParams, jOffsets)
+
+
+class Assign(ConsumerStrategy):
+ """
+ .. note:: Experimental
+
+ Assign a fixed collection of TopicPartitions.
+ """
+
+ def __init__(self, topicPartitions, kafkaParams, offsets=None):
+ """
+ Assign a fixed collection of TopicPartitions.
+
+ :param topicPartitions: List of TopicPartitions to assign.
+ :param kafkaParams: kafka parameters.
+ :param offsets: offsets to begin at on initial startup. If no offset is given for a
+ TopicPartition, the committed offset (if applicable) or kafka param
+ auto.offset.reset will be used.
+ """
+ self.topicPartitions = set(topicPartitions)
+ self.kafkaParams = kafkaParams
+ self.offsets = dict() if offsets is None else offsets
+
+ def _jConsumerStrategy(self, helper):
+ jTopicPartitions = [i._jTopicPartition(helper) for i in self.topicPartitions]
+ jOffsets = dict([k._jTopicPartition(helper), v] for (k, v) in self.offsets.items())
+ return helper.createAssign(set(jTopicPartitions), self.kafkaParams, jOffsets)
+
+
+class TopicPartition(object):
+ """
+ Represents a specific top and partition for Kafka.
+ """
+
+ def __init__(self, topic, partition):
+ """
+ Create a Python TopicPartition to map to the Java related object
+ :param topic: Kafka topic name.
+ :param partition: Kafka partition id.
+ """
+ self._topic = topic
+ self._partition = partition
+
+ def _jTopicPartition(self, helper):
+ return helper.createTopicPartition(self._topic, self._partition)
+
+ def __eq__(self, other):
+ if isinstance(other, self.__class__):
+ return (self._topic == other._topic
+ and self._partition == other._partition)
+ else:
+ return False
+
+ def __ne__(self, other):
+ return not self.__eq__(other)
+
+ def __hash__(self):
+ return hash(self._topic) ^ hash(self._partition)
+
+
+class KafkaConsumerRecord(object):
+ """
+ Kafka consumer record fetch from Kafka brokers, including metadata information and message.
+ """
+
+ def __init__(self, topic, partition, offset, timestamp, timestampType, checksum,
+ serializedKeySize, serializedValueSize, key, value):
+ self.topic = topic
+ self.partition = partition
+ self.offset = offset
+ self.timestamp = timestamp
+ self.timestampType = timestampType
+ self.checksum = checksum
+ self.serializedKeySize = serializedKeySize
+ self.serializedValueSize = serializedValueSize
+ self._rawKey = key
+ self._rawValue = value
+ self._keyDecoder = utf8_decoder
+ self._valueDecoder = utf8_decoder
+
+ def __str__(self):
+ return "Kafka ConsumerRecords(topic: %s, partition: %d, offset: %d, timestamp: %d, " \
--- End diff --
Sorry, I will fix it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14340: [SPARK-16534][Streaming][Kafka] Add Python API support f...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14340
**[Test build #62877 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62877/consoleFull)** for PR 14340 at commit [`d155c8d`](https://github.com/apache/spark/commit/d155c8d819a7fc2f26c1a91cafc37c6d70c5e89b).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14340: [SPARK-16534][Streaming][Kafka] Add Python API support f...
Posted by jerryshao <gi...@git.apache.org>.
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/14340
Thanks a lot @rxin for your comments, let me close it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14340: [SPARK-16534][Streaming][Kafka] Add Python API support f...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14340
**[Test build #62877 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62877/consoleFull)** for PR 14340 at commit [`d155c8d`](https://github.com/apache/spark/commit/d155c8d819a7fc2f26c1a91cafc37c6d70c5e89b).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14340: [SPARK-16534][Streaming][Kafka] Add Python API support f...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14340
**[Test build #62871 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62871/consoleFull)** for PR 14340 at commit [`9e6895a`](https://github.com/apache/spark/commit/9e6895a47ef403ef76ef605829bdaa8b90e5a2b7).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14340: [SPARK-16534][Streaming][Kafka] Add Python API support f...
Posted by maver1ck <gi...@git.apache.org>.
Github user maver1ck commented on the issue:
https://github.com/apache/spark/pull/14340
@rxin,
Production streaming jobs can be written in Python and me and my company are example.
I wrote a little bit more in Jira. I think it's better place for discussion.
https://issues.apache.org/jira/browse/SPARK-16534?focusedCommentId=15491107&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15491107
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14340: [SPARK-16534][Streaming][Kafka] Add Python API support f...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14340
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62877/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14340: [SPARK-16534][Streaming][Kafka] Add Python API support f...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14340
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62822/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14340: [SPARK-16534][Streaming][Kafka] Add Python API support f...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14340
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14340: [SPARK-16534][Streaming][Kafka] Add Python API support f...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on the issue:
https://github.com/apache/spark/pull/14340
So I would like to -1 this patch. I think it's been a mistake to support dstream in Python -- yes it satisfies a checkbox and Spark could claim there's support for streaming in Python. However, the tooling and maturity for working with streaming data (both in Spark and the more broad ecosystem) is simply not there. It is a big baggage to maintain, and creates a the wrong impression that production streaming jobs can be written in Python.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14340: [SPARK-16534][Streaming][Kafka] Add Python API support f...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14340
**[Test build #62807 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62807/consoleFull)** for PR 14340 at commit [`38a89f6`](https://github.com/apache/spark/commit/38a89f68dac408453ad8cdb871d2b64f23d758e1).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14340: [SPARK-16534][Streaming][Kafka] Add Python API su...
Posted by koeninger <gi...@git.apache.org>.
Github user koeninger commented on a diff in the pull request:
https://github.com/apache/spark/pull/14340#discussion_r72118143
--- Diff: external/kafka-0-10/src/main/scala/org/apache/spark/streaming/kafka010/KafkaUtils.scala ---
@@ -177,3 +182,172 @@ object KafkaUtils extends Logging {
}
}
}
+
+private[kafka010] class KafkaUtilsPythonHelper extends Logging {
+ import KafkaUtilsPythonHelper._
+
+ def createDirectStream(
+ jssc: JavaStreamingContext,
+ locationStrategy: LocationStrategy,
+ consumerStrategy: ConsumerStrategy[Array[Byte], Array[Byte]]
+ ): JavaDStream[Array[Byte]] = {
+ fixKafkaParams(consumerStrategy.executorKafkaParams)
--- End diff --
executorKafkaParams is a def, not a val. fixKafkaParams is making destructive changes to the returned map. I think this should make a defensive copy first.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14340: [SPARK-16534][Streaming][Kafka] Add Python API su...
Posted by jerryshao <gi...@git.apache.org>.
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/14340#discussion_r72177558
--- Diff: external/kafka-0-10/src/main/scala/org/apache/spark/streaming/kafka010/KafkaUtils.scala ---
@@ -177,3 +182,172 @@ object KafkaUtils extends Logging {
}
}
}
+
+private[kafka010] class KafkaUtilsPythonHelper extends Logging {
+ import KafkaUtilsPythonHelper._
+
+ def createDirectStream(
+ jssc: JavaStreamingContext,
+ locationStrategy: LocationStrategy,
+ consumerStrategy: ConsumerStrategy[Array[Byte], Array[Byte]]
+ ): JavaDStream[Array[Byte]] = {
+ fixKafkaParams(consumerStrategy.executorKafkaParams)
--- End diff --
I see, I will move this into python, that will be more easier to do.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14340: [SPARK-16534][Streaming][Kafka] Add Python API su...
Posted by koeninger <gi...@git.apache.org>.
Github user koeninger commented on a diff in the pull request:
https://github.com/apache/spark/pull/14340#discussion_r72123421
--- Diff: python/pyspark/streaming/kafka010.py ---
@@ -0,0 +1,370 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+from pyspark.rdd import RDD
+from pyspark.serializers import AutoBatchedSerializer, PickleSerializer
+from pyspark.streaming import DStream
+from pyspark.streaming.kafka import KafkaDStream, KafkaRDD, OffsetRange
+
+__all__ = ['Assign', 'KafkaConsumerRecord', 'KafkaUtils', 'PreferBrokers', 'PreferConsistent',
+ 'PreferFixed', 'Subscribe', 'SubscribePattern', 'TopicPartition', 'utf8_decoder']
+
+
+def utf8_decoder(s):
+ """ Decode the unicode as UTF-8 """
+ if s is None:
+ return None
+ return s.decode('utf-8')
+
+
+class KafkaUtils(object):
+
+ @staticmethod
+ def createDirectStream(ssc, locationStrategy, consumerStrategy,
+ keyDecoder=utf8_decoder, valueDecoder=utf8_decoder):
+ """
+ .. note:: Experimental
+
+ Create an input stream that directly pulls messages from Kafka 0.10 brokers with different
+ location strategy and consumer strategy.
+
+ This does not use Zookeeper to store offsets. The consumed offsets are tracked
+ by the stream itself. For interoperability with Kafka monitoring tools that depend on
+ Zookeeper, you have to update Kafka/Zookeeper yourself from the streaming application.
+ You can access the offsets used in each batch from the generated RDDs (see
+
+ To recover from driver failures, you have to enable checkpointing in the StreamingContext.
+ The information on consumed offset can be recovered from the checkpoint.
+ See the programming guide for details (constraints, etc.).
+
+ :param ssc: StreamingContext object,
+ :param locationStrategy: Strategy to schedule consumers for a given TopicPartition on an
+ executor.
+ :param consumerStrategy: Choices of how to create and configure underlying Kafka
+ Consumers on driver and executors.
+ :param keyDecoder: A function to decode key (default is utf8_decoder).
+ :param valueDecoder: A function to decode value (default is utf8_decoder).
+ :return: A DStream object.
+ """
+
+ helper = KafkaUtils._get_helper(ssc._sc)
+ ser = AutoBatchedSerializer(PickleSerializer())
+
+ jlocationStrategy = locationStrategy._jLocationStrategy(helper)
+ jconsumerStrategy = consumerStrategy._jConsumerStrategy(helper)
+
+ jstream = helper.createDirectStream(ssc._jssc, jlocationStrategy, jconsumerStrategy)
+
+ def func(m):
+ m._set_key_deserializer(keyDecoder)
+ m._set_value_deserializer(valueDecoder)
+ return m
+
+ stream = DStream(jstream, ssc, ser).map(func)
+
+ return KafkaDStream(stream._jdstream, ssc, stream._jrdd_deserializer)
+
+ @staticmethod
+ def createRDD(sc, kafkaParams, offsetRanges, locationStrategy,
+ keyDecoder=utf8_decoder, valueDecoder=utf8_decoder):
+ """
+ .. note:: Experimental
+
+ Create a Kafka RDD using offset ranges and location strategy.
+
+ :param sc: SparkContext object.
+ :param kafkaParams: Additional params for Kafka.
+ :param offsetRanges: list of offsetRange to specify topic:partition:[start, end) to consume.
+ :param locationStrategy: Strategy to schedule consumers for a given TopicPartition on an
+ executor.
+ :param keyDecoder: A function to decode key (default is utf8_decoder).
+ :param valueDecoder: A function to decode value (default is utf8_decoder).
+ :return: A RDD object.
+ """
+
+ if not isinstance(kafkaParams, dict):
+ raise TypeError("kafkaParams should be dict")
+
+ helper = KafkaUtils._get_helper(sc)
+ joffsetRanges = [o._jOffsetRange(helper) for o in offsetRanges]
+ jlocationStrategy = locationStrategy._jLocationStrategy(helper)
+
+ jrdd = helper.createRDD(sc._jsc, kafkaParams, joffsetRanges, jlocationStrategy)
+
+ def func(m):
+ m._set_key_deserializer(keyDecoder)
+ m._set_value_deserializer(valueDecoder)
+ return m
+
+ rdd = RDD(jrdd, sc).map(func)
+
+ return KafkaRDD(rdd._jrdd, sc, rdd._jrdd_deserializer)
+
+ @staticmethod
+ def _get_helper(sc):
+ try:
+ helper = sc._jvm.org.apache.spark.streaming.kafka010.KafkaUtilsPythonHelper()
+ KafkaRDD.set_helper(helper)
+ return helper
+ except TypeError as e:
+ if str(e) == "'JavaPackage' object is not callable":
+ KafkaUtils._printErrorMsg(sc)
+ raise
+
+ @staticmethod
+ def _printErrorMsg(sc):
+ print("""
+________________________________________________________________________________________________
+
+ Spark Streaming's Kafka libraries not found in class path. Try one of the following.
+
+ 1. Include the Kafka library and its dependencies with in the
+ spark-submit command as
+
+ $ bin/spark-submit --packages org.apache.spark:spark-streaming-kafka-0-10:%s ...
+
+ 2. Download the JAR of the artifact from Maven Central http://search.maven.org/,
+ Group Id = org.apache.spark, Artifact Id = spark-streaming-kafka-0-10-assembly, Version = %s.
+ Then, include the jar in the spark-submit command as
+
+ $ bin/spark-submit --jars <spark-streaming-kafka-0-10-assembly.jar> ...
+
+________________________________________________________________________________________________
+
+""" % (sc.version, sc.version))
+
+
+class LocationStrategy(object):
+ """
+ .. note:: Experimental
+
+ A python wrapper of Scala LocationStrategy.
+ """
+
+ def _jLocationStrategy(self, helper):
+ pass
+
+
+class PreferBrokers(LocationStrategy):
+ """
+ .. note:: Experimental
+
+ Use this only if your executors are on the same nodes as your kafka brokers.
+
+ """
+ def _jLocationStrategy(self, helper):
+ return helper.createPreferBrokers()
+
+
+class PreferConsistent(LocationStrategy):
+ """
+ .. note:: Experimental
+
+ Use this in most cases, it will consistently distribute partitions across all executors.
+ """
+
+ def _jLocationStrategy(self, helper):
+ return helper.createPreferConsistent()
+
+
+class PreferFixed(LocationStrategy):
+ """
+ .. note:: Experimental
+
+ Use this to place particular TopicPartitions on particular hosts if your load is uneven. Any
+ TopicPartition not specified in the map will use a consistent location.
+ """
+
+ def __init__(self, hostMap):
+ """
+ Python wrapper of Scala PreferFixed.
+
+ :param hostMap: A dict of TopicPartition to hostname.
+ """
+ self.hostMap = hostMap
+
+ def _jLocationStrategy(self, helper):
+ jhostMap = dict([(k._jTopicPartition(helper), v) for (k, v) in self.hostMap.items()])
+ return helper.createPreferFixed(jhostMap)
+
+
+class ConsumerStrategy(object):
+ """
+ .. note:: Experimental
+
+ A python wrapper of Scala ConsumerStrategy.
+ """
+
+ def _jConsumerStrategy(self, helper):
+ pass
+
+
+class Subscribe(ConsumerStrategy):
+ """
+ .. note:: Experimental
+
+ Subscribe to a collection of topics.
+ """
+
+ def __init__(self, topics, kafkaParams, offsets=None):
+ """
+ Subscribe to a collection of topics.
+
+ :param topics: List of topics to subscribe.
+ :param kafkaParams: Kafka parameters.
+ :param offsets: offsets to begin at on initial startup. If no offset is given for a
+ TopicPartition, the committed offset (if applicable) or kafka param
+ auto.offset.reset will be used.
+ """
+ self.topics = set(topics)
+ self.kafkaParams = kafkaParams
+ self.offsets = dict() if offsets is None else offsets
+
+ def _jConsumerStrategy(self, helper):
+ jOffsets = dict([k._jTopicPartition(helper), v] for (k, v) in self.offsets.items())
+ return helper.createSubscribe(self.topics, self.kafkaParams, jOffsets)
+
+
+class SubscribePattern(ConsumerStrategy):
+ """
+ .. note:: Experimental
+
+ Subscribe to all topics matching specified pattern to get dynamically assigned partitions.
+ """
+
+ def __init__(self, pattern, kafkaParams, offsets=None):
+ """
+ Subscribe to all topics matching specified pattern to get dynamically assigned partitions.
+
+ :param pattern: pattern to subscribe to.
+ :param kafkaParams: Kafka parameters.
+ :param offsets: offsets to begin at on initial startup. If no offset is given for a
+ TopicPartition, the committed offset (if applicable) or kafka param
+ auto.offset.reset will be used.
+ """
+ self.pattern = pattern
+ self.kafkaParams = kafkaParams
+ self.offsets = dict() if offsets is None else offsets
+
+ def _jConsumerStrategy(self, helper):
+ jOffsets = dict([k._jTopicPartition(helper), v] for (k, v) in self.offsets.items())
+ return helper.createSubscribePattern(self.pattern, self.kafkaParams, jOffsets)
+
+
+class Assign(ConsumerStrategy):
+ """
+ .. note:: Experimental
+
+ Assign a fixed collection of TopicPartitions.
+ """
+
+ def __init__(self, topicPartitions, kafkaParams, offsets=None):
+ """
+ Assign a fixed collection of TopicPartitions.
+
+ :param topicPartitions: List of TopicPartitions to assign.
+ :param kafkaParams: kafka parameters.
+ :param offsets: offsets to begin at on initial startup. If no offset is given for a
+ TopicPartition, the committed offset (if applicable) or kafka param
+ auto.offset.reset will be used.
+ """
+ self.topicPartitions = set(topicPartitions)
+ self.kafkaParams = kafkaParams
+ self.offsets = dict() if offsets is None else offsets
+
+ def _jConsumerStrategy(self, helper):
+ jTopicPartitions = [i._jTopicPartition(helper) for i in self.topicPartitions]
+ jOffsets = dict([k._jTopicPartition(helper), v] for (k, v) in self.offsets.items())
+ return helper.createAssign(set(jTopicPartitions), self.kafkaParams, jOffsets)
+
+
+class TopicPartition(object):
+ """
+ Represents a specific top and partition for Kafka.
+ """
+
+ def __init__(self, topic, partition):
+ """
+ Create a Python TopicPartition to map to the Java related object
+ :param topic: Kafka topic name.
+ :param partition: Kafka partition id.
+ """
+ self._topic = topic
+ self._partition = partition
+
+ def _jTopicPartition(self, helper):
+ return helper.createTopicPartition(self._topic, self._partition)
+
+ def __eq__(self, other):
+ if isinstance(other, self.__class__):
+ return (self._topic == other._topic
+ and self._partition == other._partition)
+ else:
+ return False
+
+ def __ne__(self, other):
+ return not self.__eq__(other)
+
+ def __hash__(self):
+ return hash(self._topic) ^ hash(self._partition)
+
+
+class KafkaConsumerRecord(object):
+ """
+ Kafka consumer record fetch from Kafka brokers, including metadata information and message.
+ """
+
+ def __init__(self, topic, partition, offset, timestamp, timestampType, checksum,
+ serializedKeySize, serializedValueSize, key, value):
+ self.topic = topic
+ self.partition = partition
+ self.offset = offset
+ self.timestamp = timestamp
+ self.timestampType = timestampType
+ self.checksum = checksum
+ self.serializedKeySize = serializedKeySize
+ self.serializedValueSize = serializedValueSize
+ self._rawKey = key
+ self._rawValue = value
+ self._keyDecoder = utf8_decoder
+ self._valueDecoder = utf8_decoder
+
+ def __str__(self):
+ return "Kafka ConsumerRecords(topic: %s, partition: %d, offset: %d, timestamp: %d, " \
--- End diff --
This is a single consumer record, not plural, right?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14340: [SPARK-16534][Streaming][Kafka] Add Python API support f...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14340
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62784/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14340: [SPARK-16534][Streaming][Kafka] Add Python API support f...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14340
**[Test build #62807 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62807/consoleFull)** for PR 14340 at commit [`38a89f6`](https://github.com/apache/spark/commit/38a89f68dac408453ad8cdb871d2b64f23d758e1).
* This patch **fails PySpark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14340: [SPARK-16534][Streaming][Kafka] Add Python API support f...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14340
**[Test build #62784 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62784/consoleFull)** for PR 14340 at commit [`dcca337`](https://github.com/apache/spark/commit/dcca3371da0c73bbcadb4331a2869e0303bd8ae8).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14340: [SPARK-16534][Streaming][Kafka] Add Python API support f...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14340
**[Test build #62784 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62784/consoleFull)** for PR 14340 at commit [`dcca337`](https://github.com/apache/spark/commit/dcca3371da0c73bbcadb4331a2869e0303bd8ae8).
* This patch **fails PySpark unit tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `class Kafka010StreamTests(PySparkStreamingTestCase):`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14340: [SPARK-16534][Streaming][Kafka] Add Python API support f...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14340
**[Test build #62822 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62822/consoleFull)** for PR 14340 at commit [`beb5c1c`](https://github.com/apache/spark/commit/beb5c1cd72a24e0b564ce4a936296f22a552a99e).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14340: [SPARK-16534][Streaming][Kafka] Add Python API su...
Posted by jerryshao <gi...@git.apache.org>.
Github user jerryshao closed the pull request at:
https://github.com/apache/spark/pull/14340
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14340: [SPARK-16534][Streaming][Kafka] Add Python API support f...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14340
**[Test build #62822 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62822/consoleFull)** for PR 14340 at commit [`beb5c1c`](https://github.com/apache/spark/commit/beb5c1cd72a24e0b564ce4a936296f22a552a99e).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14340: [SPARK-16534][Streaming][Kafka] Add Python API support f...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14340
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14340: [SPARK-16534][Streaming][Kafka] Add Python API support f...
Posted by koeninger <gi...@git.apache.org>.
Github user koeninger commented on the issue:
https://github.com/apache/spark/pull/14340
@jerryshao agreed that this is probably as good as possible for the Python api and the limitations make sense
Generally LGTM, the only remaining comments I had were minor wording things.
Thanks for working on this.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14340: [SPARK-16534][Streaming][Kafka] Add Python API support f...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14340
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62807/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14340: [SPARK-16534][Streaming][Kafka] Add Python API su...
Posted by jerryshao <gi...@git.apache.org>.
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/14340#discussion_r72177164
--- Diff: external/kafka-0-10/src/main/scala/org/apache/spark/streaming/kafka010/KafkaUtils.scala ---
@@ -177,3 +182,172 @@ object KafkaUtils extends Logging {
}
}
}
+
+private[kafka010] class KafkaUtilsPythonHelper extends Logging {
+ import KafkaUtilsPythonHelper._
+
+ def createDirectStream(
+ jssc: JavaStreamingContext,
+ locationStrategy: LocationStrategy,
+ consumerStrategy: ConsumerStrategy[Array[Byte], Array[Byte]]
+ ): JavaDStream[Array[Byte]] = {
+ fixKafkaParams(consumerStrategy.executorKafkaParams)
+ val stream = KafkaUtils.createDirectStream(jssc.ssc, locationStrategy, consumerStrategy)
+ .map { r =>
+ PythonConsumerRecord(r.topic(), r.partition(), r.offset(), r.timestamp(),
+ r.timestampType().toString, r.checksum(), r.serializedKeySize(), r.serializedValueSize(),
+ r.key(), r.value())
+ }.mapPartitions(picklerIterator)
+ new JavaDStream(stream)
+ }
+
+ def createRDD(
+ jsc: JavaSparkContext,
+ kafkaParams: ju.Map[String, Object],
+ offsetRanges: ju.List[OffsetRange],
+ locationStrategy: LocationStrategy
+ ): JavaRDD[Array[Byte]] = {
+ fixKafkaParams(kafkaParams)
+ val rdd = KafkaUtils.createRDD[Array[Byte], Array[Byte]](
+ jsc.sc,
+ kafkaParams,
+ offsetRanges.toArray(new Array[OffsetRange](offsetRanges.size())),
+ locationStrategy)
+ .map { r =>
+ PythonConsumerRecord(r.topic(), r.partition(), r.offset(), r.timestamp(),
+ r.timestampType().toString, r.checksum(), r.serializedKeySize(),
+ r.serializedValueSize(), r.key(), r.value())
+ }.mapPartitions(picklerIterator)
+ new JavaRDD(rdd)
+ }
+
+ private def fixKafkaParams(kafkaParams: ju.Map[String, Object]): Unit = {
+ val decoder = classOf[ByteArrayDeserializer].getCanonicalName
+
+ val keyDecoder = kafkaParams.get(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG)
+ if (keyDecoder != null && keyDecoder != decoder) {
+ logWarning(s"${ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG} $keyDecoder is not supported " +
+ s"for python Kafka API, overriding with $decoder")
+ }
+ kafkaParams.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, decoder)
+
+ val valueDecoder = kafkaParams.get(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG)
+ if (valueDecoder != null && valueDecoder != decoder) {
+ logWarning(s"${ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG} $valueDecoder is not " +
+ s"supported for python Kafka API, overriding with $decoder")
+ }
+ kafkaParams.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, decoder)
+ }
+
+ // Helper functions to convert Python object to Java object
+ def createOffsetRange(topic: String, partition: JInt, fromOffset: JLong, untilOffset: JLong
+ ): OffsetRange = OffsetRange.create(topic, partition, fromOffset, untilOffset)
+
+ def createPreferBrokers(): LocationStrategy = LocationStrategies.PreferBrokers
+
+ def createPreferConsistent(): LocationStrategy = LocationStrategies.PreferConsistent
+
+ def createPreferFixed(hostMap: ju.Map[TopicPartition, String]): LocationStrategy = {
+ LocationStrategies.PreferFixed(hostMap)
+ }
+
+ def createTopicPartition(topic: String, partition: JInt): TopicPartition =
+ new TopicPartition(topic, partition)
+
+ def createSubscribe(
+ topics: ju.Set[String],
+ kafkaParams: ju.Map[String, Object],
+ offsets: ju.Map[TopicPartition, JLong]): ConsumerStrategy[Array[Byte], Array[Byte]] =
+ ConsumerStrategies.Subscribe(topics, kafkaParams, offsets)
+
+ def createSubscribePattern(
+ pattern: String,
+ kafkaParams: ju.Map[String, Object],
+ offsets: ju.Map[TopicPartition, JLong]): ConsumerStrategy[Array[Byte], Array[Byte]] = {
+ ConsumerStrategies.SubscribePattern(ju.regex.Pattern.compile(pattern), kafkaParams, offsets)
+ }
+
+ def createAssign(
+ topicPartitions: ju.Set[TopicPartition],
+ kafkaParams: ju.Map[String, Object],
+ offsets: ju.Map[TopicPartition, JLong]): ConsumerStrategy[Array[Byte], Array[Byte]] = {
+ ConsumerStrategies.Assign(topicPartitions, kafkaParams, offsets)
+ }
+
+ def offsetRangesOfKafkaRDD(rdd: RDD[_]): ju.List[OffsetRange] = {
--- End diff --
Sure, I will also add this.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14340: [SPARK-16534][Streaming][Kafka] Add Python API support f...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14340
**[Test build #62870 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62870/consoleFull)** for PR 14340 at commit [`828051a`](https://github.com/apache/spark/commit/828051a02d5051c2e37150167df12660184beb18).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14340: [SPARK-16534][Streaming][Kafka] Add Python API support f...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14340
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14340: [SPARK-16534][Streaming][Kafka] Add Python API su...
Posted by koeninger <gi...@git.apache.org>.
Github user koeninger commented on a diff in the pull request:
https://github.com/apache/spark/pull/14340#discussion_r72122274
--- Diff: python/pyspark/streaming/kafka010.py ---
@@ -0,0 +1,370 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+from pyspark.rdd import RDD
+from pyspark.serializers import AutoBatchedSerializer, PickleSerializer
+from pyspark.streaming import DStream
+from pyspark.streaming.kafka import KafkaDStream, KafkaRDD, OffsetRange
+
+__all__ = ['Assign', 'KafkaConsumerRecord', 'KafkaUtils', 'PreferBrokers', 'PreferConsistent',
+ 'PreferFixed', 'Subscribe', 'SubscribePattern', 'TopicPartition', 'utf8_decoder']
+
+
+def utf8_decoder(s):
+ """ Decode the unicode as UTF-8 """
+ if s is None:
+ return None
+ return s.decode('utf-8')
+
+
+class KafkaUtils(object):
+
+ @staticmethod
+ def createDirectStream(ssc, locationStrategy, consumerStrategy,
+ keyDecoder=utf8_decoder, valueDecoder=utf8_decoder):
+ """
+ .. note:: Experimental
+
+ Create an input stream that directly pulls messages from Kafka 0.10 brokers with different
+ location strategy and consumer strategy.
+
+ This does not use Zookeeper to store offsets. The consumed offsets are tracked
+ by the stream itself. For interoperability with Kafka monitoring tools that depend on
+ Zookeeper, you have to update Kafka/Zookeeper yourself from the streaming application.
+ You can access the offsets used in each batch from the generated RDDs (see
+
+ To recover from driver failures, you have to enable checkpointing in the StreamingContext.
+ The information on consumed offset can be recovered from the checkpoint.
+ See the programming guide for details (constraints, etc.).
+
+ :param ssc: StreamingContext object,
+ :param locationStrategy: Strategy to schedule consumers for a given TopicPartition on an
--- End diff --
The scala / javadoc has a note on what the most common option to pass in is (preferConsistent and subscribe), probably worth adding here
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14340: [SPARK-16534][Streaming][Kafka] Add Python API support f...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14340
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14340: [SPARK-16534][Streaming][Kafka] Add Python API su...
Posted by koeninger <gi...@git.apache.org>.
Github user koeninger commented on a diff in the pull request:
https://github.com/apache/spark/pull/14340#discussion_r72121957
--- Diff: python/pyspark/streaming/kafka010.py ---
@@ -0,0 +1,370 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+from pyspark.rdd import RDD
+from pyspark.serializers import AutoBatchedSerializer, PickleSerializer
+from pyspark.streaming import DStream
+from pyspark.streaming.kafka import KafkaDStream, KafkaRDD, OffsetRange
+
+__all__ = ['Assign', 'KafkaConsumerRecord', 'KafkaUtils', 'PreferBrokers', 'PreferConsistent',
+ 'PreferFixed', 'Subscribe', 'SubscribePattern', 'TopicPartition', 'utf8_decoder']
+
+
+def utf8_decoder(s):
+ """ Decode the unicode as UTF-8 """
+ if s is None:
+ return None
+ return s.decode('utf-8')
+
+
+class KafkaUtils(object):
+
+ @staticmethod
+ def createDirectStream(ssc, locationStrategy, consumerStrategy,
+ keyDecoder=utf8_decoder, valueDecoder=utf8_decoder):
+ """
+ .. note:: Experimental
+
+ Create an input stream that directly pulls messages from Kafka 0.10 brokers with different
+ location strategy and consumer strategy.
+
+ This does not use Zookeeper to store offsets. The consumed offsets are tracked
--- End diff --
I don't think this verbiage is as relevant to Kafka 0.10, since the kafka managed api for offsets uses kafka topics, not zookeeper.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org