You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Jinxia Liu (JIRA)" <ji...@apache.org> on 2016/06/15 02:29:37 UTC
[jira] [Comment Edited] (SPARK-12177) Update KafkaDStreams to new
Kafka 0.10 Consumer API
[ https://issues.apache.org/jira/browse/SPARK-12177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15331007#comment-15331007 ]
Jinxia Liu edited comment on SPARK-12177 at 6/15/16 2:29 AM:
-------------------------------------------------------------
[~cody@koeninger.org] thanks for contributing the connector for kafka0.9 and kafka0.10.
I used your kafka0.10 connector and ran into some problems, would you mind looking at them?
1. when build using "mvn clean package", there is error about not passing the test case in DirectKafkaStreamSuite:
offset recovery *** FAILED ***
The code passed to eventually never returned normally. Attempted 196 times over 10.031047939 seconds. Last failure message: 55 did not equal 210. (DirectKafkaStreamSuite.scala:337)
2. another problem is(with kafka0.9 connector as well), can we add a wrapper, something like CreateDirectKafkaStream as in kafka0.8 connector, to wrap up the DirectKafkaStream constructor?
The benefit is that user does not need to know the kafka consumer APIs, in order to use the connector.
E.g.: the kafka consumer in the connector gets assigned a collection of TopicPartition, in most cases, all the partitions for given topic, if no wrapper, user needs to exploit the kafka consumer API to first retrieve the partitionInfo. Using the wrapper, user only needs to provide the topics, and such info can be passed to consumer inside the wrapper without the users knowledge.
was (Author: jinxliu@ebay.com):
[~cody@koeninger.org] thanks for contributing the connector for kafka0.9 and kafka0.10.
I used your kafka0.10 connector and ran into some problems, would you mind looking at them?
1. when build using "mvn clean package", there is error about not passing the test case in DirectKafkaStreamSuite:
offset recovery *** FAILED ***
The code passed to eventually never returned normally. Attempted 196 times over 10.031047939 seconds. Last failure message: 55 did not equal 210. (DirectKafkaStreamSuite.scala:337)
2. another problem is(with kafka0.9 connector as well), can we add a wrapper, something like CreateDirectKafkaStream in kafka0.8 connector, to wrap up the DirectKafkaStream constructor?
The benefit is that user does not need to know the kafka consumer APIs, in order to use the connector.
E.g.: the kafka consumer in the connector gets assigned a collection of TopicPartition, in most cases, all the partitions for given topic, if no wrapper, user needs to exploit the kafka consumer API to first retrieve the partitionInfo. Using the wrapper, user only needs to provide the topics, and such info can be passed to consumer inside the wrapper without the users knowledge.
> Update KafkaDStreams to new Kafka 0.10 Consumer API
> ---------------------------------------------------
>
> Key: SPARK-12177
> URL: https://issues.apache.org/jira/browse/SPARK-12177
> Project: Spark
> Issue Type: Improvement
> Components: Streaming
> Affects Versions: 1.6.0
> Reporter: Nikita Tarasenko
> Labels: consumer, kafka
>
> Kafka 0.9 already released and it introduce new consumer API that not compatible with old one. So, I added new consumer api. I made separate classes in package org.apache.spark.streaming.kafka.v09 with changed API. I didn't remove old classes for more backward compatibility. User will not need to change his old spark applications when he uprgade to new Spark version.
> Please rewiew my changes
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org