You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Xuelin Cao <xu...@gmail.com> on 2015/02/05 04:44:10 UTC
When will Spark Streaming supports Kafka-simple consumer API?
Hi,
In our environment, Kafka can only be used with simple consumer API,
like storm spout does.
And, also, I found there are suggestions that " Kafka connector of
Spark should not be used in production
<http://markmail.org/message/2lb776ta5sq6lgtw> because it is based on the
high-level consumer API of Kafka."
So, my question is, when will spark streaming supports Kafka simple
consumer API?
Re: When will Spark Streaming supports Kafka-simple consumer API?
Posted by "Xuelin Cao.2015" <xu...@gmail.com>.
Hi, Tathagata
Thanks for the information, I'm trying to build 1.3 snapshot and make
another try.
There are 2 reasons for why we use Kafka SimpleConsumer API
1. Previously, in our company, all of the real time processing system
were build on Apache Storm. So, the kafka environment is set to only
support SimpleConsumer API. The kafka environment is controlled by another
group of engineers in our company, and for some reasons I don't know, they
only support SimpleConsumer API.
2. There is a document advises do not use kafka + spark streaming in
the production environment, due to spark streaming only supports high level
API. see
*http://www.michael-noll.com/blog/2014/10/01/kafka-spark-streaming-integration-example-tutorial/#known-issues-in-spark-streaming
<http://www.michael-noll.com/blog/2014/10/01/kafka-spark-streaming-integration-example-tutorial/#known-issues-in-spark-streaming>*
I'm not quite sure whether the advise is with bias to spark
streaming. But, since we don't have any successful project as our
reference, we need to be careful about it.
On Thu, Feb 5, 2015 at 12:28 PM, Tathagata Das [via Apache Spark Developers
List] <ml...@n3.nabble.com> wrote:
> 1. There is already a third-party low-level kafka receiver -
> http://spark-packages.org/package/5
> 2. There is a new experimental Kafka stream that will be available in
> Spark
> 1.3 release. This is based on the low level API, and might suffice your
> purpose. JIRA - https://issues.apache.org/jira/browse/SPARK-4964
>
> Can you elaborate on why you have to use SimpleConsumer in your
> environment?
>
> TD
>
>
> On Wed, Feb 4, 2015 at 7:44 PM, Xuelin Cao <[hidden email]
> <http:///user/SendEmail.jtp?type=node&node=10477&i=0>> wrote:
>
> > Hi,
> >
> > In our environment, Kafka can only be used with simple consumer
> API,
> > like storm spout does.
> >
> > And, also, I found there are suggestions that " Kafka connector of
> > Spark should not be used in production
> > <http://markmail.org/message/2lb776ta5sq6lgtw> because it is based on
> the
> > high-level consumer API of Kafka."
> >
> > So, my question is, when will spark streaming supports Kafka simple
> > consumer API?
> >
>
>
> ------------------------------
> If you reply to this email, your message will be added to the discussion
> below:
>
> http://apache-spark-developers-list.1001551.n3.nabble.com/When-will-Spark-Streaming-supports-Kafka-simple-consumer-API-tp10476p10477.html
> To start a new topic under Apache Spark Developers List, email
> ml-node+s1001551n1h40@n3.nabble.com
> To unsubscribe from Apache Spark Developers List, click here
> <http://apache-spark-developers-list.1001551.n3.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=1&code=eHVlbGluY2FvMjAxNEBnbWFpbC5jb218MXwtOTc3NDY2MzAy>
> .
> NAML
> <http://apache-spark-developers-list.1001551.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>
--
View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/When-will-Spark-Streaming-supports-Kafka-simple-consumer-API-tp10476p10480.html
Sent from the Apache Spark Developers List mailing list archive at Nabble.com.
Re: When will Spark Streaming supports Kafka-simple consumer API?
Posted by Tathagata Das <ta...@gmail.com>.
1. There is already a third-party low-level kafka receiver -
http://spark-packages.org/package/5
2. There is a new experimental Kafka stream that will be available in Spark
1.3 release. This is based on the low level API, and might suffice your
purpose. JIRA - https://issues.apache.org/jira/browse/SPARK-4964
Can you elaborate on why you have to use SimpleConsumer in your environment?
TD
On Wed, Feb 4, 2015 at 7:44 PM, Xuelin Cao <xu...@gmail.com> wrote:
> Hi,
>
> In our environment, Kafka can only be used with simple consumer API,
> like storm spout does.
>
> And, also, I found there are suggestions that " Kafka connector of
> Spark should not be used in production
> <http://markmail.org/message/2lb776ta5sq6lgtw> because it is based on the
> high-level consumer API of Kafka."
>
> So, my question is, when will spark streaming supports Kafka simple
> consumer API?
>