You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by mykidong <my...@gmail.com> on 2015/01/15 04:58:46 UTC
How to replay consuming messages from kafka using spark streaming?
Hi,
My Spark Streaming Job is doing like kafka etl to HDFS.
For instance, every 10 min. my streaming job is retrieving messages from
kafka, and save them as avro files onto hdfs.
My question is, if worker fails to write avro to hdfs, sometimes, I want to
replay consuming messages from the last succeeded kafka offset again.
I think, Spark Streaming Kafka Receiver is written using Kafka High Level
Consumer API, not Simple Consumer API.
Any idea how to replay kafka consuming in spark streaming?
- Kidong.
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/How-to-replay-consuming-messages-from-kafka-using-spark-streaming-tp21145.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: How to replay consuming messages from kafka using spark
streaming?
Posted by mykidong <my...@gmail.com>.
Hi,
I have written spark streaming kafka receiver using kafka simple consumer
api:
https://github.com/mykidong/spark-kafka-simple-consumer-receiver
This kafka receiver can be used as alternative to the current spark
streaming kafka receiver which is just written in high level kafka consumer
api.
With this kafka receiver, the kafka message offset control can be done more
easier for Receiver Woker Node Failure and Driver Node Failure.
- Kidong.
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/How-to-replay-consuming-messages-from-kafka-using-spark-streaming-tp21145p21343.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
RE: How to replay consuming messages from kafka using spark
streaming?
Posted by "Shao, Saisai" <sa...@intel.com>.
I think there're two solutions:
1. Enable write ahead log in Spark Streaming if you're using Spark 1.2.
2. Using third-party Kafka consumer (https://github.com/dibbhatt/kafka-spark-consumer).
Thanks
Saisai
-----Original Message-----
From: mykidong [mailto:mykidong@gmail.com]
Sent: Thursday, January 15, 2015 11:59 AM
To: user@spark.apache.org
Subject: How to replay consuming messages from kafka using spark streaming?
Hi,
My Spark Streaming Job is doing like kafka etl to HDFS.
For instance, every 10 min. my streaming job is retrieving messages from kafka, and save them as avro files onto hdfs.
My question is, if worker fails to write avro to hdfs, sometimes, I want to replay consuming messages from the last succeeded kafka offset again.
I think, Spark Streaming Kafka Receiver is written using Kafka High Level Consumer API, not Simple Consumer API.
Any idea how to replay kafka consuming in spark streaming?
- Kidong.
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/How-to-replay-consuming-messages-from-kafka-using-spark-streaming-tp21145.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org For additional commands, e-mail: user-help@spark.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: How to replay consuming messages from kafka using spark streaming?
Posted by Cody Koeninger <co...@koeninger.org>.
Take a look at the implementation linked from here
https://issues.apache.org/jira/browse/SPARK-4964
see if that would meet your needs
On Wed, Jan 14, 2015 at 9:58 PM, mykidong <my...@gmail.com> wrote:
> Hi,
>
> My Spark Streaming Job is doing like kafka etl to HDFS.
> For instance, every 10 min. my streaming job is retrieving messages from
> kafka, and save them as avro files onto hdfs.
> My question is, if worker fails to write avro to hdfs, sometimes, I want to
> replay consuming messages from the last succeeded kafka offset again.
> I think, Spark Streaming Kafka Receiver is written using Kafka High Level
> Consumer API, not Simple Consumer API.
>
> Any idea how to replay kafka consuming in spark streaming?
>
> - Kidong.
>
>
>
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/How-to-replay-consuming-messages-from-kafka-using-spark-streaming-tp21145.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>