You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Rasika Pohankar <ra...@gmail.com> on 2015/01/18 18:47:32 UTC
Spark Streaming with Kafka
I am using Spark Streaming to process data received through Kafka. The
Spark version is 1.2.0. I have written the code in Java and am compiling it
using sbt. The program runs and receives data from Kafka and processes it
as well. But it stops receiving data suddenly after some time( it has run
for an hour up till now while receiving data from Kafka and then always
stopped receiving). The program continues to run, it only stops receiving
data. After a while, sometimes it starts and sometimes doesn't. So I stop
the program and start again.
Earlier I was using Spark 1.0.0. Upgraded to check if the problem was in
that version. But after upgrading also, it is happening.
Is this a known issue? Can someone please help.
Thanking you.
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-Streaming-with-Kafka-tp21222.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
RE: Spark Streaming with Kafka
Posted by "Shao, Saisai" <sa...@intel.com>.
Hi,
Could you please describing your monitored symptom a little more specifically, like is there any exceptions you monitored with Kafka, Zookeeper, what other exceptions from Spark side you monitored? With only this short description can hardly find any clue.
Thanks
Jerry
From: Eduardo Alfaia [mailto:e.costaalfaia@unibs.it]
Sent: Monday, January 19, 2015 1:58 AM
To: Rasika Pohankar; user@spark.apache.org
Subject: R: Spark Streaming with Kafka
I have the same issue.
________________________________
Da: Rasika Pohankar<ma...@gmail.com>
Inviato: 18/01/2015 18:48
A: user@spark.apache.org<ma...@spark.apache.org>
Oggetto: Spark Streaming with Kafka
I am using Spark Streaming to process data received through Kafka. The Spark version is 1.2.0. I have written the code in Java and am compiling it using sbt. The program runs and receives data from Kafka and processes it as well. But it stops receiving data suddenly after some time( it has run for an hour up till now while receiving data from Kafka and then always stopped receiving). The program continues to run, it only stops receiving data. After a while, sometimes it starts and sometimes doesn't. So I stop the program and start again.
Earlier I was using Spark 1.0.0. Upgraded to check if the problem was in that version. But after upgrading also, it is happening.
Is this a known issue? Can someone please help.
Thanking you.
________________________________
View this message in context: Spark Streaming with Kafka<http://apache-spark-user-list.1001560.n3.nabble.com/Spark-Streaming-with-Kafka-tp21222.html>
Sent from the Apache Spark User List mailing list archive<http://apache-spark-user-list.1001560.n3.nabble.com/> at Nabble.com.
Informativa sulla Privacy: http://www.unibs.it/node/8155
R: Spark Streaming with Kafka
Posted by Eduardo Alfaia <e....@unibs.it>.
I have the same issue.
----- Messaggio originale -----
Da: "Rasika Pohankar" <ra...@gmail.com>
Inviato: 18/01/2015 18:48
A: "user@spark.apache.org" <us...@spark.apache.org>
Oggetto: Spark Streaming with Kafka
I am using Spark Streaming to process data received through Kafka. The Spark version is 1.2.0. I have written the code in Java and am compiling it using sbt. The program runs and receives data from Kafka and processes it as well. But it stops receiving data suddenly after some time( it has run for an hour up till now while receiving data from Kafka and then always stopped receiving). The program continues to run, it only stops receiving data. After a while, sometimes it starts and sometimes doesn't. So I stop the program and start again.
Earlier I was using Spark 1.0.0. Upgraded to check if the problem was in that version. But after upgrading also, it is happening.
Is this a known issue? Can someone please help.
Thanking you.
View this message in context: Spark Streaming with Kafka
Sent from the Apache Spark User List mailing list archive at Nabble.com.
--
Informativa sulla Privacy: http://www.unibs.it/node/8155
Re: Spark Streaming with Kafka
Posted by Rasika Pohankar <ra...@gmail.com>.
Hi firemonk9,
Sorry, its been too long but I just saw this. I hope you were able to
resolve it. FWIW, we were able to solve this with the help of the Low Level
Kafka Consumer, instead of the inbuilt Kafka consumer in Spark, from here:
https://github.com/dibbhatt/kafka-spark-consumer/.
Regards,
Rasika.
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-Streaming-with-Kafka-tp21222p27014.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Spark Streaming with Kafka
Posted by Dibyendu Bhattacharya <di...@gmail.com>.
You can probably try the Low Level Consumer option with Spark 1.2
https://github.com/dibbhatt/kafka-spark-consumer
This Consumer can recover from any underlying failure of Spark Platform or
Kafka and either retry or restart the receiver. This is being working
nicely for us.
Regards,
Dibyendu
On Wed, Jan 21, 2015 at 7:46 AM, firemonk9 <dh...@gmail.com>
wrote:
> Hi,
>
> I am having similar issues. Have you found any resolution ?
>
> Thank you
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/Spark-Streaming-with-Kafka-tp21222p21276.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>
Re: Spark Streaming with Kafka
Posted by firemonk9 <dh...@gmail.com>.
Hi,
I am having similar issues. Have you found any resolution ?
Thank you
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-Streaming-with-Kafka-tp21222p21276.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org