You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by vaibhavrtk1 <le...@gmail.com> on 2016/02/22 20:06:00 UTC
Read from kafka after application is restarted
Hi
I am using kafka with spark streaming 1.3.0 . When the spark application is
not running kafka is still receiving messages. When i start the application
those messages which have already been received when spark was not running
are not processed. I am using a unreliable receiver based approach.
What can I do to process earlier messages also, which came while application
was shut down?
PS: If application was down for a long time can i also limit the max number
of message consumed in one batch interval?
Regards
Vaibhav
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Read-from-kafka-after-application-is-restarted-tp26291.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Read from kafka after application is restarted
Posted by Chitturi Padma <le...@gmail.com>.
Hi Vaibhav,
Please try with Kafka direct API approach. Is this not working ?
-- Padma Ch
On Tue, Feb 23, 2016 at 12:36 AM, vaibhavrtk1 [via Apache Spark User List] <
ml-node+s1001560n26291h66@n3.nabble.com> wrote:
> Hi
>
> I am using kafka with spark streaming 1.3.0 . When the spark application
> is not running kafka is still receiving messages. When i start the
> application those messages which have already been received when spark was
> not running are not processed. I am using a unreliable receiver based
> approach.
>
> What can I do to process earlier messages also, which came while
> application was shut down?
>
> PS: If application was down for a long time can i also limit the max
> number of message consumed in one batch interval?
>
> Regards
> Vaibhav
>
> ------------------------------
> If you reply to this email, your message will be added to the discussion
> below:
>
> http://apache-spark-user-list.1001560.n3.nabble.com/Read-from-kafka-after-application-is-restarted-tp26291.html
> To start a new topic under Apache Spark User List, email
> ml-node+s1001560n1h76@n3.nabble.com
> To unsubscribe from Apache Spark User List, click here
> <http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=1&code=bGVhcm5pbmdzLmNoaXR0dXJpQGdtYWlsLmNvbXwxfC03NzExMjUwMg==>
> .
> NAML
> <http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Read-from-kafka-after-application-is-restarted-tp26291p26292.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
Re: Read from kafka after application is restarted
Posted by Ted Yu <yu...@gmail.com>.
For receiver approach, have you tried Ryan's workaround ?
Btw I don't see the errors you faced because there was no attachment.
> On Feb 23, 2016, at 3:39 AM, vaibhavrtk1 <le...@gmail.com> wrote:
>
> Hello
>
> I have tried with Direct API but i am getting this an error, which is being tracked here https://issues.apache.org/jira/browse/SPARK-5594
>
> I also tried using Receiver approach with Write Ahead Logs ,then this issue comes
> https://issues.apache.org/jira/browse/SPARK-12407
>
> In both cases it seems it is not able to get the broadcasted variable from checkpoint directory.
> Attached is the screenshot of errors I faced with both approaches.
>
> What do you guys suggest for solving this issue?
>
>
> Vaibhav Nagpal
> 9535433788
>
>
>> On Tue, Feb 23, 2016 at 1:50 PM, Gideon [via Apache Spark User List] <[hidden email]> wrote:
>> Regarding the spark streaming receiver - can't you just use Kafka direct receivers with checkpoints? So when you restart your application it will read where it last stopped and continue from there
>> Regarding limiting the number of messages - you can do that by setting spark.streaming.receiver.maxRate. Read more about it here
>>
>> If you reply to this email, your message will be added to the discussion below:
>> http://apache-spark-user-list.1001560.n3.nabble.com/Read-from-kafka-after-application-is-restarted-tp26291p26303.html
>> To unsubscribe from Read from kafka after application is restarted, click here.
>> NAML
>
>
> Capture.JPG (222K) Download Attachment
> Capture1.JPG (169K) Download Attachment
>
> View this message in context: Re: Read from kafka after application is restarted
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
Re: Read from kafka after application is restarted
Posted by Chitturi Padma <le...@gmail.com>.
Hi Vaibhav,
As you said, from the second link, I can figure out that, it is not able
to cast the class when it is trying to read from checkpoint. Can you try
explicit casting like asInstanceOf[T] for the broad casted value ?
>From the bug, looks like it affects version 1.5. Try sample wordcount
program with same spark 1.3 version and try to re-produce the error. If you
are able to, then change the version of spark to 1.5 or 1.4 and see if the
issue is seen. If not, then possibly it should have been fixed in latest
version.
As direct API is introduced in spark 1.3, it is likely to have bugs.
On Tue, Feb 23, 2016 at 5:09 PM, vaibhavrtk1 [via Apache Spark User List] <
ml-node+s1001560n26304h69@n3.nabble.com> wrote:
> Hello
>
> I have tried with Direct API but i am getting this an error, which is
> being tracked here https://issues.apache.org/jira/browse/SPARK-5594
>
> I also tried using Receiver approach with Write Ahead Logs ,then this
> issue comes
> https://issues.apache.org/jira/browse/SPARK-12407
>
> In both cases it seems it is not able to get the broadcasted variable from
> checkpoint directory.
> Attached is the screenshot of errors I faced with both approaches.
>
> What do you guys suggest for solving this issue?
>
>
> *Vaibhav Nagpal*
> 9535433788
> <https://in.linkedin.com/in/vaibhav-nagpal-48237870>
>
> On Tue, Feb 23, 2016 at 1:50 PM, Gideon [via Apache Spark User List] <[hidden
> email] <http:///user/SendEmail.jtp?type=node&node=26304&i=0>> wrote:
>
>> Regarding the spark streaming receiver - can't you just use Kafka direct
>> receivers with checkpoints? So when you restart your application it will
>> read where it last stopped and continue from there
>> Regarding limiting the number of messages - you can do that by setting
>> spark.streaming.receiver.maxRate. Read more about it here
>> <http://spark.apache.org/docs/latest/configuration.html>
>>
>> ------------------------------
>> If you reply to this email, your message will be added to the discussion
>> below:
>>
>> http://apache-spark-user-list.1001560.n3.nabble.com/Read-from-kafka-after-application-is-restarted-tp26291p26303.html
>> To unsubscribe from Read from kafka after application is restarted, click
>> here.
>> NAML
>> <http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>>
>
>
> *Capture.JPG* (222K) Download Attachment
> <http://apache-spark-user-list.1001560.n3.nabble.com/attachment/26304/0/Capture.JPG>
> *Capture1.JPG* (169K) Download Attachment
> <http://apache-spark-user-list.1001560.n3.nabble.com/attachment/26304/1/Capture1.JPG>
>
>
> ------------------------------
> If you reply to this email, your message will be added to the discussion
> below:
>
> http://apache-spark-user-list.1001560.n3.nabble.com/Read-from-kafka-after-application-is-restarted-tp26291p26304.html
> To start a new topic under Apache Spark User List, email
> ml-node+s1001560n1h76@n3.nabble.com
> To unsubscribe from Apache Spark User List, click here
> <http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=1&code=bGVhcm5pbmdzLmNoaXR0dXJpQGdtYWlsLmNvbXwxfC03NzExMjUwMg==>
> .
> NAML
> <http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Read-from-kafka-after-application-is-restarted-tp26291p26316.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
Re: Read from kafka after application is restarted
Posted by vaibhavrtk1 <le...@gmail.com>.
Hello
I have tried with Direct API but i am getting this an error, which is being
tracked here https://issues.apache.org/jira/browse/SPARK-5594
I also tried using Receiver approach with Write Ahead Logs ,then this issue
comes
https://issues.apache.org/jira/browse/SPARK-12407
In both cases it seems it is not able to get the broadcasted variable from
checkpoint directory.
Attached is the screenshot of errors I faced with both approaches.
What do you guys suggest for solving this issue?
*Vaibhav Nagpal*
9535433788
<https://in.linkedin.com/in/vaibhav-nagpal-48237870>
On Tue, Feb 23, 2016 at 1:50 PM, Gideon [via Apache Spark User List] <
ml-node+s1001560n26303h41@n3.nabble.com> wrote:
> Regarding the spark streaming receiver - can't you just use Kafka direct
> receivers with checkpoints? So when you restart your application it will
> read where it last stopped and continue from there
> Regarding limiting the number of messages - you can do that by setting
> spark.streaming.receiver.maxRate. Read more about it here
> <http://spark.apache.org/docs/latest/configuration.html>
>
> ------------------------------
> If you reply to this email, your message will be added to the discussion
> below:
>
> http://apache-spark-user-list.1001560.n3.nabble.com/Read-from-kafka-after-application-is-restarted-tp26291p26303.html
> To unsubscribe from Read from kafka after application is restarted, click
> here
> <http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=26291&code=bGVhcm5pbmdzLnZhaWJoYXZAZ21haWwuY29tfDI2MjkxfDkwMjIyMzc4Nw==>
> .
> NAML
> <http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>
Capture.JPG (222K) <http://apache-spark-user-list.1001560.n3.nabble.com/attachment/26304/0/Capture.JPG>
Capture1.JPG (169K) <http://apache-spark-user-list.1001560.n3.nabble.com/attachment/26304/1/Capture1.JPG>
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Read-from-kafka-after-application-is-restarted-tp26291p26304.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
Re: Read from kafka after application is restarted
Posted by Gideon <gi...@volcanodata.com>.
Regarding the spark streaming receiver - can't you just use Kafka direct
receivers with checkpoints? So when you restart your application it will
read where it last stopped and continue from there
Regarding limiting the number of messages - you can do that by setting
spark.streaming.receiver.maxRate. Read more about it here
<http://spark.apache.org/docs/latest/configuration.html>
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Read-from-kafka-after-application-is-restarted-tp26291p26303.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Read from kafka after application is restarted
Posted by Cody Koeninger <co...@koeninger.org>.
The direct stream will let you do both of those things. Is there a reason
you want to use receivers?
http://spark.apache.org/docs/latest/streaming-kafka-integration.html
http://spark.apache.org/docs/latest/configuration.html#spark-streaming
look for maxRatePerPartition
On Mon, Feb 22, 2016 at 1:06 PM, vaibhavrtk1 <le...@gmail.com>
wrote:
> Hi
>
> I am using kafka with spark streaming 1.3.0 . When the spark application is
> not running kafka is still receiving messages. When i start the application
> those messages which have already been received when spark was not running
> are not processed. I am using a unreliable receiver based approach.
>
> What can I do to process earlier messages also, which came while
> application
> was shut down?
>
> PS: If application was down for a long time can i also limit the max number
> of message consumed in one batch interval?
>
> Regards
> Vaibhav
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/Read-from-kafka-after-application-is-restarted-tp26291.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>