Structured Streaming + Kafka Integration unable to read new messages after sometimes

[NikhilP <ni...@ril.com>]

I working on structured streaming reading the data from kafka and writng it
back to kafka.I am facing a weird issue,Sometime the jobs stops reading the
data from kafka topics and after deleting the checkpoint directory it starts
again reading the kafka topics.

can anybody help me in resolving the issue?






--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Structured-Streaming-Kafka-Integration-unable-to-read-new-messages-after-sometimes-tp29055.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org

Re: Structured Streaming + Kafka Integration unable to read new messages after sometimes

[Jacek Laskowski <ja...@japila.pl>]

Hi,

Any logs you could share? Anything about the query itself? Watermarked?
Aggregation? How long does it work fine? Is this somehow stable in its
instability? What version of Spark and Kafka?

Pozdrawiam,
Jacek Laskowski
http://blog.japila.pl

On 11 Aug 2017 11:29, "NikhilP" <ni...@ril.com> wrote:

> I working on structured streaming reading the data from kafka and writng it
> back to kafka.I am facing a weird issue,Sometime the jobs stops reading the
> data from kafka topics and after deleting the checkpoint directory it
> starts
> again reading the kafka topics.
>
> can anybody help me in resolving the issue?
>
>
>
>
>
>
> --
> View this message in context: http://apache-spark-user-list.
> 1001560.n3.nabble.com/Structured-Streaming-Kafka-
> Integration-unable-to-read-new-messages-after-sometimes-tp29055.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: user-unsubscribe@spark.apache.org
>
>