[structured-streaming][kafka] Will the Kafka readstream timeout after connections.max.idle.ms 540000 ms ?

[karthikjay <as...@gmail.com>]

Hi all,

We are running into a scenario where the structured streaming job is exiting
after a while specifically when the Kafka topic is not getting any data.
From the job logs, I see this connections.max.idle.ms = 540000. Does that
mean the spark readstream will close when it does not get data for 540000
milliseconds ? If yes, how do I override this ?



--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org

Re: [structured-streaming][kafka] Will the Kafka readstream timeout after connections.max.idle.ms 540000 ms ?

["Shixiong(Ryan) Zhu" <sh...@databricks.com>]

The streaming query should keep polling data from Kafka. When the query was
stopped, did you see any exception?

Best Regards,
Shixiong Zhu
Databricks Inc.
shixiong@databricks.com

databricks.com

[image: http://databricks.com] <http://databricks.com/>


<https://databricks.com/sparkaisummit/north-america>

On Tue, May 15, 2018 at 6:25 PM, karthikjay <as...@gmail.com> wrote:

> Hi all,
>
> We are running into a scenario where the structured streaming job is
> exiting
> after a while specifically when the Kafka topic is not getting any data.
> From the job logs, I see this connections.max.idle.ms = 540000. Does that
> mean the spark readstream will close when it does not get data for 540000
> milliseconds ? If yes, how do I override this ?
>
>
>
> --
> Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: user-unsubscribe@spark.apache.org
>
>