You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by karan alang <ka...@gmail.com> on 2022/07/18 05:04:46 UTC
Strimzi Kafka on GKE(GCP) - org.apache.kafka.common.errors.TimeoutException
Hello All,
I've a Strimzi Kafka cluster on GKE, and i've a Spark structured streaming
job which runs on Dataproc, picks up data from Kafka and does some
processing.
It seems to be constantly giving the following Timeout 'warning' message :
```
22/06/22 19:40:49 WARN org.apache.spark.scheduler.TaskSetManager: Lost
task 1.1 in stage 26.0 (TID 243)
(versa-structured-stream1-w-0.c.versa-sml-googl.internal executor 8):
org.apache.kafka.common.errors.TimeoutException: Timeout of 60000ms
expired before the position for partition
syslog.ueba-us4.v1.versa.demo2-1 could be determined
```
This is causing the program to fail intermittently.
Attached are the screenshot of logs i see in the Spark History Server UI.
Any pointers on what needs to be done to fix/debug the issue ?
tia!
Additional details are in the stackoverflow link :
https://stackoverflow.com/questions/72721259/strimzi-kafka-on-gke-org-apache-kafka-common-errors-timeoutexception
[image: Screen Shot 2022-07-17 at 9.57.33 PM.png][image: Screen Shot
2022-07-17 at 9.56.59 PM.png][image: Screen Shot 2022-07-17 at 9.50.56
PM.png]
Re: Strimzi Kafka on GKE(GCP) - org.apache.kafka.common.errors.TimeoutException
Posted by Artemis User <ar...@dtechspace.com>.
I am not sure what you mean by "not trying to access Kafka". Kafka is a
streaming data source so once configured, it will function behind the
scene. As you indicated, you do work with a Kafka cluster. Please
take a look at the link to the Kafka error code, it provides some
insight on what types of coding practice may cause the problem.
On 7/18/22 2:32 PM, karan alang wrote:
>
> well .. surprisingly, the Warning is showing up even when i'm not
> trying to access Kafka, actually working with the Dataframe !
> Is that expected ?
>
> On Mon, Jul 18, 2022 at 11:29 AM karan alang <ka...@gmail.com>
> wrote:
>
>
> well .. surprisingly, the Warning is showing up even when i'm not
> trying to access Kafka, actually working with the Dataframe !
>
>
> On Mon, Jul 18, 2022 at 7:24 AM Artemis User
> <ar...@dtechspace.com> wrote:
>
> Looks like you may have several problems:
>
> 1. EOFError - Your data schema may incorrect that causes the
> read_int raise an exception. You may want to check your
> Kakfa settings and data schema to ensure that your integer
> fields are indeed represented as integers (e.g., in JSON,
> don't use quote around integer numbers).
> 2. Kafka error code 1894 - This issue should be resolved in
> newer version of Kafka
> (https://issues.apache.org/jira/browse/KAFKA-1894). Please
> make sure you have upgraded Kafka to version 2.0.0 and later..
>
> Hope this helps....
> On 7/18/22 1:04 AM, karan alang wrote:
>>
>> Hello All,
>>
>> I've a Strimzi Kafka cluster on GKE, and i've a Spark
>> structured streaming job which runs on Dataproc, picks up
>> data from Kafka and does some processing.
>>
>> It seems to be constantly giving the following Timeout
>> 'warning' message :
>>
>> ```
>>
>> |22/06/22 19:40:49 WARN
>> org.apache.spark.scheduler.TaskSetManager: Lost task 1.1 in
>> stage 26.0 (TID 243)
>> (versa-structured-stream1-w-0.c.versa-sml-googl.internal
>> executor 8): org.apache.kafka.common.errors.TimeoutException:
>> Timeout of 60000ms expired before the position for partition
>> syslog.ueba-us4.v1.versa.demo2-1 could be determined|
>>
>> ```
>>
>> This is causing the program to fail intermittently.
>>
>> Attached are the screenshot of logs i see in the Spark
>> History Server UI.
>>
>> Any pointers on what needs to be done to fix/debug the issue ?
>>
>> tia!
>>
>>
>> Additional details are in the stackoverflow link :
>>
>> https://stackoverflow.com/questions/72721259/strimzi-kafka-on-gke-org-apache-kafka-common-errors-timeoutexception
>>
>>
>> Screen Shot 2022-07-17 at 9.57.33 PM.pngScreen Shot
>> 2022-07-17 at 9.56.59 PM.pngScreen Shot 2022-07-17 at 9.50.56
>> PM.png
>>
>>
>>
>
Re: Strimzi Kafka on GKE(GCP) - org.apache.kafka.common.errors.TimeoutException
Posted by karan alang <ka...@gmail.com>.
well .. surprisingly, the Warning is showing up even when i'm not trying to
access Kafka, actually working with the Dataframe !
Is that expected ?
On Mon, Jul 18, 2022 at 11:29 AM karan alang <ka...@gmail.com> wrote:
>
> well .. surprisingly, the Warning is showing up even when i'm not trying
> to access Kafka, actually working with the Dataframe !
>
>
> On Mon, Jul 18, 2022 at 7:24 AM Artemis User <ar...@dtechspace.com>
> wrote:
>
>> Looks like you may have several problems:
>>
>> 1. EOFError - Your data schema may incorrect that causes the read_int
>> raise an exception. You may want to check your Kakfa settings and data
>> schema to ensure that your integer fields are indeed represented as
>> integers (e.g., in JSON, don't use quote around integer numbers).
>> 2. Kafka error code 1894 - This issue should be resolved in newer
>> version of Kafka (https://issues.apache.org/jira/browse/KAFKA-1894).
>> Please make sure you have upgraded Kafka to version 2.0.0 and later..
>>
>> Hope this helps....
>> On 7/18/22 1:04 AM, karan alang wrote:
>>
>> Hello All,
>>
>> I've a Strimzi Kafka cluster on GKE, and i've a Spark structured
>> streaming job which runs on Dataproc, picks up data from Kafka and does
>> some processing.
>>
>> It seems to be constantly giving the following Timeout 'warning' message :
>>
>> ```
>>
>> 22/06/22 19:40:49 WARN org.apache.spark.scheduler.TaskSetManager: Lost task 1.1 in stage 26.0 (TID 243) (versa-structured-stream1-w-0.c.versa-sml-googl.internal executor 8): org.apache.kafka.common.errors.TimeoutException: Timeout of 60000ms expired before the position for partition syslog.ueba-us4.v1.versa.demo2-1 could be determined
>>
>> ```
>>
>> This is causing the program to fail intermittently.
>>
>> Attached are the screenshot of logs i see in the Spark History Server UI.
>>
>> Any pointers on what needs to be done to fix/debug the issue ?
>>
>> tia!
>>
>>
>> Additional details are in the stackoverflow link :
>>
>>
>> https://stackoverflow.com/questions/72721259/strimzi-kafka-on-gke-org-apache-kafka-common-errors-timeoutexception
>>
>>
>> [image: Screen Shot 2022-07-17 at 9.57.33 PM.png][image: Screen Shot
>> 2022-07-17 at 9.56.59 PM.png][image: Screen Shot 2022-07-17 at 9.50.56
>> PM.png]
>>
>>
>>
>>
>>
>>
>>
Re: Strimzi Kafka on GKE(GCP) - org.apache.kafka.common.errors.TimeoutException
Posted by karan alang <ka...@gmail.com>.
well .. surprisingly, the Warning is showing up even when i'm not trying to
access Kafka, actually working with the Dataframe !
On Mon, Jul 18, 2022 at 7:24 AM Artemis User <ar...@dtechspace.com> wrote:
> Looks like you may have several problems:
>
> 1. EOFError - Your data schema may incorrect that causes the read_int
> raise an exception. You may want to check your Kakfa settings and data
> schema to ensure that your integer fields are indeed represented as
> integers (e.g., in JSON, don't use quote around integer numbers).
> 2. Kafka error code 1894 - This issue should be resolved in newer
> version of Kafka (https://issues.apache.org/jira/browse/KAFKA-1894).
> Please make sure you have upgraded Kafka to version 2.0.0 and later..
>
> Hope this helps....
> On 7/18/22 1:04 AM, karan alang wrote:
>
> Hello All,
>
> I've a Strimzi Kafka cluster on GKE, and i've a Spark structured streaming
> job which runs on Dataproc, picks up data from Kafka and does some
> processing.
>
> It seems to be constantly giving the following Timeout 'warning' message :
>
> ```
>
> 22/06/22 19:40:49 WARN org.apache.spark.scheduler.TaskSetManager: Lost task 1.1 in stage 26.0 (TID 243) (versa-structured-stream1-w-0.c.versa-sml-googl.internal executor 8): org.apache.kafka.common.errors.TimeoutException: Timeout of 60000ms expired before the position for partition syslog.ueba-us4.v1.versa.demo2-1 could be determined
>
> ```
>
> This is causing the program to fail intermittently.
>
> Attached are the screenshot of logs i see in the Spark History Server UI.
>
> Any pointers on what needs to be done to fix/debug the issue ?
>
> tia!
>
>
> Additional details are in the stackoverflow link :
>
>
> https://stackoverflow.com/questions/72721259/strimzi-kafka-on-gke-org-apache-kafka-common-errors-timeoutexception
>
>
> [image: Screen Shot 2022-07-17 at 9.57.33 PM.png][image: Screen Shot
> 2022-07-17 at 9.56.59 PM.png][image: Screen Shot 2022-07-17 at 9.50.56
> PM.png]
>
>
>
>
>
>
>
Re: Strimzi Kafka on GKE(GCP) - org.apache.kafka.common.errors.TimeoutException
Posted by Artemis User <ar...@dtechspace.com>.
Looks like you may have several problems:
1. EOFError - Your data schema may incorrect that causes the read_int
raise an exception. You may want to check your Kakfa settings and
data schema to ensure that your integer fields are indeed
represented as integers (e.g., in JSON, don't use quote around
integer numbers).
2. Kafka error code 1894 - This issue should be resolved in newer
version of Kafka
(https://issues.apache.org/jira/browse/KAFKA-1894). Please make
sure you have upgraded Kafka to version 2.0.0 and later..
Hope this helps....
On 7/18/22 1:04 AM, karan alang wrote:
>
> Hello All,
>
> I've a Strimzi Kafka cluster on GKE, and i've a Spark structured
> streaming job which runs on Dataproc, picks up data from Kafka and
> does some processing.
>
> It seems to be constantly giving the following Timeout 'warning' message :
>
> ```
>
> |22/06/22 19:40:49 WARN org.apache.spark.scheduler.TaskSetManager:
> Lost task 1.1 in stage 26.0 (TID 243)
> (versa-structured-stream1-w-0.c.versa-sml-googl.internal executor 8):
> org.apache.kafka.common.errors.TimeoutException: Timeout of 60000ms
> expired before the position for partition
> syslog.ueba-us4.v1.versa.demo2-1 could be determined|
>
> ```
>
> This is causing the program to fail intermittently.
>
> Attached are the screenshot of logs i see in the Spark History Server UI.
>
> Any pointers on what needs to be done to fix/debug the issue ?
>
> tia!
>
>
> Additional details are in the stackoverflow link :
>
> https://stackoverflow.com/questions/72721259/strimzi-kafka-on-gke-org-apache-kafka-common-errors-timeoutexception
>
>
> Screen Shot 2022-07-17 at 9.57.33 PM.pngScreen Shot 2022-07-17 at
> 9.56.59 PM.pngScreen Shot 2022-07-17 at 9.50.56 PM.png
>
>
>
Re: Strimzi Kafka on GKE(GCP) - org.apache.kafka.common.errors.TimeoutException
Posted by Gourav Sengupta <go...@gmail.com>.
Hi,
If you have a chance use AWS, it is a million times easier to get things
done there with native installations of Kafka, and Kinesis, which are even
compatible with SPARK.
Regards,
Gourav
On Mon, Jul 18, 2022 at 6:07 AM karan alang <ka...@gmail.com> wrote:
> Hello All,
>
> I've a Strimzi Kafka cluster on GKE, and i've a Spark structured streaming
> job which runs on Dataproc, picks up data from Kafka and does some
> processing.
>
> It seems to be constantly giving the following Timeout 'warning' message :
>
> ```
>
> 22/06/22 19:40:49 WARN org.apache.spark.scheduler.TaskSetManager: Lost task 1.1 in stage 26.0 (TID 243) (versa-structured-stream1-w-0.c.versa-sml-googl.internal executor 8): org.apache.kafka.common.errors.TimeoutException: Timeout of 60000ms expired before the position for partition syslog.ueba-us4.v1.versa.demo2-1 could be determined
>
> ```
>
> This is causing the program to fail intermittently.
>
> Attached are the screenshot of logs i see in the Spark History Server UI.
>
> Any pointers on what needs to be done to fix/debug the issue ?
>
> tia!
>
>
> Additional details are in the stackoverflow link :
>
>
> https://stackoverflow.com/questions/72721259/strimzi-kafka-on-gke-org-apache-kafka-common-errors-timeoutexception
>
>
> [image: Screen Shot 2022-07-17 at 9.57.33 PM.png][image: Screen Shot
> 2022-07-17 at 9.56.59 PM.png][image: Screen Shot 2022-07-17 at 9.50.56
> PM.png]
>
>
>
>
>