You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by JF Chen <da...@gmail.com> on 2019/03/06 03:08:29 UTC
"java.lang.AssertionError: assertion failed: Failed to get records
for **** after polling for 180000" error
When my kafka executor reads data from kafka, sometimes it throws the error
"java.lang.AssertionError: assertion failed: Failed to get records for ****
after polling for 180000" , which after 3 minutes of executing.
The data waiting for read is not so huge, which is about 1GB. And other
partitions read by other tasks are very fast, the error always occurs on
some specific executor..
Regard,
Junfeng Chen
Re: "java.lang.AssertionError: assertion failed: Failed to get
records for **** after polling for 180000" error
Posted by Shyam P <sh...@gmail.com>.
Would be better if you share some code block to understand it better.
Else would be difficult to provide answer.
~Shyam
On Wed, Mar 6, 2019 at 8:38 AM JF Chen <da...@gmail.com> wrote:
> When my kafka executor reads data from kafka, sometimes it throws the
> error "java.lang.AssertionError: assertion failed: Failed to get records
> for **** after polling for 180000" , which after 3 minutes of executing.
> The data waiting for read is not so huge, which is about 1GB. And other
> partitions read by other tasks are very fast, the error always occurs on
> some specific executor..
>
> Regard,
> Junfeng Chen
>
Re: "java.lang.AssertionError: assertion failed: Failed to get
records for **** after polling for 180000" error
Posted by JF Chen <da...@gmail.com>.
Hi
The max bytes setting should be enough, because if the tasks fail, it read
the data from kafka very fast as normal.
The request.timeout.ms I set is 180 seconds.
I think it should be time out setting or max bandwidth setting because of
the reason that it recoveries and read the same partition very fast after
the tasks are marked failed.
Regard,
Junfeng Chen
On Wed, Mar 6, 2019 at 4:01 PM Akshay Bhardwaj <
akshay.bhardwaj1988@gmail.com> wrote:
> Sorry message sent as incomplete.
>
> To better debug the issue, please check the below config properties:
>
> - At Kafka consumer properties
> - max.partition.fetch.bytes within spark kafka consumer. If not set
> for consumer then the global config at broker level.
> - request.timeout.ms
> - At spark's configurations
> - spark.streaming.kafka.consumer.poll.ms
> - spark.network.timeout (If the above is not set, then poll.ms is
> default to spark.network.timeout)
>
>
> Generally I have faced this issue if spark.streaming.kafka.
> consumer.poll.ms is less than request.timeout.ms
>
> Also, what is the average kafka record message size in bytes?
>
>
>
> Akshay Bhardwaj
> +91-97111-33849
>
>
> On Wed, Mar 6, 2019 at 1:26 PM Akshay Bhardwaj <
> akshay.bhardwaj1988@gmail.com> wrote:
>
>> Hi,
>>
>> To better debug the issue, please check the below config properties:
>>
>> - max.partition.fetch.bytes within spark kafka consumer. If not set
>> for consumer then the global config at broker level.
>> - spark.streaming.kafka.consumer.poll.ms
>> - spark.network.timeout (If the above is not set, then poll.ms is
>> default to spark.network.timeout)
>> -
>> -
>>
>> Akshay Bhardwaj
>> +91-97111-33849
>>
>>
>> On Wed, Mar 6, 2019 at 8:39 AM JF Chen <da...@gmail.com> wrote:
>>
>>> When my kafka executor reads data from kafka, sometimes it throws the
>>> error "java.lang.AssertionError: assertion failed: Failed to get records
>>> for **** after polling for 180000" , which after 3 minutes of executing.
>>> The data waiting for read is not so huge, which is about 1GB. And other
>>> partitions read by other tasks are very fast, the error always occurs on
>>> some specific executor..
>>>
>>> Regard,
>>> Junfeng Chen
>>>
>>
Re: "java.lang.AssertionError: assertion failed: Failed to get
records for **** after polling for 180000" error
Posted by Akshay Bhardwaj <ak...@gmail.com>.
Sorry message sent as incomplete.
To better debug the issue, please check the below config properties:
- At Kafka consumer properties
- max.partition.fetch.bytes within spark kafka consumer. If not set
for consumer then the global config at broker level.
- request.timeout.ms
- At spark's configurations
- spark.streaming.kafka.consumer.poll.ms
- spark.network.timeout (If the above is not set, then poll.ms is
default to spark.network.timeout)
Generally I have faced this issue if spark.streaming.kafka.consumer.poll.ms is
less than request.timeout.ms
Also, what is the average kafka record message size in bytes?
Akshay Bhardwaj
+91-97111-33849
On Wed, Mar 6, 2019 at 1:26 PM Akshay Bhardwaj <
akshay.bhardwaj1988@gmail.com> wrote:
> Hi,
>
> To better debug the issue, please check the below config properties:
>
> - max.partition.fetch.bytes within spark kafka consumer. If not set
> for consumer then the global config at broker level.
> - spark.streaming.kafka.consumer.poll.ms
> - spark.network.timeout (If the above is not set, then poll.ms is
> default to spark.network.timeout)
> -
> -
>
> Akshay Bhardwaj
> +91-97111-33849
>
>
> On Wed, Mar 6, 2019 at 8:39 AM JF Chen <da...@gmail.com> wrote:
>
>> When my kafka executor reads data from kafka, sometimes it throws the
>> error "java.lang.AssertionError: assertion failed: Failed to get records
>> for **** after polling for 180000" , which after 3 minutes of executing.
>> The data waiting for read is not so huge, which is about 1GB. And other
>> partitions read by other tasks are very fast, the error always occurs on
>> some specific executor..
>>
>> Regard,
>> Junfeng Chen
>>
>
Re: "java.lang.AssertionError: assertion failed: Failed to get
records for **** after polling for 180000" error
Posted by Akshay Bhardwaj <ak...@gmail.com>.
Hi,
To better debug the issue, please check the below config properties:
- max.partition.fetch.bytes within spark kafka consumer. If not set for
consumer then the global config at broker level.
- spark.streaming.kafka.consumer.poll.ms
- spark.network.timeout (If the above is not set, then poll.ms is
default to spark.network.timeout)
-
-
Akshay Bhardwaj
+91-97111-33849
On Wed, Mar 6, 2019 at 8:39 AM JF Chen <da...@gmail.com> wrote:
> When my kafka executor reads data from kafka, sometimes it throws the
> error "java.lang.AssertionError: assertion failed: Failed to get records
> for **** after polling for 180000" , which after 3 minutes of executing.
> The data waiting for read is not so huge, which is about 1GB. And other
> partitions read by other tasks are very fast, the error always occurs on
> some specific executor..
>
> Regard,
> Junfeng Chen
>