You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Prasanna kumar <pr...@gmail.com> on 2022/01/12 14:47:08 UTC
BroadCast Operator not working higher parallelism
Hi,
FLINK version 1.12.7
We have the following job.
Source Read from Kafka apply broadcast rules and then write output to
kafka.
We are reading from S3 and writing to broadcast state.
This works till parallelism of 4 . See image (1)
For >4, only a few tasks are populated.
This means that the rules are not updated resulting in a large number of
rows unprocessed.
Is this an existing bug ?
If so has theis been fixed in the following versions ?
Image 1
[image: Screen Shot 2022-01-12 at 8.03.38 PM.png]
Image 2 ( See many sub tasks with 0 in received and 0 in sent)
[image: Screen Shot 2022-01-12 at 8.10.07 PM.png]
Thanks,
Prasanna.
Re: BroadCast Operator not working higher parallelism
Posted by Prasanna kumar <pr...@gmail.com>.
Actually the metrics in the UI shown in wrong and i tested and verified
that all the broadcast state operators got the records.
On Wed, Jan 12, 2022 at 8:34 PM Roman Khachatryan <ro...@apache.org> wrote:
> Hi,
>
> I see the same number of records sent by Custom File Source in total - on
> both images it's 4.
> Are you concerned about downstream tasks? Do you see that they receive
> different number of records from this source?
>
> Regards,
> Roman
>
>
> On Wed, Jan 12, 2022 at 3:47 PM Prasanna kumar <
> prasannakumarramani@gmail.com> wrote:
>
>> Hi,
>> FLINK version 1.12.7
>> We have the following job.
>> Source Read from Kafka apply broadcast rules and then write output to
>> kafka.
>> We are reading from S3 and writing to broadcast state.
>>
>> This works till parallelism of 4 . See image (1)
>> For >4, only a few tasks are populated.
>> This means that the rules are not updated resulting in a large number of
>> rows unprocessed.
>>
>> Is this an existing bug ?
>> If so has theis been fixed in the following versions ?
>>
>> Image 1
>> [image: Screen Shot 2022-01-12 at 8.03.38 PM.png]
>>
>>
>> Image 2 ( See many sub tasks with 0 in received and 0 in sent)
>>
>> [image: Screen Shot 2022-01-12 at 8.10.07 PM.png]
>>
>> Thanks,
>> Prasanna.
>>
>
Re: BroadCast Operator not working higher parallelism
Posted by Roman Khachatryan <ro...@apache.org>.
Hi,
I see the same number of records sent by Custom File Source in total - on
both images it's 4.
Are you concerned about downstream tasks? Do you see that they receive
different number of records from this source?
Regards,
Roman
On Wed, Jan 12, 2022 at 3:47 PM Prasanna kumar <
prasannakumarramani@gmail.com> wrote:
> Hi,
> FLINK version 1.12.7
> We have the following job.
> Source Read from Kafka apply broadcast rules and then write output to
> kafka.
> We are reading from S3 and writing to broadcast state.
>
> This works till parallelism of 4 . See image (1)
> For >4, only a few tasks are populated.
> This means that the rules are not updated resulting in a large number of
> rows unprocessed.
>
> Is this an existing bug ?
> If so has theis been fixed in the following versions ?
>
> Image 1
> [image: Screen Shot 2022-01-12 at 8.03.38 PM.png]
>
>
> Image 2 ( See many sub tasks with 0 in received and 0 in sent)
>
> [image: Screen Shot 2022-01-12 at 8.10.07 PM.png]
>
> Thanks,
> Prasanna.
>