Kafka Connector is not reading from an specific Kafka Partition

[Juan Romero <js...@gmail.com>]

Hi guys.

Im facing a problem related to the kafka connector reader in apache beam
over dataflow. Actually the topic has 3 partitions : 0 ,1 and 2.

Kafka is deployed over confluent cloud and in the confluent console we can
see the list the messages and its respective partition:

[image: image.png]


The messages in partition 1 and 2 are read by the pipeline but the messages
that allows to partition 0 are not read by the pipeline. For example this
ones are not read:



[image: image.png]

When i saw the dataflow logs seems that effectively the pipeline is only
reading from partitions 1 and 2 and not for 0 :

[image: image.png]

Really we dont have any weird configuration. In fact the normal behavior of
kafka is read from all the partitions.

I don't know if anyone had had this problem. Let me know guys. I appreciate
your help.

Re: Kafka Connector is not reading from an specific Kafka Partition

[Megha Siddavanahalli via user <us...@beam.apache.org>]

Hi Juan,

It might be worth checking your Beam Kafka consumer configuration - can you
please share that with us?

Thanks!


On Tue, May 7, 2024 at 7:36 AM Juan Romero <js...@gmail.com> wrote:

> Hi guys.
>
> Im facing a problem related to the kafka connector reader in apache beam
> over dataflow. Actually the topic has 3 partitions : 0 ,1 and 2.
>
> Kafka is deployed over confluent cloud and in the confluent console we can
> see the list the messages and its respective partition:
>
> [image: image.png]
>
>
> The messages in partition 1 and 2 are read by the pipeline but the
> messages that allows to partition 0 are not read by the pipeline. For
> example this ones are not read:
>
>
>
> [image: image.png]
>
> When i saw the dataflow logs seems that effectively the pipeline is only
> reading from partitions 1 and 2 and not for 0 :
>
> [image: image.png]
>
> Really we dont have any weird configuration. In fact the normal behavior
> of kafka is read from all the partitions.
>
> I don't know if anyone had had this problem. Let me know guys. I
> appreciate your help.
>