You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by "Chen, Mason" <ma...@sony.com> on 2020/05/28 06:08:13 UTC

Flink Kafka Connector Source Parallelism

Hi all,

I’m currently trying to understand Flink’s Kafka Connector and how parallelism affects it. So, I am running the flink playground click count job and the parallelism is set to 2 by default.


However, I don’t see the 2nd subtask of the Kafka Connector sending any records: https://imgur.com/cA5ucSg. Do I need to rebalance after reading from kafka?

```
clicks = clicks
   .keyBy(ClickEvent::getPage)
   .map(new BackpressureMap())
   .name("Backpressure");
```

`clicks` is the kafka click stream. From my reading in the operator docs, it seems counterintuitive to do a `rebalance()` when I am already doing a `keyBy()`.

So, my questions:

1. How do I make use of the 2nd subtask?
2. Does the number of partitions have some sort of correspondence with the parallelism of the source operator? If so, is there a general statement to be made about parallelism across all source operators?

Thanks,
Mason

Re: Flink Kafka Connector Source Parallelism

Posted by Robert Metzger <rm...@apache.org>.
Hi Mason,
your understanding is correct.

On Thu, May 28, 2020 at 8:23 AM Chen, Mason <ma...@sony.com> wrote:

> I think I may have just answered my own question. There’s only one Kafka
> partition, so the maximum parallelism is one and it doesn’t really make
> sense to make another kafka consumer under the same group id. What threw me
> off is that there’s a 2nd subtask for the kafka source created even
> though it’s not actually doing anything. So, it seems a general statement
> can be made that (# kafka partitions) >= (# parallelism of flink kafka
> source)…well I guess you could have more parallelism than kafka partitions,
> but the extra subtasks will not doing anything.
>
>
>
> *From: *"Chen, Mason" <ma...@sony.com>
> *Date: *Wednesday, May 27, 2020 at 11:09 PM
> *To: *"user@flink.apache.org" <us...@flink.apache.org>
> *Subject: *Flink Kafka Connector Source Parallelism
>
>
>
> Hi all,
>
>
>
> I’m currently trying to understand Flink’s Kafka Connector and how
> parallelism affects it. So, I am running the flink playground click count
> job and the parallelism is set to 2 by default.
>
>
> However, I don’t see the 2nd subtask of the Kafka Connector sending any records: https://imgur.com/cA5ucSg. Do I need to rebalance after reading from kafka?
>
> ```
> clicks = clicks
>    .keyBy(ClickEvent::getPage)
>    .map(*new *BackpressureMap())
>    .name(*"Backpressure"*);
> ```
>
>
>
> `clicks` is the kafka click stream. From my reading in the operator docs,
> it seems counterintuitive to do a `rebalance()` when I am already doing a
> `keyBy()`.
>
> So, my questions:
>
> 1. How do I make use of the 2nd subtask?
>
> 2. Does the number of partitions have some sort of correspondence with the
> parallelism of the source operator? If so, is there a general statement to
> be made about parallelism across all source operators?
>
>
>
> Thanks,
>
> Mason
>

Re: Flink Kafka Connector Source Parallelism

Posted by "Chen, Mason" <ma...@sony.com>.
I think I may have just answered my own question. There’s only one Kafka partition, so the maximum parallelism is one and it doesn’t really make sense to make another kafka consumer under the same group id. What threw me off is that there’s a 2nd subtask for the kafka source created even though it’s not actually doing anything. So, it seems a general statement can be made that (# kafka partitions) >= (# parallelism of flink kafka source)…well I guess you could have more parallelism than kafka partitions, but the extra subtasks will not doing anything.

From: "Chen, Mason" <ma...@sony.com>
Date: Wednesday, May 27, 2020 at 11:09 PM
To: "user@flink.apache.org" <us...@flink.apache.org>
Subject: Flink Kafka Connector Source Parallelism

Hi all,

I’m currently trying to understand Flink’s Kafka Connector and how parallelism affects it. So, I am running the flink playground click count job and the parallelism is set to 2 by default.



However, I don’t see the 2nd subtask of the Kafka Connector sending any records: https://imgur.com/cA5ucSg. Do I need to rebalance after reading from kafka?

```
clicks = clicks
   .keyBy(ClickEvent::getPage)
   .map(new BackpressureMap())
   .name("Backpressure");
```

`clicks` is the kafka click stream. From my reading in the operator docs, it seems counterintuitive to do a `rebalance()` when I am already doing a `keyBy()`.

So, my questions:

1. How do I make use of the 2nd subtask?
2. Does the number of partitions have some sort of correspondence with the parallelism of the source operator? If so, is there a general statement to be made about parallelism across all source operators?

Thanks,
Mason