Question about relationship between operator instances and keys

[haocheng <hi...@icloud.com>]

Hi community!

So many keys are computed separately in each operator parallel, how does Flink achieve this? Let an operator instance processes 100 keys, is there 100 thread that are managed by the operator instance thread?

I know that 
1. an operator many have many parallel instances at the same time, 
2. each of parallel instances is responsible to a “key group”, and each key group is composed of many keys. 
3. Several operator parallel instances may come from the same TaskManager and each of them is a thread.

Re: Question about relationship between operator instances and keys

[David Morávek <dm...@apache.org>]

Hi haocheng,

in short it works as follows:

- Each parallel instance of an operator is responsible for one to N key
groups.
- Each parallel instance belongs to a slot, which is tied with a single
thread (slot may actually introduce multiple subtasks)
- # of keygroups for each operator = max parallelism; keygroups are
distributed between subtasks ("parallel instances of the operator")
- We use hash partitioning to assign keys to KeyGroup

For more details please refer to the Flink documentation [1]. You might
also find Alex's YouTube series [2] or Fabian's and Vasiliki's book [3]
helpful for learning about Flink architecture.

[1]
https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/concepts/stateful-stream-processing/
[2]
https://www.ververica.com/blog/presenting-our-streaming-concepts-introduction-to-flink-video-series
[3]
https://www.amazon.com/Stream-Processing-Apache-Flink-Implementation/dp/149197429X

Best,
D.


On Thu, Dec 2, 2021 at 1:20 PM haocheng <hi...@icloud.com> wrote:

> Hi community!
>
> So many keys are computed separately in each operator parallel, how does
> Flink achieve this? Let an operator instance processes 100 keys, is there
> 100 thread that are managed by the operator instance thread?
>
> I know that
> 1. an operator many have many parallel instances at the same time,
> 2. each of parallel instances is responsible to a “key group”, and each
> key group is composed of many keys.
> 3. Several operator parallel instances may come from the same TaskManager
> and each of them is a thread.