You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Heidi Hazem Mohamed <H....@nu.edu.eg> on 2019/10/27 15:03:50 UTC
Custom Partitioning with keyed state
Hi,
What I want : I have my own partitioning technique that generates keys for DataStream tuples ,those keys range are equal to the number of nodes in the clusters like if I set the parallelism equal to 4 the generated keys will be 0,1,2 and 3 and so on and then every key should be partitioned to the same node to do such more keyed processing using keyed state.
What happened: I have implemented my logic using the keyBy so I can use a keyed state but it suffers from a great skewness some of the nodes had received no records and other ones received more than one. I have tried to use custom partitioning it did the physical partitioning as I want but I can not use the keyed state with it without using keyBy.
What I expect (questions): Is there a way to control the skewness or enforce keys to be parallelized over the available nodes? or Is there a way to overwrite the partitioning technique used in keyBy? or Is there a way to use a keyed state with custom partitioning?
Best Regards
Heidy Hazem
[cid:178eec23-6233-4b22-b8a3-59051181b0fc]Heidy Hazem– Teaching assistant, School of Information Technology and Computer Science (Formerly, CIT, Communication Engineering, and Information Technology School)
T: +201000 63-25-63 office: UB1-room 701
26th July Corridor, Sheikh Zayed, Giza, Egypt
www.nu.edu.eg<http://www.nu.edu.eg/> | www.facebook.com/NileUniversity<http://www.facebook.com/NileUniversity>
Re: Custom Partitioning with keyed state
Posted by Congxian Qiu <qc...@gmail.com>.
Hi
Have you tried the key selector function[1]?
[1]
https://ci.apache.org/projects/flink/flink-docs-stable/dev/api_concepts.html#define-keys-using-key-selector-functions
Best,
Congxian
Heidi Hazem Mohamed <H....@nu.edu.eg> 于2019年10月27日周日 下午11:04写道：
> Hi,
>
> What I want : I have my own partitioning technique that generates keys for
> DataStream tuples ,those keys range are equal to the number of nodes in the
> clusters like if I set the parallelism equal to 4 the generated keys will
> be 0,1,2 and 3 and so on and then every key should be partitioned to the
> same node to do such more keyed processing using keyed state.
>
> What happened: I have implemented my logic using the keyBy so I can use a
> keyed state but it suffers from a great skewness some of the nodes had
> received no records and other ones received more than one. I have tried to
> use custom partitioning it did the physical partitioning as I want but I
> can not use the keyed state with it without using keyBy.
>
> What I expect (questions): Is there a way to control the skewness or
> enforce keys to be parallelized over the available nodes? or Is there a way
> to overwrite the partitioning technique used in keyBy? or Is there a way to
> use a keyed state with custom partitioning?
>
> *Best Regards*
>
> *Heidy Hazem*
>
> *Heidy Hazem–* *Teaching assistant, School of Information Technology and
> Computer Science (Formerly, CIT, Communication Engineering, and Information
> Technology School)*
> *T*: +201000 63-25-63 *office*: UB1-room 701
>
> 26th July Corridor, Sheikh Zayed, Giza, Egypt
> *www.nu.edu.eg* <http://www.nu.edu.eg/>* | *
> *www.facebook.com/NileUniversity* <http://www.facebook.com/NileUniversity>
>
>