You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Thomas Wang <w...@datability.io> on 2022/07/10 13:29:45 UTC
DataStream.keyBy() with keys determined at run time
Hi,
I have a use case where I need to call DataStream.keyBy() with keys loaded
from a configuration. The number of keys and their data types are variables
and is determined by the configuration. Once the configuration is loaded,
they won't change. I'm trying to use the following key selector, but it
looks like I cannot use Tuple as the key type here. Is there any way I can
work around this as the rest of the logic of my application is the same.
Thank you!
public class SimpleRecordKeySelector<T extends BaseRecord> implements
KeySelector<T, Tuple>
Thomas
Re: DataStream.keyBy() with keys determined at run time
Posted by Thomas Wang <w...@datability.io>.
Hemanga, the issue is that the number of keys is unknown at the compile
time.
I ended up using yidan's suggestion and serialized all keys into a string.
Thanks for the suggestion.
Thomas
On Sun, Jul 10, 2022 at 7:05 PM yidan zhao <hi...@gmail.com> wrote:
> You can use string, and serialize all keys to a string.
>
> Hemanga Borah <bo...@gmail.com> 于2022年7月11日周一 09:49写道：
> >
> > Here is the documentation of the Tuple class:
> https://nightlies.apache.org/flink/flink-docs-master/api/java/org/apache/flink/api/java/tuple/Tuple.html
> >
> > If you need a concrete class, you can go from Tuple0 to Tuple25.
> >
> > On Sun, Jul 10, 2022 at 5:43 PM Thomas Wang <w...@datability.io> wrote:
> >>
> >> I didn't copy the exact error message, but basically the idea of the
> error message is that I cannot use the abstract class Tuple and instead, I
> should use Tuple1, Tuple2 and etc.
> >>
> >> Thomas
> >>
> >> On Sun, Jul 10, 2022 at 12:47 PM Hemanga Borah <bo...@gmail.com>
> wrote:
> >>>
> >>> What error do you see?
> >>>
> >>> On Sun, Jul 10, 2022 at 6:30 AM Thomas Wang <w...@datability.io> wrote:
> >>>>
> >>>> Hi,
> >>>>
> >>>> I have a use case where I need to call DataStream.keyBy() with keys
> loaded from a configuration. The number of keys and their data types are
> variables and is determined by the configuration. Once the configuration is
> loaded, they won't change. I'm trying to use the following key selector,
> but it looks like I cannot use Tuple as the key type here. Is there any way
> I can work around this as the rest of the logic of my application is the
> same. Thank you!
> >>>>
> >>>> public class SimpleRecordKeySelector<T extends BaseRecord> implements
> KeySelector<T, Tuple>
> >>>>
> >>>> Thomas
> >>>>
>
Re: DataStream.keyBy() with keys determined at run time
Posted by yidan zhao <hi...@gmail.com>.
You can use string, and serialize all keys to a string.
Hemanga Borah <bo...@gmail.com> 于2022年7月11日周一 09:49写道：
>
> Here is the documentation of the Tuple class: https://nightlies.apache.org/flink/flink-docs-master/api/java/org/apache/flink/api/java/tuple/Tuple.html
>
> If you need a concrete class, you can go from Tuple0 to Tuple25.
>
> On Sun, Jul 10, 2022 at 5:43 PM Thomas Wang <w...@datability.io> wrote:
>>
>> I didn't copy the exact error message, but basically the idea of the error message is that I cannot use the abstract class Tuple and instead, I should use Tuple1, Tuple2 and etc.
>>
>> Thomas
>>
>> On Sun, Jul 10, 2022 at 12:47 PM Hemanga Borah <bo...@gmail.com> wrote:
>>>
>>> What error do you see?
>>>
>>> On Sun, Jul 10, 2022 at 6:30 AM Thomas Wang <w...@datability.io> wrote:
>>>>
>>>> Hi,
>>>>
>>>> I have a use case where I need to call DataStream.keyBy() with keys loaded from a configuration. The number of keys and their data types are variables and is determined by the configuration. Once the configuration is loaded, they won't change. I'm trying to use the following key selector, but it looks like I cannot use Tuple as the key type here. Is there any way I can work around this as the rest of the logic of my application is the same. Thank you!
>>>>
>>>> public class SimpleRecordKeySelector<T extends BaseRecord> implements KeySelector<T, Tuple>
>>>>
>>>> Thomas
>>>>
Re: DataStream.keyBy() with keys determined at run time
Posted by Hemanga Borah <bo...@gmail.com>.
Here is the documentation of the Tuple class:
https://nightlies.apache.org/flink/flink-docs-master/api/java/org/apache/flink/api/java/tuple/Tuple.html
If you need a concrete class, you can go from Tuple0 to Tuple25.
On Sun, Jul 10, 2022 at 5:43 PM Thomas Wang <w...@datability.io> wrote:
> I didn't copy the exact error message, but basically the idea of the error
> message is that I cannot use the abstract class Tuple and instead, I should
> use Tuple1, Tuple2 and etc.
>
> Thomas
>
> On Sun, Jul 10, 2022 at 12:47 PM Hemanga Borah <bo...@gmail.com>
> wrote:
>
>> What error do you see?
>>
>> On Sun, Jul 10, 2022 at 6:30 AM Thomas Wang <w...@datability.io> wrote:
>>
>>> Hi,
>>>
>>> I have a use case where I need to call DataStream.keyBy() with keys
>>> loaded from a configuration. The number of keys and their data types are
>>> variables and is determined by the configuration. Once the configuration is
>>> loaded, they won't change. I'm trying to use the following key selector,
>>> but it looks like I cannot use Tuple as the key type here. Is there any way
>>> I can work around this as the rest of the logic of my application is the
>>> same. Thank you!
>>>
>>> public class SimpleRecordKeySelector<T extends BaseRecord> implements
>>> KeySelector<T, Tuple>
>>>
>>> Thomas
>>>
>>>
Re: DataStream.keyBy() with keys determined at run time
Posted by Thomas Wang <w...@datability.io>.
I didn't copy the exact error message, but basically the idea of the error
message is that I cannot use the abstract class Tuple and instead, I should
use Tuple1, Tuple2 and etc.
Thomas
On Sun, Jul 10, 2022 at 12:47 PM Hemanga Borah <bo...@gmail.com>
wrote:
> What error do you see?
>
> On Sun, Jul 10, 2022 at 6:30 AM Thomas Wang <w...@datability.io> wrote:
>
>> Hi,
>>
>> I have a use case where I need to call DataStream.keyBy() with keys
>> loaded from a configuration. The number of keys and their data types are
>> variables and is determined by the configuration. Once the configuration is
>> loaded, they won't change. I'm trying to use the following key selector,
>> but it looks like I cannot use Tuple as the key type here. Is there any way
>> I can work around this as the rest of the logic of my application is the
>> same. Thank you!
>>
>> public class SimpleRecordKeySelector<T extends BaseRecord> implements
>> KeySelector<T, Tuple>
>>
>> Thomas
>>
>>
Re: DataStream.keyBy() with keys determined at run time
Posted by Hemanga Borah <bo...@gmail.com>.
What error do you see?
On Sun, Jul 10, 2022 at 6:30 AM Thomas Wang <w...@datability.io> wrote:
> Hi,
>
> I have a use case where I need to call DataStream.keyBy() with keys loaded
> from a configuration. The number of keys and their data types are variables
> and is determined by the configuration. Once the configuration is loaded,
> they won't change. I'm trying to use the following key selector, but it
> looks like I cannot use Tuple as the key type here. Is there any way I can
> work around this as the rest of the logic of my application is the same.
> Thank you!
>
> public class SimpleRecordKeySelector<T extends BaseRecord> implements
> KeySelector<T, Tuple>
>
> Thomas
>
>