Pass a method as parameter

[Soheil Pourbafrani <so...@gmail.com>]

Is it possible in Flink to write a method A that we can give it a
written function's name B and function A will apply function B to
DataStream?

Re: Pass a method as parameter

[Rong Rong <wa...@gmail.com>]

Hi Soheil,

I think I knew what you meant by passing a function's name B to another
method A: assuming in function "A" you are trying to dynamically load
another function "B" based on either (1) some characteristic of the message
in your data stream, or (2) some configuration during start up.
And I further assume here "method A" is part of your job graph, otherwise
it would be as easy to just get rid of it.

You can definitely include "function B" in your job JAR and ship it over to
the execution cluster; and then simple use Classloader to load by name and
instantiate. However I am not sure this is the best programming paradigm.

To add to Hequn's comment [1], some alternatives you can try to eliminate
the need to load function "B" during execution:
- Consolidate all of the "function B variances" into one function, and use
a *filterBy* or a *keyBy* [2] for choosing different code path upon message
arrival;
- You can use broadcast state [3] to assist you when choosing different
code path if it depends on some external configurations;

Thanks,
Rong

[2]
https://ci.apache.org/projects/flink/flink-docs-master/dev/stream/operators/index.html
[3]
https://ci.apache.org/projects/flink/flink-docs-release-1.5/dev/stream/state/broadcast_state.html


On Sun, Jul 8, 2018 at 5:50 AM Hequn Cheng <ch...@gmail.com> wrote:

> Hi Soheil,
>
> What do you mean by "give it a written function's name B" and "function A
> will apply function B"? Do you mean function A override B?
> Perhaps the DataStream api guide[1] may gives you some guidance.
>
> Best, Hequn
>
> [1]
> https://ci.apache.org/projects/flink/flink-docs-master/dev/datastream_api.html#flink-datastream-api-programming-guide
>
> On Sun, Jul 8, 2018 at 1:21 PM, Soheil Pourbafrani <so...@gmail.com>
> wrote:
>
>> Is it possible in Flink to write a method A that we can give it a
>> written function's name B and function A will apply function B to
>> DataStream?
>>
>
>

Re: Pass a method as parameter

[Hequn Cheng <ch...@gmail.com>]

Hi Soheil,

What do you mean by "give it a written function's name B" and "function A
will apply function B"? Do you mean function A override B?
Perhaps the DataStream api guide[1] may gives you some guidance.

Best, Hequn

[1]
https://ci.apache.org/projects/flink/flink-docs-master/dev/datastream_api.html#flink-datastream-api-programming-guide

On Sun, Jul 8, 2018 at 1:21 PM, Soheil Pourbafrani <so...@gmail.com>
wrote:

> Is it possible in Flink to write a method A that we can give it a
> written function's name B and function A will apply function B to
> DataStream?
>