Passing Configuration & State

[Navneeth Krishnan <re...@gmail.com>]

Hi All,

I have developed a streaming pipeline in java and I need to pass some of
the configuration parameters that are passed during program startup to user
functions. I used the below link as reference.

https://ci.apache.org/projects/flink/flink-docs-release-1.3/dev/best_practices.html

I have tried withParameters & setGlobalJobParameters but that doesn't seem
to work. The parameters are blank inside my user function when deployed in
a cluster. I have also tried passing the parameters inside the constructor
of my user function and this seem to work on local but not in cluster mode,
again the parameters are blank.

Is there a recommended way to pass the program parameters to user function
classes?

Also, I have scenario where the state created inside a user function has to
passed around to multiple classes. Is there a state registry or something
from which I can retrieve a registered state and use or should I implement
my own?

Thanks in advance.

Regards,
Navneeth

Re: Passing Configuration & State

[Fabian Hueske <fh...@gmail.com>]

Hi Navneeth,

the configuring user function using a Configuration object and setting the
parameters in the open() method of a RichFunction is no longer recommended.
In fact, that only works for the DataSet API and has not been added for the
DataStream API. The open() method with the Configuration parameter still
exists because DataSet and DataStream API share these interfaces.

The recommended way to configure user functions is via constructor
parameters while defining the streaming application. The function object
that you create there is serialize and shipped to the workers.
Hence, you can do something like:

DataStream<X> input = ...
DataStream<Y> output = input.map(new MyMapFunction(param1, param2));

where param1 and param2 are stored in MyMapFunction, serialized and shipped
with the MyMapFunction instance.

Regarding sharing state objects inside of a user function. You can access
the state object from the RuntimeContext.
However, you must make sure that the state is only accessed within function
calls of the user function, i.e, you should not leak it to other operator
through a singleton that holds state objects.

Hope this helps,
Fabian

2017-10-26 16:54 GMT+02:00 Navneeth Krishnan <re...@gmail.com>:

> Hi All,
>
> I have developed a streaming pipeline in java and I need to pass some of
> the configuration parameters that are passed during program startup to user
> functions. I used the below link as reference.
>
> https://ci.apache.org/projects/flink/flink-docs-release-1.3/dev/best_
> practices.html
>
> I have tried withParameters & setGlobalJobParameters but that doesn't
> seem to work. The parameters are blank inside my user function when
> deployed in a cluster. I have also tried passing the parameters inside the
> constructor of my user function and this seem to work on local but not in
> cluster mode, again the parameters are blank.
>
> Is there a recommended way to pass the program parameters to user function
> classes?
>
> Also, I have scenario where the state created inside a user function has
> to passed around to multiple classes. Is there a state registry or
> something from which I can retrieve a registered state and use or should I
> implement my own?
>
> Thanks in advance.
>
> Regards,
> Navneeth
>