How to interact with a running flink application?

[Anchit Jatana <de...@gmail.com>]

Hi All,

*Brief:* I have a use case where I need to interact with a running flink
application.

*Detail:*

My Flink application has a *Kafka source*, *an operator processing on the
content received* from the Kafka stream(this operator is using a lookup
from an external source file to accomplish the processing of the Kafka
content). If the content of the file kept at the same source location
changes, I need to notify the operator to update its lookup content loaded
in the memory and continue its processing of Kafka content with the new
loaded lookup content without stopping the Flink application.

Is there a way where I can "interact with the running Flink Application"
through some event or something to notify the application to make some
changes in its operation without stopping the application.

Thank you!

Regards,
Anchit

Re: How to interact with a running flink application?

[Anchit Jatana <de...@gmail.com>]

Hi Ufuk,

Thanks for your help, I'm working on using the suggested approach to
address my use case.

Regards,
Anchit


On Wed, Sep 28, 2016 at 12:48 AM, Ufuk Celebi <uc...@apache.org> wrote:

> Hey Anchit,
>
> the usual recommendation for this is to use a CoMap/CoFlatMap
> operator, where the second input are the lookup location changes. You
> can then use this input to update the location.
>
> Search for CoMap/CoFlatMap here:
> https://ci.apache.org/projects/flink/flink-docs-master/dev/datastream_api.
> html#datastream-transformations
>
> Best,
>
> Ufuk
>
> On Wed, Sep 28, 2016 at 9:37 AM, Anchit Jatana
> <de...@gmail.com> wrote:
> > Hi All,
> >
> > Brief: I have a use case where I need to interact with a running flink
> > application.
> >
> > Detail:
> >
> > My Flink application has a Kafka source, an operator processing on the
> > content received from the Kafka stream(this operator is using a lookup
> from
> > an external source file to accomplish the processing of the Kafka
> content).
> > If the content of the file kept at the same source location changes, I
> need
> > to notify the operator to update its lookup content loaded in the memory
> and
> > continue its processing of Kafka content with the new loaded lookup
> content
> > without stopping the Flink application.
> >
> > Is there a way where I can "interact with the running Flink Application"
> > through some event or something to notify the application to make some
> > changes in its operation without stopping the application.
> >
> > Thank you!
> >
> > Regards,
> > Anchit
>

Re: How to interact with a running flink application?

[Ufuk Celebi <uc...@apache.org>]

Hey Anchit,

the usual recommendation for this is to use a CoMap/CoFlatMap
operator, where the second input are the lookup location changes. You
can then use this input to update the location.

Search for CoMap/CoFlatMap here:
https://ci.apache.org/projects/flink/flink-docs-master/dev/datastream_api.html#datastream-transformations

Best,

Ufuk

On Wed, Sep 28, 2016 at 9:37 AM, Anchit Jatana
<de...@gmail.com> wrote:
> Hi All,
>
> Brief: I have a use case where I need to interact with a running flink
> application.
>
> Detail:
>
> My Flink application has a Kafka source, an operator processing on the
> content received from the Kafka stream(this operator is using a lookup from
> an external source file to accomplish the processing of the Kafka content).
> If the content of the file kept at the same source location changes, I need
> to notify the operator to update its lookup content loaded in the memory and
> continue its processing of Kafka content with the new loaded lookup content
> without stopping the Flink application.
>
> Is there a way where I can "interact with the running Flink Application"
> through some event or something to notify the application to make some
> changes in its operation without stopping the application.
>
> Thank you!
>
> Regards,
> Anchit