You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@flink.apache.org by Yashoda Krishna T <ya...@unbxd.com> on 2023/02/14 15:00:11 UTC

DI in flink

Does flink support dependency injection in flink task functions in java? If
not is there an alternative?

RE: DI in flink

Posted by Schwalbe Matthias <Ma...@viseca.ch>.

Hi Yashoda,

I use Spring-Boot to setup my job networks and DI-compose streaming components like operators/functions etc.
Important part is that all components need to be serializable in order for this to work.
Specific task implementations are a little more difficult (little experience) to set up in a DI way. If I’m not mistaken, Flink uses factories for this.

Sincere greetings

Thias


From: Yashoda Krishna T <ya...@unbxd.com>
Sent: Wednesday, February 15, 2023 6:19 AM
To: Austin Cawley-Edwards <au...@gmail.com>
Cc: user <us...@flink.apache.org>
Subject: Re: DI in flink

Thanks Austin.
I can make use of Rich functions to solve my problem.

Thanks
Yashoda

On Wed, Feb 15, 2023 at 12:42 AM Austin Cawley-Edwards <au...@gmail.com>> wrote:
(note: please keep user@flink.apache.org<ma...@flink.apache.org> included in replies)

Ah, I see. Then no, this is not provided by Flink. When I've used dependency inject with Flink in the past, I instantiated everything in the `open()` method of the Flink Rich* classes. Could you solve this by having a common base Sink class or builder that does the configuring? I'm just wondering why it's necessary to solve it in Flink itself.

Best,
Austin

On Tue, Feb 14, 2023 at 11:05 AM Yashoda Krishna T <ya...@unbxd.com>> wrote:
This is my use case.
I have a sink function to push streaming data to S3. And I have a class lets call S3ConnProvider that provides me a connection object to S3, and a class lets say S3Util that has functions over S3 which injects S3ConnProvider.
If dependency injection works I can inject S3Util alone in my SinkFunction class. If not I have to initialize S3ConnProvider first and then S3Util.
This can become complex if there are too many initializations required depending on the use case.
Diese Nachricht ist ausschliesslich für den Adressaten bestimmt und beinhaltet unter Umständen vertrauliche Mitteilungen. Da die Vertraulichkeit von e-Mail-Nachrichten nicht gewährleistet werden kann, übernehmen wir keine Haftung für die Gewährung der Vertraulichkeit und Unversehrtheit dieser Mitteilung. Bei irrtümlicher Zustellung bitten wir Sie um Benachrichtigung per e-Mail und um Löschung dieser Nachricht sowie eventueller Anhänge. Jegliche unberechtigte Verwendung oder Verbreitung dieser Informationen ist streng verboten.

This message is intended only for the named recipient and may contain confidential or privileged information. As the confidentiality of email communication cannot be guaranteed, we do not accept any responsibility for the confidentiality and the intactness of this message. If you have received it in error, please advise the sender by return e-mail and delete this message and any attachments. Any unauthorised use or dissemination of this information is strictly prohibited.

Re: DI in flink

Posted by Yashoda Krishna T <ya...@unbxd.com>.

Thanks Austin.
I can make use of Rich functions to solve my problem.

Thanks
Yashoda

On Wed, Feb 15, 2023 at 12:42 AM Austin Cawley-Edwards <
austin.cawley@gmail.com> wrote:

> (note: please keep user@flink.apache.org included in replies)
>
> Ah, I see. Then no, this is not provided by Flink. When I've used
> dependency inject with Flink in the past, I instantiated everything in the
> `open()` method of the Flink Rich* classes. Could you solve this by having
> a common base Sink class or builder that does the configuring? I'm just
> wondering why it's necessary to solve it in Flink itself.
>
> Best,
> Austin
>
> On Tue, Feb 14, 2023 at 11:05 AM Yashoda Krishna T <
> yashoda.krishna@unbxd.com> wrote:
>
>> This is my use case.
>> I have a sink function to push streaming data to S3. And I have a class
>> lets call S3ConnProvider that provides me a connection object to S3, and a
>> class lets say S3Util that has functions over S3 which injects
>> S3ConnProvider.
>> If dependency injection works I can inject S3Util alone in my
>> SinkFunction class. If not I have to initialize S3ConnProvider first and
>> then S3Util.
>> This can become complex if there are too many initializations
>> required depending on the use case.
>>
>>>

Re: DI in flink

Posted by Austin Cawley-Edwards <au...@gmail.com>.

(note: please keep user@flink.apache.org included in replies)

Ah, I see. Then no, this is not provided by Flink. When I've used
dependency inject with Flink in the past, I instantiated everything in the
`open()` method of the Flink Rich* classes. Could you solve this by having
a common base Sink class or builder that does the configuring? I'm just
wondering why it's necessary to solve it in Flink itself.

Best,
Austin

On Tue, Feb 14, 2023 at 11:05 AM Yashoda Krishna T <
yashoda.krishna@unbxd.com> wrote:

> This is my use case.
> I have a sink function to push streaming data to S3. And I have a class
> lets call S3ConnProvider that provides me a connection object to S3, and a
> class lets say S3Util that has functions over S3 which injects
> S3ConnProvider.
> If dependency injection works I can inject S3Util alone in my SinkFunction
> class. If not I have to initialize S3ConnProvider first and then S3Util.
> This can become complex if there are too many initializations
> required depending on the use case.
>
>>

Re: DI in flink

Posted by Austin Cawley-Edwards <au...@gmail.com>.

What would be the benefits and features over what can be done in user land?

On Tue, Feb 14, 2023 at 10:41 Yashoda Krishna T <ya...@unbxd.com>
wrote:

> Hi Austin
>
> Yes this can be done in Usrr land.
> Can we do it in flink land too?
>
> Thanks
> Yashoda
>
> On Tue, 14 Feb 2023, 9:05 pm Austin Cawley-Edwards, <
> austin.cawley@gmail.com> wrote:
>
>> Hey Yashoda,
>>
>> This can be done in userland (eg with Dagger <https://dagger.dev/>)
>> unless you're wanting Flink to do something in addition?
>>
>> Best,
>> Austin
>>
>> On Tue, Feb 14, 2023 at 10:01 AM Yashoda Krishna T <
>> yashoda.krishna@unbxd.com> wrote:
>>
>>> Does flink support dependency injection in flink task functions in java?
>>> If not is there an alternative?
>>>
>>

Re: DI in flink

Posted by Austin Cawley-Edwards <au...@gmail.com>.

Hey Yashoda,

This can be done in userland (eg with Dagger <https://dagger.dev/>) unless
you're wanting Flink to do something in addition?

Best,
Austin

On Tue, Feb 14, 2023 at 10:01 AM Yashoda Krishna T <
yashoda.krishna@unbxd.com> wrote:

> Does flink support dependency injection in flink task functions in java?
> If not is there an alternative?
>