You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Andrew Roberts <ar...@fuze.com> on 2017/12/15 21:23:01 UTC

Job-level close()?

Hello,

I’m writing a Flink operator that connects to a database, and running it in parallel results in issues due to the singleton nature of the connection pool in the library I’m working with. The operator needs to close the connection pool when it’s done, but only when ALL parallel instances are done. If one subtask finishes first, then it closes the pool out from under the subtasks that are still working. Currently, I’m using a reference-counting hack that’s pretty brittle and unsatisfying. Are there any plans to add a user-definable job-level cleanup interface?

Thanks

Andrew
-- 
*Confidentiality Notice: The information contained in this e-mail and any
attachments may be confidential. If you are not an intended recipient, you
are hereby notified that any dissemination, distribution or copying of this
e-mail is strictly prohibited. If you have received this e-mail in error,
please notify the sender and permanently delete the e-mail and any
attachments immediately. You should not retain, copy or use this e-mail or
any attachment for any purpose, nor disclose all or any part of the
contents to any other person. Thank you.*

Re: Job-level close()?

Posted by Eron Wright <er...@gmail.com>.
Since the connection pool is per-TM, and a given job may span numerous TMs,
I think you're imagining a per-TM cleanup.

Your idea of using reference counting seems like the best option.  Maintain
a static count of open instances.  Use a synchronization block to manage
the count and to conditionally open and close the connection pool.


On Mon, Dec 18, 2017 at 1:58 AM, Fabian Hueske <fh...@gmail.com> wrote:

> Hi Andrew,
>
> I'm not aware of such a plan.
> Another way to address such issues is to run multiple TaskManagers with a
> single slot. In that case, only one subtask is executed per TM processes.
>
> Best, Fabian
>
> 2017-12-15 22:23 GMT+01:00 Andrew Roberts <ar...@fuze.com>:
>
>> Hello,
>>
>> I’m writing a Flink operator that connects to a database, and running it
>> in parallel results in issues due to the singleton nature of the connection
>> pool in the library I’m working with. The operator needs to close the
>> connection pool when it’s done, but only when ALL parallel instances are
>> done. If one subtask finishes first, then it closes the pool out from under
>> the subtasks that are still working. Currently, I’m using a
>> reference-counting hack that’s pretty brittle and unsatisfying. Are there
>> any plans to add a user-definable job-level cleanup interface?
>>
>> Thanks
>>
>> Andrew
>> --
>> *Confidentiality Notice: The information contained in this e-mail and any
>> attachments may be confidential. If you are not an intended recipient, you
>> are hereby notified that any dissemination, distribution or copying of
>> this
>> e-mail is strictly prohibited. If you have received this e-mail in error,
>> please notify the sender and permanently delete the e-mail and any
>> attachments immediately. You should not retain, copy or use this e-mail or
>> any attachment for any purpose, nor disclose all or any part of the
>> contents to any other person. Thank you.*
>>
>
>

Re: Job-level close()?

Posted by Fabian Hueske <fh...@gmail.com>.
Hi Andrew,

I'm not aware of such a plan.
Another way to address such issues is to run multiple TaskManagers with a
single slot. In that case, only one subtask is executed per TM processes.

Best, Fabian

2017-12-15 22:23 GMT+01:00 Andrew Roberts <ar...@fuze.com>:

> Hello,
>
> I’m writing a Flink operator that connects to a database, and running it
> in parallel results in issues due to the singleton nature of the connection
> pool in the library I’m working with. The operator needs to close the
> connection pool when it’s done, but only when ALL parallel instances are
> done. If one subtask finishes first, then it closes the pool out from under
> the subtasks that are still working. Currently, I’m using a
> reference-counting hack that’s pretty brittle and unsatisfying. Are there
> any plans to add a user-definable job-level cleanup interface?
>
> Thanks
>
> Andrew
> --
> *Confidentiality Notice: The information contained in this e-mail and any
> attachments may be confidential. If you are not an intended recipient, you
> are hereby notified that any dissemination, distribution or copying of this
> e-mail is strictly prohibited. If you have received this e-mail in error,
> please notify the sender and permanently delete the e-mail and any
> attachments immediately. You should not retain, copy or use this e-mail or
> any attachment for any purpose, nor disclose all or any part of the
> contents to any other person. Thank you.*
>