You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Kamal Mittal via user <us...@flink.apache.org> on 2023/08/02 02:49:28 UTC
Flink operator task opens threads internally
Hello Community,
I have a operator pipeline like as below, is it ok if "source" task opens threads by using java thread pool and parallelize the work?
This is needed for accepting multiple client socket connections in "single custom source server socket function".
Single Custom source server socket function - ServerSocket serversocket = new ServerSocket(9999);
Now using thread pool accept multiple connections in separate threads = new Runnable () -> serversocket.accept();
[cid:image001.png@01D9C51A.0C833FD0]
Rgds,
Kamal
RE: Flink operator task opens threads internally
Posted by Kamal Mittal via user <us...@flink.apache.org>.
Thanks for info.
Attached the POC code for reference. Class ServerCustomSocketStreamFunction.java is custom server socket source and class FlinkClientApp.java is main program.
From: liu ron <ro...@gmail.com>
Sent: 05 August 2023 02:28 PM
To: Kamal Mittal <ka...@ericsson.com>
Cc: Paul Lam <pa...@gmail.com>; user@flink.apache.org
Subject: Re: Flink operator task opens threads internally
Hi, Kamal
Based on your context, it is okay, but the server socket function must be singleton. Otherwise, there will be port conflict problems.
One more question, would you be so kind as to provide the PoC code? Maybe it helps to us share more views.
Best,
Ron
Kamal Mittal via user <us...@flink.apache.org>> 于2023年8月5日周六 10:50写道：
Hello,
Can you please share views for below?
Rgds,
Kamal
From: Kamal Mittal <ka...@ericsson.com>>
Sent: 04 August 2023 11:19 AM
To: Kamal Mittal <ka...@ericsson.com>>; Paul Lam <pa...@gmail.com>>
Cc: user@flink.apache.org<ma...@flink.apache.org>
Subject: RE: Flink operator task opens threads internally
Hello,
I hope you have the context of use case clear now as described in below mail as well as here again –
Flink application will be opening a server socket as part of custom source over a fixed port and client will be opening client sockets for Flink server socket. To read data from each client socket, Flink server socket will open a thread after accepting the connection which can be 1000 in nos.
Flink server socket application is running over K8s environment.
Do let me know please that it is ok to open multiple threads for accepting client socket connections inside flink application?
Rgds,
Kamal
From: Kamal Mittal via user <us...@flink.apache.org>>
Sent: 03 August 2023 09:27 AM
To: Paul Lam <pa...@gmail.com>>
Cc: user@flink.apache.org<ma...@flink.apache.org>
Subject: RE: Flink operator task opens threads internally
Hello,
We have a client sending TCP traffic towards Flink application and to support that there is server socket (Configurable port) opened which is accepting socket connections.
So to accept multiple client connections we have used a thread pool (configurable size) which will execute in a flink task slot.
Rgds,
KAmal
From: Paul Lam <pa...@gmail.com>>
Sent: 03 August 2023 08:59 AM
To: Kamal Mittal <ka...@ericsson.com>>
Cc: user@flink.apache.org<ma...@flink.apache.org>
Subject: Re: Flink operator task opens threads internally
Hi Kamal,
It’s okay if you don’t mind the data order.
But it’s not very commonly seen to accept client sockets from Flink jobs, as
the socket server address is dynamic and requires service discovery.
Would you like to share more about the background?
Best,
Paul Lam
2023年8月3日 10:26，Kamal Mittal via user <us...@flink.apache.org>> 写道：
Hello Community,
Please share views for the below mail.
Rgds,
Kamal
From: Kamal Mittal via user <us...@flink.apache.org>>
Sent: 02 August 2023 08:19 AM
To: user@flink.apache.org<ma...@flink.apache.org>
Subject: Flink operator task opens threads internally
Hello Community,
I have a operator pipeline like as below, is it ok if “source” task opens threads by using java thread pool and parallelize the work?
This is needed for accepting multiple client socket connections in “single custom source server socket function”.
Single Custom source server socket function – ServerSocket serversocket = new ServerSocket(9999);
Now using thread pool accept multiple connections in separate threads = new Runnable () -> serversocket.accept();
<image001.png>
Rgds,
Kamal
Re: Flink operator task opens threads internally
Posted by liu ron <ro...@gmail.com>.
Hi, Kamal
Based on your context, it is okay, but the server socket function must be
singleton. Otherwise, there will be port conflict problems.
One more question, would you be so kind as to provide the PoC code? Maybe
it helps to us share more views.
Best,
Ron
Kamal Mittal via user <us...@flink.apache.org> 于2023年8月5日周六 10:50写道：
> Hello,
>
>
>
> Can you please share views for below?
>
>
>
> Rgds,
>
> Kamal
>
>
>
> *From:* Kamal Mittal <ka...@ericsson.com>
> *Sent:* 04 August 2023 11:19 AM
> *To:* Kamal Mittal <ka...@ericsson.com>; Paul Lam <
> paullin3280@gmail.com>
> *Cc:* user@flink.apache.org
> *Subject:* RE: Flink operator task opens threads internally
>
>
>
> Hello,
>
>
>
> I hope you have the context of use case clear now as described in below
> mail as well as here again –
>
>
>
> Flink application will be opening a server socket as part of custom source
> over a fixed port and client will be opening client sockets for Flink
> server socket. To read data from each client socket, Flink server socket
> will open a thread after accepting the connection which can be 1000 in nos.
>
>
>
> Flink server socket application is running over K8s environment.
>
>
>
> Do let me know please that it is ok to open multiple threads for accepting
> client socket connections inside flink application?
>
>
>
> Rgds,
>
> Kamal
>
>
>
> *From:* Kamal Mittal via user <us...@flink.apache.org>
> *Sent:* 03 August 2023 09:27 AM
> *To:* Paul Lam <pa...@gmail.com>
> *Cc:* user@flink.apache.org
> *Subject:* RE: Flink operator task opens threads internally
>
>
>
> Hello,
>
>
>
> We have a client sending TCP traffic towards Flink application and to
> support that there is server socket (Configurable port) opened which is
> accepting socket connections.
>
>
>
> So to accept multiple client connections we have used a thread pool
> (configurable size) which will execute in a flink task slot.
>
>
>
> Rgds,
>
> KAmal
>
>
>
> *From:* Paul Lam <pa...@gmail.com>
> *Sent:* 03 August 2023 08:59 AM
> *To:* Kamal Mittal <ka...@ericsson.com>
> *Cc:* user@flink.apache.org
> *Subject:* Re: Flink operator task opens threads internally
>
>
>
> Hi Kamal,
>
>
>
> It’s okay if you don’t mind the data order.
>
>
>
> But it’s not very commonly seen to accept client sockets from Flink jobs,
> as
>
> the socket server address is dynamic and requires service discovery.
>
>
>
> Would you like to share more about the background?
>
>
> Best,
>
> Paul Lam
>
>
>
> 2023年8月3日 10:26，Kamal Mittal via user <us...@flink.apache.org> 写道：
>
>
>
> Hello Community,
>
>
>
> Please share views for the below mail.
>
>
>
> Rgds,
>
> Kamal
>
>
>
> *From:* Kamal Mittal via user <us...@flink.apache.org>
> *Sent:* 02 August 2023 08:19 AM
> *To:* user@flink.apache.org
> *Subject:* Flink operator task opens threads internally
>
>
>
> Hello Community,
>
>
>
> I have a operator pipeline like as below, is it ok if “source” task opens
> threads by using java thread pool and parallelize the work?
>
>
>
> This is needed for accepting multiple client socket connections in “single
> custom source server socket function”.
>
>
>
> Single Custom source server socket function – ServerSocket *serversocket* =
> new ServerSocket(9999);
>
> Now using thread pool accept multiple connections in separate threads = new
> *Runnable* () -> *serversocket*.accept();
>
>
>
>
>
> <image001.png>
>
>
>
> Rgds,
>
> Kamal
>
>
>
RE: Flink operator task opens threads internally
Posted by Kamal Mittal via user <us...@flink.apache.org>.
Hello,
Can you please share views for below?
Rgds,
Kamal
From: Kamal Mittal <ka...@ericsson.com>
Sent: 04 August 2023 11:19 AM
To: Kamal Mittal <ka...@ericsson.com>; Paul Lam <pa...@gmail.com>
Cc: user@flink.apache.org
Subject: RE: Flink operator task opens threads internally
Hello,
I hope you have the context of use case clear now as described in below mail as well as here again –
Flink application will be opening a server socket as part of custom source over a fixed port and client will be opening client sockets for Flink server socket. To read data from each client socket, Flink server socket will open a thread after accepting the connection which can be 1000 in nos.
Flink server socket application is running over K8s environment.
Do let me know please that it is ok to open multiple threads for accepting client socket connections inside flink application?
Rgds,
Kamal
From: Kamal Mittal via user <us...@flink.apache.org>>
Sent: 03 August 2023 09:27 AM
To: Paul Lam <pa...@gmail.com>>
Cc: user@flink.apache.org<ma...@flink.apache.org>
Subject: RE: Flink operator task opens threads internally
Hello,
We have a client sending TCP traffic towards Flink application and to support that there is server socket (Configurable port) opened which is accepting socket connections.
So to accept multiple client connections we have used a thread pool (configurable size) which will execute in a flink task slot.
Rgds,
KAmal
From: Paul Lam <pa...@gmail.com>>
Sent: 03 August 2023 08:59 AM
To: Kamal Mittal <ka...@ericsson.com>>
Cc: user@flink.apache.org<ma...@flink.apache.org>
Subject: Re: Flink operator task opens threads internally
Hi Kamal,
It’s okay if you don’t mind the data order.
But it’s not very commonly seen to accept client sockets from Flink jobs, as
the socket server address is dynamic and requires service discovery.
Would you like to share more about the background?
Best,
Paul Lam
2023年8月3日 10:26，Kamal Mittal via user <us...@flink.apache.org>> 写道：
Hello Community,
Please share views for the below mail.
Rgds,
Kamal
From: Kamal Mittal via user <us...@flink.apache.org>>
Sent: 02 August 2023 08:19 AM
To: user@flink.apache.org<ma...@flink.apache.org>
Subject: Flink operator task opens threads internally
Hello Community,
I have a operator pipeline like as below, is it ok if “source” task opens threads by using java thread pool and parallelize the work?
This is needed for accepting multiple client socket connections in “single custom source server socket function”.
Single Custom source server socket function – ServerSocket serversocket = new ServerSocket(9999);
Now using thread pool accept multiple connections in separate threads = new Runnable () -> serversocket.accept();
<image001.png>
Rgds,
Kamal
RE: Flink operator task opens threads internally
Posted by Kamal Mittal via user <us...@flink.apache.org>.
Hello,
I hope you have the context of use case clear now as described in below mail as well as here again –
Flink application will be opening a server socket as part of custom source over a fixed port and client will be opening client sockets for Flink server socket. To read data from each client socket, Flink server socket will open a thread after accepting the connection which can be 1000 in nos.
Flink server socket application is running over K8s environment.
Do let me know please that it is ok to open multiple threads for accepting client socket connections inside flink application?
Rgds,
Kamal
From: Kamal Mittal via user <us...@flink.apache.org>
Sent: 03 August 2023 09:27 AM
To: Paul Lam <pa...@gmail.com>
Cc: user@flink.apache.org
Subject: RE: Flink operator task opens threads internally
Hello,
We have a client sending TCP traffic towards Flink application and to support that there is server socket (Configurable port) opened which is accepting socket connections.
So to accept multiple client connections we have used a thread pool (configurable size) which will execute in a flink task slot.
Rgds,
KAmal
From: Paul Lam <pa...@gmail.com>>
Sent: 03 August 2023 08:59 AM
To: Kamal Mittal <ka...@ericsson.com>>
Cc: user@flink.apache.org<ma...@flink.apache.org>
Subject: Re: Flink operator task opens threads internally
Hi Kamal,
It’s okay if you don’t mind the data order.
But it’s not very commonly seen to accept client sockets from Flink jobs, as
the socket server address is dynamic and requires service discovery.
Would you like to share more about the background?
Best,
Paul Lam
2023年8月3日 10:26，Kamal Mittal via user <us...@flink.apache.org>> 写道：
Hello Community,
Please share views for the below mail.
Rgds,
Kamal
From: Kamal Mittal via user <us...@flink.apache.org>>
Sent: 02 August 2023 08:19 AM
To: user@flink.apache.org<ma...@flink.apache.org>
Subject: Flink operator task opens threads internally
Hello Community,
I have a operator pipeline like as below, is it ok if “source” task opens threads by using java thread pool and parallelize the work?
This is needed for accepting multiple client socket connections in “single custom source server socket function”.
Single Custom source server socket function – ServerSocket serversocket = new ServerSocket(9999);
Now using thread pool accept multiple connections in separate threads = new Runnable () -> serversocket.accept();
<image001.png>
Rgds,
Kamal
RE: Flink operator task opens threads internally
Posted by Kamal Mittal via user <us...@flink.apache.org>.
Hello,
We have a client sending TCP traffic towards Flink application and to support that there is server socket (Configurable port) opened which is accepting socket connections.
So to accept multiple client connections we have used a thread pool (configurable size) which will execute in a flink task slot.
Rgds,
KAmal
From: Paul Lam <pa...@gmail.com>
Sent: 03 August 2023 08:59 AM
To: Kamal Mittal <ka...@ericsson.com>
Cc: user@flink.apache.org
Subject: Re: Flink operator task opens threads internally
Hi Kamal,
It’s okay if you don’t mind the data order.
But it’s not very commonly seen to accept client sockets from Flink jobs, as
the socket server address is dynamic and requires service discovery.
Would you like to share more about the background?
Best,
Paul Lam
2023年8月3日 10:26，Kamal Mittal via user <us...@flink.apache.org>> 写道：
Hello Community,
Please share views for the below mail.
Rgds,
Kamal
From: Kamal Mittal via user <us...@flink.apache.org>>
Sent: 02 August 2023 08:19 AM
To: user@flink.apache.org<ma...@flink.apache.org>
Subject: Flink operator task opens threads internally
Hello Community,
I have a operator pipeline like as below, is it ok if “source” task opens threads by using java thread pool and parallelize the work?
This is needed for accepting multiple client socket connections in “single custom source server socket function”.
Single Custom source server socket function – ServerSocket serversocket = new ServerSocket(9999);
Now using thread pool accept multiple connections in separate threads = new Runnable () -> serversocket.accept();
<image001.png>
Rgds,
Kamal
Re: Flink operator task opens threads internally
Posted by Paul Lam <pa...@gmail.com>.
Hi Kamal,
It’s okay if you don’t mind the data order.
But it’s not very commonly seen to accept client sockets from Flink jobs, as
the socket server address is dynamic and requires service discovery.
Would you like to share more about the background?
Best,
Paul Lam
> 2023年8月3日 10:26，Kamal Mittal via user <us...@flink.apache.org> 写道：
>
> Hello Community,
>
> Please share views for the below mail.
>
> Rgds,
> Kamal
>
> From: Kamal Mittal via user <us...@flink.apache.org>
> Sent: 02 August 2023 08:19 AM
> To: user@flink.apache.org
> Subject: Flink operator task opens threads internally
>
> Hello Community,
>
> I have a operator pipeline like as below, is it ok if “source” task opens threads by using java thread pool and parallelize the work?
>
> This is needed for accepting multiple client socket connections in “single custom source server socket function”.
>
> Single Custom source server socket function – ServerSocket serversocket = new ServerSocket(9999);
> Now using thread pool accept multiple connections in separate threads = new Runnable () -> serversocket.accept();
>
>
> <image001.png>
>
> Rgds,
> Kamal
RE: Flink operator task opens threads internally
Posted by Kamal Mittal via user <us...@flink.apache.org>.
Hello Community,
Please share views for the below mail.
Rgds,
Kamal
From: Kamal Mittal via user <us...@flink.apache.org>
Sent: 02 August 2023 08:19 AM
To: user@flink.apache.org
Subject: Flink operator task opens threads internally
Hello Community,
I have a operator pipeline like as below, is it ok if "source" task opens threads by using java thread pool and parallelize the work?
This is needed for accepting multiple client socket connections in "single custom source server socket function".
Single Custom source server socket function - ServerSocket serversocket = new ServerSocket(9999);
Now using thread pool accept multiple connections in separate threads = new Runnable () -> serversocket.accept();
[cid:image001.png@01D9C5DF.F6FDD730]
Rgds,
Kamal