You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Kamal Mittal via user <us...@flink.apache.org> on 2023/08/01 02:43:28 UTC
RE: Flink netty connector for TCP source
Thanks.
I looked at the link for custom data sources, one query here that how to make sure for Split enumerator to execute on Job Manager rather than at Task manager?
From: liu ron <ro...@gmail.com>
Sent: 31 July 2023 10:06 PM
To: user@flink.apache.org
Subject: Re: Flink netty connector for TCP source
Hi, Kamal
Currently, we don't provide the connector like TCP source in Flink main repo. If you need this connector, you can try to implement it refer to the FLIP-27 source docs
[1] https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/sources/
Best,
Ron
Shammon FY <zj...@gmail.com>> 于2023年7月27日周四 11:23写道：
I cannot find any information about netty source in flink website and it is not in the connector list[1], so I'm think that it is not supported by flink community
[1] https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/datastream/overview/
Best,
Shammon FY
On Thu, Jul 27, 2023 at 10:53 AM Kamal Mittal <ka...@ericsson.com>> wrote:
Hello Shammon,
Yes socket text stream I am aware of but was thinking if something like as ‘https://github.com/apache/bahir-flink/tree/master/flink-connector-netty<https://protect2.fireeye.com/v1/url?k=31323334-501d5122-313273af-454445555731-ffc853ff875d9ed3&q=1&e=30c3fa9e-7c08-4486-8017-0d78c3c714ac&u=https%3A%2F%2Fgithub.com%2Fapache%2Fbahir-flink%2Ftree%2Fmaster%2Fflink-connector-netty>’ is also supported by Flink?
Rgds,
Kamal
From: Shammon FY <zj...@gmail.com>>
Sent: 27 July 2023 08:15 AM
To: Kamal Mittal <ka...@ericsson.com>>
Cc: user@flink.apache.org<ma...@flink.apache.org>
Subject: Re: Flink netty connector for TCP source
Hi Kamal,
There's socket text stream in `DataStream` and you can refer to [1] for more details.
[1] https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/overview/#example-program
Best,
Shammon FY
On Wed, Jul 26, 2023 at 4:26 PM Kamal Mittal via user <us...@flink.apache.org>> wrote:
Hello,
Does flink provides netty connector for custom TCP source?
Any documentation details please share?
Rgds,
Kamal
Re: Flink netty connector for TCP source
Posted by Shammon FY <zj...@gmail.com>.
Yes, you are right !
On Thu, Aug 3, 2023 at 1:35 PM Kamal Mittal <ka...@ericsson.com>
wrote:
> Hello Shammon,
>
>
>
> As it is said one split enumerator for one source means multiple sub-tasks
> of that source (if parallelism >1) will use same split enumerator instance
> right?
>
>
>
> Rgds,
>
> Kamal
>
>
>
> *From:* Shammon FY <zj...@gmail.com>
> *Sent:* 03 August 2023 10:54 AM
> *To:* Kamal Mittal <ka...@ericsson.com>
> *Cc:* user@flink.apache.org
> *Subject:* Re: Flink netty connector for TCP source
>
>
>
> Hi Kamal,
>
>
>
> I think it depends on your practical application. In general, the built-in
> source in Flink such as kafka or hive will proactively fetch splits from
> source, instead of starting a service with a source service pushed over.
>
>
>
> Returning to the issue of port conflicts, you may need to check if the
> specified port in the split enumerator is being used by other services, or
> if random ports can be used. Generally speaking, there will only be one
> split enumerator for one source, but when the job master fails to restart,
> the split enumerator of the previous job may not be completely shutdown. If
> they use the same port, it may also cause conflicts.
>
>
>
> Best,
>
> Shammon FY
>
>
>
> On Thu, Aug 3, 2023 at 11:57 AM Kamal Mittal <ka...@ericsson.com>
> wrote:
>
> Hello Shammon,
>
>
>
> Please have a look for below and share views.
>
>
>
> Rgds,
>
> Kamal
>
>
>
> *From:* Kamal Mittal via user <us...@flink.apache.org>
> *Sent:* 02 August 2023 08:02 AM
> *To:* Shammon FY <zj...@gmail.com>; user@flink.apache.org
> *Subject:* RE: Flink netty connector for TCP source
>
>
>
> Thanks Shammon.
>
>
>
> Purpose of opening server socket in Split Enumerator was that it has only
> one instance per source and so the server socket too (port binding can
> happen only once). And then accepted Socket connections
> (serversocket.accept()) will act as splits which will be further processed
> by readers.
>
>
>
> Let me know please if there is issue you see in above understanding or
> some other way you can suggest. Issue is that server socket can only bind a
> port once over a machine node and that’s why we thought to open it in split
> enumerator which is called only once and per source.
>
>
>
> Rgds,
>
> Kamal
>
>
>
> *From:* Shammon FY <zj...@gmail.com>
> *Sent:* 02 August 2023 07:48 AM
> *To:* Kamal Mittal <ka...@ericsson.com>; user@flink.apache.org
> *Subject:* Re: Flink netty connector for TCP source
>
>
>
> Hi Kamal,
>
>
>
> It confuses me a little that what's the purpose of opening a server socket
> in SplitEnumerator? Currently there will be only one SplitEnumerator
> instance in JobManager for each one source not each source subtask. If
> there's only one source in your job, no matter how much parallelism this
> source has, there will be only one SplitEnumerator instance in JM which can
> connect and send events to the source subtasks.
>
>
>
> Additionally, if you want to read data from a netty source, you can
> implement your NettySplitEnumerator to get splits from netty source and
> assign them to a NettySourceReader which will read data from netty source
> according to the splits.
>
>
>
> Best,
>
> Shammon FY
>
>
>
>
>
> On Tue, Aug 1, 2023 at 12:02 PM Kamal Mittal via user <
> user@flink.apache.org> wrote:
>
> Hello,
>
>
>
> I created a custom server socket source in which opened server socket in
> split enumerator, source parallelism is =2 and it gives error while running
> - “Address is already in use” i.e. it tried to open two server sockets as
> per parallelism which is not correct as parallelism is applicable for
> source operator and not for split enumerator?
>
>
>
> Please correct me if above understanding is not correct.
>
>
>
> Rgds,
>
> Kamal
>
>
>
> *From:* Hang Ruan <ru...@gmail.com>
> *Sent:* 01 August 2023 08:55 AM
> *To:* Kamal Mittal <ka...@ericsson.com>
> *Cc:* liu ron <ro...@gmail.com>; user@flink.apache.org
> *Subject:* Re: Flink netty connector for TCP source
>
>
>
> Hi, Kamal.
>
>
>
> The SplitEnumerator is contained in the SourceCoordinator. They are only
> used in JM.
>
>
>
> Best,
>
> Hang
>
>
>
> Kamal Mittal via user <us...@flink.apache.org> 于2023年8月1日周二 10:43写道：
>
> Thanks.
>
>
>
> I looked at the link for custom data sources, one query here that how to
> make sure for Split enumerator to execute on Job Manager rather than at
> Task manager?
>
>
>
> *From:* liu ron <ro...@gmail.com>
> *Sent:* 31 July 2023 10:06 PM
> *To:* user@flink.apache.org
> *Subject:* Re: Flink netty connector for TCP source
>
>
>
> Hi, Kamal
>
> Currently, we don't provide the connector like TCP source in Flink main
> repo. If you need this connector, you can try to implement it refer to the
> FLIP-27 source docs
>
>
>
> [1]
> https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/sources/
>
>
>
> Best,
>
> Ron
>
>
>
> Shammon FY <zj...@gmail.com> 于2023年7月27日周四 11:23写道：
>
> I cannot find any information about netty source in flink website and it
> is not in the connector list[1], so I'm think that it is not supported by
> flink community
>
>
>
> [1]
> https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/datastream/overview/
>
>
>
> Best,
>
> Shammon FY
>
>
>
> On Thu, Jul 27, 2023 at 10:53 AM Kamal Mittal <ka...@ericsson.com>
> wrote:
>
> Hello Shammon,
>
>
>
> Yes socket text stream I am aware of but was thinking if something like as
> ‘https://github.com/apache/bahir-flink/tree/master/flink-connector-netty
> <https://protect2.fireeye.com/v1/url?k=31323334-501d5122-313273af-454445555731-ffc853ff875d9ed3&q=1&e=30c3fa9e-7c08-4486-8017-0d78c3c714ac&u=https%3A%2F%2Fgithub.com%2Fapache%2Fbahir-flink%2Ftree%2Fmaster%2Fflink-connector-netty>’
> is also supported by Flink?
>
>
>
> Rgds,
>
> Kamal
>
>
>
> *From:* Shammon FY <zj...@gmail.com>
> *Sent:* 27 July 2023 08:15 AM
> *To:* Kamal Mittal <ka...@ericsson.com>
> *Cc:* user@flink.apache.org
> *Subject:* Re: Flink netty connector for TCP source
>
>
>
> Hi Kamal,
>
>
>
> There's socket text stream in `DataStream` and you can refer to [1] for
> more details.
>
>
>
> [1]
> https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/overview/#example-program
>
>
>
> Best,
>
> Shammon FY
>
>
>
> On Wed, Jul 26, 2023 at 4:26 PM Kamal Mittal via user <
> user@flink.apache.org> wrote:
>
> Hello,
>
>
>
> Does flink provides netty connector for custom TCP source?
>
>
>
> Any documentation details please share?
>
>
>
> Rgds,
>
> Kamal
>
>
RE: Flink netty connector for TCP source
Posted by Kamal Mittal via user <us...@flink.apache.org>.
Hello Shammon,
As it is said one split enumerator for one source means multiple sub-tasks of that source (if parallelism >1) will use same split enumerator instance right?
Rgds,
Kamal
From: Shammon FY <zj...@gmail.com>
Sent: 03 August 2023 10:54 AM
To: Kamal Mittal <ka...@ericsson.com>
Cc: user@flink.apache.org
Subject: Re: Flink netty connector for TCP source
Hi Kamal,
I think it depends on your practical application. In general, the built-in source in Flink such as kafka or hive will proactively fetch splits from source, instead of starting a service with a source service pushed over.
Returning to the issue of port conflicts, you may need to check if the specified port in the split enumerator is being used by other services, or if random ports can be used. Generally speaking, there will only be one split enumerator for one source, but when the job master fails to restart, the split enumerator of the previous job may not be completely shutdown. If they use the same port, it may also cause conflicts.
Best,
Shammon FY
On Thu, Aug 3, 2023 at 11:57 AM Kamal Mittal <ka...@ericsson.com>> wrote:
Hello Shammon,
Please have a look for below and share views.
Rgds,
Kamal
From: Kamal Mittal via user <us...@flink.apache.org>>
Sent: 02 August 2023 08:02 AM
To: Shammon FY <zj...@gmail.com>>; user@flink.apache.org<ma...@flink.apache.org>
Subject: RE: Flink netty connector for TCP source
Thanks Shammon.
Purpose of opening server socket in Split Enumerator was that it has only one instance per source and so the server socket too (port binding can happen only once). And then accepted Socket connections (serversocket.accept()) will act as splits which will be further processed by readers.
Let me know please if there is issue you see in above understanding or some other way you can suggest. Issue is that server socket can only bind a port once over a machine node and that’s why we thought to open it in split enumerator which is called only once and per source.
Rgds,
Kamal
From: Shammon FY <zj...@gmail.com>>
Sent: 02 August 2023 07:48 AM
To: Kamal Mittal <ka...@ericsson.com>>; user@flink.apache.org<ma...@flink.apache.org>
Subject: Re: Flink netty connector for TCP source
Hi Kamal,
It confuses me a little that what's the purpose of opening a server socket in SplitEnumerator? Currently there will be only one SplitEnumerator instance in JobManager for each one source not each source subtask. If there's only one source in your job, no matter how much parallelism this source has, there will be only one SplitEnumerator instance in JM which can connect and send events to the source subtasks.
Additionally, if you want to read data from a netty source, you can implement your NettySplitEnumerator to get splits from netty source and assign them to a NettySourceReader which will read data from netty source according to the splits.
Best,
Shammon FY
On Tue, Aug 1, 2023 at 12:02 PM Kamal Mittal via user <us...@flink.apache.org>> wrote:
Hello,
I created a custom server socket source in which opened server socket in split enumerator, source parallelism is =2 and it gives error while running - “Address is already in use” i.e. it tried to open two server sockets as per parallelism which is not correct as parallelism is applicable for source operator and not for split enumerator?
Please correct me if above understanding is not correct.
Rgds,
Kamal
From: Hang Ruan <ru...@gmail.com>>
Sent: 01 August 2023 08:55 AM
To: Kamal Mittal <ka...@ericsson.com>>
Cc: liu ron <ro...@gmail.com>>; user@flink.apache.org<ma...@flink.apache.org>
Subject: Re: Flink netty connector for TCP source
Hi, Kamal.
The SplitEnumerator is contained in the SourceCoordinator. They are only used in JM.
Best,
Hang
Kamal Mittal via user <us...@flink.apache.org>> 于2023年8月1日周二 10:43写道：
Thanks.
I looked at the link for custom data sources, one query here that how to make sure for Split enumerator to execute on Job Manager rather than at Task manager?
From: liu ron <ro...@gmail.com>>
Sent: 31 July 2023 10:06 PM
To: user@flink.apache.org<ma...@flink.apache.org>
Subject: Re: Flink netty connector for TCP source
Hi, Kamal
Currently, we don't provide the connector like TCP source in Flink main repo. If you need this connector, you can try to implement it refer to the FLIP-27 source docs
[1] https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/sources/
Best,
Ron
Shammon FY <zj...@gmail.com>> 于2023年7月27日周四 11:23写道：
I cannot find any information about netty source in flink website and it is not in the connector list[1], so I'm think that it is not supported by flink community
[1] https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/datastream/overview/
Best,
Shammon FY
On Thu, Jul 27, 2023 at 10:53 AM Kamal Mittal <ka...@ericsson.com>> wrote:
Hello Shammon,
Yes socket text stream I am aware of but was thinking if something like as ‘https://github.com/apache/bahir-flink/tree/master/flink-connector-netty<https://protect2.fireeye.com/v1/url?k=31323334-501d5122-313273af-454445555731-ffc853ff875d9ed3&q=1&e=30c3fa9e-7c08-4486-8017-0d78c3c714ac&u=https%3A%2F%2Fgithub.com%2Fapache%2Fbahir-flink%2Ftree%2Fmaster%2Fflink-connector-netty>’ is also supported by Flink?
Rgds,
Kamal
From: Shammon FY <zj...@gmail.com>>
Sent: 27 July 2023 08:15 AM
To: Kamal Mittal <ka...@ericsson.com>>
Cc: user@flink.apache.org<ma...@flink.apache.org>
Subject: Re: Flink netty connector for TCP source
Hi Kamal,
There's socket text stream in `DataStream` and you can refer to [1] for more details.
[1] https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/overview/#example-program
Best,
Shammon FY
On Wed, Jul 26, 2023 at 4:26 PM Kamal Mittal via user <us...@flink.apache.org>> wrote:
Hello,
Does flink provides netty connector for custom TCP source?
Any documentation details please share?
Rgds,
Kamal
Re: Flink netty connector for TCP source
Posted by Shammon FY <zj...@gmail.com>.
Hi Kamal,
I think it depends on your practical application. In general, the built-in
source in Flink such as kafka or hive will proactively fetch splits from
source, instead of starting a service with a source service pushed over.
Returning to the issue of port conflicts, you may need to check if the
specified port in the split enumerator is being used by other services, or
if random ports can be used. Generally speaking, there will only be one
split enumerator for one source, but when the job master fails to restart,
the split enumerator of the previous job may not be completely shutdown. If
they use the same port, it may also cause conflicts.
Best,
Shammon FY
On Thu, Aug 3, 2023 at 11:57 AM Kamal Mittal <ka...@ericsson.com>
wrote:
> Hello Shammon,
>
>
>
> Please have a look for below and share views.
>
>
>
> Rgds,
>
> Kamal
>
>
>
> *From:* Kamal Mittal via user <us...@flink.apache.org>
> *Sent:* 02 August 2023 08:02 AM
> *To:* Shammon FY <zj...@gmail.com>; user@flink.apache.org
> *Subject:* RE: Flink netty connector for TCP source
>
>
>
> Thanks Shammon.
>
>
>
> Purpose of opening server socket in Split Enumerator was that it has only
> one instance per source and so the server socket too (port binding can
> happen only once). And then accepted Socket connections
> (serversocket.accept()) will act as splits which will be further processed
> by readers.
>
>
>
> Let me know please if there is issue you see in above understanding or
> some other way you can suggest. Issue is that server socket can only bind a
> port once over a machine node and that’s why we thought to open it in split
> enumerator which is called only once and per source.
>
>
>
> Rgds,
>
> Kamal
>
>
>
> *From:* Shammon FY <zj...@gmail.com>
> *Sent:* 02 August 2023 07:48 AM
> *To:* Kamal Mittal <ka...@ericsson.com>; user@flink.apache.org
> *Subject:* Re: Flink netty connector for TCP source
>
>
>
> Hi Kamal,
>
>
>
> It confuses me a little that what's the purpose of opening a server socket
> in SplitEnumerator? Currently there will be only one SplitEnumerator
> instance in JobManager for each one source not each source subtask. If
> there's only one source in your job, no matter how much parallelism this
> source has, there will be only one SplitEnumerator instance in JM which can
> connect and send events to the source subtasks.
>
>
>
> Additionally, if you want to read data from a netty source, you can
> implement your NettySplitEnumerator to get splits from netty source and
> assign them to a NettySourceReader which will read data from netty source
> according to the splits.
>
>
>
> Best,
>
> Shammon FY
>
>
>
>
>
> On Tue, Aug 1, 2023 at 12:02 PM Kamal Mittal via user <
> user@flink.apache.org> wrote:
>
> Hello,
>
>
>
> I created a custom server socket source in which opened server socket in
> split enumerator, source parallelism is =2 and it gives error while running
> - “Address is already in use” i.e. it tried to open two server sockets as
> per parallelism which is not correct as parallelism is applicable for
> source operator and not for split enumerator?
>
>
>
> Please correct me if above understanding is not correct.
>
>
>
> Rgds,
>
> Kamal
>
>
>
> *From:* Hang Ruan <ru...@gmail.com>
> *Sent:* 01 August 2023 08:55 AM
> *To:* Kamal Mittal <ka...@ericsson.com>
> *Cc:* liu ron <ro...@gmail.com>; user@flink.apache.org
> *Subject:* Re: Flink netty connector for TCP source
>
>
>
> Hi, Kamal.
>
>
>
> The SplitEnumerator is contained in the SourceCoordinator. They are only
> used in JM.
>
>
>
> Best,
>
> Hang
>
>
>
> Kamal Mittal via user <us...@flink.apache.org> 于2023年8月1日周二 10:43写道：
>
> Thanks.
>
>
>
> I looked at the link for custom data sources, one query here that how to
> make sure for Split enumerator to execute on Job Manager rather than at
> Task manager?
>
>
>
> *From:* liu ron <ro...@gmail.com>
> *Sent:* 31 July 2023 10:06 PM
> *To:* user@flink.apache.org
> *Subject:* Re: Flink netty connector for TCP source
>
>
>
> Hi, Kamal
>
> Currently, we don't provide the connector like TCP source in Flink main
> repo. If you need this connector, you can try to implement it refer to the
> FLIP-27 source docs
>
>
>
> [1]
> https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/sources/
>
>
>
> Best,
>
> Ron
>
>
>
> Shammon FY <zj...@gmail.com> 于2023年7月27日周四 11:23写道：
>
> I cannot find any information about netty source in flink website and it
> is not in the connector list[1], so I'm think that it is not supported by
> flink community
>
>
>
> [1]
> https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/datastream/overview/
>
>
>
> Best,
>
> Shammon FY
>
>
>
> On Thu, Jul 27, 2023 at 10:53 AM Kamal Mittal <ka...@ericsson.com>
> wrote:
>
> Hello Shammon,
>
>
>
> Yes socket text stream I am aware of but was thinking if something like as
> ‘https://github.com/apache/bahir-flink/tree/master/flink-connector-netty
> <https://protect2.fireeye.com/v1/url?k=31323334-501d5122-313273af-454445555731-ffc853ff875d9ed3&q=1&e=30c3fa9e-7c08-4486-8017-0d78c3c714ac&u=https%3A%2F%2Fgithub.com%2Fapache%2Fbahir-flink%2Ftree%2Fmaster%2Fflink-connector-netty>’
> is also supported by Flink?
>
>
>
> Rgds,
>
> Kamal
>
>
>
> *From:* Shammon FY <zj...@gmail.com>
> *Sent:* 27 July 2023 08:15 AM
> *To:* Kamal Mittal <ka...@ericsson.com>
> *Cc:* user@flink.apache.org
> *Subject:* Re: Flink netty connector for TCP source
>
>
>
> Hi Kamal,
>
>
>
> There's socket text stream in `DataStream` and you can refer to [1] for
> more details.
>
>
>
> [1]
> https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/overview/#example-program
>
>
>
> Best,
>
> Shammon FY
>
>
>
> On Wed, Jul 26, 2023 at 4:26 PM Kamal Mittal via user <
> user@flink.apache.org> wrote:
>
> Hello,
>
>
>
> Does flink provides netty connector for custom TCP source?
>
>
>
> Any documentation details please share?
>
>
>
> Rgds,
>
> Kamal
>
>
RE: Flink netty connector for TCP source
Posted by Kamal Mittal via user <us...@flink.apache.org>.
Hello Shammon,
Please have a look for below and share views.
Rgds,
Kamal
From: Kamal Mittal via user <us...@flink.apache.org>
Sent: 02 August 2023 08:02 AM
To: Shammon FY <zj...@gmail.com>; user@flink.apache.org
Subject: RE: Flink netty connector for TCP source
Thanks Shammon.
Purpose of opening server socket in Split Enumerator was that it has only one instance per source and so the server socket too (port binding can happen only once). And then accepted Socket connections (serversocket.accept()) will act as splits which will be further processed by readers.
Let me know please if there is issue you see in above understanding or some other way you can suggest. Issue is that server socket can only bind a port once over a machine node and that’s why we thought to open it in split enumerator which is called only once and per source.
Rgds,
Kamal
From: Shammon FY <zj...@gmail.com>>
Sent: 02 August 2023 07:48 AM
To: Kamal Mittal <ka...@ericsson.com>>; user@flink.apache.org<ma...@flink.apache.org>
Subject: Re: Flink netty connector for TCP source
Hi Kamal,
It confuses me a little that what's the purpose of opening a server socket in SplitEnumerator? Currently there will be only one SplitEnumerator instance in JobManager for each one source not each source subtask. If there's only one source in your job, no matter how much parallelism this source has, there will be only one SplitEnumerator instance in JM which can connect and send events to the source subtasks.
Additionally, if you want to read data from a netty source, you can implement your NettySplitEnumerator to get splits from netty source and assign them to a NettySourceReader which will read data from netty source according to the splits.
Best,
Shammon FY
On Tue, Aug 1, 2023 at 12:02 PM Kamal Mittal via user <us...@flink.apache.org>> wrote:
Hello,
I created a custom server socket source in which opened server socket in split enumerator, source parallelism is =2 and it gives error while running - “Address is already in use” i.e. it tried to open two server sockets as per parallelism which is not correct as parallelism is applicable for source operator and not for split enumerator?
Please correct me if above understanding is not correct.
Rgds,
Kamal
From: Hang Ruan <ru...@gmail.com>>
Sent: 01 August 2023 08:55 AM
To: Kamal Mittal <ka...@ericsson.com>>
Cc: liu ron <ro...@gmail.com>>; user@flink.apache.org<ma...@flink.apache.org>
Subject: Re: Flink netty connector for TCP source
Hi, Kamal.
The SplitEnumerator is contained in the SourceCoordinator. They are only used in JM.
Best,
Hang
Kamal Mittal via user <us...@flink.apache.org>> 于2023年8月1日周二 10:43写道：
Thanks.
I looked at the link for custom data sources, one query here that how to make sure for Split enumerator to execute on Job Manager rather than at Task manager?
From: liu ron <ro...@gmail.com>>
Sent: 31 July 2023 10:06 PM
To: user@flink.apache.org<ma...@flink.apache.org>
Subject: Re: Flink netty connector for TCP source
Hi, Kamal
Currently, we don't provide the connector like TCP source in Flink main repo. If you need this connector, you can try to implement it refer to the FLIP-27 source docs
[1] https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/sources/
Best,
Ron
Shammon FY <zj...@gmail.com>> 于2023年7月27日周四 11:23写道：
I cannot find any information about netty source in flink website and it is not in the connector list[1], so I'm think that it is not supported by flink community
[1] https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/datastream/overview/
Best,
Shammon FY
On Thu, Jul 27, 2023 at 10:53 AM Kamal Mittal <ka...@ericsson.com>> wrote:
Hello Shammon,
Yes socket text stream I am aware of but was thinking if something like as ‘https://github.com/apache/bahir-flink/tree/master/flink-connector-netty<https://protect2.fireeye.com/v1/url?k=31323334-501d5122-313273af-454445555731-ffc853ff875d9ed3&q=1&e=30c3fa9e-7c08-4486-8017-0d78c3c714ac&u=https%3A%2F%2Fgithub.com%2Fapache%2Fbahir-flink%2Ftree%2Fmaster%2Fflink-connector-netty>’ is also supported by Flink?
Rgds,
Kamal
From: Shammon FY <zj...@gmail.com>>
Sent: 27 July 2023 08:15 AM
To: Kamal Mittal <ka...@ericsson.com>>
Cc: user@flink.apache.org<ma...@flink.apache.org>
Subject: Re: Flink netty connector for TCP source
Hi Kamal,
There's socket text stream in `DataStream` and you can refer to [1] for more details.
[1] https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/overview/#example-program
Best,
Shammon FY
On Wed, Jul 26, 2023 at 4:26 PM Kamal Mittal via user <us...@flink.apache.org>> wrote:
Hello,
Does flink provides netty connector for custom TCP source?
Any documentation details please share?
Rgds,
Kamal
RE: Flink netty connector for TCP source
Posted by Kamal Mittal via user <us...@flink.apache.org>.
Thanks Shammon.
Purpose of opening server socket in Split Enumerator was that it has only one instance per source and so the server socket too (port binding can happen only once). And then accepted Socket connections (serversocket.accept()) will act as splits which will be further processed by readers.
Let me know please if there is issue you see in above understanding or some other way you can suggest. Issue is that server socket can only bind a port once over a machine node and that’s why we thought to open it in split enumerator which is called only once and per source.
Rgds,
Kamal
From: Shammon FY <zj...@gmail.com>
Sent: 02 August 2023 07:48 AM
To: Kamal Mittal <ka...@ericsson.com>; user@flink.apache.org
Subject: Re: Flink netty connector for TCP source
Hi Kamal,
It confuses me a little that what's the purpose of opening a server socket in SplitEnumerator? Currently there will be only one SplitEnumerator instance in JobManager for each one source not each source subtask. If there's only one source in your job, no matter how much parallelism this source has, there will be only one SplitEnumerator instance in JM which can connect and send events to the source subtasks.
Additionally, if you want to read data from a netty source, you can implement your NettySplitEnumerator to get splits from netty source and assign them to a NettySourceReader which will read data from netty source according to the splits.
Best,
Shammon FY
On Tue, Aug 1, 2023 at 12:02 PM Kamal Mittal via user <us...@flink.apache.org>> wrote:
Hello,
I created a custom server socket source in which opened server socket in split enumerator, source parallelism is =2 and it gives error while running - “Address is already in use” i.e. it tried to open two server sockets as per parallelism which is not correct as parallelism is applicable for source operator and not for split enumerator?
Please correct me if above understanding is not correct.
Rgds,
Kamal
From: Hang Ruan <ru...@gmail.com>>
Sent: 01 August 2023 08:55 AM
To: Kamal Mittal <ka...@ericsson.com>>
Cc: liu ron <ro...@gmail.com>>; user@flink.apache.org<ma...@flink.apache.org>
Subject: Re: Flink netty connector for TCP source
Hi, Kamal.
The SplitEnumerator is contained in the SourceCoordinator. They are only used in JM.
Best,
Hang
Kamal Mittal via user <us...@flink.apache.org>> 于2023年8月1日周二 10:43写道：
Thanks.
I looked at the link for custom data sources, one query here that how to make sure for Split enumerator to execute on Job Manager rather than at Task manager?
From: liu ron <ro...@gmail.com>>
Sent: 31 July 2023 10:06 PM
To: user@flink.apache.org<ma...@flink.apache.org>
Subject: Re: Flink netty connector for TCP source
Hi, Kamal
Currently, we don't provide the connector like TCP source in Flink main repo. If you need this connector, you can try to implement it refer to the FLIP-27 source docs
[1] https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/sources/
Best,
Ron
Shammon FY <zj...@gmail.com>> 于2023年7月27日周四 11:23写道：
I cannot find any information about netty source in flink website and it is not in the connector list[1], so I'm think that it is not supported by flink community
[1] https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/datastream/overview/
Best,
Shammon FY
On Thu, Jul 27, 2023 at 10:53 AM Kamal Mittal <ka...@ericsson.com>> wrote:
Hello Shammon,
Yes socket text stream I am aware of but was thinking if something like as ‘https://github.com/apache/bahir-flink/tree/master/flink-connector-netty<https://protect2.fireeye.com/v1/url?k=31323334-501d5122-313273af-454445555731-ffc853ff875d9ed3&q=1&e=30c3fa9e-7c08-4486-8017-0d78c3c714ac&u=https%3A%2F%2Fgithub.com%2Fapache%2Fbahir-flink%2Ftree%2Fmaster%2Fflink-connector-netty>’ is also supported by Flink?
Rgds,
Kamal
From: Shammon FY <zj...@gmail.com>>
Sent: 27 July 2023 08:15 AM
To: Kamal Mittal <ka...@ericsson.com>>
Cc: user@flink.apache.org<ma...@flink.apache.org>
Subject: Re: Flink netty connector for TCP source
Hi Kamal,
There's socket text stream in `DataStream` and you can refer to [1] for more details.
[1] https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/overview/#example-program
Best,
Shammon FY
On Wed, Jul 26, 2023 at 4:26 PM Kamal Mittal via user <us...@flink.apache.org>> wrote:
Hello,
Does flink provides netty connector for custom TCP source?
Any documentation details please share?
Rgds,
Kamal
Re: Flink netty connector for TCP source
Posted by Shammon FY <zj...@gmail.com>.
Hi Kamal,
It confuses me a little that what's the purpose of opening a server socket
in SplitEnumerator? Currently there will be only one SplitEnumerator
instance in JobManager for each one source not each source subtask. If
there's only one source in your job, no matter how much parallelism this
source has, there will be only one SplitEnumerator instance in JM which can
connect and send events to the source subtasks.
Additionally, if you want to read data from a netty source, you can
implement your NettySplitEnumerator to get splits from netty source and
assign them to a NettySourceReader which will read data from netty source
according to the splits.
Best,
Shammon FY
On Tue, Aug 1, 2023 at 12:02 PM Kamal Mittal via user <us...@flink.apache.org>
wrote:
> Hello,
>
>
>
> I created a custom server socket source in which opened server socket in
> split enumerator, source parallelism is =2 and it gives error while running
> - “Address is already in use” i.e. it tried to open two server sockets as
> per parallelism which is not correct as parallelism is applicable for
> source operator and not for split enumerator?
>
>
>
> Please correct me if above understanding is not correct.
>
>
>
> Rgds,
>
> Kamal
>
>
>
> *From:* Hang Ruan <ru...@gmail.com>
> *Sent:* 01 August 2023 08:55 AM
> *To:* Kamal Mittal <ka...@ericsson.com>
> *Cc:* liu ron <ro...@gmail.com>; user@flink.apache.org
> *Subject:* Re: Flink netty connector for TCP source
>
>
>
> Hi, Kamal.
>
>
>
> The SplitEnumerator is contained in the SourceCoordinator. They are only
> used in JM.
>
>
>
> Best,
>
> Hang
>
>
>
> Kamal Mittal via user <us...@flink.apache.org> 于2023年8月1日周二 10:43写道：
>
> Thanks.
>
>
>
> I looked at the link for custom data sources, one query here that how to
> make sure for Split enumerator to execute on Job Manager rather than at
> Task manager?
>
>
>
> *From:* liu ron <ro...@gmail.com>
> *Sent:* 31 July 2023 10:06 PM
> *To:* user@flink.apache.org
> *Subject:* Re: Flink netty connector for TCP source
>
>
>
> Hi, Kamal
>
> Currently, we don't provide the connector like TCP source in Flink main
> repo. If you need this connector, you can try to implement it refer to the
> FLIP-27 source docs
>
>
>
> [1]
> https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/sources/
>
>
>
> Best,
>
> Ron
>
>
>
> Shammon FY <zj...@gmail.com> 于2023年7月27日周四 11:23写道：
>
> I cannot find any information about netty source in flink website and it
> is not in the connector list[1], so I'm think that it is not supported by
> flink community
>
>
>
> [1]
> https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/datastream/overview/
>
>
>
> Best,
>
> Shammon FY
>
>
>
> On Thu, Jul 27, 2023 at 10:53 AM Kamal Mittal <ka...@ericsson.com>
> wrote:
>
> Hello Shammon,
>
>
>
> Yes socket text stream I am aware of but was thinking if something like as
> ‘https://github.com/apache/bahir-flink/tree/master/flink-connector-netty
> <https://protect2.fireeye.com/v1/url?k=31323334-501d5122-313273af-454445555731-ffc853ff875d9ed3&q=1&e=30c3fa9e-7c08-4486-8017-0d78c3c714ac&u=https%3A%2F%2Fgithub.com%2Fapache%2Fbahir-flink%2Ftree%2Fmaster%2Fflink-connector-netty>’
> is also supported by Flink?
>
>
>
> Rgds,
>
> Kamal
>
>
>
> *From:* Shammon FY <zj...@gmail.com>
> *Sent:* 27 July 2023 08:15 AM
> *To:* Kamal Mittal <ka...@ericsson.com>
> *Cc:* user@flink.apache.org
> *Subject:* Re: Flink netty connector for TCP source
>
>
>
> Hi Kamal,
>
>
>
> There's socket text stream in `DataStream` and you can refer to [1] for
> more details.
>
>
>
> [1]
> https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/overview/#example-program
>
>
>
> Best,
>
> Shammon FY
>
>
>
> On Wed, Jul 26, 2023 at 4:26 PM Kamal Mittal via user <
> user@flink.apache.org> wrote:
>
> Hello,
>
>
>
> Does flink provides netty connector for custom TCP source?
>
>
>
> Any documentation details please share?
>
>
>
> Rgds,
>
> Kamal
>
>
RE: Flink netty connector for TCP source
Posted by Kamal Mittal via user <us...@flink.apache.org>.
Hello,
I created a custom server socket source in which opened server socket in split enumerator, source parallelism is =2 and it gives error while running - “Address is already in use” i.e. it tried to open two server sockets as per parallelism which is not correct as parallelism is applicable for source operator and not for split enumerator?
Please correct me if above understanding is not correct.
Rgds,
Kamal
From: Hang Ruan <ru...@gmail.com>
Sent: 01 August 2023 08:55 AM
To: Kamal Mittal <ka...@ericsson.com>
Cc: liu ron <ro...@gmail.com>; user@flink.apache.org
Subject: Re: Flink netty connector for TCP source
Hi, Kamal.
The SplitEnumerator is contained in the SourceCoordinator. They are only used in JM.
Best,
Hang
Kamal Mittal via user <us...@flink.apache.org>> 于2023年8月1日周二 10:43写道：
Thanks.
I looked at the link for custom data sources, one query here that how to make sure for Split enumerator to execute on Job Manager rather than at Task manager?
From: liu ron <ro...@gmail.com>>
Sent: 31 July 2023 10:06 PM
To: user@flink.apache.org<ma...@flink.apache.org>
Subject: Re: Flink netty connector for TCP source
Hi, Kamal
Currently, we don't provide the connector like TCP source in Flink main repo. If you need this connector, you can try to implement it refer to the FLIP-27 source docs
[1] https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/sources/
Best,
Ron
Shammon FY <zj...@gmail.com>> 于2023年7月27日周四 11:23写道：
I cannot find any information about netty source in flink website and it is not in the connector list[1], so I'm think that it is not supported by flink community
[1] https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/datastream/overview/
Best,
Shammon FY
On Thu, Jul 27, 2023 at 10:53 AM Kamal Mittal <ka...@ericsson.com>> wrote:
Hello Shammon,
Yes socket text stream I am aware of but was thinking if something like as ‘https://github.com/apache/bahir-flink/tree/master/flink-connector-netty<https://protect2.fireeye.com/v1/url?k=31323334-501d5122-313273af-454445555731-ffc853ff875d9ed3&q=1&e=30c3fa9e-7c08-4486-8017-0d78c3c714ac&u=https%3A%2F%2Fgithub.com%2Fapache%2Fbahir-flink%2Ftree%2Fmaster%2Fflink-connector-netty>’ is also supported by Flink?
Rgds,
Kamal
From: Shammon FY <zj...@gmail.com>>
Sent: 27 July 2023 08:15 AM
To: Kamal Mittal <ka...@ericsson.com>>
Cc: user@flink.apache.org<ma...@flink.apache.org>
Subject: Re: Flink netty connector for TCP source
Hi Kamal,
There's socket text stream in `DataStream` and you can refer to [1] for more details.
[1] https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/overview/#example-program
Best,
Shammon FY
On Wed, Jul 26, 2023 at 4:26 PM Kamal Mittal via user <us...@flink.apache.org>> wrote:
Hello,
Does flink provides netty connector for custom TCP source?
Any documentation details please share?
Rgds,
Kamal
Re: Flink netty connector for TCP source
Posted by Hang Ruan <ru...@gmail.com>.
Hi, Kamal.
The SplitEnumerator is contained in the SourceCoordinator. They are only
used in JM.
Best,
Hang
Kamal Mittal via user <us...@flink.apache.org> 于2023年8月1日周二 10:43写道：
> Thanks.
>
>
>
> I looked at the link for custom data sources, one query here that how to
> make sure for Split enumerator to execute on Job Manager rather than at
> Task manager?
>
>
>
> *From:* liu ron <ro...@gmail.com>
> *Sent:* 31 July 2023 10:06 PM
> *To:* user@flink.apache.org
> *Subject:* Re: Flink netty connector for TCP source
>
>
>
> Hi, Kamal
>
> Currently, we don't provide the connector like TCP source in Flink main
> repo. If you need this connector, you can try to implement it refer to the
> FLIP-27 source docs
>
>
>
> [1]
> https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/sources/
>
>
>
> Best,
>
> Ron
>
>
>
> Shammon FY <zj...@gmail.com> 于2023年7月27日周四 11:23写道：
>
> I cannot find any information about netty source in flink website and it
> is not in the connector list[1], so I'm think that it is not supported by
> flink community
>
>
>
> [1]
> https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/datastream/overview/
>
>
>
> Best,
>
> Shammon FY
>
>
>
> On Thu, Jul 27, 2023 at 10:53 AM Kamal Mittal <ka...@ericsson.com>
> wrote:
>
> Hello Shammon,
>
>
>
> Yes socket text stream I am aware of but was thinking if something like as
> ‘https://github.com/apache/bahir-flink/tree/master/flink-connector-netty
> <https://protect2.fireeye.com/v1/url?k=31323334-501d5122-313273af-454445555731-ffc853ff875d9ed3&q=1&e=30c3fa9e-7c08-4486-8017-0d78c3c714ac&u=https%3A%2F%2Fgithub.com%2Fapache%2Fbahir-flink%2Ftree%2Fmaster%2Fflink-connector-netty>’
> is also supported by Flink?
>
>
>
> Rgds,
>
> Kamal
>
>
>
> *From:* Shammon FY <zj...@gmail.com>
> *Sent:* 27 July 2023 08:15 AM
> *To:* Kamal Mittal <ka...@ericsson.com>
> *Cc:* user@flink.apache.org
> *Subject:* Re: Flink netty connector for TCP source
>
>
>
> Hi Kamal,
>
>
>
> There's socket text stream in `DataStream` and you can refer to [1] for
> more details.
>
>
>
> [1]
> https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/overview/#example-program
>
>
>
> Best,
>
> Shammon FY
>
>
>
> On Wed, Jul 26, 2023 at 4:26 PM Kamal Mittal via user <
> user@flink.apache.org> wrote:
>
> Hello,
>
>
>
> Does flink provides netty connector for custom TCP source?
>
>
>
> Any documentation details please share?
>
>
>
> Rgds,
>
> Kamal
>
>