You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by sidhant gupta <si...@gmail.com> on 2020/08/24 14:01:12 UTC
How jobmanager and task manager communicates with each other ?
Hi User
How jobmanager and task manager communicates with each other ? How to set
connection between jobmanager and task manager running in different/same
ec2 instance ? Is it http or tcp ? How the service discovery works ?
Thanks
Sidhant Gupta
Re: How jobmanager and task manager communicates with each other ?
Posted by Andrey Zagrebin <az...@apache.org>.
Hi Sidhant,
(1) If we are not using Flink's HA services then how we can dynamically
> configure task manager nodes to connect to job manager? Any suggestions or
> best practices?
Not sure what you mean by 'dynamically'.
I think you have to restart the task manager with the new configuration
to connect to another job manager.
(2) Which and how flink's HA service can be used for the service discovery
> of job manager ?
You can check the docs for the zookeeper implementation of the HA in Flink
[1]
Best,
Andrey
[1]
https://ci.apache.org/projects/flink/flink-docs-release-1.11/ops/jobmanager_high_availability.html
On Tue, Aug 25, 2020 at 5:45 PM sidhant gupta <si...@gmail.com> wrote:
> Hi Till,
>
> Thanks for the reply.
>
> (1) If we are not using Flink's HA services then how we can dynamically
> configure task manager nodes to connect to job manager? Any suggestions or
> best practices?
>
> (2) Which and how flink's HA service can be used for the service discovery
> of job manager ?
>
> Regards
> Sidhant Gupta
>
>
> On Tue, Aug 25, 2020, 11:51 AM Till Rohrmann <tr...@apache.org> wrote:
>
>> Hi Sidhant,
>>
>> the cluster components use tcp to communicate with each other. If you are
>> not using Flink's HA services, then the TaskManager nodes need to be
>> configured with the JobManager's address to connect to them. If you are
>> using HA services, then the service discovery happens through the HA
>> services. One requirement for Flink to work is that the different cluster
>> nodes on which a Flink process is started can communicate with each other.
>>
>> Cheers,
>> Till
>>
>> On Mon, Aug 24, 2020 at 6:26 PM sidhant gupta <si...@gmail.com>
>> wrote:
>>
>>> ++dev@flink.apache.org
>>>
>>> On Mon, Aug 24, 2020, 7:31 PM sidhant gupta <si...@gmail.com> wrote:
>>>
>>> > Hi User
>>> >
>>> > How jobmanager and task manager communicates with each other ? How to
>>> set
>>> > connection between jobmanager and task manager running in
>>> different/same
>>> > ec2 instance ? Is it http or tcp ? How the service discovery works ?
>>> >
>>> > Thanks
>>> > Sidhant Gupta
>>> >
>>>
>>
Re: How jobmanager and task manager communicates with each other ?
Posted by Andrey Zagrebin <az...@apache.org>.
Hi Sidhant,
(1) If we are not using Flink's HA services then how we can dynamically
> configure task manager nodes to connect to job manager? Any suggestions or
> best practices?
Not sure what you mean by 'dynamically'.
I think you have to restart the task manager with the new configuration
to connect to another job manager.
(2) Which and how flink's HA service can be used for the service discovery
> of job manager ?
You can check the docs for the zookeeper implementation of the HA in Flink
[1]
Best,
Andrey
[1]
https://ci.apache.org/projects/flink/flink-docs-release-1.11/ops/jobmanager_high_availability.html
On Tue, Aug 25, 2020 at 5:45 PM sidhant gupta <si...@gmail.com> wrote:
> Hi Till,
>
> Thanks for the reply.
>
> (1) If we are not using Flink's HA services then how we can dynamically
> configure task manager nodes to connect to job manager? Any suggestions or
> best practices?
>
> (2) Which and how flink's HA service can be used for the service discovery
> of job manager ?
>
> Regards
> Sidhant Gupta
>
>
> On Tue, Aug 25, 2020, 11:51 AM Till Rohrmann <tr...@apache.org> wrote:
>
>> Hi Sidhant,
>>
>> the cluster components use tcp to communicate with each other. If you are
>> not using Flink's HA services, then the TaskManager nodes need to be
>> configured with the JobManager's address to connect to them. If you are
>> using HA services, then the service discovery happens through the HA
>> services. One requirement for Flink to work is that the different cluster
>> nodes on which a Flink process is started can communicate with each other.
>>
>> Cheers,
>> Till
>>
>> On Mon, Aug 24, 2020 at 6:26 PM sidhant gupta <si...@gmail.com>
>> wrote:
>>
>>> ++dev@flink.apache.org
>>>
>>> On Mon, Aug 24, 2020, 7:31 PM sidhant gupta <si...@gmail.com> wrote:
>>>
>>> > Hi User
>>> >
>>> > How jobmanager and task manager communicates with each other ? How to
>>> set
>>> > connection between jobmanager and task manager running in
>>> different/same
>>> > ec2 instance ? Is it http or tcp ? How the service discovery works ?
>>> >
>>> > Thanks
>>> > Sidhant Gupta
>>> >
>>>
>>
Re: How jobmanager and task manager communicates with each other ?
Posted by sidhant gupta <si...@gmail.com>.
Hi Till,
Thanks for the reply.
(1) If we are not using Flink's HA services then how we can dynamically
configure task manager nodes to connect to job manager? Any suggestions or
best practices?
(2) Which and how flink's HA service can be used for the service discovery
of job manager ?
Regards
Sidhant Gupta
On Tue, Aug 25, 2020, 11:51 AM Till Rohrmann <tr...@apache.org> wrote:
> Hi Sidhant,
>
> the cluster components use tcp to communicate with each other. If you are
> not using Flink's HA services, then the TaskManager nodes need to be
> configured with the JobManager's address to connect to them. If you are
> using HA services, then the service discovery happens through the HA
> services. One requirement for Flink to work is that the different cluster
> nodes on which a Flink process is started can communicate with each other.
>
> Cheers,
> Till
>
> On Mon, Aug 24, 2020 at 6:26 PM sidhant gupta <si...@gmail.com> wrote:
>
>> ++dev@flink.apache.org
>>
>> On Mon, Aug 24, 2020, 7:31 PM sidhant gupta <si...@gmail.com> wrote:
>>
>> > Hi User
>> >
>> > How jobmanager and task manager communicates with each other ? How to
>> set
>> > connection between jobmanager and task manager running in different/same
>> > ec2 instance ? Is it http or tcp ? How the service discovery works ?
>> >
>> > Thanks
>> > Sidhant Gupta
>> >
>>
>
Re: How jobmanager and task manager communicates with each other ?
Posted by sidhant gupta <si...@gmail.com>.
Hi Till,
Thanks for the reply.
(1) If we are not using Flink's HA services then how we can dynamically
configure task manager nodes to connect to job manager? Any suggestions or
best practices?
(2) Which and how flink's HA service can be used for the service discovery
of job manager ?
Regards
Sidhant Gupta
On Tue, Aug 25, 2020, 11:51 AM Till Rohrmann <tr...@apache.org> wrote:
> Hi Sidhant,
>
> the cluster components use tcp to communicate with each other. If you are
> not using Flink's HA services, then the TaskManager nodes need to be
> configured with the JobManager's address to connect to them. If you are
> using HA services, then the service discovery happens through the HA
> services. One requirement for Flink to work is that the different cluster
> nodes on which a Flink process is started can communicate with each other.
>
> Cheers,
> Till
>
> On Mon, Aug 24, 2020 at 6:26 PM sidhant gupta <si...@gmail.com> wrote:
>
>> ++dev@flink.apache.org
>>
>> On Mon, Aug 24, 2020, 7:31 PM sidhant gupta <si...@gmail.com> wrote:
>>
>> > Hi User
>> >
>> > How jobmanager and task manager communicates with each other ? How to
>> set
>> > connection between jobmanager and task manager running in different/same
>> > ec2 instance ? Is it http or tcp ? How the service discovery works ?
>> >
>> > Thanks
>> > Sidhant Gupta
>> >
>>
>
Re: How jobmanager and task manager communicates with each other ?
Posted by Till Rohrmann <tr...@apache.org>.
Hi Sidhant,
the cluster components use tcp to communicate with each other. If you are
not using Flink's HA services, then the TaskManager nodes need to be
configured with the JobManager's address to connect to them. If you are
using HA services, then the service discovery happens through the HA
services. One requirement for Flink to work is that the different cluster
nodes on which a Flink process is started can communicate with each other.
Cheers,
Till
On Mon, Aug 24, 2020 at 6:26 PM sidhant gupta <si...@gmail.com> wrote:
> ++dev@flink.apache.org
>
> On Mon, Aug 24, 2020, 7:31 PM sidhant gupta <si...@gmail.com> wrote:
>
> > Hi User
> >
> > How jobmanager and task manager communicates with each other ? How to set
> > connection between jobmanager and task manager running in different/same
> > ec2 instance ? Is it http or tcp ? How the service discovery works ?
> >
> > Thanks
> > Sidhant Gupta
> >
>
Re: How jobmanager and task manager communicates with each other ?
Posted by Till Rohrmann <tr...@apache.org>.
Hi Sidhant,
the cluster components use tcp to communicate with each other. If you are
not using Flink's HA services, then the TaskManager nodes need to be
configured with the JobManager's address to connect to them. If you are
using HA services, then the service discovery happens through the HA
services. One requirement for Flink to work is that the different cluster
nodes on which a Flink process is started can communicate with each other.
Cheers,
Till
On Mon, Aug 24, 2020 at 6:26 PM sidhant gupta <si...@gmail.com> wrote:
> ++dev@flink.apache.org
>
> On Mon, Aug 24, 2020, 7:31 PM sidhant gupta <si...@gmail.com> wrote:
>
> > Hi User
> >
> > How jobmanager and task manager communicates with each other ? How to set
> > connection between jobmanager and task manager running in different/same
> > ec2 instance ? Is it http or tcp ? How the service discovery works ?
> >
> > Thanks
> > Sidhant Gupta
> >
>
Re: How jobmanager and task manager communicates with each other ?
Posted by sidhant gupta <si...@gmail.com>.
++dev@flink.apache.org
On Mon, Aug 24, 2020, 7:31 PM sidhant gupta <si...@gmail.com> wrote:
> Hi User
>
> How jobmanager and task manager communicates with each other ? How to set
> connection between jobmanager and task manager running in different/same
> ec2 instance ? Is it http or tcp ? How the service discovery works ?
>
> Thanks
> Sidhant Gupta
>
Re: How jobmanager and task manager communicates with each other ?
Posted by sidhant gupta <si...@gmail.com>.
++dev@flink.apache.org
On Mon, Aug 24, 2020, 7:31 PM sidhant gupta <si...@gmail.com> wrote:
> Hi User
>
> How jobmanager and task manager communicates with each other ? How to set
> connection between jobmanager and task manager running in different/same
> ec2 instance ? Is it http or tcp ? How the service discovery works ?
>
> Thanks
> Sidhant Gupta
>