You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@tez.apache.org by Johannes Zillmann <jz...@googlemail.com> on 2014/07/29 15:29:29 UTC
tez.am.client.am.port-range
Question about 'tez.am.client.am.port-range’.
Seems to have no effect setting this before client initialization.
Debugged DAGClientServer and the setting comes through properly and is handed over to new RPC.Builder(conf)….setPortRangeConfig()...build()
So i’m guessing this is a bug with the RPC lib !? Can anybody confirm ?
Johannes
Re: tez.am.client.am.port-range
Posted by "Jianfeng (Jeff) Zhang" <jz...@hortonworks.com>.
Hi Johannes,
setPortRangeConfig is for setting the port range for RPC server which has
its own threads ( including listener, handlers ). DAGClientServer is in
another thread which is not shared with thread of RPC server. Check the
following code in DAGClientServer.java, we will get the bindAddress ( host
& port ) and register it to RM (YarnTaskSchedulerService). Then the client
could get the bindAddress by getting ApplicationReport (DAGClientRPCImpl),
and then use it to connect to AM. Let me know if it is still not clear for
you.
// DAGClientServer.java
server.start();
bindAddress = NetUtils.getConnectAddress(server);
LOG.info("Instantiated DAGClientRPCServer at " + bindAddress);
Best Regards,
Jeff Zhang
On Tue, Jul 29, 2014 at 6:29 AM, Johannes Zillmann <jzillmann@googlemail.com
> wrote:
> Question about 'tez.am.client.am.port-range’.
>
> Seems to have no effect setting this before client initialization.
> Debugged DAGClientServer and the setting comes through properly and is
> handed over to new RPC.Builder(conf)….setPortRangeConfig()...build()
>
> So i’m guessing this is a bug with the RPC lib !? Can anybody confirm ?
>
> Johannes
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: tez.am.client.am.port-range
Posted by "Jianfeng (Jeff) Zhang" <jz...@hortonworks.com>.
It looks like we are not using the RPC library correctly, When building
Server, the portRangeConfig should be the name of port range configuration
rather than the value.
BTW, the RPC builder api is indeed a little confusing, the first impression
of the portRangeConfig would be the value for most of people IMO.
RPC.Server server = new RPC.Builder(conf).setProtocol(pbProtocol)
.setInstance(blockingService).setBindAddress(addr.getHostName())
.setPort(addr.getPort()).setNumHandlers(numHandlers).setVerbose(
false)
.setPortRangeConfig(portRangeConfig) // should be name of
portRange configuration
.setSecretManager(secretManager)
.build();
Best Regards,
Jeff Zhang
On Wed, Jul 30, 2014 at 7:01 AM, Johannes Zillmann <jzillmann@googlemail.com
> wrote:
> Thanks for response, created
> https://issues.apache.org/jira/browse/TEZ-1342
>
> Johannes
>
> On 30 Jul 2014, at 07:48, Siddharth Seth <ss...@apache.org> wrote:
>
> > Tried this out (with Hadoop-2.4 libraries included with Tez); I see the
> same behaviour where the port range config is being completely ignored; the
> correct value is available in the DAGClientServer.
> > @Johannes, could you please open a TEZ jira - so that we can track this
> for Tez. On the Hadoop side - for one thing, better documentation is
> required on how this parameter needs to be specified. This is used by the
> MapReduce AppMaster as well - and as far as I know does work. This does
> need verification though.
> >
> > @Jeff, I believe this value is supposed to be used when starting up the
> RPC server - once that is setup, we fetch the relevant port information and
> register it with the RM so that the Client gets it in an application report.
> >
> >
> > On Tue, Jul 29, 2014 at 6:29 AM, Johannes Zillmann <
> jzillmann@googlemail.com> wrote:
> > Question about 'tez.am.client.am.port-range’.
> >
> > Seems to have no effect setting this before client initialization.
> > Debugged DAGClientServer and the setting comes through properly and is
> handed over to new RPC.Builder(conf)….setPortRangeConfig()...build()
> >
> > So i’m guessing this is a bug with the RPC lib !? Can anybody confirm ?
> >
> > Johannes
> >
>
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: tez.am.client.am.port-range
Posted by Johannes Zillmann <jz...@googlemail.com>.
Thanks for response, created https://issues.apache.org/jira/browse/TEZ-1342
Johannes
On 30 Jul 2014, at 07:48, Siddharth Seth <ss...@apache.org> wrote:
> Tried this out (with Hadoop-2.4 libraries included with Tez); I see the same behaviour where the port range config is being completely ignored; the correct value is available in the DAGClientServer.
> @Johannes, could you please open a TEZ jira - so that we can track this for Tez. On the Hadoop side - for one thing, better documentation is required on how this parameter needs to be specified. This is used by the MapReduce AppMaster as well - and as far as I know does work. This does need verification though.
>
> @Jeff, I believe this value is supposed to be used when starting up the RPC server - once that is setup, we fetch the relevant port information and register it with the RM so that the Client gets it in an application report.
>
>
> On Tue, Jul 29, 2014 at 6:29 AM, Johannes Zillmann <jz...@googlemail.com> wrote:
> Question about 'tez.am.client.am.port-range’.
>
> Seems to have no effect setting this before client initialization.
> Debugged DAGClientServer and the setting comes through properly and is handed over to new RPC.Builder(conf)….setPortRangeConfig()...build()
>
> So i’m guessing this is a bug with the RPC lib !? Can anybody confirm ?
>
> Johannes
>
Re: tez.am.client.am.port-range
Posted by Siddharth Seth <ss...@apache.org>.
Tried this out (with Hadoop-2.4 libraries included with Tez); I see the
same behaviour where the port range config is being completely ignored; the
correct value is available in the DAGClientServer.
@Johannes, could you please open a TEZ jira - so that we can track this for
Tez. On the Hadoop side - for one thing, better documentation is required
on how this parameter needs to be specified. This is used by the MapReduce
AppMaster as well - and as far as I know does work. This does need
verification though.
@Jeff, I believe this value is supposed to be used when starting up the RPC
server - once that is setup, we fetch the relevant port information and
register it with the RM so that the Client gets it in an application report.
On Tue, Jul 29, 2014 at 6:29 AM, Johannes Zillmann <jzillmann@googlemail.com
> wrote:
> Question about 'tez.am.client.am.port-range’.
>
> Seems to have no effect setting this before client initialization.
> Debugged DAGClientServer and the setting comes through properly and is
> handed over to new RPC.Builder(conf)….setPortRangeConfig()...build()
>
> So i’m guessing this is a bug with the RPC lib !? Can anybody confirm ?
>
> Johannes