You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-user@hadoop.apache.org by Li Li <fa...@gmail.com> on 2015/12/08 10:27:59 UTC
nodemanager listen on 0.0.0.0
I want to setup a yarn cluster. It's not allowed to visit from
public network so I changed many configuration files to let all
service bind to the internal network address.
All thing is fine exception the nodemanager which bind
0.0.0.0:13562. I searched the default-mapred-default.xml and found it
mapreduce.shuffle.port.
Which configuration file should I change to let nodemanager listen
on internal address instead of 0.0.0.0?
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org
For additional commands, e-mail: user-help@hadoop.apache.org
Re: nodemanager listen on 0.0.0.0
Posted by Harsh J <ha...@cloudera.com>.
Thank you Varun, https://issues.apache.org/jira/browse/MAPREDUCE-5938 is
the one.
On Tue, Dec 8, 2015 at 10:02 PM Varun Saxena <vs...@gmail.com>
wrote:
> There is already a JIRA for it - MAPREDUCE-5938(and other related JIRAs'
> MAPREDUCE-6402 and YARN-4119)
>
> Regards,
> Varun Saxena
>
> On Tue, Dec 8, 2015 at 9:56 PM, Harsh J <ha...@cloudera.com> wrote:
>
>> Hello,
>>
>> Could you file a JIRA for this please? Currently the ShuffleHandler will
>> always bind to wildcard address due to the code being that way (in both
>> branch-2 and trunk presently:
>> https://github.com/apache/hadoop/blob/release-2.7.1/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/ShuffleHandler.java#L388
>> ).
>>
>> Your current workaround would be to use a firewall to drop out external
>> connections to that port.
>>
>> On Tue, Dec 8, 2015 at 9:30 PM Varun Vasudev <vv...@apache.org> wrote:
>>
>>> Hi Li,
>>>
>>> What have you set yarn.nodemanager.bind-host to? From
>>> https://hadoop.apache.org/docs/r2.7.1/hadoop-yarn/hadoop-yarn-common/yarn-default.xml
>>> -
>>>
>>> "The actual address the server will bind to. If this optional address is
>>> set, the RPC and webapp servers will bind to this address and the
>>> port specified in
>>> yarn.nodemanager.address and yarn.nodemanager.webapp.address,
>>> respectively. This is
>>> most useful for making NM listen to all interfaces by setting to
>>> 0.0.0.0”
>>>
>>> -Varun
>>>
>>>
>>>
>>> On 12/8/15, 2:57 PM, "Li Li" <fa...@gmail.com> wrote:
>>>
>>> > I want to setup a yarn cluster. It's not allowed to visit from
>>> >public network so I changed many configuration files to let all
>>> >service bind to the internal network address.
>>> > All thing is fine exception the nodemanager which bind
>>> >0.0.0.0:13562. I searched the default-mapred-default.xml and found it
>>> >mapreduce.shuffle.port.
>>> > Which configuration file should I change to let nodemanager listen
>>> >on internal address instead of 0.0.0.0?
>>> >
>>> >---------------------------------------------------------------------
>>> >To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org
>>> >For additional commands, e-mail: user-help@hadoop.apache.org
>>> >
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org
>>> For additional commands, e-mail: user-help@hadoop.apache.org
>>>
>>>
>
Re: nodemanager listen on 0.0.0.0
Posted by Harsh J <ha...@cloudera.com>.
Thank you Varun, https://issues.apache.org/jira/browse/MAPREDUCE-5938 is
the one.
On Tue, Dec 8, 2015 at 10:02 PM Varun Saxena <vs...@gmail.com>
wrote:
> There is already a JIRA for it - MAPREDUCE-5938(and other related JIRAs'
> MAPREDUCE-6402 and YARN-4119)
>
> Regards,
> Varun Saxena
>
> On Tue, Dec 8, 2015 at 9:56 PM, Harsh J <ha...@cloudera.com> wrote:
>
>> Hello,
>>
>> Could you file a JIRA for this please? Currently the ShuffleHandler will
>> always bind to wildcard address due to the code being that way (in both
>> branch-2 and trunk presently:
>> https://github.com/apache/hadoop/blob/release-2.7.1/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/ShuffleHandler.java#L388
>> ).
>>
>> Your current workaround would be to use a firewall to drop out external
>> connections to that port.
>>
>> On Tue, Dec 8, 2015 at 9:30 PM Varun Vasudev <vv...@apache.org> wrote:
>>
>>> Hi Li,
>>>
>>> What have you set yarn.nodemanager.bind-host to? From
>>> https://hadoop.apache.org/docs/r2.7.1/hadoop-yarn/hadoop-yarn-common/yarn-default.xml
>>> -
>>>
>>> "The actual address the server will bind to. If this optional address is
>>> set, the RPC and webapp servers will bind to this address and the
>>> port specified in
>>> yarn.nodemanager.address and yarn.nodemanager.webapp.address,
>>> respectively. This is
>>> most useful for making NM listen to all interfaces by setting to
>>> 0.0.0.0”
>>>
>>> -Varun
>>>
>>>
>>>
>>> On 12/8/15, 2:57 PM, "Li Li" <fa...@gmail.com> wrote:
>>>
>>> > I want to setup a yarn cluster. It's not allowed to visit from
>>> >public network so I changed many configuration files to let all
>>> >service bind to the internal network address.
>>> > All thing is fine exception the nodemanager which bind
>>> >0.0.0.0:13562. I searched the default-mapred-default.xml and found it
>>> >mapreduce.shuffle.port.
>>> > Which configuration file should I change to let nodemanager listen
>>> >on internal address instead of 0.0.0.0?
>>> >
>>> >---------------------------------------------------------------------
>>> >To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org
>>> >For additional commands, e-mail: user-help@hadoop.apache.org
>>> >
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org
>>> For additional commands, e-mail: user-help@hadoop.apache.org
>>>
>>>
>
Re: nodemanager listen on 0.0.0.0
Posted by Harsh J <ha...@cloudera.com>.
Thank you Varun, https://issues.apache.org/jira/browse/MAPREDUCE-5938 is
the one.
On Tue, Dec 8, 2015 at 10:02 PM Varun Saxena <vs...@gmail.com>
wrote:
> There is already a JIRA for it - MAPREDUCE-5938(and other related JIRAs'
> MAPREDUCE-6402 and YARN-4119)
>
> Regards,
> Varun Saxena
>
> On Tue, Dec 8, 2015 at 9:56 PM, Harsh J <ha...@cloudera.com> wrote:
>
>> Hello,
>>
>> Could you file a JIRA for this please? Currently the ShuffleHandler will
>> always bind to wildcard address due to the code being that way (in both
>> branch-2 and trunk presently:
>> https://github.com/apache/hadoop/blob/release-2.7.1/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/ShuffleHandler.java#L388
>> ).
>>
>> Your current workaround would be to use a firewall to drop out external
>> connections to that port.
>>
>> On Tue, Dec 8, 2015 at 9:30 PM Varun Vasudev <vv...@apache.org> wrote:
>>
>>> Hi Li,
>>>
>>> What have you set yarn.nodemanager.bind-host to? From
>>> https://hadoop.apache.org/docs/r2.7.1/hadoop-yarn/hadoop-yarn-common/yarn-default.xml
>>> -
>>>
>>> "The actual address the server will bind to. If this optional address is
>>> set, the RPC and webapp servers will bind to this address and the
>>> port specified in
>>> yarn.nodemanager.address and yarn.nodemanager.webapp.address,
>>> respectively. This is
>>> most useful for making NM listen to all interfaces by setting to
>>> 0.0.0.0”
>>>
>>> -Varun
>>>
>>>
>>>
>>> On 12/8/15, 2:57 PM, "Li Li" <fa...@gmail.com> wrote:
>>>
>>> > I want to setup a yarn cluster. It's not allowed to visit from
>>> >public network so I changed many configuration files to let all
>>> >service bind to the internal network address.
>>> > All thing is fine exception the nodemanager which bind
>>> >0.0.0.0:13562. I searched the default-mapred-default.xml and found it
>>> >mapreduce.shuffle.port.
>>> > Which configuration file should I change to let nodemanager listen
>>> >on internal address instead of 0.0.0.0?
>>> >
>>> >---------------------------------------------------------------------
>>> >To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org
>>> >For additional commands, e-mail: user-help@hadoop.apache.org
>>> >
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org
>>> For additional commands, e-mail: user-help@hadoop.apache.org
>>>
>>>
>
Re: nodemanager listen on 0.0.0.0
Posted by Harsh J <ha...@cloudera.com>.
Thank you Varun, https://issues.apache.org/jira/browse/MAPREDUCE-5938 is
the one.
On Tue, Dec 8, 2015 at 10:02 PM Varun Saxena <vs...@gmail.com>
wrote:
> There is already a JIRA for it - MAPREDUCE-5938(and other related JIRAs'
> MAPREDUCE-6402 and YARN-4119)
>
> Regards,
> Varun Saxena
>
> On Tue, Dec 8, 2015 at 9:56 PM, Harsh J <ha...@cloudera.com> wrote:
>
>> Hello,
>>
>> Could you file a JIRA for this please? Currently the ShuffleHandler will
>> always bind to wildcard address due to the code being that way (in both
>> branch-2 and trunk presently:
>> https://github.com/apache/hadoop/blob/release-2.7.1/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/ShuffleHandler.java#L388
>> ).
>>
>> Your current workaround would be to use a firewall to drop out external
>> connections to that port.
>>
>> On Tue, Dec 8, 2015 at 9:30 PM Varun Vasudev <vv...@apache.org> wrote:
>>
>>> Hi Li,
>>>
>>> What have you set yarn.nodemanager.bind-host to? From
>>> https://hadoop.apache.org/docs/r2.7.1/hadoop-yarn/hadoop-yarn-common/yarn-default.xml
>>> -
>>>
>>> "The actual address the server will bind to. If this optional address is
>>> set, the RPC and webapp servers will bind to this address and the
>>> port specified in
>>> yarn.nodemanager.address and yarn.nodemanager.webapp.address,
>>> respectively. This is
>>> most useful for making NM listen to all interfaces by setting to
>>> 0.0.0.0”
>>>
>>> -Varun
>>>
>>>
>>>
>>> On 12/8/15, 2:57 PM, "Li Li" <fa...@gmail.com> wrote:
>>>
>>> > I want to setup a yarn cluster. It's not allowed to visit from
>>> >public network so I changed many configuration files to let all
>>> >service bind to the internal network address.
>>> > All thing is fine exception the nodemanager which bind
>>> >0.0.0.0:13562. I searched the default-mapred-default.xml and found it
>>> >mapreduce.shuffle.port.
>>> > Which configuration file should I change to let nodemanager listen
>>> >on internal address instead of 0.0.0.0?
>>> >
>>> >---------------------------------------------------------------------
>>> >To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org
>>> >For additional commands, e-mail: user-help@hadoop.apache.org
>>> >
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org
>>> For additional commands, e-mail: user-help@hadoop.apache.org
>>>
>>>
>
Re: nodemanager listen on 0.0.0.0
Posted by Varun Saxena <vs...@gmail.com>.
There is already a JIRA for it - MAPREDUCE-5938(and other related JIRAs'
MAPREDUCE-6402 and YARN-4119)
Regards,
Varun Saxena
On Tue, Dec 8, 2015 at 9:56 PM, Harsh J <ha...@cloudera.com> wrote:
> Hello,
>
> Could you file a JIRA for this please? Currently the ShuffleHandler will
> always bind to wildcard address due to the code being that way (in both
> branch-2 and trunk presently:
> https://github.com/apache/hadoop/blob/release-2.7.1/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/ShuffleHandler.java#L388
> ).
>
> Your current workaround would be to use a firewall to drop out external
> connections to that port.
>
> On Tue, Dec 8, 2015 at 9:30 PM Varun Vasudev <vv...@apache.org> wrote:
>
>> Hi Li,
>>
>> What have you set yarn.nodemanager.bind-host to? From
>> https://hadoop.apache.org/docs/r2.7.1/hadoop-yarn/hadoop-yarn-common/yarn-default.xml
>> -
>>
>> "The actual address the server will bind to. If this optional address is
>> set, the RPC and webapp servers will bind to this address and the
>> port specified in
>> yarn.nodemanager.address and yarn.nodemanager.webapp.address,
>> respectively. This is
>> most useful for making NM listen to all interfaces by setting to
>> 0.0.0.0”
>>
>> -Varun
>>
>>
>>
>> On 12/8/15, 2:57 PM, "Li Li" <fa...@gmail.com> wrote:
>>
>> > I want to setup a yarn cluster. It's not allowed to visit from
>> >public network so I changed many configuration files to let all
>> >service bind to the internal network address.
>> > All thing is fine exception the nodemanager which bind
>> >0.0.0.0:13562. I searched the default-mapred-default.xml and found it
>> >mapreduce.shuffle.port.
>> > Which configuration file should I change to let nodemanager listen
>> >on internal address instead of 0.0.0.0?
>> >
>> >---------------------------------------------------------------------
>> >To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org
>> >For additional commands, e-mail: user-help@hadoop.apache.org
>> >
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org
>> For additional commands, e-mail: user-help@hadoop.apache.org
>>
>>
Re: nodemanager listen on 0.0.0.0
Posted by Varun Saxena <vs...@gmail.com>.
There is already a JIRA for it - MAPREDUCE-5938(and other related JIRAs'
MAPREDUCE-6402 and YARN-4119)
Regards,
Varun Saxena
On Tue, Dec 8, 2015 at 9:56 PM, Harsh J <ha...@cloudera.com> wrote:
> Hello,
>
> Could you file a JIRA for this please? Currently the ShuffleHandler will
> always bind to wildcard address due to the code being that way (in both
> branch-2 and trunk presently:
> https://github.com/apache/hadoop/blob/release-2.7.1/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/ShuffleHandler.java#L388
> ).
>
> Your current workaround would be to use a firewall to drop out external
> connections to that port.
>
> On Tue, Dec 8, 2015 at 9:30 PM Varun Vasudev <vv...@apache.org> wrote:
>
>> Hi Li,
>>
>> What have you set yarn.nodemanager.bind-host to? From
>> https://hadoop.apache.org/docs/r2.7.1/hadoop-yarn/hadoop-yarn-common/yarn-default.xml
>> -
>>
>> "The actual address the server will bind to. If this optional address is
>> set, the RPC and webapp servers will bind to this address and the
>> port specified in
>> yarn.nodemanager.address and yarn.nodemanager.webapp.address,
>> respectively. This is
>> most useful for making NM listen to all interfaces by setting to
>> 0.0.0.0”
>>
>> -Varun
>>
>>
>>
>> On 12/8/15, 2:57 PM, "Li Li" <fa...@gmail.com> wrote:
>>
>> > I want to setup a yarn cluster. It's not allowed to visit from
>> >public network so I changed many configuration files to let all
>> >service bind to the internal network address.
>> > All thing is fine exception the nodemanager which bind
>> >0.0.0.0:13562. I searched the default-mapred-default.xml and found it
>> >mapreduce.shuffle.port.
>> > Which configuration file should I change to let nodemanager listen
>> >on internal address instead of 0.0.0.0?
>> >
>> >---------------------------------------------------------------------
>> >To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org
>> >For additional commands, e-mail: user-help@hadoop.apache.org
>> >
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org
>> For additional commands, e-mail: user-help@hadoop.apache.org
>>
>>
Re: nodemanager listen on 0.0.0.0
Posted by Varun Saxena <vs...@gmail.com>.
There is already a JIRA for it - MAPREDUCE-5938(and other related JIRAs'
MAPREDUCE-6402 and YARN-4119)
Regards,
Varun Saxena
On Tue, Dec 8, 2015 at 9:56 PM, Harsh J <ha...@cloudera.com> wrote:
> Hello,
>
> Could you file a JIRA for this please? Currently the ShuffleHandler will
> always bind to wildcard address due to the code being that way (in both
> branch-2 and trunk presently:
> https://github.com/apache/hadoop/blob/release-2.7.1/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/ShuffleHandler.java#L388
> ).
>
> Your current workaround would be to use a firewall to drop out external
> connections to that port.
>
> On Tue, Dec 8, 2015 at 9:30 PM Varun Vasudev <vv...@apache.org> wrote:
>
>> Hi Li,
>>
>> What have you set yarn.nodemanager.bind-host to? From
>> https://hadoop.apache.org/docs/r2.7.1/hadoop-yarn/hadoop-yarn-common/yarn-default.xml
>> -
>>
>> "The actual address the server will bind to. If this optional address is
>> set, the RPC and webapp servers will bind to this address and the
>> port specified in
>> yarn.nodemanager.address and yarn.nodemanager.webapp.address,
>> respectively. This is
>> most useful for making NM listen to all interfaces by setting to
>> 0.0.0.0”
>>
>> -Varun
>>
>>
>>
>> On 12/8/15, 2:57 PM, "Li Li" <fa...@gmail.com> wrote:
>>
>> > I want to setup a yarn cluster. It's not allowed to visit from
>> >public network so I changed many configuration files to let all
>> >service bind to the internal network address.
>> > All thing is fine exception the nodemanager which bind
>> >0.0.0.0:13562. I searched the default-mapred-default.xml and found it
>> >mapreduce.shuffle.port.
>> > Which configuration file should I change to let nodemanager listen
>> >on internal address instead of 0.0.0.0?
>> >
>> >---------------------------------------------------------------------
>> >To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org
>> >For additional commands, e-mail: user-help@hadoop.apache.org
>> >
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org
>> For additional commands, e-mail: user-help@hadoop.apache.org
>>
>>
Re: nodemanager listen on 0.0.0.0
Posted by Varun Saxena <vs...@gmail.com>.
There is already a JIRA for it - MAPREDUCE-5938(and other related JIRAs'
MAPREDUCE-6402 and YARN-4119)
Regards,
Varun Saxena
On Tue, Dec 8, 2015 at 9:56 PM, Harsh J <ha...@cloudera.com> wrote:
> Hello,
>
> Could you file a JIRA for this please? Currently the ShuffleHandler will
> always bind to wildcard address due to the code being that way (in both
> branch-2 and trunk presently:
> https://github.com/apache/hadoop/blob/release-2.7.1/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/ShuffleHandler.java#L388
> ).
>
> Your current workaround would be to use a firewall to drop out external
> connections to that port.
>
> On Tue, Dec 8, 2015 at 9:30 PM Varun Vasudev <vv...@apache.org> wrote:
>
>> Hi Li,
>>
>> What have you set yarn.nodemanager.bind-host to? From
>> https://hadoop.apache.org/docs/r2.7.1/hadoop-yarn/hadoop-yarn-common/yarn-default.xml
>> -
>>
>> "The actual address the server will bind to. If this optional address is
>> set, the RPC and webapp servers will bind to this address and the
>> port specified in
>> yarn.nodemanager.address and yarn.nodemanager.webapp.address,
>> respectively. This is
>> most useful for making NM listen to all interfaces by setting to
>> 0.0.0.0”
>>
>> -Varun
>>
>>
>>
>> On 12/8/15, 2:57 PM, "Li Li" <fa...@gmail.com> wrote:
>>
>> > I want to setup a yarn cluster. It's not allowed to visit from
>> >public network so I changed many configuration files to let all
>> >service bind to the internal network address.
>> > All thing is fine exception the nodemanager which bind
>> >0.0.0.0:13562. I searched the default-mapred-default.xml and found it
>> >mapreduce.shuffle.port.
>> > Which configuration file should I change to let nodemanager listen
>> >on internal address instead of 0.0.0.0?
>> >
>> >---------------------------------------------------------------------
>> >To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org
>> >For additional commands, e-mail: user-help@hadoop.apache.org
>> >
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org
>> For additional commands, e-mail: user-help@hadoop.apache.org
>>
>>
Re: nodemanager listen on 0.0.0.0
Posted by Harsh J <ha...@cloudera.com>.
Hello,
Could you file a JIRA for this please? Currently the ShuffleHandler will
always bind to wildcard address due to the code being that way (in both
branch-2 and trunk presently:
https://github.com/apache/hadoop/blob/release-2.7.1/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/ShuffleHandler.java#L388
).
Your current workaround would be to use a firewall to drop out external
connections to that port.
On Tue, Dec 8, 2015 at 9:30 PM Varun Vasudev <vv...@apache.org> wrote:
> Hi Li,
>
> What have you set yarn.nodemanager.bind-host to? From
> https://hadoop.apache.org/docs/r2.7.1/hadoop-yarn/hadoop-yarn-common/yarn-default.xml
> -
>
> "The actual address the server will bind to. If this optional address is
> set, the RPC and webapp servers will bind to this address and the
> port specified in
> yarn.nodemanager.address and yarn.nodemanager.webapp.address,
> respectively. This is
> most useful for making NM listen to all interfaces by setting to
> 0.0.0.0”
>
> -Varun
>
>
>
> On 12/8/15, 2:57 PM, "Li Li" <fa...@gmail.com> wrote:
>
> > I want to setup a yarn cluster. It's not allowed to visit from
> >public network so I changed many configuration files to let all
> >service bind to the internal network address.
> > All thing is fine exception the nodemanager which bind
> >0.0.0.0:13562. I searched the default-mapred-default.xml and found it
> >mapreduce.shuffle.port.
> > Which configuration file should I change to let nodemanager listen
> >on internal address instead of 0.0.0.0?
> >
> >---------------------------------------------------------------------
> >To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org
> >For additional commands, e-mail: user-help@hadoop.apache.org
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: user-help@hadoop.apache.org
>
>
Re: nodemanager listen on 0.0.0.0
Posted by Harsh J <ha...@cloudera.com>.
Hello,
Could you file a JIRA for this please? Currently the ShuffleHandler will
always bind to wildcard address due to the code being that way (in both
branch-2 and trunk presently:
https://github.com/apache/hadoop/blob/release-2.7.1/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/ShuffleHandler.java#L388
).
Your current workaround would be to use a firewall to drop out external
connections to that port.
On Tue, Dec 8, 2015 at 9:30 PM Varun Vasudev <vv...@apache.org> wrote:
> Hi Li,
>
> What have you set yarn.nodemanager.bind-host to? From
> https://hadoop.apache.org/docs/r2.7.1/hadoop-yarn/hadoop-yarn-common/yarn-default.xml
> -
>
> "The actual address the server will bind to. If this optional address is
> set, the RPC and webapp servers will bind to this address and the
> port specified in
> yarn.nodemanager.address and yarn.nodemanager.webapp.address,
> respectively. This is
> most useful for making NM listen to all interfaces by setting to
> 0.0.0.0”
>
> -Varun
>
>
>
> On 12/8/15, 2:57 PM, "Li Li" <fa...@gmail.com> wrote:
>
> > I want to setup a yarn cluster. It's not allowed to visit from
> >public network so I changed many configuration files to let all
> >service bind to the internal network address.
> > All thing is fine exception the nodemanager which bind
> >0.0.0.0:13562. I searched the default-mapred-default.xml and found it
> >mapreduce.shuffle.port.
> > Which configuration file should I change to let nodemanager listen
> >on internal address instead of 0.0.0.0?
> >
> >---------------------------------------------------------------------
> >To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org
> >For additional commands, e-mail: user-help@hadoop.apache.org
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: user-help@hadoop.apache.org
>
>
Re: nodemanager listen on 0.0.0.0
Posted by Harsh J <ha...@cloudera.com>.
Hello,
Could you file a JIRA for this please? Currently the ShuffleHandler will
always bind to wildcard address due to the code being that way (in both
branch-2 and trunk presently:
https://github.com/apache/hadoop/blob/release-2.7.1/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/ShuffleHandler.java#L388
).
Your current workaround would be to use a firewall to drop out external
connections to that port.
On Tue, Dec 8, 2015 at 9:30 PM Varun Vasudev <vv...@apache.org> wrote:
> Hi Li,
>
> What have you set yarn.nodemanager.bind-host to? From
> https://hadoop.apache.org/docs/r2.7.1/hadoop-yarn/hadoop-yarn-common/yarn-default.xml
> -
>
> "The actual address the server will bind to. If this optional address is
> set, the RPC and webapp servers will bind to this address and the
> port specified in
> yarn.nodemanager.address and yarn.nodemanager.webapp.address,
> respectively. This is
> most useful for making NM listen to all interfaces by setting to
> 0.0.0.0”
>
> -Varun
>
>
>
> On 12/8/15, 2:57 PM, "Li Li" <fa...@gmail.com> wrote:
>
> > I want to setup a yarn cluster. It's not allowed to visit from
> >public network so I changed many configuration files to let all
> >service bind to the internal network address.
> > All thing is fine exception the nodemanager which bind
> >0.0.0.0:13562. I searched the default-mapred-default.xml and found it
> >mapreduce.shuffle.port.
> > Which configuration file should I change to let nodemanager listen
> >on internal address instead of 0.0.0.0?
> >
> >---------------------------------------------------------------------
> >To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org
> >For additional commands, e-mail: user-help@hadoop.apache.org
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: user-help@hadoop.apache.org
>
>
Re: nodemanager listen on 0.0.0.0
Posted by Harsh J <ha...@cloudera.com>.
Hello,
Could you file a JIRA for this please? Currently the ShuffleHandler will
always bind to wildcard address due to the code being that way (in both
branch-2 and trunk presently:
https://github.com/apache/hadoop/blob/release-2.7.1/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/ShuffleHandler.java#L388
).
Your current workaround would be to use a firewall to drop out external
connections to that port.
On Tue, Dec 8, 2015 at 9:30 PM Varun Vasudev <vv...@apache.org> wrote:
> Hi Li,
>
> What have you set yarn.nodemanager.bind-host to? From
> https://hadoop.apache.org/docs/r2.7.1/hadoop-yarn/hadoop-yarn-common/yarn-default.xml
> -
>
> "The actual address the server will bind to. If this optional address is
> set, the RPC and webapp servers will bind to this address and the
> port specified in
> yarn.nodemanager.address and yarn.nodemanager.webapp.address,
> respectively. This is
> most useful for making NM listen to all interfaces by setting to
> 0.0.0.0”
>
> -Varun
>
>
>
> On 12/8/15, 2:57 PM, "Li Li" <fa...@gmail.com> wrote:
>
> > I want to setup a yarn cluster. It's not allowed to visit from
> >public network so I changed many configuration files to let all
> >service bind to the internal network address.
> > All thing is fine exception the nodemanager which bind
> >0.0.0.0:13562. I searched the default-mapred-default.xml and found it
> >mapreduce.shuffle.port.
> > Which configuration file should I change to let nodemanager listen
> >on internal address instead of 0.0.0.0?
> >
> >---------------------------------------------------------------------
> >To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org
> >For additional commands, e-mail: user-help@hadoop.apache.org
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: user-help@hadoop.apache.org
>
>
Re: nodemanager listen on 0.0.0.0
Posted by Varun Vasudev <vv...@apache.org>.
Hi Li,
What have you set yarn.nodemanager.bind-host to? From https://hadoop.apache.org/docs/r2.7.1/hadoop-yarn/hadoop-yarn-common/yarn-default.xml -
"The actual address the server will bind to. If this optional address is
set, the RPC and webapp servers will bind to this address and the port specified in
yarn.nodemanager.address and yarn.nodemanager.webapp.address, respectively. This is
most useful for making NM listen to all interfaces by setting to 0.0.0.0”
-Varun
On 12/8/15, 2:57 PM, "Li Li" <fa...@gmail.com> wrote:
> I want to setup a yarn cluster. It's not allowed to visit from
>public network so I changed many configuration files to let all
>service bind to the internal network address.
> All thing is fine exception the nodemanager which bind
>0.0.0.0:13562. I searched the default-mapred-default.xml and found it
>mapreduce.shuffle.port.
> Which configuration file should I change to let nodemanager listen
>on internal address instead of 0.0.0.0?
>
>---------------------------------------------------------------------
>To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org
>For additional commands, e-mail: user-help@hadoop.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org
For additional commands, e-mail: user-help@hadoop.apache.org
Re: nodemanager listen on 0.0.0.0
Posted by Li Li <fa...@gmail.com>.
the hadoop cluster may be used as crawler so it should visit public network.
On Tue, Dec 8, 2015 at 7:18 PM, Namikaze Minato <ll...@gmail.com> wrote:
> Listening on 0.0.0.0 means listening on all interfaces. Do you really
> NEED to have it listen only on internal addresses? If you're not
> connected to a public network it should not matter, right?
>
> Anyway, your cluster should work as-is.
>
> Regards,
> LLoyd
>
> PS: FYI, your e-mail ended up in spams.
>
> On 8 December 2015 at 10:27, Li Li <fa...@gmail.com> wrote:
>> I want to setup a yarn cluster. It's not allowed to visit from
>> public network so I changed many configuration files to let all
>> service bind to the internal network address.
>> All thing is fine exception the nodemanager which bind
>> 0.0.0.0:13562. I searched the default-mapred-default.xml and found it
>> mapreduce.shuffle.port.
>> Which configuration file should I change to let nodemanager listen
>> on internal address instead of 0.0.0.0?
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org
>> For additional commands, e-mail: user-help@hadoop.apache.org
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org
For additional commands, e-mail: user-help@hadoop.apache.org
Re: nodemanager listen on 0.0.0.0
Posted by Li Li <fa...@gmail.com>.
the hadoop cluster may be used as crawler so it should visit public network.
On Tue, Dec 8, 2015 at 7:18 PM, Namikaze Minato <ll...@gmail.com> wrote:
> Listening on 0.0.0.0 means listening on all interfaces. Do you really
> NEED to have it listen only on internal addresses? If you're not
> connected to a public network it should not matter, right?
>
> Anyway, your cluster should work as-is.
>
> Regards,
> LLoyd
>
> PS: FYI, your e-mail ended up in spams.
>
> On 8 December 2015 at 10:27, Li Li <fa...@gmail.com> wrote:
>> I want to setup a yarn cluster. It's not allowed to visit from
>> public network so I changed many configuration files to let all
>> service bind to the internal network address.
>> All thing is fine exception the nodemanager which bind
>> 0.0.0.0:13562. I searched the default-mapred-default.xml and found it
>> mapreduce.shuffle.port.
>> Which configuration file should I change to let nodemanager listen
>> on internal address instead of 0.0.0.0?
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org
>> For additional commands, e-mail: user-help@hadoop.apache.org
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org
For additional commands, e-mail: user-help@hadoop.apache.org
Re: nodemanager listen on 0.0.0.0
Posted by Li Li <fa...@gmail.com>.
the hadoop cluster may be used as crawler so it should visit public network.
On Tue, Dec 8, 2015 at 7:18 PM, Namikaze Minato <ll...@gmail.com> wrote:
> Listening on 0.0.0.0 means listening on all interfaces. Do you really
> NEED to have it listen only on internal addresses? If you're not
> connected to a public network it should not matter, right?
>
> Anyway, your cluster should work as-is.
>
> Regards,
> LLoyd
>
> PS: FYI, your e-mail ended up in spams.
>
> On 8 December 2015 at 10:27, Li Li <fa...@gmail.com> wrote:
>> I want to setup a yarn cluster. It's not allowed to visit from
>> public network so I changed many configuration files to let all
>> service bind to the internal network address.
>> All thing is fine exception the nodemanager which bind
>> 0.0.0.0:13562. I searched the default-mapred-default.xml and found it
>> mapreduce.shuffle.port.
>> Which configuration file should I change to let nodemanager listen
>> on internal address instead of 0.0.0.0?
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org
>> For additional commands, e-mail: user-help@hadoop.apache.org
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org
For additional commands, e-mail: user-help@hadoop.apache.org
Re: nodemanager listen on 0.0.0.0
Posted by Li Li <fa...@gmail.com>.
the hadoop cluster may be used as crawler so it should visit public network.
On Tue, Dec 8, 2015 at 7:18 PM, Namikaze Minato <ll...@gmail.com> wrote:
> Listening on 0.0.0.0 means listening on all interfaces. Do you really
> NEED to have it listen only on internal addresses? If you're not
> connected to a public network it should not matter, right?
>
> Anyway, your cluster should work as-is.
>
> Regards,
> LLoyd
>
> PS: FYI, your e-mail ended up in spams.
>
> On 8 December 2015 at 10:27, Li Li <fa...@gmail.com> wrote:
>> I want to setup a yarn cluster. It's not allowed to visit from
>> public network so I changed many configuration files to let all
>> service bind to the internal network address.
>> All thing is fine exception the nodemanager which bind
>> 0.0.0.0:13562. I searched the default-mapred-default.xml and found it
>> mapreduce.shuffle.port.
>> Which configuration file should I change to let nodemanager listen
>> on internal address instead of 0.0.0.0?
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org
>> For additional commands, e-mail: user-help@hadoop.apache.org
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org
For additional commands, e-mail: user-help@hadoop.apache.org
Re: nodemanager listen on 0.0.0.0
Posted by Namikaze Minato <ll...@gmail.com>.
Listening on 0.0.0.0 means listening on all interfaces. Do you really
NEED to have it listen only on internal addresses? If you're not
connected to a public network it should not matter, right?
Anyway, your cluster should work as-is.
Regards,
LLoyd
PS: FYI, your e-mail ended up in spams.
On 8 December 2015 at 10:27, Li Li <fa...@gmail.com> wrote:
> I want to setup a yarn cluster. It's not allowed to visit from
> public network so I changed many configuration files to let all
> service bind to the internal network address.
> All thing is fine exception the nodemanager which bind
> 0.0.0.0:13562. I searched the default-mapred-default.xml and found it
> mapreduce.shuffle.port.
> Which configuration file should I change to let nodemanager listen
> on internal address instead of 0.0.0.0?
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: user-help@hadoop.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org
For additional commands, e-mail: user-help@hadoop.apache.org
Re: nodemanager listen on 0.0.0.0
Posted by Varun Vasudev <vv...@apache.org>.
Hi Li,
What have you set yarn.nodemanager.bind-host to? From https://hadoop.apache.org/docs/r2.7.1/hadoop-yarn/hadoop-yarn-common/yarn-default.xml -
"The actual address the server will bind to. If this optional address is
set, the RPC and webapp servers will bind to this address and the port specified in
yarn.nodemanager.address and yarn.nodemanager.webapp.address, respectively. This is
most useful for making NM listen to all interfaces by setting to 0.0.0.0”
-Varun
On 12/8/15, 2:57 PM, "Li Li" <fa...@gmail.com> wrote:
> I want to setup a yarn cluster. It's not allowed to visit from
>public network so I changed many configuration files to let all
>service bind to the internal network address.
> All thing is fine exception the nodemanager which bind
>0.0.0.0:13562. I searched the default-mapred-default.xml and found it
>mapreduce.shuffle.port.
> Which configuration file should I change to let nodemanager listen
>on internal address instead of 0.0.0.0?
>
>---------------------------------------------------------------------
>To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org
>For additional commands, e-mail: user-help@hadoop.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org
For additional commands, e-mail: user-help@hadoop.apache.org
Re: nodemanager listen on 0.0.0.0
Posted by Namikaze Minato <ll...@gmail.com>.
Listening on 0.0.0.0 means listening on all interfaces. Do you really
NEED to have it listen only on internal addresses? If you're not
connected to a public network it should not matter, right?
Anyway, your cluster should work as-is.
Regards,
LLoyd
PS: FYI, your e-mail ended up in spams.
On 8 December 2015 at 10:27, Li Li <fa...@gmail.com> wrote:
> I want to setup a yarn cluster. It's not allowed to visit from
> public network so I changed many configuration files to let all
> service bind to the internal network address.
> All thing is fine exception the nodemanager which bind
> 0.0.0.0:13562. I searched the default-mapred-default.xml and found it
> mapreduce.shuffle.port.
> Which configuration file should I change to let nodemanager listen
> on internal address instead of 0.0.0.0?
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: user-help@hadoop.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org
For additional commands, e-mail: user-help@hadoop.apache.org
Re: nodemanager listen on 0.0.0.0
Posted by Namikaze Minato <ll...@gmail.com>.
Listening on 0.0.0.0 means listening on all interfaces. Do you really
NEED to have it listen only on internal addresses? If you're not
connected to a public network it should not matter, right?
Anyway, your cluster should work as-is.
Regards,
LLoyd
PS: FYI, your e-mail ended up in spams.
On 8 December 2015 at 10:27, Li Li <fa...@gmail.com> wrote:
> I want to setup a yarn cluster. It's not allowed to visit from
> public network so I changed many configuration files to let all
> service bind to the internal network address.
> All thing is fine exception the nodemanager which bind
> 0.0.0.0:13562. I searched the default-mapred-default.xml and found it
> mapreduce.shuffle.port.
> Which configuration file should I change to let nodemanager listen
> on internal address instead of 0.0.0.0?
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: user-help@hadoop.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org
For additional commands, e-mail: user-help@hadoop.apache.org
Re: nodemanager listen on 0.0.0.0
Posted by Varun Vasudev <vv...@apache.org>.
Hi Li,
What have you set yarn.nodemanager.bind-host to? From https://hadoop.apache.org/docs/r2.7.1/hadoop-yarn/hadoop-yarn-common/yarn-default.xml -
"The actual address the server will bind to. If this optional address is
set, the RPC and webapp servers will bind to this address and the port specified in
yarn.nodemanager.address and yarn.nodemanager.webapp.address, respectively. This is
most useful for making NM listen to all interfaces by setting to 0.0.0.0”
-Varun
On 12/8/15, 2:57 PM, "Li Li" <fa...@gmail.com> wrote:
> I want to setup a yarn cluster. It's not allowed to visit from
>public network so I changed many configuration files to let all
>service bind to the internal network address.
> All thing is fine exception the nodemanager which bind
>0.0.0.0:13562. I searched the default-mapred-default.xml and found it
>mapreduce.shuffle.port.
> Which configuration file should I change to let nodemanager listen
>on internal address instead of 0.0.0.0?
>
>---------------------------------------------------------------------
>To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org
>For additional commands, e-mail: user-help@hadoop.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org
For additional commands, e-mail: user-help@hadoop.apache.org
Re: nodemanager listen on 0.0.0.0
Posted by Varun Vasudev <vv...@apache.org>.
Hi Li,
What have you set yarn.nodemanager.bind-host to? From https://hadoop.apache.org/docs/r2.7.1/hadoop-yarn/hadoop-yarn-common/yarn-default.xml -
"The actual address the server will bind to. If this optional address is
set, the RPC and webapp servers will bind to this address and the port specified in
yarn.nodemanager.address and yarn.nodemanager.webapp.address, respectively. This is
most useful for making NM listen to all interfaces by setting to 0.0.0.0”
-Varun
On 12/8/15, 2:57 PM, "Li Li" <fa...@gmail.com> wrote:
> I want to setup a yarn cluster. It's not allowed to visit from
>public network so I changed many configuration files to let all
>service bind to the internal network address.
> All thing is fine exception the nodemanager which bind
>0.0.0.0:13562. I searched the default-mapred-default.xml and found it
>mapreduce.shuffle.port.
> Which configuration file should I change to let nodemanager listen
>on internal address instead of 0.0.0.0?
>
>---------------------------------------------------------------------
>To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org
>For additional commands, e-mail: user-help@hadoop.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org
For additional commands, e-mail: user-help@hadoop.apache.org
Re: nodemanager listen on 0.0.0.0
Posted by Namikaze Minato <ll...@gmail.com>.
Listening on 0.0.0.0 means listening on all interfaces. Do you really
NEED to have it listen only on internal addresses? If you're not
connected to a public network it should not matter, right?
Anyway, your cluster should work as-is.
Regards,
LLoyd
PS: FYI, your e-mail ended up in spams.
On 8 December 2015 at 10:27, Li Li <fa...@gmail.com> wrote:
> I want to setup a yarn cluster. It's not allowed to visit from
> public network so I changed many configuration files to let all
> service bind to the internal network address.
> All thing is fine exception the nodemanager which bind
> 0.0.0.0:13562. I searched the default-mapred-default.xml and found it
> mapreduce.shuffle.port.
> Which configuration file should I change to let nodemanager listen
> on internal address instead of 0.0.0.0?
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: user-help@hadoop.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org
For additional commands, e-mail: user-help@hadoop.apache.org