You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Michael Bieniosek (JIRA)" <ji...@apache.org> on 2007/04/04 19:26:32 UTC
[jira] Created: (HADOOP-1202) Need to be able to set jobtracker &
namenode to bind to 0.0.0.0
Need to be able to set jobtracker & namenode to bind to 0.0.0.0
---------------------------------------------------------------
Key: HADOOP-1202
URL: https://issues.apache.org/jira/browse/HADOOP-1202
Project: Hadoop
Issue Type: Bug
Reporter: Michael Bieniosek
Attachments: bind-all-addresses.patch
Currently, the namenode will bind to the hostname in fs.default.name, and the jobtracker will bind to the hostname in mapred.job.tracker. These names are also reported to datanode and tasktracker clients.
Consequently, putting a fqhn in these fields causes the server to only listen on the interface serving that fqhn. Putting 0.0.0.0 in these fields causes the server to report 0.0.0.0 as its address to clients, which confuses the clients and causes them to fail.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1202) Need to be able to set jobtracker &
namenode to bind to 0.0.0.0
Posted by "Michael Bieniosek (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12486827 ]
Michael Bieniosek commented on HADOOP-1202:
-------------------------------------------
In the case of the namenode, the problem is that fs.default.name is used for two purposes:
1) What address should I bind?
2) What address should I report to my clients?
As far as I can tell, the hostname passed to RPC.getServer is used only for purpose 1, and the hostname passed to FSNameSystem is used only for purpose 2.
My patch only changes the hostname passed to RPC.getServer. In the case of the 1-arg NameNode, the purpose-2-hostname is taken from fs.default.name, passed to init, and explicitly used to set this.nameNodeAddress. The value of nameNodeAddress passed back to fs.default.name is the purpose-2-hostname. The result of dfs.namenode.bindAddress is only passed to RPC.getServer.
> Need to be able to set jobtracker & namenode to bind to 0.0.0.0
> ---------------------------------------------------------------
>
> Key: HADOOP-1202
> URL: https://issues.apache.org/jira/browse/HADOOP-1202
> Project: Hadoop
> Issue Type: Bug
> Reporter: Michael Bieniosek
> Attachments: bind-all-addresses.patch
>
>
> Currently, the namenode will bind to the hostname specified in configuration in fs.default.name, and the jobtracker will bind to the hostname in mapred.job.tracker. These names are also reported to datanode and tasktracker clients.
> Consequently, putting a fqhn in these fields causes the server to only listen on the interface serving that fqhn. Putting 0.0.0.0 in these fields causes the server to report 0.0.0.0 as its address to clients, which confuses the clients and causes them to fail.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1202) Need to be able to set jobtracker &
namenode to bind to 0.0.0.0
Posted by "Konstantin Shvachko (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12486750 ]
Konstantin Shvachko commented on HADOOP-1202:
---------------------------------------------
-1 on introducing a new configuration parameter having the same meaning as the previous one.
You are trying to separate input and output configuration parameters.
We should NOT use config parameters as an output of any function.
I think this is a confusion with the 0 port introduced in HADOOP-1085.
I would rather write 2 different constructors for the NameNode.
One will be used in tests with MiniDFSCluster, which would always poll the port,
or even better if the MiniClusters polls, and just passes the correct address into the NameNode constructor.
Another constructor is for a regular name-node startup just as it was before 1085.
I think this a cleaner an less confusing approach.
Same for the JobTracker.
> Need to be able to set jobtracker & namenode to bind to 0.0.0.0
> ---------------------------------------------------------------
>
> Key: HADOOP-1202
> URL: https://issues.apache.org/jira/browse/HADOOP-1202
> Project: Hadoop
> Issue Type: Bug
> Reporter: Michael Bieniosek
> Attachments: bind-all-addresses.patch
>
>
> Currently, the namenode will bind to the hostname specified in configuration in fs.default.name, and the jobtracker will bind to the hostname in mapred.job.tracker. These names are also reported to datanode and tasktracker clients.
> Consequently, putting a fqhn in these fields causes the server to only listen on the interface serving that fqhn. Putting 0.0.0.0 in these fields causes the server to report 0.0.0.0 as its address to clients, which confuses the clients and causes them to fail.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1202) Need to be able to set jobtracker &
namenode to bind to 0.0.0.0
Posted by "Konstantin Shvachko (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12486818 ]
Konstantin Shvachko commented on HADOOP-1202:
---------------------------------------------
In case of DFS
<name>fs.default.name</name>
<description>
......
determine the host, port, etc. for a filesystem.</description>
</property>
Lets see what happens to this parameter after your patch.
NameNode takes host from "dfs.namenode.bindAddress", combines it with the port
extracted from "fs.default.name" and generates an address, which is set back to "fs.default.name".
So the host part of "fs.default.name" does not play any role in configuring the name-node.
You are replacing this part with "dfs.namenode.bindAddress". This is my objection.
If port=0, then it will be also overwritten (introduced in another patch).
So "fs.default.name" will be completely replaced after the name-node constructor.
NameNode() receives fs.default.name as an input via the conf parameter. I called it an output parameter,
since changes made to it inside NameNode() are passed back to the calling procedure.
The rest is about how I think it should be implemented all together.
Can you solve 0.0.0.0 problem without introducing new config parameters?
> Need to be able to set jobtracker & namenode to bind to 0.0.0.0
> ---------------------------------------------------------------
>
> Key: HADOOP-1202
> URL: https://issues.apache.org/jira/browse/HADOOP-1202
> Project: Hadoop
> Issue Type: Bug
> Reporter: Michael Bieniosek
> Attachments: bind-all-addresses.patch
>
>
> Currently, the namenode will bind to the hostname specified in configuration in fs.default.name, and the jobtracker will bind to the hostname in mapred.job.tracker. These names are also reported to datanode and tasktracker clients.
> Consequently, putting a fqhn in these fields causes the server to only listen on the interface serving that fqhn. Putting 0.0.0.0 in these fields causes the server to report 0.0.0.0 as its address to clients, which confuses the clients and causes them to fail.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1202) Need to be able to set jobtracker &
namenode to bind to 0.0.0.0
Posted by "Michael Bieniosek (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12489573 ]
Michael Bieniosek commented on HADOOP-1202:
-------------------------------------------
My jobtracker needs to talk to the tasktrackers, and my namenode needs to talk to my datanodes. I need to talk to my jobtracker to give it tasks, and I need to talk to my namenode to retrieve the results of my jobs.
My hadoop cluster lives on a different network than I do, so I can't talk to it directly. As a special exception, my jobtracker and namenode have names in both networks, so if I apply this patch, everyone can talk to them. Without the patch, hadoop won't let the jobtracker and namenode talk in both networks.
> Need to be able to set jobtracker & namenode to bind to 0.0.0.0
> ---------------------------------------------------------------
>
> Key: HADOOP-1202
> URL: https://issues.apache.org/jira/browse/HADOOP-1202
> Project: Hadoop
> Issue Type: Bug
> Reporter: Michael Bieniosek
> Attachments: bind-all-addresses.patch
>
>
> Currently, the namenode will bind to the hostname specified in configuration in fs.default.name, and the jobtracker will bind to the hostname in mapred.job.tracker. These names are also reported to datanode and tasktracker clients.
> Consequently, putting a fqhn in these fields causes the server to only listen on the interface serving that fqhn. Putting 0.0.0.0 in these fields causes the server to report 0.0.0.0 as its address to clients, which confuses the clients and causes them to fail.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1202) Need to be able to set jobtracker &
namenode to bind to 0.0.0.0
Posted by "Michael Bieniosek (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Michael Bieniosek updated HADOOP-1202:
--------------------------------------
Status: Patch Available (was: Open)
I propose adding two new configuration variables, dfs.namenode.bindAddress and mapred.job.tracker.bindAddress.
> Need to be able to set jobtracker & namenode to bind to 0.0.0.0
> ---------------------------------------------------------------
>
> Key: HADOOP-1202
> URL: https://issues.apache.org/jira/browse/HADOOP-1202
> Project: Hadoop
> Issue Type: Bug
> Reporter: Michael Bieniosek
> Attachments: bind-all-addresses.patch
>
>
> Currently, the namenode will bind to the hostname in fs.default.name, and the jobtracker will bind to the hostname in mapred.job.tracker. These names are also reported to datanode and tasktracker clients.
> Consequently, putting a fqhn in these fields causes the server to only listen on the interface serving that fqhn. Putting 0.0.0.0 in these fields causes the server to report 0.0.0.0 as its address to clients, which confuses the clients and causes them to fail.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1202) Need to be able to set jobtracker &
namenode to bind to 0.0.0.0
Posted by "Jeremy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12491757 ]
Jeremy commented on HADOOP-1202:
--------------------------------
I agree that there are some configuration issues here that need to be addressed.
In my particular situation, I have had to use IP addresses instead of FQDN's for the properties fs.default.name and mapred.job.tracker. This is because the server that is running my NameNode and JobTracker is resolving it's FQDN name to 127.0.0.1, and consequently, the services are then being bound to 127.0.0.1 and are not accessible from other machines on the network.
This could be addressed by adjusting the way the FQDN is resolved, however, my hosting provider would have to make that change and in general this seems like something that should be configurable in Hadoop.
> Need to be able to set jobtracker & namenode to bind to 0.0.0.0
> ---------------------------------------------------------------
>
> Key: HADOOP-1202
> URL: https://issues.apache.org/jira/browse/HADOOP-1202
> Project: Hadoop
> Issue Type: Bug
> Reporter: Michael Bieniosek
> Attachments: bind-all-addresses.patch
>
>
> Currently, the namenode will bind to the hostname specified in configuration in fs.default.name, and the jobtracker will bind to the hostname in mapred.job.tracker. These names are also reported to datanode and tasktracker clients.
> Consequently, putting a fqhn in these fields causes the server to only listen on the interface serving that fqhn. Putting 0.0.0.0 in these fields causes the server to report 0.0.0.0 as its address to clients, which confuses the clients and causes them to fail.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1202) Jobtracker & Namenode can only bind
to one interface
Posted by "Michael Bieniosek (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Michael Bieniosek updated HADOOP-1202:
--------------------------------------
Description:
There is no way to tell the jobtracker & namenode to listen on more than one interface. Because of firewall rules etc., my datanodes/tasktrackers and human users live on different networks, and so talk to the jobtracker/namenode on different interfaces.
Currently, the namenode will bind to the hostname specified in configuration in fs.default.name, and the jobtracker will bind to the hostname in mapred.job.tracker. These names are also reported to datanode and tasktracker clients.
Consequently, putting a fqhn in these fields causes the server to only listen on the interface serving that fqhn. Putting 0.0.0.0 in these fields causes the server to report 0.0.0.0 as its address to clients, which confuses the clients and causes them to fail.
was:
Currently, the namenode will bind to the hostname specified in configuration in fs.default.name, and the jobtracker will bind to the hostname in mapred.job.tracker. These names are also reported to datanode and tasktracker clients.
Consequently, putting a fqhn in these fields causes the server to only listen on the interface serving that fqhn. Putting 0.0.0.0 in these fields causes the server to report 0.0.0.0 as its address to clients, which confuses the clients and causes them to fail.
Summary: Jobtracker & Namenode can only bind to one interface (was: Need to be able to set jobtracker & namenode to bind to 0.0.0.0)
Improve description
> Jobtracker & Namenode can only bind to one interface
> ----------------------------------------------------
>
> Key: HADOOP-1202
> URL: https://issues.apache.org/jira/browse/HADOOP-1202
> Project: Hadoop
> Issue Type: Bug
> Reporter: Michael Bieniosek
> Attachments: bind-all-addresses.patch
>
>
> There is no way to tell the jobtracker & namenode to listen on more than one interface. Because of firewall rules etc., my datanodes/tasktrackers and human users live on different networks, and so talk to the jobtracker/namenode on different interfaces.
> Currently, the namenode will bind to the hostname specified in configuration in fs.default.name, and the jobtracker will bind to the hostname in mapred.job.tracker. These names are also reported to datanode and tasktracker clients.
> Consequently, putting a fqhn in these fields causes the server to only listen on the interface serving that fqhn. Putting 0.0.0.0 in these fields causes the server to report 0.0.0.0 as its address to clients, which confuses the clients and causes them to fail.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1202) Need to be able to set jobtracker &
namenode to bind to 0.0.0.0
Posted by "Michael Bieniosek (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12487155 ]
Michael Bieniosek commented on HADOOP-1202:
-------------------------------------------
> How do expect cluster to work well when config is inconsistent? Config is the only way for datanode to find where namenode is located. Binding to 0.0.0.0 does not fix misconfiguration.
My problem is that my namenode and jobtracker have more than one name, because of the way the firewall is set up. Internally, it listens on interface 1, where it can talk to the datanodes and tasktrackers, while externally it listens on interface 2, where it can talk to the rest of the network. The namenode/jobtracker need to listen on both interfaces, which happen to have different dns names and different ips.
> Need to be able to set jobtracker & namenode to bind to 0.0.0.0
> ---------------------------------------------------------------
>
> Key: HADOOP-1202
> URL: https://issues.apache.org/jira/browse/HADOOP-1202
> Project: Hadoop
> Issue Type: Bug
> Reporter: Michael Bieniosek
> Attachments: bind-all-addresses.patch
>
>
> Currently, the namenode will bind to the hostname specified in configuration in fs.default.name, and the jobtracker will bind to the hostname in mapred.job.tracker. These names are also reported to datanode and tasktracker clients.
> Consequently, putting a fqhn in these fields causes the server to only listen on the interface serving that fqhn. Putting 0.0.0.0 in these fields causes the server to report 0.0.0.0 as its address to clients, which confuses the clients and causes them to fail.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1202) Need to be able to set jobtracker &
namenode to bind to 0.0.0.0
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12487161 ]
Raghu Angadi commented on HADOOP-1202:
--------------------------------------
I do think it will be useful to be able to bind to all interfaces, at least in some special cases.
Just out of curiosity, in your set up what is in 'rest of the network' that needs to talk to namenode?. I think I still don't follow your set up.
> Need to be able to set jobtracker & namenode to bind to 0.0.0.0
> ---------------------------------------------------------------
>
> Key: HADOOP-1202
> URL: https://issues.apache.org/jira/browse/HADOOP-1202
> Project: Hadoop
> Issue Type: Bug
> Reporter: Michael Bieniosek
> Attachments: bind-all-addresses.patch
>
>
> Currently, the namenode will bind to the hostname specified in configuration in fs.default.name, and the jobtracker will bind to the hostname in mapred.job.tracker. These names are also reported to datanode and tasktracker clients.
> Consequently, putting a fqhn in these fields causes the server to only listen on the interface serving that fqhn. Putting 0.0.0.0 in these fields causes the server to report 0.0.0.0 as its address to clients, which confuses the clients and causes them to fail.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1202) Need to be able to set jobtracker &
namenode to bind to 0.0.0.0
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12486720 ]
Hadoop QA commented on HADOOP-1202:
-----------------------------------
+1, because http://issues.apache.org/jira/secure/attachment/12354936/bind-all-addresses.patch applied and successfully tested against trunk revision http://svn.apache.org/repos/asf/lucene/hadoop/trunk/525505. Results are at http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch
> Need to be able to set jobtracker & namenode to bind to 0.0.0.0
> ---------------------------------------------------------------
>
> Key: HADOOP-1202
> URL: https://issues.apache.org/jira/browse/HADOOP-1202
> Project: Hadoop
> Issue Type: Bug
> Reporter: Michael Bieniosek
> Attachments: bind-all-addresses.patch
>
>
> Currently, the namenode will bind to the hostname specified in configuration in fs.default.name, and the jobtracker will bind to the hostname in mapred.job.tracker. These names are also reported to datanode and tasktracker clients.
> Consequently, putting a fqhn in these fields causes the server to only listen on the interface serving that fqhn. Putting 0.0.0.0 in these fields causes the server to report 0.0.0.0 as its address to clients, which confuses the clients and causes them to fail.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1202) Need to be able to set jobtracker &
namenode to bind to 0.0.0.0
Posted by "Michael Bieniosek (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Michael Bieniosek updated HADOOP-1202:
--------------------------------------
Attachment: bind-all-addresses.patch
> Need to be able to set jobtracker & namenode to bind to 0.0.0.0
> ---------------------------------------------------------------
>
> Key: HADOOP-1202
> URL: https://issues.apache.org/jira/browse/HADOOP-1202
> Project: Hadoop
> Issue Type: Bug
> Reporter: Michael Bieniosek
> Attachments: bind-all-addresses.patch
>
>
> Currently, the namenode will bind to the hostname in fs.default.name, and the jobtracker will bind to the hostname in mapred.job.tracker. These names are also reported to datanode and tasktracker clients.
> Consequently, putting a fqhn in these fields causes the server to only listen on the interface serving that fqhn. Putting 0.0.0.0 in these fields causes the server to report 0.0.0.0 as its address to clients, which confuses the clients and causes them to fail.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1202) Jobtracker & Namenode can only bind
to one interface
Posted by "Fredrik Hedberg (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Fredrik Hedberg updated HADOOP-1202:
------------------------------------
Attachment: bind-all-addresses-2.patch
Michael's patch updated to apply to trunk.
> Jobtracker & Namenode can only bind to one interface
> ----------------------------------------------------
>
> Key: HADOOP-1202
> URL: https://issues.apache.org/jira/browse/HADOOP-1202
> Project: Hadoop
> Issue Type: Bug
> Reporter: Michael Bieniosek
> Attachments: bind-all-addresses-2.patch, bind-all-addresses.patch
>
>
> There is no way to tell the jobtracker & namenode to listen on more than one interface. Because of firewall rules etc., my datanodes/tasktrackers and human users live on different networks, and so talk to the jobtracker/namenode on different interfaces.
> Currently, the namenode will bind to the hostname specified in configuration in fs.default.name, and the jobtracker will bind to the hostname in mapred.job.tracker. These names are also reported to datanode and tasktracker clients.
> Consequently, putting a fqhn in these fields causes the server to only listen on the interface serving that fqhn. Putting 0.0.0.0 in these fields causes the server to report 0.0.0.0 as its address to clients, which confuses the clients and causes them to fail.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1202) Need to be able to set jobtracker &
namenode to bind to 0.0.0.0
Posted by "Tom White (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tom White updated HADOOP-1202:
------------------------------
Status: Open (was: Patch Available)
Marking as Open until consensus is reached...
> Need to be able to set jobtracker & namenode to bind to 0.0.0.0
> ---------------------------------------------------------------
>
> Key: HADOOP-1202
> URL: https://issues.apache.org/jira/browse/HADOOP-1202
> Project: Hadoop
> Issue Type: Bug
> Reporter: Michael Bieniosek
> Attachments: bind-all-addresses.patch
>
>
> Currently, the namenode will bind to the hostname specified in configuration in fs.default.name, and the jobtracker will bind to the hostname in mapred.job.tracker. These names are also reported to datanode and tasktracker clients.
> Consequently, putting a fqhn in these fields causes the server to only listen on the interface serving that fqhn. Putting 0.0.0.0 in these fields causes the server to report 0.0.0.0 as its address to clients, which confuses the clients and causes them to fail.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1202) Jobtracker & Namenode can only bind
to one interface
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12491806 ]
Raghu Angadi commented on HADOOP-1202:
--------------------------------------
bq. ... I need to talk to my namenode to retrieve the results of my jobs. ...
So you need to read some files from HDFS?. ie. you need to be able to talk to a datanode on the same interface that datanode talks to Namenode ( since that is the ip address namenode gives you ). This jira will help you only if you currently can talk to datanodes but not to namenode. Correct?
> Jobtracker & Namenode can only bind to one interface
> ----------------------------------------------------
>
> Key: HADOOP-1202
> URL: https://issues.apache.org/jira/browse/HADOOP-1202
> Project: Hadoop
> Issue Type: Bug
> Reporter: Michael Bieniosek
> Attachments: bind-all-addresses.patch
>
>
> There is no way to tell the jobtracker & namenode to listen on more than one interface. Because of firewall rules etc., my datanodes/tasktrackers and human users live on different networks, and so talk to the jobtracker/namenode on different interfaces.
> Currently, the namenode will bind to the hostname specified in configuration in fs.default.name, and the jobtracker will bind to the hostname in mapred.job.tracker. These names are also reported to datanode and tasktracker clients.
> Consequently, putting a fqhn in these fields causes the server to only listen on the interface serving that fqhn. Putting 0.0.0.0 in these fields causes the server to report 0.0.0.0 as its address to clients, which confuses the clients and causes them to fail.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1202) Need to be able to set jobtracker &
namenode to bind to 0.0.0.0
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12487139 ]
Raghu Angadi commented on HADOOP-1202:
--------------------------------------
> The datanodes get very confused if the dns name they have configured for the namenode is not exactly the same as the name the namenode is configured with.
How do expect cluster to work well when config is inconsistent? Config is the only way for datanode to find where namenode is located. Binding to 0.0.0.0 does not fix misconfiguration.
But I think it will be useful to be able to specify bind address. I think the default value for bind address should be "default" which would imply what we do now. Most users will never need to look at it. On some special clusters it could be set to desired value.
> Need to be able to set jobtracker & namenode to bind to 0.0.0.0
> ---------------------------------------------------------------
>
> Key: HADOOP-1202
> URL: https://issues.apache.org/jira/browse/HADOOP-1202
> Project: Hadoop
> Issue Type: Bug
> Reporter: Michael Bieniosek
> Attachments: bind-all-addresses.patch
>
>
> Currently, the namenode will bind to the hostname specified in configuration in fs.default.name, and the jobtracker will bind to the hostname in mapred.job.tracker. These names are also reported to datanode and tasktracker clients.
> Consequently, putting a fqhn in these fields causes the server to only listen on the interface serving that fqhn. Putting 0.0.0.0 in these fields causes the server to report 0.0.0.0 as its address to clients, which confuses the clients and causes them to fail.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1202) Need to be able to set jobtracker &
namenode to bind to 0.0.0.0
Posted by "Michael Bieniosek (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Michael Bieniosek updated HADOOP-1202:
--------------------------------------
Description:
Currently, the namenode will bind to the hostname specified in configuration in fs.default.name, and the jobtracker will bind to the hostname in mapred.job.tracker. These names are also reported to datanode and tasktracker clients.
Consequently, putting a fqhn in these fields causes the server to only listen on the interface serving that fqhn. Putting 0.0.0.0 in these fields causes the server to report 0.0.0.0 as its address to clients, which confuses the clients and causes them to fail.
was:
Currently, the namenode will bind to the hostname in fs.default.name, and the jobtracker will bind to the hostname in mapred.job.tracker. These names are also reported to datanode and tasktracker clients.
Consequently, putting a fqhn in these fields causes the server to only listen on the interface serving that fqhn. Putting 0.0.0.0 in these fields causes the server to report 0.0.0.0 as its address to clients, which confuses the clients and causes them to fail.
> Need to be able to set jobtracker & namenode to bind to 0.0.0.0
> ---------------------------------------------------------------
>
> Key: HADOOP-1202
> URL: https://issues.apache.org/jira/browse/HADOOP-1202
> Project: Hadoop
> Issue Type: Bug
> Reporter: Michael Bieniosek
> Attachments: bind-all-addresses.patch
>
>
> Currently, the namenode will bind to the hostname specified in configuration in fs.default.name, and the jobtracker will bind to the hostname in mapred.job.tracker. These names are also reported to datanode and tasktracker clients.
> Consequently, putting a fqhn in these fields causes the server to only listen on the interface serving that fqhn. Putting 0.0.0.0 in these fields causes the server to report 0.0.0.0 as its address to clients, which confuses the clients and causes them to fail.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1202) Need to be able to set jobtracker &
namenode to bind to 0.0.0.0
Posted by "Michael Bieniosek (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12486751 ]
Michael Bieniosek commented on HADOOP-1202:
-------------------------------------------
Konstantin,
I'm not sure I understand your objection. How are these config parameters the output of a function? Which previous configuration parameter has the same meaning?
My change doesn't affect the server ports at all.
> Need to be able to set jobtracker & namenode to bind to 0.0.0.0
> ---------------------------------------------------------------
>
> Key: HADOOP-1202
> URL: https://issues.apache.org/jira/browse/HADOOP-1202
> Project: Hadoop
> Issue Type: Bug
> Reporter: Michael Bieniosek
> Attachments: bind-all-addresses.patch
>
>
> Currently, the namenode will bind to the hostname specified in configuration in fs.default.name, and the jobtracker will bind to the hostname in mapred.job.tracker. These names are also reported to datanode and tasktracker clients.
> Consequently, putting a fqhn in these fields causes the server to only listen on the interface serving that fqhn. Putting 0.0.0.0 in these fields causes the server to report 0.0.0.0 as its address to clients, which confuses the clients and causes them to fail.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1202) Need to be able to set jobtracker &
namenode to bind to 0.0.0.0
Posted by "Michael Bieniosek (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12487136 ]
Michael Bieniosek commented on HADOOP-1202:
-------------------------------------------
By client, I mean a datanode. The datanodes get very confused if the dns name they have configured for the namenode is not exactly the same as the name the namenode is configured with. I suppose it might be possible to fix the datanode side, but that would still mean that I would have to set fs.default.name to 0.0.0.0 for the namenode.
> Need to be able to set jobtracker & namenode to bind to 0.0.0.0
> ---------------------------------------------------------------
>
> Key: HADOOP-1202
> URL: https://issues.apache.org/jira/browse/HADOOP-1202
> Project: Hadoop
> Issue Type: Bug
> Reporter: Michael Bieniosek
> Attachments: bind-all-addresses.patch
>
>
> Currently, the namenode will bind to the hostname specified in configuration in fs.default.name, and the jobtracker will bind to the hostname in mapred.job.tracker. These names are also reported to datanode and tasktracker clients.
> Consequently, putting a fqhn in these fields causes the server to only listen on the interface serving that fqhn. Putting 0.0.0.0 in these fields causes the server to report 0.0.0.0 as its address to clients, which confuses the clients and causes them to fail.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1202) Need to be able to set jobtracker &
namenode to bind to 0.0.0.0
Posted by "Konstantin Shvachko (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12487129 ]
Konstantin Shvachko commented on HADOOP-1202:
---------------------------------------------
> In the case of the namenode, the problem is that fs.default.name is used for two purposes:
> 1) What address should I bind?
Yes, that's the purpose.
> 2) What address should I report to my clients?
This is a very bad purpose imo.
Configuration parameters should not be used for that.
I'd prefer if I could declare the conf parameter in the init() method as "const" to prevent it from modification.
There are other ways, like calling directly NameNode.getNameNodeAddress()
Not from the client of course but by modifying "fs.default.name" just before calling the client.
What kind of clients are we talking here?
> Need to be able to set jobtracker & namenode to bind to 0.0.0.0
> ---------------------------------------------------------------
>
> Key: HADOOP-1202
> URL: https://issues.apache.org/jira/browse/HADOOP-1202
> Project: Hadoop
> Issue Type: Bug
> Reporter: Michael Bieniosek
> Attachments: bind-all-addresses.patch
>
>
> Currently, the namenode will bind to the hostname specified in configuration in fs.default.name, and the jobtracker will bind to the hostname in mapred.job.tracker. These names are also reported to datanode and tasktracker clients.
> Consequently, putting a fqhn in these fields causes the server to only listen on the interface serving that fqhn. Putting 0.0.0.0 in these fields causes the server to report 0.0.0.0 as its address to clients, which confuses the clients and causes them to fail.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.