You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Carlos Valiente (JIRA)" <ji...@apache.org> on 2009/04/05 20:25:12 UTC
[jira] Created: (HADOOP-5626) SecondaryNamenode may report
incorrect info host name
SecondaryNamenode may report incorrect info host name
-----------------------------------------------------
Key: HADOOP-5626
URL: https://issues.apache.org/jira/browse/HADOOP-5626
Project: Hadoop Core
Issue Type: Bug
Components: dfs
Reporter: Carlos Valiente
Priority: Minor
Fix For: 0.21.0
Attachments: HADOOP-5626.patch
I have set up {{dfs.secondary.http.address}} like this:
{code}
<property>
<name>dfs.secondary.http.address</name>
<value>secondary.example.com:50090</value>
</property>
{code}
In my setup {{secondary.example.com}} resolves to an IP address (say, 192.168.0.10) which is not the same as the host's name (as returned by {{InetAddress.getLocalHost().getHostAddress()}}, say 192.168.0.1).
In this situation, edit log related transfers fail. From the namenode log:
{code}
2009-04-05 13:32:39,128 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from 192.168.0.10
2009-04-05 13:32:39,168 WARN org.mortbay.log: /getimage: java.io.IOException: GetImage failed. java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
at java.net.Socket.connect(Socket.java:519)
at java.net.Socket.connect(Socket.java:469)
at sun.net.NetworkClient.doConnect(NetworkClient.java:163)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:394)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:529)
at sun.net.www.http.HttpClient.<init>(HttpClient.java:233)
at sun.net.www.http.HttpClient.New(HttpClient.java:306)
at sun.net.www.http.HttpClient.New(HttpClient.java:323)
at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:837)
at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:778)
at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:703)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1026)
at org.apache.hadoop.hdfs.server.namenode.TransferFsImage.getFileClient(TransferFsImage.java:151)
...
{code}
>From the secondary namenode log:
{code}
2009-04-05 13:42:39,238 ERROR org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: Exception in doCheckpoint:
2009-04-05 13:42:39,238 ERROR org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: java.io.FileNotFoundException: http://nn.example.com:50070/getimage?putimage=1&port=50090&machine=
192.168.0.1&token=-19:1243068779:0:1238929357000:1238929031783
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1288)
at org.apache.hadoop.hdfs.server.namenode.TransferFsImage.getFileClient(TransferFsImage.java:151)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.putFSImage(SecondaryNameNode.java:294)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doCheckpoint(SecondaryNameNode.java:333)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.run(SecondaryNameNode.java:239)
at java.lang.Thread.run(Thread.java:619)
{code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5626) SecondaryNamenode may report
incorrect info host name
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Todd Lipcon updated HADOOP-5626:
--------------------------------
Attachment: hadoop-5626.txt
Fixes the behavior described in this ticket. Modifications to test improve the speed (100 seconds to 30 seconds on my machine where 0.0.0.0 lookup is very slow) and also verify the new behavior (this test fails with the old behavior as noted in HADOOP-3694)
> SecondaryNamenode may report incorrect info host name
> -----------------------------------------------------
>
> Key: HADOOP-5626
> URL: https://issues.apache.org/jira/browse/HADOOP-5626
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Reporter: Carlos Valiente
> Priority: Minor
> Fix For: 0.21.0
>
> Attachments: HADOOP-5626.patch, hadoop-5626.txt
>
>
> I have set up {{dfs.secondary.http.address}} like this:
> {code}
> <property>
> <name>dfs.secondary.http.address</name>
> <value>secondary.example.com:50090</value>
> </property>
> {code}
> In my setup {{secondary.example.com}} resolves to an IP address (say, 192.168.0.10) which is not the same as the host's name (as returned by {{InetAddress.getLocalHost().getHostAddress()}}, say 192.168.0.1).
> In this situation, edit log related transfers fail. From the namenode log:
> {code}
> 2009-04-05 13:32:39,128 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from 192.168.0.10
> 2009-04-05 13:32:39,168 WARN org.mortbay.log: /getimage: java.io.IOException: GetImage failed. java.net.ConnectException: Connection refused
> at java.net.PlainSocketImpl.socketConnect(Native Method)
> at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
> at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
> at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
> at java.net.Socket.connect(Socket.java:519)
> at java.net.Socket.connect(Socket.java:469)
> at sun.net.NetworkClient.doConnect(NetworkClient.java:163)
> at sun.net.www.http.HttpClient.openServer(HttpClient.java:394)
> at sun.net.www.http.HttpClient.openServer(HttpClient.java:529)
> at sun.net.www.http.HttpClient.<init>(HttpClient.java:233)
> at sun.net.www.http.HttpClient.New(HttpClient.java:306)
> at sun.net.www.http.HttpClient.New(HttpClient.java:323)
> at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:837)
> at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:778)
> at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:703)
> at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1026)
> at org.apache.hadoop.hdfs.server.namenode.TransferFsImage.getFileClient(TransferFsImage.java:151)
> ...
> {code}
> From the secondary namenode log:
> {code}
> 2009-04-05 13:42:39,238 ERROR org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: Exception in doCheckpoint:
> 2009-04-05 13:42:39,238 ERROR org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: java.io.FileNotFoundException: http://nn.example.com:50070/getimage?putimage=1&port=50090&machine=
> 192.168.0.1&token=-19:1243068779:0:1238929357000:1238929031783
> at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1288)
> at org.apache.hadoop.hdfs.server.namenode.TransferFsImage.getFileClient(TransferFsImage.java:151)
> at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.putFSImage(SecondaryNameNode.java:294)
> at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doCheckpoint(SecondaryNameNode.java:333)
> at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.run(SecondaryNameNode.java:239)
> at java.lang.Thread.run(Thread.java:619)
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5626) SecondaryNamenode may report
incorrect info host name
Posted by "Steve Loughran (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12712235#action_12712235 ]
Steve Loughran commented on HADOOP-5626:
----------------------------------------
I think we could tease out the code that caches the local hostname and make it more public; that would save the 30s hangs whenever something tries to look it up on one of my badly configured boxes.
For Java upload, POST is good, goes through firewalls too. HttpClient is the best way to do this -it is already part of the hadoop core library dependencies. Don't waste time trying to understand java.net if you can help it.
> SecondaryNamenode may report incorrect info host name
> -----------------------------------------------------
>
> Key: HADOOP-5626
> URL: https://issues.apache.org/jira/browse/HADOOP-5626
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Reporter: Carlos Valiente
> Assignee: Todd Lipcon
> Priority: Minor
> Fix For: 0.21.0
>
> Attachments: HADOOP-5626.patch, hadoop-5626.txt
>
>
> I have set up {{dfs.secondary.http.address}} like this:
> {code}
> <property>
> <name>dfs.secondary.http.address</name>
> <value>secondary.example.com:50090</value>
> </property>
> {code}
> In my setup {{secondary.example.com}} resolves to an IP address (say, 192.168.0.10) which is not the same as the host's name (as returned by {{InetAddress.getLocalHost().getHostAddress()}}, say 192.168.0.1).
> In this situation, edit log related transfers fail. From the namenode log:
> {code}
> 2009-04-05 13:32:39,128 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from 192.168.0.10
> 2009-04-05 13:32:39,168 WARN org.mortbay.log: /getimage: java.io.IOException: GetImage failed. java.net.ConnectException: Connection refused
> at java.net.PlainSocketImpl.socketConnect(Native Method)
> at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
> at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
> at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
> at java.net.Socket.connect(Socket.java:519)
> at java.net.Socket.connect(Socket.java:469)
> at sun.net.NetworkClient.doConnect(NetworkClient.java:163)
> at sun.net.www.http.HttpClient.openServer(HttpClient.java:394)
> at sun.net.www.http.HttpClient.openServer(HttpClient.java:529)
> at sun.net.www.http.HttpClient.<init>(HttpClient.java:233)
> at sun.net.www.http.HttpClient.New(HttpClient.java:306)
> at sun.net.www.http.HttpClient.New(HttpClient.java:323)
> at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:837)
> at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:778)
> at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:703)
> at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1026)
> at org.apache.hadoop.hdfs.server.namenode.TransferFsImage.getFileClient(TransferFsImage.java:151)
> ...
> {code}
> From the secondary namenode log:
> {code}
> 2009-04-05 13:42:39,238 ERROR org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: Exception in doCheckpoint:
> 2009-04-05 13:42:39,238 ERROR org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: java.io.FileNotFoundException: http://nn.example.com:50070/getimage?putimage=1&port=50090&machine=
> 192.168.0.1&token=-19:1243068779:0:1238929357000:1238929031783
> at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1288)
> at org.apache.hadoop.hdfs.server.namenode.TransferFsImage.getFileClient(TransferFsImage.java:151)
> at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.putFSImage(SecondaryNameNode.java:294)
> at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doCheckpoint(SecondaryNameNode.java:333)
> at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.run(SecondaryNameNode.java:239)
> at java.lang.Thread.run(Thread.java:619)
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5626) SecondaryNamenode may report
incorrect info host name
Posted by "Carlos Valiente (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Carlos Valiente updated HADOOP-5626:
------------------------------------
Attachment: HADOOP-5626.patch
> SecondaryNamenode may report incorrect info host name
> -----------------------------------------------------
>
> Key: HADOOP-5626
> URL: https://issues.apache.org/jira/browse/HADOOP-5626
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Reporter: Carlos Valiente
> Priority: Minor
> Fix For: 0.21.0
>
> Attachments: HADOOP-5626.patch
>
>
> I have set up {{dfs.secondary.http.address}} like this:
> {code}
> <property>
> <name>dfs.secondary.http.address</name>
> <value>secondary.example.com:50090</value>
> </property>
> {code}
> In my setup {{secondary.example.com}} resolves to an IP address (say, 192.168.0.10) which is not the same as the host's name (as returned by {{InetAddress.getLocalHost().getHostAddress()}}, say 192.168.0.1).
> In this situation, edit log related transfers fail. From the namenode log:
> {code}
> 2009-04-05 13:32:39,128 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from 192.168.0.10
> 2009-04-05 13:32:39,168 WARN org.mortbay.log: /getimage: java.io.IOException: GetImage failed. java.net.ConnectException: Connection refused
> at java.net.PlainSocketImpl.socketConnect(Native Method)
> at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
> at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
> at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
> at java.net.Socket.connect(Socket.java:519)
> at java.net.Socket.connect(Socket.java:469)
> at sun.net.NetworkClient.doConnect(NetworkClient.java:163)
> at sun.net.www.http.HttpClient.openServer(HttpClient.java:394)
> at sun.net.www.http.HttpClient.openServer(HttpClient.java:529)
> at sun.net.www.http.HttpClient.<init>(HttpClient.java:233)
> at sun.net.www.http.HttpClient.New(HttpClient.java:306)
> at sun.net.www.http.HttpClient.New(HttpClient.java:323)
> at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:837)
> at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:778)
> at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:703)
> at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1026)
> at org.apache.hadoop.hdfs.server.namenode.TransferFsImage.getFileClient(TransferFsImage.java:151)
> ...
> {code}
> From the secondary namenode log:
> {code}
> 2009-04-05 13:42:39,238 ERROR org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: Exception in doCheckpoint:
> 2009-04-05 13:42:39,238 ERROR org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: java.io.FileNotFoundException: http://nn.example.com:50070/getimage?putimage=1&port=50090&machine=
> 192.168.0.1&token=-19:1243068779:0:1238929357000:1238929031783
> at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1288)
> at org.apache.hadoop.hdfs.server.namenode.TransferFsImage.getFileClient(TransferFsImage.java:151)
> at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.putFSImage(SecondaryNameNode.java:294)
> at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doCheckpoint(SecondaryNameNode.java:333)
> at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.run(SecondaryNameNode.java:239)
> at java.lang.Thread.run(Thread.java:619)
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5626) SecondaryNamenode may report
incorrect info host name
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Todd Lipcon updated HADOOP-5626:
--------------------------------
Assignee: Todd Lipcon
Status: Patch Available (was: Open)
> SecondaryNamenode may report incorrect info host name
> -----------------------------------------------------
>
> Key: HADOOP-5626
> URL: https://issues.apache.org/jira/browse/HADOOP-5626
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Reporter: Carlos Valiente
> Assignee: Todd Lipcon
> Priority: Minor
> Fix For: 0.21.0
>
> Attachments: HADOOP-5626.patch, hadoop-5626.txt
>
>
> I have set up {{dfs.secondary.http.address}} like this:
> {code}
> <property>
> <name>dfs.secondary.http.address</name>
> <value>secondary.example.com:50090</value>
> </property>
> {code}
> In my setup {{secondary.example.com}} resolves to an IP address (say, 192.168.0.10) which is not the same as the host's name (as returned by {{InetAddress.getLocalHost().getHostAddress()}}, say 192.168.0.1).
> In this situation, edit log related transfers fail. From the namenode log:
> {code}
> 2009-04-05 13:32:39,128 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from 192.168.0.10
> 2009-04-05 13:32:39,168 WARN org.mortbay.log: /getimage: java.io.IOException: GetImage failed. java.net.ConnectException: Connection refused
> at java.net.PlainSocketImpl.socketConnect(Native Method)
> at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
> at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
> at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
> at java.net.Socket.connect(Socket.java:519)
> at java.net.Socket.connect(Socket.java:469)
> at sun.net.NetworkClient.doConnect(NetworkClient.java:163)
> at sun.net.www.http.HttpClient.openServer(HttpClient.java:394)
> at sun.net.www.http.HttpClient.openServer(HttpClient.java:529)
> at sun.net.www.http.HttpClient.<init>(HttpClient.java:233)
> at sun.net.www.http.HttpClient.New(HttpClient.java:306)
> at sun.net.www.http.HttpClient.New(HttpClient.java:323)
> at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:837)
> at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:778)
> at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:703)
> at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1026)
> at org.apache.hadoop.hdfs.server.namenode.TransferFsImage.getFileClient(TransferFsImage.java:151)
> ...
> {code}
> From the secondary namenode log:
> {code}
> 2009-04-05 13:42:39,238 ERROR org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: Exception in doCheckpoint:
> 2009-04-05 13:42:39,238 ERROR org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: java.io.FileNotFoundException: http://nn.example.com:50070/getimage?putimage=1&port=50090&machine=
> 192.168.0.1&token=-19:1243068779:0:1238929357000:1238929031783
> at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1288)
> at org.apache.hadoop.hdfs.server.namenode.TransferFsImage.getFileClient(TransferFsImage.java:151)
> at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.putFSImage(SecondaryNameNode.java:294)
> at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doCheckpoint(SecondaryNameNode.java:333)
> at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.run(SecondaryNameNode.java:239)
> at java.lang.Thread.run(Thread.java:619)
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5626) SecondaryNamenode may report
incorrect info host name
Posted by "Jakob Homan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12713657#action_12713657 ]
Jakob Homan commented on HADOOP-5626:
-------------------------------------
bq. For now, I think making some small concrete steps is the best route. History has shown that it's difficult (and takes a lot of time) to get larger sweeping changes through the development pipeline, so we should work towards some small achievable goals while planning out the better solution.
Service location issues are certainly worth looking at and you're right that HADOOP-4383 is a good place to start. My comment dealt with your specific patch, which as it stands, is introducing a bit of code duplication in terms of resolving the full hostname of the backup node and secondary namenode. I'd recommend updating your patch to avoid this by utilizing the net utils package as a repository of this rather generic function. No need to introduce two ways of doing the same thing. :)
> SecondaryNamenode may report incorrect info host name
> -----------------------------------------------------
>
> Key: HADOOP-5626
> URL: https://issues.apache.org/jira/browse/HADOOP-5626
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Reporter: Carlos Valiente
> Assignee: Todd Lipcon
> Priority: Minor
> Fix For: 0.21.0
>
> Attachments: HADOOP-5626.patch, hadoop-5626.txt
>
>
> I have set up {{dfs.secondary.http.address}} like this:
> {code}
> <property>
> <name>dfs.secondary.http.address</name>
> <value>secondary.example.com:50090</value>
> </property>
> {code}
> In my setup {{secondary.example.com}} resolves to an IP address (say, 192.168.0.10) which is not the same as the host's name (as returned by {{InetAddress.getLocalHost().getHostAddress()}}, say 192.168.0.1).
> In this situation, edit log related transfers fail. From the namenode log:
> {code}
> 2009-04-05 13:32:39,128 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from 192.168.0.10
> 2009-04-05 13:32:39,168 WARN org.mortbay.log: /getimage: java.io.IOException: GetImage failed. java.net.ConnectException: Connection refused
> at java.net.PlainSocketImpl.socketConnect(Native Method)
> at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
> at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
> at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
> at java.net.Socket.connect(Socket.java:519)
> at java.net.Socket.connect(Socket.java:469)
> at sun.net.NetworkClient.doConnect(NetworkClient.java:163)
> at sun.net.www.http.HttpClient.openServer(HttpClient.java:394)
> at sun.net.www.http.HttpClient.openServer(HttpClient.java:529)
> at sun.net.www.http.HttpClient.<init>(HttpClient.java:233)
> at sun.net.www.http.HttpClient.New(HttpClient.java:306)
> at sun.net.www.http.HttpClient.New(HttpClient.java:323)
> at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:837)
> at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:778)
> at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:703)
> at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1026)
> at org.apache.hadoop.hdfs.server.namenode.TransferFsImage.getFileClient(TransferFsImage.java:151)
> ...
> {code}
> From the secondary namenode log:
> {code}
> 2009-04-05 13:42:39,238 ERROR org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: Exception in doCheckpoint:
> 2009-04-05 13:42:39,238 ERROR org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: java.io.FileNotFoundException: http://nn.example.com:50070/getimage?putimage=1&port=50090&machine=
> 192.168.0.1&token=-19:1243068779:0:1238929357000:1238929031783
> at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1288)
> at org.apache.hadoop.hdfs.server.namenode.TransferFsImage.getFileClient(TransferFsImage.java:151)
> at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.putFSImage(SecondaryNameNode.java:294)
> at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doCheckpoint(SecondaryNameNode.java:333)
> at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.run(SecondaryNameNode.java:239)
> at java.lang.Thread.run(Thread.java:619)
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5626) SecondaryNamenode may report
incorrect info host name
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12712221#action_12712221 ]
Todd Lipcon commented on HADOOP-5626:
-------------------------------------
Steve: that would require the name of the local interface. In the default configuration, that isn't defined, unless I'm missing something.
Here's an alternate solution that I like better: I don't see any reason why the 2NN specifies its own IP address as the machine=<ip> parameter to the putimage request. There are two other options:
- The NN could simply use the requester address of the HTTP request to determine the 2NN's IP.
- The 2NN could simply use an HTTP PUT or POST to upload the checkpoint rather than requesting a GET callback. I don't know a lot about the Java HTTP facilities, but I assume this isn't too difficult.
> SecondaryNamenode may report incorrect info host name
> -----------------------------------------------------
>
> Key: HADOOP-5626
> URL: https://issues.apache.org/jira/browse/HADOOP-5626
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Reporter: Carlos Valiente
> Assignee: Todd Lipcon
> Priority: Minor
> Fix For: 0.21.0
>
> Attachments: HADOOP-5626.patch, hadoop-5626.txt
>
>
> I have set up {{dfs.secondary.http.address}} like this:
> {code}
> <property>
> <name>dfs.secondary.http.address</name>
> <value>secondary.example.com:50090</value>
> </property>
> {code}
> In my setup {{secondary.example.com}} resolves to an IP address (say, 192.168.0.10) which is not the same as the host's name (as returned by {{InetAddress.getLocalHost().getHostAddress()}}, say 192.168.0.1).
> In this situation, edit log related transfers fail. From the namenode log:
> {code}
> 2009-04-05 13:32:39,128 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from 192.168.0.10
> 2009-04-05 13:32:39,168 WARN org.mortbay.log: /getimage: java.io.IOException: GetImage failed. java.net.ConnectException: Connection refused
> at java.net.PlainSocketImpl.socketConnect(Native Method)
> at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
> at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
> at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
> at java.net.Socket.connect(Socket.java:519)
> at java.net.Socket.connect(Socket.java:469)
> at sun.net.NetworkClient.doConnect(NetworkClient.java:163)
> at sun.net.www.http.HttpClient.openServer(HttpClient.java:394)
> at sun.net.www.http.HttpClient.openServer(HttpClient.java:529)
> at sun.net.www.http.HttpClient.<init>(HttpClient.java:233)
> at sun.net.www.http.HttpClient.New(HttpClient.java:306)
> at sun.net.www.http.HttpClient.New(HttpClient.java:323)
> at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:837)
> at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:778)
> at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:703)
> at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1026)
> at org.apache.hadoop.hdfs.server.namenode.TransferFsImage.getFileClient(TransferFsImage.java:151)
> ...
> {code}
> From the secondary namenode log:
> {code}
> 2009-04-05 13:42:39,238 ERROR org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: Exception in doCheckpoint:
> 2009-04-05 13:42:39,238 ERROR org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: java.io.FileNotFoundException: http://nn.example.com:50070/getimage?putimage=1&port=50090&machine=
> 192.168.0.1&token=-19:1243068779:0:1238929357000:1238929031783
> at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1288)
> at org.apache.hadoop.hdfs.server.namenode.TransferFsImage.getFileClient(TransferFsImage.java:151)
> at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.putFSImage(SecondaryNameNode.java:294)
> at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doCheckpoint(SecondaryNameNode.java:333)
> at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.run(SecondaryNameNode.java:239)
> at java.lang.Thread.run(Thread.java:619)
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5626) SecondaryNamenode may report
incorrect info host name
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12711864#action_12711864 ]
Todd Lipcon commented on HADOOP-5626:
-------------------------------------
This is the issue that prevented TestCheckpoint from passing in HADOOP-3694.
My patch is the same spirit as Carlos's but uses InetAddress.isAnyLocalAddress instead of a string compare. Uploading shortly.
> SecondaryNamenode may report incorrect info host name
> -----------------------------------------------------
>
> Key: HADOOP-5626
> URL: https://issues.apache.org/jira/browse/HADOOP-5626
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Reporter: Carlos Valiente
> Priority: Minor
> Fix For: 0.21.0
>
> Attachments: HADOOP-5626.patch
>
>
> I have set up {{dfs.secondary.http.address}} like this:
> {code}
> <property>
> <name>dfs.secondary.http.address</name>
> <value>secondary.example.com:50090</value>
> </property>
> {code}
> In my setup {{secondary.example.com}} resolves to an IP address (say, 192.168.0.10) which is not the same as the host's name (as returned by {{InetAddress.getLocalHost().getHostAddress()}}, say 192.168.0.1).
> In this situation, edit log related transfers fail. From the namenode log:
> {code}
> 2009-04-05 13:32:39,128 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from 192.168.0.10
> 2009-04-05 13:32:39,168 WARN org.mortbay.log: /getimage: java.io.IOException: GetImage failed. java.net.ConnectException: Connection refused
> at java.net.PlainSocketImpl.socketConnect(Native Method)
> at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
> at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
> at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
> at java.net.Socket.connect(Socket.java:519)
> at java.net.Socket.connect(Socket.java:469)
> at sun.net.NetworkClient.doConnect(NetworkClient.java:163)
> at sun.net.www.http.HttpClient.openServer(HttpClient.java:394)
> at sun.net.www.http.HttpClient.openServer(HttpClient.java:529)
> at sun.net.www.http.HttpClient.<init>(HttpClient.java:233)
> at sun.net.www.http.HttpClient.New(HttpClient.java:306)
> at sun.net.www.http.HttpClient.New(HttpClient.java:323)
> at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:837)
> at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:778)
> at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:703)
> at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1026)
> at org.apache.hadoop.hdfs.server.namenode.TransferFsImage.getFileClient(TransferFsImage.java:151)
> ...
> {code}
> From the secondary namenode log:
> {code}
> 2009-04-05 13:42:39,238 ERROR org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: Exception in doCheckpoint:
> 2009-04-05 13:42:39,238 ERROR org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: java.io.FileNotFoundException: http://nn.example.com:50070/getimage?putimage=1&port=50090&machine=
> 192.168.0.1&token=-19:1243068779:0:1238929357000:1238929031783
> at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1288)
> at org.apache.hadoop.hdfs.server.namenode.TransferFsImage.getFileClient(TransferFsImage.java:151)
> at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.putFSImage(SecondaryNameNode.java:294)
> at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doCheckpoint(SecondaryNameNode.java:333)
> at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.run(SecondaryNameNode.java:239)
> at java.lang.Thread.run(Thread.java:619)
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5626) SecondaryNamenode may report
incorrect info host name
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12713318#action_12713318 ]
Todd Lipcon commented on HADOOP-5626:
-------------------------------------
Jakob and Steve: I definitely agree that we need a more centralized common way of doing this. I think HADOOP-4383 is a good place to suggest doing a more general overhaul of how services locate each other. I wrote up a brief proposal to Steve in a private email which I'll copy paste into that JIRA.
For now, I think making some small concrete steps is the best route. History has shown that it's difficult (and takes a lot of time) to get larger sweeping changes through the development pipeline, so we should work towards some small achievable goals while planning out the better solution.
> SecondaryNamenode may report incorrect info host name
> -----------------------------------------------------
>
> Key: HADOOP-5626
> URL: https://issues.apache.org/jira/browse/HADOOP-5626
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Reporter: Carlos Valiente
> Assignee: Todd Lipcon
> Priority: Minor
> Fix For: 0.21.0
>
> Attachments: HADOOP-5626.patch, hadoop-5626.txt
>
>
> I have set up {{dfs.secondary.http.address}} like this:
> {code}
> <property>
> <name>dfs.secondary.http.address</name>
> <value>secondary.example.com:50090</value>
> </property>
> {code}
> In my setup {{secondary.example.com}} resolves to an IP address (say, 192.168.0.10) which is not the same as the host's name (as returned by {{InetAddress.getLocalHost().getHostAddress()}}, say 192.168.0.1).
> In this situation, edit log related transfers fail. From the namenode log:
> {code}
> 2009-04-05 13:32:39,128 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from 192.168.0.10
> 2009-04-05 13:32:39,168 WARN org.mortbay.log: /getimage: java.io.IOException: GetImage failed. java.net.ConnectException: Connection refused
> at java.net.PlainSocketImpl.socketConnect(Native Method)
> at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
> at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
> at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
> at java.net.Socket.connect(Socket.java:519)
> at java.net.Socket.connect(Socket.java:469)
> at sun.net.NetworkClient.doConnect(NetworkClient.java:163)
> at sun.net.www.http.HttpClient.openServer(HttpClient.java:394)
> at sun.net.www.http.HttpClient.openServer(HttpClient.java:529)
> at sun.net.www.http.HttpClient.<init>(HttpClient.java:233)
> at sun.net.www.http.HttpClient.New(HttpClient.java:306)
> at sun.net.www.http.HttpClient.New(HttpClient.java:323)
> at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:837)
> at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:778)
> at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:703)
> at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1026)
> at org.apache.hadoop.hdfs.server.namenode.TransferFsImage.getFileClient(TransferFsImage.java:151)
> ...
> {code}
> From the secondary namenode log:
> {code}
> 2009-04-05 13:42:39,238 ERROR org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: Exception in doCheckpoint:
> 2009-04-05 13:42:39,238 ERROR org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: java.io.FileNotFoundException: http://nn.example.com:50070/getimage?putimage=1&port=50090&machine=
> 192.168.0.1&token=-19:1243068779:0:1238929357000:1238929031783
> at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1288)
> at org.apache.hadoop.hdfs.server.namenode.TransferFsImage.getFileClient(TransferFsImage.java:151)
> at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.putFSImage(SecondaryNameNode.java:294)
> at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doCheckpoint(SecondaryNameNode.java:333)
> at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.run(SecondaryNameNode.java:239)
> at java.lang.Thread.run(Thread.java:619)
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5626) SecondaryNamenode may report
incorrect info host name
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12712859#action_12712859 ]
Todd Lipcon commented on HADOOP-5626:
-------------------------------------
The test failures are unrelated (job history and capacity scheduler, both mapred whereas this only touches hdfs)
While I think the discussion above is valuable, since this blocks HADOOP-5888 I'd like for this to be committed and we can continue work on making image transfer more sane in another JIRA.
As for switching to POST rather than the "GET with a callback to another GET", I looked into this today and, while easy, I'm not sure it's a great idea. I think there are some people who are relying on the GetImageServlet on the 2NN for backup purposes, so removing the Jetty from the 2NN would break that.
> SecondaryNamenode may report incorrect info host name
> -----------------------------------------------------
>
> Key: HADOOP-5626
> URL: https://issues.apache.org/jira/browse/HADOOP-5626
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Reporter: Carlos Valiente
> Assignee: Todd Lipcon
> Priority: Minor
> Fix For: 0.21.0
>
> Attachments: HADOOP-5626.patch, hadoop-5626.txt
>
>
> I have set up {{dfs.secondary.http.address}} like this:
> {code}
> <property>
> <name>dfs.secondary.http.address</name>
> <value>secondary.example.com:50090</value>
> </property>
> {code}
> In my setup {{secondary.example.com}} resolves to an IP address (say, 192.168.0.10) which is not the same as the host's name (as returned by {{InetAddress.getLocalHost().getHostAddress()}}, say 192.168.0.1).
> In this situation, edit log related transfers fail. From the namenode log:
> {code}
> 2009-04-05 13:32:39,128 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from 192.168.0.10
> 2009-04-05 13:32:39,168 WARN org.mortbay.log: /getimage: java.io.IOException: GetImage failed. java.net.ConnectException: Connection refused
> at java.net.PlainSocketImpl.socketConnect(Native Method)
> at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
> at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
> at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
> at java.net.Socket.connect(Socket.java:519)
> at java.net.Socket.connect(Socket.java:469)
> at sun.net.NetworkClient.doConnect(NetworkClient.java:163)
> at sun.net.www.http.HttpClient.openServer(HttpClient.java:394)
> at sun.net.www.http.HttpClient.openServer(HttpClient.java:529)
> at sun.net.www.http.HttpClient.<init>(HttpClient.java:233)
> at sun.net.www.http.HttpClient.New(HttpClient.java:306)
> at sun.net.www.http.HttpClient.New(HttpClient.java:323)
> at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:837)
> at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:778)
> at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:703)
> at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1026)
> at org.apache.hadoop.hdfs.server.namenode.TransferFsImage.getFileClient(TransferFsImage.java:151)
> ...
> {code}
> From the secondary namenode log:
> {code}
> 2009-04-05 13:42:39,238 ERROR org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: Exception in doCheckpoint:
> 2009-04-05 13:42:39,238 ERROR org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: java.io.FileNotFoundException: http://nn.example.com:50070/getimage?putimage=1&port=50090&machine=
> 192.168.0.1&token=-19:1243068779:0:1238929357000:1238929031783
> at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1288)
> at org.apache.hadoop.hdfs.server.namenode.TransferFsImage.getFileClient(TransferFsImage.java:151)
> at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.putFSImage(SecondaryNameNode.java:294)
> at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doCheckpoint(SecondaryNameNode.java:333)
> at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.run(SecondaryNameNode.java:239)
> at java.lang.Thread.run(Thread.java:619)
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5626) SecondaryNamenode may report
incorrect info host name
Posted by "Jakob Homan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12713190#action_12713190 ]
Jakob Homan commented on HADOOP-5626:
-------------------------------------
In BackupNode::getHttpServerAddress this issue also arises and is handled by using the rpcAddress, which has already been resolved to a real address. It would be good to move your solution to NetUtils and access it from both the SecondaryNameNode and the BackupNode, to avoid code duplication.
> SecondaryNamenode may report incorrect info host name
> -----------------------------------------------------
>
> Key: HADOOP-5626
> URL: https://issues.apache.org/jira/browse/HADOOP-5626
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Reporter: Carlos Valiente
> Assignee: Todd Lipcon
> Priority: Minor
> Fix For: 0.21.0
>
> Attachments: HADOOP-5626.patch, hadoop-5626.txt
>
>
> I have set up {{dfs.secondary.http.address}} like this:
> {code}
> <property>
> <name>dfs.secondary.http.address</name>
> <value>secondary.example.com:50090</value>
> </property>
> {code}
> In my setup {{secondary.example.com}} resolves to an IP address (say, 192.168.0.10) which is not the same as the host's name (as returned by {{InetAddress.getLocalHost().getHostAddress()}}, say 192.168.0.1).
> In this situation, edit log related transfers fail. From the namenode log:
> {code}
> 2009-04-05 13:32:39,128 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from 192.168.0.10
> 2009-04-05 13:32:39,168 WARN org.mortbay.log: /getimage: java.io.IOException: GetImage failed. java.net.ConnectException: Connection refused
> at java.net.PlainSocketImpl.socketConnect(Native Method)
> at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
> at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
> at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
> at java.net.Socket.connect(Socket.java:519)
> at java.net.Socket.connect(Socket.java:469)
> at sun.net.NetworkClient.doConnect(NetworkClient.java:163)
> at sun.net.www.http.HttpClient.openServer(HttpClient.java:394)
> at sun.net.www.http.HttpClient.openServer(HttpClient.java:529)
> at sun.net.www.http.HttpClient.<init>(HttpClient.java:233)
> at sun.net.www.http.HttpClient.New(HttpClient.java:306)
> at sun.net.www.http.HttpClient.New(HttpClient.java:323)
> at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:837)
> at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:778)
> at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:703)
> at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1026)
> at org.apache.hadoop.hdfs.server.namenode.TransferFsImage.getFileClient(TransferFsImage.java:151)
> ...
> {code}
> From the secondary namenode log:
> {code}
> 2009-04-05 13:42:39,238 ERROR org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: Exception in doCheckpoint:
> 2009-04-05 13:42:39,238 ERROR org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: java.io.FileNotFoundException: http://nn.example.com:50070/getimage?putimage=1&port=50090&machine=
> 192.168.0.1&token=-19:1243068779:0:1238929357000:1238929031783
> at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1288)
> at org.apache.hadoop.hdfs.server.namenode.TransferFsImage.getFileClient(TransferFsImage.java:151)
> at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.putFSImage(SecondaryNameNode.java:294)
> at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doCheckpoint(SecondaryNameNode.java:333)
> at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.run(SecondaryNameNode.java:239)
> at java.lang.Thread.run(Thread.java:619)
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5626) SecondaryNamenode may report
incorrect info host name
Posted by "Steve Loughran (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12712217#action_12712217 ]
Steve Loughran commented on HADOOP-5626:
----------------------------------------
Todd,
-can you use the code in DNS to get localhost reliably on more systems?
> SecondaryNamenode may report incorrect info host name
> -----------------------------------------------------
>
> Key: HADOOP-5626
> URL: https://issues.apache.org/jira/browse/HADOOP-5626
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Reporter: Carlos Valiente
> Assignee: Todd Lipcon
> Priority: Minor
> Fix For: 0.21.0
>
> Attachments: HADOOP-5626.patch, hadoop-5626.txt
>
>
> I have set up {{dfs.secondary.http.address}} like this:
> {code}
> <property>
> <name>dfs.secondary.http.address</name>
> <value>secondary.example.com:50090</value>
> </property>
> {code}
> In my setup {{secondary.example.com}} resolves to an IP address (say, 192.168.0.10) which is not the same as the host's name (as returned by {{InetAddress.getLocalHost().getHostAddress()}}, say 192.168.0.1).
> In this situation, edit log related transfers fail. From the namenode log:
> {code}
> 2009-04-05 13:32:39,128 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from 192.168.0.10
> 2009-04-05 13:32:39,168 WARN org.mortbay.log: /getimage: java.io.IOException: GetImage failed. java.net.ConnectException: Connection refused
> at java.net.PlainSocketImpl.socketConnect(Native Method)
> at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
> at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
> at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
> at java.net.Socket.connect(Socket.java:519)
> at java.net.Socket.connect(Socket.java:469)
> at sun.net.NetworkClient.doConnect(NetworkClient.java:163)
> at sun.net.www.http.HttpClient.openServer(HttpClient.java:394)
> at sun.net.www.http.HttpClient.openServer(HttpClient.java:529)
> at sun.net.www.http.HttpClient.<init>(HttpClient.java:233)
> at sun.net.www.http.HttpClient.New(HttpClient.java:306)
> at sun.net.www.http.HttpClient.New(HttpClient.java:323)
> at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:837)
> at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:778)
> at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:703)
> at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1026)
> at org.apache.hadoop.hdfs.server.namenode.TransferFsImage.getFileClient(TransferFsImage.java:151)
> ...
> {code}
> From the secondary namenode log:
> {code}
> 2009-04-05 13:42:39,238 ERROR org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: Exception in doCheckpoint:
> 2009-04-05 13:42:39,238 ERROR org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: java.io.FileNotFoundException: http://nn.example.com:50070/getimage?putimage=1&port=50090&machine=
> 192.168.0.1&token=-19:1243068779:0:1238929357000:1238929031783
> at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1288)
> at org.apache.hadoop.hdfs.server.namenode.TransferFsImage.getFileClient(TransferFsImage.java:151)
> at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.putFSImage(SecondaryNameNode.java:294)
> at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doCheckpoint(SecondaryNameNode.java:333)
> at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.run(SecondaryNameNode.java:239)
> at java.lang.Thread.run(Thread.java:619)
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5626) SecondaryNamenode may report
incorrect info host name
Posted by "Steve Loughran (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12713211#action_12713211 ]
Steve Loughran commented on HADOOP-5626:
----------------------------------------
+1 to having something common
I think we also ought to spell out the minimum requirements of hadoop on a network, something like
/etc/hosts is well configured (local hostname doesnt map to 127.0.0.1 or ::1)
dns works
rDNS works
..etc
> SecondaryNamenode may report incorrect info host name
> -----------------------------------------------------
>
> Key: HADOOP-5626
> URL: https://issues.apache.org/jira/browse/HADOOP-5626
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Reporter: Carlos Valiente
> Assignee: Todd Lipcon
> Priority: Minor
> Fix For: 0.21.0
>
> Attachments: HADOOP-5626.patch, hadoop-5626.txt
>
>
> I have set up {{dfs.secondary.http.address}} like this:
> {code}
> <property>
> <name>dfs.secondary.http.address</name>
> <value>secondary.example.com:50090</value>
> </property>
> {code}
> In my setup {{secondary.example.com}} resolves to an IP address (say, 192.168.0.10) which is not the same as the host's name (as returned by {{InetAddress.getLocalHost().getHostAddress()}}, say 192.168.0.1).
> In this situation, edit log related transfers fail. From the namenode log:
> {code}
> 2009-04-05 13:32:39,128 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from 192.168.0.10
> 2009-04-05 13:32:39,168 WARN org.mortbay.log: /getimage: java.io.IOException: GetImage failed. java.net.ConnectException: Connection refused
> at java.net.PlainSocketImpl.socketConnect(Native Method)
> at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
> at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
> at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
> at java.net.Socket.connect(Socket.java:519)
> at java.net.Socket.connect(Socket.java:469)
> at sun.net.NetworkClient.doConnect(NetworkClient.java:163)
> at sun.net.www.http.HttpClient.openServer(HttpClient.java:394)
> at sun.net.www.http.HttpClient.openServer(HttpClient.java:529)
> at sun.net.www.http.HttpClient.<init>(HttpClient.java:233)
> at sun.net.www.http.HttpClient.New(HttpClient.java:306)
> at sun.net.www.http.HttpClient.New(HttpClient.java:323)
> at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:837)
> at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:778)
> at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:703)
> at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1026)
> at org.apache.hadoop.hdfs.server.namenode.TransferFsImage.getFileClient(TransferFsImage.java:151)
> ...
> {code}
> From the secondary namenode log:
> {code}
> 2009-04-05 13:42:39,238 ERROR org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: Exception in doCheckpoint:
> 2009-04-05 13:42:39,238 ERROR org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: java.io.FileNotFoundException: http://nn.example.com:50070/getimage?putimage=1&port=50090&machine=
> 192.168.0.1&token=-19:1243068779:0:1238929357000:1238929031783
> at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1288)
> at org.apache.hadoop.hdfs.server.namenode.TransferFsImage.getFileClient(TransferFsImage.java:151)
> at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.putFSImage(SecondaryNameNode.java:294)
> at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doCheckpoint(SecondaryNameNode.java:333)
> at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.run(SecondaryNameNode.java:239)
> at java.lang.Thread.run(Thread.java:619)
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5626) SecondaryNamenode may report
incorrect info host name
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Todd Lipcon updated HADOOP-5626:
--------------------------------
Status: Open (was: Patch Available)
Removing "Patch available" status to incorporate feedback and refactor the change into a static method in NetUtils.
> SecondaryNamenode may report incorrect info host name
> -----------------------------------------------------
>
> Key: HADOOP-5626
> URL: https://issues.apache.org/jira/browse/HADOOP-5626
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Reporter: Carlos Valiente
> Assignee: Todd Lipcon
> Priority: Minor
> Fix For: 0.21.0
>
> Attachments: HADOOP-5626.patch, hadoop-5626.txt
>
>
> I have set up {{dfs.secondary.http.address}} like this:
> {code}
> <property>
> <name>dfs.secondary.http.address</name>
> <value>secondary.example.com:50090</value>
> </property>
> {code}
> In my setup {{secondary.example.com}} resolves to an IP address (say, 192.168.0.10) which is not the same as the host's name (as returned by {{InetAddress.getLocalHost().getHostAddress()}}, say 192.168.0.1).
> In this situation, edit log related transfers fail. From the namenode log:
> {code}
> 2009-04-05 13:32:39,128 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from 192.168.0.10
> 2009-04-05 13:32:39,168 WARN org.mortbay.log: /getimage: java.io.IOException: GetImage failed. java.net.ConnectException: Connection refused
> at java.net.PlainSocketImpl.socketConnect(Native Method)
> at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
> at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
> at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
> at java.net.Socket.connect(Socket.java:519)
> at java.net.Socket.connect(Socket.java:469)
> at sun.net.NetworkClient.doConnect(NetworkClient.java:163)
> at sun.net.www.http.HttpClient.openServer(HttpClient.java:394)
> at sun.net.www.http.HttpClient.openServer(HttpClient.java:529)
> at sun.net.www.http.HttpClient.<init>(HttpClient.java:233)
> at sun.net.www.http.HttpClient.New(HttpClient.java:306)
> at sun.net.www.http.HttpClient.New(HttpClient.java:323)
> at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:837)
> at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:778)
> at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:703)
> at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1026)
> at org.apache.hadoop.hdfs.server.namenode.TransferFsImage.getFileClient(TransferFsImage.java:151)
> ...
> {code}
> From the secondary namenode log:
> {code}
> 2009-04-05 13:42:39,238 ERROR org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: Exception in doCheckpoint:
> 2009-04-05 13:42:39,238 ERROR org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: java.io.FileNotFoundException: http://nn.example.com:50070/getimage?putimage=1&port=50090&machine=
> 192.168.0.1&token=-19:1243068779:0:1238929357000:1238929031783
> at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1288)
> at org.apache.hadoop.hdfs.server.namenode.TransferFsImage.getFileClient(TransferFsImage.java:151)
> at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.putFSImage(SecondaryNameNode.java:294)
> at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doCheckpoint(SecondaryNameNode.java:333)
> at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.run(SecondaryNameNode.java:239)
> at java.lang.Thread.run(Thread.java:619)
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5626) SecondaryNamenode may report
incorrect info host name
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12713005#action_12713005 ]
dhruba borthakur commented on HADOOP-5626:
------------------------------------------
> As for switching to POST rather than the "GET with a callback to another GET", I
I remember that when this was being designed the first time, I was told that there could be "permission" related issues to do a POST vs a GET. The POST is like a "write" somewhat, whereas a GET is more like a "read".
> SecondaryNamenode may report incorrect info host name
> -----------------------------------------------------
>
> Key: HADOOP-5626
> URL: https://issues.apache.org/jira/browse/HADOOP-5626
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Reporter: Carlos Valiente
> Assignee: Todd Lipcon
> Priority: Minor
> Fix For: 0.21.0
>
> Attachments: HADOOP-5626.patch, hadoop-5626.txt
>
>
> I have set up {{dfs.secondary.http.address}} like this:
> {code}
> <property>
> <name>dfs.secondary.http.address</name>
> <value>secondary.example.com:50090</value>
> </property>
> {code}
> In my setup {{secondary.example.com}} resolves to an IP address (say, 192.168.0.10) which is not the same as the host's name (as returned by {{InetAddress.getLocalHost().getHostAddress()}}, say 192.168.0.1).
> In this situation, edit log related transfers fail. From the namenode log:
> {code}
> 2009-04-05 13:32:39,128 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from 192.168.0.10
> 2009-04-05 13:32:39,168 WARN org.mortbay.log: /getimage: java.io.IOException: GetImage failed. java.net.ConnectException: Connection refused
> at java.net.PlainSocketImpl.socketConnect(Native Method)
> at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
> at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
> at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
> at java.net.Socket.connect(Socket.java:519)
> at java.net.Socket.connect(Socket.java:469)
> at sun.net.NetworkClient.doConnect(NetworkClient.java:163)
> at sun.net.www.http.HttpClient.openServer(HttpClient.java:394)
> at sun.net.www.http.HttpClient.openServer(HttpClient.java:529)
> at sun.net.www.http.HttpClient.<init>(HttpClient.java:233)
> at sun.net.www.http.HttpClient.New(HttpClient.java:306)
> at sun.net.www.http.HttpClient.New(HttpClient.java:323)
> at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:837)
> at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:778)
> at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:703)
> at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1026)
> at org.apache.hadoop.hdfs.server.namenode.TransferFsImage.getFileClient(TransferFsImage.java:151)
> ...
> {code}
> From the secondary namenode log:
> {code}
> 2009-04-05 13:42:39,238 ERROR org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: Exception in doCheckpoint:
> 2009-04-05 13:42:39,238 ERROR org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: java.io.FileNotFoundException: http://nn.example.com:50070/getimage?putimage=1&port=50090&machine=
> 192.168.0.1&token=-19:1243068779:0:1238929357000:1238929031783
> at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1288)
> at org.apache.hadoop.hdfs.server.namenode.TransferFsImage.getFileClient(TransferFsImage.java:151)
> at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.putFSImage(SecondaryNameNode.java:294)
> at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doCheckpoint(SecondaryNameNode.java:333)
> at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.run(SecondaryNameNode.java:239)
> at java.lang.Thread.run(Thread.java:619)
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.