You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Jakob Homan (JIRA)" <ji...@apache.org> on 2010/04/06 01:54:27 UTC
[jira] Created: (HADOOP-6682) NetUtils:normalizeHostName does not
process hostnames starting with [a-f] correctly
NetUtils:normalizeHostName does not process hostnames starting with [a-f] correctly
-----------------------------------------------------------------------------------
Key: HADOOP-6682
URL: https://issues.apache.org/jira/browse/HADOOP-6682
Project: Hadoop Common
Issue Type: Bug
Components: io
Reporter: Jakob Homan
public static String normalizeHostName(String name) {
if (Character.digit(name.charAt(0), 16) != -1) {
return name;
This code is attempting to short-circuit the hostname->ip resolution on the assumption that if name starts with a digit, it's already an ip address. This is of questionable value, but because it checks for a hex digit, it will fail on names starting with [a-f]. Such names will not be converted to an ip address, but be returned unchanged.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-6682) NetUtils:normalizeHostName does not
process hostnames starting with [a-f] correctly
Posted by "Jakob Homan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jakob Homan updated HADOOP-6682:
--------------------------------
Status: Patch Available (was: Open)
submitting patch.
> NetUtils:normalizeHostName does not process hostnames starting with [a-f] correctly
> -----------------------------------------------------------------------------------
>
> Key: HADOOP-6682
> URL: https://issues.apache.org/jira/browse/HADOOP-6682
> Project: Hadoop Common
> Issue Type: Bug
> Components: io
> Reporter: Jakob Homan
> Attachments: HADOOP-6682-Y20.patch, HADOOP-6682.patch
>
>
> public static String normalizeHostName(String name) {
> if (Character.digit(name.charAt(0), 16) != -1) {
> return name;
> This code is attempting to short-circuit the hostname->ip resolution on the assumption that if name starts with a digit, it's already an ip address. This is of questionable value, but because it checks for a hex digit, it will fail on names starting with [a-f]. Such names will not be converted to an ip address, but be returned unchanged.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6682) NetUtils:normalizeHostName does not
process hostnames starting with [a-f] correctly
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12878196#action_12878196 ]
Hudson commented on HADOOP-6682:
--------------------------------
Integrated in Hadoop-Common-trunk-Commit #301 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk-Commit/301/])
HADOOP-6682. NetUtils:normalizeHostName does not process hostnames starting with [a-f] correctly.
> NetUtils:normalizeHostName does not process hostnames starting with [a-f] correctly
> -----------------------------------------------------------------------------------
>
> Key: HADOOP-6682
> URL: https://issues.apache.org/jira/browse/HADOOP-6682
> Project: Hadoop Common
> Issue Type: Bug
> Components: io
> Reporter: Jakob Homan
> Assignee: Jakob Homan
> Fix For: 0.22.0
>
> Attachments: HADOOP-6682-Y20.patch, HADOOP-6682.patch
>
>
> public static String normalizeHostName(String name) {
> if (Character.digit(name.charAt(0), 16) != -1) {
> return name;
> This code is attempting to short-circuit the hostname->ip resolution on the assumption that if name starts with a digit, it's already an ip address. This is of questionable value, but because it checks for a hex digit, it will fail on names starting with [a-f]. Such names will not be converted to an ip address, but be returned unchanged.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6682) NetUtils:normalizeHostName does not
process hostnames starting with [a-f] correctly
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12878252#action_12878252 ]
Hudson commented on HADOOP-6682:
--------------------------------
Integrated in Hadoop-Common-trunk #364 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/364/])
HADOOP-6682. NetUtils:normalizeHostName does not process hostnames starting with [a-f] correctly.
> NetUtils:normalizeHostName does not process hostnames starting with [a-f] correctly
> -----------------------------------------------------------------------------------
>
> Key: HADOOP-6682
> URL: https://issues.apache.org/jira/browse/HADOOP-6682
> Project: Hadoop Common
> Issue Type: Bug
> Components: io
> Reporter: Jakob Homan
> Assignee: Jakob Homan
> Fix For: 0.22.0
>
> Attachments: HADOOP-6682-Y20.patch, HADOOP-6682.patch
>
>
> public static String normalizeHostName(String name) {
> if (Character.digit(name.charAt(0), 16) != -1) {
> return name;
> This code is attempting to short-circuit the hostname->ip resolution on the assumption that if name starts with a digit, it's already an ip address. This is of questionable value, but because it checks for a hex digit, it will fail on names starting with [a-f]. Such names will not be converted to an ip address, but be returned unchanged.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-6682) NetUtils:normalizeHostName does not
process hostnames starting with [a-f] correctly
Posted by "Jakob Homan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jakob Homan updated HADOOP-6682:
--------------------------------
Status: Resolved (was: Patch Available)
Hadoop Flags: [Reviewed]
Assignee: Jakob Homan
Fix Version/s: 0.22.0
Resolution: Fixed
I've committed this. Resolving as fixed.
> NetUtils:normalizeHostName does not process hostnames starting with [a-f] correctly
> -----------------------------------------------------------------------------------
>
> Key: HADOOP-6682
> URL: https://issues.apache.org/jira/browse/HADOOP-6682
> Project: Hadoop Common
> Issue Type: Bug
> Components: io
> Reporter: Jakob Homan
> Assignee: Jakob Homan
> Fix For: 0.22.0
>
> Attachments: HADOOP-6682-Y20.patch, HADOOP-6682.patch
>
>
> public static String normalizeHostName(String name) {
> if (Character.digit(name.charAt(0), 16) != -1) {
> return name;
> This code is attempting to short-circuit the hostname->ip resolution on the assumption that if name starts with a digit, it's already an ip address. This is of questionable value, but because it checks for a hex digit, it will fail on names starting with [a-f]. Such names will not be converted to an ip address, but be returned unchanged.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-6682) NetUtils:normalizeHostName does not
process hostnames starting with [a-f] correctly
Posted by "Jakob Homan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jakob Homan updated HADOOP-6682:
--------------------------------
Attachment: HADOOP-6682.patch
Patch for trunk. Same as Y20 patch. Just changes the base to 10 rather than 16.
Hong, I like your approach and will open another JIRA to address the general shortcoming of the normalizeHostname function. For this one I'd like to take as little risk as possible.
I considered writing a unit test for the fix that would try to normalize some host name that starts with an offending hostname (ask.com), but this would violate the unit-test-shouldn't-talk-to-outside-resources guideline, and couldn't think of a better way to test it. If someone has one, let me and I'll add it. Otherwise, I think it's good to go without one.
> NetUtils:normalizeHostName does not process hostnames starting with [a-f] correctly
> -----------------------------------------------------------------------------------
>
> Key: HADOOP-6682
> URL: https://issues.apache.org/jira/browse/HADOOP-6682
> Project: Hadoop Common
> Issue Type: Bug
> Components: io
> Reporter: Jakob Homan
> Attachments: HADOOP-6682-Y20.patch, HADOOP-6682.patch
>
>
> public static String normalizeHostName(String name) {
> if (Character.digit(name.charAt(0), 16) != -1) {
> return name;
> This code is attempting to short-circuit the hostname->ip resolution on the assumption that if name starts with a digit, it's already an ip address. This is of questionable value, but because it checks for a hex digit, it will fail on names starting with [a-f]. Such names will not be converted to an ip address, but be returned unchanged.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-6682) NetUtils:normalizeHostName does not
process hostnames starting with [a-f] correctly
Posted by "Jakob Homan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jakob Homan updated HADOOP-6682:
--------------------------------
Status: Open (was: Patch Available)
I call bogus on the javadoc warning, since this patch doesn't affect that. Re-run Hudson for test failure. I don't see it on my machine, but let's see if it re-occurs.
> NetUtils:normalizeHostName does not process hostnames starting with [a-f] correctly
> -----------------------------------------------------------------------------------
>
> Key: HADOOP-6682
> URL: https://issues.apache.org/jira/browse/HADOOP-6682
> Project: Hadoop Common
> Issue Type: Bug
> Components: io
> Reporter: Jakob Homan
> Attachments: HADOOP-6682-Y20.patch, HADOOP-6682.patch
>
>
> public static String normalizeHostName(String name) {
> if (Character.digit(name.charAt(0), 16) != -1) {
> return name;
> This code is attempting to short-circuit the hostname->ip resolution on the assumption that if name starts with a digit, it's already an ip address. This is of questionable value, but because it checks for a hex digit, it will fail on names starting with [a-f]. Such names will not be converted to an ip address, but be returned unchanged.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6682) NetUtils:normalizeHostName does not
process hostnames starting with [a-f] correctly
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12878183#action_12878183 ]
Hadoop QA commented on HADOOP-6682:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12446933/HADOOP-6682.patch
against trunk revision 953910.
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
-1 javadoc. The javadoc tool appears to have generated 1 warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/81/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/81/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/81/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/81/console
This message is automatically generated.
> NetUtils:normalizeHostName does not process hostnames starting with [a-f] correctly
> -----------------------------------------------------------------------------------
>
> Key: HADOOP-6682
> URL: https://issues.apache.org/jira/browse/HADOOP-6682
> Project: Hadoop Common
> Issue Type: Bug
> Components: io
> Reporter: Jakob Homan
> Attachments: HADOOP-6682-Y20.patch, HADOOP-6682.patch
>
>
> public static String normalizeHostName(String name) {
> if (Character.digit(name.charAt(0), 16) != -1) {
> return name;
> This code is attempting to short-circuit the hostname->ip resolution on the assumption that if name starts with a digit, it's already an ip address. This is of questionable value, but because it checks for a hex digit, it will fail on names starting with [a-f]. Such names will not be converted to an ip address, but be returned unchanged.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6682) NetUtils:normalizeHostName does not
process hostnames starting with [a-f] correctly
Posted by "Devaraj Das (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12878172#action_12878172 ]
Devaraj Das commented on HADOOP-6682:
-------------------------------------
+1
> NetUtils:normalizeHostName does not process hostnames starting with [a-f] correctly
> -----------------------------------------------------------------------------------
>
> Key: HADOOP-6682
> URL: https://issues.apache.org/jira/browse/HADOOP-6682
> Project: Hadoop Common
> Issue Type: Bug
> Components: io
> Reporter: Jakob Homan
> Attachments: HADOOP-6682-Y20.patch, HADOOP-6682.patch
>
>
> public static String normalizeHostName(String name) {
> if (Character.digit(name.charAt(0), 16) != -1) {
> return name;
> This code is attempting to short-circuit the hostname->ip resolution on the assumption that if name starts with a digit, it's already an ip address. This is of questionable value, but because it checks for a hex digit, it will fail on names starting with [a-f]. Such names will not be converted to an ip address, but be returned unchanged.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6682) NetUtils:normalizeHostName does not
process hostnames starting with [a-f] correctly
Posted by "Hong Tang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12853740#action_12853740 ]
Hong Tang commented on HADOOP-6682:
-----------------------------------
The regex IPV4BKN needs some change because in job history dots (.) are escaped with double backslashes.
> NetUtils:normalizeHostName does not process hostnames starting with [a-f] correctly
> -----------------------------------------------------------------------------------
>
> Key: HADOOP-6682
> URL: https://issues.apache.org/jira/browse/HADOOP-6682
> Project: Hadoop Common
> Issue Type: Bug
> Components: io
> Reporter: Jakob Homan
>
> public static String normalizeHostName(String name) {
> if (Character.digit(name.charAt(0), 16) != -1) {
> return name;
> This code is attempting to short-circuit the hostname->ip resolution on the assumption that if name starts with a digit, it's already an ip address. This is of questionable value, but because it checks for a hex digit, it will fail on names starting with [a-f]. Such names will not be converted to an ip address, but be returned unchanged.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-6682) NetUtils:normalizeHostName does not
process hostnames starting with [a-f] correctly
Posted by "Jakob Homan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jakob Homan updated HADOOP-6682:
--------------------------------
Status: Patch Available (was: Open)
> NetUtils:normalizeHostName does not process hostnames starting with [a-f] correctly
> -----------------------------------------------------------------------------------
>
> Key: HADOOP-6682
> URL: https://issues.apache.org/jira/browse/HADOOP-6682
> Project: Hadoop Common
> Issue Type: Bug
> Components: io
> Reporter: Jakob Homan
> Attachments: HADOOP-6682-Y20.patch, HADOOP-6682.patch
>
>
> public static String normalizeHostName(String name) {
> if (Character.digit(name.charAt(0), 16) != -1) {
> return name;
> This code is attempting to short-circuit the hostname->ip resolution on the assumption that if name starts with a digit, it's already an ip address. This is of questionable value, but because it checks for a hex digit, it will fail on names starting with [a-f]. Such names will not be converted to an ip address, but be returned unchanged.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-6682) NetUtils:normalizeHostName does not
process hostnames starting with [a-f] correctly
Posted by "Jakob Homan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jakob Homan updated HADOOP-6682:
--------------------------------
Attachment: HADOOP-6682-Y20.patch
Patch for Y20, which we forgot to upload at the time. Not for commit.
> NetUtils:normalizeHostName does not process hostnames starting with [a-f] correctly
> -----------------------------------------------------------------------------------
>
> Key: HADOOP-6682
> URL: https://issues.apache.org/jira/browse/HADOOP-6682
> Project: Hadoop Common
> Issue Type: Bug
> Components: io
> Reporter: Jakob Homan
> Attachments: HADOOP-6682-Y20.patch
>
>
> public static String normalizeHostName(String name) {
> if (Character.digit(name.charAt(0), 16) != -1) {
> return name;
> This code is attempting to short-circuit the hostname->ip resolution on the assumption that if name starts with a digit, it's already an ip address. This is of questionable value, but because it checks for a hex digit, it will fail on names starting with [a-f]. Such names will not be converted to an ip address, but be returned unchanged.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6682) NetUtils:normalizeHostName does not
process hostnames starting with [a-f] correctly
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12878167#action_12878167 ]
Hadoop QA commented on HADOOP-6682:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12446933/HADOOP-6682.patch
against trunk revision 953910.
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
-1 javadoc. The javadoc tool appears to have generated 1 warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/79/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/79/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/79/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/79/console
This message is automatically generated.
> NetUtils:normalizeHostName does not process hostnames starting with [a-f] correctly
> -----------------------------------------------------------------------------------
>
> Key: HADOOP-6682
> URL: https://issues.apache.org/jira/browse/HADOOP-6682
> Project: Hadoop Common
> Issue Type: Bug
> Components: io
> Reporter: Jakob Homan
> Attachments: HADOOP-6682-Y20.patch, HADOOP-6682.patch
>
>
> public static String normalizeHostName(String name) {
> if (Character.digit(name.charAt(0), 16) != -1) {
> return name;
> This code is attempting to short-circuit the hostname->ip resolution on the assumption that if name starts with a digit, it's already an ip address. This is of questionable value, but because it checks for a hex digit, it will fail on names starting with [a-f]. Such names will not be converted to an ip address, but be returned unchanged.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6682) NetUtils:normalizeHostName does not
process hostnames starting with [a-f] correctly
Posted by "Hong Tang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12853739#action_12853739 ]
Hong Tang commented on HADOOP-6682:
-----------------------------------
Similar issue came up in MAPREDUCE-1222. The following code might be useful:
{noformat}
+ static Pattern IP_PATTERN;
+
+ static {
+ // 0-255
+ String IPV4BK1 = "(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)";
+ // .b.c.d - where b/c/d are 0-255, and optionally adding two more
+ // backslashes before each period
+ String IPV4BKN = "(?:\\\\?\\." + IPV4BK1 + "){3}";
+ String IPV4_PATTERN = IPV4BK1 + IPV4BKN;
+
+ // first hexadecimal number
+ String IPV6BK1 = "(?:[0-9a-fA-F]{1,4})";
+ // remaining 7 hexadecimal numbers, each preceded with ":".
+ String IPV6BKN = "(?::" + IPV6BK1 + "){7}";
+ String IPV6_PATTERN = IPV6BK1 + IPV6BKN;
+
+ IP_PATTERN = Pattern.compile(
+ "^(?:" + IPV4_PATTERN + "|" + IPV6_PATTERN + ")$");
+ }
+
+
+ static boolean isIPAddress(String hostname) {
+ return IP_PATTERN.matcher(hostname).matches();
+ }
{noformat}
> NetUtils:normalizeHostName does not process hostnames starting with [a-f] correctly
> -----------------------------------------------------------------------------------
>
> Key: HADOOP-6682
> URL: https://issues.apache.org/jira/browse/HADOOP-6682
> Project: Hadoop Common
> Issue Type: Bug
> Components: io
> Reporter: Jakob Homan
>
> public static String normalizeHostName(String name) {
> if (Character.digit(name.charAt(0), 16) != -1) {
> return name;
> This code is attempting to short-circuit the hostname->ip resolution on the assumption that if name starts with a digit, it's already an ip address. This is of questionable value, but because it checks for a hex digit, it will fail on names starting with [a-f]. Such names will not be converted to an ip address, but be returned unchanged.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.