You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "smengcl (via GitHub)" <gi...@apache.org> on 2026/10/21 19:36:16 UTC
[PR] HDFS-18014. Fix slow peer lookup for IPv6 DataNodes in DatanodeManager [hadoop]
smengcl opened a new pull request, #8852:
URL: https://github.com/apache/hadoop/pull/8852
Generated-by: Claude Code (Opus 5.5)
### Description of PR
`DatanodeManager#getDnDescriptorsFromIpAddr` took the host from a slow peer key with `split(":")[0]`. For an IPv6 info address such as `2001:db8::1:9864` or `[2001:db8::1]:9864` that yields `2001` or `[2001`, so the DataNode was never found and slow node exclusion did nothing on IPv6 clusters.
The host is now everything before the last `:`, with surrounding brackets removed. IPv4 and hostname keys parse the same as before.
### How was this patch tested?
Added `TestDatanodeManager#testSlowPeersResolvedForIpv4AndIpv6`, which registers IPv4 and IPv6 DataNodes, reports them as slow peers using plain IPv4, plain IPv6 and bracketed IPv6 info addresses, and checks that `getAllSlowDataNodes` returns all three. It fails without the fix. `TestReplicationPolicyExcludeSlowNodes` still passes.
### For code changes:
- [x] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
- [ ] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
- [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files?
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HDFS-18014. Fix slow peer lookup for IPv6 DataNodes in DatanodeManager [hadoop]
Posted by "hfutatzhanghb (via GitHub)" <gi...@apache.org>.
hfutatzhanghb commented on PR #8852:
URL: https://github.com/apache/hadoop/pull/8852#issuecomment-6334339840
LGTM. +1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HDFS-18014. Fix slow peer lookup for IPv6 DataNodes in DatanodeManager [hadoop]
Posted by "jojochuang (via GitHub)" <gi...@apache.org>.
jojochuang merged PR #8852:
URL: https://github.com/apache/hadoop/pull/8852
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HDFS-18014. Fix slow peer lookup for IPv6 DataNodes in DatanodeManager [hadoop]
Posted by "smengcl (via GitHub)" <gi...@apache.org>.
smengcl commented on code in PR #8852:
URL: https://github.com/apache/hadoop/pull/8852#discussion_r4343233925
##########
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java:
##########
@@ -2264,7 +2264,11 @@ private List<DatanodeDescriptor> getDnDescriptorsFromIpAddr(List<String> nodes)
if (StringUtils.isBlank(node) || !node.contains(IP_PORT_SEPARATOR)) {
continue;
}
- String ipAddr = node.split(IP_PORT_SEPARATOR)[0];
+ // Port is after the last ':'; IPv6 hosts may be bracketed.
+ String ipAddr = node.substring(0, node.lastIndexOf(IP_PORT_SEPARATOR));
+ if (ipAddr.startsWith("[") && ipAddr.endsWith("]")) {
+ ipAddr = ipAddr.substring(1, ipAddr.length() - 1);
+ }
Review Comment:
Slow peer keys always come from `DatanodeID#getInfoAddr()` or `getInfoSecureAddr()`, which always append the port, so a bare IPv6 literal never reaches this method. If one did, the extracted host would not match any registered node and the key would be skipped, which is the same outcome as for any other malformed key (and the same as before this change). Leaving the guard as is to keep the patch narrow.
##########
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestDatanodeManager.java:
##########
@@ -195,6 +197,42 @@ public void testHost2NodeMapCorrectAfterReregister()
assertNotNull(dm.getDatanodeByHostName(hostnameNew), "should be a node with new hostname");
}
+ /**
+ * Slow peer reports keyed by IPv4, IPv6 and bracketed IPv6 info addresses
+ * must all resolve to their registered datanodes.
+ */
+ @Test
+ public void testSlowPeersResolvedForIpv4AndIpv6() throws IOException {
+ FSNamesystem fsn = Mockito.mock(FSNamesystem.class);
+ Mockito.when(fsn.hasWriteLock()).thenReturn(true);
+ Mockito.when(fsn.hasWriteLock(RwLockMode.BM)).thenReturn(true);
+ Configuration conf = new Configuration();
+ conf.setBoolean(DFSConfigKeys.DFS_DATANODE_PEER_STATS_ENABLED_KEY, true);
+ DatanodeManager dm = mockDatanodeManager(fsn, conf);
+ try {
+ String[] ips = {"1.2.3.4", "2001:db8::1", "2001:db8::2", "5.6.7.8"};
+ for (int i = 0; i < ips.length; i++) {
+ dm.registerDatanode(new DatanodeRegistration(
+ new DatanodeID(ips[i], "", "uuid" + i, 9866, 9864, 0, 9867), null, null, "version"));
+ }
+ DatanodeDescriptor ipv4 = dm.getDatanodeByHost(ips[0]);
+ DatanodeDescriptor ipv6 = dm.getDatanodeByHost(ips[1]);
+ DatanodeDescriptor ipv6Bracketed = dm.getDatanodeByHost(ips[2]);
+
+ SlowPeerTracker tracker = dm.getSlowPeerTracker();
+ OutlierMetrics metrics = new OutlierMetrics(0.0, 0.0, 0.0, 5.0);
+ tracker.addReport(ipv4.getInfoAddr(), ips[3], metrics);
+ tracker.addReport(ipv6.getInfoAddr(), ips[3], metrics);
Review Comment:
Good point. On trunk `getInfoAddr()` is unbracketed, but HDFS-18010 will bracket it, and then both cases would be bracketed. The test now spells out the three keys (`1.2.3.4:9864`, `2001:db8::1:9864`, `[2001:db8::2]:9864`) so both IPv6 forms stay covered.
##########
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestDatanodeManager.java:
##########
@@ -195,6 +197,42 @@ public void testHost2NodeMapCorrectAfterReregister()
assertNotNull(dm.getDatanodeByHostName(hostnameNew), "should be a node with new hostname");
}
+ /**
+ * Slow peer reports keyed by IPv4, IPv6 and bracketed IPv6 info addresses
+ * must all resolve to their registered datanodes.
+ */
+ @Test
+ public void testSlowPeersResolvedForIpv4AndIpv6() throws IOException {
+ FSNamesystem fsn = Mockito.mock(FSNamesystem.class);
+ Mockito.when(fsn.hasWriteLock()).thenReturn(true);
+ Mockito.when(fsn.hasWriteLock(RwLockMode.BM)).thenReturn(true);
+ Configuration conf = new Configuration();
+ conf.setBoolean(DFSConfigKeys.DFS_DATANODE_PEER_STATS_ENABLED_KEY, true);
+ DatanodeManager dm = mockDatanodeManager(fsn, conf);
+ try {
+ String[] ips = {"1.2.3.4", "2001:db8::1", "2001:db8::2", "5.6.7.8"};
+ for (int i = 0; i < ips.length; i++) {
+ dm.registerDatanode(new DatanodeRegistration(
+ new DatanodeID(ips[i], "", "uuid" + i, 9866, 9864, 0, 9867), null, null, "version"));
+ }
+ DatanodeDescriptor ipv4 = dm.getDatanodeByHost(ips[0]);
+ DatanodeDescriptor ipv6 = dm.getDatanodeByHost(ips[1]);
+ DatanodeDescriptor ipv6Bracketed = dm.getDatanodeByHost(ips[2]);
+
+ SlowPeerTracker tracker = dm.getSlowPeerTracker();
+ OutlierMetrics metrics = new OutlierMetrics(0.0, 0.0, 0.0, 5.0);
+ tracker.addReport(ipv4.getInfoAddr(), ips[3], metrics);
+ tracker.addReport(ipv6.getInfoAddr(), ips[3], metrics);
+ tracker.addReport(NetUtils.getHostPortString(ips[2], ipv6Bracketed.getInfoPort()), ips[3],
+ metrics);
+
+ assertThat(dm.getAllSlowDataNodes()).containsExactlyInAnyOrder(ipv4, ipv6, ipv6Bracketed);
Review Comment:
Renamed to `dnIpv4`, `dnIpv6PlainKey` and `dnIpv6BracketedKey` so the names describe the key form each node is reported with.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HDFS-18014. Fix slow peer lookup for IPv6 DataNodes in DatanodeManager [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #8852:
URL: https://github.com/apache/hadoop/pull/8852#issuecomment-6329299195
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 24s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 1 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 30m 36s | | trunk passed |
| +1 :green_heart: | compile | 0m 57s | | trunk passed with JDK Ubuntu-21.0.12.1+1-1-24.04.4-Ubuntu |
| +1 :green_heart: | compile | 0m 59s | | trunk passed with JDK Ubuntu-17.0.20.1+1-1-24.04-Ubuntu |
| +1 :green_heart: | checkstyle | 0m 58s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 4s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 49s | | trunk passed with JDK Ubuntu-21.0.12.1+1-1-24.04.4-Ubuntu |
| +1 :green_heart: | javadoc | 0m 52s | | trunk passed with JDK Ubuntu-17.0.20.1+1-1-24.04-Ubuntu |
| +1 :green_heart: | spotbugs | 2m 20s | | trunk passed |
| +1 :green_heart: | shadedclient | 18m 49s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 49s | | the patch passed |
| +1 :green_heart: | compile | 0m 43s | | the patch passed with JDK Ubuntu-21.0.12.1+1-1-24.04.4-Ubuntu |
| +1 :green_heart: | javac | 0m 43s | | the patch passed |
| +1 :green_heart: | compile | 0m 44s | | the patch passed with JDK Ubuntu-17.0.20.1+1-1-24.04-Ubuntu |
| +1 :green_heart: | javac | 0m 44s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 0m 44s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 52s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 38s | | the patch passed with JDK Ubuntu-21.0.12.1+1-1-24.04.4-Ubuntu |
| +1 :green_heart: | javadoc | 0m 38s | | the patch passed with JDK Ubuntu-17.0.20.1+1-1-24.04-Ubuntu |
| +1 :green_heart: | spotbugs | 2m 23s | | the patch passed |
| +1 :green_heart: | shadedclient | 18m 35s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| -1 :x: | unit | 180m 44s | [/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8852/1/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt) | hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 32s | | The patch does not generate ASF License warnings. |
| | | 264m 21s | | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.hdfs.server.namenode.TestNameNodeMXBean |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.56 ServerAPI=1.56 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8852/1/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/8852 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 4a23cebae94c 5.15.0-190-generic #200-Ubuntu SMP Fri Aug 7 15:06:04 UTC 2026 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / a550b44154b09ce0483f7e926866e5622b508628 |
| Default Java | Ubuntu-17.0.20.1+1-1-24.04-Ubuntu |
| Multi-JDK versions | /usr/lib/jvm/java-21-openjdk-amd64:Ubuntu-21.0.12.1+1-1-24.04.4-Ubuntu /usr/lib/jvm/java-17-openjdk-amd64:Ubuntu-17.0.20.1+1-1-24.04-Ubuntu |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8852/1/testReport/ |
| Max. process+thread count | 3587 (vs. ulimit of 10000) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8852/1/console |
| versions | git=2.43.0 maven=3.9.15 spotbugs=4.9.7 |
| Powered by | Apache Yetus 0.14.1 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HDFS-18014. Fix slow peer lookup for IPv6 DataNodes in DatanodeManager [hadoop]
Posted by "jojochuang (via GitHub)" <gi...@apache.org>.
jojochuang commented on PR #8852:
URL: https://github.com/apache/hadoop/pull/8852#issuecomment-6335545245
Merged. Thanks @smengcl @hfutatzhanghb for review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HDFS-18014. Fix slow peer lookup for IPv6 DataNodes in DatanodeManager [hadoop]
Posted by "smengcl (via GitHub)" <gi...@apache.org>.
smengcl commented on PR #8852:
URL: https://github.com/apache/hadoop/pull/8852#issuecomment-6336912881
Thanks @jojochuang @hfutatzhanghb for the reviews.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HDFS-18014. Fix slow peer lookup for IPv6 DataNodes in DatanodeManager [hadoop]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #8852:
URL: https://github.com/apache/hadoop/pull/8852#discussion_r4343100649
##########
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestDatanodeManager.java:
##########
@@ -195,6 +197,42 @@ public void testHost2NodeMapCorrectAfterReregister()
assertNotNull(dm.getDatanodeByHostName(hostnameNew), "should be a node with new hostname");
}
+ /**
+ * Slow peer reports keyed by IPv4, IPv6 and bracketed IPv6 info addresses
+ * must all resolve to their registered datanodes.
+ */
+ @Test
+ public void testSlowPeersResolvedForIpv4AndIpv6() throws IOException {
+ FSNamesystem fsn = Mockito.mock(FSNamesystem.class);
+ Mockito.when(fsn.hasWriteLock()).thenReturn(true);
+ Mockito.when(fsn.hasWriteLock(RwLockMode.BM)).thenReturn(true);
+ Configuration conf = new Configuration();
+ conf.setBoolean(DFSConfigKeys.DFS_DATANODE_PEER_STATS_ENABLED_KEY, true);
+ DatanodeManager dm = mockDatanodeManager(fsn, conf);
+ try {
+ String[] ips = {"1.2.3.4", "2001:db8::1", "2001:db8::2", "5.6.7.8"};
+ for (int i = 0; i < ips.length; i++) {
+ dm.registerDatanode(new DatanodeRegistration(
+ new DatanodeID(ips[i], "", "uuid" + i, 9866, 9864, 0, 9867), null, null, "version"));
+ }
+ DatanodeDescriptor ipv4 = dm.getDatanodeByHost(ips[0]);
+ DatanodeDescriptor ipv6 = dm.getDatanodeByHost(ips[1]);
+ DatanodeDescriptor ipv6Bracketed = dm.getDatanodeByHost(ips[2]);
+
+ SlowPeerTracker tracker = dm.getSlowPeerTracker();
+ OutlierMetrics metrics = new OutlierMetrics(0.0, 0.0, 0.0, 5.0);
+ tracker.addReport(ipv4.getInfoAddr(), ips[3], metrics);
+ tracker.addReport(ipv6.getInfoAddr(), ips[3], metrics);
Review Comment:
This test intends to cover 'plain IPv6' and 'bracketed IPv6' slow-peer keys, but `ipv6.getInfoAddr()` may already be bracketed depending on how the info address is formatted in the implementation, which can make the 'plain IPv6' case not actually exercised. To ensure the intended coverage, construct the unbracketed key explicitly (e.g., `ips[1] + \":\" + ipv6.getInfoPort()`) and keep the bracketed case via `NetUtils.getHostPortString(...)`.
##########
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestDatanodeManager.java:
##########
@@ -195,6 +197,42 @@ public void testHost2NodeMapCorrectAfterReregister()
assertNotNull(dm.getDatanodeByHostName(hostnameNew), "should be a node with new hostname");
}
+ /**
+ * Slow peer reports keyed by IPv4, IPv6 and bracketed IPv6 info addresses
+ * must all resolve to their registered datanodes.
+ */
+ @Test
+ public void testSlowPeersResolvedForIpv4AndIpv6() throws IOException {
+ FSNamesystem fsn = Mockito.mock(FSNamesystem.class);
+ Mockito.when(fsn.hasWriteLock()).thenReturn(true);
+ Mockito.when(fsn.hasWriteLock(RwLockMode.BM)).thenReturn(true);
+ Configuration conf = new Configuration();
+ conf.setBoolean(DFSConfigKeys.DFS_DATANODE_PEER_STATS_ENABLED_KEY, true);
+ DatanodeManager dm = mockDatanodeManager(fsn, conf);
+ try {
+ String[] ips = {"1.2.3.4", "2001:db8::1", "2001:db8::2", "5.6.7.8"};
+ for (int i = 0; i < ips.length; i++) {
+ dm.registerDatanode(new DatanodeRegistration(
+ new DatanodeID(ips[i], "", "uuid" + i, 9866, 9864, 0, 9867), null, null, "version"));
+ }
+ DatanodeDescriptor ipv4 = dm.getDatanodeByHost(ips[0]);
+ DatanodeDescriptor ipv6 = dm.getDatanodeByHost(ips[1]);
+ DatanodeDescriptor ipv6Bracketed = dm.getDatanodeByHost(ips[2]);
+
+ SlowPeerTracker tracker = dm.getSlowPeerTracker();
+ OutlierMetrics metrics = new OutlierMetrics(0.0, 0.0, 0.0, 5.0);
+ tracker.addReport(ipv4.getInfoAddr(), ips[3], metrics);
+ tracker.addReport(ipv6.getInfoAddr(), ips[3], metrics);
+ tracker.addReport(NetUtils.getHostPortString(ips[2], ipv6Bracketed.getInfoPort()), ips[3],
+ metrics);
+
+ assertThat(dm.getAllSlowDataNodes()).containsExactlyInAnyOrder(ipv4, ipv6, ipv6Bracketed);
Review Comment:
The variable name `ipv6Bracketed` is misleading: it holds a `DatanodeDescriptor` looked up by an unbracketed IPv6 literal (`ips[2]`), and the bracketed form is only used later when constructing the slow-peer key. Renaming to something like `ipv6ForBracketedKey` (or `ipv6_2`) would make the test intent clearer.
##########
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java:
##########
@@ -2264,7 +2264,11 @@ private List<DatanodeDescriptor> getDnDescriptorsFromIpAddr(List<String> nodes)
if (StringUtils.isBlank(node) || !node.contains(IP_PORT_SEPARATOR)) {
continue;
}
- String ipAddr = node.split(IP_PORT_SEPARATOR)[0];
+ // Port is after the last ':'; IPv6 hosts may be bracketed.
+ String ipAddr = node.substring(0, node.lastIndexOf(IP_PORT_SEPARATOR));
+ if (ipAddr.startsWith("[") && ipAddr.endsWith("]")) {
+ ipAddr = ipAddr.substring(1, ipAddr.length() - 1);
+ }
Review Comment:
The current guard `!node.contains(\":\")` is too weak: a bare IPv6 address (no port) contains ':' and will be mis-parsed by `substring(0, lastIndexOf(':'))`, producing an incorrect host that can never match (and potentially hiding malformed inputs). Consider tightening parsing to require an actual host+port form: for bracketed IPv6 require a `\"]:\"` delimiter; otherwise split on the last ':' only if the suffix is a valid numeric port (and skip/continue if not). This prevents accidentally treating plain IPv6 literals as host:port keys.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HDFS-18014. Fix slow peer lookup for IPv6 DataNodes in DatanodeManager [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #8852:
URL: https://github.com/apache/hadoop/pull/8852#issuecomment-6329352638
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 24s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 1 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 30m 31s | | trunk passed |
| +1 :green_heart: | compile | 0m 57s | | trunk passed with JDK Ubuntu-21.0.12.1+1-1-24.04.4-Ubuntu |
| +1 :green_heart: | compile | 1m 0s | | trunk passed with JDK Ubuntu-17.0.20.1+1-1-24.04-Ubuntu |
| +1 :green_heart: | checkstyle | 1m 2s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 3s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 53s | | trunk passed with JDK Ubuntu-21.0.12.1+1-1-24.04.4-Ubuntu |
| +1 :green_heart: | javadoc | 0m 49s | | trunk passed with JDK Ubuntu-17.0.20.1+1-1-24.04-Ubuntu |
| +1 :green_heart: | spotbugs | 2m 21s | | trunk passed |
| +1 :green_heart: | shadedclient | 18m 25s | | branch has no errors when building and testing our client artifacts. |
| -0 :warning: | patch | 18m 44s | | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 50s | | the patch passed |
| +1 :green_heart: | compile | 0m 48s | | the patch passed with JDK Ubuntu-21.0.12.1+1-1-24.04.4-Ubuntu |
| +1 :green_heart: | javac | 0m 48s | | the patch passed |
| +1 :green_heart: | compile | 0m 48s | | the patch passed with JDK Ubuntu-17.0.20.1+1-1-24.04-Ubuntu |
| +1 :green_heart: | javac | 0m 48s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 0m 46s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 53s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 36s | | the patch passed with JDK Ubuntu-21.0.12.1+1-1-24.04.4-Ubuntu |
| +1 :green_heart: | javadoc | 0m 44s | | the patch passed with JDK Ubuntu-17.0.20.1+1-1-24.04-Ubuntu |
| +1 :green_heart: | spotbugs | 2m 8s | | the patch passed |
| +1 :green_heart: | shadedclient | 17m 50s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 182m 21s | | hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 33s | | The patch does not generate ASF License warnings. |
| | | 265m 19s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.56 ServerAPI=1.56 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8852/2/artifact/out/Dockerfile |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 14f80b3de17c 5.15.0-190-generic #200-Ubuntu SMP Fri Aug 7 15:06:04 UTC 2026 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 35bbaa10e886f10947f35f1ae32e86e6c4c52ef1 |
| Default Java | Ubuntu-17.0.20.1+1-1-24.04-Ubuntu |
| Multi-JDK versions | /usr/lib/jvm/java-21-openjdk-amd64:Ubuntu-21.0.12.1+1-1-24.04.4-Ubuntu /usr/lib/jvm/java-17-openjdk-amd64:Ubuntu-17.0.20.1+1-1-24.04-Ubuntu |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8852/2/testReport/ |
| Max. process+thread count | 3954 (vs. ulimit of 10000) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8852/2/console |
| versions | git=2.43.0 maven=3.9.15 spotbugs=4.9.7 |
| Powered by | Apache Yetus 0.14.1 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HDFS-18014. Fix slow peer lookup for IPv6 DataNodes in DatanodeManager [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #8852:
URL: https://github.com/apache/hadoop/pull/8852#issuecomment-6332650947
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 29s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 1 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 27m 20s | | trunk passed |
| +1 :green_heart: | compile | 1m 1s | | trunk passed with JDK Ubuntu-21.0.12.1+1-1-24.04.4-Ubuntu |
| +1 :green_heart: | compile | 1m 3s | | trunk passed with JDK Ubuntu-17.0.20.1+1-1-24.04-Ubuntu |
| +1 :green_heart: | checkstyle | 1m 7s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 7s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 53s | | trunk passed with JDK Ubuntu-21.0.12.1+1-1-24.04.4-Ubuntu |
| +1 :green_heart: | javadoc | 0m 55s | | trunk passed with JDK Ubuntu-17.0.20.1+1-1-24.04-Ubuntu |
| +1 :green_heart: | spotbugs | 2m 17s | | trunk passed |
| +1 :green_heart: | shadedclient | 17m 42s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 45s | | the patch passed |
| +1 :green_heart: | compile | 0m 42s | | the patch passed with JDK Ubuntu-21.0.12.1+1-1-24.04.4-Ubuntu |
| +1 :green_heart: | javac | 0m 42s | | the patch passed |
| +1 :green_heart: | compile | 0m 43s | | the patch passed with JDK Ubuntu-17.0.20.1+1-1-24.04-Ubuntu |
| +1 :green_heart: | javac | 0m 43s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 0m 43s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 49s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 32s | | the patch passed with JDK Ubuntu-21.0.12.1+1-1-24.04.4-Ubuntu |
| +1 :green_heart: | javadoc | 0m 36s | | the patch passed with JDK Ubuntu-17.0.20.1+1-1-24.04-Ubuntu |
| +1 :green_heart: | spotbugs | 2m 8s | | the patch passed |
| +1 :green_heart: | shadedclient | 16m 43s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 183m 57s | | hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 31s | | The patch does not generate ASF License warnings. |
| | | 261m 43s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.56 ServerAPI=1.56 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8852/3/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/8852 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux cab5c9e9a4bb 5.15.0-190-generic #200-Ubuntu SMP Fri Aug 7 15:06:04 UTC 2026 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / badd8c78ae05707459f14f6c2672632f7be59eb7 |
| Default Java | Ubuntu-17.0.20.1+1-1-24.04-Ubuntu |
| Multi-JDK versions | /usr/lib/jvm/java-21-openjdk-amd64:Ubuntu-21.0.12.1+1-1-24.04.4-Ubuntu /usr/lib/jvm/java-17-openjdk-amd64:Ubuntu-17.0.20.1+1-1-24.04-Ubuntu |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8852/3/testReport/ |
| Max. process+thread count | 3825 (vs. ulimit of 10000) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8852/3/console |
| versions | git=2.43.0 maven=3.9.15 spotbugs=4.9.7 |
| Powered by | Apache Yetus 0.14.1 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org