You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "ASF GitHub Bot (Jira)" <ji...@apache.org> on 2022/07/29 15:08:00 UTC
[jira] [Work logged] (HDFS-16700) RBF: Record the real client IP carried by the Router in the NameNode log
[ https://issues.apache.org/jira/browse/HDFS-16700?focusedWorklogId=796416&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-796416 ]
ASF GitHub Bot logged work on HDFS-16700:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 29/Jul/22 15:07
Start Date: 29/Jul/22 15:07
Worklog Time Spent: 10m
Work Description: goiri commented on code in PR #4659:
URL: https://github.com/apache/hadoop/pull/4659#discussion_r933363776
##########
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Server.java:
##########
@@ -1351,6 +1357,33 @@ private class ResponseParams {
@Override
public String toString() {
+ boolean isCallerContextEnabled = conf.getBoolean(
+ HADOOP_CALLER_CONTEXT_ENABLED_KEY,
+ HADOOP_CALLER_CONTEXT_ENABLED_DEFAULT);
+ CallerContext context = getCallerContext();
+ if (isCallerContextEnabled && context != null && context.isContextValid()) {
+ String cc = context.getContext();
+ String clientIp = "";
+ String ipKey = CallerContext.CLIENT_IP_STR +
+ CallerContext.Builder.KEY_VALUE_SEPARATOR;
+ int posn = cc.indexOf(ipKey);
+ if (posn != -1) {
+ posn += ipKey.length();
+ int end = cc.indexOf(",", posn);
+ clientIp = (end == -1 ? cc.substring(posn) : cc.substring(posn, end));
+ }
+ String clientPort = "";
+ String portKey = CallerContext.CLIENT_PORT_STR +
Review Comment:
Single line.
##########
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Server.java:
##########
@@ -1351,6 +1357,33 @@ private class ResponseParams {
@Override
public String toString() {
+ boolean isCallerContextEnabled = conf.getBoolean(
+ HADOOP_CALLER_CONTEXT_ENABLED_KEY,
+ HADOOP_CALLER_CONTEXT_ENABLED_DEFAULT);
+ CallerContext context = getCallerContext();
+ if (isCallerContextEnabled && context != null && context.isContextValid()) {
+ String cc = context.getContext();
+ String clientIp = "";
+ String ipKey = CallerContext.CLIENT_IP_STR +
Review Comment:
Single line.
##########
hadoop-hdfs-project/hadoop-hdfs-rbf/src/test/java/org/apache/hadoop/hdfs/server/federation/router/TestRouterRpc.java:
##########
@@ -2070,6 +2071,23 @@ public void testSetBalancerBandwidth() throws Exception {
}, 100, 60 * 1000);
}
+ @Test
+ public void testRecordRealClientIp() {
+ GenericTestUtils.LogCapturer logger =
+ GenericTestUtils.LogCapturer.captureLogs(Server.LOG);
+ CallerContext.setCurrent(
+ new CallerContext.Builder(
+ "clientContext,clientIp:2.2.2.2,clientPort:1234").build());
+ try {
+ String filePath = "/test/f.log";
+ routerProtocol.getBlockLocations(filePath, 0, 100);
+ } catch (Exception e) {
+ // do nothing
+ } finally {
+ assertTrue(logger.getOutput().contains("2.2.2.2:1234"));
Review Comment:
Can we test with HADOOP_CALLER_CONTEXT_ENABLED_KEY set to true and false?
##########
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Server.java:
##########
@@ -1351,6 +1357,33 @@ private class ResponseParams {
@Override
public String toString() {
+ boolean isCallerContextEnabled = conf.getBoolean(
+ HADOOP_CALLER_CONTEXT_ENABLED_KEY,
+ HADOOP_CALLER_CONTEXT_ENABLED_DEFAULT);
+ CallerContext context = getCallerContext();
+ if (isCallerContextEnabled && context != null && context.isContextValid()) {
+ String cc = context.getContext();
Review Comment:
There is a lot of parsing logic here. It would be better to extract it and make it more general and add specific tests.
Issue Time Tracking
-------------------
Worklog Id: (was: 796416)
Time Spent: 20m (was: 10m)
> RBF: Record the real client IP carried by the Router in the NameNode log
> ------------------------------------------------------------------------
>
> Key: HDFS-16700
> URL: https://issues.apache.org/jira/browse/HDFS-16700
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode, rbf
> Affects Versions: 3.3.3
> Reporter: JiangHua Zhu
> Assignee: JiangHua Zhu
> Priority: Major
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Here are some logs recorded by the NameNode when using RBF:
> {code:java}
> 2022-07-28 19:31:07,126 INFO ipc.Server: IPC Server handler 8 on default port 8020, call Call#127 Retry#0 org.apache.hadoop.hdfs.protocol.ClientProtocol.getFileInfo from 172.10.100.67:58001
> {code}
> The ip information here is still the router. If the real client ip is recorded, it will more clearly express where the request comes from.
> E.g:
> {code:java}
> 2022-07-29 19:31:07,126 INFO ipc.Server: IPC Server handler 8 on default port 8020, call Call#127 Retry#0 org.apache.hadoop.hdfs.protocol.ClientProtocol.getFileInfo from 172.10.100.67:58001, client=172.111.65.123:43232
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org