You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "ASF GitHub Bot (Jira)" <ji...@apache.org> on 2023/09/22 20:26:00 UTC
[jira] [Commented] (HDFS-17181) WebHDFS not considering whether a DN is good when called from outside the cluster
[ https://issues.apache.org/jira/browse/HDFS-17181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17768155#comment-17768155 ]
ASF GitHub Bot commented on HDFS-17181:
---------------------------------------
lfrancke opened a new pull request, #6108:
URL: https://github.com/apache/hadoop/pull/6108
<!--
Thanks for sending a pull request!
1. If this is your first time, please read our contributor guidelines: https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute
2. Make sure your PR title starts with JIRA issue id, e.g., 'HADOOP-17799. Your PR title ...'.
-->
### Description of PR
https://issues.apache.org/jira/browse/HDFS-17181
When calling WebHDFS to create a file it will happily redirect to nodes that are in maintenance.
The reason is in the `chooseDatanode` method in `NamenodeWebHdfsMethods` where it will only call the `BlockPlacementPolicy` (which considers all these edge cases) in case the `remoteAddr` (i.e. the address making the request to WebHDFS) is also running a DataNode.
In all other cases it just refers to `NetworkTopology#chooseRandom` which does not consider any of these circumstances (e.g. load, maintenance).
I don't understand the reason to not just always refer to the placement policy so this PR fixes that
### How was this patch tested?
We tested it in production and ran the unit tests (had some issues with those so let's see what the builds here say)
### For code changes:
- [x] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
> WebHDFS not considering whether a DN is good when called from outside the cluster
> ---------------------------------------------------------------------------------
>
> Key: HDFS-17181
> URL: https://issues.apache.org/jira/browse/HDFS-17181
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode, webhdfs
> Affects Versions: 3.3.6
> Reporter: Lars Francke
> Priority: Major
> Attachments: Test_fix_for_HDFS-171811.patch
>
>
> When calling WebHDFS to create a file (I'm sure the same problem occurs for other actions e.g. OPEN but I haven't checked all of them yet) it will happily redirect to nodes that are in maintenance.
> The reason is in the [{{chooseDatanode}}|https://github.com/apache/hadoop/blob/rel/release-3.3.6/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/web/resources/NamenodeWebHdfsMethods.java#L307C9-L315] method in {{NamenodeWebHdfsMethods}} where it will only call the {{BlockPlacementPolicy}} (which considers all these edge cases) in case the {{remoteAddr}} (i.e. the address making the request to WebHDFS) is also running a DataNode.
>
> In all other cases it just refers to [{{NetworkTopology#chooseRandom}}|https://github.com/apache/hadoop/blob/rel/release-3.3.6/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/web/resources/NamenodeWebHdfsMethods.java#L342-L343] which does not consider any of these circumstances (e.g. load, maintenance).
> I don't understand the reason to not just always refer to the placement policy and we're currently testing a patch to do just that.
> I have attached a draft patch for now.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org