[jira] [Commented] (HDFS-17844) Flaky test: testViewFsMultipleExportPoint may encounter race condition which leads to namenode ID collisions

["ASF GitHub Bot (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HDFS-17844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18024385#comment-18024385 ] 

ASF GitHub Bot commented on HDFS-17844:
---------------------------------------

Wang-Benjamin opened a new pull request, #8009:
URL: https://github.com/apache/hadoop/pull/8009

   ## Problem
   There could be a race condition in generating namenode ID. What is problematic is that different URIs resolve to the same namenode ID, which leads to data integrity issues and security problems. The logic of the underlying function used to detect such problem is incorrect as it throws an error whenever the namenode exists even if the URI is the same. As a result, `testViewFsMultipleExportPoint()` has a nondeterministic behavior when the test execution order and timing change.
   
   Command line used to identify the nondeterminism of `testViewFsMultipleExportPoint()`:
   
   ```
   mvn -pl hadoop-hdfs-project/hadoop-hdfs-nfs \
       -Dcheckstyle.skip=true -Drat.skip=true \
       edu.illinois:nondex-maven-plugin:2.1.7:nondex \
       -Dtest=org.apache.hadoop.hdfs.nfs.nfs3.TestExportsTable#testViewFsMultipleExportPoint \
       -DnondexRuns=20 |& tee nondex-$(date +%s).log
   ```
   Error example:
   ```
   [ERROR]   TestExportsTable.testViewFsMultipleExportPoint:111 » FileSystem FS:viewfs, Namenode ID collision for path:/hdfs2 nnid:2130740544 uri being added:hdfs://localhost:34111/ existing uri:hdfs://localhost:34111/
   ```
   
   ## Patch
   The proposed fix has changed the collision detection logic in `prepareAddressMap()` so that it will throw errors only when different URIs resolves to the same namenode ID. 
   
   In addition, the namenode ID generation logic in `getNamenodeId()` is modified where a robust hash function combines URI and address strings, so it now reduces the chance of hash collision.
   
   
   




> Flaky test: testViewFsMultipleExportPoint may encounter race condition which leads to namenode ID collisions
> ------------------------------------------------------------------------------------------------------------
>
>                 Key: HDFS-17844
>                 URL: https://issues.apache.org/jira/browse/HDFS-17844
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: namenode, test
>            Reporter: Yakun Wang
>            Priority: Major
>              Labels: patch
>
> There could be a race condition in namenode ID generation, causing ID collisions. However, the underlying collision detection logic is incorrect because it throws an error as long as the namenode ID exists, even if the URI is the same. Same namenode ID with same URI is safe. What needs to be detected is that different URIs resolve to the same namenode ID, which is a security problem.
> Error output:
> `[ERROR] Errors: 
> [ERROR]   TestExportsTable.testViewFsMultipleExportPoint:111 » FileSystem FS:viewfs, Namenode ID collision for path:/hdfs2 nnid:2130740544 uri being added:hdfs://localhost:34111/ existing uri:hdfs://localhost:34111/`
> Functions that are related to collision detection logic: 
> # prepareAddressMap in hadoop/hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/java/org/apache/hadoop/hdfs/nfs/nfs3/DFSClientCache.java
> # getNamenodeId in hadoop/hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/java/org/apache/hadoop/hdfs/nfs/nfs3/Nfs3Utils.java



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org