You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "cxzl25 (via GitHub)" <gi...@apache.org> on 2026/10/15 10:02:50 UTC

[PR] HDFS-17996. Make the regular file check in DirectoryScanner configurable [hadoop]

cxzl25 opened a new pull request, #8825:
URL: https://github.com/apache/hadoop/pull/8825

   ### Description of PR
   
   HDFS-16316 added a check to DirectoryScanner#scan that verifies, for every
   block, that both the block file and the meta file are regular files
   (FileUtil#isRegularFile with NOFOLLOW_LINKS). The check runs during
   reconciliation, after the reports of all volumes have been compiled, merged
   and sorted by block id. It therefore issues two extra lstat calls per block
   in block id order, which interleaves all volumes and sub-directories and has
   no locality at all.
   
   
   ### How was this patch tested?
   
   
   ### For code changes:
   
   - [ ] Does the title of this PR start with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
   - [ ] Object storage: Have the integration tests been executed and the endpoint
         declared according to the connector-specific documentation? *Note: Automated CI
         testing doesn't cover all cases so manual testing with cloud storage is still
         required.*
   - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
   - [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files?
   
   ### AI Tooling
   
   If an AI tool was used:
   
   - [ ] The PR includes the phrase "Contains content generated by <tool>"
         where <tool> is the name of the AI tool used.
   - [ ] My use of AI contributions follows the ASF legal policy
         https://www.apache.org/legal/generative-tooling.html
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Re: [PR] HDFS-17996. Make the regular file check in DirectoryScanner configurable [hadoop]

Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #8825:
URL: https://github.com/apache/hadoop/pull/8825#issuecomment-6196510103

   :confetti_ball: **+1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime |  Logfile | Comment |
   |:----:|----------:|--------:|:--------:|:-------:|
   | +0 :ok: |  reexec  |   0m 35s |  |  Docker mode activated.  |
   |||| _ Prechecks _ |
   | +1 :green_heart: |  dupname  |   0m  0s |  |  No case conflicting files found.  |
   | +0 :ok: |  codespell  |   0m  0s |  |  codespell was not available.  |
   | +0 :ok: |  detsecrets  |   0m  0s |  |  detect-secrets was not available.  |
   | +0 :ok: |  xmllint  |   0m  0s |  |  xmllint was not available.  |
   | +1 :green_heart: |  @author  |   0m  0s |  |  The patch does not contain any @author tags.  |
   | +1 :green_heart: |  test4tests  |   0m  0s |  |  The patch appears to include 1 new or modified test files.  |
   |||| _ trunk Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |  47m 42s |  |  trunk passed  |
   | +1 :green_heart: |  compile  |   1m 50s |  |  trunk passed with JDK Ubuntu-21.0.12.1+1-1-24.04.4-Ubuntu  |
   | +1 :green_heart: |  compile  |   1m 50s |  |  trunk passed with JDK Ubuntu-17.0.20.1+1-1-24.04-Ubuntu  |
   | +1 :green_heart: |  checkstyle  |   1m 51s |  |  trunk passed  |
   | +1 :green_heart: |  mvnsite  |   2m  1s |  |  trunk passed  |
   | +1 :green_heart: |  javadoc  |   1m 32s |  |  trunk passed with JDK Ubuntu-21.0.12.1+1-1-24.04.4-Ubuntu  |
   | +1 :green_heart: |  javadoc  |   1m 29s |  |  trunk passed with JDK Ubuntu-17.0.20.1+1-1-24.04-Ubuntu  |
   | +1 :green_heart: |  spotbugs  |   4m 35s |  |  trunk passed  |
   | +1 :green_heart: |  shadedclient  |  35m 18s |  |  branch has no errors when building and testing our client artifacts.  |
   |||| _ Patch Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |   1m 23s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |   1m 18s |  |  the patch passed with JDK Ubuntu-21.0.12.1+1-1-24.04.4-Ubuntu  |
   | +1 :green_heart: |  javac  |   1m 18s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |   1m 21s |  |  the patch passed with JDK Ubuntu-17.0.20.1+1-1-24.04-Ubuntu  |
   | +1 :green_heart: |  javac  |   1m 21s |  |  the patch passed  |
   | +1 :green_heart: |  blanks  |   0m  0s |  |  The patch has no blanks issues.  |
   | +1 :green_heart: |  checkstyle  |   1m 16s |  |  the patch passed  |
   | +1 :green_heart: |  mvnsite  |   1m 27s |  |  the patch passed  |
   | +1 :green_heart: |  javadoc  |   1m  1s |  |  the patch passed with JDK Ubuntu-21.0.12.1+1-1-24.04.4-Ubuntu  |
   | +1 :green_heart: |  javadoc  |   1m  2s |  |  the patch passed with JDK Ubuntu-17.0.20.1+1-1-24.04-Ubuntu  |
   | +1 :green_heart: |  spotbugs  |   3m 52s |  |  the patch passed  |
   | +1 :green_heart: |  shadedclient  |  31m 12s |  |  patch has no errors when building and testing our client artifacts.  |
   |||| _ Other Tests _ |
   | +1 :green_heart: |  unit  | 218m 14s |  |  hadoop-hdfs in the patch passed.  |
   | +1 :green_heart: |  asflicense  |   0m 40s |  |  The patch does not generate ASF License warnings.  |
   |  |   | 359m 37s |  |  |
   
   
   | Subsystem | Report/Notes |
   |----------:|:-------------|
   | Docker | ClientAPI=1.56 ServerAPI=1.56 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8825/1/artifact/out/Dockerfile |
   | GITHUB PR | https://github.com/apache/hadoop/pull/8825 |
   | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint |
   | uname | Linux 1f999d0a1848 5.15.0-190-generic #200-Ubuntu SMP Fri Aug 7 15:06:04 UTC 2026 x86_64 x86_64 x86_64 GNU/Linux |
   | Build tool | maven |
   | Personality | dev-support/bin/hadoop.sh |
   | git revision | trunk / 2f67be228875316682b38a54aa8d4820987816b1 |
   | Default Java | Ubuntu-17.0.20.1+1-1-24.04-Ubuntu |
   | Multi-JDK versions | /usr/lib/jvm/java-21-openjdk-amd64:Ubuntu-21.0.12.1+1-1-24.04.4-Ubuntu /usr/lib/jvm/java-17-openjdk-amd64:Ubuntu-17.0.20.1+1-1-24.04-Ubuntu |
   |  Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8825/1/testReport/ |
   | Max. process+thread count | 3334 (vs. ulimit of 10000) |
   | modules | C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs |
   | Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8825/1/console |
   | versions | git=2.43.0 maven=3.9.15 spotbugs=4.9.7 |
   | Powered by | Apache Yetus 0.14.1 https://yetus.apache.org |
   
   
   This message was automatically generated.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Re: [PR] HDFS-17996. Make the regular file check in DirectoryScanner configurable [hadoop]

Posted by "slfan1989 (via GitHub) via common-issues" <co...@hadoop.apache.org>.
slfan1989 commented on code in PR #8825:
URL: https://github.com/apache/hadoop/pull/8825#discussion_r4572815470


##########
hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml:
##########
@@ -946,6 +946,20 @@
   </description>
 </property>
 
+<property>
+  <name>dfs.datanode.directoryscan.regular.file.check</name>
+  <value>true</value>
+  <description>Whether the directory scanner verifies that the block file and
+  the meta file of every block are regular files (not directories, symlinks
+  or special files). The check issues two extra lstat calls per block after
+  all volumes have been listed, in block id order across all volumes. On
+  DataNodes whose number of files exceeds the kernel inode cache this second
+  pass is a random read on every volume and can take hours; set this to false
+  to skip it. Blocks that differ in other ways are still checked for regular

Review Comment:
   Thanks for keeping the check enabled by default. 
   
   Could we narrow the last sentence? When a replica exists on disk but is missing from `volumeMap`, `checkAndUpdate()` adds it and returns before checking the file type. 
   
   I verified that this path can add a symlinked replica with the option disabled. 
   
   Please clarify that only some reconciliation paths still check for regular files.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org