[jira] [Commented] (HDFS-17824) DataNode fails to start when dfs.datanode.directoryscan.threads is 0 or negative

["ASF GitHub Bot (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HDFS-17824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18063820#comment-18063820 ] 

ASF GitHub Bot commented on HDFS-17824:
---------------------------------------

deepujain opened a new pull request, #8309:
URL: https://github.com/apache/hadoop/pull/8309

   ### Summary
   
   When `dfs.datanode.directoryscan.threads` is set to 0 or a negative value, the DataNode throws `IllegalArgumentException` from `Executors.newFixedThreadPool(threads)` during DirectoryScanner initialization and fails to start. This change validates the config and uses the default (1) when the value is non-positive, with a warning log, so the DataNode starts and the operator sees a clear message.
   
   ### Change
   
   - **DirectoryScanner**: After reading `dfs.datanode.directoryscan.threads`, if the value is ≤ 0, log a warning (parameter name, invalid value, default) and set threads to `DFS_DATANODE_DIRECTORYSCAN_THREADS_DEFAULT` (1) before creating the thread pool. Matches the existing pattern used for `reconcile.blocks.batch.size` and `reconcile.blocks.batch.interval` in the same class.
   - **TestDirectoryScanner**: New test `testInvalidDirectoryScanThreadsUsesDefault()` — set `directoryscan.threads` to 0, build a minimal cluster, create and start DirectoryScanner; verifies no exception and scanner runs (HDFS-17824).
   
   ### JIRA
   
   Fixes HDFS-17824
   




> DataNode fails to start when dfs.datanode.directoryscan.threads is 0 or negative
> --------------------------------------------------------------------------------
>
>                 Key: HDFS-17824
>                 URL: https://issues.apache.org/jira/browse/HDFS-17824
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: datanode
>    Affects Versions: 2.8.5
>            Reporter: AMC-team
>            Assignee: AMC-team
>            Priority: Major
>         Attachments: HDFS-17824.000.patch
>
>
> When {{dfs.datanode.directoryscan.threads}} is configured to {{0}} or a negative value, the DataNode initialization path constructs the directory scanner with {{{}Executors.newFixedThreadPool(threads){}}}. Because the thread count is non-positive, the JDK throws {{{}java.lang.IllegalArgumentException{}}}, and the DataNode fails to start. The code currently does not validate this configuration before creating the thread pool.
> *Steps to Reproduce*
> 1. Set the properties above in {{{}hdfs-site.xml{}}}.
> 2. Start or restart the DataNode.
> 3. Observe the following stack trace and process exit:
> {code:java}
> java.lang.IllegalArgumentException
>   at java.util.concurrent.ThreadPoolExecutor.<init>(ThreadPoolExecutor.java:1314)
>   at java.util.concurrent.ThreadPoolExecutor.<init>(ThreadPoolExecutor.java:1237)
>   at java.util.concurrent.Executors.newFixedThreadPool(Executors.java:151)
>   at org.apache.hadoop.hdfs.server.datanode.DirectoryScanner.<init>(DirectoryScanner.java:* )
>   at org.apache.hadoop.hdfs.server.datanode.DataNode.initDirectoryScanner(DataNode.java:* )
>   at org.apache.hadoop.hdfs.server.datanode.DataNode.initBlockPool(DataNode.java:1543)
>   ...
>  {code}
> *Actual Result* 
> DataNode startup aborts with IllegalArgumentException during DirectoryScanner initialization; the node never becomes available.
> *Expected Result*
> Non-positive values of {{dfs.datanode.directoryscan.threads}} are handled gracefully—either rejected early with a clear configuration error naming the key and offending value, or automatically clamped to a safe default (≥1) with a {{{}WARN{}}}, rather than crashing the process.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org