You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-dev@hadoop.apache.org by "TangLin (Jira)" <ji...@apache.org> on 2026/08/26 02:55:00 UTC
[jira] [Created] (HDFS-17968) When a single block's meta file is missing or inaccessible on a DataNode volume, the entire block pool scan on that volume is terminated for up to 3 weeks (VolumeScanner) or the entire DiskBalancer plan execution is aborted.
TangLin created HDFS-17968:
------------------------------
Summary: When a single block's meta file is missing or inaccessible on a DataNode volume, the entire block pool scan on that volume is terminated for up to 3 weeks (VolumeScanner) or the entire DiskBalancer plan execution is aborted.
Key: HDFS-17968
URL: https://issues.apache.org/jira/browse/HDFS-17968
Project: Hadoop HDFS
Issue Type: Bug
Reporter: TangLin
Problem Summary
Phenomenon
When a single block's meta file is missing or inaccessible on a DataNode volume, the entire block pool scan on that
volume is terminated for up to 3 weeks (VolumeScanner) or the entire DiskBalancer plan execution is aborted.
- VolumeScanner: The BlockIterator is marked as EOF (atEnd=true). No further blocks on that volume are scanned until
scanPeriodMs (default 504 hours) elapses and the iterator is rewound.
- DiskBalancer: The copyBlocks loop exits. The plan must be manually resubmitted.
Root Cause
HDFS-9735 added a findMetaFile() call inside BlockIterator.nextBlock() to populate generationStamp and numBytes on the
returned ExtendedBlock, so that DiskBalancer could directly use the iterator result to call moveBlockAcrossVolumes().
When findMetaFile() throws IOException (meta file missing, directory I/O error, etc.), the catch block in
FsVolumeImpl.BlockIterator.nextBlock() sets state.atEnd = true before re-throwing. This permanently marks the iterator
as exhausted, even though only one block was problematic.
Key Insight
The original HDFS-7430 design did not access meta files during iteration — it only parsed block IDs from filenames.
The findMetaFile dependency was introduced later by HDFS-9735 specifically for DiskBalancer support, and it is the
direct cause of this cascading failure.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org