[jira] [Created] (HDFS-9230) Report space overhead of unfinalized upgrade/rollingUpgrade

["Xiaoyu Yao (JIRA)" <ji...@apache.org>]

Xiaoyu Yao created HDFS-9230:
--------------------------------

             Summary: Report space overhead of unfinalized upgrade/rollingUpgrade
                 Key: HDFS-9230
                 URL: https://issues.apache.org/jira/browse/HDFS-9230
             Project: Hadoop HDFS
          Issue Type: Improvement
          Components: HDFS
            Reporter: Xiaoyu Yao


DataNodes do not delete block files during upgrades to allow rollback. This is often confusing to administrators since they sometimes delete files before finalize upgrade but don't see the DFS used space reduce.

Ideally, HDFS should report the un-finalized upgrade overhead along with its message on NN UI "Upgrade in progress. Not yet finalized." Or, this can be improve with better NN UI message and document that space won't be reclaimed for deletion until upgrade is finalized.

For non-rolling upgrade, it is not easy to track this due to hard link. Say NN initialized upgrade at T1, the block files on DNs that exist before T1 are still under 'current' directory but is just a hard link to 'previous' directory. When those files are deleted after T1 due to deletion, the block file usage on DN won't get deleted until upgrade is finalized. 

So we need to book keeping files created before T1 but deleted after T1 as the un-finalized upgrade overhead here.

For rolling upgrade, it is relative easy to track space overhead as we are not using hard link.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)