You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "ASF GitHub Bot (Jira)" <ji...@apache.org> on 2025/09/26 00:22:00 UTC
[jira] [Commented] (HDFS-17524) OIV: add Transformed processor which reconstructs an fsimage from another fsimage file
[ https://issues.apache.org/jira/browse/HDFS-17524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18022939#comment-18022939 ]
ASF GitHub Bot commented on HDFS-17524:
---------------------------------------
github-actions[bot] commented on PR #6820:
URL: https://github.com/apache/hadoop/pull/6820#issuecomment-3336332180
We're closing this stale PR because it has been open for 100 days with no activity. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you feel like this was a mistake, or you would like to continue working on it, please feel free to re-open it and ask for a committer to remove the stale tag and review again.
Thanks all for your contribution.
> OIV: add Transformed processor which reconstructs an fsimage from another fsimage file
> --------------------------------------------------------------------------------------
>
> Key: HDFS-17524
> URL: https://issues.apache.org/jira/browse/HDFS-17524
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: tools
> Affects Versions: 3.2.0, 3.3.4
> Reporter: Xiaobao Wu
> Priority: Major
> Labels: pull-request-available
>
> *Background：*
> The Image file generated by the existing Hadoop 3.3.4 version cannot be forward compatible . In the high version of HDFS, the fsimage file conversion tool is provided to support the generation of forward compatible fsimage file to support the downgrade operation.
> {*}Description{*}：
> Because there are differences in the structure and loading methods of some Sections between high and low versions of fsimage files, especially the StringTable Section. This will make it impossible to downgrade to a lower version of HDFS ( e.g., 3.1.1 ) in higher versions ( e.g., 3.3.4 ), because when the lower version of HDFS loads the fsimage file generated by the higher version of HDFS, there will be an ArrayIndexOutOfBoundsException.
>
> The code differences are as follows：
> {code:java}
> // 3.3.4
> static SerialNumberManager.StringTable loadStringTable(InputStream in)
> throws IOException {
> ··· ···
> SerialNumberManager.StringTable stringTable = SerialNumberManager.newStringTable(s.getNumEntry(), s.getMaskBits());
> for (int i = 0; i < s.getNumEntry(); ++i) {
> FsImageProto.StringTableSection.Entry e = FsImageProto .StringTableSection.Entry.parseDelimitedFrom(in);
> stringTable.put(e.getId(), e.getStr());
> }
> return stringTable;
> }
> // 3.1.1
> static String[] loadStringTable(InputStream in) throws IOException {
> ··· ···
> String[] stringTable = new String[s.getNumEntry() + 1];
> for (int i = 0; i < s.getNumEntry(); ++i) {
> FsImageProto.StringTableSection.Entry e = FsImageProto
> .StringTableSection.Entry.parseDelimitedFrom(in);
> // ArrayIndexOutOfBoundsException is triggered when loading a higher version of the fsimage file.
> stringTable[e.getId()] = e.getStr();
> }
> return stringTable;
> }{code}
> {*}Solution{*}：
> Solution Reference from HDFS-17463
> !http://www.kdocs.cn/api/v3/office/copy/Mm0rd3BzNEx2Y29zaUdsQkczVnRUV2JwR2RvVWNVdk9aT3dRc2czUXRYdit1ekZ4UmN3UWFLN0hwOTZidnJ1L2ZxaW5PaUNHRmU1bGNyS3lRUGZRbE1vR2I4MlQvS0ppOUZxbVRnQ2o2SUNJZGFoeVNzMUFjR2tKTStsTjZpUTFwanpmcTRML0JFTDJHcXV4aGpESVFXS1RTeEkyZk5sb25LOEEyT0lHbDJydVlIZEJ2dXlyYVozM2pkZGdacEtWQnR3SUQ0MXUwV1RINTMyaDluV2FRTWNjS2p5Nm0rZngzbGNGdEd4cFpLdjFpWUtWK2UyMDZhVVFYUWVHZXlwZEQ0c25MWU93NFY0PQ==/attach/object/K3TLVNAYAAQFQ?|width=693!
> From the figure, it can be seen that the Id arrangement of StringTable in the fsimage file has changed from a compact arrangement to a decentralized arrangement, that is, USER, GROUP and XATTR are no longer mixed. The arrangement is divided into different storage areas and arranged separately.
> * With the sub-sections feature introduced in HDFS-14617, Protobuf can support compatible reading.
> * When saving fsimage files in high and low versions, the main difference is the arrangement of Entry（e.g., USER, GROUP, and XATTR ） in StringTable.
> * We will add a conversion tool to convert the Id arrangement of the high version fsimage file StringTable to a compact arrangement, so that the low version can be compatible with this format fsimage file.
>
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org