[jira] [Updated] (HDFS-5223) Allow edit log/fsimage format changes without changing layout version

["Colin Patrick McCabe (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HDFS-5223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Colin Patrick McCabe updated HDFS-5223:
---------------------------------------

    Attachment: HDFS-5223.004.patch

Here is a first version implementing the feature flag approach.  It's still a bit rough and needs some unit tests.  The basic idea is to place a LayoutFlags object inside the FSNamesystem, and supply the flags object whenever we need to roll the edit logs, etc.  We also store these flags in the FSImage header and FSEditLog header.  The flags are supplied during edit log opcode serialization and deserialization.

After installing new software, and ensuring that it's stable, the user performs an allowLayoutFlag RPC to finalize the upgrade.  Once the new layout flag has been added to the FSImage and edit log, downgrade is not possible.  We may want to add a downgrade feature flag tool later, possibly as part of NameNode recovery mode.  Since the old version cannot support the new features, by definition this will involve data loss.

I didn't implement this for BKJM, so it's frozen at LayoutFlags.NONE.  We can address this later in a follow-on change.

> Allow edit log/fsimage format changes without changing layout version
> ---------------------------------------------------------------------
>
>                 Key: HDFS-5223
>                 URL: https://issues.apache.org/jira/browse/HDFS-5223
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: namenode
>    Affects Versions: 2.1.1-beta
>            Reporter: Aaron T. Myers
>            Assignee: Colin Patrick McCabe
>         Attachments: HDFS-5223.004.patch
>
>
> Currently all HDFS on-disk formats are version by the single layout version. This means that even for changes which might be backward compatible, like the addition of a new edit log op code, we must go through the full `namenode -upgrade' process which requires coordination with DNs, etc. HDFS should support a lighter weight alternative.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)