You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "dhruba borthakur (JIRA)" <ji...@apache.org> on 2006/12/08 06:23:21 UTC

[jira] Created: (HADOOP-807) splitting DFS version number

splitting DFS version number
----------------------------

                 Key: HADOOP-807
                 URL: http://issues.apache.org/jira/browse/HADOOP-807
             Project: Hadoop
          Issue Type: Bug
          Components: dfs
            Reporter: dhruba borthakur
         Assigned To: dhruba borthakur


In the current code, there is a enum named DFS_CURRENT_VERSION. It is being used for three different purposes:

1. It is stored in the fsimage and represents the format of the fsimage.
2. It is stored in the "storage" file by the Datanodes. Thus, it represents the format of data on Datanodes.
3. It is used as a version of the Datanode Protocol.

The current implementation makes it difficult to change the fsimage format without affecting the Datanode Protocol. My proposal is to introduce three new constants, one for each of the functionality described above.

In the current code, we have:
    public static final int DFS_CURRENT_VERSION = -3;

Instead, we can have:
    public static final int DFS_FSIMAGE_CURRENT_VERSION = -3;
    public static final int DFS_DATANODEPROTOCOL_CURRENT_VERSION = -3;
    public static final int DFS_DATASTORAGE_CURRENT_VERSION = -3;

The first one is associated with the format of the fsimage/edits; the second one is associated with the communication procotol between the namenode and the datanode; the third one is associated with the format of the datastore on the datanodes (e.g. directory fanout).

Please comment.

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Re: [jira] Created: (HADOOP-807) splitting DFS version number

Posted by Nigel Daley <nd...@yahoo-inc.com>.
My only concern with this is the potential burden of documenting and  
testing the interoperability of these independently changing versions  
-- especially given that we don't currently have any tests in this  
area.  In my experience, generating compatibility matrices can be  
tricky business.


On Dec 7, 2006, at 9:23 PM, dhruba borthakur (JIRA) wrote:

> splitting DFS version number
> ----------------------------
>
>                  Key: HADOOP-807
>                  URL: http://issues.apache.org/jira/browse/HADOOP-807
>              Project: Hadoop
>           Issue Type: Bug
>           Components: dfs
>             Reporter: dhruba borthakur
>          Assigned To: dhruba borthakur
>
>
> In the current code, there is a enum named DFS_CURRENT_VERSION. It  
> is being used for three different purposes:
>
> 1. It is stored in the fsimage and represents the format of the  
> fsimage.
> 2. It is stored in the "storage" file by the Datanodes. Thus, it  
> represents the format of data on Datanodes.
> 3. It is used as a version of the Datanode Protocol.
>
> The current implementation makes it difficult to change the fsimage  
> format without affecting the Datanode Protocol. My proposal is to  
> introduce three new constants, one for each of the functionality  
> described above.
>
> In the current code, we have:
>     public static final int DFS_CURRENT_VERSION = -3;
>
> Instead, we can have:
>     public static final int DFS_FSIMAGE_CURRENT_VERSION = -3;
>     public static final int DFS_DATANODEPROTOCOL_CURRENT_VERSION = -3;
>     public static final int DFS_DATASTORAGE_CURRENT_VERSION = -3;
>
> The first one is associated with the format of the fsimage/edits;  
> the second one is associated with the communication procotol  
> between the namenode and the datanode; the third one is associated  
> with the format of the datastore on the datanodes (e.g. directory  
> fanout).
>
> Please comment.
>
> -- 
> This message is automatically generated by JIRA.
> -
> If you think it was sent incorrectly contact one of the  
> administrators: http://issues.apache.org/jira/secure/ 
> Administrators.jspa
> -
> For more information on JIRA, see: http://www.atlassian.com/ 
> software/jira
>
>


[jira] Commented: (HADOOP-807) splitting DFS version number

Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
    [ http://issues.apache.org/jira/browse/HADOOP-807?page=comments#action_12457015 ] 
            
dhruba borthakur commented on HADOOP-807:
-----------------------------------------

I think the fsimage version and the datanode-storage-version are unrelated to one another. Thus a compatibility matrix does not arise.

The namenode need not care what format the datanode is storing its data. The namenode cares about the fsimage version and the Datanode protocol version. Similarly, the datanode cares about the datanode-storage-version and the Datanode protocol version. The compatibility question arises *only* if the DatanodeProtocol changes. 

But I agree with you that if we say that all cluster nodes have to be upgraded at the same time, then we can live with a single data format version for both namenodes and datanodes.

> splitting DFS version number
> ----------------------------
>
>                 Key: HADOOP-807
>                 URL: http://issues.apache.org/jira/browse/HADOOP-807
>             Project: Hadoop
>          Issue Type: Bug
>          Components: dfs
>            Reporter: dhruba borthakur
>         Assigned To: dhruba borthakur
>
> In the current code, there is a enum named DFS_CURRENT_VERSION. It is being used for three different purposes:
> 1. It is stored in the fsimage and represents the format of the fsimage.
> 2. It is stored in the "storage" file by the Datanodes. Thus, it represents the format of data on Datanodes.
> 3. It is used as a version of the Datanode Protocol.
> The current implementation makes it difficult to change the fsimage format without affecting the Datanode Protocol. My proposal is to introduce three new constants, one for each of the functionality described above.
> In the current code, we have:
>     public static final int DFS_CURRENT_VERSION = -3;
> Instead, we can have:
>     public static final int DFS_FSIMAGE_CURRENT_VERSION = -3;
>     public static final int DFS_DATANODEPROTOCOL_CURRENT_VERSION = -3;
>     public static final int DFS_DATASTORAGE_CURRENT_VERSION = -3;
> The first one is associated with the format of the fsimage/edits; the second one is associated with the communication procotol between the namenode and the datanode; the third one is associated with the format of the datastore on the datanodes (e.g. directory fanout).
> Please comment.

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] Commented: (HADOOP-807) splitting DFS version number

Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
    [ http://issues.apache.org/jira/browse/HADOOP-807?page=comments#action_12456950 ] 
            
Raghu Angadi commented on HADOOP-807:
-------------------------------------


Should we have another VERSION for namenode? It has files other than fsimage ( new VERSION file, edits file etc).

Also DFS_DATA_VERSION is only used by Datanode, then it should declared in a Datanode class or name should imply it.


> splitting DFS version number
> ----------------------------
>
>                 Key: HADOOP-807
>                 URL: http://issues.apache.org/jira/browse/HADOOP-807
>             Project: Hadoop
>          Issue Type: Bug
>          Components: dfs
>            Reporter: dhruba borthakur
>         Assigned To: dhruba borthakur
>
> In the current code, there is a enum named DFS_CURRENT_VERSION. It is being used for three different purposes:
> 1. It is stored in the fsimage and represents the format of the fsimage.
> 2. It is stored in the "storage" file by the Datanodes. Thus, it represents the format of data on Datanodes.
> 3. It is used as a version of the Datanode Protocol.
> The current implementation makes it difficult to change the fsimage format without affecting the Datanode Protocol. My proposal is to introduce three new constants, one for each of the functionality described above.
> In the current code, we have:
>     public static final int DFS_CURRENT_VERSION = -3;
> Instead, we can have:
>     public static final int DFS_FSIMAGE_CURRENT_VERSION = -3;
>     public static final int DFS_DATANODEPROTOCOL_CURRENT_VERSION = -3;
>     public static final int DFS_DATASTORAGE_CURRENT_VERSION = -3;
> The first one is associated with the format of the fsimage/edits; the second one is associated with the communication procotol between the namenode and the datanode; the third one is associated with the format of the datastore on the datanodes (e.g. directory fanout).
> Please comment.

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] Commented: (HADOOP-807) splitting DFS version number

Posted by "Konstantin Shvachko (JIRA)" <ji...@apache.org>.
    [ http://issues.apache.org/jira/browse/HADOOP-807?page=comments#action_12457011 ] 
            
Konstantin Shvachko commented on HADOOP-807:
--------------------------------------------

1. & 2. I think these two are closely related and we should keep the same version for the data layout
for fsimage and data-nodes. If we split them then we will have to support a table of compatible versions
or something similar.
Currently we have simple criteria: different versions are incompatible.
I understand that some changes on the namenode do not necessarily make incompatible data-nodes
running old versions and vice versa. But sometimes they do, and that is what DFS_CURRENT_VERSION
is used for.

3. Yes there is a different version for Datanode protocol, which is DatanodeProtocol.versionID.

I don't think we should split versions at this point, until we define our versioning approach in general.
I agree, we should change the name of the constant.

> splitting DFS version number
> ----------------------------
>
>                 Key: HADOOP-807
>                 URL: http://issues.apache.org/jira/browse/HADOOP-807
>             Project: Hadoop
>          Issue Type: Bug
>          Components: dfs
>            Reporter: dhruba borthakur
>         Assigned To: dhruba borthakur
>
> In the current code, there is a enum named DFS_CURRENT_VERSION. It is being used for three different purposes:
> 1. It is stored in the fsimage and represents the format of the fsimage.
> 2. It is stored in the "storage" file by the Datanodes. Thus, it represents the format of data on Datanodes.
> 3. It is used as a version of the Datanode Protocol.
> The current implementation makes it difficult to change the fsimage format without affecting the Datanode Protocol. My proposal is to introduce three new constants, one for each of the functionality described above.
> In the current code, we have:
>     public static final int DFS_CURRENT_VERSION = -3;
> Instead, we can have:
>     public static final int DFS_FSIMAGE_CURRENT_VERSION = -3;
>     public static final int DFS_DATANODEPROTOCOL_CURRENT_VERSION = -3;
>     public static final int DFS_DATASTORAGE_CURRENT_VERSION = -3;
> The first one is associated with the format of the fsimage/edits; the second one is associated with the communication procotol between the namenode and the datanode; the third one is associated with the format of the datastore on the datanodes (e.g. directory fanout).
> Please comment.

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] Commented: (HADOOP-807) splitting DFS version number

Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
    [ http://issues.apache.org/jira/browse/HADOOP-807?page=comments#action_12456986 ] 
            
dhruba borthakur commented on HADOOP-807:
-----------------------------------------

Nigel's comment (for the record):

"My only concern with this is the potential burden of documenting and testing the interoperability of these independently changing versions
-- especially given that we don't currently have any tests in this area.  In my experience, generating compatibility matrices can be tricky business."

I agree with the above comment. Testing an old datanode with a newer datanode can be burdensome. But the only other alternative is to require that all nodes in the cluster are upgraded at the same time.

> splitting DFS version number
> ----------------------------
>
>                 Key: HADOOP-807
>                 URL: http://issues.apache.org/jira/browse/HADOOP-807
>             Project: Hadoop
>          Issue Type: Bug
>          Components: dfs
>            Reporter: dhruba borthakur
>         Assigned To: dhruba borthakur
>
> In the current code, there is a enum named DFS_CURRENT_VERSION. It is being used for three different purposes:
> 1. It is stored in the fsimage and represents the format of the fsimage.
> 2. It is stored in the "storage" file by the Datanodes. Thus, it represents the format of data on Datanodes.
> 3. It is used as a version of the Datanode Protocol.
> The current implementation makes it difficult to change the fsimage format without affecting the Datanode Protocol. My proposal is to introduce three new constants, one for each of the functionality described above.
> In the current code, we have:
>     public static final int DFS_CURRENT_VERSION = -3;
> Instead, we can have:
>     public static final int DFS_FSIMAGE_CURRENT_VERSION = -3;
>     public static final int DFS_DATANODEPROTOCOL_CURRENT_VERSION = -3;
>     public static final int DFS_DATASTORAGE_CURRENT_VERSION = -3;
> The first one is associated with the format of the fsimage/edits; the second one is associated with the communication procotol between the namenode and the datanode; the third one is associated with the format of the datastore on the datanodes (e.g. directory fanout).
> Please comment.

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] Commented: (HADOOP-807) splitting DFS version number

Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
    [ http://issues.apache.org/jira/browse/HADOOP-807?page=comments#action_12456699 ] 
            
dhruba borthakur commented on HADOOP-807:
-----------------------------------------

Actually, a clarification about the comment above. The DFS_CURRENT_VERSION is used by the datanode to register itself with the namenode (this is seperate from the Datanode Protocol version).  This means that if we change the format of the fsimage and install a new namenode on the cluster, it cannot talk with older datanodes.  Maybe this is acceptable for now.


> splitting DFS version number
> ----------------------------
>
>                 Key: HADOOP-807
>                 URL: http://issues.apache.org/jira/browse/HADOOP-807
>             Project: Hadoop
>          Issue Type: Bug
>          Components: dfs
>            Reporter: dhruba borthakur
>         Assigned To: dhruba borthakur
>
> In the current code, there is a enum named DFS_CURRENT_VERSION. It is being used for three different purposes:
> 1. It is stored in the fsimage and represents the format of the fsimage.
> 2. It is stored in the "storage" file by the Datanodes. Thus, it represents the format of data on Datanodes.
> 3. It is used as a version of the Datanode Protocol.
> The current implementation makes it difficult to change the fsimage format without affecting the Datanode Protocol. My proposal is to introduce three new constants, one for each of the functionality described above.
> In the current code, we have:
>     public static final int DFS_CURRENT_VERSION = -3;
> Instead, we can have:
>     public static final int DFS_FSIMAGE_CURRENT_VERSION = -3;
>     public static final int DFS_DATANODEPROTOCOL_CURRENT_VERSION = -3;
>     public static final int DFS_DATASTORAGE_CURRENT_VERSION = -3;
> The first one is associated with the format of the fsimage/edits; the second one is associated with the communication procotol between the namenode and the datanode; the third one is associated with the format of the datastore on the datanodes (e.g. directory fanout).
> Please comment.

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] Commented: (HADOOP-807) splitting DFS version number

Posted by "Konstantin Shvachko (JIRA)" <ji...@apache.org>.
    [ http://issues.apache.org/jira/browse/HADOOP-807?page=comments#action_12457042 ] 
            
Konstantin Shvachko commented on HADOOP-807:
--------------------------------------------

Unified storage version ensures that all system components are in sync.

We wanted all system components to run the same software version.
The reason is that we have seen cases (before versions where introduced)
when some data-nodes where not shutdown properly and caused different kinds of problems.
Using a unified storage version somewhat relaxes the same software requirement.
If we want to further relax this requirement we need a clearly defined versioning story.
Which might not be trivial. There are different approaches, here are just some

http://www.cs.cmu.edu/~srini/Papers/publications/dead/usenix05-distapp/usenix05.pdf
http://publications.csail.mit.edu/tmp/MIT-CSAIL-TR-2005-078.pdf



> splitting DFS version number
> ----------------------------
>
>                 Key: HADOOP-807
>                 URL: http://issues.apache.org/jira/browse/HADOOP-807
>             Project: Hadoop
>          Issue Type: Bug
>          Components: dfs
>            Reporter: dhruba borthakur
>         Assigned To: dhruba borthakur
>
> In the current code, there is a enum named DFS_CURRENT_VERSION. It is being used for three different purposes:
> 1. It is stored in the fsimage and represents the format of the fsimage.
> 2. It is stored in the "storage" file by the Datanodes. Thus, it represents the format of data on Datanodes.
> 3. It is used as a version of the Datanode Protocol.
> The current implementation makes it difficult to change the fsimage format without affecting the Datanode Protocol. My proposal is to introduce three new constants, one for each of the functionality described above.
> In the current code, we have:
>     public static final int DFS_CURRENT_VERSION = -3;
> Instead, we can have:
>     public static final int DFS_FSIMAGE_CURRENT_VERSION = -3;
>     public static final int DFS_DATANODEPROTOCOL_CURRENT_VERSION = -3;
>     public static final int DFS_DATASTORAGE_CURRENT_VERSION = -3;
> The first one is associated with the format of the fsimage/edits; the second one is associated with the communication procotol between the namenode and the datanode; the third one is associated with the format of the datastore on the datanodes (e.g. directory fanout).
> Please comment.

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] Resolved: (HADOOP-807) splitting DFS version number

Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HADOOP-807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

dhruba borthakur resolved HADOOP-807.
-------------------------------------

    Resolution: Won't Fix

I propose that we resolve this because because a more-exhaustive versioning scheme is part of hadoop-702.

> splitting DFS version number
> ----------------------------
>
>                 Key: HADOOP-807
>                 URL: https://issues.apache.org/jira/browse/HADOOP-807
>             Project: Hadoop
>          Issue Type: Bug
>          Components: dfs
>            Reporter: dhruba borthakur
>         Assigned To: dhruba borthakur
>
> In the current code, there is a enum named DFS_CURRENT_VERSION. It is being used for three different purposes:
> 1. It is stored in the fsimage and represents the format of the fsimage.
> 2. It is stored in the "storage" file by the Datanodes. Thus, it represents the format of data on Datanodes.
> 3. It is used as a version of the Datanode Protocol.
> The current implementation makes it difficult to change the fsimage format without affecting the Datanode Protocol. My proposal is to introduce three new constants, one for each of the functionality described above.
> In the current code, we have:
>     public static final int DFS_CURRENT_VERSION = -3;
> Instead, we can have:
>     public static final int DFS_FSIMAGE_CURRENT_VERSION = -3;
>     public static final int DFS_DATANODEPROTOCOL_CURRENT_VERSION = -3;
>     public static final int DFS_DATASTORAGE_CURRENT_VERSION = -3;
> The first one is associated with the format of the fsimage/edits; the second one is associated with the communication procotol between the namenode and the datanode; the third one is associated with the format of the datastore on the datanodes (e.g. directory fanout).
> Please comment.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HADOOP-807) splitting DFS version number

Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
    [ http://issues.apache.org/jira/browse/HADOOP-807?page=comments#action_12456918 ] 
            
Doug Cutting commented on HADOOP-807:
-------------------------------------

This sounds reasonable to me.  Those names are rather long for my taste.  I'd probably use something like:
   DFS_IMAGE_VERSION
   DFS_PROTOCOL_VERSION
   DFS_DATA_VERSION
The DFS_ is probably even redundant in every context where these are used, since they're already enclosed in a DFS-specific class.


> splitting DFS version number
> ----------------------------
>
>                 Key: HADOOP-807
>                 URL: http://issues.apache.org/jira/browse/HADOOP-807
>             Project: Hadoop
>          Issue Type: Bug
>          Components: dfs
>            Reporter: dhruba borthakur
>         Assigned To: dhruba borthakur
>
> In the current code, there is a enum named DFS_CURRENT_VERSION. It is being used for three different purposes:
> 1. It is stored in the fsimage and represents the format of the fsimage.
> 2. It is stored in the "storage" file by the Datanodes. Thus, it represents the format of data on Datanodes.
> 3. It is used as a version of the Datanode Protocol.
> The current implementation makes it difficult to change the fsimage format without affecting the Datanode Protocol. My proposal is to introduce three new constants, one for each of the functionality described above.
> In the current code, we have:
>     public static final int DFS_CURRENT_VERSION = -3;
> Instead, we can have:
>     public static final int DFS_FSIMAGE_CURRENT_VERSION = -3;
>     public static final int DFS_DATANODEPROTOCOL_CURRENT_VERSION = -3;
>     public static final int DFS_DATASTORAGE_CURRENT_VERSION = -3;
> The first one is associated with the format of the fsimage/edits; the second one is associated with the communication procotol between the namenode and the datanode; the third one is associated with the format of the datastore on the datanodes (e.g. directory fanout).
> Please comment.

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira