You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "zhangwei (JIRA)" <ji...@apache.org> on 2009/02/06 03:07:59 UTC

[jira] Created: (HADOOP-5181) Add md5sum facility in dfsshell

Add md5sum facility in dfsshell
-------------------------------

                 Key: HADOOP-5181
                 URL: https://issues.apache.org/jira/browse/HADOOP-5181
             Project: Hadoop Core
          Issue Type: New Feature
          Components: dfs
            Reporter: zhangwei


I think it would be usefull to add md5sum (or anyone else) to dfsshell ,and the facility can verify the file on hdfs.It can confirm the file is integrity after copyFromLocal or copyToLocal.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HADOOP-5181) Add md5sum facility in dfsshell

Posted by "zhangwei (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-5181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12671061#action_12671061 ] 

zhangwei commented on HADOOP-5181:
----------------------------------

Maybe I don't say clearly. 
It's not relevant with the crc verify. It's a tool that help peple to see the file's checksum on hdfs.
If people want to get the file's checksum ,he has to only download  it to local and run "md5sum".

In many cases,developer want to check a large set of mapred output files to confirm the program in testing runs correctly,
and they can also differ the older and new version program's output to find out the new program's tiny bug.


> Add md5sum facility in dfsshell
> -------------------------------
>
>                 Key: HADOOP-5181
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5181
>             Project: Hadoop Core
>          Issue Type: New Feature
>          Components: dfs
>            Reporter: zhangwei
>
> I think it would be usefull to add md5sum (or anyone else) to dfsshell ,and the facility can verify the file on hdfs.It can confirm the file is integrity after copyFromLocal or copyToLocal.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HADOOP-5181) Add md5sum facility in dfsshell

Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-5181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12671038#action_12671038 ] 

dhruba borthakur commented on HADOOP-5181:
------------------------------------------

dfsshell uses the dfs client, which in turn, verifies that the crc's match the data of the file that is being fetched. There is a crc for every 512 bytes in the file. So, I am not sure what additional verification is achieved by incorporating md5 in dfsshell.

> Add md5sum facility in dfsshell
> -------------------------------
>
>                 Key: HADOOP-5181
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5181
>             Project: Hadoop Core
>          Issue Type: New Feature
>          Components: dfs
>            Reporter: zhangwei
>
> I think it would be usefull to add md5sum (or anyone else) to dfsshell ,and the facility can verify the file on hdfs.It can confirm the file is integrity after copyFromLocal or copyToLocal.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HADOOP-5181) Add md5sum facility in dfsshell

Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-5181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12671238#action_12671238 ] 

Doug Cutting commented on HADOOP-5181:
--------------------------------------

I think the request is for shell access to:

http://hadoop.apache.org/core/docs/current/api/org/apache/hadoop/fs/FileSystem.html#getFileChecksum(org.apache.hadoop.fs.Path)

This seems reasonable to me.  It provides access to the checksum that's already computed & stored.

We could print the checksum as hex.  Perhaps we could even print it as a column with '-lsll' or somesuch.


> Add md5sum facility in dfsshell
> -------------------------------
>
>                 Key: HADOOP-5181
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5181
>             Project: Hadoop Core
>          Issue Type: New Feature
>          Components: dfs
>            Reporter: zhangwei
>
> I think it would be usefull to add md5sum (or anyone else) to dfsshell ,and the facility can verify the file on hdfs.It can confirm the file is integrity after copyFromLocal or copyToLocal.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.