You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Adam Muise (JIRA)" <ji...@apache.org> on 2012/09/25 20:40:07 UTC
[jira] [Created] (HDFS-3971) Add a resume feature to the
copyFromLocal and put commands
Adam Muise created HDFS-3971:
--------------------------------
Summary: Add a resume feature to the copyFromLocal and put commands
Key: HDFS-3971
URL: https://issues.apache.org/jira/browse/HDFS-3971
Project: Hadoop HDFS
Issue Type: New Feature
Components: tools
Affects Versions: 2.0.1-alpha
Reporter: Adam Muise
Priority: Minor
Fix For: 2.0.1-alpha
Add a resume feature to the copyFromLocal command. Failures in large transfers result in a great deal of wasted time. For large files, it would be good to be able to continue from the last good block onwards. The file would have to be unavailable to other clients for reads or regular writes until the "resume" process was completed.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3971) Add a resume feature to the
copyFromLocal and put commands
Posted by "Robert Joseph Evans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13463192#comment-13463192 ]
Robert Joseph Evans commented on HDFS-3971:
-------------------------------------------
It almost sounds like you want to turn this into something like rsync. I think it would be much more useful to just add in an rsync command with a simmilar set of features and flags then trying to reinvent it piecemeal. Then it can look at time stamps on the files, and possibly checksums as well, to pick up where it left off on a failure.
> Add a resume feature to the copyFromLocal and put commands
> ----------------------------------------------------------
>
> Key: HDFS-3971
> URL: https://issues.apache.org/jira/browse/HDFS-3971
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: tools
> Affects Versions: 2.0.1-alpha
> Reporter: Adam Muise
> Priority: Minor
> Fix For: 2.0.1-alpha
>
>
> Add a resume feature to the copyFromLocal command. Failures in large transfers result in a great deal of wasted time. For large files, it would be good to be able to continue from the last good block onwards. The file would have to be unavailable to other clients for reads or regular writes until the "resume" process was completed.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3971) Add a resume feature to the
copyFromLocal and put commands
Posted by "Eli Collins (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13464475#comment-13464475 ]
Eli Collins commented on HDFS-3971:
-----------------------------------
Yea, something like rsync or Sqoop for file systems seems more appropriate.
> Add a resume feature to the copyFromLocal and put commands
> ----------------------------------------------------------
>
> Key: HDFS-3971
> URL: https://issues.apache.org/jira/browse/HDFS-3971
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: tools
> Affects Versions: 2.0.1-alpha
> Reporter: Adam Muise
> Priority: Minor
> Fix For: 2.0.1-alpha
>
>
> Add a resume feature to the copyFromLocal command. Failures in large transfers result in a great deal of wasted time. For large files, it would be good to be able to continue from the last good block onwards. The file would have to be unavailable to other clients for reads or regular writes until the "resume" process was completed.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3971) Add a resume feature to the
copyFromLocal and put commands
Posted by "Adam Muise (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13479100#comment-13479100 ]
Adam Muise commented on HDFS-3971:
----------------------------------
Yes, this should probably look like rsync.
No, sqoop does not support this use case.
> Add a resume feature to the copyFromLocal and put commands
> ----------------------------------------------------------
>
> Key: HDFS-3971
> URL: https://issues.apache.org/jira/browse/HDFS-3971
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: tools
> Affects Versions: 2.0.1-alpha
> Reporter: Adam Muise
> Priority: Minor
> Fix For: 2.0.1-alpha
>
>
> Add a resume feature to the copyFromLocal command. Failures in large transfers result in a great deal of wasted time. For large files, it would be good to be able to continue from the last good block onwards. The file would have to be unavailable to other clients for reads or regular writes until the "resume" process was completed.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira