You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Runping Qi (JIRA)" <ji...@apache.org> on 2007/10/13 01:35:50 UTC
[jira] Created: (HADOOP-2048) DISTCP mapper should report progress
more often
DISTCP mapper should report progress more often
-----------------------------------------------
Key: HADOOP-2048
URL: https://issues.apache.org/jira/browse/HADOOP-2048
Project: Hadoop
Issue Type: Bug
Reporter: Runping Qi
When I ran DISTCP to copy files from one dfs to another, I noticed that some mappers got killed due to failing to report status for 606 seconds.
I noticed that the mappers try to make a progress report for every 32MB copied. A better way to ensure progress is to use a time interval since last report.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HADOOP-2048) DISTCP mapper should report progress
more often
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas reassigned HADOOP-2048:
-------------------------------------
Assignee: Chris Douglas
> DISTCP mapper should report progress more often
> -----------------------------------------------
>
> Key: HADOOP-2048
> URL: https://issues.apache.org/jira/browse/HADOOP-2048
> Project: Hadoop
> Issue Type: Bug
> Reporter: Runping Qi
> Assignee: Chris Douglas
>
> When I ran DISTCP to copy files from one dfs to another, I noticed that some mappers got killed due to failing to report status for 606 seconds.
> I noticed that the mappers try to make a progress report for every 32MB copied. A better way to ensure progress is to use a time interval since last report.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2048) DISTCP mapper should report progress
more often
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas updated HADOOP-2048:
----------------------------------
Status: Patch Available (was: Open)
> DISTCP mapper should report progress more often
> -----------------------------------------------
>
> Key: HADOOP-2048
> URL: https://issues.apache.org/jira/browse/HADOOP-2048
> Project: Hadoop
> Issue Type: Bug
> Reporter: Runping Qi
> Assignee: Chris Douglas
> Attachments: 2048-2.patch, 2048.patch
>
>
> When I ran DISTCP to copy files from one dfs to another, I noticed that some mappers got killed due to failing to report status for 606 seconds.
> I noticed that the mappers try to make a progress report for every 32MB copied. A better way to ensure progress is to use a time interval since last report.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2048) DISTCP mapper should report progress
more often
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Owen O'Malley updated HADOOP-2048:
----------------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
I just committed this. Thanks, Chris!
> DISTCP mapper should report progress more often
> -----------------------------------------------
>
> Key: HADOOP-2048
> URL: https://issues.apache.org/jira/browse/HADOOP-2048
> Project: Hadoop
> Issue Type: Bug
> Reporter: Runping Qi
> Assignee: Chris Douglas
> Priority: Blocker
> Fix For: 0.15.0
>
> Attachments: 2048-3.patch, 2048-4.patch, 2048-5.patch
>
>
> When I ran DISTCP to copy files from one dfs to another, I noticed that some mappers got killed due to failing to report status for 606 seconds.
> I noticed that the mappers try to make a progress report for every 32MB copied. A better way to ensure progress is to use a time interval since last report.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2048) DISTCP mapper should report progress
more often
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas updated HADOOP-2048:
----------------------------------
Status: Patch Available (was: Open)
> DISTCP mapper should report progress more often
> -----------------------------------------------
>
> Key: HADOOP-2048
> URL: https://issues.apache.org/jira/browse/HADOOP-2048
> Project: Hadoop
> Issue Type: Bug
> Reporter: Runping Qi
> Assignee: Chris Douglas
> Priority: Blocker
> Fix For: 0.15.0
>
> Attachments: 2048-3.patch, 2048-4.patch
>
>
> When I ran DISTCP to copy files from one dfs to another, I noticed that some mappers got killed due to failing to report status for 606 seconds.
> I noticed that the mappers try to make a progress report for every 32MB copied. A better way to ensure progress is to use a time interval since last report.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2048) DISTCP mapper should report progress
more often
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas updated HADOOP-2048:
----------------------------------
Attachment: 2048.patch
Report status at least every mapred.task.timeout / 8 milliseconds (default 1:15) or after every 10MB copied.
> DISTCP mapper should report progress more often
> -----------------------------------------------
>
> Key: HADOOP-2048
> URL: https://issues.apache.org/jira/browse/HADOOP-2048
> Project: Hadoop
> Issue Type: Bug
> Reporter: Runping Qi
> Assignee: Chris Douglas
> Attachments: 2048.patch
>
>
> When I ran DISTCP to copy files from one dfs to another, I noticed that some mappers got killed due to failing to report status for 606 seconds.
> I noticed that the mappers try to make a progress report for every 32MB copied. A better way to ensure progress is to use a time interval since last report.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2048) DISTCP mapper should report progress
more often
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas updated HADOOP-2048:
----------------------------------
Status: Patch Available (was: Open)
> DISTCP mapper should report progress more often
> -----------------------------------------------
>
> Key: HADOOP-2048
> URL: https://issues.apache.org/jira/browse/HADOOP-2048
> Project: Hadoop
> Issue Type: Bug
> Reporter: Runping Qi
> Assignee: Chris Douglas
> Attachments: 2048.patch
>
>
> When I ran DISTCP to copy files from one dfs to another, I noticed that some mappers got killed due to failing to report status for 606 seconds.
> I noticed that the mappers try to make a progress report for every 32MB copied. A better way to ensure progress is to use a time interval since last report.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2048) DISTCP mapper should report
progress more often
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12537682 ]
Hudson commented on HADOOP-2048:
--------------------------------
Integrated in Hadoop-Nightly #282 (See [http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Nightly/282/])
> DISTCP mapper should report progress more often
> -----------------------------------------------
>
> Key: HADOOP-2048
> URL: https://issues.apache.org/jira/browse/HADOOP-2048
> Project: Hadoop
> Issue Type: Bug
> Reporter: Runping Qi
> Assignee: Chris Douglas
> Priority: Blocker
> Fix For: 0.15.0
>
> Attachments: 2048-3.patch, 2048-4.patch, 2048-5.patch
>
>
> When I ran DISTCP to copy files from one dfs to another, I noticed that some mappers got killed due to failing to report status for 606 seconds.
> I noticed that the mappers try to make a progress report for every 32MB copied. A better way to ensure progress is to use a time interval since last report.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2048) DISTCP mapper should report progress
more often
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas updated HADOOP-2048:
----------------------------------
Status: Patch Available (was: Open)
> DISTCP mapper should report progress more often
> -----------------------------------------------
>
> Key: HADOOP-2048
> URL: https://issues.apache.org/jira/browse/HADOOP-2048
> Project: Hadoop
> Issue Type: Bug
> Reporter: Runping Qi
> Assignee: Chris Douglas
> Attachments: 2048-2.patch, 2048-3.patch, 2048.patch
>
>
> When I ran DISTCP to copy files from one dfs to another, I noticed that some mappers got killed due to failing to report status for 606 seconds.
> I noticed that the mappers try to make a progress report for every 32MB copied. A better way to ensure progress is to use a time interval since last report.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2048) DISTCP mapper should report
progress more often
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12534495 ]
Hadoop QA commented on HADOOP-2048:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12367674/2048.patch
against trunk revision r584336.
@author +1. The patch does not contain any @author tags.
javadoc +1. The javadoc tool did not generate any warning messages.
javac +1. The applied patch does not generate any new compiler warnings.
findbugs -1. The patch appears to introduce 1 new Findbugs warnings.
core tests +1. The patch passed core unit tests.
contrib tests +1. The patch passed contrib unit tests.
Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/941/testReport/
Findbugs warnings: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/941/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/941/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/941/console
This message is automatically generated.
> DISTCP mapper should report progress more often
> -----------------------------------------------
>
> Key: HADOOP-2048
> URL: https://issues.apache.org/jira/browse/HADOOP-2048
> Project: Hadoop
> Issue Type: Bug
> Reporter: Runping Qi
> Assignee: Chris Douglas
> Attachments: 2048.patch
>
>
> When I ran DISTCP to copy files from one dfs to another, I noticed that some mappers got killed due to failing to report status for 606 seconds.
> I noticed that the mappers try to make a progress report for every 32MB copied. A better way to ensure progress is to use a time interval since last report.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2048) DISTCP mapper should report progress
more often
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas updated HADOOP-2048:
----------------------------------
Attachment: 2048-3.patch
Given HADOOP-2049 and HADOOP-2050, changing the way distcp handles failures seems warranted. This deletes partial copies and fails the map in FSCopyFilesMapper::close. Though it may repeat work if 'overwrite' or 'update' are set, it should handle retries properly.
> DISTCP mapper should report progress more often
> -----------------------------------------------
>
> Key: HADOOP-2048
> URL: https://issues.apache.org/jira/browse/HADOOP-2048
> Project: Hadoop
> Issue Type: Bug
> Reporter: Runping Qi
> Assignee: Chris Douglas
> Attachments: 2048-2.patch, 2048-3.patch, 2048.patch
>
>
> When I ran DISTCP to copy files from one dfs to another, I noticed that some mappers got killed due to failing to report status for 606 seconds.
> I noticed that the mappers try to make a progress report for every 32MB copied. A better way to ensure progress is to use a time interval since last report.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2048) DISTCP mapper should report
progress more often
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12534973 ]
Hadoop QA commented on HADOOP-2048:
-----------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12367772/2048-2.patch
against trunk revision r584844.
@author +1. The patch does not contain any @author tags.
javadoc +1. The javadoc tool did not generate any warning messages.
javac +1. The applied patch does not generate any new compiler warnings.
findbugs +1. The patch does not introduce any new Findbugs warnings.
core tests +1. The patch passed core unit tests.
contrib tests +1. The patch passed contrib unit tests.
Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/947/testReport/
Findbugs warnings: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/947/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/947/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/947/console
This message is automatically generated.
> DISTCP mapper should report progress more often
> -----------------------------------------------
>
> Key: HADOOP-2048
> URL: https://issues.apache.org/jira/browse/HADOOP-2048
> Project: Hadoop
> Issue Type: Bug
> Reporter: Runping Qi
> Assignee: Chris Douglas
> Attachments: 2048-2.patch, 2048.patch
>
>
> When I ran DISTCP to copy files from one dfs to another, I noticed that some mappers got killed due to failing to report status for 606 seconds.
> I noticed that the mappers try to make a progress report for every 32MB copied. A better way to ensure progress is to use a time interval since last report.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2048) DISTCP mapper should report
progress more often
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12535044 ]
Hadoop QA commented on HADOOP-2048:
-----------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12367791/2048-3.patch
against trunk revision r584939.
@author +1. The patch does not contain any @author tags.
javadoc +1. The javadoc tool did not generate any warning messages.
javac +1. The applied patch does not generate any new compiler warnings.
findbugs +1. The patch does not introduce any new Findbugs warnings.
core tests +1. The patch passed core unit tests.
contrib tests +1. The patch passed contrib unit tests.
Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/951/testReport/
Findbugs warnings: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/951/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/951/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/951/console
This message is automatically generated.
> DISTCP mapper should report progress more often
> -----------------------------------------------
>
> Key: HADOOP-2048
> URL: https://issues.apache.org/jira/browse/HADOOP-2048
> Project: Hadoop
> Issue Type: Bug
> Reporter: Runping Qi
> Assignee: Chris Douglas
> Attachments: 2048-2.patch, 2048-3.patch, 2048.patch
>
>
> When I ran DISTCP to copy files from one dfs to another, I noticed that some mappers got killed due to failing to report status for 606 seconds.
> I noticed that the mappers try to make a progress report for every 32MB copied. A better way to ensure progress is to use a time interval since last report.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2048) DISTCP mapper should report progress
more often
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas updated HADOOP-2048:
----------------------------------
Attachment: 2048-5.patch
Added counters
> DISTCP mapper should report progress more often
> -----------------------------------------------
>
> Key: HADOOP-2048
> URL: https://issues.apache.org/jira/browse/HADOOP-2048
> Project: Hadoop
> Issue Type: Bug
> Reporter: Runping Qi
> Assignee: Chris Douglas
> Priority: Blocker
> Fix For: 0.15.0
>
> Attachments: 2048-3.patch, 2048-4.patch, 2048-5.patch
>
>
> When I ran DISTCP to copy files from one dfs to another, I noticed that some mappers got killed due to failing to report status for 606 seconds.
> I noticed that the mappers try to make a progress report for every 32MB copied. A better way to ensure progress is to use a time interval since last report.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2048) DISTCP mapper should report progress
more often
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas updated HADOOP-2048:
----------------------------------
Attachment: (was: 2048-2.patch)
> DISTCP mapper should report progress more often
> -----------------------------------------------
>
> Key: HADOOP-2048
> URL: https://issues.apache.org/jira/browse/HADOOP-2048
> Project: Hadoop
> Issue Type: Bug
> Reporter: Runping Qi
> Assignee: Chris Douglas
> Priority: Blocker
> Fix For: 0.15.0
>
> Attachments: 2048-3.patch
>
>
> When I ran DISTCP to copy files from one dfs to another, I noticed that some mappers got killed due to failing to report status for 606 seconds.
> I noticed that the mappers try to make a progress report for every 32MB copied. A better way to ensure progress is to use a time interval since last report.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2048) DISTCP mapper should report progress
more often
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas updated HADOOP-2048:
----------------------------------
Attachment: (was: 2048.patch)
> DISTCP mapper should report progress more often
> -----------------------------------------------
>
> Key: HADOOP-2048
> URL: https://issues.apache.org/jira/browse/HADOOP-2048
> Project: Hadoop
> Issue Type: Bug
> Reporter: Runping Qi
> Assignee: Chris Douglas
> Attachments: 2048.patch
>
>
> When I ran DISTCP to copy files from one dfs to another, I noticed that some mappers got killed due to failing to report status for 606 seconds.
> I noticed that the mappers try to make a progress report for every 32MB copied. A better way to ensure progress is to use a time interval since last report.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2048) DISTCP mapper should report progress
more often
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas updated HADOOP-2048:
----------------------------------
Attachment: 2048-4.patch
Addressed comments
> DISTCP mapper should report progress more often
> -----------------------------------------------
>
> Key: HADOOP-2048
> URL: https://issues.apache.org/jira/browse/HADOOP-2048
> Project: Hadoop
> Issue Type: Bug
> Reporter: Runping Qi
> Assignee: Chris Douglas
> Priority: Blocker
> Fix For: 0.15.0
>
> Attachments: 2048-3.patch, 2048-4.patch
>
>
> When I ran DISTCP to copy files from one dfs to another, I noticed that some mappers got killed due to failing to report status for 606 seconds.
> I noticed that the mappers try to make a progress report for every 32MB copied. A better way to ensure progress is to use a time interval since last report.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2048) DISTCP mapper should report progress
more often
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas updated HADOOP-2048:
----------------------------------
Status: Open (was: Patch Available)
> DISTCP mapper should report progress more often
> -----------------------------------------------
>
> Key: HADOOP-2048
> URL: https://issues.apache.org/jira/browse/HADOOP-2048
> Project: Hadoop
> Issue Type: Bug
> Reporter: Runping Qi
> Assignee: Chris Douglas
> Priority: Blocker
> Fix For: 0.15.0
>
> Attachments: 2048-3.patch, 2048-4.patch
>
>
> When I ran DISTCP to copy files from one dfs to another, I noticed that some mappers got killed due to failing to report status for 606 seconds.
> I noticed that the mappers try to make a progress report for every 32MB copied. A better way to ensure progress is to use a time interval since last report.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2048) DISTCP mapper should report progress
more often
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas updated HADOOP-2048:
----------------------------------
Attachment: (was: 2048.patch)
> DISTCP mapper should report progress more often
> -----------------------------------------------
>
> Key: HADOOP-2048
> URL: https://issues.apache.org/jira/browse/HADOOP-2048
> Project: Hadoop
> Issue Type: Bug
> Reporter: Runping Qi
> Assignee: Chris Douglas
> Priority: Blocker
> Fix For: 0.15.0
>
> Attachments: 2048-3.patch
>
>
> When I ran DISTCP to copy files from one dfs to another, I noticed that some mappers got killed due to failing to report status for 606 seconds.
> I noticed that the mappers try to make a progress report for every 32MB copied. A better way to ensure progress is to use a time interval since last report.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2048) DISTCP mapper should report progress
more often
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Owen O'Malley updated HADOOP-2048:
----------------------------------
Fix Version/s: 0.15.0
Priority: Blocker (was: Major)
Description:
When I ran DISTCP to copy files from one dfs to another, I noticed that some mappers got killed due to failing to report status for 606 seconds.
I noticed that the mappers try to make a progress report for every 32MB copied. A better way to ensure progress is to use a time interval since last report.
was:
When I ran DISTCP to copy files from one dfs to another, I noticed that some mappers got killed due to failing to report status for 606 seconds.
I noticed that the mappers try to make a progress report for every 32MB copied. A better way to ensure progress is to use a time interval since last report.
This is a very important patch for distcp to ensure that the copy worked correctly and should be part of 15.
> DISTCP mapper should report progress more often
> -----------------------------------------------
>
> Key: HADOOP-2048
> URL: https://issues.apache.org/jira/browse/HADOOP-2048
> Project: Hadoop
> Issue Type: Bug
> Reporter: Runping Qi
> Assignee: Chris Douglas
> Priority: Blocker
> Fix For: 0.15.0
>
> Attachments: 2048-2.patch, 2048-3.patch, 2048.patch
>
>
> When I ran DISTCP to copy files from one dfs to another, I noticed that some mappers got killed due to failing to report status for 606 seconds.
> I noticed that the mappers try to make a progress report for every 32MB copied. A better way to ensure progress is to use a time interval since last report.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2048) DISTCP mapper should report progress
more often
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas updated HADOOP-2048:
----------------------------------
Status: Patch Available (was: Open)
> DISTCP mapper should report progress more often
> -----------------------------------------------
>
> Key: HADOOP-2048
> URL: https://issues.apache.org/jira/browse/HADOOP-2048
> Project: Hadoop
> Issue Type: Bug
> Reporter: Runping Qi
> Assignee: Chris Douglas
> Attachments: 2048.patch
>
>
> When I ran DISTCP to copy files from one dfs to another, I noticed that some mappers got killed due to failing to report status for 606 seconds.
> I noticed that the mappers try to make a progress report for every 32MB copied. A better way to ensure progress is to use a time interval since last report.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2048) DISTCP mapper should report progress
more often
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas updated HADOOP-2048:
----------------------------------
Status: Open (was: Patch Available)
> DISTCP mapper should report progress more often
> -----------------------------------------------
>
> Key: HADOOP-2048
> URL: https://issues.apache.org/jira/browse/HADOOP-2048
> Project: Hadoop
> Issue Type: Bug
> Reporter: Runping Qi
> Assignee: Chris Douglas
> Attachments: 2048-2.patch, 2048.patch
>
>
> When I ran DISTCP to copy files from one dfs to another, I noticed that some mappers got killed due to failing to report status for 606 seconds.
> I noticed that the mappers try to make a progress report for every 32MB copied. A better way to ensure progress is to use a time interval since last report.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2048) DISTCP mapper should report progress
more often
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas updated HADOOP-2048:
----------------------------------
Status: Open (was: Patch Available)
> DISTCP mapper should report progress more often
> -----------------------------------------------
>
> Key: HADOOP-2048
> URL: https://issues.apache.org/jira/browse/HADOOP-2048
> Project: Hadoop
> Issue Type: Bug
> Reporter: Runping Qi
> Assignee: Chris Douglas
> Attachments: 2048.patch
>
>
> When I ran DISTCP to copy files from one dfs to another, I noticed that some mappers got killed due to failing to report status for 606 seconds.
> I noticed that the mappers try to make a progress report for every 32MB copied. A better way to ensure progress is to use a time interval since last report.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2048) DISTCP mapper should report progress
more often
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Owen O'Malley updated HADOOP-2048:
----------------------------------
Status: Open (was: Patch Available)
A couple of issues:
1. Please use better/longer variable names.
2. The failures shouldn't be stored, but always logged at the INFO level.
3. I'd change the bfailed flag to failureCount and have the final exception record the number of failures.
4. Don't bother doing a time limit on the status reporting. The framework already limits it down to once a second.
5. Just use the status message to record # bytes copied, # files copied, # failures, since particular failures will be overwritten too quickly. You just want the user to know that there is something to look at in the logs.
> DISTCP mapper should report progress more often
> -----------------------------------------------
>
> Key: HADOOP-2048
> URL: https://issues.apache.org/jira/browse/HADOOP-2048
> Project: Hadoop
> Issue Type: Bug
> Reporter: Runping Qi
> Assignee: Chris Douglas
> Priority: Blocker
> Fix For: 0.15.0
>
> Attachments: 2048-2.patch, 2048-3.patch, 2048.patch
>
>
> When I ran DISTCP to copy files from one dfs to another, I noticed that some mappers got killed due to failing to report status for 606 seconds.
> I noticed that the mappers try to make a progress report for every 32MB copied. A better way to ensure progress is to use a time interval since last report.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2048) DISTCP mapper should report progress
more often
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas updated HADOOP-2048:
----------------------------------
Attachment: 2048-2.patch
Updated status reporting (HADOOP-2052)
> DISTCP mapper should report progress more often
> -----------------------------------------------
>
> Key: HADOOP-2048
> URL: https://issues.apache.org/jira/browse/HADOOP-2048
> Project: Hadoop
> Issue Type: Bug
> Reporter: Runping Qi
> Assignee: Chris Douglas
> Attachments: 2048-2.patch, 2048.patch
>
>
> When I ran DISTCP to copy files from one dfs to another, I noticed that some mappers got killed due to failing to report status for 606 seconds.
> I noticed that the mappers try to make a progress report for every 32MB copied. A better way to ensure progress is to use a time interval since last report.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2048) DISTCP mapper should report progress
more often
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas updated HADOOP-2048:
----------------------------------
Attachment: 2048.patch
Fixed findbugs warning
> DISTCP mapper should report progress more often
> -----------------------------------------------
>
> Key: HADOOP-2048
> URL: https://issues.apache.org/jira/browse/HADOOP-2048
> Project: Hadoop
> Issue Type: Bug
> Reporter: Runping Qi
> Assignee: Chris Douglas
> Attachments: 2048.patch
>
>
> When I ran DISTCP to copy files from one dfs to another, I noticed that some mappers got killed due to failing to report status for 606 seconds.
> I noticed that the mappers try to make a progress report for every 32MB copied. A better way to ensure progress is to use a time interval since last report.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2048) DISTCP mapper should report progress
more often
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas updated HADOOP-2048:
----------------------------------
Status: Patch Available (was: Open)
> DISTCP mapper should report progress more often
> -----------------------------------------------
>
> Key: HADOOP-2048
> URL: https://issues.apache.org/jira/browse/HADOOP-2048
> Project: Hadoop
> Issue Type: Bug
> Reporter: Runping Qi
> Assignee: Chris Douglas
> Priority: Blocker
> Fix For: 0.15.0
>
> Attachments: 2048-3.patch, 2048-4.patch, 2048-5.patch
>
>
> When I ran DISTCP to copy files from one dfs to another, I noticed that some mappers got killed due to failing to report status for 606 seconds.
> I noticed that the mappers try to make a progress report for every 32MB copied. A better way to ensure progress is to use a time interval since last report.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2048) DISTCP mapper should report progress
more often
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas updated HADOOP-2048:
----------------------------------
Status: Open (was: Patch Available)
> DISTCP mapper should report progress more often
> -----------------------------------------------
>
> Key: HADOOP-2048
> URL: https://issues.apache.org/jira/browse/HADOOP-2048
> Project: Hadoop
> Issue Type: Bug
> Reporter: Runping Qi
> Assignee: Chris Douglas
> Attachments: 2048-2.patch, 2048-3.patch, 2048.patch
>
>
> When I ran DISTCP to copy files from one dfs to another, I noticed that some mappers got killed due to failing to report status for 606 seconds.
> I noticed that the mappers try to make a progress report for every 32MB copied. A better way to ensure progress is to use a time interval since last report.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.