You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Siddharth Seth (Created) (JIRA)" <ji...@apache.org> on 2011/12/06 01:33:40 UTC
[jira] [Created] (MAPREDUCE-3512) Batch jobHistory disk flushes
Batch jobHistory disk flushes
-----------------------------
Key: MAPREDUCE-3512
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3512
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: mr-am, mrv2
Affects Versions: 0.23.0
Reporter: Siddharth Seth
The mr-am flushes each individual job history event to disk for AM recovery. The history even handler ends up with a significant backlog for tests like MAPREDUCE-3402.
History events could be batched up based on num records / time / TaskFinishedEvents to reduce the number of DFS writes - with the potential drawback of having to rerun some tasks during AM recovery.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3512) Batch jobHistory disk flushes
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13184669#comment-13184669 ]
Hudson commented on MAPREDUCE-3512:
-----------------------------------
Integrated in Hadoop-Mapreduce-0.23-Commit #383 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/383/])
MAPREDUCE-3512. Batching JobHistory flushing to DFS so that we don't flush for every event slowing down AM. Contributed by Siddarth Seth.
svn merge --ignore-ancestry -c 1230353 ../../trunk/
vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1230354
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobHistoryEventHandler.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/jobhistory
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/jobhistory/TestJobHistoryEventHandler.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java
> Batch jobHistory disk flushes
> -----------------------------
>
> Key: MAPREDUCE-3512
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3512
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mr-am, mrv2
> Affects Versions: 0.23.0
> Reporter: Siddharth Seth
> Assignee: Siddharth Seth
> Priority: Blocker
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3512-20120111.txt, MR3512.txt, MR3512.txt
>
>
> The mr-am flushes each individual job history event to disk for AM recovery. The history even handler ends up with a significant backlog for tests like MAPREDUCE-3402.
> History events could be batched up based on num records / time / TaskFinishedEvents to reduce the number of DFS writes - with the potential drawback of having to rerun some tasks during AM recovery.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3512) Batch jobHistory disk flushes
Posted by "Siddharth Seth (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13183737#comment-13183737 ]
Siddharth Seth commented on MAPREDUCE-3512:
-------------------------------------------
Had run some crude benchmarks - writing out about 40MB of data (iirc). With a flush after every 0.5KB - this took roughly 35 seconds. With one flush at the end - around 6 seconds. Flush every 500KB - about 12 seconds.
Also, a benchmark like AMScalability (100K map tasks) currently takes over 5 minutes after job complete to finish the history file.
> Batch jobHistory disk flushes
> -----------------------------
>
> Key: MAPREDUCE-3512
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3512
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mr-am, mrv2
> Affects Versions: 0.23.0
> Reporter: Siddharth Seth
> Assignee: Siddharth Seth
> Priority: Blocker
> Attachments: MR3512.txt, MR3512.txt
>
>
> The mr-am flushes each individual job history event to disk for AM recovery. The history even handler ends up with a significant backlog for tests like MAPREDUCE-3402.
> History events could be batched up based on num records / time / TaskFinishedEvents to reduce the number of DFS writes - with the potential drawback of having to rerun some tasks during AM recovery.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3512) Batch jobHistory disk flushes
Posted by "Amol Kekre (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13184282#comment-13184282 ]
Amol Kekre commented on MAPREDUCE-3512:
---------------------------------------
can we get this jira committed?
> Batch jobHistory disk flushes
> -----------------------------
>
> Key: MAPREDUCE-3512
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3512
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mr-am, mrv2
> Affects Versions: 0.23.0
> Reporter: Siddharth Seth
> Assignee: Siddharth Seth
> Priority: Blocker
> Attachments: MR3512.txt, MR3512.txt
>
>
> The mr-am flushes each individual job history event to disk for AM recovery. The history even handler ends up with a significant backlog for tests like MAPREDUCE-3402.
> History events could be batched up based on num records / time / TaskFinishedEvents to reduce the number of DFS writes - with the potential drawback of having to rerun some tasks during AM recovery.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3512) Batch jobHistory disk flushes
Posted by "Siddharth Seth (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Siddharth Seth updated MAPREDUCE-3512:
--------------------------------------
Status: Patch Available (was: Open)
> Batch jobHistory disk flushes
> -----------------------------
>
> Key: MAPREDUCE-3512
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3512
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mr-am, mrv2
> Affects Versions: 0.23.0
> Reporter: Siddharth Seth
> Assignee: Siddharth Seth
> Priority: Blocker
> Attachments: MR3512.txt, MR3512.txt
>
>
> The mr-am flushes each individual job history event to disk for AM recovery. The history even handler ends up with a significant backlog for tests like MAPREDUCE-3402.
> History events could be batched up based on num records / time / TaskFinishedEvents to reduce the number of DFS writes - with the potential drawback of having to rerun some tasks during AM recovery.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3512) Batch jobHistory disk flushes
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13184900#comment-13184900 ]
Hudson commented on MAPREDUCE-3512:
-----------------------------------
Integrated in Hadoop-Hdfs-trunk #923 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/923/])
MAPREDUCE-3512. Batching JobHistory flushing to DFS so that we don't flush for every event slowing down AM. Contributed by Siddarth Seth.
vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1230353
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobHistoryEventHandler.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/jobhistory
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/jobhistory/TestJobHistoryEventHandler.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java
> Batch jobHistory disk flushes
> -----------------------------
>
> Key: MAPREDUCE-3512
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3512
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mr-am, mrv2
> Affects Versions: 0.23.0
> Reporter: Siddharth Seth
> Assignee: Siddharth Seth
> Priority: Blocker
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3512-20120111.txt, MR3512.txt, MR3512.txt
>
>
> The mr-am flushes each individual job history event to disk for AM recovery. The history even handler ends up with a significant backlog for tests like MAPREDUCE-3402.
> History events could be batched up based on num records / time / TaskFinishedEvents to reduce the number of DFS writes - with the potential drawback of having to rerun some tasks during AM recovery.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3512) Batch jobHistory disk flushes
Posted by "Vinod Kumar Vavilapalli (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vinod Kumar Vavilapalli updated MAPREDUCE-3512:
-----------------------------------------------
Attachment: MAPREDUCE-3512-20120111.txt
Patch with trivial edits.
> Batch jobHistory disk flushes
> -----------------------------
>
> Key: MAPREDUCE-3512
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3512
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mr-am, mrv2
> Affects Versions: 0.23.0
> Reporter: Siddharth Seth
> Assignee: Siddharth Seth
> Priority: Blocker
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3512-20120111.txt, MR3512.txt, MR3512.txt
>
>
> The mr-am flushes each individual job history event to disk for AM recovery. The history even handler ends up with a significant backlog for tests like MAPREDUCE-3402.
> History events could be batched up based on num records / time / TaskFinishedEvents to reduce the number of DFS writes - with the potential drawback of having to rerun some tasks during AM recovery.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3512) Batch jobHistory disk flushes
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13184667#comment-13184667 ]
Hudson commented on MAPREDUCE-3512:
-----------------------------------
Integrated in Hadoop-Common-trunk-Commit #1534 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1534/])
MAPREDUCE-3512. Batching JobHistory flushing to DFS so that we don't flush for every event slowing down AM. Contributed by Siddarth Seth.
vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1230353
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobHistoryEventHandler.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/jobhistory
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/jobhistory/TestJobHistoryEventHandler.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java
> Batch jobHistory disk flushes
> -----------------------------
>
> Key: MAPREDUCE-3512
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3512
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mr-am, mrv2
> Affects Versions: 0.23.0
> Reporter: Siddharth Seth
> Assignee: Siddharth Seth
> Priority: Blocker
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3512-20120111.txt, MR3512.txt, MR3512.txt
>
>
> The mr-am flushes each individual job history event to disk for AM recovery. The history even handler ends up with a significant backlog for tests like MAPREDUCE-3402.
> History events could be batched up based on num records / time / TaskFinishedEvents to reduce the number of DFS writes - with the potential drawback of having to rerun some tasks during AM recovery.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3512) Batch jobHistory disk flushes
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13184575#comment-13184575 ]
Hadoop QA commented on MAPREDUCE-3512:
--------------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12510270/MAPREDUCE-3512-20120111.txt
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 2 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed unit tests in .
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1598//testReport/
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1598//console
This message is automatically generated.
> Batch jobHistory disk flushes
> -----------------------------
>
> Key: MAPREDUCE-3512
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3512
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mr-am, mrv2
> Affects Versions: 0.23.0
> Reporter: Siddharth Seth
> Assignee: Siddharth Seth
> Priority: Blocker
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3512-20120111.txt, MR3512.txt, MR3512.txt
>
>
> The mr-am flushes each individual job history event to disk for AM recovery. The history even handler ends up with a significant backlog for tests like MAPREDUCE-3402.
> History events could be batched up based on num records / time / TaskFinishedEvents to reduce the number of DFS writes - with the potential drawback of having to rerun some tasks during AM recovery.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3512) Batch jobHistory disk flushes
Posted by "Mahadev konar (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13164861#comment-13164861 ]
Mahadev konar commented on MAPREDUCE-3512:
------------------------------------------
@Vinod,
The only issue I see with that is that we might end up having only part of the event logged when an AM crashes. Am not sure if the history event handler can handle incomplete events.
> Batch jobHistory disk flushes
> -----------------------------
>
> Key: MAPREDUCE-3512
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3512
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mr-am, mrv2
> Affects Versions: 0.23.0
> Reporter: Siddharth Seth
>
> The mr-am flushes each individual job history event to disk for AM recovery. The history even handler ends up with a significant backlog for tests like MAPREDUCE-3402.
> History events could be batched up based on num records / time / TaskFinishedEvents to reduce the number of DFS writes - with the potential drawback of having to rerun some tasks during AM recovery.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3512) Batch jobHistory disk flushes
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13184935#comment-13184935 ]
Hudson commented on MAPREDUCE-3512:
-----------------------------------
Integrated in Hadoop-Mapreduce-0.23-Build #158 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/158/])
MAPREDUCE-3512. Batching JobHistory flushing to DFS so that we don't flush for every event slowing down AM. Contributed by Siddarth Seth.
svn merge --ignore-ancestry -c 1230353 ../../trunk/
vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1230354
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobHistoryEventHandler.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/jobhistory
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/jobhistory/TestJobHistoryEventHandler.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java
> Batch jobHistory disk flushes
> -----------------------------
>
> Key: MAPREDUCE-3512
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3512
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mr-am, mrv2
> Affects Versions: 0.23.0
> Reporter: Siddharth Seth
> Assignee: Siddharth Seth
> Priority: Blocker
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3512-20120111.txt, MR3512.txt, MR3512.txt
>
>
> The mr-am flushes each individual job history event to disk for AM recovery. The history even handler ends up with a significant backlog for tests like MAPREDUCE-3402.
> History events could be batched up based on num records / time / TaskFinishedEvents to reduce the number of DFS writes - with the potential drawback of having to rerun some tasks during AM recovery.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3512) Batch jobHistory disk flushes
Posted by "Vinod Kumar Vavilapalli (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vinod Kumar Vavilapalli updated MAPREDUCE-3512:
-----------------------------------------------
Hadoop Flags: Reviewed
Status: Patch Available (was: Open)
> Batch jobHistory disk flushes
> -----------------------------
>
> Key: MAPREDUCE-3512
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3512
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mr-am, mrv2
> Affects Versions: 0.23.0
> Reporter: Siddharth Seth
> Assignee: Siddharth Seth
> Priority: Blocker
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3512-20120111.txt, MR3512.txt, MR3512.txt
>
>
> The mr-am flushes each individual job history event to disk for AM recovery. The history even handler ends up with a significant backlog for tests like MAPREDUCE-3402.
> History events could be batched up based on num records / time / TaskFinishedEvents to reduce the number of DFS writes - with the potential drawback of having to rerun some tasks during AM recovery.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3512) Batch jobHistory disk flushes
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13184659#comment-13184659 ]
Hudson commented on MAPREDUCE-3512:
-----------------------------------
Integrated in Hadoop-Common-0.23-Commit #371 (See [https://builds.apache.org/job/Hadoop-Common-0.23-Commit/371/])
MAPREDUCE-3512. Batching JobHistory flushing to DFS so that we don't flush for every event slowing down AM. Contributed by Siddarth Seth.
svn merge --ignore-ancestry -c 1230353 ../../trunk/
vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1230354
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobHistoryEventHandler.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/jobhistory
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/jobhistory/TestJobHistoryEventHandler.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java
> Batch jobHistory disk flushes
> -----------------------------
>
> Key: MAPREDUCE-3512
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3512
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mr-am, mrv2
> Affects Versions: 0.23.0
> Reporter: Siddharth Seth
> Assignee: Siddharth Seth
> Priority: Blocker
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3512-20120111.txt, MR3512.txt, MR3512.txt
>
>
> The mr-am flushes each individual job history event to disk for AM recovery. The history even handler ends up with a significant backlog for tests like MAPREDUCE-3402.
> History events could be batched up based on num records / time / TaskFinishedEvents to reduce the number of DFS writes - with the potential drawback of having to rerun some tasks during AM recovery.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3512) Batch jobHistory disk flushes
Posted by "Vinod Kumar Vavilapalli (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vinod Kumar Vavilapalli updated MAPREDUCE-3512:
-----------------------------------------------
Resolution: Fixed
Release Note: Batching JobHistory flushing to DFS so that we don't flush for every event slowing down AM.
Status: Resolved (was: Patch Available)
Just committed this to trunk and branch-0.23. Thanks Sid!!!!!
> Batch jobHistory disk flushes
> -----------------------------
>
> Key: MAPREDUCE-3512
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3512
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mr-am, mrv2
> Affects Versions: 0.23.0
> Reporter: Siddharth Seth
> Assignee: Siddharth Seth
> Priority: Blocker
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3512-20120111.txt, MR3512.txt, MR3512.txt
>
>
> The mr-am flushes each individual job history event to disk for AM recovery. The history even handler ends up with a significant backlog for tests like MAPREDUCE-3402.
> History events could be batched up based on num records / time / TaskFinishedEvents to reduce the number of DFS writes - with the potential drawback of having to rerun some tasks during AM recovery.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3512) Batch jobHistory disk flushes
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13184664#comment-13184664 ]
Hudson commented on MAPREDUCE-3512:
-----------------------------------
Integrated in Hadoop-Hdfs-trunk-Commit #1607 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1607/])
MAPREDUCE-3512. Batching JobHistory flushing to DFS so that we don't flush for every event slowing down AM. Contributed by Siddarth Seth.
vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1230353
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobHistoryEventHandler.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/jobhistory
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/jobhistory/TestJobHistoryEventHandler.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java
> Batch jobHistory disk flushes
> -----------------------------
>
> Key: MAPREDUCE-3512
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3512
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mr-am, mrv2
> Affects Versions: 0.23.0
> Reporter: Siddharth Seth
> Assignee: Siddharth Seth
> Priority: Blocker
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3512-20120111.txt, MR3512.txt, MR3512.txt
>
>
> The mr-am flushes each individual job history event to disk for AM recovery. The history even handler ends up with a significant backlog for tests like MAPREDUCE-3402.
> History events could be batched up based on num records / time / TaskFinishedEvents to reduce the number of DFS writes - with the potential drawback of having to rerun some tasks during AM recovery.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-3512) Batch jobHistory disk flushes
Posted by "Siddharth Seth (Assigned) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Siddharth Seth reassigned MAPREDUCE-3512:
-----------------------------------------
Assignee: Siddharth Seth
> Batch jobHistory disk flushes
> -----------------------------
>
> Key: MAPREDUCE-3512
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3512
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mr-am, mrv2
> Affects Versions: 0.23.0
> Reporter: Siddharth Seth
> Assignee: Siddharth Seth
> Priority: Critical
>
> The mr-am flushes each individual job history event to disk for AM recovery. The history even handler ends up with a significant backlog for tests like MAPREDUCE-3402.
> History events could be batched up based on num records / time / TaskFinishedEvents to reduce the number of DFS writes - with the potential drawback of having to rerun some tasks during AM recovery.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3512) Batch jobHistory disk flushes
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13184912#comment-13184912 ]
Hudson commented on MAPREDUCE-3512:
-----------------------------------
Integrated in Hadoop-Hdfs-0.23-Build #136 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/136/])
MAPREDUCE-3512. Batching JobHistory flushing to DFS so that we don't flush for every event slowing down AM. Contributed by Siddarth Seth.
svn merge --ignore-ancestry -c 1230353 ../../trunk/
vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1230354
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobHistoryEventHandler.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/jobhistory
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/jobhistory/TestJobHistoryEventHandler.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java
> Batch jobHistory disk flushes
> -----------------------------
>
> Key: MAPREDUCE-3512
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3512
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mr-am, mrv2
> Affects Versions: 0.23.0
> Reporter: Siddharth Seth
> Assignee: Siddharth Seth
> Priority: Blocker
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3512-20120111.txt, MR3512.txt, MR3512.txt
>
>
> The mr-am flushes each individual job history event to disk for AM recovery. The history even handler ends up with a significant backlog for tests like MAPREDUCE-3402.
> History events could be batched up based on num records / time / TaskFinishedEvents to reduce the number of DFS writes - with the potential drawback of having to rerun some tasks during AM recovery.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3512) Batch jobHistory disk flushes
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13184946#comment-13184946 ]
Hudson commented on MAPREDUCE-3512:
-----------------------------------
Integrated in Hadoop-Mapreduce-trunk #956 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/956/])
MAPREDUCE-3512. Batching JobHistory flushing to DFS so that we don't flush for every event slowing down AM. Contributed by Siddarth Seth.
vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1230353
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobHistoryEventHandler.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/jobhistory
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/jobhistory/TestJobHistoryEventHandler.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java
> Batch jobHistory disk flushes
> -----------------------------
>
> Key: MAPREDUCE-3512
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3512
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mr-am, mrv2
> Affects Versions: 0.23.0
> Reporter: Siddharth Seth
> Assignee: Siddharth Seth
> Priority: Blocker
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3512-20120111.txt, MR3512.txt, MR3512.txt
>
>
> The mr-am flushes each individual job history event to disk for AM recovery. The history even handler ends up with a significant backlog for tests like MAPREDUCE-3402.
> History events could be batched up based on num records / time / TaskFinishedEvents to reduce the number of DFS writes - with the potential drawback of having to rerun some tasks during AM recovery.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3512) Batch jobHistory disk flushes
Posted by "Vinod Kumar Vavilapalli (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13184524#comment-13184524 ]
Vinod Kumar Vavilapalli commented on MAPREDUCE-3512:
----------------------------------------------------
+1 overall..
I am making trivial edits myself to avoid another iteration - more comments to explain what is happening, and a few variable renames.
> Batch jobHistory disk flushes
> -----------------------------
>
> Key: MAPREDUCE-3512
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3512
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mr-am, mrv2
> Affects Versions: 0.23.0
> Reporter: Siddharth Seth
> Assignee: Siddharth Seth
> Priority: Blocker
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3512-20120111.txt, MR3512.txt, MR3512.txt
>
>
> The mr-am flushes each individual job history event to disk for AM recovery. The history even handler ends up with a significant backlog for tests like MAPREDUCE-3402.
> History events could be batched up based on num records / time / TaskFinishedEvents to reduce the number of DFS writes - with the potential drawback of having to rerun some tasks during AM recovery.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3512) Batch jobHistory disk flushes
Posted by "Vinod Kumar Vavilapalli (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vinod Kumar Vavilapalli updated MAPREDUCE-3512:
-----------------------------------------------
Fix Version/s: 0.23.1
Status: Open (was: Patch Available)
> Batch jobHistory disk flushes
> -----------------------------
>
> Key: MAPREDUCE-3512
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3512
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mr-am, mrv2
> Affects Versions: 0.23.0
> Reporter: Siddharth Seth
> Assignee: Siddharth Seth
> Priority: Blocker
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3512-20120111.txt, MR3512.txt, MR3512.txt
>
>
> The mr-am flushes each individual job history event to disk for AM recovery. The history even handler ends up with a significant backlog for tests like MAPREDUCE-3402.
> History events could be batched up based on num records / time / TaskFinishedEvents to reduce the number of DFS writes - with the potential drawback of having to rerun some tasks during AM recovery.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3512) Batch jobHistory disk flushes
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13184676#comment-13184676 ]
Hudson commented on MAPREDUCE-3512:
-----------------------------------
Integrated in Hadoop-Mapreduce-trunk-Commit #1552 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1552/])
MAPREDUCE-3512. Batching JobHistory flushing to DFS so that we don't flush for every event slowing down AM. Contributed by Siddarth Seth.
vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1230353
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobHistoryEventHandler.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/jobhistory
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/jobhistory/TestJobHistoryEventHandler.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java
> Batch jobHistory disk flushes
> -----------------------------
>
> Key: MAPREDUCE-3512
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3512
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mr-am, mrv2
> Affects Versions: 0.23.0
> Reporter: Siddharth Seth
> Assignee: Siddharth Seth
> Priority: Blocker
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3512-20120111.txt, MR3512.txt, MR3512.txt
>
>
> The mr-am flushes each individual job history event to disk for AM recovery. The history even handler ends up with a significant backlog for tests like MAPREDUCE-3402.
> History events could be batched up based on num records / time / TaskFinishedEvents to reduce the number of DFS writes - with the potential drawback of having to rerun some tasks during AM recovery.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3512) Batch jobHistory disk flushes
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13183763#comment-13183763 ]
Hadoop QA commented on MAPREDUCE-3512:
--------------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12510129/MR3512.txt
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 2 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed unit tests in .
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1586//testReport/
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1586//console
This message is automatically generated.
> Batch jobHistory disk flushes
> -----------------------------
>
> Key: MAPREDUCE-3512
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3512
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mr-am, mrv2
> Affects Versions: 0.23.0
> Reporter: Siddharth Seth
> Assignee: Siddharth Seth
> Priority: Blocker
> Attachments: MR3512.txt, MR3512.txt
>
>
> The mr-am flushes each individual job history event to disk for AM recovery. The history even handler ends up with a significant backlog for tests like MAPREDUCE-3402.
> History events could be batched up based on num records / time / TaskFinishedEvents to reduce the number of DFS writes - with the potential drawback of having to rerun some tasks during AM recovery.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3512) Batch jobHistory disk flushes
Posted by "Sharad Agarwal (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13166514#comment-13166514 ]
Sharad Agarwal commented on MAPREDUCE-3512:
-------------------------------------------
bq. Am not sure if the history event handler can handle incomplete events.
it can't. in that case recovery will be aborted and it will fallback to running all tasks from start.
hflush has to happen at event boundaries. I knew it that hflush on every call may slow things up, but just didn't want to do premature optimization. the simple fix is to just put the events in bounded queue and do write + hflush when full. It has downside of some tasks being rerun on recovery but thats completely ok.
As suggested above, additionally doing it on TaskFinishedEvent will even be more optimal.
> Batch jobHistory disk flushes
> -----------------------------
>
> Key: MAPREDUCE-3512
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3512
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mr-am, mrv2
> Affects Versions: 0.23.0
> Reporter: Siddharth Seth
>
> The mr-am flushes each individual job history event to disk for AM recovery. The history even handler ends up with a significant backlog for tests like MAPREDUCE-3402.
> History events could be batched up based on num records / time / TaskFinishedEvents to reduce the number of DFS writes - with the potential drawback of having to rerun some tasks during AM recovery.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3512) Batch jobHistory disk flushes
Posted by "Mahadev konar (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mahadev konar updated MAPREDUCE-3512:
-------------------------------------
Priority: Critical (was: Major)
> Batch jobHistory disk flushes
> -----------------------------
>
> Key: MAPREDUCE-3512
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3512
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mr-am, mrv2
> Affects Versions: 0.23.0
> Reporter: Siddharth Seth
> Priority: Critical
>
> The mr-am flushes each individual job history event to disk for AM recovery. The history even handler ends up with a significant backlog for tests like MAPREDUCE-3402.
> History events could be batched up based on num records / time / TaskFinishedEvents to reduce the number of DFS writes - with the potential drawback of having to rerun some tasks during AM recovery.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3512) Batch jobHistory disk flushes
Posted by "Siddharth Seth (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Siddharth Seth updated MAPREDUCE-3512:
--------------------------------------
Attachment: MR3512.txt
Updated with unit tests.
> Batch jobHistory disk flushes
> -----------------------------
>
> Key: MAPREDUCE-3512
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3512
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mr-am, mrv2
> Affects Versions: 0.23.0
> Reporter: Siddharth Seth
> Assignee: Siddharth Seth
> Priority: Blocker
> Attachments: MR3512.txt, MR3512.txt
>
>
> The mr-am flushes each individual job history event to disk for AM recovery. The history even handler ends up with a significant backlog for tests like MAPREDUCE-3402.
> History events could be batched up based on num records / time / TaskFinishedEvents to reduce the number of DFS writes - with the potential drawback of having to rerun some tasks during AM recovery.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3512) Batch jobHistory disk flushes
Posted by "Vinod Kumar Vavilapalli (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13164852#comment-13164852 ]
Vinod Kumar Vavilapalli commented on MAPREDUCE-3512:
----------------------------------------------------
Do we have enough information to see if the write call itself is taking time or the subsequent (h)flush ? A micro benchmark perhaps? If the writes are already buffered by the DFSClient, then the blame goes to the sync call, in which case, we can simply fix this by doing the sync every so often instead of doing it always.
> Batch jobHistory disk flushes
> -----------------------------
>
> Key: MAPREDUCE-3512
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3512
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mr-am, mrv2
> Affects Versions: 0.23.0
> Reporter: Siddharth Seth
>
> The mr-am flushes each individual job history event to disk for AM recovery. The history even handler ends up with a significant backlog for tests like MAPREDUCE-3402.
> History events could be batched up based on num records / time / TaskFinishedEvents to reduce the number of DFS writes - with the potential drawback of having to rerun some tasks during AM recovery.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3512) Batch jobHistory disk flushes
Posted by "Siddharth Seth (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13164810#comment-13164810 ]
Siddharth Seth commented on MAPREDUCE-3512:
-------------------------------------------
bq. Perhaps what we want to do is to have a non-blocking check of the event queue so we can batch all events currently on the queue up to a given number of events in a single write. This way if there are not very many events we do more writes and the events are output quickly but if we start to fall behind in the writes then we start batching them up into bigger chunks which are more efficient.
Yep. That's a nice simple way of dealing with the queue and takes care of short/long running tasks.
> Batch jobHistory disk flushes
> -----------------------------
>
> Key: MAPREDUCE-3512
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3512
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mr-am, mrv2
> Affects Versions: 0.23.0
> Reporter: Siddharth Seth
>
> The mr-am flushes each individual job history event to disk for AM recovery. The history even handler ends up with a significant backlog for tests like MAPREDUCE-3402.
> History events could be batched up based on num records / time / TaskFinishedEvents to reduce the number of DFS writes - with the potential drawback of having to rerun some tasks during AM recovery.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3512) Batch jobHistory disk flushes
Posted by "Robert Joseph Evans (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13164483#comment-13164483 ]
Robert Joseph Evans commented on MAPREDUCE-3512:
------------------------------------------------
Unless we are some how stopping Tasks from doing any work until the event it written out to the history file, batching up the writes will reduce the number of tasks that have to rerun on AM Recovery. This is because we already have the events batched in the queue and if we crash while they are in the queue we cannot recover them.
Perhaps what we want to do is to have a non-blocking check of the event queue so we can batch all events currently on the queue up to a given number of events in a single write. This way if there are not very many events we do more writes and the events are output quickly but if we start to fall behind in the writes then we start batching them up into bigger chunks which are more efficient.
> Batch jobHistory disk flushes
> -----------------------------
>
> Key: MAPREDUCE-3512
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3512
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mr-am, mrv2
> Affects Versions: 0.23.0
> Reporter: Siddharth Seth
>
> The mr-am flushes each individual job history event to disk for AM recovery. The history even handler ends up with a significant backlog for tests like MAPREDUCE-3402.
> History events could be batched up based on num records / time / TaskFinishedEvents to reduce the number of DFS writes - with the potential drawback of having to rerun some tasks during AM recovery.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3512) Batch jobHistory disk flushes
Posted by "Siddharth Seth (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Siddharth Seth updated MAPREDUCE-3512:
--------------------------------------
Attachment: MR3512.txt
Initial patch - will upload another one later with some of the TODOs resolved and with unit tests.
> Batch jobHistory disk flushes
> -----------------------------
>
> Key: MAPREDUCE-3512
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3512
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mr-am, mrv2
> Affects Versions: 0.23.0
> Reporter: Siddharth Seth
> Assignee: Siddharth Seth
> Priority: Blocker
> Attachments: MR3512.txt
>
>
> The mr-am flushes each individual job history event to disk for AM recovery. The history even handler ends up with a significant backlog for tests like MAPREDUCE-3402.
> History events could be batched up based on num records / time / TaskFinishedEvents to reduce the number of DFS writes - with the potential drawback of having to rerun some tasks during AM recovery.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3512) Batch jobHistory disk flushes
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13184666#comment-13184666 ]
Hudson commented on MAPREDUCE-3512:
-----------------------------------
Integrated in Hadoop-Hdfs-0.23-Commit #361 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/361/])
MAPREDUCE-3512. Batching JobHistory flushing to DFS so that we don't flush for every event slowing down AM. Contributed by Siddarth Seth.
svn merge --ignore-ancestry -c 1230353 ../../trunk/
vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1230354
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobHistoryEventHandler.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/jobhistory
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/jobhistory/TestJobHistoryEventHandler.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java
> Batch jobHistory disk flushes
> -----------------------------
>
> Key: MAPREDUCE-3512
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3512
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mr-am, mrv2
> Affects Versions: 0.23.0
> Reporter: Siddharth Seth
> Assignee: Siddharth Seth
> Priority: Blocker
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3512-20120111.txt, MR3512.txt, MR3512.txt
>
>
> The mr-am flushes each individual job history event to disk for AM recovery. The history even handler ends up with a significant backlog for tests like MAPREDUCE-3402.
> History events could be batched up based on num records / time / TaskFinishedEvents to reduce the number of DFS writes - with the potential drawback of having to rerun some tasks during AM recovery.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3512) Batch jobHistory disk flushes
Posted by "Vinod Kumar Vavilapalli (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vinod Kumar Vavilapalli updated MAPREDUCE-3512:
-----------------------------------------------
Issue Type: Sub-task (was: Improvement)
Parent: MAPREDUCE-3561
> Batch jobHistory disk flushes
> -----------------------------
>
> Key: MAPREDUCE-3512
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3512
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mr-am, mrv2
> Affects Versions: 0.23.0
> Reporter: Siddharth Seth
>
> The mr-am flushes each individual job history event to disk for AM recovery. The history even handler ends up with a significant backlog for tests like MAPREDUCE-3402.
> History events could be batched up based on num records / time / TaskFinishedEvents to reduce the number of DFS writes - with the potential drawback of having to rerun some tasks during AM recovery.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira