You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Ravi Gummadi (Created) (JIRA)" <ji...@apache.org> on 2011/12/23 10:50:30 UTC
[jira] [Created] (MAPREDUCE-3597) Provide a way to access other
info of history file from Rumentool
Provide a way to access other info of history file from Rumentool
-----------------------------------------------------------------
Key: MAPREDUCE-3597
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3597
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: tools/rumen
Affects Versions: 0.24.0
Reporter: Ravi Gummadi
Assignee: Ravi Gummadi
Fix For: 0.24.0
As the trace file generated by Rumen TraceBuilder is skipping some of the info like job counters, task counters, etc. we need a way to access "other info available in history file which is not dumped to trace file". This is useful for components which want to parse history files and get info. These components can directly use/leverage "Rumen's parsing of history files across hadoop releases" and get history info in a consistent way for further analysis/processing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3597) Provide a way to access other
info of history file from Rumentool
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13190029#comment-13190029 ]
Hudson commented on MAPREDUCE-3597:
-----------------------------------
Integrated in Hadoop-Common-0.23-Commit #394 (See [https://builds.apache.org/job/Hadoop-Common-0.23-Commit/394/])
Merge -r 1222694:1222695 from trunk to branch. FIXES: MAPREDUCE-3597
tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1234086
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/tools/rumen/TestRumenJobTraces.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/tools/data/rumen/small-trace-test/counters-test-trace.json.gz
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/tools/data/rumen/small-trace-test/dispatch-sample-v20-jt-log.gz
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/tools/data/rumen/small-trace-test/dispatch-trace-output.json.gz
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/tools/org/apache/hadoop/tools/rumen/JobBuilder.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/tools/org/apache/hadoop/tools/rumen/JobHistoryUtils.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/tools/org/apache/hadoop/tools/rumen/LoggedJob.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/tools/org/apache/hadoop/tools/rumen/ParsedJob.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/tools/org/apache/hadoop/tools/rumen/ParsedTask.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/tools/org/apache/hadoop/tools/rumen/ParsedTaskAttempt.java
> Provide a way to access other info of history file from Rumentool
> -----------------------------------------------------------------
>
> Key: MAPREDUCE-3597
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3597
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: tools/rumen
> Affects Versions: 0.24.0
> Reporter: Ravi Gummadi
> Assignee: Ravi Gummadi
> Fix For: 0.24.0
>
> Attachments: 3597.branch-1.v1.patch, 3597.branch-1.v2.patch, 3597.v0.patch, 3597.v1.patch, MAPREDUCE-3597_branch-0.23.patch
>
>
> As the trace file generated by Rumen TraceBuilder is skipping some of the info like job counters, task counters, etc. we need a way to access "other info available in history file which is not dumped to trace file". This is useful for components which want to parse history files and get info. These components can directly use/leverage "Rumen's parsing of history files across hadoop releases" and get history info in a consistent way for further analysis/processing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3597) Provide a way to access other
info of history file from Rumentool
Posted by "Ravi Gummadi (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ravi Gummadi updated MAPREDUCE-3597:
------------------------------------
Attachment: 3597.branch-1.v2.1.patch
Attaching new patch for branch-1 incorporating Amar's review comments.
> Provide a way to access other info of history file from Rumentool
> -----------------------------------------------------------------
>
> Key: MAPREDUCE-3597
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3597
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: tools/rumen
> Affects Versions: 0.24.0
> Reporter: Ravi Gummadi
> Assignee: Ravi Gummadi
> Fix For: 0.24.0
>
> Attachments: 3597.branch-0.23.patch, 3597.branch-1.v1.patch, 3597.branch-1.v2.1.patch, 3597.branch-1.v2.patch, 3597.v0.patch, 3597.v1.patch, MAPREDUCE-3597_branch-0.23.patch
>
>
> As the trace file generated by Rumen TraceBuilder is skipping some of the info like job counters, task counters, etc. we need a way to access "other info available in history file which is not dumped to trace file". This is useful for components which want to parse history files and get info. These components can directly use/leverage "Rumen's parsing of history files across hadoop releases" and get history info in a consistent way for further analysis/processing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3597) Provide a way to access other
info of history file from Rumentool
Posted by "Amar Kamat (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13196816#comment-13196816 ]
Amar Kamat commented on MAPREDUCE-3597:
---------------------------------------
+1. Looks good to me.
> Provide a way to access other info of history file from Rumentool
> -----------------------------------------------------------------
>
> Key: MAPREDUCE-3597
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3597
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: tools/rumen
> Affects Versions: 0.24.0
> Reporter: Ravi Gummadi
> Assignee: Ravi Gummadi
> Fix For: 0.24.0
>
> Attachments: 3597.branch-0.23.patch, 3597.branch-1.v1.patch, 3597.branch-1.v2.1.patch, 3597.branch-1.v2.2.patch, 3597.branch-1.v2.patch, 3597.v0.patch, 3597.v1.patch, MAPREDUCE-3597_branch-0.23.patch
>
>
> As the trace file generated by Rumen TraceBuilder is skipping some of the info like job counters, task counters, etc. we need a way to access "other info available in history file which is not dumped to trace file". This is useful for components which want to parse history files and get info. These components can directly use/leverage "Rumen's parsing of history files across hadoop releases" and get history info in a consistent way for further analysis/processing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3597) Provide a way to access other
info of history file from Rumentool
Posted by "Amar Kamat (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13179309#comment-13179309 ]
Amar Kamat commented on MAPREDUCE-3597:
---------------------------------------
Ravi, is it possible to port this to branch-1?
> Provide a way to access other info of history file from Rumentool
> -----------------------------------------------------------------
>
> Key: MAPREDUCE-3597
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3597
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: tools/rumen
> Affects Versions: 0.24.0
> Reporter: Ravi Gummadi
> Assignee: Ravi Gummadi
> Fix For: 0.24.0
>
> Attachments: 3597.v0.patch, 3597.v1.patch
>
>
> As the trace file generated by Rumen TraceBuilder is skipping some of the info like job counters, task counters, etc. we need a way to access "other info available in history file which is not dumped to trace file". This is useful for components which want to parse history files and get info. These components can directly use/leverage "Rumen's parsing of history files across hadoop releases" and get history info in a consistent way for further analysis/processing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3597) Provide a way to access other
info of history file from Rumentool
Posted by "Vinod Kumar Vavilapalli (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vinod Kumar Vavilapalli updated MAPREDUCE-3597:
-----------------------------------------------
Affects Version/s: 0.23.1
Fix Version/s: 0.23.1
> Provide a way to access other info of history file from Rumentool
> -----------------------------------------------------------------
>
> Key: MAPREDUCE-3597
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3597
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: tools/rumen
> Affects Versions: 0.23.1, 0.24.0
> Reporter: Ravi Gummadi
> Assignee: Ravi Gummadi
> Fix For: 0.23.1, 0.24.0
>
> Attachments: 3597.branch-0.23.patch, 3597.branch-1.v1.patch, 3597.branch-1.v2.1.patch, 3597.branch-1.v2.2.patch, 3597.branch-1.v2.patch, 3597.v0.patch, 3597.v1.patch, MAPREDUCE-3597_branch-0.23.patch
>
>
> As the trace file generated by Rumen TraceBuilder is skipping some of the info like job counters, task counters, etc. we need a way to access "other info available in history file which is not dumped to trace file". This is useful for components which want to parse history files and get info. These components can directly use/leverage "Rumen's parsing of history files across hadoop releases" and get history info in a consistent way for further analysis/processing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3597) Provide a way to access other
info of history file from Rumentool
Posted by "Ravi Gummadi (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ravi Gummadi updated MAPREDUCE-3597:
------------------------------------
Attachment: 3597.v0.patch
Attaching a patch that adds API for accessing the other info (which is not writtent to trace file).
> Provide a way to access other info of history file from Rumentool
> -----------------------------------------------------------------
>
> Key: MAPREDUCE-3597
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3597
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: tools/rumen
> Affects Versions: 0.24.0
> Reporter: Ravi Gummadi
> Assignee: Ravi Gummadi
> Fix For: 0.24.0
>
> Attachments: 3597.v0.patch
>
>
> As the trace file generated by Rumen TraceBuilder is skipping some of the info like job counters, task counters, etc. we need a way to access "other info available in history file which is not dumped to trace file". This is useful for components which want to parse history files and get info. These components can directly use/leverage "Rumen's parsing of history files across hadoop releases" and get history info in a consistent way for further analysis/processing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3597) Provide a way to access other
info of history file from Rumentool
Posted by "Eli Collins (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eli Collins updated MAPREDUCE-3597:
-----------------------------------
Target Version/s: 0.23.1 (was: 0.24.0)
Seems like a good candidate for branch 23.
> Provide a way to access other info of history file from Rumentool
> -----------------------------------------------------------------
>
> Key: MAPREDUCE-3597
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3597
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: tools/rumen
> Affects Versions: 0.24.0
> Reporter: Ravi Gummadi
> Assignee: Ravi Gummadi
> Fix For: 0.24.0
>
> Attachments: 3597.v0.patch, 3597.v1.patch
>
>
> As the trace file generated by Rumen TraceBuilder is skipping some of the info like job counters, task counters, etc. we need a way to access "other info available in history file which is not dumped to trace file". This is useful for components which want to parse history files and get info. These components can directly use/leverage "Rumen's parsing of history files across hadoop releases" and get history info in a consistent way for further analysis/processing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3597) Provide a way to access other
info of history file from Rumentool
Posted by "Ravi Gummadi (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ravi Gummadi updated MAPREDUCE-3597:
------------------------------------
Attachment: 3597.branch-1.v2.2.patch
Attaching new patch by changing testcase a little bit.
> Provide a way to access other info of history file from Rumentool
> -----------------------------------------------------------------
>
> Key: MAPREDUCE-3597
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3597
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: tools/rumen
> Affects Versions: 0.24.0
> Reporter: Ravi Gummadi
> Assignee: Ravi Gummadi
> Fix For: 0.24.0
>
> Attachments: 3597.branch-0.23.patch, 3597.branch-1.v1.patch, 3597.branch-1.v2.1.patch, 3597.branch-1.v2.2.patch, 3597.branch-1.v2.patch, 3597.v0.patch, 3597.v1.patch, MAPREDUCE-3597_branch-0.23.patch
>
>
> As the trace file generated by Rumen TraceBuilder is skipping some of the info like job counters, task counters, etc. we need a way to access "other info available in history file which is not dumped to trace file". This is useful for components which want to parse history files and get info. These components can directly use/leverage "Rumen's parsing of history files across hadoop releases" and get history info in a consistent way for further analysis/processing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3597) Provide a way to access other
info of history file from Rumentool
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13190395#comment-13190395 ]
Hudson commented on MAPREDUCE-3597:
-----------------------------------
Integrated in Hadoop-Hdfs-0.23-Build #145 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/145/])
Merge -r 1222694:1222695 from trunk to branch. FIXES: MAPREDUCE-3597
tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1234086
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/tools/rumen/TestRumenJobTraces.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/tools/data/rumen/small-trace-test/counters-test-trace.json.gz
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/tools/data/rumen/small-trace-test/dispatch-sample-v20-jt-log.gz
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/tools/data/rumen/small-trace-test/dispatch-trace-output.json.gz
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/tools/org/apache/hadoop/tools/rumen/JobBuilder.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/tools/org/apache/hadoop/tools/rumen/JobHistoryUtils.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/tools/org/apache/hadoop/tools/rumen/LoggedJob.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/tools/org/apache/hadoop/tools/rumen/ParsedJob.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/tools/org/apache/hadoop/tools/rumen/ParsedTask.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/tools/org/apache/hadoop/tools/rumen/ParsedTaskAttempt.java
> Provide a way to access other info of history file from Rumentool
> -----------------------------------------------------------------
>
> Key: MAPREDUCE-3597
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3597
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: tools/rumen
> Affects Versions: 0.24.0
> Reporter: Ravi Gummadi
> Assignee: Ravi Gummadi
> Fix For: 0.24.0
>
> Attachments: 3597.branch-1.v1.patch, 3597.branch-1.v2.patch, 3597.v0.patch, 3597.v1.patch, MAPREDUCE-3597_branch-0.23.patch
>
>
> As the trace file generated by Rumen TraceBuilder is skipping some of the info like job counters, task counters, etc. we need a way to access "other info available in history file which is not dumped to trace file". This is useful for components which want to parse history files and get info. These components can directly use/leverage "Rumen's parsing of history files across hadoop releases" and get history info in a consistent way for further analysis/processing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3597) Provide a way to access other
info of history file from Rumentool
Posted by "Ravi Gummadi (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ravi Gummadi updated MAPREDUCE-3597:
------------------------------------
Attachment: 3597.branch-1.v1.patch
Attaching patch for branch-1.
> Provide a way to access other info of history file from Rumentool
> -----------------------------------------------------------------
>
> Key: MAPREDUCE-3597
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3597
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: tools/rumen
> Affects Versions: 0.24.0
> Reporter: Ravi Gummadi
> Assignee: Ravi Gummadi
> Fix For: 0.24.0
>
> Attachments: 3597.branch-1.v1.patch, 3597.v0.patch, 3597.v1.patch
>
>
> As the trace file generated by Rumen TraceBuilder is skipping some of the info like job counters, task counters, etc. we need a way to access "other info available in history file which is not dumped to trace file". This is useful for components which want to parse history files and get info. These components can directly use/leverage "Rumen's parsing of history files across hadoop releases" and get history info in a consistent way for further analysis/processing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3597) Provide a way to access other
info of history file from Rumentool
Posted by "Ravi Gummadi (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13190942#comment-13190942 ]
Ravi Gummadi commented on MAPREDUCE-3597:
-----------------------------------------
The above commit messages for branch-0.23 say that the 3 .gz files are modified. So commit seems to be fine, but the patch uploaded here in JIRA seems to be missing these files' changes. I will upload a patch for branch-0.23, which actually got committed and is correct.
> Provide a way to access other info of history file from Rumentool
> -----------------------------------------------------------------
>
> Key: MAPREDUCE-3597
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3597
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: tools/rumen
> Affects Versions: 0.24.0
> Reporter: Ravi Gummadi
> Assignee: Ravi Gummadi
> Fix For: 0.24.0
>
> Attachments: 3597.branch-1.v1.patch, 3597.branch-1.v2.patch, 3597.v0.patch, 3597.v1.patch, MAPREDUCE-3597_branch-0.23.patch
>
>
> As the trace file generated by Rumen TraceBuilder is skipping some of the info like job counters, task counters, etc. we need a way to access "other info available in history file which is not dumped to trace file". This is useful for components which want to parse history files and get info. These components can directly use/leverage "Rumen's parsing of history files across hadoop releases" and get history info in a consistent way for further analysis/processing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3597) Provide a way to access other
info of history file from Rumentool
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13175405#comment-13175405 ]
Hadoop QA commented on MAPREDUCE-3597:
--------------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12508512/3597.v1.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 12 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed unit tests in .
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1504//testReport/
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1504//console
This message is automatically generated.
> Provide a way to access other info of history file from Rumentool
> -----------------------------------------------------------------
>
> Key: MAPREDUCE-3597
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3597
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: tools/rumen
> Affects Versions: 0.24.0
> Reporter: Ravi Gummadi
> Assignee: Ravi Gummadi
> Fix For: 0.24.0
>
> Attachments: 3597.v0.patch, 3597.v1.patch
>
>
> As the trace file generated by Rumen TraceBuilder is skipping some of the info like job counters, task counters, etc. we need a way to access "other info available in history file which is not dumped to trace file". This is useful for components which want to parse history files and get info. These components can directly use/leverage "Rumen's parsing of history files across hadoop releases" and get history info in a consistent way for further analysis/processing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3597) Provide a way to access other
info of history file from Rumentool
Posted by "Amar Kamat (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amar Kamat updated MAPREDUCE-3597:
----------------------------------
Tags: parsed objects rumen
Resolution: Fixed
Release Note: Rumen now provides {{Parsed*}} objects. These objects provide extra information that are not provided by {{Logged*}} objects.
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
I just committed this to trunk. Thanks Ravi!
> Provide a way to access other info of history file from Rumentool
> -----------------------------------------------------------------
>
> Key: MAPREDUCE-3597
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3597
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: tools/rumen
> Affects Versions: 0.24.0
> Reporter: Ravi Gummadi
> Assignee: Ravi Gummadi
> Fix For: 0.24.0
>
> Attachments: 3597.v0.patch, 3597.v1.patch
>
>
> As the trace file generated by Rumen TraceBuilder is skipping some of the info like job counters, task counters, etc. we need a way to access "other info available in history file which is not dumped to trace file". This is useful for components which want to parse history files and get info. These components can directly use/leverage "Rumen's parsing of history files across hadoop releases" and get history info in a consistent way for further analysis/processing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3597) Provide a way to access other
info of history file from Rumentool
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13175455#comment-13175455 ]
Hudson commented on MAPREDUCE-3597:
-----------------------------------
Integrated in Hadoop-Common-trunk-Commit #1468 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1468/])
MAPREDUCE-3597. [Rumen] Rumen should provide APIs to access all the job-history related information.
amarrk : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1222695
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/tools/rumen/TestRumenJobTraces.java
* /hadoop/common/trunk/hadoop-mapreduce-project/src/test/tools/data/rumen/small-trace-test/counters-test-trace.json.gz
* /hadoop/common/trunk/hadoop-mapreduce-project/src/test/tools/data/rumen/small-trace-test/dispatch-sample-v20-jt-log.gz
* /hadoop/common/trunk/hadoop-mapreduce-project/src/test/tools/data/rumen/small-trace-test/dispatch-trace-output.json.gz
* /hadoop/common/trunk/hadoop-mapreduce-project/src/tools/org/apache/hadoop/tools/rumen/JobBuilder.java
* /hadoop/common/trunk/hadoop-mapreduce-project/src/tools/org/apache/hadoop/tools/rumen/JobHistoryUtils.java
* /hadoop/common/trunk/hadoop-mapreduce-project/src/tools/org/apache/hadoop/tools/rumen/LoggedJob.java
* /hadoop/common/trunk/hadoop-mapreduce-project/src/tools/org/apache/hadoop/tools/rumen/ParsedJob.java
* /hadoop/common/trunk/hadoop-mapreduce-project/src/tools/org/apache/hadoop/tools/rumen/ParsedTask.java
* /hadoop/common/trunk/hadoop-mapreduce-project/src/tools/org/apache/hadoop/tools/rumen/ParsedTaskAttempt.java
> Provide a way to access other info of history file from Rumentool
> -----------------------------------------------------------------
>
> Key: MAPREDUCE-3597
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3597
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: tools/rumen
> Affects Versions: 0.24.0
> Reporter: Ravi Gummadi
> Assignee: Ravi Gummadi
> Fix For: 0.24.0
>
> Attachments: 3597.v0.patch, 3597.v1.patch
>
>
> As the trace file generated by Rumen TraceBuilder is skipping some of the info like job counters, task counters, etc. we need a way to access "other info available in history file which is not dumped to trace file". This is useful for components which want to parse history files and get info. These components can directly use/leverage "Rumen's parsing of history files across hadoop releases" and get history info in a consistent way for further analysis/processing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3597) Provide a way to access other
info of history file from Rumentool
Posted by "Ravi Gummadi (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ravi Gummadi updated MAPREDUCE-3597:
------------------------------------
Attachment: 3597.v1.patch
Attaching new patch with testcases for hadoop20 history files also.
> Provide a way to access other info of history file from Rumentool
> -----------------------------------------------------------------
>
> Key: MAPREDUCE-3597
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3597
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: tools/rumen
> Affects Versions: 0.24.0
> Reporter: Ravi Gummadi
> Assignee: Ravi Gummadi
> Fix For: 0.24.0
>
> Attachments: 3597.v0.patch, 3597.v1.patch
>
>
> As the trace file generated by Rumen TraceBuilder is skipping some of the info like job counters, task counters, etc. we need a way to access "other info available in history file which is not dumped to trace file". This is useful for components which want to parse history files and get info. These components can directly use/leverage "Rumen's parsing of history files across hadoop releases" and get history info in a consistent way for further analysis/processing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3597) Provide a way to access other
info of history file from Rumentool
Posted by "Alejandro Abdelnur (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alejandro Abdelnur updated MAPREDUCE-3597:
------------------------------------------
Attachment: MAPREDUCE-3597_branch-0.23.patch
patch for branch-0.23 by Ahmed (originally posted in MAPREDUCE-3582)
> Provide a way to access other info of history file from Rumentool
> -----------------------------------------------------------------
>
> Key: MAPREDUCE-3597
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3597
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: tools/rumen
> Affects Versions: 0.24.0
> Reporter: Ravi Gummadi
> Assignee: Ravi Gummadi
> Fix For: 0.24.0
>
> Attachments: 3597.branch-1.v1.patch, 3597.branch-1.v2.patch, 3597.v0.patch, 3597.v1.patch, MAPREDUCE-3597_branch-0.23.patch
>
>
> As the trace file generated by Rumen TraceBuilder is skipping some of the info like job counters, task counters, etc. we need a way to access "other info available in history file which is not dumped to trace file". This is useful for components which want to parse history files and get info. These components can directly use/leverage "Rumen's parsing of history files across hadoop releases" and get history info in a consistent way for further analysis/processing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3597) Provide a way to access other
info of history file from Rumentool
Posted by "Ravi Gummadi (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13196817#comment-13196817 ]
Ravi Gummadi commented on MAPREDUCE-3597:
-----------------------------------------
Unit tests and test-patch passed for the patch for branch-1.
> Provide a way to access other info of history file from Rumentool
> -----------------------------------------------------------------
>
> Key: MAPREDUCE-3597
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3597
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: tools/rumen
> Affects Versions: 0.24.0
> Reporter: Ravi Gummadi
> Assignee: Ravi Gummadi
> Fix For: 0.24.0
>
> Attachments: 3597.branch-0.23.patch, 3597.branch-1.v1.patch, 3597.branch-1.v2.1.patch, 3597.branch-1.v2.2.patch, 3597.branch-1.v2.patch, 3597.v0.patch, 3597.v1.patch, MAPREDUCE-3597_branch-0.23.patch
>
>
> As the trace file generated by Rumen TraceBuilder is skipping some of the info like job counters, task counters, etc. we need a way to access "other info available in history file which is not dumped to trace file". This is useful for components which want to parse history files and get info. These components can directly use/leverage "Rumen's parsing of history files across hadoop releases" and get history info in a consistent way for further analysis/processing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3597) Provide a way to access other
info of history file from Rumentool
Posted by "Ravi Gummadi (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ravi Gummadi updated MAPREDUCE-3597:
------------------------------------
Status: Patch Available (was: Open)
> Provide a way to access other info of history file from Rumentool
> -----------------------------------------------------------------
>
> Key: MAPREDUCE-3597
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3597
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: tools/rumen
> Affects Versions: 0.24.0
> Reporter: Ravi Gummadi
> Assignee: Ravi Gummadi
> Fix For: 0.24.0
>
> Attachments: 3597.v0.patch
>
>
> As the trace file generated by Rumen TraceBuilder is skipping some of the info like job counters, task counters, etc. we need a way to access "other info available in history file which is not dumped to trace file". This is useful for components which want to parse history files and get info. These components can directly use/leverage "Rumen's parsing of history files across hadoop releases" and get history info in a consistent way for further analysis/processing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3597) Provide a way to access other
info of history file from Rumentool
Posted by "Ravi Gummadi (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ravi Gummadi updated MAPREDUCE-3597:
------------------------------------
Attachment: 3597.branch-1.v2.patch
Attaching updated patch for branch-1 fixing some issues.
> Provide a way to access other info of history file from Rumentool
> -----------------------------------------------------------------
>
> Key: MAPREDUCE-3597
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3597
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: tools/rumen
> Affects Versions: 0.24.0
> Reporter: Ravi Gummadi
> Assignee: Ravi Gummadi
> Fix For: 0.24.0
>
> Attachments: 3597.branch-1.v1.patch, 3597.branch-1.v2.patch, 3597.v0.patch, 3597.v1.patch
>
>
> As the trace file generated by Rumen TraceBuilder is skipping some of the info like job counters, task counters, etc. we need a way to access "other info available in history file which is not dumped to trace file". This is useful for components which want to parse history files and get info. These components can directly use/leverage "Rumen's parsing of history files across hadoop releases" and get history info in a consistent way for further analysis/processing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3597) Provide a way to access other
info of history file from Rumentool
Posted by "Ravi Gummadi (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ravi Gummadi updated MAPREDUCE-3597:
------------------------------------
Attachment: 3597.branch-0.23.patch
Attaching the correct patch for branch-0.23. This got committed already.
> Provide a way to access other info of history file from Rumentool
> -----------------------------------------------------------------
>
> Key: MAPREDUCE-3597
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3597
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: tools/rumen
> Affects Versions: 0.24.0
> Reporter: Ravi Gummadi
> Assignee: Ravi Gummadi
> Fix For: 0.24.0
>
> Attachments: 3597.branch-0.23.patch, 3597.branch-1.v1.patch, 3597.branch-1.v2.patch, 3597.v0.patch, 3597.v1.patch, MAPREDUCE-3597_branch-0.23.patch
>
>
> As the trace file generated by Rumen TraceBuilder is skipping some of the info like job counters, task counters, etc. we need a way to access "other info available in history file which is not dumped to trace file". This is useful for components which want to parse history files and get info. These components can directly use/leverage "Rumen's parsing of history files across hadoop releases" and get history info in a consistent way for further analysis/processing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3597) Provide a way to access other
info of history file from Rumentool
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13175378#comment-13175378 ]
Hadoop QA commented on MAPREDUCE-3597:
--------------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12508503/3597.v0.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 12 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed unit tests in .
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1502//testReport/
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1502//console
This message is automatically generated.
> Provide a way to access other info of history file from Rumentool
> -----------------------------------------------------------------
>
> Key: MAPREDUCE-3597
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3597
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: tools/rumen
> Affects Versions: 0.24.0
> Reporter: Ravi Gummadi
> Assignee: Ravi Gummadi
> Fix For: 0.24.0
>
> Attachments: 3597.v0.patch
>
>
> As the trace file generated by Rumen TraceBuilder is skipping some of the info like job counters, task counters, etc. we need a way to access "other info available in history file which is not dumped to trace file". This is useful for components which want to parse history files and get info. These components can directly use/leverage "Rumen's parsing of history files across hadoop releases" and get history info in a consistent way for further analysis/processing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3597) Provide a way to access other
info of history file from Rumentool
Posted by "Ravi Gummadi (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ravi Gummadi updated MAPREDUCE-3597:
------------------------------------
Status: Patch Available (was: Open)
> Provide a way to access other info of history file from Rumentool
> -----------------------------------------------------------------
>
> Key: MAPREDUCE-3597
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3597
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: tools/rumen
> Affects Versions: 0.24.0
> Reporter: Ravi Gummadi
> Assignee: Ravi Gummadi
> Fix For: 0.24.0
>
> Attachments: 3597.v0.patch, 3597.v1.patch
>
>
> As the trace file generated by Rumen TraceBuilder is skipping some of the info like job counters, task counters, etc. we need a way to access "other info available in history file which is not dumped to trace file". This is useful for components which want to parse history files and get info. These components can directly use/leverage "Rumen's parsing of history files across hadoop releases" and get history info in a consistent way for further analysis/processing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3597) Provide a way to access other
info of history file from Rumentool
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13190410#comment-13190410 ]
Hudson commented on MAPREDUCE-3597:
-----------------------------------
Integrated in Hadoop-Mapreduce-0.23-Build #167 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/167/])
Merge -r 1222694:1222695 from trunk to branch. FIXES: MAPREDUCE-3597
tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1234086
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/tools/rumen/TestRumenJobTraces.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/tools/data/rumen/small-trace-test/counters-test-trace.json.gz
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/tools/data/rumen/small-trace-test/dispatch-sample-v20-jt-log.gz
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/tools/data/rumen/small-trace-test/dispatch-trace-output.json.gz
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/tools/org/apache/hadoop/tools/rumen/JobBuilder.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/tools/org/apache/hadoop/tools/rumen/JobHistoryUtils.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/tools/org/apache/hadoop/tools/rumen/LoggedJob.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/tools/org/apache/hadoop/tools/rumen/ParsedJob.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/tools/org/apache/hadoop/tools/rumen/ParsedTask.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/tools/org/apache/hadoop/tools/rumen/ParsedTaskAttempt.java
> Provide a way to access other info of history file from Rumentool
> -----------------------------------------------------------------
>
> Key: MAPREDUCE-3597
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3597
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: tools/rumen
> Affects Versions: 0.24.0
> Reporter: Ravi Gummadi
> Assignee: Ravi Gummadi
> Fix For: 0.24.0
>
> Attachments: 3597.branch-1.v1.patch, 3597.branch-1.v2.patch, 3597.v0.patch, 3597.v1.patch, MAPREDUCE-3597_branch-0.23.patch
>
>
> As the trace file generated by Rumen TraceBuilder is skipping some of the info like job counters, task counters, etc. we need a way to access "other info available in history file which is not dumped to trace file". This is useful for components which want to parse history files and get info. These components can directly use/leverage "Rumen's parsing of history files across hadoop releases" and get history info in a consistent way for further analysis/processing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3597) Provide a way to access other
info of history file from Rumentool
Posted by "Alejandro Abdelnur (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13190026#comment-13190026 ]
Alejandro Abdelnur commented on MAPREDUCE-3597:
-----------------------------------------------
committed to branch-0.23
> Provide a way to access other info of history file from Rumentool
> -----------------------------------------------------------------
>
> Key: MAPREDUCE-3597
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3597
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: tools/rumen
> Affects Versions: 0.24.0
> Reporter: Ravi Gummadi
> Assignee: Ravi Gummadi
> Fix For: 0.24.0
>
> Attachments: 3597.branch-1.v1.patch, 3597.branch-1.v2.patch, 3597.v0.patch, 3597.v1.patch, MAPREDUCE-3597_branch-0.23.patch
>
>
> As the trace file generated by Rumen TraceBuilder is skipping some of the info like job counters, task counters, etc. we need a way to access "other info available in history file which is not dumped to trace file". This is useful for components which want to parse history files and get info. These components can directly use/leverage "Rumen's parsing of history files across hadoop releases" and get history info in a consistent way for further analysis/processing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3597) Provide a way to access other
info of history file from Rumentool
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13175454#comment-13175454 ]
Hudson commented on MAPREDUCE-3597:
-----------------------------------
Integrated in Hadoop-Hdfs-trunk-Commit #1540 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1540/])
MAPREDUCE-3597. [Rumen] Rumen should provide APIs to access all the job-history related information.
amarrk : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1222695
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/tools/rumen/TestRumenJobTraces.java
* /hadoop/common/trunk/hadoop-mapreduce-project/src/test/tools/data/rumen/small-trace-test/counters-test-trace.json.gz
* /hadoop/common/trunk/hadoop-mapreduce-project/src/test/tools/data/rumen/small-trace-test/dispatch-sample-v20-jt-log.gz
* /hadoop/common/trunk/hadoop-mapreduce-project/src/test/tools/data/rumen/small-trace-test/dispatch-trace-output.json.gz
* /hadoop/common/trunk/hadoop-mapreduce-project/src/tools/org/apache/hadoop/tools/rumen/JobBuilder.java
* /hadoop/common/trunk/hadoop-mapreduce-project/src/tools/org/apache/hadoop/tools/rumen/JobHistoryUtils.java
* /hadoop/common/trunk/hadoop-mapreduce-project/src/tools/org/apache/hadoop/tools/rumen/LoggedJob.java
* /hadoop/common/trunk/hadoop-mapreduce-project/src/tools/org/apache/hadoop/tools/rumen/ParsedJob.java
* /hadoop/common/trunk/hadoop-mapreduce-project/src/tools/org/apache/hadoop/tools/rumen/ParsedTask.java
* /hadoop/common/trunk/hadoop-mapreduce-project/src/tools/org/apache/hadoop/tools/rumen/ParsedTaskAttempt.java
> Provide a way to access other info of history file from Rumentool
> -----------------------------------------------------------------
>
> Key: MAPREDUCE-3597
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3597
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: tools/rumen
> Affects Versions: 0.24.0
> Reporter: Ravi Gummadi
> Assignee: Ravi Gummadi
> Fix For: 0.24.0
>
> Attachments: 3597.v0.patch, 3597.v1.patch
>
>
> As the trace file generated by Rumen TraceBuilder is skipping some of the info like job counters, task counters, etc. we need a way to access "other info available in history file which is not dumped to trace file". This is useful for components which want to parse history files and get info. These components can directly use/leverage "Rumen's parsing of history files across hadoop releases" and get history info in a consistent way for further analysis/processing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3597) Provide a way to access other
info of history file from Rumentool
Posted by "Amar Kamat (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13175445#comment-13175445 ]
Amar Kamat commented on MAPREDUCE-3597:
---------------------------------------
{{test-patch}} passed on my local box. Rumen and Gridmix tests passed.
> Provide a way to access other info of history file from Rumentool
> -----------------------------------------------------------------
>
> Key: MAPREDUCE-3597
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3597
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: tools/rumen
> Affects Versions: 0.24.0
> Reporter: Ravi Gummadi
> Assignee: Ravi Gummadi
> Fix For: 0.24.0
>
> Attachments: 3597.v0.patch, 3597.v1.patch
>
>
> As the trace file generated by Rumen TraceBuilder is skipping some of the info like job counters, task counters, etc. we need a way to access "other info available in history file which is not dumped to trace file". This is useful for components which want to parse history files and get info. These components can directly use/leverage "Rumen's parsing of history files across hadoop releases" and get history info in a consistent way for further analysis/processing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3597) Provide a way to access other
info of history file from Rumentool
Posted by "Matt Foley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Matt Foley updated MAPREDUCE-3597:
----------------------------------
Target Version/s: 0.23.1, 1.1.0 (was: 0.23.1)
Fix Version/s: 1.1.0
> Provide a way to access other info of history file from Rumentool
> -----------------------------------------------------------------
>
> Key: MAPREDUCE-3597
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3597
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: tools/rumen
> Affects Versions: 0.23.1, 0.24.0
> Reporter: Ravi Gummadi
> Assignee: Ravi Gummadi
> Fix For: 0.23.1, 1.1.0
>
> Attachments: 3597.branch-0.23.patch, 3597.branch-1.v1.patch, 3597.branch-1.v2.1.patch, 3597.branch-1.v2.2.patch, 3597.branch-1.v2.patch, 3597.v0.patch, 3597.v1.patch, MAPREDUCE-3597_branch-0.23.patch
>
>
> As the trace file generated by Rumen TraceBuilder is skipping some of the info like job counters, task counters, etc. we need a way to access "other info available in history file which is not dumped to trace file". This is useful for components which want to parse history files and get info. These components can directly use/leverage "Rumen's parsing of history files across hadoop releases" and get history info in a consistent way for further analysis/processing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3597) Provide a way to access other
info of history file from Rumentool
Posted by "Amar Kamat (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13189633#comment-13189633 ]
Amar Kamat commented on MAPREDUCE-3597:
---------------------------------------
Few questions:
1. You have added a new {{getQueueName()}} API to {{JobSubmittedEvent}} class. Can you add some test case or validation lines for this newly added API?
2. Is there a testcase testing the map/reduce job level counters? As I understand, these counters got added recently and not available in the test logs. Is it possible to run a MR job (or reuse logs from other test scenarios) for the same?
3. Is it possible for the conf entires (e.g. queue name etc) to be null? Is it safe to check for nullity before setting the field in Parsed/LoggedTask from the conf?
> Provide a way to access other info of history file from Rumentool
> -----------------------------------------------------------------
>
> Key: MAPREDUCE-3597
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3597
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: tools/rumen
> Affects Versions: 0.24.0
> Reporter: Ravi Gummadi
> Assignee: Ravi Gummadi
> Fix For: 0.24.0
>
> Attachments: 3597.branch-1.v1.patch, 3597.branch-1.v2.patch, 3597.v0.patch, 3597.v1.patch
>
>
> As the trace file generated by Rumen TraceBuilder is skipping some of the info like job counters, task counters, etc. we need a way to access "other info available in history file which is not dumped to trace file". This is useful for components which want to parse history files and get info. These components can directly use/leverage "Rumen's parsing of history files across hadoop releases" and get history info in a consistent way for further analysis/processing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3597) Provide a way to access other
info of history file from Rumentool
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13175468#comment-13175468 ]
Hudson commented on MAPREDUCE-3597:
-----------------------------------
Integrated in Hadoop-Mapreduce-trunk-Commit #1489 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1489/])
MAPREDUCE-3597. [Rumen] Rumen should provide APIs to access all the job-history related information.
amarrk : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1222695
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/tools/rumen/TestRumenJobTraces.java
* /hadoop/common/trunk/hadoop-mapreduce-project/src/test/tools/data/rumen/small-trace-test/counters-test-trace.json.gz
* /hadoop/common/trunk/hadoop-mapreduce-project/src/test/tools/data/rumen/small-trace-test/dispatch-sample-v20-jt-log.gz
* /hadoop/common/trunk/hadoop-mapreduce-project/src/test/tools/data/rumen/small-trace-test/dispatch-trace-output.json.gz
* /hadoop/common/trunk/hadoop-mapreduce-project/src/tools/org/apache/hadoop/tools/rumen/JobBuilder.java
* /hadoop/common/trunk/hadoop-mapreduce-project/src/tools/org/apache/hadoop/tools/rumen/JobHistoryUtils.java
* /hadoop/common/trunk/hadoop-mapreduce-project/src/tools/org/apache/hadoop/tools/rumen/LoggedJob.java
* /hadoop/common/trunk/hadoop-mapreduce-project/src/tools/org/apache/hadoop/tools/rumen/ParsedJob.java
* /hadoop/common/trunk/hadoop-mapreduce-project/src/tools/org/apache/hadoop/tools/rumen/ParsedTask.java
* /hadoop/common/trunk/hadoop-mapreduce-project/src/tools/org/apache/hadoop/tools/rumen/ParsedTaskAttempt.java
> Provide a way to access other info of history file from Rumentool
> -----------------------------------------------------------------
>
> Key: MAPREDUCE-3597
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3597
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: tools/rumen
> Affects Versions: 0.24.0
> Reporter: Ravi Gummadi
> Assignee: Ravi Gummadi
> Fix For: 0.24.0
>
> Attachments: 3597.v0.patch, 3597.v1.patch
>
>
> As the trace file generated by Rumen TraceBuilder is skipping some of the info like job counters, task counters, etc. we need a way to access "other info available in history file which is not dumped to trace file". This is useful for components which want to parse history files and get info. These components can directly use/leverage "Rumen's parsing of history files across hadoop releases" and get history info in a consistent way for further analysis/processing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3597) Provide a way to access other
info of history file from Rumentool
Posted by "Ravi Gummadi (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13190931#comment-13190931 ]
Ravi Gummadi commented on MAPREDUCE-3597:
-----------------------------------------
The patch for branch-0.23 is not having the binary files' changes (changes to .gz files).
> Provide a way to access other info of history file from Rumentool
> -----------------------------------------------------------------
>
> Key: MAPREDUCE-3597
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3597
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: tools/rumen
> Affects Versions: 0.24.0
> Reporter: Ravi Gummadi
> Assignee: Ravi Gummadi
> Fix For: 0.24.0
>
> Attachments: 3597.branch-1.v1.patch, 3597.branch-1.v2.patch, 3597.v0.patch, 3597.v1.patch, MAPREDUCE-3597_branch-0.23.patch
>
>
> As the trace file generated by Rumen TraceBuilder is skipping some of the info like job counters, task counters, etc. we need a way to access "other info available in history file which is not dumped to trace file". This is useful for components which want to parse history files and get info. These components can directly use/leverage "Rumen's parsing of history files across hadoop releases" and get history info in a consistent way for further analysis/processing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3597) Provide a way to access other
info of history file from Rumentool
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13175720#comment-13175720 ]
Hudson commented on MAPREDUCE-3597:
-----------------------------------
Integrated in Hadoop-Mapreduce-trunk #937 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/937/])
MAPREDUCE-3597. [Rumen] Rumen should provide APIs to access all the job-history related information.
amarrk : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1222695
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/tools/rumen/TestRumenJobTraces.java
* /hadoop/common/trunk/hadoop-mapreduce-project/src/test/tools/data/rumen/small-trace-test/counters-test-trace.json.gz
* /hadoop/common/trunk/hadoop-mapreduce-project/src/test/tools/data/rumen/small-trace-test/dispatch-sample-v20-jt-log.gz
* /hadoop/common/trunk/hadoop-mapreduce-project/src/test/tools/data/rumen/small-trace-test/dispatch-trace-output.json.gz
* /hadoop/common/trunk/hadoop-mapreduce-project/src/tools/org/apache/hadoop/tools/rumen/JobBuilder.java
* /hadoop/common/trunk/hadoop-mapreduce-project/src/tools/org/apache/hadoop/tools/rumen/JobHistoryUtils.java
* /hadoop/common/trunk/hadoop-mapreduce-project/src/tools/org/apache/hadoop/tools/rumen/LoggedJob.java
* /hadoop/common/trunk/hadoop-mapreduce-project/src/tools/org/apache/hadoop/tools/rumen/ParsedJob.java
* /hadoop/common/trunk/hadoop-mapreduce-project/src/tools/org/apache/hadoop/tools/rumen/ParsedTask.java
* /hadoop/common/trunk/hadoop-mapreduce-project/src/tools/org/apache/hadoop/tools/rumen/ParsedTaskAttempt.java
> Provide a way to access other info of history file from Rumentool
> -----------------------------------------------------------------
>
> Key: MAPREDUCE-3597
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3597
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: tools/rumen
> Affects Versions: 0.24.0
> Reporter: Ravi Gummadi
> Assignee: Ravi Gummadi
> Fix For: 0.24.0
>
> Attachments: 3597.v0.patch, 3597.v1.patch
>
>
> As the trace file generated by Rumen TraceBuilder is skipping some of the info like job counters, task counters, etc. we need a way to access "other info available in history file which is not dumped to trace file". This is useful for components which want to parse history files and get info. These components can directly use/leverage "Rumen's parsing of history files across hadoop releases" and get history info in a consistent way for further analysis/processing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3597) Provide a way to access other
info of history file from Rumentool
Posted by "Amar Kamat (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13185576#comment-13185576 ]
Amar Kamat commented on MAPREDUCE-3597:
---------------------------------------
The patch looks good to me. It seems that branch-1 Rumen is aware of pre and post 21 changes. We need to be sure of the implications.
> Provide a way to access other info of history file from Rumentool
> -----------------------------------------------------------------
>
> Key: MAPREDUCE-3597
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3597
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: tools/rumen
> Affects Versions: 0.24.0
> Reporter: Ravi Gummadi
> Assignee: Ravi Gummadi
> Fix For: 0.24.0
>
> Attachments: 3597.branch-1.v1.patch, 3597.v0.patch, 3597.v1.patch
>
>
> As the trace file generated by Rumen TraceBuilder is skipping some of the info like job counters, task counters, etc. we need a way to access "other info available in history file which is not dumped to trace file". This is useful for components which want to parse history files and get info. These components can directly use/leverage "Rumen's parsing of history files across hadoop releases" and get history info in a consistent way for further analysis/processing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3597) Provide a way to access other
info of history file from Rumentool
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13175714#comment-13175714 ]
Hudson commented on MAPREDUCE-3597:
-----------------------------------
Integrated in Hadoop-Hdfs-trunk #904 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/904/])
MAPREDUCE-3597. [Rumen] Rumen should provide APIs to access all the job-history related information.
amarrk : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1222695
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/tools/rumen/TestRumenJobTraces.java
* /hadoop/common/trunk/hadoop-mapreduce-project/src/test/tools/data/rumen/small-trace-test/counters-test-trace.json.gz
* /hadoop/common/trunk/hadoop-mapreduce-project/src/test/tools/data/rumen/small-trace-test/dispatch-sample-v20-jt-log.gz
* /hadoop/common/trunk/hadoop-mapreduce-project/src/test/tools/data/rumen/small-trace-test/dispatch-trace-output.json.gz
* /hadoop/common/trunk/hadoop-mapreduce-project/src/tools/org/apache/hadoop/tools/rumen/JobBuilder.java
* /hadoop/common/trunk/hadoop-mapreduce-project/src/tools/org/apache/hadoop/tools/rumen/JobHistoryUtils.java
* /hadoop/common/trunk/hadoop-mapreduce-project/src/tools/org/apache/hadoop/tools/rumen/LoggedJob.java
* /hadoop/common/trunk/hadoop-mapreduce-project/src/tools/org/apache/hadoop/tools/rumen/ParsedJob.java
* /hadoop/common/trunk/hadoop-mapreduce-project/src/tools/org/apache/hadoop/tools/rumen/ParsedTask.java
* /hadoop/common/trunk/hadoop-mapreduce-project/src/tools/org/apache/hadoop/tools/rumen/ParsedTaskAttempt.java
> Provide a way to access other info of history file from Rumentool
> -----------------------------------------------------------------
>
> Key: MAPREDUCE-3597
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3597
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: tools/rumen
> Affects Versions: 0.24.0
> Reporter: Ravi Gummadi
> Assignee: Ravi Gummadi
> Fix For: 0.24.0
>
> Attachments: 3597.v0.patch, 3597.v1.patch
>
>
> As the trace file generated by Rumen TraceBuilder is skipping some of the info like job counters, task counters, etc. we need a way to access "other info available in history file which is not dumped to trace file". This is useful for components which want to parse history files and get info. These components can directly use/leverage "Rumen's parsing of history files across hadoop releases" and get history info in a consistent way for further analysis/processing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3597) Provide a way to access other
info of history file from Rumentool
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13190033#comment-13190033 ]
Hudson commented on MAPREDUCE-3597:
-----------------------------------
Integrated in Hadoop-Mapreduce-0.23-Commit #408 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/408/])
Merge -r 1222694:1222695 from trunk to branch. FIXES: MAPREDUCE-3597
tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1234086
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/tools/rumen/TestRumenJobTraces.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/tools/data/rumen/small-trace-test/counters-test-trace.json.gz
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/tools/data/rumen/small-trace-test/dispatch-sample-v20-jt-log.gz
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/tools/data/rumen/small-trace-test/dispatch-trace-output.json.gz
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/tools/org/apache/hadoop/tools/rumen/JobBuilder.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/tools/org/apache/hadoop/tools/rumen/JobHistoryUtils.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/tools/org/apache/hadoop/tools/rumen/LoggedJob.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/tools/org/apache/hadoop/tools/rumen/ParsedJob.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/tools/org/apache/hadoop/tools/rumen/ParsedTask.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/tools/org/apache/hadoop/tools/rumen/ParsedTaskAttempt.java
> Provide a way to access other info of history file from Rumentool
> -----------------------------------------------------------------
>
> Key: MAPREDUCE-3597
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3597
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: tools/rumen
> Affects Versions: 0.24.0
> Reporter: Ravi Gummadi
> Assignee: Ravi Gummadi
> Fix For: 0.24.0
>
> Attachments: 3597.branch-1.v1.patch, 3597.branch-1.v2.patch, 3597.v0.patch, 3597.v1.patch, MAPREDUCE-3597_branch-0.23.patch
>
>
> As the trace file generated by Rumen TraceBuilder is skipping some of the info like job counters, task counters, etc. we need a way to access "other info available in history file which is not dumped to trace file". This is useful for components which want to parse history files and get info. These components can directly use/leverage "Rumen's parsing of history files across hadoop releases" and get history info in a consistent way for further analysis/processing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3597) Provide a way to access other
info of history file from Rumentool
Posted by "Ravi Gummadi (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ravi Gummadi updated MAPREDUCE-3597:
------------------------------------
Status: Open (was: Patch Available)
> Provide a way to access other info of history file from Rumentool
> -----------------------------------------------------------------
>
> Key: MAPREDUCE-3597
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3597
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: tools/rumen
> Affects Versions: 0.24.0
> Reporter: Ravi Gummadi
> Assignee: Ravi Gummadi
> Fix For: 0.24.0
>
> Attachments: 3597.v0.patch, 3597.v1.patch
>
>
> As the trace file generated by Rumen TraceBuilder is skipping some of the info like job counters, task counters, etc. we need a way to access "other info available in history file which is not dumped to trace file". This is useful for components which want to parse history files and get info. These components can directly use/leverage "Rumen's parsing of history files across hadoop releases" and get history info in a consistent way for further analysis/processing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3597) Provide a way to access other
info of history file from Rumentool
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13190027#comment-13190027 ]
Hudson commented on MAPREDUCE-3597:
-----------------------------------
Integrated in Hadoop-Hdfs-0.23-Commit #385 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/385/])
Merge -r 1222694:1222695 from trunk to branch. FIXES: MAPREDUCE-3597
tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1234086
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/tools/rumen/TestRumenJobTraces.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/tools/data/rumen/small-trace-test/counters-test-trace.json.gz
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/tools/data/rumen/small-trace-test/dispatch-sample-v20-jt-log.gz
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/tools/data/rumen/small-trace-test/dispatch-trace-output.json.gz
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/tools/org/apache/hadoop/tools/rumen/JobBuilder.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/tools/org/apache/hadoop/tools/rumen/JobHistoryUtils.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/tools/org/apache/hadoop/tools/rumen/LoggedJob.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/tools/org/apache/hadoop/tools/rumen/ParsedJob.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/tools/org/apache/hadoop/tools/rumen/ParsedTask.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/tools/org/apache/hadoop/tools/rumen/ParsedTaskAttempt.java
> Provide a way to access other info of history file from Rumentool
> -----------------------------------------------------------------
>
> Key: MAPREDUCE-3597
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3597
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: tools/rumen
> Affects Versions: 0.24.0
> Reporter: Ravi Gummadi
> Assignee: Ravi Gummadi
> Fix For: 0.24.0
>
> Attachments: 3597.branch-1.v1.patch, 3597.branch-1.v2.patch, 3597.v0.patch, 3597.v1.patch, MAPREDUCE-3597_branch-0.23.patch
>
>
> As the trace file generated by Rumen TraceBuilder is skipping some of the info like job counters, task counters, etc. we need a way to access "other info available in history file which is not dumped to trace file". This is useful for components which want to parse history files and get info. These components can directly use/leverage "Rumen's parsing of history files across hadoop releases" and get history info in a consistent way for further analysis/processing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3597) Provide a way to access other
info of history file from Rumentool
Posted by "Ravi Gummadi (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13196831#comment-13196831 ]
Ravi Gummadi commented on MAPREDUCE-3597:
-----------------------------------------
I just committed this to branch-1.
> Provide a way to access other info of history file from Rumentool
> -----------------------------------------------------------------
>
> Key: MAPREDUCE-3597
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3597
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: tools/rumen
> Affects Versions: 0.24.0
> Reporter: Ravi Gummadi
> Assignee: Ravi Gummadi
> Fix For: 0.24.0
>
> Attachments: 3597.branch-0.23.patch, 3597.branch-1.v1.patch, 3597.branch-1.v2.1.patch, 3597.branch-1.v2.2.patch, 3597.branch-1.v2.patch, 3597.v0.patch, 3597.v1.patch, MAPREDUCE-3597_branch-0.23.patch
>
>
> As the trace file generated by Rumen TraceBuilder is skipping some of the info like job counters, task counters, etc. we need a way to access "other info available in history file which is not dumped to trace file". This is useful for components which want to parse history files and get info. These components can directly use/leverage "Rumen's parsing of history files across hadoop releases" and get history info in a consistent way for further analysis/processing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira