You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Hong Tang (JIRA)" <ji...@apache.org> on 2009/09/09 23:53:57 UTC
[jira] Created: (MAPREDUCE-966) Rumen interface improvement
Rumen interface improvement
---------------------------
Key: MAPREDUCE-966
URL: https://issues.apache.org/jira/browse/MAPREDUCE-966
Project: Hadoop Map/Reduce
Issue Type: Improvement
Affects Versions: 0.21.0
Reporter: Hong Tang
Assignee: Hong Tang
Rumen could expose a cleaner interface to simplify the integration with other tools.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-966) Rumen interface improvement
Posted by "Hong Tang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hong Tang updated MAPREDUCE-966:
--------------------------------
Status: Patch Available (was: Open)
> Rumen interface improvement
> ---------------------------
>
> Key: MAPREDUCE-966
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-966
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Affects Versions: 0.21.0
> Reporter: Hong Tang
> Assignee: Hong Tang
> Attachments: mapreduce-966-20090910-3.patch
>
>
> Rumen could expose a cleaner interface to simplify the integration with other tools.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-966) Rumen interface improvement
Posted by "Hong Tang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hong Tang updated MAPREDUCE-966:
--------------------------------
Status: Patch Available (was: Open)
> Rumen interface improvement
> ---------------------------
>
> Key: MAPREDUCE-966
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-966
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Affects Versions: 0.21.0
> Reporter: Hong Tang
> Assignee: Hong Tang
> Attachments: mapreduce-966-20090910-3.patch, mapreduce-966-20090910-4.patch
>
>
> Rumen could expose a cleaner interface to simplify the integration with other tools.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-966) Rumen interface improvement
Posted by "Hong Tang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12753309#action_12753309 ]
Hong Tang commented on MAPREDUCE-966:
-------------------------------------
Proposed changes:
- Isolate the tools dependent on Rumen to three simple interfaces:
## JobStory (describing a MapReduce job).
## ClusterStory (describing the cluster setup and topology etc)
## JobStoryProducer that produces a sequence of jobs.
Accordingly, ZombieJob adapts a LoggedJob to JobStory, and ZombieCluster adapts LoggedNetworkTopology to ClusterStory (indirectly through AbstractClusterStory). Finally ZombieJobProducer reads rumen traces and produces a sequence of JobStory instances.
- Encapsulate the logic of JSON parsing within Rumen and remove the Parser class. Two reader classes are added to parse json encoded LoggedJob and LoggedNetworkTopology (JobTraceReader and ClusterTopologyReader). No throw of Json-specific exceptions from the interface.
- Better sanity check in ZombieJob, and fill in made-up data if the source data are missing or invalid.
> Rumen interface improvement
> ---------------------------
>
> Key: MAPREDUCE-966
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-966
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Affects Versions: 0.21.0
> Reporter: Hong Tang
> Assignee: Hong Tang
>
> Rumen could expose a cleaner interface to simplify the integration with other tools.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-966) Rumen interface improvement
Posted by "Hong Tang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12753943#action_12753943 ]
Hong Tang commented on MAPREDUCE-966:
-------------------------------------
The attached patch implements the changes proposed above. During the process, we also fixed a few minor issues:
- Changed the usage of java io API (File etc) to hadoop Path, FileSystem, Configuration.
- Upgraded the tests to junit 4.
- Use the newly added JsonObjectMapperParser to replace customized json parsing in TestRumenJobTraces.
- Replaced the usage of Vector to List in unit tests.
- Fixed an NPE bug in HadoopLogAnalyzer.
- Added an API getOutcome() in JobStory.
- Fixed a bug in ParsedHost where it fails to parse rack names contain non-digital characters.
- Fixed a bug where ZombieJob.getTaskInfo() returns uninitialized TaskInfo objects.
- Adapt the usage of newly added ClusterStory in ZombieJob.
> Rumen interface improvement
> ---------------------------
>
> Key: MAPREDUCE-966
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-966
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Affects Versions: 0.21.0
> Reporter: Hong Tang
> Assignee: Hong Tang
> Attachments: mapreduce-966-20090910-3.patch
>
>
> Rumen could expose a cleaner interface to simplify the integration with other tools.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-966) Rumen interface improvement
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas updated MAPREDUCE-966:
------------------------------------
Resolution: Fixed
Fix Version/s: 0.21.0
Hadoop Flags: [Reviewed]
Status: Resolved (was: Patch Available)
+1
I committed this. Thanks, Hong!
> Rumen interface improvement
> ---------------------------
>
> Key: MAPREDUCE-966
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-966
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Affects Versions: 0.21.0
> Reporter: Hong Tang
> Assignee: Hong Tang
> Fix For: 0.21.0
>
> Attachments: mapreduce-966-20090910-3.patch, mapreduce-966-20090910-4.patch
>
>
> Rumen could expose a cleaner interface to simplify the integration with other tools.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-966) Rumen interface improvement
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12754504#action_12754504 ]
Hudson commented on MAPREDUCE-966:
----------------------------------
Integrated in Hadoop-Mapreduce-trunk-Commit #32 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/32/])
. Modify Rumen to clean up interfaces and simplify integration
with other tools. Contributed by Hong Tang
> Rumen interface improvement
> ---------------------------
>
> Key: MAPREDUCE-966
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-966
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Affects Versions: 0.21.0
> Reporter: Hong Tang
> Assignee: Hong Tang
> Fix For: 0.21.0
>
> Attachments: mapreduce-966-20090910-3.patch, mapreduce-966-20090910-4.patch
>
>
> Rumen could expose a cleaner interface to simplify the integration with other tools.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-966) Rumen interface improvement
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12753991#action_12753991 ]
Hadoop QA commented on MAPREDUCE-966:
-------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12419252/mapreduce-966-20090910-3.patch
against trunk revision 813585.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 15 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
-1 findbugs. The patch appears to introduce 2 new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/60/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/60/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/60/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/60/console
This message is automatically generated.
> Rumen interface improvement
> ---------------------------
>
> Key: MAPREDUCE-966
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-966
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Affects Versions: 0.21.0
> Reporter: Hong Tang
> Assignee: Hong Tang
> Attachments: mapreduce-966-20090910-3.patch
>
>
> Rumen could expose a cleaner interface to simplify the integration with other tools.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-966) Rumen interface improvement
Posted by "Hong Tang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hong Tang updated MAPREDUCE-966:
--------------------------------
Status: Open (was: Patch Available)
> Rumen interface improvement
> ---------------------------
>
> Key: MAPREDUCE-966
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-966
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Affects Versions: 0.21.0
> Reporter: Hong Tang
> Assignee: Hong Tang
> Attachments: mapreduce-966-20090910-3.patch, mapreduce-966-20090910-4.patch
>
>
> Rumen could expose a cleaner interface to simplify the integration with other tools.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-966) Rumen interface improvement
Posted by "Hong Tang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hong Tang updated MAPREDUCE-966:
--------------------------------
Attachment: mapreduce-966-20090910-3.patch
> Rumen interface improvement
> ---------------------------
>
> Key: MAPREDUCE-966
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-966
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Affects Versions: 0.21.0
> Reporter: Hong Tang
> Assignee: Hong Tang
> Attachments: mapreduce-966-20090910-3.patch
>
>
> Rumen could expose a cleaner interface to simplify the integration with other tools.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-966) Rumen interface improvement
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12754070#action_12754070 ]
Hadoop QA commented on MAPREDUCE-966:
-------------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12419278/mapreduce-966-20090910-4.patch
against trunk revision 813660.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 15 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/62/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/62/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/62/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/62/console
This message is automatically generated.
> Rumen interface improvement
> ---------------------------
>
> Key: MAPREDUCE-966
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-966
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Affects Versions: 0.21.0
> Reporter: Hong Tang
> Assignee: Hong Tang
> Attachments: mapreduce-966-20090910-3.patch, mapreduce-966-20090910-4.patch
>
>
> Rumen could expose a cleaner interface to simplify the integration with other tools.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-966) Rumen interface improvement
Posted by "Amar Kamat (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amar Kamat updated MAPREDUCE-966:
---------------------------------
Component/s: tools/rumen
> Rumen interface improvement
> ---------------------------
>
> Key: MAPREDUCE-966
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-966
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: tools/rumen
> Affects Versions: 0.21.0
> Reporter: Hong Tang
> Assignee: Hong Tang
> Fix For: 0.21.0
>
> Attachments: mapreduce-966-20090910-3.patch, mapreduce-966-20090910-4.patch
>
>
> Rumen could expose a cleaner interface to simplify the integration with other tools.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-966) Rumen interface improvement
Posted by "Hong Tang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hong Tang updated MAPREDUCE-966:
--------------------------------
Attachment: mapreduce-966-20090910-4.patch
Patch addresses the findbugs warnings.
> Rumen interface improvement
> ---------------------------
>
> Key: MAPREDUCE-966
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-966
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Affects Versions: 0.21.0
> Reporter: Hong Tang
> Assignee: Hong Tang
> Attachments: mapreduce-966-20090910-3.patch, mapreduce-966-20090910-4.patch
>
>
> Rumen could expose a cleaner interface to simplify the integration with other tools.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.