You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Robert Joseph Evans (Created) (JIRA)" <ji...@apache.org> on 2012/03/23 16:01:27 UTC
[jira] [Created] (MAPREDUCE-4059) The history server should have a
separate pluggable storage/query interface
The history server should have a separate pluggable storage/query interface
---------------------------------------------------------------------------
Key: MAPREDUCE-4059
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4059
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: mrv2
Affects Versions: 0.24.0, 0.23.3
Reporter: Robert Joseph Evans
Assignee: Robert Joseph Evans
The history server currently caches all parsed jobs in RAM. These jobs can be very large because of counters. It would be nice to have a pluggable interface for the cacheing and querying of the cached data so that we can play around with different implementations. Also just for cleanness of the code it would be nice to split the very large JobHistoryServer.java into a few smaller ones that are more understandable and readable.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4059) The history server should have a
separate pluggable storage/query interface
Posted by "Thomas Graves (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Thomas Graves updated MAPREDUCE-4059:
-------------------------------------
Resolution: Fixed
Fix Version/s: 0.23.3
Status: Resolved (was: Patch Available)
+1. Thanks Bobby!
I've committed this to trunk, branch-2, and branch-0.23
> The history server should have a separate pluggable storage/query interface
> ---------------------------------------------------------------------------
>
> Key: MAPREDUCE-4059
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4059
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
> Affects Versions: 0.24.0, 0.23.3
> Reporter: Robert Joseph Evans
> Assignee: Robert Joseph Evans
> Fix For: 0.23.3
>
> Attachments: MR-4059.txt, MR-4059.txt, MR-4059.txt, MR-4059.txt, MR-4059.txt, MR-4059.txt, MR-4059.txt
>
>
> The history server currently caches all parsed jobs in RAM. These jobs can be very large because of counters. It would be nice to have a pluggable interface for the cacheing and querying of the cached data so that we can play around with different implementations. Also just for cleanness of the code it would be nice to split the very large JobHistoryServer.java into a few smaller ones that are more understandable and readable.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4059) The history server should have
a separate pluggable storage/query interface
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13245547#comment-13245547 ]
Hadoop QA commented on MAPREDUCE-4059:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12521166/MR-4059.txt
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 21 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests:
org.apache.hadoop.mapreduce.v2.TestMiniMRProxyUser
org.apache.hadoop.mapreduce.v2.TestSpeculativeExecution
org.apache.hadoop.mapred.TestSpecialCharactersInOutputPath
org.apache.hadoop.mapreduce.v2.TestRMNMInfo
org.apache.hadoop.mapred.TestReduceFetchFromPartialMem
org.apache.hadoop.mapred.TestLazyOutput
org.apache.hadoop.mapreduce.TestChild
org.apache.hadoop.mapred.TestReduceFetch
org.apache.hadoop.mapred.TestMiniMRBringup
org.apache.hadoop.mapred.TestMiniMRClientCluster
org.apache.hadoop.mapreduce.lib.output.TestJobOutputCommitter
org.apache.hadoop.mapreduce.v2.TestMROldApiJobs
org.apache.hadoop.mapreduce.v2.TestNonExistentJob
org.apache.hadoop.mapred.TestMiniMRChildTask
org.apache.hadoop.mapred.TestJobCounters
org.apache.hadoop.mapreduce.v2.TestUberAM
org.apache.hadoop.conf.TestNoDefaultsJobConf
org.apache.hadoop.mapred.TestJobSysDirWithDFS
org.apache.hadoop.mapreduce.v2.TestMRJobs
org.apache.hadoop.mapred.TestClientRedirect
org.apache.hadoop.mapred.TestJobCleanup
org.apache.hadoop.mapreduce.TestMapReduceLazyOutput
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2135//testReport/
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2135//console
This message is automatically generated.
> The history server should have a separate pluggable storage/query interface
> ---------------------------------------------------------------------------
>
> Key: MAPREDUCE-4059
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4059
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
> Affects Versions: 0.24.0, 0.23.3
> Reporter: Robert Joseph Evans
> Assignee: Robert Joseph Evans
> Attachments: MR-4059.txt, MR-4059.txt, MR-4059.txt, MR-4059.txt
>
>
> The history server currently caches all parsed jobs in RAM. These jobs can be very large because of counters. It would be nice to have a pluggable interface for the cacheing and querying of the cached data so that we can play around with different implementations. Also just for cleanness of the code it would be nice to split the very large JobHistoryServer.java into a few smaller ones that are more understandable and readable.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4059) The history server should have a
separate pluggable storage/query interface
Posted by "Robert Joseph Evans (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Joseph Evans updated MAPREDUCE-4059:
-------------------------------------------
Status: Patch Available (was: Open)
> The history server should have a separate pluggable storage/query interface
> ---------------------------------------------------------------------------
>
> Key: MAPREDUCE-4059
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4059
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
> Affects Versions: 0.24.0, 0.23.3
> Reporter: Robert Joseph Evans
> Assignee: Robert Joseph Evans
> Attachments: MR-4059.txt, MR-4059.txt
>
>
> The history server currently caches all parsed jobs in RAM. These jobs can be very large because of counters. It would be nice to have a pluggable interface for the cacheing and querying of the cached data so that we can play around with different implementations. Also just for cleanness of the code it would be nice to split the very large JobHistoryServer.java into a few smaller ones that are more understandable and readable.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4059) The history server should have a
separate pluggable storage/query interface
Posted by "Robert Joseph Evans (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Joseph Evans updated MAPREDUCE-4059:
-------------------------------------------
Status: Open (was: Patch Available)
Canceling patch to address Tom's comments. I will probably file a new JIRA to think more deeply about what the interface should look like.
> The history server should have a separate pluggable storage/query interface
> ---------------------------------------------------------------------------
>
> Key: MAPREDUCE-4059
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4059
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
> Affects Versions: 0.24.0, 0.23.3
> Reporter: Robert Joseph Evans
> Assignee: Robert Joseph Evans
> Attachments: MR-4059.txt, MR-4059.txt, MR-4059.txt, MR-4059.txt, MR-4059.txt, MR-4059.txt
>
>
> The history server currently caches all parsed jobs in RAM. These jobs can be very large because of counters. It would be nice to have a pluggable interface for the cacheing and querying of the cached data so that we can play around with different implementations. Also just for cleanness of the code it would be nice to split the very large JobHistoryServer.java into a few smaller ones that are more understandable and readable.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4059) The history server should have
a separate pluggable storage/query interface
Posted by "Robert Joseph Evans (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13244218#comment-13244218 ]
Robert Joseph Evans commented on MAPREDUCE-4059:
------------------------------------------------
Yes there is some overlap between this and MAPREDUCE-3971. MAPREDUCE-3971 is to update the web service to support this functionality and anything else that is necessary. This JIRA is to provide an API layer that will abstract away the caching and query of the jobs from how that data is used, whether it be a web service or a web GUI or an RPC call. The idea is that eventually I can experiment with different back-ends that will be better at storing and querying the data than the simple LRU cache that we have now.
I did not add in the offset to the web service because there is still work that needs to be done to update documentation, verify that it works correctly, add in tests, etc. If you would prefer I can remove the offset parameter and just leave the count parameter, and then add it back in as part of MAPREDUCE-3971.
> The history server should have a separate pluggable storage/query interface
> ---------------------------------------------------------------------------
>
> Key: MAPREDUCE-4059
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4059
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
> Affects Versions: 0.24.0, 0.23.3
> Reporter: Robert Joseph Evans
> Assignee: Robert Joseph Evans
> Attachments: MR-4059.txt, MR-4059.txt
>
>
> The history server currently caches all parsed jobs in RAM. These jobs can be very large because of counters. It would be nice to have a pluggable interface for the cacheing and querying of the cached data so that we can play around with different implementations. Also just for cleanness of the code it would be nice to split the very large JobHistoryServer.java into a few smaller ones that are more understandable and readable.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4059) The history server should have a
separate pluggable storage/query interface
Posted by "Robert Joseph Evans (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Joseph Evans updated MAPREDUCE-4059:
-------------------------------------------
Attachment: MR-4059.txt
One more update. There was an issue where the job state was not being set in the job report for a partial job. This resulted in the query using just partial jobs caused it to fail.
> The history server should have a separate pluggable storage/query interface
> ---------------------------------------------------------------------------
>
> Key: MAPREDUCE-4059
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4059
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
> Affects Versions: 0.24.0, 0.23.3
> Reporter: Robert Joseph Evans
> Assignee: Robert Joseph Evans
> Attachments: MR-4059.txt, MR-4059.txt, MR-4059.txt, MR-4059.txt
>
>
> The history server currently caches all parsed jobs in RAM. These jobs can be very large because of counters. It would be nice to have a pluggable interface for the cacheing and querying of the cached data so that we can play around with different implementations. Also just for cleanness of the code it would be nice to split the very large JobHistoryServer.java into a few smaller ones that are more understandable and readable.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4059) The history server should have
a separate pluggable storage/query interface
Posted by "Robert Joseph Evans (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13249905#comment-13249905 ]
Robert Joseph Evans commented on MAPREDUCE-4059:
------------------------------------------------
{quote}org.apache.hadoop.yarn.server.resourcemanager.TestClientRMService
org.apache.hadoop.yarn.server.resourcemanager.resourcetracker.TestNMExpiry
org.apache.hadoop.yarn.server.resourcemanager.TestAMAuthorization
org.apache.hadoop.yarn.server.resourcemanager.TestApplicationACLs{quote}
These tests have been failing randomly and are not related to the patch.
> The history server should have a separate pluggable storage/query interface
> ---------------------------------------------------------------------------
>
> Key: MAPREDUCE-4059
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4059
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
> Affects Versions: 0.24.0, 0.23.3
> Reporter: Robert Joseph Evans
> Assignee: Robert Joseph Evans
> Attachments: MR-4059.txt, MR-4059.txt, MR-4059.txt, MR-4059.txt, MR-4059.txt, MR-4059.txt
>
>
> The history server currently caches all parsed jobs in RAM. These jobs can be very large because of counters. It would be nice to have a pluggable interface for the cacheing and querying of the cached data so that we can play around with different implementations. Also just for cleanness of the code it would be nice to split the very large JobHistoryServer.java into a few smaller ones that are more understandable and readable.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4059) The history server should have
a separate pluggable storage/query interface
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13237011#comment-13237011 ]
Hadoop QA commented on MAPREDUCE-4059:
--------------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12519694/MR-4059.txt
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 21 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed unit tests in .
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2094//testReport/
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2094//console
This message is automatically generated.
> The history server should have a separate pluggable storage/query interface
> ---------------------------------------------------------------------------
>
> Key: MAPREDUCE-4059
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4059
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
> Affects Versions: 0.24.0, 0.23.3
> Reporter: Robert Joseph Evans
> Assignee: Robert Joseph Evans
> Attachments: MR-4059.txt, MR-4059.txt
>
>
> The history server currently caches all parsed jobs in RAM. These jobs can be very large because of counters. It would be nice to have a pluggable interface for the cacheing and querying of the cached data so that we can play around with different implementations. Also just for cleanness of the code it would be nice to split the very large JobHistoryServer.java into a few smaller ones that are more understandable and readable.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4059) The history server should have
a separate pluggable storage/query interface
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13250944#comment-13250944 ]
Hudson commented on MAPREDUCE-4059:
-----------------------------------
Integrated in Hadoop-Mapreduce-trunk-Commit #2053 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2053/])
MAPREDUCE-4059. The history server should have a separate pluggable storage/query interface. (Robert Evans via tgraves). (Revision 1311896)
Result = SUCCESS
tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1311896
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/jobhistory/JHAdminConfig.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/jobhistory/JobHistoryUtils.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/CachedHistoryStorage.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/HistoryContext.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/HistoryFileManager.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/HistoryStorage.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/JobHistory.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/PartialJob.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/webapp/HsWebApp.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/webapp/HsWebServices.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/TestJobHistoryParsing.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/webapp/TestHsWebServices.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/webapp/TestHsWebServicesAttempts.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/webapp/TestHsWebServicesJobConf.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/webapp/TestHsWebServicesJobs.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/webapp/TestHsWebServicesJobsQuery.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/webapp/TestHsWebServicesTasks.java
> The history server should have a separate pluggable storage/query interface
> ---------------------------------------------------------------------------
>
> Key: MAPREDUCE-4059
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4059
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
> Affects Versions: 0.24.0, 0.23.3
> Reporter: Robert Joseph Evans
> Assignee: Robert Joseph Evans
> Fix For: 0.23.3
>
> Attachments: MR-4059.txt, MR-4059.txt, MR-4059.txt, MR-4059.txt, MR-4059.txt, MR-4059.txt, MR-4059.txt
>
>
> The history server currently caches all parsed jobs in RAM. These jobs can be very large because of counters. It would be nice to have a pluggable interface for the cacheing and querying of the cached data so that we can play around with different implementations. Also just for cleanness of the code it would be nice to split the very large JobHistoryServer.java into a few smaller ones that are more understandable and readable.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4059) The history server should have a
separate pluggable storage/query interface
Posted by "Robert Joseph Evans (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Joseph Evans updated MAPREDUCE-4059:
-------------------------------------------
Status: Patch Available (was: Open)
> The history server should have a separate pluggable storage/query interface
> ---------------------------------------------------------------------------
>
> Key: MAPREDUCE-4059
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4059
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
> Affects Versions: 0.24.0, 0.23.3
> Reporter: Robert Joseph Evans
> Assignee: Robert Joseph Evans
> Attachments: MR-4059.txt
>
>
> The history server currently caches all parsed jobs in RAM. These jobs can be very large because of counters. It would be nice to have a pluggable interface for the cacheing and querying of the cached data so that we can play around with different implementations. Also just for cleanness of the code it would be nice to split the very large JobHistoryServer.java into a few smaller ones that are more understandable and readable.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4059) The history server should have
a separate pluggable storage/query interface
Posted by "Robert Joseph Evans (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13242635#comment-13242635 ]
Robert Joseph Evans commented on MAPREDUCE-4059:
------------------------------------------------
Oh I should also remove my debugging log messages that are logged at error!!
> The history server should have a separate pluggable storage/query interface
> ---------------------------------------------------------------------------
>
> Key: MAPREDUCE-4059
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4059
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
> Affects Versions: 0.24.0, 0.23.3
> Reporter: Robert Joseph Evans
> Assignee: Robert Joseph Evans
> Attachments: MR-4059.txt, MR-4059.txt
>
>
> The history server currently caches all parsed jobs in RAM. These jobs can be very large because of counters. It would be nice to have a pluggable interface for the cacheing and querying of the cached data so that we can play around with different implementations. Also just for cleanness of the code it would be nice to split the very large JobHistoryServer.java into a few smaller ones that are more understandable and readable.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4059) The history server should have
a separate pluggable storage/query interface
Posted by "Robert Joseph Evans (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13236656#comment-13236656 ]
Robert Joseph Evans commented on MAPREDUCE-4059:
------------------------------------------------
I for got to add that I did not change any tests yet. The patch should be backwards compatible with all existing tests, and I would like some feedback on the API before I start trying to add in more tests.
> The history server should have a separate pluggable storage/query interface
> ---------------------------------------------------------------------------
>
> Key: MAPREDUCE-4059
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4059
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
> Affects Versions: 0.24.0, 0.23.3
> Reporter: Robert Joseph Evans
> Assignee: Robert Joseph Evans
> Attachments: MR-4059.txt
>
>
> The history server currently caches all parsed jobs in RAM. These jobs can be very large because of counters. It would be nice to have a pluggable interface for the cacheing and querying of the cached data so that we can play around with different implementations. Also just for cleanness of the code it would be nice to split the very large JobHistoryServer.java into a few smaller ones that are more understandable and readable.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4059) The history server should have a
separate pluggable storage/query interface
Posted by "Robert Joseph Evans (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Joseph Evans updated MAPREDUCE-4059:
-------------------------------------------
Attachment: MR-4059.txt
changes from .equals to ==
> The history server should have a separate pluggable storage/query interface
> ---------------------------------------------------------------------------
>
> Key: MAPREDUCE-4059
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4059
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
> Affects Versions: 0.24.0, 0.23.3
> Reporter: Robert Joseph Evans
> Assignee: Robert Joseph Evans
> Attachments: MR-4059.txt, MR-4059.txt, MR-4059.txt, MR-4059.txt, MR-4059.txt
>
>
> The history server currently caches all parsed jobs in RAM. These jobs can be very large because of counters. It would be nice to have a pluggable interface for the cacheing and querying of the cached data so that we can play around with different implementations. Also just for cleanness of the code it would be nice to split the very large JobHistoryServer.java into a few smaller ones that are more understandable and readable.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Closed] (MAPREDUCE-4059) The history server should have a
separate pluggable storage/query interface
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy closed MAPREDUCE-4059.
------------------------------------
> The history server should have a separate pluggable storage/query interface
> ---------------------------------------------------------------------------
>
> Key: MAPREDUCE-4059
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4059
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
> Affects Versions: 0.24.0, 0.23.3
> Reporter: Robert Joseph Evans
> Assignee: Robert Joseph Evans
> Fix For: 0.23.3, 2.0.2-alpha
>
> Attachments: MR-4059.txt, MR-4059.txt, MR-4059.txt, MR-4059.txt, MR-4059.txt, MR-4059.txt, MR-4059.txt
>
>
> The history server currently caches all parsed jobs in RAM. These jobs can be very large because of counters. It would be nice to have a pluggable interface for the cacheing and querying of the cached data so that we can play around with different implementations. Also just for cleanness of the code it would be nice to split the very large JobHistoryServer.java into a few smaller ones that are more understandable and readable.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4059) The history server should have
a separate pluggable storage/query interface
Posted by "Robert Joseph Evans (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13242632#comment-13242632 ]
Robert Joseph Evans commented on MAPREDUCE-4059:
------------------------------------------------
That is a great catch. I did totally miss the possibility of int overflow, and with count being negative. Will update the patch. Thanks.
> The history server should have a separate pluggable storage/query interface
> ---------------------------------------------------------------------------
>
> Key: MAPREDUCE-4059
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4059
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
> Affects Versions: 0.24.0, 0.23.3
> Reporter: Robert Joseph Evans
> Assignee: Robert Joseph Evans
> Attachments: MR-4059.txt, MR-4059.txt
>
>
> The history server currently caches all parsed jobs in RAM. These jobs can be very large because of counters. It would be nice to have a pluggable interface for the cacheing and querying of the cached data so that we can play around with different implementations. Also just for cleanness of the code it would be nice to split the very large JobHistoryServer.java into a few smaller ones that are more understandable and readable.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4059) The history server should have
a separate pluggable storage/query interface
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13236688#comment-13236688 ]
Hadoop QA commented on MAPREDUCE-4059:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12519635/MR-4059.txt
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
-1 findbugs. The patch appears to introduce 4 new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests:
org.apache.hadoop.mapreduce.v2.hs.webapp.TestHsWebServicesJobs
org.apache.hadoop.mapreduce.v2.hs.webapp.TestHsWebServicesTasks
org.apache.hadoop.mapreduce.v2.hs.webapp.TestHsWebServicesAttempts
org.apache.hadoop.mapreduce.v2.hs.webapp.TestHsWebServices
org.apache.hadoop.mapreduce.v2.hs.webapp.TestHsWebServicesJobConf
org.apache.hadoop.mapreduce.v2.hs.webapp.TestHsWebServicesJobsQuery
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2089//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2089//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-hs.html
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2089//console
This message is automatically generated.
> The history server should have a separate pluggable storage/query interface
> ---------------------------------------------------------------------------
>
> Key: MAPREDUCE-4059
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4059
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
> Affects Versions: 0.24.0, 0.23.3
> Reporter: Robert Joseph Evans
> Assignee: Robert Joseph Evans
> Attachments: MR-4059.txt
>
>
> The history server currently caches all parsed jobs in RAM. These jobs can be very large because of counters. It would be nice to have a pluggable interface for the cacheing and querying of the cached data so that we can play around with different implementations. Also just for cleanness of the code it would be nice to split the very large JobHistoryServer.java into a few smaller ones that are more understandable and readable.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4059) The history server should have a
separate pluggable storage/query interface
Posted by "Robert Joseph Evans (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Joseph Evans updated MAPREDUCE-4059:
-------------------------------------------
Status: Patch Available (was: Open)
> The history server should have a separate pluggable storage/query interface
> ---------------------------------------------------------------------------
>
> Key: MAPREDUCE-4059
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4059
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
> Affects Versions: 0.24.0, 0.23.3
> Reporter: Robert Joseph Evans
> Assignee: Robert Joseph Evans
> Attachments: MR-4059.txt, MR-4059.txt, MR-4059.txt, MR-4059.txt, MR-4059.txt, MR-4059.txt
>
>
> The history server currently caches all parsed jobs in RAM. These jobs can be very large because of counters. It would be nice to have a pluggable interface for the cacheing and querying of the cached data so that we can play around with different implementations. Also just for cleanness of the code it would be nice to split the very large JobHistoryServer.java into a few smaller ones that are more understandable and readable.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4059) The history server should have
a separate pluggable storage/query interface
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13250793#comment-13250793 ]
Hadoop QA commented on MAPREDUCE-4059:
--------------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12522112/MR-4059.txt
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 7 new or modified test files.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed unit tests in .
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2183//testReport/
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2183//console
This message is automatically generated.
> The history server should have a separate pluggable storage/query interface
> ---------------------------------------------------------------------------
>
> Key: MAPREDUCE-4059
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4059
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
> Affects Versions: 0.24.0, 0.23.3
> Reporter: Robert Joseph Evans
> Assignee: Robert Joseph Evans
> Attachments: MR-4059.txt, MR-4059.txt, MR-4059.txt, MR-4059.txt, MR-4059.txt, MR-4059.txt, MR-4059.txt
>
>
> The history server currently caches all parsed jobs in RAM. These jobs can be very large because of counters. It would be nice to have a pluggable interface for the cacheing and querying of the cached data so that we can play around with different implementations. Also just for cleanness of the code it would be nice to split the very large JobHistoryServer.java into a few smaller ones that are more understandable and readable.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4059) The history server should have
a separate pluggable storage/query interface
Posted by "Bhallamudi Venkata Siva Kamesh (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13242600#comment-13242600 ]
Bhallamudi Venkata Siva Kamesh commented on MAPREDUCE-4059:
-----------------------------------------------------------
Hi Robert,
I just have gone through a portion of the patch and have few comments/doubts on the patch.
1. {code:title=CachedHistoryStorage.java|borderStyle=solid}
if(offset == null || offset < 0) offset = 0l;
if(count == null) count = Long.MAX_VALUE;
long at = 0;
long end = offset + count - 1;
LOG.error("Looking for entries starting at "+offset+" with a length of "+count+" so ending at "+end);
for (Job job : jobs) {
LOG.error("Looking at job END: "+at+" <= "+end);
if(at > end) {
break;
}
{code}
Suppose say, in the above code offset is set as 100 and count is not initialized, so count will be initialized as Long.MAX_VALUE.
If so, *end* may get negative value (because of integer overflow) and hence for loop will be exited in the first iteration itself.
But I think, we should return entries starting from 100th entry.
Am I missing anything here?
2. I think it is better to add sanity check like this in CachedHistoryStorage#getPartialJobs()
{noformat}
if(offset > jobs.size()) {
return allJobs;
}
{noformat}
> The history server should have a separate pluggable storage/query interface
> ---------------------------------------------------------------------------
>
> Key: MAPREDUCE-4059
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4059
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
> Affects Versions: 0.24.0, 0.23.3
> Reporter: Robert Joseph Evans
> Assignee: Robert Joseph Evans
> Attachments: MR-4059.txt, MR-4059.txt
>
>
> The history server currently caches all parsed jobs in RAM. These jobs can be very large because of counters. It would be nice to have a pluggable interface for the cacheing and querying of the cached data so that we can play around with different implementations. Also just for cleanness of the code it would be nice to split the very large JobHistoryServer.java into a few smaller ones that are more understandable and readable.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4059) The history server should have
a separate pluggable storage/query interface
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13251512#comment-13251512 ]
Hudson commented on MAPREDUCE-4059:
-----------------------------------
Integrated in Hadoop-Hdfs-0.23-Build #224 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/224/])
merge -r 1311896:1311897 from branch-2. FIXES: MAPREDUCE-4059 (Revision 1311898)
Result = SUCCESS
tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1311898
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/jobhistory/JHAdminConfig.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/jobhistory/JobHistoryUtils.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/CachedHistoryStorage.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/HistoryContext.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/HistoryFileManager.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/HistoryStorage.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/JobHistory.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/PartialJob.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/webapp/HsWebApp.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/webapp/HsWebServices.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/TestJobHistoryParsing.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/webapp/TestHsWebServices.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/webapp/TestHsWebServicesAttempts.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/webapp/TestHsWebServicesJobConf.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/webapp/TestHsWebServicesJobs.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/webapp/TestHsWebServicesJobsQuery.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/webapp/TestHsWebServicesTasks.java
> The history server should have a separate pluggable storage/query interface
> ---------------------------------------------------------------------------
>
> Key: MAPREDUCE-4059
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4059
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
> Affects Versions: 0.24.0, 0.23.3
> Reporter: Robert Joseph Evans
> Assignee: Robert Joseph Evans
> Fix For: 0.23.3
>
> Attachments: MR-4059.txt, MR-4059.txt, MR-4059.txt, MR-4059.txt, MR-4059.txt, MR-4059.txt, MR-4059.txt
>
>
> The history server currently caches all parsed jobs in RAM. These jobs can be very large because of counters. It would be nice to have a pluggable interface for the cacheing and querying of the cached data so that we can play around with different implementations. Also just for cleanness of the code it would be nice to split the very large JobHistoryServer.java into a few smaller ones that are more understandable and readable.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4059) The history server should have a
separate pluggable storage/query interface
Posted by "Robert Joseph Evans (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Joseph Evans updated MAPREDUCE-4059:
-------------------------------------------
Attachment: MR-4059.txt
This patch does not change any of the code. I just had eclipse reformat some of the files so they fit the codeing guidelines.
I also filed MAPREDUCE-4131 to address the issue of the HistoryStorage interface.
> The history server should have a separate pluggable storage/query interface
> ---------------------------------------------------------------------------
>
> Key: MAPREDUCE-4059
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4059
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
> Affects Versions: 0.24.0, 0.23.3
> Reporter: Robert Joseph Evans
> Assignee: Robert Joseph Evans
> Attachments: MR-4059.txt, MR-4059.txt, MR-4059.txt, MR-4059.txt, MR-4059.txt, MR-4059.txt, MR-4059.txt
>
>
> The history server currently caches all parsed jobs in RAM. These jobs can be very large because of counters. It would be nice to have a pluggable interface for the cacheing and querying of the cached data so that we can play around with different implementations. Also just for cleanness of the code it would be nice to split the very large JobHistoryServer.java into a few smaller ones that are more understandable and readable.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4059) The history server should have
a separate pluggable storage/query interface
Posted by "Robert Joseph Evans (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13246263#comment-13246263 ]
Robert Joseph Evans commented on MAPREDUCE-4059:
------------------------------------------------
Yes it is better to use == instead of .equals when comparing enums. I was copying the behavior of the previous code that was part of the web service.
The HistoryFileManager manages History files. It is ideally the only one that can interact with them for thread safety reasons. These concurrency issues are addressed in a follow on JIRA MAPREDUCE-3972.
The data from the history files could be stored some place else in the future, but it will always start out being written to HDFS. It just may be transfered to another location after it is written to HDFS. Initially OtherStorages are likely to act like a cache and provide faster access to data already stored in HDFS. Once issues with that are worked out. And if the community feels comfortable with doing so it may change to storing data that is not backed by files in HDFS.
> The history server should have a separate pluggable storage/query interface
> ---------------------------------------------------------------------------
>
> Key: MAPREDUCE-4059
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4059
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
> Affects Versions: 0.24.0, 0.23.3
> Reporter: Robert Joseph Evans
> Assignee: Robert Joseph Evans
> Attachments: MR-4059.txt, MR-4059.txt, MR-4059.txt, MR-4059.txt
>
>
> The history server currently caches all parsed jobs in RAM. These jobs can be very large because of counters. It would be nice to have a pluggable interface for the cacheing and querying of the cached data so that we can play around with different implementations. Also just for cleanness of the code it would be nice to split the very large JobHistoryServer.java into a few smaller ones that are more understandable and readable.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4059) The history server should have
a separate pluggable storage/query interface
Posted by "Bhallamudi Venkata Siva Kamesh (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13243181#comment-13243181 ]
Bhallamudi Venkata Siva Kamesh commented on MAPREDUCE-4059:
-----------------------------------------------------------
Hi Robert,
Again I have some comments/doubts on the attached patch
There is an API
{noformat}JobsInfo getPartialJobs(Long offset, Long count, String user, String queue, Long sBegin, Long sEnd, Long fBegin, Long fEnd);{noformat}
which supports returning only a limited number of jobs (no of jobs = offset + count - 1). But [MAPREDUCE-3971|https://issues.apache.org/jira/browse/MAPREDUCE-3971] is also trying to solve the same issue. So are we solving [MAPREDUCE-3971|https://issues.apache.org/jira/browse/MAPREDUCE-3971] also as part of this issue or am I missing anything here?
Also I *think* it is better to add query param like
{noformat}@QueryParam("offset") String offset{noformat}
in HsWebServices#getJobs method to support offset. In the patch offset is hardcoded as 0.Or any reason for doing so?
Also I *think* it is better to use LinkedHashMap in CachedHistoryStorage#getAllPartialJobs and LinkedList in HistoryFileManager.getAllMetaInfo for reasons I metioned at [MAPREDUCE-3971|https://issues.apache.org/jira/browse/MAPREDUCE-3971?focusedCommentId=13226923&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13226923]. Do you see any problems with these?
> The history server should have a separate pluggable storage/query interface
> ---------------------------------------------------------------------------
>
> Key: MAPREDUCE-4059
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4059
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
> Affects Versions: 0.24.0, 0.23.3
> Reporter: Robert Joseph Evans
> Assignee: Robert Joseph Evans
> Attachments: MR-4059.txt, MR-4059.txt
>
>
> The history server currently caches all parsed jobs in RAM. These jobs can be very large because of counters. It would be nice to have a pluggable interface for the cacheing and querying of the cached data so that we can play around with different implementations. Also just for cleanness of the code it would be nice to split the very large JobHistoryServer.java into a few smaller ones that are more understandable and readable.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4059) The history server should have a
separate pluggable storage/query interface
Posted by "Robert Joseph Evans (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Joseph Evans updated MAPREDUCE-4059:
-------------------------------------------
Attachment: MR-4059.txt
Updating the patch. Upmerged to trunk and addressed dome of the comments so far. I have not touched The internal data structures just yet. I figure we can look at those as part of MAPREDUCE-3973 and MAPREDUCE-3971.
> The history server should have a separate pluggable storage/query interface
> ---------------------------------------------------------------------------
>
> Key: MAPREDUCE-4059
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4059
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
> Affects Versions: 0.24.0, 0.23.3
> Reporter: Robert Joseph Evans
> Assignee: Robert Joseph Evans
> Attachments: MR-4059.txt, MR-4059.txt, MR-4059.txt
>
>
> The history server currently caches all parsed jobs in RAM. These jobs can be very large because of counters. It would be nice to have a pluggable interface for the cacheing and querying of the cached data so that we can play around with different implementations. Also just for cleanness of the code it would be nice to split the very large JobHistoryServer.java into a few smaller ones that are more understandable and readable.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4059) The history server should have
a separate pluggable storage/query interface
Posted by "Robert Joseph Evans (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13245553#comment-13245553 ]
Robert Joseph Evans commented on MAPREDUCE-4059:
------------------------------------------------
Yes that would probably work too, but if someone does happen to put in a count that is close to LONG_MAX we can still run into this situation, so I would prefer to just leave the code as is.
> The history server should have a separate pluggable storage/query interface
> ---------------------------------------------------------------------------
>
> Key: MAPREDUCE-4059
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4059
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
> Affects Versions: 0.24.0, 0.23.3
> Reporter: Robert Joseph Evans
> Assignee: Robert Joseph Evans
> Attachments: MR-4059.txt, MR-4059.txt, MR-4059.txt, MR-4059.txt
>
>
> The history server currently caches all parsed jobs in RAM. These jobs can be very large because of counters. It would be nice to have a pluggable interface for the cacheing and querying of the cached data so that we can play around with different implementations. Also just for cleanness of the code it would be nice to split the very large JobHistoryServer.java into a few smaller ones that are more understandable and readable.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4059) The history server should have
a separate pluggable storage/query interface
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13250935#comment-13250935 ]
Hudson commented on MAPREDUCE-4059:
-----------------------------------
Integrated in Hadoop-Common-trunk-Commit #2040 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2040/])
MAPREDUCE-4059. The history server should have a separate pluggable storage/query interface. (Robert Evans via tgraves). (Revision 1311896)
Result = SUCCESS
tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1311896
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/jobhistory/JHAdminConfig.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/jobhistory/JobHistoryUtils.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/CachedHistoryStorage.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/HistoryContext.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/HistoryFileManager.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/HistoryStorage.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/JobHistory.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/PartialJob.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/webapp/HsWebApp.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/webapp/HsWebServices.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/TestJobHistoryParsing.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/webapp/TestHsWebServices.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/webapp/TestHsWebServicesAttempts.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/webapp/TestHsWebServicesJobConf.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/webapp/TestHsWebServicesJobs.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/webapp/TestHsWebServicesJobsQuery.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/webapp/TestHsWebServicesTasks.java
> The history server should have a separate pluggable storage/query interface
> ---------------------------------------------------------------------------
>
> Key: MAPREDUCE-4059
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4059
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
> Affects Versions: 0.24.0, 0.23.3
> Reporter: Robert Joseph Evans
> Assignee: Robert Joseph Evans
> Fix For: 0.23.3
>
> Attachments: MR-4059.txt, MR-4059.txt, MR-4059.txt, MR-4059.txt, MR-4059.txt, MR-4059.txt, MR-4059.txt
>
>
> The history server currently caches all parsed jobs in RAM. These jobs can be very large because of counters. It would be nice to have a pluggable interface for the cacheing and querying of the cached data so that we can play around with different implementations. Also just for cleanness of the code it would be nice to split the very large JobHistoryServer.java into a few smaller ones that are more understandable and readable.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4059) The history server should have
a separate pluggable storage/query interface
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13249895#comment-13249895 ]
Hadoop QA commented on MAPREDUCE-4059:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12521964/MR-4059.txt
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 7 new or modified test files.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests:
org.apache.hadoop.yarn.server.resourcemanager.TestClientRMService
org.apache.hadoop.yarn.server.resourcemanager.resourcetracker.TestNMExpiry
org.apache.hadoop.yarn.server.resourcemanager.TestAMAuthorization
org.apache.hadoop.yarn.server.resourcemanager.TestApplicationACLs
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2174//testReport/
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2174//console
This message is automatically generated.
> The history server should have a separate pluggable storage/query interface
> ---------------------------------------------------------------------------
>
> Key: MAPREDUCE-4059
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4059
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
> Affects Versions: 0.24.0, 0.23.3
> Reporter: Robert Joseph Evans
> Assignee: Robert Joseph Evans
> Attachments: MR-4059.txt, MR-4059.txt, MR-4059.txt, MR-4059.txt, MR-4059.txt, MR-4059.txt
>
>
> The history server currently caches all parsed jobs in RAM. These jobs can be very large because of counters. It would be nice to have a pluggable interface for the cacheing and querying of the cached data so that we can play around with different implementations. Also just for cleanness of the code it would be nice to split the very large JobHistoryServer.java into a few smaller ones that are more understandable and readable.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4059) The history server should have
a separate pluggable storage/query interface
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13246378#comment-13246378 ]
Hadoop QA commented on MAPREDUCE-4059:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12521311/MR-4059.txt
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 21 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests:
org.apache.hadoop.mapreduce.v2.TestMiniMRProxyUser
org.apache.hadoop.mapreduce.v2.TestSpeculativeExecution
org.apache.hadoop.mapred.TestSpecialCharactersInOutputPath
org.apache.hadoop.mapreduce.v2.TestRMNMInfo
org.apache.hadoop.mapred.TestReduceFetchFromPartialMem
org.apache.hadoop.mapred.TestLazyOutput
org.apache.hadoop.mapreduce.TestChild
org.apache.hadoop.mapred.TestReduceFetch
org.apache.hadoop.mapred.TestMiniMRBringup
org.apache.hadoop.mapred.TestMiniMRClientCluster
org.apache.hadoop.mapreduce.lib.output.TestJobOutputCommitter
org.apache.hadoop.mapreduce.v2.TestMROldApiJobs
org.apache.hadoop.mapreduce.v2.TestNonExistentJob
org.apache.hadoop.mapred.TestMiniMRChildTask
org.apache.hadoop.mapred.TestJobCounters
org.apache.hadoop.mapreduce.v2.TestUberAM
org.apache.hadoop.conf.TestNoDefaultsJobConf
org.apache.hadoop.mapred.TestJobSysDirWithDFS
org.apache.hadoop.mapreduce.v2.TestMRJobs
org.apache.hadoop.mapred.TestClientRedirect
org.apache.hadoop.mapred.TestJobCleanup
org.apache.hadoop.mapreduce.TestMapReduceLazyOutput
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2145//testReport/
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2145//console
This message is automatically generated.
> The history server should have a separate pluggable storage/query interface
> ---------------------------------------------------------------------------
>
> Key: MAPREDUCE-4059
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4059
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
> Affects Versions: 0.24.0, 0.23.3
> Reporter: Robert Joseph Evans
> Assignee: Robert Joseph Evans
> Attachments: MR-4059.txt, MR-4059.txt, MR-4059.txt, MR-4059.txt, MR-4059.txt
>
>
> The history server currently caches all parsed jobs in RAM. These jobs can be very large because of counters. It would be nice to have a pluggable interface for the cacheing and querying of the cached data so that we can play around with different implementations. Also just for cleanness of the code it would be nice to split the very large JobHistoryServer.java into a few smaller ones that are more understandable and readable.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4059) The history server should have a
separate pluggable storage/query interface
Posted by "Robert Joseph Evans (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Joseph Evans updated MAPREDUCE-4059:
-------------------------------------------
Status: Patch Available (was: Open)
> The history server should have a separate pluggable storage/query interface
> ---------------------------------------------------------------------------
>
> Key: MAPREDUCE-4059
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4059
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
> Affects Versions: 0.24.0, 0.23.3
> Reporter: Robert Joseph Evans
> Assignee: Robert Joseph Evans
> Attachments: MR-4059.txt, MR-4059.txt, MR-4059.txt, MR-4059.txt
>
>
> The history server currently caches all parsed jobs in RAM. These jobs can be very large because of counters. It would be nice to have a pluggable interface for the cacheing and querying of the cached data so that we can play around with different implementations. Also just for cleanness of the code it would be nice to split the very large JobHistoryServer.java into a few smaller ones that are more understandable and readable.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4059) The history server should have
a separate pluggable storage/query interface
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13251604#comment-13251604 ]
Hudson commented on MAPREDUCE-4059:
-----------------------------------
Integrated in Hadoop-Mapreduce-trunk #1046 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1046/])
MAPREDUCE-4059. The history server should have a separate pluggable storage/query interface. (Robert Evans via tgraves). (Revision 1311896)
Result = SUCCESS
tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1311896
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/jobhistory/JHAdminConfig.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/jobhistory/JobHistoryUtils.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/CachedHistoryStorage.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/HistoryContext.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/HistoryFileManager.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/HistoryStorage.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/JobHistory.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/PartialJob.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/webapp/HsWebApp.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/webapp/HsWebServices.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/TestJobHistoryParsing.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/webapp/TestHsWebServices.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/webapp/TestHsWebServicesAttempts.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/webapp/TestHsWebServicesJobConf.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/webapp/TestHsWebServicesJobs.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/webapp/TestHsWebServicesJobsQuery.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/webapp/TestHsWebServicesTasks.java
> The history server should have a separate pluggable storage/query interface
> ---------------------------------------------------------------------------
>
> Key: MAPREDUCE-4059
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4059
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
> Affects Versions: 0.24.0, 0.23.3
> Reporter: Robert Joseph Evans
> Assignee: Robert Joseph Evans
> Fix For: 0.23.3
>
> Attachments: MR-4059.txt, MR-4059.txt, MR-4059.txt, MR-4059.txt, MR-4059.txt, MR-4059.txt, MR-4059.txt
>
>
> The history server currently caches all parsed jobs in RAM. These jobs can be very large because of counters. It would be nice to have a pluggable interface for the cacheing and querying of the cached data so that we can play around with different implementations. Also just for cleanness of the code it would be nice to split the very large JobHistoryServer.java into a few smaller ones that are more understandable and readable.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4059) The history server should have a
separate pluggable storage/query interface
Posted by "Robert Joseph Evans (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Joseph Evans updated MAPREDUCE-4059:
-------------------------------------------
Attachment: MR-4059.txt
OK I fixed the findbugs issues and the test failures. I made a change there the web services needed teh HistoryContext instead of the AppContext and I forgot to update the bind in the tests.
> The history server should have a separate pluggable storage/query interface
> ---------------------------------------------------------------------------
>
> Key: MAPREDUCE-4059
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4059
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
> Affects Versions: 0.24.0, 0.23.3
> Reporter: Robert Joseph Evans
> Assignee: Robert Joseph Evans
> Attachments: MR-4059.txt, MR-4059.txt
>
>
> The history server currently caches all parsed jobs in RAM. These jobs can be very large because of counters. It would be nice to have a pluggable interface for the cacheing and querying of the cached data so that we can play around with different implementations. Also just for cleanness of the code it would be nice to split the very large JobHistoryServer.java into a few smaller ones that are more understandable and readable.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4059) The history server should have
a separate pluggable storage/query interface
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13251553#comment-13251553 ]
Hudson commented on MAPREDUCE-4059:
-----------------------------------
Integrated in Hadoop-Hdfs-trunk #1011 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1011/])
MAPREDUCE-4059. The history server should have a separate pluggable storage/query interface. (Robert Evans via tgraves). (Revision 1311896)
Result = FAILURE
tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1311896
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/jobhistory/JHAdminConfig.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/jobhistory/JobHistoryUtils.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/CachedHistoryStorage.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/HistoryContext.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/HistoryFileManager.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/HistoryStorage.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/JobHistory.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/PartialJob.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/webapp/HsWebApp.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/webapp/HsWebServices.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/TestJobHistoryParsing.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/webapp/TestHsWebServices.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/webapp/TestHsWebServicesAttempts.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/webapp/TestHsWebServicesJobConf.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/webapp/TestHsWebServicesJobs.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/webapp/TestHsWebServicesJobsQuery.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/webapp/TestHsWebServicesTasks.java
> The history server should have a separate pluggable storage/query interface
> ---------------------------------------------------------------------------
>
> Key: MAPREDUCE-4059
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4059
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
> Affects Versions: 0.24.0, 0.23.3
> Reporter: Robert Joseph Evans
> Assignee: Robert Joseph Evans
> Fix For: 0.23.3
>
> Attachments: MR-4059.txt, MR-4059.txt, MR-4059.txt, MR-4059.txt, MR-4059.txt, MR-4059.txt, MR-4059.txt
>
>
> The history server currently caches all parsed jobs in RAM. These jobs can be very large because of counters. It would be nice to have a pluggable interface for the cacheing and querying of the cached data so that we can play around with different implementations. Also just for cleanness of the code it would be nice to split the very large JobHistoryServer.java into a few smaller ones that are more understandable and readable.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4059) The history server should have a
separate pluggable storage/query interface
Posted by "Robert Joseph Evans (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Joseph Evans updated MAPREDUCE-4059:
-------------------------------------------
Status: Patch Available (was: Open)
> The history server should have a separate pluggable storage/query interface
> ---------------------------------------------------------------------------
>
> Key: MAPREDUCE-4059
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4059
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
> Affects Versions: 0.24.0, 0.23.3
> Reporter: Robert Joseph Evans
> Assignee: Robert Joseph Evans
> Attachments: MR-4059.txt, MR-4059.txt, MR-4059.txt, MR-4059.txt, MR-4059.txt
>
>
> The history server currently caches all parsed jobs in RAM. These jobs can be very large because of counters. It would be nice to have a pluggable interface for the cacheing and querying of the cached data so that we can play around with different implementations. Also just for cleanness of the code it would be nice to split the very large JobHistoryServer.java into a few smaller ones that are more understandable and readable.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4059) The history server should have
a separate pluggable storage/query interface
Posted by "Bhallamudi Venkata Siva Kamesh (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13246244#comment-13246244 ]
Bhallamudi Venkata Siva Kamesh commented on MAPREDUCE-4059:
-----------------------------------------------------------
A minor comment: I think, while comparing enum types it is better to use "==" rather than "equals" method.
{code:title=CachedHistoryStorage.java|borderStyle=solid}
if(jobState != null && !jobState.equals(report.getJobState())) {
continue;
}
{code}
Also please go through the following point and clarify
If all the implementations of the HistoryStorage interacts with the HistoryFileManager manager, I think, it is better to make HistoryStorage as an
abstract class, and moving the common boilerplate code to HistoryStorage like init etc.... Currently HistoryFileManager is the only class that is
used to interact with history files stored in the HDFS. But as per my understading of [MAPREDUCE-3973|https://issues.apache.org/jira/browse/MAPREDUCE-3973?focusedCommentId=13226450&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13226450], these job history files can be stored in a differnt store as well?. If this is the case how to manage files stored in a different store (any different implementation)?
> The history server should have a separate pluggable storage/query interface
> ---------------------------------------------------------------------------
>
> Key: MAPREDUCE-4059
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4059
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
> Affects Versions: 0.24.0, 0.23.3
> Reporter: Robert Joseph Evans
> Assignee: Robert Joseph Evans
> Attachments: MR-4059.txt, MR-4059.txt, MR-4059.txt, MR-4059.txt
>
>
> The history server currently caches all parsed jobs in RAM. These jobs can be very large because of counters. It would be nice to have a pluggable interface for the cacheing and querying of the cached data so that we can play around with different implementations. Also just for cleanness of the code it would be nice to split the very large JobHistoryServer.java into a few smaller ones that are more understandable and readable.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4059) The history server should have
a separate pluggable storage/query interface
Posted by "Thomas Graves (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13250075#comment-13250075 ]
Thomas Graves commented on MAPREDUCE-4059:
------------------------------------------
For the most part looks good.
can you make the formatting in CachedHistoryStorage.java consistent - some places are missing a space between if and (.
I think it would be more extensible to have the getPartialJobs interface take a class rather then all the individual query parameters. If you want to push this off to another Jira I'm fine with that since its marked as InterfaceStability.Unstable.
> The history server should have a separate pluggable storage/query interface
> ---------------------------------------------------------------------------
>
> Key: MAPREDUCE-4059
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4059
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
> Affects Versions: 0.24.0, 0.23.3
> Reporter: Robert Joseph Evans
> Assignee: Robert Joseph Evans
> Attachments: MR-4059.txt, MR-4059.txt, MR-4059.txt, MR-4059.txt, MR-4059.txt, MR-4059.txt
>
>
> The history server currently caches all parsed jobs in RAM. These jobs can be very large because of counters. It would be nice to have a pluggable interface for the cacheing and querying of the cached data so that we can play around with different implementations. Also just for cleanness of the code it would be nice to split the very large JobHistoryServer.java into a few smaller ones that are more understandable and readable.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4059) The history server should have a
separate pluggable storage/query interface
Posted by "Robert Joseph Evans (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Joseph Evans updated MAPREDUCE-4059:
-------------------------------------------
Attachment: MR-4059.txt
No code changes just upmerged.
> The history server should have a separate pluggable storage/query interface
> ---------------------------------------------------------------------------
>
> Key: MAPREDUCE-4059
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4059
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
> Affects Versions: 0.24.0, 0.23.3
> Reporter: Robert Joseph Evans
> Assignee: Robert Joseph Evans
> Attachments: MR-4059.txt, MR-4059.txt, MR-4059.txt, MR-4059.txt, MR-4059.txt, MR-4059.txt
>
>
> The history server currently caches all parsed jobs in RAM. These jobs can be very large because of counters. It would be nice to have a pluggable interface for the cacheing and querying of the cached data so that we can play around with different implementations. Also just for cleanness of the code it would be nice to split the very large JobHistoryServer.java into a few smaller ones that are more understandable and readable.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4059) The history server should have
a separate pluggable storage/query interface
Posted by "Bhallamudi Venkata Siva Kamesh (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13245541#comment-13245541 ]
Bhallamudi Venkata Siva Kamesh commented on MAPREDUCE-4059:
-----------------------------------------------------------
Hi Robert,
if count is null, I think assigning count as jobs.size() seems a good option for me. If so, I think we won't have any overflows and can eliminate the following piece of the code.
{code:xml}
if(end < 0) { //due to overflow
end = Long.MAX_VALUE;
}
{code}
Any comments?
> The history server should have a separate pluggable storage/query interface
> ---------------------------------------------------------------------------
>
> Key: MAPREDUCE-4059
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4059
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
> Affects Versions: 0.24.0, 0.23.3
> Reporter: Robert Joseph Evans
> Assignee: Robert Joseph Evans
> Attachments: MR-4059.txt, MR-4059.txt, MR-4059.txt, MR-4059.txt
>
>
> The history server currently caches all parsed jobs in RAM. These jobs can be very large because of counters. It would be nice to have a pluggable interface for the cacheing and querying of the cached data so that we can play around with different implementations. Also just for cleanness of the code it would be nice to split the very large JobHistoryServer.java into a few smaller ones that are more understandable and readable.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4059) The history server should have a
separate pluggable storage/query interface
Posted by "Robert Joseph Evans (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Joseph Evans updated MAPREDUCE-4059:
-------------------------------------------
Status: Open (was: Patch Available)
Looking into the findbugs issues and the test failures.
> The history server should have a separate pluggable storage/query interface
> ---------------------------------------------------------------------------
>
> Key: MAPREDUCE-4059
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4059
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
> Affects Versions: 0.24.0, 0.23.3
> Reporter: Robert Joseph Evans
> Assignee: Robert Joseph Evans
> Attachments: MR-4059.txt
>
>
> The history server currently caches all parsed jobs in RAM. These jobs can be very large because of counters. It would be nice to have a pluggable interface for the cacheing and querying of the cached data so that we can play around with different implementations. Also just for cleanness of the code it would be nice to split the very large JobHistoryServer.java into a few smaller ones that are more understandable and readable.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4059) The history server should have a
separate pluggable storage/query interface
Posted by "Robert Joseph Evans (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Joseph Evans updated MAPREDUCE-4059:
-------------------------------------------
Status: Open (was: Patch Available)
> The history server should have a separate pluggable storage/query interface
> ---------------------------------------------------------------------------
>
> Key: MAPREDUCE-4059
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4059
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
> Affects Versions: 0.24.0, 0.23.3
> Reporter: Robert Joseph Evans
> Assignee: Robert Joseph Evans
> Attachments: MR-4059.txt, MR-4059.txt, MR-4059.txt, MR-4059.txt, MR-4059.txt
>
>
> The history server currently caches all parsed jobs in RAM. These jobs can be very large because of counters. It would be nice to have a pluggable interface for the cacheing and querying of the cached data so that we can play around with different implementations. Also just for cleanness of the code it would be nice to split the very large JobHistoryServer.java into a few smaller ones that are more understandable and readable.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4059) The history server should have
a separate pluggable storage/query interface
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13250934#comment-13250934 ]
Hudson commented on MAPREDUCE-4059:
-----------------------------------
Integrated in Hadoop-Hdfs-trunk-Commit #2114 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2114/])
MAPREDUCE-4059. The history server should have a separate pluggable storage/query interface. (Robert Evans via tgraves). (Revision 1311896)
Result = SUCCESS
tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1311896
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/jobhistory/JHAdminConfig.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/jobhistory/JobHistoryUtils.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/CachedHistoryStorage.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/HistoryContext.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/HistoryFileManager.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/HistoryStorage.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/JobHistory.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/PartialJob.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/webapp/HsWebApp.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/webapp/HsWebServices.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/TestJobHistoryParsing.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/webapp/TestHsWebServices.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/webapp/TestHsWebServicesAttempts.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/webapp/TestHsWebServicesJobConf.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/webapp/TestHsWebServicesJobs.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/webapp/TestHsWebServicesJobsQuery.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/webapp/TestHsWebServicesTasks.java
> The history server should have a separate pluggable storage/query interface
> ---------------------------------------------------------------------------
>
> Key: MAPREDUCE-4059
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4059
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
> Affects Versions: 0.24.0, 0.23.3
> Reporter: Robert Joseph Evans
> Assignee: Robert Joseph Evans
> Fix For: 0.23.3
>
> Attachments: MR-4059.txt, MR-4059.txt, MR-4059.txt, MR-4059.txt, MR-4059.txt, MR-4059.txt, MR-4059.txt
>
>
> The history server currently caches all parsed jobs in RAM. These jobs can be very large because of counters. It would be nice to have a pluggable interface for the cacheing and querying of the cached data so that we can play around with different implementations. Also just for cleanness of the code it would be nice to split the very large JobHistoryServer.java into a few smaller ones that are more understandable and readable.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4059) The history server should have a
separate pluggable storage/query interface
Posted by "Robert Joseph Evans (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Joseph Evans updated MAPREDUCE-4059:
-------------------------------------------
Attachment: MR-4059.txt
This patch is a little large, but there really is almost no new code. JobHistory.java was split up, with most of the original code going into HistoryFileManager.java. Some of the code went into CachedHistoryStorage.java, and a few DEFAULT_ values were moved into JHAdminConfig.
Also some of the query code that was used by the web service was also moved into CachedHistoryStorage.java
I belive that the HistoryStorage API will likely evolve in the future so I have marked it as unstable. It is a first shot at what we may want it to look like.
> The history server should have a separate pluggable storage/query interface
> ---------------------------------------------------------------------------
>
> Key: MAPREDUCE-4059
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4059
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
> Affects Versions: 0.24.0, 0.23.3
> Reporter: Robert Joseph Evans
> Assignee: Robert Joseph Evans
> Attachments: MR-4059.txt
>
>
> The history server currently caches all parsed jobs in RAM. These jobs can be very large because of counters. It would be nice to have a pluggable interface for the cacheing and querying of the cached data so that we can play around with different implementations. Also just for cleanness of the code it would be nice to split the very large JobHistoryServer.java into a few smaller ones that are more understandable and readable.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4059) The history server should have a
separate pluggable storage/query interface
Posted by "Robert Joseph Evans (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Joseph Evans updated MAPREDUCE-4059:
-------------------------------------------
Status: Open (was: Patch Available)
Up-merging patch.
> The history server should have a separate pluggable storage/query interface
> ---------------------------------------------------------------------------
>
> Key: MAPREDUCE-4059
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4059
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
> Affects Versions: 0.24.0, 0.23.3
> Reporter: Robert Joseph Evans
> Assignee: Robert Joseph Evans
> Attachments: MR-4059.txt, MR-4059.txt, MR-4059.txt, MR-4059.txt, MR-4059.txt, MR-4059.txt
>
>
> The history server currently caches all parsed jobs in RAM. These jobs can be very large because of counters. It would be nice to have a pluggable interface for the cacheing and querying of the cached data so that we can play around with different implementations. Also just for cleanness of the code it would be nice to split the very large JobHistoryServer.java into a few smaller ones that are more understandable and readable.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4059) The history server should have a
separate pluggable storage/query interface
Posted by "Robert Joseph Evans (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Joseph Evans updated MAPREDUCE-4059:
-------------------------------------------
Status: Open (was: Patch Available)
Addressing review comments.
> The history server should have a separate pluggable storage/query interface
> ---------------------------------------------------------------------------
>
> Key: MAPREDUCE-4059
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4059
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
> Affects Versions: 0.24.0, 0.23.3
> Reporter: Robert Joseph Evans
> Assignee: Robert Joseph Evans
> Attachments: MR-4059.txt, MR-4059.txt
>
>
> The history server currently caches all parsed jobs in RAM. These jobs can be very large because of counters. It would be nice to have a pluggable interface for the cacheing and querying of the cached data so that we can play around with different implementations. Also just for cleanness of the code it would be nice to split the very large JobHistoryServer.java into a few smaller ones that are more understandable and readable.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4059) The history server should have a
separate pluggable storage/query interface
Posted by "Robert Joseph Evans (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Joseph Evans updated MAPREDUCE-4059:
-------------------------------------------
Status: Patch Available (was: Open)
> The history server should have a separate pluggable storage/query interface
> ---------------------------------------------------------------------------
>
> Key: MAPREDUCE-4059
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4059
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
> Affects Versions: 0.24.0, 0.23.3
> Reporter: Robert Joseph Evans
> Assignee: Robert Joseph Evans
> Attachments: MR-4059.txt, MR-4059.txt, MR-4059.txt, MR-4059.txt, MR-4059.txt, MR-4059.txt, MR-4059.txt
>
>
> The history server currently caches all parsed jobs in RAM. These jobs can be very large because of counters. It would be nice to have a pluggable interface for the cacheing and querying of the cached data so that we can play around with different implementations. Also just for cleanness of the code it would be nice to split the very large JobHistoryServer.java into a few smaller ones that are more understandable and readable.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira