You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by "Dmytro Molkov (JIRA)" <ji...@apache.org> on 2010/05/18 20:41:51 UTC
[jira] Created: (HIVE-1349) Do not pull counters of non initialized
jobs
Do not pull counters of non initialized jobs
--------------------------------------------
Key: HIVE-1349
URL: https://issues.apache.org/jira/browse/HIVE-1349
Project: Hadoop Hive
Issue Type: Improvement
Reporter: Dmytro Molkov
Right now hive client puts a lot of pressure on the jobtracker by pulling the counters of the job that has just been submitted and is in the process of initializing tasks.
While the job initialized tasks and reads the splits file it is holding the JobInProgress lock. The getCounters method acquires a JobTracker lock and waits for the JobInProgress lock. Since the read of the file can take a couple (a couple of tens of seconds) the job tracker is effectively blocked for that period of time.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1349) Do not pull counters of non initialized
jobs
Posted by "Dmytro Molkov (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dmytro Molkov updated HIVE-1349:
--------------------------------
Status: Open (was: Patch Available)
> Do not pull counters of non initialized jobs
> --------------------------------------------
>
> Key: HIVE-1349
> URL: https://issues.apache.org/jira/browse/HIVE-1349
> Project: Hadoop Hive
> Issue Type: Improvement
> Reporter: Dmytro Molkov
> Attachments: HIVE-1349.2.patch, HIVE-1349.patch
>
>
> Right now hive client puts a lot of pressure on the jobtracker by pulling the counters of the job that has just been submitted and is in the process of initializing tasks.
> While the job initialized tasks and reads the splits file it is holding the JobInProgress lock. The getCounters method acquires a JobTracker lock and waits for the JobInProgress lock. Since the read of the file can take a couple (a couple of tens of seconds) the job tracker is effectively blocked for that period of time.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1349) Do not pull counters of non
initialized jobs
Posted by "Dmytro Molkov (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12868977#action_12868977 ]
Dmytro Molkov commented on HIVE-1349:
-------------------------------------
Thanks Paul, I just reached the same conclusion looking at shims.
Will try to get patch ready again soon.
> Do not pull counters of non initialized jobs
> --------------------------------------------
>
> Key: HIVE-1349
> URL: https://issues.apache.org/jira/browse/HIVE-1349
> Project: Hadoop Hive
> Issue Type: Improvement
> Reporter: Dmytro Molkov
> Attachments: HIVE-1349.2.patch, HIVE-1349.patch
>
>
> Right now hive client puts a lot of pressure on the jobtracker by pulling the counters of the job that has just been submitted and is in the process of initializing tasks.
> While the job initialized tasks and reads the splits file it is holding the JobInProgress lock. The getCounters method acquires a JobTracker lock and waits for the JobInProgress lock. Since the read of the file can take a couple (a couple of tens of seconds) the job tracker is effectively blocked for that period of time.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1349) Do not pull counters of non initialized
jobs
Posted by "Dmytro Molkov (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dmytro Molkov updated HIVE-1349:
--------------------------------
Attachment: HIVE-1349.2.patch
Added the stuff to the configuration.
> Do not pull counters of non initialized jobs
> --------------------------------------------
>
> Key: HIVE-1349
> URL: https://issues.apache.org/jira/browse/HIVE-1349
> Project: Hadoop Hive
> Issue Type: Improvement
> Reporter: Dmytro Molkov
> Attachments: HIVE-1349.2.patch, HIVE-1349.patch
>
>
> Right now hive client puts a lot of pressure on the jobtracker by pulling the counters of the job that has just been submitted and is in the process of initializing tasks.
> While the job initialized tasks and reads the splits file it is holding the JobInProgress lock. The getCounters method acquires a JobTracker lock and waits for the JobInProgress lock. Since the read of the file can take a couple (a couple of tens of seconds) the job tracker is effectively blocked for that period of time.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1349) Do not pull counters of non initialized
jobs
Posted by "Dmytro Molkov (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dmytro Molkov updated HIVE-1349:
--------------------------------
Status: Patch Available (was: Open)
> Do not pull counters of non initialized jobs
> --------------------------------------------
>
> Key: HIVE-1349
> URL: https://issues.apache.org/jira/browse/HIVE-1349
> Project: Hadoop Hive
> Issue Type: Improvement
> Reporter: Dmytro Molkov
> Attachments: HIVE-1349.2.patch, HIVE-1349.patch
>
>
> Right now hive client puts a lot of pressure on the jobtracker by pulling the counters of the job that has just been submitted and is in the process of initializing tasks.
> While the job initialized tasks and reads the splits file it is holding the JobInProgress lock. The getCounters method acquires a JobTracker lock and waits for the JobInProgress lock. Since the read of the file can take a couple (a couple of tens of seconds) the job tracker is effectively blocked for that period of time.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1349) Do not pull counters of non initialized
jobs
Posted by "Ning Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ning Zhang updated HIVE-1349:
-----------------------------
Status: Resolved (was: Patch Available)
Fix Version/s: 0.6.0
Resolution: Fixed
Committed. Thanks Dmytro!
> Do not pull counters of non initialized jobs
> --------------------------------------------
>
> Key: HIVE-1349
> URL: https://issues.apache.org/jira/browse/HIVE-1349
> Project: Hadoop Hive
> Issue Type: Improvement
> Reporter: Dmytro Molkov
> Fix For: 0.6.0
>
> Attachments: HIVE-1349.2.patch, HIVE-1349.3.patch, HIVE-1349.patch
>
>
> Right now hive client puts a lot of pressure on the jobtracker by pulling the counters of the job that has just been submitted and is in the process of initializing tasks.
> While the job initialized tasks and reads the splits file it is holding the JobInProgress lock. The getCounters method acquires a JobTracker lock and waits for the JobInProgress lock. Since the read of the file can take a couple (a couple of tens of seconds) the job tracker is effectively blocked for that period of time.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1349) Do not pull counters of non initialized
jobs
Posted by "Dmytro Molkov (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dmytro Molkov updated HIVE-1349:
--------------------------------
Attachment: HIVE-1349.3.patch
OK. Have a look at Shims implementation, does this look right?
I will submit for tests right away.
> Do not pull counters of non initialized jobs
> --------------------------------------------
>
> Key: HIVE-1349
> URL: https://issues.apache.org/jira/browse/HIVE-1349
> Project: Hadoop Hive
> Issue Type: Improvement
> Reporter: Dmytro Molkov
> Attachments: HIVE-1349.2.patch, HIVE-1349.3.patch, HIVE-1349.patch
>
>
> Right now hive client puts a lot of pressure on the jobtracker by pulling the counters of the job that has just been submitted and is in the process of initializing tasks.
> While the job initialized tasks and reads the splits file it is holding the JobInProgress lock. The getCounters method acquires a JobTracker lock and waits for the JobInProgress lock. Since the read of the file can take a couple (a couple of tens of seconds) the job tracker is effectively blocked for that period of time.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1349) Do not pull counters of non initialized
jobs
Posted by "Dmytro Molkov (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dmytro Molkov updated HIVE-1349:
--------------------------------
Status: Patch Available (was: Open)
Submitting the patch for testing, it doesn't include a unit test with it, but I am not really sure how to test this kind of change. If you have an idea - please let me know.
> Do not pull counters of non initialized jobs
> --------------------------------------------
>
> Key: HIVE-1349
> URL: https://issues.apache.org/jira/browse/HIVE-1349
> Project: Hadoop Hive
> Issue Type: Improvement
> Reporter: Dmytro Molkov
> Attachments: HIVE-1349.patch
>
>
> Right now hive client puts a lot of pressure on the jobtracker by pulling the counters of the job that has just been submitted and is in the process of initializing tasks.
> While the job initialized tasks and reads the splits file it is holding the JobInProgress lock. The getCounters method acquires a JobTracker lock and waits for the JobInProgress lock. Since the read of the file can take a couple (a couple of tens of seconds) the job tracker is effectively blocked for that period of time.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1349) Do not pull counters of non
initialized jobs
Posted by "Ning Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12868944#action_12868944 ]
Ning Zhang commented on HIVE-1349:
----------------------------------
This patch does not compile with hadoop 0.17. rc.getStatus() seems not available in 0.17. Can you do ant -Dhadoop.version=0.17.2.1 clean package to verify?
> Do not pull counters of non initialized jobs
> --------------------------------------------
>
> Key: HIVE-1349
> URL: https://issues.apache.org/jira/browse/HIVE-1349
> Project: Hadoop Hive
> Issue Type: Improvement
> Reporter: Dmytro Molkov
> Attachments: HIVE-1349.2.patch, HIVE-1349.patch
>
>
> Right now hive client puts a lot of pressure on the jobtracker by pulling the counters of the job that has just been submitted and is in the process of initializing tasks.
> While the job initialized tasks and reads the splits file it is holding the JobInProgress lock. The getCounters method acquires a JobTracker lock and waits for the JobInProgress lock. Since the read of the file can take a couple (a couple of tens of seconds) the job tracker is effectively blocked for that period of time.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1349) Do not pull counters of non initialized
jobs
Posted by "Dmytro Molkov (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dmytro Molkov updated HIVE-1349:
--------------------------------
Attachment: HIVE-1349.patch
Hey guys, take a look at the patch please.
When the job is created the status is PREP, it is switched to RUNNING when the setup is finished, so while the job status is PREP we are skipping the polling of counters.
The poll interval is also made configurable, so hive can hit jobtrackers getCounter which is not perfect less frequently.
> Do not pull counters of non initialized jobs
> --------------------------------------------
>
> Key: HIVE-1349
> URL: https://issues.apache.org/jira/browse/HIVE-1349
> Project: Hadoop Hive
> Issue Type: Improvement
> Reporter: Dmytro Molkov
> Attachments: HIVE-1349.patch
>
>
> Right now hive client puts a lot of pressure on the jobtracker by pulling the counters of the job that has just been submitted and is in the process of initializing tasks.
> While the job initialized tasks and reads the splits file it is holding the JobInProgress lock. The getCounters method acquires a JobTracker lock and waits for the JobInProgress lock. Since the read of the file can take a couple (a couple of tens of seconds) the job tracker is effectively blocked for that period of time.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1349) Do not pull counters of non
initialized jobs
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12868849#action_12868849 ]
Namit Jain commented on HIVE-1349:
----------------------------------
I will take a look
> Do not pull counters of non initialized jobs
> --------------------------------------------
>
> Key: HIVE-1349
> URL: https://issues.apache.org/jira/browse/HIVE-1349
> Project: Hadoop Hive
> Issue Type: Improvement
> Reporter: Dmytro Molkov
> Attachments: HIVE-1349.patch
>
>
> Right now hive client puts a lot of pressure on the jobtracker by pulling the counters of the job that has just been submitted and is in the process of initializing tasks.
> While the job initialized tasks and reads the splits file it is holding the JobInProgress lock. The getCounters method acquires a JobTracker lock and waits for the JobInProgress lock. Since the read of the file can take a couple (a couple of tens of seconds) the job tracker is effectively blocked for that period of time.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1349) Do not pull counters of non
initialized jobs
Posted by "Ning Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12868858#action_12868858 ]
Ning Zhang commented on HIVE-1349:
----------------------------------
Can you add the new hive parameter to conf/hive-default.xml as well? Other than that it looks good.
> Do not pull counters of non initialized jobs
> --------------------------------------------
>
> Key: HIVE-1349
> URL: https://issues.apache.org/jira/browse/HIVE-1349
> Project: Hadoop Hive
> Issue Type: Improvement
> Reporter: Dmytro Molkov
> Attachments: HIVE-1349.patch
>
>
> Right now hive client puts a lot of pressure on the jobtracker by pulling the counters of the job that has just been submitted and is in the process of initializing tasks.
> While the job initialized tasks and reads the splits file it is holding the JobInProgress lock. The getCounters method acquires a JobTracker lock and waits for the JobInProgress lock. Since the read of the file can take a couple (a couple of tens of seconds) the job tracker is effectively blocked for that period of time.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1349) Do not pull counters of non
initialized jobs
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12868856#action_12868856 ]
Namit Jain commented on HIVE-1349:
----------------------------------
Can you add the description of the new parameter: hive.exec.counters.pull.interval: in hive-default.xml ?
> Do not pull counters of non initialized jobs
> --------------------------------------------
>
> Key: HIVE-1349
> URL: https://issues.apache.org/jira/browse/HIVE-1349
> Project: Hadoop Hive
> Issue Type: Improvement
> Reporter: Dmytro Molkov
> Attachments: HIVE-1349.patch
>
>
> Right now hive client puts a lot of pressure on the jobtracker by pulling the counters of the job that has just been submitted and is in the process of initializing tasks.
> While the job initialized tasks and reads the splits file it is holding the JobInProgress lock. The getCounters method acquires a JobTracker lock and waits for the JobInProgress lock. Since the read of the file can take a couple (a couple of tens of seconds) the job tracker is effectively blocked for that period of time.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1349) Do not pull counters of non initialized
jobs
Posted by "Dmytro Molkov (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dmytro Molkov updated HIVE-1349:
--------------------------------
Status: Patch Available (was: Open)
> Do not pull counters of non initialized jobs
> --------------------------------------------
>
> Key: HIVE-1349
> URL: https://issues.apache.org/jira/browse/HIVE-1349
> Project: Hadoop Hive
> Issue Type: Improvement
> Reporter: Dmytro Molkov
> Attachments: HIVE-1349.2.patch, HIVE-1349.3.patch, HIVE-1349.patch
>
>
> Right now hive client puts a lot of pressure on the jobtracker by pulling the counters of the job that has just been submitted and is in the process of initializing tasks.
> While the job initialized tasks and reads the splits file it is holding the JobInProgress lock. The getCounters method acquires a JobTracker lock and waits for the JobInProgress lock. Since the read of the file can take a couple (a couple of tens of seconds) the job tracker is effectively blocked for that period of time.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1349) Do not pull counters of non
initialized jobs
Posted by "Ning Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12868913#action_12868913 ]
Ning Zhang commented on HIVE-1349:
----------------------------------
+1. will commit after tests.
> Do not pull counters of non initialized jobs
> --------------------------------------------
>
> Key: HIVE-1349
> URL: https://issues.apache.org/jira/browse/HIVE-1349
> Project: Hadoop Hive
> Issue Type: Improvement
> Reporter: Dmytro Molkov
> Attachments: HIVE-1349.2.patch, HIVE-1349.patch
>
>
> Right now hive client puts a lot of pressure on the jobtracker by pulling the counters of the job that has just been submitted and is in the process of initializing tasks.
> While the job initialized tasks and reads the splits file it is holding the JobInProgress lock. The getCounters method acquires a JobTracker lock and waits for the JobInProgress lock. Since the read of the file can take a couple (a couple of tens of seconds) the job tracker is effectively blocked for that period of time.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1349) Do not pull counters of non initialized
jobs
Posted by "Dmytro Molkov (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dmytro Molkov updated HIVE-1349:
--------------------------------
Status: Open (was: Patch Available)
Yes, agreed. I somehow thought it was building it with all the versions.
Is there a way to fix non hadoop-17 versions only?
> Do not pull counters of non initialized jobs
> --------------------------------------------
>
> Key: HIVE-1349
> URL: https://issues.apache.org/jira/browse/HIVE-1349
> Project: Hadoop Hive
> Issue Type: Improvement
> Reporter: Dmytro Molkov
> Attachments: HIVE-1349.2.patch, HIVE-1349.patch
>
>
> Right now hive client puts a lot of pressure on the jobtracker by pulling the counters of the job that has just been submitted and is in the process of initializing tasks.
> While the job initialized tasks and reads the splits file it is holding the JobInProgress lock. The getCounters method acquires a JobTracker lock and waits for the JobInProgress lock. Since the read of the file can take a couple (a couple of tens of seconds) the job tracker is effectively blocked for that period of time.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1349) Do not pull counters of non
initialized jobs
Posted by "Paul Yang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12868975#action_12868975 ]
Paul Yang commented on HIVE-1349:
---------------------------------
Yes, take a look at the HadoopShims interface - there's a set of classes that implements this interface, with a a different implementation for each major Hadoop version. You can add a method to the interface that does rc.getStatus() and implement the method for the shim classes > 0.17.
> Do not pull counters of non initialized jobs
> --------------------------------------------
>
> Key: HIVE-1349
> URL: https://issues.apache.org/jira/browse/HIVE-1349
> Project: Hadoop Hive
> Issue Type: Improvement
> Reporter: Dmytro Molkov
> Attachments: HIVE-1349.2.patch, HIVE-1349.patch
>
>
> Right now hive client puts a lot of pressure on the jobtracker by pulling the counters of the job that has just been submitted and is in the process of initializing tasks.
> While the job initialized tasks and reads the splits file it is holding the JobInProgress lock. The getCounters method acquires a JobTracker lock and waits for the JobInProgress lock. Since the read of the file can take a couple (a couple of tens of seconds) the job tracker is effectively blocked for that period of time.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1349) Do not pull counters of non
initialized jobs
Posted by "Ning Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12869003#action_12869003 ]
Ning Zhang commented on HIVE-1349:
----------------------------------
looks good. testing now
> Do not pull counters of non initialized jobs
> --------------------------------------------
>
> Key: HIVE-1349
> URL: https://issues.apache.org/jira/browse/HIVE-1349
> Project: Hadoop Hive
> Issue Type: Improvement
> Reporter: Dmytro Molkov
> Attachments: HIVE-1349.2.patch, HIVE-1349.3.patch, HIVE-1349.patch
>
>
> Right now hive client puts a lot of pressure on the jobtracker by pulling the counters of the job that has just been submitted and is in the process of initializing tasks.
> While the job initialized tasks and reads the splits file it is holding the JobInProgress lock. The getCounters method acquires a JobTracker lock and waits for the JobInProgress lock. Since the read of the file can take a couple (a couple of tens of seconds) the job tracker is effectively blocked for that period of time.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.