You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Amareshwari Sriramadasu (JIRA)" <ji...@apache.org> on 2010/05/10 07:09:48 UTC
[jira] Created: (MAPREDUCE-1779) Should we provide a way to know
JobTracker's memory info from client?
Should we provide a way to know JobTracker's memory info from client?
---------------------------------------------------------------------
Key: MAPREDUCE-1779
URL: https://issues.apache.org/jira/browse/MAPREDUCE-1779
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: client, jobtracker
Affects Versions: 0.21.0
Reporter: Amareshwari Sriramadasu
Fix For: 0.22.0
In HADOOP-4435, in branch 0.20, getClusterStatus() method returns JobTracker's used memory and total memory.
But these details are missed in new api (through MAPREDUCE-777).
If these details are needed only for web UI, I don't think they are needed for client.
So, should we provide a way to know JobTracker's memory info from client?
If yes, an api should be added in org.apache.hadoop.mapreduce.Cluster for the same.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1779) Should we provide a way to know
JobTracker's memory info from client?
Posted by "Amareshwari Sriramadasu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1779?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amareshwari Sriramadasu updated MAPREDUCE-1779:
-----------------------------------------------
Fix Version/s: 0.22.0
(was: 0.21.0)
Moving it to 0.22 as per Dhruba's comment.
bq. There are applications that are using this information currently exposed via the ClusterStatus API? is there an alternative way to retrieving them?
Branch 0.21 does not have a way to get this memory information from clients.
> Should we provide a way to know JobTracker's memory info from client?
> ---------------------------------------------------------------------
>
> Key: MAPREDUCE-1779
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1779
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: client, jobtracker
> Affects Versions: 0.21.0
> Reporter: Amareshwari Sriramadasu
> Priority: Blocker
> Fix For: 0.22.0
>
>
> In HADOOP-4435, in branch 0.20, getClusterStatus() method returns JobTracker's used memory and total memory.
> But these details are missed in new api (through MAPREDUCE-777).
> If these details are needed only for web UI, I don't think they are needed for client.
> So, should we provide a way to know JobTracker's memory info from client?
> If yes, an api should be added in org.apache.hadoop.mapreduce.Cluster for the same.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1779) Should we provide a way to know
JobTracker's memory info from client?
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12868104#action_12868104 ]
dhruba borthakur commented on MAPREDUCE-1779:
---------------------------------------------
Two reasons:
1. Hive actually throttles new job-submissions if the heap memory on the JT exceeds a certain threshold.
2. It is also needed to monitor the health of the JT.
> Should we provide a way to know JobTracker's memory info from client?
> ---------------------------------------------------------------------
>
> Key: MAPREDUCE-1779
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1779
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: client, jobtracker
> Affects Versions: 0.21.0
> Reporter: Amareshwari Sriramadasu
> Priority: Blocker
> Fix For: 0.22.0
>
>
> In HADOOP-4435, in branch 0.20, getClusterStatus() method returns JobTracker's used memory and total memory.
> But these details are missed in new api (through MAPREDUCE-777).
> If these details are needed only for web UI, I don't think they are needed for client.
> So, should we provide a way to know JobTracker's memory info from client?
> If yes, an api should be added in org.apache.hadoop.mapreduce.Cluster for the same.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1779) Should we provide a way to know
JobTracker's memory info from client?
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12868269#action_12868269 ]
Arun C Murthy commented on MAPREDUCE-1779:
------------------------------------------
bq. It is also needed to monitor the health of the JT.
This is already available in JVM Metrics.
bq. Hive actually throttles new job-submissions if the heap memory on the JT exceeds a certain threshold.
Again, I'd like to re-iterate that this 'feature' causes *serious* performance issues on the JobTracker - JNI calls are *very* expensive, a rouge client for this feature can easily cause *severe* harm to the JobTracker and hence the entire cluster.
Hive can use JVM Metrics for the same functionality given that this is already available in JVM Metrics. Thus, I'm -1 on this feature.
> Should we provide a way to know JobTracker's memory info from client?
> ---------------------------------------------------------------------
>
> Key: MAPREDUCE-1779
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1779
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: client, jobtracker
> Affects Versions: 0.21.0
> Reporter: Amareshwari Sriramadasu
> Priority: Blocker
> Fix For: 0.22.0
>
>
> In HADOOP-4435, in branch 0.20, getClusterStatus() method returns JobTracker's used memory and total memory.
> But these details are missed in new api (through MAPREDUCE-777).
> If these details are needed only for web UI, I don't think they are needed for client.
> So, should we provide a way to know JobTracker's memory info from client?
> If yes, an api should be added in org.apache.hadoop.mapreduce.Cluster for the same.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1779) Should we provide a way to know
JobTracker's memory info from client?
Posted by "Amar Kamat (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12868099#action_12868099 ]
Amar Kamat commented on MAPREDUCE-1779:
---------------------------------------
Why would client applications need JobTracker's memory information. I think the reason we added it to ClusterStatus was that its maintained at one place and its passed it to the webui for display. I dont think JobTracker's memory information should be a part of ClusterStatus. If at all some admins require it, it should be made available via MRAdmin. I dont see any reason why client should be aware of JobTracker's memory details.
> Should we provide a way to know JobTracker's memory info from client?
> ---------------------------------------------------------------------
>
> Key: MAPREDUCE-1779
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1779
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: client, jobtracker
> Affects Versions: 0.21.0
> Reporter: Amareshwari Sriramadasu
> Priority: Blocker
> Fix For: 0.22.0
>
>
> In HADOOP-4435, in branch 0.20, getClusterStatus() method returns JobTracker's used memory and total memory.
> But these details are missed in new api (through MAPREDUCE-777).
> If these details are needed only for web UI, I don't think they are needed for client.
> So, should we provide a way to know JobTracker's memory info from client?
> If yes, an api should be added in org.apache.hadoop.mapreduce.Cluster for the same.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1779) Should we provide a way to know
JobTracker's memory info from client?
Posted by "Luke Lu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12865983#action_12865983 ]
Luke Lu commented on MAPREDUCE-1779:
------------------------------------
Per discussions with Chris and Arun, the JT memory stats should not be in ClusterStatus.
> Should we provide a way to know JobTracker's memory info from client?
> ---------------------------------------------------------------------
>
> Key: MAPREDUCE-1779
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1779
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: client, jobtracker
> Affects Versions: 0.21.0
> Reporter: Amareshwari Sriramadasu
> Fix For: 0.22.0
>
>
> In HADOOP-4435, in branch 0.20, getClusterStatus() method returns JobTracker's used memory and total memory.
> But these details are missed in new api (through MAPREDUCE-777).
> If these details are needed only for web UI, I don't think they are needed for client.
> So, should we provide a way to know JobTracker's memory info from client?
> If yes, an api should be added in org.apache.hadoop.mapreduce.Cluster for the same.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1779) Should we provide a way to know
JobTracker's memory info from client?
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12868302#action_12868302 ]
dhruba borthakur commented on MAPREDUCE-1779:
---------------------------------------------
Sounds like a fine idea to me, if this data is already available via JVM Metrics.
> Should we provide a way to know JobTracker's memory info from client?
> ---------------------------------------------------------------------
>
> Key: MAPREDUCE-1779
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1779
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: client, jobtracker
> Affects Versions: 0.21.0
> Reporter: Amareshwari Sriramadasu
> Priority: Blocker
> Fix For: 0.22.0
>
>
> In HADOOP-4435, in branch 0.20, getClusterStatus() method returns JobTracker's used memory and total memory.
> But these details are missed in new api (through MAPREDUCE-777).
> If these details are needed only for web UI, I don't think they are needed for client.
> So, should we provide a way to know JobTracker's memory info from client?
> If yes, an api should be added in org.apache.hadoop.mapreduce.Cluster for the same.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1779) Should we provide a way to know
JobTracker's memory info from client?
Posted by "Amareshwari Sriramadasu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1779?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amareshwari Sriramadasu updated MAPREDUCE-1779:
-----------------------------------------------
Fix Version/s: 0.21.0
(was: 0.22.0)
Priority: Blocker (was: Major)
> Should we provide a way to know JobTracker's memory info from client?
> ---------------------------------------------------------------------
>
> Key: MAPREDUCE-1779
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1779
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: client, jobtracker
> Affects Versions: 0.21.0
> Reporter: Amareshwari Sriramadasu
> Priority: Blocker
> Fix For: 0.21.0
>
>
> In HADOOP-4435, in branch 0.20, getClusterStatus() method returns JobTracker's used memory and total memory.
> But these details are missed in new api (through MAPREDUCE-777).
> If these details are needed only for web UI, I don't think they are needed for client.
> So, should we provide a way to know JobTracker's memory info from client?
> If yes, an api should be added in org.apache.hadoop.mapreduce.Cluster for the same.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1779) Should we provide a way to know
JobTracker's memory info from client?
Posted by "Amareshwari Sriramadasu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12866491#action_12866491 ]
Amareshwari Sriramadasu commented on MAPREDUCE-1779:
----------------------------------------------------
bq. Does this mean that the webUi will not show the JT memory usage anymore?
No. Web UI will continue to show memory usage. Since JobTracker is the one serving web ui, It need not pass memory info through getClusterStatus() api.
bq. There are applications that are using this information currently exposed via the ClusterStatus API?
I think we should add another api like getJTMemoryInfo() to retrieve the information from Applications. Memory info can not be added to getClusterStatus() because
* it is not cluster info.
* getClusterStatus() is synchronized and we cannot introduce jni calls into this. See MAPREDUCE-1683.
> Should we provide a way to know JobTracker's memory info from client?
> ---------------------------------------------------------------------
>
> Key: MAPREDUCE-1779
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1779
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: client, jobtracker
> Affects Versions: 0.21.0
> Reporter: Amareshwari Sriramadasu
> Fix For: 0.22.0
>
>
> In HADOOP-4435, in branch 0.20, getClusterStatus() method returns JobTracker's used memory and total memory.
> But these details are missed in new api (through MAPREDUCE-777).
> If these details are needed only for web UI, I don't think they are needed for client.
> So, should we provide a way to know JobTracker's memory info from client?
> If yes, an api should be added in org.apache.hadoop.mapreduce.Cluster for the same.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1779) Should we provide a way to know
JobTracker's memory info from client?
Posted by "Tom White (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12867682#action_12867682 ]
Tom White commented on MAPREDUCE-1779:
--------------------------------------
Does this need to be a blocker? We could add this in a later (point) release, no?
> Should we provide a way to know JobTracker's memory info from client?
> ---------------------------------------------------------------------
>
> Key: MAPREDUCE-1779
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1779
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: client, jobtracker
> Affects Versions: 0.21.0
> Reporter: Amareshwari Sriramadasu
> Priority: Blocker
> Fix For: 0.21.0
>
>
> In HADOOP-4435, in branch 0.20, getClusterStatus() method returns JobTracker's used memory and total memory.
> But these details are missed in new api (through MAPREDUCE-777).
> If these details are needed only for web UI, I don't think they are needed for client.
> So, should we provide a way to know JobTracker's memory info from client?
> If yes, an api should be added in org.apache.hadoop.mapreduce.Cluster for the same.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1779) Should we provide a way to know
JobTracker's memory info from client?
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12867729#action_12867729 ]
dhruba borthakur commented on MAPREDUCE-1779:
---------------------------------------------
I am fine moving it to 0.22.
> Should we provide a way to know JobTracker's memory info from client?
> ---------------------------------------------------------------------
>
> Key: MAPREDUCE-1779
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1779
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: client, jobtracker
> Affects Versions: 0.21.0
> Reporter: Amareshwari Sriramadasu
> Priority: Blocker
> Fix For: 0.21.0
>
>
> In HADOOP-4435, in branch 0.20, getClusterStatus() method returns JobTracker's used memory and total memory.
> But these details are missed in new api (through MAPREDUCE-777).
> If these details are needed only for web UI, I don't think they are needed for client.
> So, should we provide a way to know JobTracker's memory info from client?
> If yes, an api should be added in org.apache.hadoop.mapreduce.Cluster for the same.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1779) Should we provide a way to know
JobTracker's memory info from client?
Posted by "Amareshwari Sriramadasu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1779?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amareshwari Sriramadasu updated MAPREDUCE-1779:
-----------------------------------------------
Moving it to 0.22 as per Dhruba's comment.
bq. There are applications that are using this information currently exposed via the ClusterStatus API? is there an alternative way to retrieving them?
Branch 0.21 does not have a way to get this memory information from clients.
> Should we provide a way to know JobTracker's memory info from client?
> ---------------------------------------------------------------------
>
> Key: MAPREDUCE-1779
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1779
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: client, jobtracker
> Affects Versions: 0.21.0
> Reporter: Amareshwari Sriramadasu
> Priority: Blocker
> Fix For: 0.22.0
>
>
> In HADOOP-4435, in branch 0.20, getClusterStatus() method returns JobTracker's used memory and total memory.
> But these details are missed in new api (through MAPREDUCE-777).
> If these details are needed only for web UI, I don't think they are needed for client.
> So, should we provide a way to know JobTracker's memory info from client?
> If yes, an api should be added in org.apache.hadoop.mapreduce.Cluster for the same.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1779) Should we provide a way to know
JobTracker's memory info from client?
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12866487#action_12866487 ]
dhruba borthakur commented on MAPREDUCE-1779:
---------------------------------------------
1. Does this mean that the webUi will not show the JT memory usage anymore?
2. There are applications that are using this information currently exposed via the ClusterStatus API? is there an alternative way to retrieving them?
> Should we provide a way to know JobTracker's memory info from client?
> ---------------------------------------------------------------------
>
> Key: MAPREDUCE-1779
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1779
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: client, jobtracker
> Affects Versions: 0.21.0
> Reporter: Amareshwari Sriramadasu
> Fix For: 0.22.0
>
>
> In HADOOP-4435, in branch 0.20, getClusterStatus() method returns JobTracker's used memory and total memory.
> But these details are missed in new api (through MAPREDUCE-777).
> If these details are needed only for web UI, I don't think they are needed for client.
> So, should we provide a way to know JobTracker's memory info from client?
> If yes, an api should be added in org.apache.hadoop.mapreduce.Cluster for the same.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (MAPREDUCE-1779) Should we provide a way to know
JobTracker's memory info from client?
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1779?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy resolved MAPREDUCE-1779.
--------------------------------------
Resolution: Won't Fix
Thanks Dhruba. Closing as 'wontfix'.
> Should we provide a way to know JobTracker's memory info from client?
> ---------------------------------------------------------------------
>
> Key: MAPREDUCE-1779
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1779
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: client, jobtracker
> Affects Versions: 0.21.0
> Reporter: Amareshwari Sriramadasu
> Priority: Blocker
> Fix For: 0.22.0
>
>
> In HADOOP-4435, in branch 0.20, getClusterStatus() method returns JobTracker's used memory and total memory.
> But these details are missed in new api (through MAPREDUCE-777).
> If these details are needed only for web UI, I don't think they are needed for client.
> So, should we provide a way to know JobTracker's memory info from client?
> If yes, an api should be added in org.apache.hadoop.mapreduce.Cluster for the same.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.