You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ambari.apache.org by Dmytro Sen <ds...@hortonworks.com> on 2015/03/04 01:51:49 UTC
Review Request 31713: Query for FLUME_HANDLER metrics takes several
minutes
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31713/
-----------------------------------------------------------
Review request for Ambari, Myroslav Papirkovskyy and Sid Wagle.
Bugs: AMBARI-9914
https://issues.apache.org/jira/browse/AMBARI-9914
Repository: ambari
Description
-------
API call:
curl -u admin:admin "http://host1:8080/api/v1/clusters/cc/components/?ServiceComponentInfo/component_name=FLUME_HANDLER|ServiceComponentInfo/component_name=APP_TIMELINE_SERVER|ServiceComponentInfo/category=MASTER&fields=ServiceComponentInfo/Version,ServiceComponentInfo/StartTime,ServiceComponentInfo/HeapMemoryUsed,ServiceComponentInfo/HeapMemoryMax,ServiceComponentInfo/service_name,host_components/HostRoles/host_name,host_components/HostRoles/state,host_components/HostRoles/maintenance_state,host_components/HostRoles/stale_configs,host_components/HostRoles/ha_state,host_components/HostRoles/desired_admin_state,host_components/metrics/jvm/memHeapUsedM,host_components/metrics/jvm/HeapMemoryMax,host_components/metrics/jvm/HeapMemoryUsed,host_components/metrics/jvm/memHeapCommittedM,host_components/metrics/mapred/jobtracker/trackers_decommissioned,host_components/metrics/cpu/cpu_wio,host_components/metrics/rpc/RpcQueueTime_avg_time,host_components/metrics/dfs/FSNamesystem/*,host_componen
ts/metrics/dfs/namenode/Version,host_components/metrics/dfs/namenode/LiveNodes,host_components/metrics/dfs/namenode/DeadNodes,host_components/metrics/dfs/namenode/DecomNodes,host_components/metrics/dfs/namenode/TotalFiles,host_components/metrics/dfs/namenode/UpgradeFinalized,host_components/metrics/dfs/namenode/Safemode,host_components/metrics/runtime/StartTime,host_components/metrics/flume/flume,host_components/processes/HostComponentProcess,host_components/metrics/hbase/master/IsActiveMaster,ServiceComponentInfo/MasterStartTime,ServiceComponentInfo/MasterActiveTime,ServiceComponentInfo/AverageLoad,ServiceComponentInfo/Revision,ServiceComponentInfo/RegionsInTransition,metrics/api/v1/cluster/summary,metrics/api/v1/topology/summary,host_components/metrics/yarn/Queue,ServiceComponentInfo/rm_metrics/cluster/activeNMcount,ServiceComponentInfo/rm_metrics/cluster/lostNMcount,ServiceComponentInfo/rm_metrics/cluster/unhealthyNMcount,ServiceComponentInfo/rm_metrics/cluster/rebootedNMcount,Se
rviceComponentInfo/rm_metrics/cluster/decommissionedNMcount&minimal_response=true&_=1425140480545"
Takes several minutes on bigger clusters
Diffs
-----
ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/PhoenixTransactSQL.java 80c6545
ambari-server/src/main/resources/ganglia_properties.json 4ba2f84
Diff: https://reviews.apache.org/r/31713/diff/
Testing
-------
all tests passed.
updated only metainfoirmation and SQL queries
Thanks,
Dmytro Sen
Re: Review Request 31713: Query for FLUME_HANDLER metrics takes
several minutes
Posted by Sid Wagle <sw...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31713/#review75262
-----------------------------------------------------------
Ship it!
Ship It!
- Sid Wagle
On March 4, 2015, 11:25 p.m., Dmytro Sen wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/31713/
> -----------------------------------------------------------
>
> (Updated March 4, 2015, 11:25 p.m.)
>
>
> Review request for Ambari, Myroslav Papirkovskyy and Sid Wagle.
>
>
> Bugs: AMBARI-9914
> https://issues.apache.org/jira/browse/AMBARI-9914
>
>
> Repository: ambari
>
>
> Description
> -------
>
> API call:
> curl -u admin:admin "http://host1:8080/api/v1/clusters/cc/components/?ServiceComponentInfo/component_name=FLUME_HANDLER|ServiceComponentInfo/component_name=APP_TIMELINE_SERVER|ServiceComponentInfo/category=MASTER&fields=ServiceComponentInfo/Version,ServiceComponentInfo/StartTime,ServiceComponentInfo/HeapMemoryUsed,ServiceComponentInfo/HeapMemoryMax,ServiceComponentInfo/service_name,host_components/HostRoles/host_name,host_components/HostRoles/state,host_components/HostRoles/maintenance_state,host_components/HostRoles/stale_configs,host_components/HostRoles/ha_state,host_components/HostRoles/desired_admin_state,host_components/metrics/jvm/memHeapUsedM,host_components/metrics/jvm/HeapMemoryMax,host_components/metrics/jvm/HeapMemoryUsed,host_components/metrics/jvm/memHeapCommittedM,host_components/metrics/mapred/jobtracker/trackers_decommissioned,host_components/metrics/cpu/cpu_wio,host_components/metrics/rpc/RpcQueueTime_avg_time,host_components/metrics/dfs/FSNamesystem/*,host_compon
ents/metrics/dfs/namenode/Version,host_components/metrics/dfs/namenode/LiveNodes,host_components/metrics/dfs/namenode/DeadNodes,host_components/metrics/dfs/namenode/DecomNodes,host_components/metrics/dfs/namenode/TotalFiles,host_components/metrics/dfs/namenode/UpgradeFinalized,host_components/metrics/dfs/namenode/Safemode,host_components/metrics/runtime/StartTime,host_components/metrics/flume/flume,host_components/processes/HostComponentProcess,host_components/metrics/hbase/master/IsActiveMaster,ServiceComponentInfo/MasterStartTime,ServiceComponentInfo/MasterActiveTime,ServiceComponentInfo/AverageLoad,ServiceComponentInfo/Revision,ServiceComponentInfo/RegionsInTransition,metrics/api/v1/cluster/summary,metrics/api/v1/topology/summary,host_components/metrics/yarn/Queue,ServiceComponentInfo/rm_metrics/cluster/activeNMcount,ServiceComponentInfo/rm_metrics/cluster/lostNMcount,ServiceComponentInfo/rm_metrics/cluster/unhealthyNMcount,ServiceComponentInfo/rm_metrics/cluster/rebootedNMcount,
ServiceComponentInfo/rm_metrics/cluster/decommissionedNMcount&minimal_response=true&_=1425140480545"
> Takes several minutes on bigger clusters
>
>
> Diffs
> -----
>
> ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/PhoenixTransactSQL.java 80c6545
>
> Diff: https://reviews.apache.org/r/31713/diff/
>
>
> Testing
> -------
>
> all tests passed.
> updated only metainfoirmation and SQL queries
>
>
> Thanks,
>
> Dmytro Sen
>
>
Re: Review Request 31713: Query for FLUME_HANDLER metrics takes
several minutes
Posted by Dmytro Sen <ds...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31713/
-----------------------------------------------------------
(Updated Март 4, 2015, 11:25 п.п.)
Review request for Ambari, Myroslav Papirkovskyy and Sid Wagle.
Bugs: AMBARI-9914
https://issues.apache.org/jira/browse/AMBARI-9914
Repository: ambari
Description
-------
API call:
curl -u admin:admin "http://host1:8080/api/v1/clusters/cc/components/?ServiceComponentInfo/component_name=FLUME_HANDLER|ServiceComponentInfo/component_name=APP_TIMELINE_SERVER|ServiceComponentInfo/category=MASTER&fields=ServiceComponentInfo/Version,ServiceComponentInfo/StartTime,ServiceComponentInfo/HeapMemoryUsed,ServiceComponentInfo/HeapMemoryMax,ServiceComponentInfo/service_name,host_components/HostRoles/host_name,host_components/HostRoles/state,host_components/HostRoles/maintenance_state,host_components/HostRoles/stale_configs,host_components/HostRoles/ha_state,host_components/HostRoles/desired_admin_state,host_components/metrics/jvm/memHeapUsedM,host_components/metrics/jvm/HeapMemoryMax,host_components/metrics/jvm/HeapMemoryUsed,host_components/metrics/jvm/memHeapCommittedM,host_components/metrics/mapred/jobtracker/trackers_decommissioned,host_components/metrics/cpu/cpu_wio,host_components/metrics/rpc/RpcQueueTime_avg_time,host_components/metrics/dfs/FSNamesystem/*,host_componen
ts/metrics/dfs/namenode/Version,host_components/metrics/dfs/namenode/LiveNodes,host_components/metrics/dfs/namenode/DeadNodes,host_components/metrics/dfs/namenode/DecomNodes,host_components/metrics/dfs/namenode/TotalFiles,host_components/metrics/dfs/namenode/UpgradeFinalized,host_components/metrics/dfs/namenode/Safemode,host_components/metrics/runtime/StartTime,host_components/metrics/flume/flume,host_components/processes/HostComponentProcess,host_components/metrics/hbase/master/IsActiveMaster,ServiceComponentInfo/MasterStartTime,ServiceComponentInfo/MasterActiveTime,ServiceComponentInfo/AverageLoad,ServiceComponentInfo/Revision,ServiceComponentInfo/RegionsInTransition,metrics/api/v1/cluster/summary,metrics/api/v1/topology/summary,host_components/metrics/yarn/Queue,ServiceComponentInfo/rm_metrics/cluster/activeNMcount,ServiceComponentInfo/rm_metrics/cluster/lostNMcount,ServiceComponentInfo/rm_metrics/cluster/unhealthyNMcount,ServiceComponentInfo/rm_metrics/cluster/rebootedNMcount,Se
rviceComponentInfo/rm_metrics/cluster/decommissionedNMcount&minimal_response=true&_=1425140480545"
Takes several minutes on bigger clusters
Diffs (updated)
-----
ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/PhoenixTransactSQL.java 80c6545
Diff: https://reviews.apache.org/r/31713/diff/
Testing
-------
all tests passed.
updated only metainfoirmation and SQL queries
Thanks,
Dmytro Sen
Re: Review Request 31713: Query for FLUME_HANDLER metrics takes
several minutes
Posted by Sid Wagle <sw...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31713/#review75253
-----------------------------------------------------------
Ship it!
Ship It!
- Sid Wagle
On March 4, 2015, 10:22 p.m., Dmytro Sen wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/31713/
> -----------------------------------------------------------
>
> (Updated March 4, 2015, 10:22 p.m.)
>
>
> Review request for Ambari, Myroslav Papirkovskyy and Sid Wagle.
>
>
> Bugs: AMBARI-9914
> https://issues.apache.org/jira/browse/AMBARI-9914
>
>
> Repository: ambari
>
>
> Description
> -------
>
> API call:
> curl -u admin:admin "http://host1:8080/api/v1/clusters/cc/components/?ServiceComponentInfo/component_name=FLUME_HANDLER|ServiceComponentInfo/component_name=APP_TIMELINE_SERVER|ServiceComponentInfo/category=MASTER&fields=ServiceComponentInfo/Version,ServiceComponentInfo/StartTime,ServiceComponentInfo/HeapMemoryUsed,ServiceComponentInfo/HeapMemoryMax,ServiceComponentInfo/service_name,host_components/HostRoles/host_name,host_components/HostRoles/state,host_components/HostRoles/maintenance_state,host_components/HostRoles/stale_configs,host_components/HostRoles/ha_state,host_components/HostRoles/desired_admin_state,host_components/metrics/jvm/memHeapUsedM,host_components/metrics/jvm/HeapMemoryMax,host_components/metrics/jvm/HeapMemoryUsed,host_components/metrics/jvm/memHeapCommittedM,host_components/metrics/mapred/jobtracker/trackers_decommissioned,host_components/metrics/cpu/cpu_wio,host_components/metrics/rpc/RpcQueueTime_avg_time,host_components/metrics/dfs/FSNamesystem/*,host_compon
ents/metrics/dfs/namenode/Version,host_components/metrics/dfs/namenode/LiveNodes,host_components/metrics/dfs/namenode/DeadNodes,host_components/metrics/dfs/namenode/DecomNodes,host_components/metrics/dfs/namenode/TotalFiles,host_components/metrics/dfs/namenode/UpgradeFinalized,host_components/metrics/dfs/namenode/Safemode,host_components/metrics/runtime/StartTime,host_components/metrics/flume/flume,host_components/processes/HostComponentProcess,host_components/metrics/hbase/master/IsActiveMaster,ServiceComponentInfo/MasterStartTime,ServiceComponentInfo/MasterActiveTime,ServiceComponentInfo/AverageLoad,ServiceComponentInfo/Revision,ServiceComponentInfo/RegionsInTransition,metrics/api/v1/cluster/summary,metrics/api/v1/topology/summary,host_components/metrics/yarn/Queue,ServiceComponentInfo/rm_metrics/cluster/activeNMcount,ServiceComponentInfo/rm_metrics/cluster/lostNMcount,ServiceComponentInfo/rm_metrics/cluster/unhealthyNMcount,ServiceComponentInfo/rm_metrics/cluster/rebootedNMcount,
ServiceComponentInfo/rm_metrics/cluster/decommissionedNMcount&minimal_response=true&_=1425140480545"
> Takes several minutes on bigger clusters
>
>
> Diffs
> -----
>
> ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/PhoenixTransactSQL.java 80c6545
> ambari-server/src/main/resources/ganglia_properties.json 4ba2f84
>
> Diff: https://reviews.apache.org/r/31713/diff/
>
>
> Testing
> -------
>
> all tests passed.
> updated only metainfoirmation and SQL queries
>
>
> Thanks,
>
> Dmytro Sen
>
>
Re: Review Request 31713: Query for FLUME_HANDLER metrics takes
several minutes
Posted by Dmytro Sen <ds...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31713/
-----------------------------------------------------------
(Updated Март 4, 2015, 10:22 п.п.)
Review request for Ambari, Myroslav Papirkovskyy and Sid Wagle.
Bugs: AMBARI-9914
https://issues.apache.org/jira/browse/AMBARI-9914
Repository: ambari
Description
-------
API call:
curl -u admin:admin "http://host1:8080/api/v1/clusters/cc/components/?ServiceComponentInfo/component_name=FLUME_HANDLER|ServiceComponentInfo/component_name=APP_TIMELINE_SERVER|ServiceComponentInfo/category=MASTER&fields=ServiceComponentInfo/Version,ServiceComponentInfo/StartTime,ServiceComponentInfo/HeapMemoryUsed,ServiceComponentInfo/HeapMemoryMax,ServiceComponentInfo/service_name,host_components/HostRoles/host_name,host_components/HostRoles/state,host_components/HostRoles/maintenance_state,host_components/HostRoles/stale_configs,host_components/HostRoles/ha_state,host_components/HostRoles/desired_admin_state,host_components/metrics/jvm/memHeapUsedM,host_components/metrics/jvm/HeapMemoryMax,host_components/metrics/jvm/HeapMemoryUsed,host_components/metrics/jvm/memHeapCommittedM,host_components/metrics/mapred/jobtracker/trackers_decommissioned,host_components/metrics/cpu/cpu_wio,host_components/metrics/rpc/RpcQueueTime_avg_time,host_components/metrics/dfs/FSNamesystem/*,host_componen
ts/metrics/dfs/namenode/Version,host_components/metrics/dfs/namenode/LiveNodes,host_components/metrics/dfs/namenode/DeadNodes,host_components/metrics/dfs/namenode/DecomNodes,host_components/metrics/dfs/namenode/TotalFiles,host_components/metrics/dfs/namenode/UpgradeFinalized,host_components/metrics/dfs/namenode/Safemode,host_components/metrics/runtime/StartTime,host_components/metrics/flume/flume,host_components/processes/HostComponentProcess,host_components/metrics/hbase/master/IsActiveMaster,ServiceComponentInfo/MasterStartTime,ServiceComponentInfo/MasterActiveTime,ServiceComponentInfo/AverageLoad,ServiceComponentInfo/Revision,ServiceComponentInfo/RegionsInTransition,metrics/api/v1/cluster/summary,metrics/api/v1/topology/summary,host_components/metrics/yarn/Queue,ServiceComponentInfo/rm_metrics/cluster/activeNMcount,ServiceComponentInfo/rm_metrics/cluster/lostNMcount,ServiceComponentInfo/rm_metrics/cluster/unhealthyNMcount,ServiceComponentInfo/rm_metrics/cluster/rebootedNMcount,Se
rviceComponentInfo/rm_metrics/cluster/decommissionedNMcount&minimal_response=true&_=1425140480545"
Takes several minutes on bigger clusters
Diffs (updated)
-----
ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/PhoenixTransactSQL.java 80c6545
ambari-server/src/main/resources/ganglia_properties.json 4ba2f84
Diff: https://reviews.apache.org/r/31713/diff/
Testing
-------
all tests passed.
updated only metainfoirmation and SQL queries
Thanks,
Dmytro Sen
Re: Review Request 31713: Query for FLUME_HANDLER metrics takes
several minutes
Posted by Sid Wagle <sw...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31713/#review75125
-----------------------------------------------------------
ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/PhoenixTransactSQL.java
<https://reviews.apache.org/r/31713/#comment122075>
This is not sufficient for making sure that the Reverse scan is used. Note: The change needs to make sure that all the columns in the filter have the same order by predicate as SERVER_TIME. This means HBase does not have to scan one column in ASC and other in DSC, in which case it has to do a client side merge after range scan.
E.g.: SELECT * FROM METRIC_RECORD WHERE METRIC_NAME = 'cpu_idle' AND HOSTNAME = 'ambari-sid-3.c.pramod-thangali.internal' AND APP_ID = 'HOST' ORDER BY metric_name desc, hostname desc, SERVER_TIME DESC LIMIT 1;
Also, the implementation should figure out the columns to put in the WHERE clause based on what is the filter criteria.
- Sid Wagle
On March 4, 2015, 12:51 a.m., Dmytro Sen wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/31713/
> -----------------------------------------------------------
>
> (Updated March 4, 2015, 12:51 a.m.)
>
>
> Review request for Ambari, Myroslav Papirkovskyy and Sid Wagle.
>
>
> Bugs: AMBARI-9914
> https://issues.apache.org/jira/browse/AMBARI-9914
>
>
> Repository: ambari
>
>
> Description
> -------
>
> API call:
> curl -u admin:admin "http://host1:8080/api/v1/clusters/cc/components/?ServiceComponentInfo/component_name=FLUME_HANDLER|ServiceComponentInfo/component_name=APP_TIMELINE_SERVER|ServiceComponentInfo/category=MASTER&fields=ServiceComponentInfo/Version,ServiceComponentInfo/StartTime,ServiceComponentInfo/HeapMemoryUsed,ServiceComponentInfo/HeapMemoryMax,ServiceComponentInfo/service_name,host_components/HostRoles/host_name,host_components/HostRoles/state,host_components/HostRoles/maintenance_state,host_components/HostRoles/stale_configs,host_components/HostRoles/ha_state,host_components/HostRoles/desired_admin_state,host_components/metrics/jvm/memHeapUsedM,host_components/metrics/jvm/HeapMemoryMax,host_components/metrics/jvm/HeapMemoryUsed,host_components/metrics/jvm/memHeapCommittedM,host_components/metrics/mapred/jobtracker/trackers_decommissioned,host_components/metrics/cpu/cpu_wio,host_components/metrics/rpc/RpcQueueTime_avg_time,host_components/metrics/dfs/FSNamesystem/*,host_compon
ents/metrics/dfs/namenode/Version,host_components/metrics/dfs/namenode/LiveNodes,host_components/metrics/dfs/namenode/DeadNodes,host_components/metrics/dfs/namenode/DecomNodes,host_components/metrics/dfs/namenode/TotalFiles,host_components/metrics/dfs/namenode/UpgradeFinalized,host_components/metrics/dfs/namenode/Safemode,host_components/metrics/runtime/StartTime,host_components/metrics/flume/flume,host_components/processes/HostComponentProcess,host_components/metrics/hbase/master/IsActiveMaster,ServiceComponentInfo/MasterStartTime,ServiceComponentInfo/MasterActiveTime,ServiceComponentInfo/AverageLoad,ServiceComponentInfo/Revision,ServiceComponentInfo/RegionsInTransition,metrics/api/v1/cluster/summary,metrics/api/v1/topology/summary,host_components/metrics/yarn/Queue,ServiceComponentInfo/rm_metrics/cluster/activeNMcount,ServiceComponentInfo/rm_metrics/cluster/lostNMcount,ServiceComponentInfo/rm_metrics/cluster/unhealthyNMcount,ServiceComponentInfo/rm_metrics/cluster/rebootedNMcount,
ServiceComponentInfo/rm_metrics/cluster/decommissionedNMcount&minimal_response=true&_=1425140480545"
> Takes several minutes on bigger clusters
>
>
> Diffs
> -----
>
> ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/PhoenixTransactSQL.java 80c6545
> ambari-server/src/main/resources/ganglia_properties.json 4ba2f84
>
> Diff: https://reviews.apache.org/r/31713/diff/
>
>
> Testing
> -------
>
> all tests passed.
> updated only metainfoirmation and SQL queries
>
>
> Thanks,
>
> Dmytro Sen
>
>