You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "ASF subversion and git services (JIRA)" <ji...@apache.org> on 2019/06/25 22:05:00 UTC
[jira] [Commented] (IMPALA-7322) Add storage wait time to profile
for operations with metadata load
[ https://issues.apache.org/jira/browse/IMPALA-7322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16872759#comment-16872759 ]
ASF subversion and git services commented on IMPALA-7322:
---------------------------------------------------------
Commit 2fd795cf56e65a43087375867dcc9890e3a27330 in impala's branch refs/heads/master from Yongzhi Chen
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=2fd795c ]
IMPALA-7322: Add storage wait time to profile
Add metrics to record storage wait time for operations with
metadata load in catalog for hdfs, kudu and hbase tables.
Pass storage wait time from catalog to fe through thrift and log
total storage load time in query profile.
Storage-load-time is the amount of time spent loading
metadata from the underlying storage layer (e.g. S3, HDFS,
Kudu, HBase), which does not include the amount of time
spending loading data from HMS.
Testing:
Ran queries that can trigger all of, none of or some of the related
tables loading. Check query profile for each query. Check catalog
metrics for each table.
Add unit tests to test_observability.py
Ran all core tests.
Sample output:
Profile:(storage-load-time is the added property):
After ran a hbase query (Metadata load finished is divided into
several lines because of limitation of commit message):
Query Compilation: 4s401ms
- Metadata load started: 661.084us (661.084us)
- Metadata load finished. loaded-tables=1/1
load-requests=1 catalog-updates=3
storage-load-time=233ms: 3s819ms (3s819ms)
- Analysis finished: 3s820ms (763.979us)
- Value transfer graph computed: 3s820ms (63.193us)
Catalog metrics(this sample is from a hdfs table):
storage-metadata-load-duration:
Count: 1
Mean rate: 0.0085
1 min. rate: 0.032
5 min. rate: 0.1386
15 min. rate: 0.177
Min (msec): 111
Max (msec): 111
Mean (msec): 111.1802
Median (msec): 111.1802
75th-% (msec): 111.1802
95th-% (msec): 111.1802
99th-% (msec): 111.1802
Change-Id: I6dde7e394b7c1c396d835ef6aa0a55930c0a8660
Reviewed-on: http://gerrit.cloudera.org:8080/12940
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
> Add storage wait time to profile for operations with metadata load
> ------------------------------------------------------------------
>
> Key: IMPALA-7322
> URL: https://issues.apache.org/jira/browse/IMPALA-7322
> Project: IMPALA
> Issue Type: Improvement
> Components: Frontend
> Affects Versions: Impala 3.0, Impala 2.12.0
> Reporter: Balazs Jeszenszky
> Assignee: Yongzhi Chen
> Priority: Major
>
> The profile of a REFRESH or of the query triggering metadata load should point out how much time was spent waiting for source systems.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org