You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "ASF subversion and git services (Jira)" <ji...@apache.org> on 2019/09/25 22:31:00 UTC
[jira] [Commented] (IMPALA-7637) Include more hash table stats in
profile
[ https://issues.apache.org/jira/browse/IMPALA-7637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16938109#comment-16938109 ]
ASF subversion and git services commented on IMPALA-7637:
---------------------------------------------------------
Commit 803323bb39aee3a3bd38e7cca544cef72500b8b6 in impala's branch refs/heads/master from Yongzhi Chen
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=803323b ]
IMPALA-7637: Add more hash table stats to profile
Add hash table counters(probes, travel and resizes) to profile.
Put hash table stats into the child profile "hash table".
Tests:
Add new test test_query_profle_hashtable.
Ran exhaustive tests.
Profile Sample:
Hash Join Builder (join_node_id=2):
...
Runtime filters: 1 of 1 Runtime Filter Published
- BuildRowsPartitionTime: 157.960us
- BuildRowsPartitioned: 100 (100)
- HashTablesBuildTime: 298.817us
- LargestPartitionPercent: 7 (7)
- MaxPartitionLevel: 0 (0)
- NumRepartitions: 0 (0)
- PartitionsCreated: 16 (16)
- PeakMemoryUsage: 17.12 KB (17536)
- RepartitionTime: 0.000ns
- SpilledPartitions: 0 (0)
Hash Table:
- HashBuckets: 256 (256)
- HashCollisions: 0 (0)
- Probes: 2.52K (2520)
- Resizes: 0 (0)
- Travel: 1.79K (178
Change-Id: I1fd875dd1af8031242fd5f5ff554d3a71aaa6f87
Reviewed-on: http://gerrit.cloudera.org:8080/14234
Reviewed-by: Sahil Takiar <st...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
> Include more hash table stats in profile
> ----------------------------------------
>
> Key: IMPALA-7637
> URL: https://issues.apache.org/jira/browse/IMPALA-7637
> Project: IMPALA
> Issue Type: Improvement
> Components: Backend
> Reporter: Tim Armstrong
> Assignee: Yongzhi Chen
> Priority: Major
> Labels: observability, ramp-up
>
> Our hash table collects some useful stats about collisions and travel length, but then we don't do anything to expose them: https://github.com/apache/impala/blob/540611e863fe99b3d3ae35f8b94a745a68b9eba2/be/src/exec/hash-table.h#L989
> We should add some of them to the profile, maybe:
> * the number of probes
> * the average travel length per probe
> * the number of hash collisions
> * (optional) the number of hash table resizes. We already have the hash table size and the resize time, which I think is sufficient to debug most problems with resizes.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org