[jira] [Comment Edited] (SPARK-20953) Add hash map metrics to aggregate and join

["Liang-Chi Hsieh (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-20953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16042419#comment-16042419 ] 

Liang-Chi Hsieh edited comment on SPARK-20953 at 6/8/17 8:50 AM:
-----------------------------------------------------------------

[~rxin] Are we just going to log an error for too great avg probe, or we also want to add avg probe to two operators' sql metrics for showing on UI?


was (Author: viirya):
[~rxin] Are we just going to log an error for too great avg probe, or we also want to add avg probe to two operators' sql metrics?

> Add hash map metrics to aggregate and join
> ------------------------------------------
>
>                 Key: SPARK-20953
>                 URL: https://issues.apache.org/jira/browse/SPARK-20953
>             Project: Spark
>          Issue Type: New Feature
>          Components: SQL
>    Affects Versions: 2.2.0
>            Reporter: Reynold Xin
>
> It would be useful if we can identify hash map collision issues early on.
> We should add avg hash map probe metric to aggregate operator and hash join operator and report them. If the avg probe is greater than a specific (configurable) threshold, we should log an error at runtime.
> The primary classes to look at are UnsafeFixedWidthAggregationMap, HashAggregateExec, HashedRelation, HashJoin.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org